SlowGuess/NAACL_2022_OCR huggingface dataset:
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Djame Seddah, Sandra Kübler, and Reut Tsarfaty. 2014. Introducing the SPMRL 2014 shared task on parsing morphologically-rich languages. In Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Canonical Languages, pages 103-109.
Satoshi Sekine and Michael Collins. 2008. evalb. http://nlp.cs.nyu.edu/evalb/.
Mitchell Stern, Jacob Andreas, and Dan Klein. 2017. A minimal span-based neural constituency parser. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistic (ACL), pages 818-827.
Ann Taylor, Arja Nurmi, Anthony Warner, Susan Pintzuk, and Terttu Nevalainen. 2006. Parsing Corpus of Early English Correspondence. Distributed through the Oxford Text Archive.
Ann Taylor, Anthony Warner, Susan Pintzuk, and Frans Beths. 2003. York-Toronto-Helsinki Parsing Corpus of Old English Prose. Distributed by the Oxford Text archive.
Text Creation Partnership. 2019. Early English Books Online. https://textcreationpartnership.org/tcp-texts/eebo-tcp-early-english-books-online/.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS), volume 30.
Joel C. Wallenberg. 2016. Extrapolation is disappearing. Language, 92(4):e237-e256.
Joel C. Wallenberg, Rachael Bailes, Christine Cuskley, and Anton Karl Ingason. 2021. Smooth signals and syntactic change. Languages, 6(2):60.
Joel C. Wallenberg, Anton Karl Ingason, Einar Freyr Sigurdhsson, and Eirikur Rögnvaldsson. 2011. Icelandic Parsing Historical Corpus (IcePaHC), v0.9. http://www.linguist.is/icelandic_treebank.
Yi Yang and Jacob Eisenstein. 2016. Part-of-speech tagging for historical English. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1318-1328.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies
and reading books. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 19-27.
A PPCEME preprocessing
A.1 Metadata
In addition to the changes described in Section 3.2, we removed the metadata under CODE, META, and REF nodes. In cases where CODE dominated a leaf, removing the leaf resulted in an ill-formed tree. The 267 trees in question were removed, as were 576 trees rooted in META (usually stage directions for a play) and 9 trees containing BREAK.
In addition, before carrying out the above modifications, we changed all instances of (CODE <paren>) and CODE <$\$\text{paren}>) to (OPAREN -LRB-) and (CPAREN -RRB-), respectively. We did this in order to retain the parentheses that otherwise, being daughters of CODE, would have been deleted.
Our counts of number of words and sentences differ slightly from Yang and Eisenstein (2016). This is probably related to small differences of preparation of the type just discussed.
A.2 PPCEME Part-of-Speech Modifications
In addition to the changes described in the main text, we changed the tag MD0 to MD. MD0 is an untensd modal, as in he will can or to can do something. There are only 4 cases, as this is an option that had mostly died out by the time of Early Modern English.
There are also cases where words that are ordinarily spelled as a single orthographic token are sometimes split into several tokens. PPCEME represents the former case with a single POS tag and the latter as a constituent whose non-terminal is the POS tag, with the words given numbered segmented POS tags - for example, (ADJ alone) vs. (ADJ (ADJ21 a) (ADJ22 lone)). We modified all such tags by removing the numbers, and appending _NT to the nonterminals, in order to more clearly distinguish between POS tags and nonterminals. In this example, the resulting structure would be (ADJ_NT (ADJ a) (ADJ lone)).
A.3 Distinctions among Verb Classes
PTB makes no distinction between main verbs and the auxiliary verbs  $be$ ,  $do$  and  $have$ , but this distinction is vital for us, since it is exactly the syntax of main (but not auxiliary) verbs that changes over the course of Early Modern English. In fact, even among the verbs with auxiliary uses, we need to distinguish  $do$  from the other auxiliaries in order
to track the rise of auxiliary  $do$ . For this reason, we do not follow Yang and Eisenstein (2016) in mapping the PPCEME tags for verbs to the smaller set used in PTB.
A.4 Function Tags
We exclude certain tags that occur very rarely in PPCEME (CLF, COM, TMC, RFL, ADT, EXL, YYY, ELAB, XXX, TAG, and TPC). Table 8 shows the frequency for each of the remaining 31 tags in the entire corpus for nonterminals with a non-empty yield. For convenience, the tags are organized into six groups. The syntactic and semantic groups are roughly similar to those groups for the PTB, as presented in Gabbard et al. (2006). The other groups include tags that differ significantly from those in the PTB, as noted in Section 3.2. For the full set of PPCEME function tags, see https://www.ling.upenn.edu/hist-corpora/annotation/terms. htm.
A.5 Empty categories
PPCEME indicates discontinuous dependencies by means of empty categories that are coindexed with a displaced constituent. Following common NLP practice, we remove both the empty categories and the co-indexing from the parser training material, and thus from the parser output. This simplifies the parsing model, and for present purposes, the absence of empty categories is irrelevant. However, if we wish to include linguistic queries in future work that make reference to empty categories, as is necessary in the general case, the parsing model will need to be augmented appropriately (Johnson, 2002; Cai et al., 2011; Chen et al., 2018).
B Cross-validation Splits
Table 9 summarizes the composition of the train/dev/test sections across the cross-validation 8 splits; specifically, the total number of documents, the total number of tokens, and the percentage of total tokens in each section. Since the partitioning process is performed at the level of PPCEME source files, and these files differ substantially in size, there is some variation in these numbers across the splits. For this reason, we report standard deviations as well as means. The final row ("OVERALL") depicts numbers for a complete split (i.e., the train/dev/test sections combined); as this is constant across each split, the entries of this
tag	description	frequency
Syntactic	37.23
SBJ	subject	21.00
OB1	direct object	11.96
OB2	indirect object	1.20
SPR	secondary predicate	0.28
MSR	measure	1.17
POS	possessive	0.86
VOC	vocative	0.77
Semantic	7.93
DIR	directional	0.50
LOC	locative	0.84
TMP	temporal	3.09
ADV	adverbial	3.50
CP only	8.83
CAR	clause-adjoined	0.55
REL	relative clause	3.36
THT	THAT clause	2.52
CMP	comparative	0.53
QUE	question	1.35
FRL	free relative	0.33
EOP	empty operator	0.19
IP only	9.67
INF	infinitive	4.59
PPL	participial	2.18
IMP	imperative	1.12
SMC	small clause	0.90
PRP	purpose	0.46
ABS	absolute	0.42
CP or IP	33.10
SUB	subordinate	14.52
MAT	matrix	12.66
SPE	direct speech	5.64
DEG	degree	0.28
Miscellaneous	3.23
PRN	parenthesesetical	2.60
RSP	resumptive	0.33
LFD	left-dislocated	0.30
Table 8: Relative frequencies of the 31 retained function tags in PPCEME. The tags are organized into 6 groups, with combined frequency by group in boldface.
row have a standard deviation of zero. As can be seen, overall the splits have kept to target 90-5-5 breakdown; e.g., the train section on average comprises  $89.65\%$  of the total tokens with a standard deviation of  $0.54\%$ .
As mentioned in the main text, the corpus consists of text from three main time periods (e1, e2, e3) $^{18}$  and we aimed to balance the time periods equally within each split, to the extent possible given that we treated the files as atomic units. Table 10 shows the breakdown by period. Similar to Table 9, mean/standard deviation for total number of documents/tokens are presented for each time period in each section. Additionally, for each time
period, it reports the mean percentage of each split (in tokens) from each time period. The marginals provide numbers combining across time periods (the "ALL PERIODS" row) and sections (the "ENTIRE SPLIT" column). For example, the training section contains on average 1,743,211.25 tokens, with on average  $32.85\%$  coming from time period e1,  $36.61\%$  from e2, and  $30.53\%$  from e3.
C ELMo Pretraining on EEBO
C.1 Text Extraction
EEBO's XML files contain a great deal of metadata and markup in addition to the source text. For each file, we extracted the core source information (title, author, date) and kept the text within  $<\mathbb{P}>$  tags, which gives at least a rough sense of the document divisions.
Following Ecay (2015, pp. 105-6), we excluded some metadata and other material embedded in the text. Information under NOTE, SPEAKER, and GAP elements were eliminated, as was information under the L ("line of verse") element, which was considered irrelevant for the linguistic searches envisioned for the final resource<sup>19</sup>. We also adopting his handling of GAP tags, which indicate the locations of OCR errors, which consists of mapping OCR errors to word-internal bullet characters - e.g., Eccl·siasticall.
C.2 Normalization
The extracted text underwent Unicode normalization to NFC form in order to eliminate spurious surface differences between tokens. The resulting text contained 642 unique characters, 381 of which occurred fewer than 200 times. Manual inspection of these uncommon characters revealed that while some of these made sense in context (e.g., within sections of Greek or Latin text), many seemed to be spurious characters due to OCR errors (e.g., WHITE RECTANGLE 0x25AD). Consequently, we elected to filter out all sentences containing characters occurring fewer than 200 times. This eliminated 4139 lines, with 9,341,966 remaining for training (consisting of 1,168,749,620 tokens).[20]
18For details regarding the PPCEME time periods (e1, e2 and e3) see https://www.ling.upenn.edu/hist-corpora/PPCEME-RELEASE-3/description.html
In future work, we will likely revise this to keep the text but with some meta-tags to indicate its origin.
20This token count differs from that cited for EEBO in the main body of the paper (1.5 billion) due to improvements in preprocessing over the course of the work.
section	# files	# tokens	% of split
train	205.88	(13.34)	1743211.25	(10441.53)	89.65	(0.54)
dev	12.50	(7.15)	101000.12	(4081.82)	5.19	(0.21)
test	13.62	(7.91)	100268.62	(7832.66)	5.16	(0.40)
OVERALL	232	(0.00)	1944480	(0.00)	100	(0.00)
Table 9: Mean number of files and tokens for train/dev/test sections across the 8 cross-validation splits (standard deviations are presented in parentheses). The percentage of tokens in each section is also presented (in the % of split column).
period	train section	dev section	test section	ENTIRE SPLIT
# files	# tokens	% train	# files	# tokens	% dev	# files	# tokens	% test	# files	# tokens	% split
e1	72.88(6.51)	572672.62(11974.31)	32.85(0.79)	4.25(3.01)	33178.50(7078.50)	32.98(7.36)	4.88(4.55)	31369.88(8193.65)	31.50(8.67)	82(0.00)	637221(0.00)	32.77(0.00)
e2	66.00(4.38)	638269.88(13490.18)	36.61(0.60)	4.00(2.51)	34844.62(6382.81)	34.40(5.41)	4.00(2.14)	35186.50(7767.44)	34.89(6.18)	74(0.00)	708301(0.00)	36.43(0.00)
e3	67.00(5.18)	532268.75(7066.41)	30.53(0.35)	4.25(3.96)	32977.00(5211.71)	32.63(4.65)	4.75(3.45)	33712.25(5592.81)	33.60(4.70)	76(0.00)	598958(0.00)	30.80(0.00)
ALLPERIODS	205.88(13.34)	1743211.25(10441.53)	100(0.00)	12.50(7.15)	101000.12(4081.82)	100(0.00)	13.62(7.91)	100268.62(7832.66)	100(0.00)	232(0.00)	1944480(0.00)	100(0.00)
Table 10: Mean number of files and tokens for train/dev/test sections within each of three time periods (e1, e2, and e3) across the 8 cross-validation splits. The  $\%$  train/dev/test columns indicate the  $\%$  of total train/dev/test tokens for each time period. Standard deviations are presented in parentheses.
C.3 Tokenization
After normalizing the extracted text into Unicode NFC form in order to eliminate spurious surface differences between tokens, we tokenized the EEBO text in accordance with PPCEME's tokenization guidelines as best we could:
1. Possessive morphemes are not separated from their host (e.g., Queen 's) (unlike in PTB).
2. Punctuation is separated except in the case of abbreviations (e.g., Mr.), token-internal hyphens (e.g., Fitz-Morris), or certain special cases (e.g., &c).
3. Roman numerals can include leading, internal, or trailing periods (e.g., .xiiii.C.).
PPCEME tokenization is straightforward in principle, but the non-standardized nature of the historical material raises various difficulties. For instance, it is easy to tell that the elided article th' should be split off (e.g., th'exchaung is tokenized as th' exchaung). But when the apostrophe is missing, the status of th is unclear (e.g., thafternoone is tokenized as th afternoonone, but thynkyth remains a single token). Another example of pervasive ambiguity is its and it's; in PPCEME, these forms were tokenized manually as one token or two, depending on whether the spelling represents the possessive form of the pronoun it or the contracted form of it is. Since EEBO's size rules out manual processing,
we resolved such ambiguities by defaulting to the more common case. In the above examples, this resulted in splitting the variants with apostrophes and not splitting the ones without.[21]
D Parser training
D.1 Parser hyperparameters
Table 11 shows the hyperparameter settings used in the Berkeley Neural Parser. These are all the default settings for these parameters. We added a parameter max_epochs, used to set the maximum number of epochs. For the cross-validation training reported, we set max_epochs = 50.
The total number of parameters in each model is presented in Table 12. As the label vocabulary size depends both on the function tag set used (e.g., ftags-0 vs ftags-31) and the split the model was trained on as, model sizes due differ subtly from run-to-run (on the order of a few thousand parameters).
D.2 Training times
Mean training times in hours for each configuration are presented in Table 13. Each training run was performed using a single NVIDIA GTX 1080 GPU.
E Pretraining comparison experiments
Table 14 expands Table 2 to include recall and precision in addition to F1.
21Future work could consider a joint tokenization-POSTaging model.
hyperparameter	value
attention_dropout	0.2
batch_size	32
char_lstm_input_dropout	0.2
checks_per_epoch	4
clip_grad_norm	0.0
d_char_emb	64
d_ff	2048
d_kv	64
d_label Hidden	256
d_model	1024
d_tagHidden	256
elmo_dropout	0.5
encoder_max_len	512
force_root Constituent	‘auto’
learning_rate	5e-05
learning_rate_warmup_steps	160
max_consecutive_decays	3
max_len_dev	0
max_len_train	0
morpho_emb_dropout	0.2
num_heads	8
num_layers	8
predict_labels	True
relu_dropout	0.1
residual_dropout	0.2
step Decay_factor	0.5
step Decay_patience	5
tag_loss_scale	5.0
max_epochs	50
Table 11: Hyperparameters used with the Berkeley Neural Parser.
config	ftags-0	ftags-10	ftags-31
bert	136M (1.47K)	136M (2.24K)	136M (3.25K)
elmo-e	27M (1.47K)	27M (2.24K)	27M (3.25K)
elmo-o	27M (1.47K)	27M (2.24K)	27M (3.25K)
char	27M (1.47K)	27M (2.24K)	27M (3.25K)
Table 12: Mean model sizes for each configuration using no function tags, 10 function tags, or 31 function tags. Standard deviations are in parentheses.
config	ftags-0	ftags-10	ftags-31
bert	24.38 (0.52)	24.88 (0.83)	30.75 (1.04)
elmo-e	26.00 (0.93)	31.50 (1.20)	31.75 (1.67)
elmo-o	26.50 (1.07)	30.75 (1.16)	31.50 (1.31)
char	13.12 (0.64)	15.62 (0.74)	16.50 (0.76)
Table 13: Mean training times (in hours) for each configuration using no function tags, 10 function tags, or 31 function tags. Standard deviations are in parentheses.
config	ftags-0	ftags-10	ftags-31
dev
elmo-e (f1)	90.92(1.86)	90.95(1.79)	90.89(1.83)
(rec)	90.57(1.96)	90.64(1.91)	90.49(1.93)
(prec)	91.27(1.76)	91.26(1.69)	91.30(1.74)
bert (f1)	90.37(1.95)	90.37(1.95)	90.37(1.98)
(rec)	90.09(2.05)	90.02(2.06)	90.00(2.08)
(prec)	90.65(1.84)	90.72(1.84)	90.74(1.87)
elmo-o (f1)	88.06(2.23)	88.13(2.25)	88.09(2.22)
(rec)	87.47(2.35)	87.51(2.45)	87.38(2.37)
(prec)	88.66(2.12)	88.76(2.06)	88.81(2.07)
char (f1)	87.28(2.34)	87.43(2.33)	87.33(2.33)
(rec)	86.80(2.45)	86.86(2.49)	86.70(2.46)
(prec)	87.76(2.24)	88.02(2.18)	87.98(2.21)
test
elmo-e (f1)	90.56(0.66)	90.62(0.71)	90.53(0.69)
(rec)	90.23(0.73)	90.31(0.80)	90.13(0.76)
(prec)	90.88(0.60)	90.93(0.64)	90.92(0.62)
bert (f1)	89.81(0.76)	89.83(0.80)	89.86(0.77)
(rec)	89.51(0.85)	89.46(0.87)	89.44(0.86)
(prec)	90.11(0.67)	90.20(0.74)	90.29(0.68)
elmo-o (f1)	87.31(0.85)	87.50(0.84)	87.38(0.81)
(rec)	86.71(0.92)	86.92(0.92)	86.69(0.90)
(prec)	87.91(0.79)	88.10(0.80)	88.08(0.73)
char (f1)	86.29(0.93)	86.50(0.96)	86.45(0.90)
(rec)	85.83(1.11)	85.89(1.06)	85.79(1.05)
(prec)	86.77(0.76)	87.12(0.88)	87.12(0.78)
Table 14: Cross-validation mean and standard deviation F1, recall, and precision parsing scores for each of the three versions of PPCEME, with no function tags, 10 function tags, or 31 function tags. "bert" is bert-base, "elmo-e" is elmo trained on EEBO, "elmo-o" is elmo with the original embeddings, and "char" is char-lstm. The F1 scores are the same as in Table 2.