SlowGuess/NAACL_2022_OCR huggingface dataset:
Seth Kulick and Neville Ryant
Linguistic Data Consortium
{skulick, nryant}@ldc.upenn.edu
Beatrice Santorini
Linguistics Dept.
beatrice@sas.upenn.edu
The Penn-Helsinki Parsing Corpus of Early Modern English (PPCEME), a 1.7-million-word treebank that is an important resource for research in syntactic change, has several properties that present potential challenges for NLP technologies. We describe these key features of PPCEME that make it challenging for parsing, including a larger and more varied set of function tags than in the Penn Treebank, and present results for this corpus using a modified version of the Berkeley Neural Parser and the approach to function tag recovery of Gabbard et al. (2006). While this approach to function tag recovery gives reasonable results, it is in some ways inappropriate for span-based parsers. We also present further evidence of the importance of in-domain pretraining for contextualized word representations. The resulting parser will be used to parse Early English Books Online, a 1.5 billion word corpus whose utility for the study of syntactic change will be greatly increased with the addition of accurate parse trees.
The Penn-Helsinki Parsing Corpus of Early Modern English (PPCEME) (Kroch et al., 2004) consists of over 1.7 million words of text from 1500 to 1712, manually annotated for phrase structure. It belongs to a family of treebanks of historical English (Taylor et al., 2003; Kroch et al., 2000b; Taylor et al., 2006; Kroch et al., 2016) and other languages (Wallenberg et al., 2011; Galves et al., 2017; Kroch and Santorini, 2021; Martineau et al., 2021) with a shared annotation philosophy and similar guidelines across languages, which form the basis for reproducible studies of syntactic change (Kroch et al., 2000a; Ecay, 2015; Wallenberg, 2016; Galves, 2020; Wallenberg et al., 2021).
PPCEME is of interest for NLP researchers as well as linguists since it differs from the Penn Treebank (PTB) (Marcus et al., 1993) along several
dimensions, and it is not obvious that parsing technologies mostly based on the PTB can transfer over successfully. While the text in the PTB is rigidly standardized, subject to editorial guidelines, the source text in PPCEME is more chaotic, with a lack of standardization of spelling and punctuation. The annotation style is similar to that of the PTB, but it has more finely-grained Part of Speech (POS) tags, some common and important structures (e.g., coordination) are annotated differently than in the PTB, and perhaps most significantly, it makes far greater use of function tags. There has however been relatively little work in the NLP community (Moon and Baldridge, 2007; Kulick et al., 2014; Yang and Eisenstein, 2016) using PPCEME or its sister corpora, the Penn Parsing Corpus of Middle English, 2nd edition (PPCME2) and the Penn Parsing Corpus of Modern British English (PPCMBE) $^{1}$ .
In this paper, we present first results for both parsing accuracy and function tag recovery for PPCEME, pointing out some limitations for the latter. Our parser is a slightly modified version of the self-attentive neural constituency parser introduced by Kitaev and Klein (2018). While constituent-parsing technology has seen notable gains in recent years, to our knowledge there has no work on recovering function tags within this framework, which is crucial for the larger context of this work, discussed below in this introduction. We use an approach to function tag recovery used earlier with the PTB (Gabbard et al., 2006) and show that while it gives reasonable results, it is inefficient and inappropriate for contemporary span-based constituency parsers, especially as the number of function tags increases.
Many of our decisions are informed by the larger context of this work: parsing the Early English Books Online (EEBO) corpus (Text Creation Partnership, 2019). While PPCEME has been used for building models of syntactic change, its usefulness for this work is limited by the fact that various
1 All three corpora are collected in Kroch (2020)
Findings of the Association for Computational Linguistics: NAACL 2022, pages 578 - 593
phenomena of interest are too sparse within it to support reliable statistical modeling of language change (e.g. distinguishing between competition (Kroch, 1989) and drift (Karjus, 2020)). In contrast, EEBO covers the same time period, 1475 to 1700, with 1.5 billion words of text – a difference of three orders of magnitude. However, EEBO's potential as a resource for linguistic research remains unrealized because it is not linguistically annotated and its size renders manual annotation infeasible.
Our goal therefore is the creation of a parsed version of EEBO consistent with the structures in PPCEME. This parsed version of EEBO will be used for linguistic research on language change by searching for structures of interest based on the phrase structure and POS tags. The parsing model trained on PPCEME will be used to parse EEBO so that it too can be used for queries for structures of linguistic interest, although at a much larger scale than previously possible. For more details, we refer to Kulick et al. (2022).
This broader concern impacts our design choices for the parser; for instance, our decision to work on constituency parsing rather than dependency parsing – linguistic queries depend on phrase structure trees. Likewise, since the function tags and POS tags are used by the linguistic queries, it critical that the parser be able to generate this information. It also informed the development of our tokenizer for EEBO. While in this paper we use standard metrics for parser evaluation, we also refer to Kulick et al. (2022) for a discussion of an alternative method of evaluation based on precision and recall for the specific queries.
The PPCEME is one of several corpora designed for historical linguistics research that share annotation styles and design decisions. Others include the second edition of the Penn-Helsinki Parsing Corpus of Middle English (Kroch et al., 2000b) (PPCME2) and the Penn Parsing Corpus of Modern British English (Kroch et al., 2016) (PPCMBE). While there has been much linguistics research based on these corpora, there is relatively little previous NLP work using them. Kulick et al. (2014) described parsing PPCMBE, while Moon and Baldridge (2007) and Yang and Eisenstein (2016) focused on POS-tagging, the former on PPCME2, and the latter on PPCEME and PPCMBE.
This previous work has discussed characteristics
of these corpora that differ from that of the PTB and the potential impact on models trained on these corpora. A focus has been on how to transform the annotation to be closer to that of the PTB, such as by transforming the phrase structure (Kulick et al., 2014) or by mapping the PPCME2 POS tag set into that of PTB (Moon and Baldridge, 2007; Yang and Eisenstein, 2016).
3 Major Differences between PPCEME and PTB
As mentioned in the introduction, the PPCEME annotation style differs from that of PTB in certain respects. We describe here two of the most significant differences in more detail as well as how they influenced our preprocessing decisions. We also give examples in Section 3.3 of the spelling variations present in the Early Modern English data.
3.1 PPCEME Complex Part-of-Speech Tags
The PPCEME POS tag set is much larger ( $n = 353$ ) than that typically used in parsing work (e.g., for PTB  $n = 36$ ). Consequently, it requires trimming to a more computationally tractable size. Of the 353 tags, 213 are complex tags intended to facilitate tracking changes in orthographic conventions over time - for instance, the development of (ADJ gentle) (NS men) to (ADJ+NS gentlemen). Since these changes are irrelevant for present purposes, we prune such tags in accordance with the Righthand Head Rule, yielding (NS gentlemen).<sup>2</sup> Certain rare cases, such as (WPRO+ADV+ADV whatsoever) or (Q+BEP+PRO albeit), are exceptions to the Righthand Head Rule. In such cases, the best simple tag is sometimes the leftmost tag and sometimes another tag entirely ((WPRO whatsoever), (P albeit)). We simply ignore this complication on the grounds that these cases are a small subset of the complex tags, which themselves are used for only about  $1\%$  of the words in the corpus. After pruning and some other minor changes discussed in Appendix A, 85 POS tags remain.
3.2 Function Tags
In phrase-structure treebanks, function tags can be appended to syntactic category labels in order to provide information about a constituent's grammatical or semantic role. The PTB uses 20 function
2Yang and Eisenstein (2016) simplify the complex tags for the same reason as we do, but keep the leftmost tag, which for English is incorrect in the general case.
tags in this way, while distinguishing other constituent roles by means of structural differences (e.g., adjoining a relative clause but not a complement clause) By contrast, PPCEME relies on function tags uniformly, largely because it has neither base NPs or VPs. As a result, PPCEME's set of function tags is larger than PTB's. Omitting a few rare types, we consider 31 function tags in the work reported below. The following tree illustrates PPCEME's use of function tags to encode central grammatical roles. The subject and indirect object are sisters, but distinguished by the function tags SBJ and OB2, respectively. MAT and SUB on the two IPs identify the higher one as a matrix clause and the lower one as a subordinate clause. Finally, THT indicates that the CP is a that complement clause (rather than, say, a relative or adverbial clause).
There has been some work on recovering function tags in PTB (Blaheta and Charniak, 2000; Blaheta, 2003; Gabbard et al., 2006; Merlo and Musillo, 2005), but overall this topic has received only limited attention. We are not aware of any work to recover the function tags in the historical corpora. As discussed in the introduction, our larger goal requires including them in the parsing model.
For the remainder of this paper, we refer to the version of PPCEME using these 31 function tags as "ftags-31". We also report results for a version using no function tags at all ("ftags-0") and for a version using ten of the tags that were of particular importance for some of the linguistic searches ("ftags-10"). For a detailed description of the tags see Section 10.
3.3 Standardization
As mentioned in the introduction, the source texts of Early Modern English text (both in PPCEME and EEBO) can have spelling variations of a sort that are not present in corpora such as the PTB. Table 1 shows some examples of these variants in the source text.
standard	variants
sheriff	sheriffe, sherife, sherif, schereffé*
showed	shew’d, shewed*, shewd, show’d, shevved
servant*	servant, seruaunt
strive	striue, stryue*
knocked	knockt, knock’d, knokyd*
asked	ask’d, askt, ask’t, askyd*
devil	divel, devill, divell, diuell, deuell, diuel, deuil
against	agaynst, againste, agaynste
Table 1: Some of the variant spellings present in the Early Modern English source text. Forms marked with an asterisk are used in example sentences in this paper.
In addition, there is also some inconsistency of punctuation in the source material, which interacts with annotation decisions concerning sentence segmentation. A period POS tag marks sentence-final punctuation and a comma POS tag marks sentence-internal punctuation, though these do not always match the actual token receiving that tag.
For example, consider the following text for one sentence/tree in PPCEME:
And when he cam | knokyd at the gatys. To whome anone one of the gentylmannys seruaintys askyd who was there I
The first | is annotated with a comma POS tag, and the second | is annotated with a period POS tag. The period receives a comma POS tag, since it is sentence-internal.5
4 Cross-validation Splits
Recently, concerns have been raised over the validity of inferences drawn from static train/dev/test splits of a corpus; for instance, see Gorman and Bedrick (2019), who evaluate the consistency of rankings of POS taggers across 20 random splits of the WSJ section of PTB. For us, this issue is particularly pressing as PPCEME contains relatively few individual source texts, thus increasing the chance that a single particularly difficult or non-representative source text will greatly skew performance on the dev/test partitions.
We therefore define an 8-fold cross-validation split, with each component split roughly matching the  $90\% -5\% -5\%$  distribution in the standard single PTB split. Within each partition (train, dev, test) of a split, we attempted to equally represent (in terms of equal word counts) each of PPCEME's three
In terms of the syntactic annotation, when he cam is a PP preceding the main verb knokyd, and To whome...there is a CP-relative clause.
3See Appendix A for the details, along with some information on function tag frequency.
4E.g. NP-SBJ becomes NP.
70-year time periods, as indicated by “e1”, “e2”, and “e3” in the filenames. Given our eventual goal of parsing all of EEBO, which encompasses all of these time periods, this step is necessary in order to adequately predict performance on that corpus.<sup>6</sup> Finally, in cases where PPCEME distributes a single source text over several annotated files, we were careful to assign all such files to the same partition. As PPCEME contains 448 annotated files, but only 232 distinct source texts, this greatly constrained how we could define the partitions. Nevertheless, we succeeded in including 209 of the 232 source texts in either a dev or test partition of one of the 8 splits. For more details on the split definitions, see Appendix B.
5 ELMo Embeddings Trained on EEBO
In recent years, contextualized word embeddings such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) have driven significant improvements on downstream NLP tasks, including POS tagging and parsing. Due to the significant overhead involved in training these representations, researchers often use pretrained models distributed by large companies, sometimes fine-tuned to the domain of interest. Although this often produces perfectly satisfactory results, in cases of significant mismatch between a test domain and standard training domains - usually sources such as text scraped from Wikipedia, BooksCorpus (Zhu et al., 2015), news text from Common Crawl (Nagel, 2016), and discussion forums (Radford et al., 2019) - pretraining on the novel domain yields significant improvements (Lee et al., 2020; Beltagy et al., 2019; Jin et al., 2019).
Because of the orthographic and syntactic differences between Early Modern English and contemporary English mentioned in Section 3, our current work involves exactly such a mismatch, and so we pretrained ELMo embeddings on EEBO. We used the same model configuration as Peters et al. (2018) for 11 epochs. Pretraining was performed using the TensorFlow implementation maintained by AllenLNP using the default model configuration. We
then integrated the resulting embeddings, which have 1,024 dimensions, into the parser model (see Section 6). Further details regarding the pretraining may be found in Appendix C.
6 Model and evaluation
6.1 Parser Architecture
We use the parsing model of Kitaev et al. (2019), which represents a constituency tree  $T$  as a set of labeled spans  $(i,j,l)$  where  $i$  and  $j$  are the beginning and ending positions of the span and  $l$  its label. Each tree is assigned a score  $s(T)$ , which is decomposed as a sum of per-span scores:
The per-span scores  $s(i,j,l)$  are themselves assigned using a neural network that takes a sequence of per-word embeddings as input, processes these embeddings using a transformer-based encoder (Vaswani et al., 2017), and then produces a span score from an MLP classifier (Stern et al., 2017). The highest-scoring valid tree is found using a variant of the CKY algorithm. POS tags are recovered using a separate classifier operating on top of the encoder output, which is jointly optimized with the span classifier. For more details, see Kitaev and Klein (2018).
We consider four approaches for producing the word embeddings that serve as input to the encoder: bert-base Contextual word embeddings are computed using BERT (Devlin et al., 2019) by retaining the output of the last layer for the final subword unit of each token. We use the bert-base model distributed via the transformers Python package<sup>10</sup>, which is pre-trained on 3.3 billion words of modern English.
elmo (orig) As above, but using ELMo (Peters et al., 2018). We use the ELMo-original model distributed by AllenNLP<sup>11</sup>, which was pre-trained on 1 billion words of modern English.
elmo (eebo) ELMo embeddings pre-trained on Early Modern English from EEBO. Uses the same model architecture as elmo (orig).
char-lstm Tokens are represented as the concatenation of the outputs of a bidirectional character-level LSTM for the final character of the token.
10https://github.com/huggingface/ transformers
11https://allennlp.org/elmo
<sup>6</sup>By contrast, Yang and Eisenstein (2016), split PPCEME into thirds by time period (rather than across time periods) for the different purpose of studying domain adaptation.
At present, we lack the computational resources for the obvious next step of pretraining BERT embeddings on EEBO, but we are pursuing access to them.
This corresponds to 2 weeks of training using 4 GTX 1080 GPUs.
9https://github.com/allenai/bilm-tf
All four of these embeddings types are character or subword based, which is particularly helpful for the PPCEME corpus due to the spelling variations. The subword units for BERT however are derived from its modern English training material.
Our implementation is based on version 0.2.0 of the Berkeley Neural Parser $^{12}$  with modifications to allow the use of ELMo. $^{13}$ . We train each of the 8 models (one for each cross-validation split) for 50 epochs, using the evalb (Sekine and Collins, 2008) score on the dev section as our criterion for saving the best model. For additional details regarding training and hyperparameters, see Appendix D.
6.2 Function Tags
We adopt the approach of Gabbard et al. (2006) for function tag recovery, in which the parser preprocessing step simply does not delete the function tags, and so nonterminals such as NP-SBJ are treated as an atomic unit. Since the decision whether to delete is part of the preprocessing, this approach does not require modifying the parser. As mentioned in Section 3.2, our preprocessing code for PPCEME creates three versions of the files, ftags-0, ftags-10, and ftags-31.
6.3 Evaluation metrics
During training, the parser uses the standard evalb on the dev set to determine the best model so far. evalb compares matching brackets between the gold and system output trees. However, since our models also predict POS tags, and evalb removes punctuation based on POS tags, inconsistent sentence lengths can arise if the gold and parsed trees have differing POS tags, resulting in "Error" sentences in the evalb output. For the scores reported in Section 7, we therefore use the modified evalb supplied with the Berkeley parser, originating from Seddah et al. (2014), which does not delete any words, so the differences in POS tags do not affect the sentence length.
Function tags are typically removed by evalb for the comparison of bracket labels, and we have not modified this. To evaluate the function tag recovery, we use the metric in Gabbard et al. (2006) which in turn follows Blaheta (2003). This evaluation compares function tags only for nonterminals that are counted as matches for evalb. For example, if a
config	ftags-0	ftags-10	ftags-31
dev
elmo-e	90.92 (1.86)	90.95 (1.79)	90.89 (1.83)
bert	90.37 (1.95)	90.37 (1.95)	90.37 (1.98)
elmo-o	88.06 (2.23)	88.13 (2.25)	88.09 (2.22)
char	87.28 (2.34)	87.43 (2.33)	87.33 (2.33)
test
elmo-e	90.56 (0.66)	90.62 (0.71)	90.53 (0.69)
bert	89.81 (0.76)	89.83 (0.80)	89.86 (0.77)
elmo-o	87.31 (0.85)	87.50 (0.84)	87.38 (0.81)
char	86.29 (0.93)	86.50 (0.96)	86.45 (0.90)
Table 2: Cross-validated F1 for the parser on PPCEME using no function tags, 10 function tags, or 31 function tags. The standard deviation for F1 is presented in parentheses. Scores are obtained using the version of evalb that does not delete punctuation, for reasons discussed in Section 6.3, and do not consider function tags in the matching of brackets. bert: bert-base, elmo-e: ELMo pretrained on EEBO, elmo-o: original ELMo embeddings, char: character LSTM
NP-SBJ node in the gold tree matches with a NP-SBJ node in the parsed tree, it is a match for SBJ, while if a NP-SBJ node in the gold tree matches with a NP-OB1 node in the parsed tree (which can happen since the function tags do not count for evalb), it is a recall error for SBJ and a precision error for OB1.[14]
POS evaluation is the standard tag accuracy as output by evalb.
7 Pretraining comparison experiments
Table 2 presents parsing results for the dev/test sections of the 8 cross-validation splits described in Section 4. The rows are the four embedding representations described in Section 6.1 and the columns are the F1 scores (evalb bracket scores, as discussed in Section 6.3) for each of the three versions (ftags-0, ftags-10, ftags-31) of the training and evaluation data. Each cell shows the mean and standard deviation for that embedding representation and function tag version over the 8 splits. As there is no great precision/recall imbalance, we relegate the corresponding precision and recall numbers to a more complete version of this table in Appendix E.
For each of the three versions of the training and evaluation data, the best F1 scores are obtained using ELMo (EEBO), followed by BERT, ELMo (orig), and, char-LSTM. Moreover, ELMo (EEBO) outperforms BERT (to varying degrees) on every single combination of split/function tags. In particular, for the version with 31 function tags, which
14We combined this function tag evaluation with our own Python reimplementation of evalb.
$^{12}$ https://github.com/nikitakit/  
self-attentive-parser  
 $^{13}$ These modifications and other relevant software are avail-  
able at https://github.com/skulick/emeparse.
config	ftags-0	ftags-10	ftags-31
dev
elmo-e	98.14 (0.72)	98.17 (0.72)	98.14 (0.69)
bert	97.75 (0.78)	97.73 (0.83)	97.71 (0.89)
elmo-o	96.87 (0.90)	96.93 (0.92)	96.91 (0.95)
char	97.17 (0.82)	97.25 (0.84)	97.23 (0.79)
test
elmo-e	98.29 (0.36)	98.30 (0.36)	98.30 (0.36)
bert	97.95 (0.35)	97.94 (0.36)	97.97 (0.35)
elmo-o	97.09 (0.45)	97.14 (0.43)	97.16 (0.46)
char	97.23 (0.41)	97.30 (0.42)	97.28 (0.45)
will be the focus of further analysis in the following sections, ELMo (EEBO) outperforms BERT by  $0.67\%$  absolute  $(t = 6.05, p < 1e - 3)$  on the test set.
The fact that BERT is consistently outperformed by ELMo (EEBO) is particularly interesting given that (a) BERT in general outperforms ELMo when trained on similar material, (b) BERT in this case was trained on three times as much material as ELMo, and (c) the BERT model parameters were fine-tuned, while the ELMo parameters were not. This result underlines the importance of pretraining on in-domain materials, a fact that has also been observed for NLP tasks in biomedical (Lee et al., 2020), financial (Araci, 2019), and legal (Chalkidis et al., 2020) domains.
Overall, we note that the scores are a few points lower than the current state of the art for PTB (96.38% F1 on the test section (Mrini et al., 2020)). As Kulick et al. (2014) point out, all of the English historical corpora lack certain brackets present in PTB (base-NPs and VPs) that are relatively "easy to get", and this tends to adversely affect their parsing scores. Specifically, they find that it impacts the F1 score for PPCMBE by about  $2.5\%$  absolute, an effect we expect to carry over to PPCEME.
8 Part-of-speech Results
While the evalb scores are of greater importance, POS accuracy also matters as the linguistic queries we plan to run on the eventual parsed EEBO sometimes refer explicitly to POS tags. We therefore consider the POS results in detail. As is apparent from Table 3, POS accuracy is highest for ELMo (EEBO) for all combinations of function tags and dev/test set, followed by BERT, then ELMo (orig); e.g., when using 31 function tags, ELMo (EEBO) outperforms BERT by  $0.33\%$  abso