SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 3: Cross-validated POS accuracy on PPCEME using no function tags, 10 function tags, or 31 function tags. Standard deviations are in parentheses. The rows and columns are analogous to those in Table 2.
tag	frequency	f1
N	11.85 (0.70)	97.21 (0.84)
P	11.59 (0.45)	99.29 (0.19)
D	7.58 (0.94)	99.71 (0.10)
PRO	7.01 (1.04)	99.72 (0.17)
,	6.71 (0.93)	99.72 (0.14)
CONJ	5.20 (0.47)	99.55 (0.22)
.	4.42 (0.38)	99.69 (0.10)
ADJ	4.34 (0.45)	95.95 (0.85)
NS	3.48 (0.40)	97.31 (0.99)
ADV	3.05 (0.27)	97.11 (0.58)
NPR	2.98 (1.07)	93.05 (4.37)
VB	2.94 (0.33)	98.68 (0.55)
PRO$	2.57 (0.43)	99.60 (0.13)
VBD	1.95 (0.78)	97.56 (0.95)
MD	1.77 (0.25)	99.43 (0.59)
Q	1.75 (0.06)	99.00 (0.24)
VAN	1.71 (0.16)	96.31 (0.90)
VBP	1.65 (0.38)	96.65 (0.72)
BEP	1.62 (0.48)	99.56 (0.31)
TO	1.50 (0.12)	99.67 (0.09)
C	1.21 (0.16)	98.81 (0.53)
WPRO	0.95 (0.12)	99.36 (0.40)
BED	0.89 (0.22)	99.75 (0.22)
NUM	0.89 (0.48)	97.47 (1.15)
VAG	0.82 (0.13)	96.41 (0.65)
NEG	0.76 (0.13)	99.77 (0.12)
BE	0.68 (0.10)	99.46 (0.29)
VBN	0.60 (0.16)	96.38 (0.98)
HVP	0.50 (0.08)	99.25 (0.94)
ADVR	0.50 (0.07)	97.41 (0.68)
RP	0.49 (0.12)	96.12 (1.48)
FW	0.45 (0.24)	87.14 (10.44)
VBI	0.41 (0.09)	92.79 (2.58)
HVD	0.35 (0.13)	99.41 (1.12)
WADV	0.31 (0.04)	97.31 (1.22)
Table 4: Cross-validated per-tag F1 of the ELMo (EEBO) configuration for the 35 most frequent POS tags on the PPCEME test section (using the 31 function tag set). The frequency column indicates the mean relative frequency of each tag in the test set. Standard deviations are in parentheses.
lute  $(t = 5.68, p < 1\mathrm{e} - 3)$  on the test set.
As the distribution of POS tags is highly unbalanced, we also break down the  $98.30\%$  score in Table 3 for (elmo-e, ftags-31, test) and report F1 by tag for the 35 most frequent POS tags (i.e., tags with a frequency  $\geq 0.30\%$ ) in the PPCEME test set (Table  $4^{15}$ ). While there is some variation among the tags, F1 is consistently high ( $>95\%$ ) with the exception of the imperative (VBI), proper noun (NPR) $^{16}$ , and foreign word (FW). The imperative
15For descriptions of PPCEME POS tags in the table, see https://www.ling.upenn.edu/hist-corpora/ annotation/terms.htm.
16The NPR score of  $93.05\%$  is likely due to known annotation inconsistencies in the corpus between N (common noun) and NPR. This also impacts the N score, which at  $97.21\%$  does lower our overall POS score since it is the most common tag.
tag is of particular concern for us as we anticipate future users of the parsed EEBO will rely heavily on it for retrieval of imperative clauses (Kulick et al., 2022). We therefore provide two examples of parser confusion between VBI and VBP (present).
8.1 Example 1
In this first example, tree (a) is the gold tree, while (b) is the parser output, which has both the wrong POS tag for stryue (strive) and the wrong function tag for the parent IP.  $\star \mathrm{con}\star$  in the gold tree is an elided subject under conjunction, and in this case refers to a subject in the previous tree. Generating the correct parse requires, as it would for a human, referring to the preceding text. In isolation, the parse with VBI is perfectly reasonable, although incorrect in context.
Another aspect of this error concerns the common NLP practice, which we follow, of removing empty categories from the parser training material (Appendix A.5). In this case, the removal of  $\star \mathrm{con}\star$  has the consequence that the training data has ordinary (i.e. non-imperative) sentences without an overt subject and a seemingly arbitrary assignment of VBP or VBI to verbs.
This example therefore shows both the need to refer to text beyond the sentence level, and how the training data is corrupted by the removal of empty categories.
8.2 Example 2
This second example shows the reverse error,
in which the gold POS tag is VBI for lay, while the parser makes it a VBP. The reason is that it incorrectly parses for Mice, and makes Mice the NP-SBJ laying the Poyson. Tagging lay as a VBP is consistent with that mistake.
9 Out-of-vocabulary span labels and resource inefficiency
As shown in Table 2, for each configuration (row) there is little effect on the F1 parsing scores among ftags-0, ftags-10, and ftags-31, a result that is consistent with Gabbard et al. (2006) for the PTB. However, including the function tags significantly affects the label vocabulary, as we discuss before going on to the analysis of the function tag performance in Section 10.
As discussed in Section 6.1, the parser uses a classifier to produce the most likely label for a span. The parser determines the label vocabulary at training time, by collecting all the possible nonterminal labels. There are two complications in this determination of the label vocabulary.
First, for unary branches, the parser combines the different nonterminals into a single label. For example, for the structure (NP (CP . . .)), which is a NP with a single child, a CP, the label is NP::CP. At parse time, if this label is the most likely for a span, then it is split up again into a parent NP and child CP node.
Second, since function tags are integrated into the label vocabulary, they multiply out over the cases with unary branching. For example, with the structure (NP (CP-FRL . . .)), for a NP with a free relative CP child, the value in the label vocabulary will be NP::CP-FRL. But there can also be instances (NP-SBJ (CP-FRL . . .)), (NP-OB1 (CP-FRL . . .)), resulting in the entries NP-SBJ::CP-FRL and NP-OB1::CP-FRL in the label vocabulary.
Table 5 depicts the impact of these two factors on the label vocabulary. Even without any function tags, the collapsing of unary branches causes the label vocabulary to increase from an average of 87.62 to an average of 334.25. With all 31 function tags, this increase is even more pronounced, with the label vocabulary increasing from 250.00 to 937.88.
The label vocabulary used by the model is the "collapsed" value for the train section. This increase of 687.88 resource requirements during training, and we found that the training time could increase
section	ftags-0	ftags-31
unary branches not collapsed
train	87.62 (1.51)	250.00 (2.27)
test	46.12 (2.17)	143.38 (6.05)
both	45.25 (1.39)	140.75 (5.76)
unary branches collapsed
train	334.25 (5.73)	937.88 (12.64)
test	117.50 (6.59)	299.88 (17.93)
both	111.12 (5.64)	278.62 (14.99)
as much as  $25\%$  between trains of ftags-0 and ftags-31.
Much of the increased resource requirements is not required, though. The "test" rows show the corresponding numbers for the eight test sections, and the "both" row shows the label values that are in both the train and test sections. Even without function tags, an average of only 111.12 of the labels in the training section occur in the test section, while the test section has an average of 6.38 labels that are not in the training section (117.50 - 111.12). The inefficiency worsens considerably with more function tags. For ftags-31, an average of only 278.62 of the 937.88 labels in the training section occur in the test section, and the test section has an average of 21.26 labels that are not in the training section (299.88 - 278.62).
If a label is in the test section but not in the label vocabulary determined by the train section, the parser will not correctly predict spans with that label. They are in effect out-of-vocabulary span labels. While the parser score is not damaged more, due to the infrequency of these OOV labels, this shows the limits of the current approach. We consider these current results to be a baseline for future improvement.
10 Function tag analysis
Table 6 shows the score for the function tags over the different configurations, with either 10 or 31 function tags. The overall score for the function tags drops drastically between ftags-10 and ftags-31. For example, on the test sections, with ELMo EEBO it falls from a mean of  $97.90\%$  with ftags-10 to a mean of  $95.55\%$  for ftags-31.
Table 5: Span label vocabulary size along two dimensions: (1) no function tags vs. all 31 tags, and (2) labels are collapsed/not collapsed on unary branches. The "train" and "test" rows indicate the span label vocabulary sizes for those sections along those dimensions. The "both" row shows the number of labels that are common to the train and test section vocabularies. All sizes are mean and standard deviation over the 8 splits.
config	ftags-10	ftags-31
dev
elmo-e	97.94 (0.51)	94.90 (1.54)
bert	97.79 (0.60)	94.40 (1.42)
elmo-o	97.13 (0.66)	93.56 (1.69)
char	97.01 (0.69)	93.44 (1.60)
test
elmo-e	97.90 (0.28)	95.55 (0.87)
bert	97.72 (0.27)	94.95 (0.94)
elmo-o	96.97 (0.28)	94.24 (0.81)
char	96.88 (0.35)	94.22 (0.82)
Table 6: Function tag scores for the different configurations, showing the mean and standard deviation over the 8 splits. The function tag score uses a comparison of the function tags for brackets that count as a match for evalb. There is no function tag score for ftags-0 since there are no function tags in that version of the training and evaluation data.
To explore these results in more detail, we pick one cell in the table – ELMo EEBO with 31 tags, for the test sections – and look at the individual scores for the 31 function tags that make up the  $95.55\%$  result, as shown in Table 7. The tags are organized into 6 groups for expository convenience. The “syntactic” and “semantic” tags are roughly similar to those groups for the PTB, as presented in Gabbard et al. (2006). The other groups include tags that are very different than in the PTB, as mentioned in Section 3.2, such as the tags restricted to CP nodes and indicating properties of the CP clause. The asterisks in Table 6 indicate the ten tags included in ftags-10. The third column shows the frequency of each tag or group of tags among all the function tags being scored in the test section, and the F1 column shows the score for the tag or group of tags.
Examining the results for the tags included in ftags-31 but not in ftags-10, the main cause for the drop in score is the SPE tag for direct speech, which is one of the most common tags, with an average  $3.91\%$  frequency, but with an average score of  $46.08\%$  F1.[17] The PPCEME lacks the consistent clues for direct speech (such as quotation marks) that would be available in newswire text adhering to modern punctuation guidelines.
For example, the sentence and he shall be welcome has an IP-MAT-SPE (matrix IP, direct speech) as the root of the entire tree in gold annotation, while in the parser output it is IP-MAT.
17The somewhat greater drop in score for the dev section splits, from  $97.94\%$  to  $94.90\%$  in Table 6, is likely because in the dev sections the SPE tag performs just as badly, but is more common (6.79% frequency).
tag	description	frequency	f1
Syntactic	39.69 (1.55)	97.41 (0.32)
*SBJ	subject	23.02 (1.09)	98.71 (0.14)
*ACC	accusative	12.42 (0.79)	96.34 (0.62)
*DTV	dative	1.48 (0.36)	92.96 (2.70)
*VOC	vocative	0.69 (0.42)	93.88 (3.27)
MSR	measure	1.13 (0.24)	92.75 (1.28)
POS	possessive	0.72 (0.23)	97.97 (0.91)
SPR	sec pred	0.24 (0.06)	76.33 (5.07)
Semantic	7.72 (0.44)	95.09 (0.73)
TMP	temporal	2.91 (0.42)	93.54 (1.32)
ADV	adverbial	3.64 (0.41)	97.25 (0.42)
LOC	locative	0.70 (0.11)	91.89 (1.84)
DIR	directional	0.48 (0.09)	92.75 (1.56)
CP only	8.67 (0.60)	94.70 (0.69)
REL	rel clause	3.41 (0.51)	95.30 (0.90)
THT	THAT clause	2.47 (0.27)	97.60 (0.63)
*QUE	question	1.24 (0.53)	95.97 (0.42)
CAR	clause-adj	0.59 (0.15)	79.98 (2.81)
CMP	comparative	0.57 (0.14)	95.80 (0.66)
FRL	free relative	0.29 (0.06)	80.72 (3.78)
EOP	empty op	0.09 (0.02)	94.26 (3.66)
IP only	9.28 (0.95)	96.21 (0.46)
*INF	infinitival	4.46 (0.39)	98.97 (0.29)
PPL	participial	2.01 (0.44)	98.44 (0.72)
*IMP	imperative	1.26 (0.25)	93.50 (1.59)
SMC	small clause	0.73 (0.16)	97.74 (0.58)
PRP	purpose	0.45 (0.07)	71.76 (3.58)
ABS	absolute	0.37 (0.19)	86.07 (3.61)
CP or IP	31.78 (2.69)	94.22 (2.59)
*MAT	matrix	12.93 (0.70)	98.58 (0.29)
*SUB	subordinate	14.64 (0.96)	98.97 (0.22)
SPE	direct speech	3.91 (3.58)	46.08 (17.76)
DEG	degree	0.30 (0.05)	88.62 (2.35)
null	Misc	2.85 (0.32)	86.57 (2.73)
*PRN	parenthetical	2.17 (0.21)	90.00 (2.10)
RSP	resumptive	0.37 (0.08)	70.33 (7.98)
LFD	left-disl	0.31 (0.07)	79.37 (4.87)
Total	100.00 (0.00)	95.55 (0.87)
Table 7: Function tag breakdown. The third column shows the frequency of the tag in the test section, while the fourth column shows its F1 score. The tags are organized into six groups and we also indicate the combined frequency and F1 score for the tags of each group.
The parser (or for that matter, a person) cannot know that the sentence is direct speech without looking at the wider context, which is a conversation spanning a few sentences. The underlying problem here is related to that of the first example of a POS error in Section 8. Future work to recover the direct speech information will need to examine discourse contexts beyond just single sentences in isolation, as the parser does now.
The other scores show some variance in accuracy, although some of the most common tags (SBJ, SUB, MAT) have the highest scores. Considering the tags as groups, there are some drops in scores among some of the tags for CP or IP constituents (e.g. purpose clauses). These require more anal
ysis, although in general the question of whether the function tag recovery is satisfactory can only be answered with reference to the larger aim of this work (that is, by determining if the tags are accurate enough to ensure accurate results from subsequent linguistic searches on automatically parsed material).
11 Conclusion and Future Work
In this work we have presented the first parsing results on the PPCEME, with a focus on challenges posed by its extensive set of function tags. Adapting an earlier approach to function tag recovery works reasonably well, while we point out some challenges for future work concerning improvements in accuracy and resource efficiency.
We have also demonstrated the continued importance of in-domain pretraining, as the parser configuration using ELMo trained on in-domain EEBO outperforms the configuration using BERT trained on modern English. We performed cross-validation to ensure that the results were robust and not an accident of a particular split.
Future work will proceed along a number of lines. Regarding the function tags, an obvious approach is to integrate a separate function tag classifier into the parser separate from one for the bare nonterminals and to combine them to produce the full nonterminal labels. We will also evaluate the function tags by their utility for linguistic searches on automatically parsed material, as mentioned at the end of Section 10.
This work was supported by NSF grant BCS 13-046668. We thank the three anonymous reviewers for their comments. We would also like to acknowledge here the tremendous debt, intellectual and in other ways, that this work owes to the late Professor Anthony Kroch.
Dogu Araci. 2019. FinBERT: Financial sentiment analysis with pre-trained language models. arXiv preprint arXiv:1908.10063.
Iz Beltagy, Arman Cohan, and Kyle Lo. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Don Blaheta. 2003. Function Tagging. Ph.D. thesis, Brown University.
Don Blaheta and Eugene Charniak. 2000. Assigning function tags to parsed text. In 1st Meeting of the North American Chapter of the Association for Computational Linguistics.
Shu Cai, David Chiang, and Yoav Goldberg. 2011. Language-independent parsing with empty elements. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 212-216, Portland, Oregon, USA. Association for Computational Linguistics.
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898-2904, Online. Association for Computational Linguistics.
Yufei Chen, Yuanyuan Zhao, Weiwei Sun, and Xiaojun Wan. 2018. Pre- and in-parsing models for neural empty category detection. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2687-2696, Melbourne, Australia. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171-4186.
Aaron Ecay. 2015. A multi-step analysis of the evolution of English do-support. Ph.D. thesis, University of Pennsylvania.
Ryan Gabbard, Seth Kulick, and Mitchell Marcus. 2006. Fully parsing the Penn Treebank. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 184-191.
Charlotte Galves. 2020. Relaxed Verb Second in Classical Portuguese. In Rebecca Woods and Sam Wolfe, editors, Rethinking Verb-Second, pages 368-395. Oxford University Press.
Charlotte Galves, Aroldo Leal de Andrade, and Pablo Faria. 2017. Tycho Brahe Parsing Corpus of Historical Portuguese. http://www.tycho.iel.unicamp.br/~tycho/corpus/texts/psd.zip.
Kyle Gorman and Steven Bedrick. 2019. We need to talk about standard splits. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2786-2791.
Qiao Jin, Bhuwan Dhingra, William W Cohen, and Xinghua Lu. 2019. Probing biomedical embeddings from language models. arXiv preprint arXiv:1904.02181.
Mark Johnson. 2002. A simple pattern-matching algorithm for recovering empty nodes and their antecedents. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 136-143, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Andres Karjus. 2020. Competition, selection and communicative need in language change: An investigation using corpora, computational modelling and experimentation. Ph.D. thesis, University of Edinburgh.
Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 3499–3505.
Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2676-2686.
Anthony Kroch. 1989. Reflexes of grammar in patterns of language change. Language Variation and Change, 1(3):199-244.
Anthony Kroch. 2020. Penn Parsing Corpora of Historical English (LDC2020T16). Web Download. Philadelphia: Linguistic Data Consortium.
Anthony Kroch and Beatrice Santorini. 2021. PennBFM Parsing Corpus of Historical French, version 1.0. https://github.com/beatrice57/mcvf-plus-ppchf.
Anthony Kroch, Beatrice Santorini, and Lauren Delfs. 2004. Penn-Helsinki Parsing Corpus of Early Modern English (PPCEME). CD-ROM, first edition, release 3. http://www.ling.upenn.edu/ppche/ppche-release-2016/PPCEME-RELEASE-3.
Anthony Kroch, Beatrice Santorini, and Ariel Diertani. 2016. Penn Parsing Corpus of Modern British English (PPCMBE2). CD-ROM, second edition, release 1. http://www.ling.upenn.edu/ppche/ppche-release-2016/PPCMBE2-RELEASE-1.
Anthony Kroch, Ann Taylor, and Donald Ringe. 2000a. The Middle English verb-second constraint: A case study in language contact and language change. In Susan Herring, Lene Schoessler, and Peter van Reenen, editors, Textual parameters in older language, pages 353-391. Benjamins.
Anthony Kroch, Ann Taylor, and Beatrice Santorini. 2000b. Penn-Helsinki Parsing Corpus of Middle English (ppcme2). CD-ROM, second edition, release 4.
Seth Kulick, Anthony Kroch, and Beatrice Santorini. 2014. The Penn Parsing Corpus of Modern British English: First parsing results and analysis. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 662-667.
Seth Kulick, Neville Ryant, and Beatrice Santorini. 2022. Parsing Early Modern English for linguistic search. In Proceedings of the Society for Computation in Linguistics 2022, pages 143-157, online. Association for Computational Linguistics.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: a pretrained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
France Martineau, Paul Hirschbühler, Anthony Kroch, and Yves Charles Morin. 2021. MCVF Corpus, parsed, version 2.0. https://github.com/beatrice57/mcvf-plus-ppchf.
Paola Merlo and Gabriele Musillo. 2005. Accurate function parsing. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 620-627. Association for Computational Linguistics.
Taesun Moon and Jason Baldridge. 2007. Part-of-speech tagging for Middle English through alignment and projection of parallel diachronic texts. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 390-399.
Khalil Mrini, Franck Dernoncourt, Quan Hung Tran, Trung Bui, Walter Chang, and Ndapa Nakashole. 2020. Rethinking self-attention: Towards interpretability in neural parsing. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 731–742.
Sebastian Nagel. 2016. Cc-news. https://commoncrawl.org/2016/10/news-dataset-available/.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 2227-2237.