SlowGuess/NAACL_2022_OCR huggingface dataset:
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proc. of NAACL-HLT, pages 4129-4138, Minneapolis, MN, USA.
Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: the  $90\%$  solution. In Proc. of HLT-NAACL, pages 57-60, New York City, USA.
Angelina Ivanova, Stephan Oepen, Lilja Øvrelid, and Dan Flickinger. 2012. Who did what to whom? a contrastive study of syntacto-semantic dependencies. In Proc. of LAW, pages 2-11, Jeju, Republic of Korea. Association for Computational Linguistics.
Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019. Probing what different NLP tasks teach machines about function word comprehension. In *Proc. of *SEM*, pages 235-249, Minneapolis, Minnesota. Association for Computational Linguistics.
Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In Proc. of ICLR.
Alexander Koller, Stephan Oepen, and Weiwei Sun. 2019. Graph-based meaning representations: Design and processing. In Proc. of ACL: Tutorial Abstracts, pages 6-11, Florence, Italy. Association for Computational Linguistics.
Artur Kulmizev, Vinit Ravishankar, Mostafa Abdou, and Joakim Nivre. 2020. Do neural language models show preferences for syntactic formalisms? In Proc. of ACL, pages 4077-4091, Online. Association for Computational Linguistics.
Ilia Kuznetsov and Iryna Gurevych. 2020. A matter of framing: The impact of linguistic formalism on probing results. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 171-182, Online. Association for Computational Linguistics.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. TACL, 4:521-535.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proc. of NAACL-HLT, pages 1073-1094, Minneapolis, Minnesota.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proc. of ICLR, New Orleans, LA, USA.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
William Merrill, Yoav Goldberg, Roy Schwartz, and Noah A. Smith. 2021. Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand? TACL, 9:1047-1060.
Piotr Mirowski and Andreas Vlachos. 2015. Dependency recurrent neural language models for sentence completion. In Proc. of ACL-IJCNLP, pages 511-517, Beijing, China. Association for Computational Linguistics.
Stefan Müller. 2020. Grammatical theory: From transformational grammar to constraint-based approaches. Language Science Press.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: a multilingual treebank collection. In Proc. of LREC, pages 1659-1666, Portoroz, Slovenia.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajic, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proc. of LREC, pages 4034-4043, Marseille, France. European Language Resources Association.
Stephan Oepen, Omri Abend, Lasha Abzianidze, Johan Bos, Jan Hajic, Daniel Hershcovich, Bin Li, Tim O'Gorman, Nianwen Xue, and Daniel Zeman. 2020. MRP 2020: The second shared task on cross-framework and cross-lingual meaning representation parsing. In Proc. of MRP at CoNLL, pages 1-22, Online. Association for Computational Linguistics.
Stephan Oepen, Omri Abend, Jan Hajic, Daniel Herschovich, Marco Kuhlmann, Tim O'Gorman, Nianwen Xue, Jayeol Chun, Milan Straka, and Zdenka Uresova. 2019. MRP 2019: Cross-framework meaning representation parsing. In Proc. of MRP at CoNLL, pages 1-27, Hong Kong. Association for Computational Linguistics.
Stephan Oepen and Jan Tore Lönning. 2006. Discriminant-based MRS banking. In Proc. of LREC, Genoa, Italy. European Language Resources Association (ELRA).
Adam Pauls and Dan Klein. 2012. Large-scale syntactic language modeling with treelets. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 959-968, Jeju Island, Korea. Association for Computational Linguistics.
Hao Peng, Roy Schwartz, and Noah A. Smith. 2019. PaLM: A hybrid parser and language model. In Proc. of EMNLP-IJCNLP, pages 3644-3651, Hong Kong, China. Association for Computational Linguistics.
Jakob Prange, Nathan Schneider, and Omri Abend. 2019a. Made for each other: Broad-coverage semantic structures meet preposition supersenses. In Proc. of CoNLL, pages 174–185, Hong Kong, China. Association for Computational Linguistics.
Jakob Prange, Nathan Schneider, and Omri Abend. 2019b. Semantically constrained multilayer annotation: the case of coreference. In Proc. of DMR, pages 164-176, Florence, Italy.
Peng Qian, Tahira Naseem, Roger Levy, and Ramón Fernández Astudillo. 2021. Structural guidance for transformer language models. In Proc. of ACL-IJCNLP, pages 3735-3745, Online. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog.
Stefan Riezler and John T. Maxwell. 2005. On some pitfalls in automatic evaluation and significance testing for MT. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 57-64, Ann Arbor, Michigan. Association for Computational Linguistics.
Michael Roth and Mirella Lapata. 2016. Neural semantic role labeling with dependency path embeddings. In Proc. of ACL, pages 1192-1202, Berlin, Germany. Association for Computational Linguistics.
Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Miloš Stanojevic, Phil Blunsom, and Chris Dyer. 2022. Transformer Grammars: Augmenting Transformer language models with syntactic inductive biases at scale. ArXiv: 2203.00633.
Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In *The Semantic Web*, pages 593–607, Cham. Springer International Publishing.
Petr Sgall, Eva Hajičová, and Jarmila Panevova. 1986. The meaning of the sentence and its semantic and pragmatic aspects. academia.
Yikang Shen, Zhouhan Lin, Chin-wei Huang, and Aaron Courville. 2018. Neural language modeling by jointly learning syntax and lexicon. In Proc. of ICLR.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In Proc. of ICLR.
Aviv Slobodkin, Leshem Choshen, and Omri Abend. 2021. Semantics-aware attention improves neural machine translation. ArXiv:2110.06920.
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proc. of EMNLP, pages 5027-5038, Brussels, Belgium. Association for Computational Linguistics.
Swabha Swayamdipta, Sam Thomson, Kenton Lee, Luke Zettlemoyer, Chris Dyer, and Noah A. Smith. 2018. Syntactic scaffolds for semantic structures. In Proc. of EMNLP, pages 3772-3782, Brussels, Belgium.
Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured Long Short-Term Memory networks. In Proc. of ACL-IJCNLP, pages 1556–1566, Beijing, China.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In Proc. of ACL, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? probing for sentence structure in contextualized word representations. In Proc. of ICLR.
Sean Trott, Tiago Timponi Torrent, Nancy Chang, and Nathan Schneider. 2020. (Re)construing Meaning in NLP. In Proc. of ACL, pages 5170-5184, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NeurIPS, pages 5998-6008, Long Beach, CA, USA.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proc. of ICLR.
Zhaofeng Wu, Hao Peng, and Noah A. Smith. 2021. Infusing Finetuning with Semantic Dependencies. TACL, 9:226-242.
Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. 2020. Perturbed masking: Parameter-free probing for analyzing and interpreting BERT. In Proc. of ACL, pages 4166-4176, Online. Association for Computational Linguistics.
Kaiyu Yang and Jia Deng. 2020. Strongly incremental constituency parsing with graph neural networks. In Proc. of NeurIPS, volume 33, pages 21687-21698. Curran Associates, Inc.
Zdeněk Žabokrtský, Daniel Zeman, and Magda Ševčíková. 2020. Sentence meaning representations across languages: What can we learn from existing frameworks? Computational Linguistics, 46(3):605-665.
A Additional Modeling Details
A.1 Selecting Anchor Nodes
In case there are multiple anchoring options (see, e.g., EDS nodes 0 vs. 1 for first token in figure 1), we use the following tie-breaker heuristics: Select the anchor node with the most parents and children; if still a tie, select the anchor with the highest node ID (tends to be hierarchically lower, i.e., vertically closer to the token anchor).
A.2 Relative Types
$REL = \langle P, B, O, T, C, R \rangle$ , namely, parents  $P_i$ , siblings  $B_{i,p}$ , grandparents  $O_{i,p}$ , aunts  $T_{i,p}$  (all indexed by parent  $p$ ), children  $C_i$ , and coparents  $R_{i,c}$  (indexed by child  $c$ ). This is the anchor node's Markov blanket, plus siblings, grandparents, and aunts. We chose this set of relations based on general notions of linguistic hierarchy (predicate-argument, head-dependent) and preliminary experiments, but without tuning for specific formalisms. Precise definitions are given in table 7. Relative nodes are permanently associated with the label of the edge that got them selected.
A.3 Representing Tokens and Labels
We use GPT-2's pretrained global embeddings (from the lowest layer, before any local contextualization) to obtain embeddings for relative token anchors in the slice-vector. When a token anchor in a linguistic graph consists of multiple BBPE tokens, we average their embeddings. We reuse the transpose of the same embedding matrix again to project the last hidden state of the token-emission MLP into the vocabulary.
SLR edge labels are encoded as one-hot vectors in the slice vectors, which lowers the potential for unnecessary random initialization variance of from-scratch embeddings.
A.4 Distinguishing Dependencies from Constituencies
While this distinction—as defined in §5.2 in terms of the anchoring mapping between graph nodes and word tokens—can be subtle for individual sentences, it nonetheless affects slice encoding. In PSD, for example, auxiliaries are unanchored, whereas in PTG they are grouped with their main predicate (figure 2).
rel	Name	Definition	γ
Pi	parent	{v : (v, ai) ∈ E}	2
Bi,p	sibling	{v : (p, v) ∈ E} ∀p ∈ Pi	2
Oi,p	grandparent	{v : (v, p) ∈ E} ∀p ∈ Pi	1
Ti,p	aunt	{v : (o, v) ∈ E ∧ o ∈ Oi,p}	2
∀p ∈ Pi
Ci	child	{v : (ai, v) ∈ E}	2
Ri,c	coparent	{n : (v, c) ∈ E} ∀c ∈ Ci	1
Table 7: Relative types and capacities.
A.5 Model Hyperparameters
We report our model and training hyperparameters in table 8. We did not perform explicit hyperparameter tuning, besides some manual testing early in development on a subset of the MRP shared task data. Those data are annotated with SLR frameworks other than the ones we compare here, and we ended up excluding them from our experiments for lack of overlap with most of the other frameworks' annotations.
A.6 Efficient Batching for R-GCN
In our incremental setting we need to apply the R-GCN to each token-level slice, which would lead to multiple days $^{15}$  of training for each model if done naively. We achieve a considerable speedup by exploiting the oracle graphs at training and evaluation time to pre-compute slices and running the R-GCN only once per sentence batch.
B.1 Sentence Filtering
To establish a common ground for comparison, we take the intersection of sentences occurring in the annotated datasets of all linguistic formalisms.
In a first step, we discard two sentences whose linguistic graph in at least one formalism is empty.18 We then select only those 35,513 traindev / 1,401 eval sentences that appear in both the MRP 2019 and 2020 datasets (the 2019 corpus contains 143/1,958 more in train-dev/eval).19 Next,
15Projected timeline based on a few iterations, which is confirmed by Yang and Deng (2020).
16For label set  $L$ . The factor 16 arises from the capacities chosen (table 7), and the extra embedding allocation is for averaged preceding unanalyzable/within-anchor tokens.
17For bidirectional label set  $L^{*}$  , which is twice as big as  $L$
18The sentence "It is." in DM and a 'sentence' consisting of the @-symbol in PTG.
19 'train-dev' refers to the data split that was used as training data in both the MRP and 2019 tasks, and which we split  $90\%$  /  $10\%$  into our training and development data. 'eval' refers to the data that was used as evaluation data in MRP 2019 and as development data in MRP 2020, and which we evaluate our
Figure 2: Example of subtle differences in constituency (PTG) and dependency (PSD) versions of the same underlying formalism, the Prague Functional Description. PTG has an abstract PRED node as well as a multiword anchor where PSD does not, which results in diverging slice representations for the last two tokens.
we take the intersection of these sentences and OntoNotes 5.0, which contains the gold PTB syntax annotations. 26,719/929 sentences remain in the train-dev/eval set. The MRP graph format operates on raw-text character offsets, while PTB and UD trees operate on word tokens. We are able reconstruct offset-based text anchors for PTB and UD from the raw text strings used in the MRP data for all but 394 train-dev / 8 eval sentences, which leaves us with the final 26,325 train-dev and 921 eval sentences.
In a few cases, where the linguistic graph has no edges, we add an artificial edge with a dummy label.
B.2 Tokenization
We follow the sentence segmentation of the Penn Treebank corpus. Within sentences, we obtain token boundaries from GPT-2's pretrained byte-level byte-pair encoding (BBPE) tokenizer. The BBPE tokens are then aligned with the formalism-dependent SLR node anchors via raw-text character offsets. Tokens that are continuations of multiword anchors in the graph ('\_reported' in PTG, figure 1); subword tokens of a single graph anchor ('N-umerous'); or are unanchored in the graph ('\_were' in EDS), are treated as unanalyzable, i.e., their slice consists of a copy of the preceding to
GPT-2
Embedding dim	768
Vocabulary	50,257
Activation	GELU
Dropout	0.1
Learning rate	1e-6
MLP
Input dim	16 * |L| + 17 * 76816
Layers	2
Hidden dims	1,024; 768
Activation	ReLU
Dropout	0.2
Learning rate	1e-4
R-GCN
Input dim	768
Layers	2
Hidden dims	768; 768
Activation	ReLU
Basis matrices	[0.1 * |L*|]17
Learning rate	1e-4
Other training settings
Epochs	10
Batch size	8
Table 8: Model and training hyperparameters
ken's slice, plus the preceding within-anchor tokens.
B.3 UD Conversion
Quasi-gold UD 2.0 trees are obtained from the UD converter released with the Java Stanford Parser v4.2.0 (https://nlp.stanford.edu/software/lex-parser.html) on the PTB trees.
B.4 PTB Labels
By convention, phrasal and functional labels in PTB are node labels. To match the labeled-edges-unlabeled-nodes format of the other formalisms, we losslessly convert them to edge labels (namely, on each node's single incoming edge), discarding the preterminal nodes' POS labels. In preliminary experiments we saw that including the POS tags is much more beneficial than phrase structure only; but since we do not include word-level tags in any of the other conditions, this would be an unfair comparison. We focus here on sentence-level structure and leave studies of word-level tags to future work.
B.5 Data Splits
We split the corpus into training/development and evaluation data following the MRP task setup. Specifically, we evaluate on the data split that was
models on.
used as evaluation data in MRP 2019 and as development data in 2020, as only for this data gold annotations in all formalisms have been released. We do not perform empirical hyperparameter tuning. In early development, a small subset of the data was used.
We report detailed results without early stopping (table 9), breakdowns by POS-class (table 10 and appendix C.1), as well as ablation experiments (table 11) for all SLR formalisms. In tables 4 and 10 and figure 3 we merge the POS tags {NOUN, PROPN} into 'noun', {ADJ, ADV} into 'mod', and {INTJ, SYM, X} into 'misc'.
C.1 Lexico-Semantic or Syntactic Knowledge?
In §6.3 we have found part-of-speech-specific patterns of model performance. But whenever, for a certain syntactic word class  $S$ , a formalism  $A$  is more conducive to next-word prediction than a formalism  $B$ , it is not clear whether this is the case because the choices get narrowed down to  $S$  itself or whether it is caused by either complementary or completely independent signals, perhaps at the lexical or semantic-structure levels.
We investigate this by rerunning the experiment with each token's UPOS tag as an additional input. If this is more or less the same information as is gained—to different extents—from the SLRs, then the results should be similar to before, and SLR-conditional differences should disappear.
A few particularly interesting POS subsets are shown in figure 3. We discuss them in order.
Among content words, nouns and verbs are similar both in terms of baseline performance and in how much easier it becomes to select the correct lexical item if the part-of-speech is known. At the same time, the individual SLR formalisms differ quite a lot in how much information they contribute about the POS class itself and about lexical choice within the part-of-speech. The respective best formalisms (EDS for nouns, PTB and UD for verbs) approximate oracle POS knowledge by themselves and still contribute substantial complementary information when the actual POS tag is revealed. In contrast, PTB does not seem to provide any useful signal about nouns to the incremental LM—neither independently nor in conjunction with the POS.
Modifiers (adjectives and adverbs) display a
rather interesting behavior: the fact that a word of this type is coming next is very hard to predict from just the preceding raw context, which makes sense since they tend to add optional meaning on top of the (obligatory) logical and grammatical content. However, once the decision to modify has been made, the contextual choice becomes much easier than that for nouns or verbs. In both cases, all SLRs are quite helpful, with UD on the lower end and EDS leading the field.
We find similar tendencies among auxiliaries ( $\approx$ function verbs) and pronouns ( $\approx$ function nouns) as with (content) verbs and (content) nouns, but naturally at a much smaller scale. Despite their functional-grammatical distribution and behavior, the semantic frameworks EDS and PTG consistently outperform the syntactic ones UD and PTB even on these 'small' words. A possible explanation for this interaction with auxiliaries in particular could be that EDS and PTG do not analyze them separately at all, but rather group them, respectively, with the preceding context $^{20}$  or their main predicate. The models might be able to leverage this to focus on things like subject-verb agreement, local cohesion, or anticipating the main predicate. More explicit syntactic analyses of auxiliaries (incrementally inaccessible forward-pointing dependencies in UD; VP-nesting in PTB), in contrast, may restrict the model from directly making these connections. Adding POS information in the input decreases SLR-dependent differences.
For 'subordinators' in the broad sense, i.e., subordinating conjunctions at the clausal level and adpositions for nominal complements, PTB and PTG are particularly well-suited. By themselves they are already at least as informative as POS, and they still add a small but noticeable complementary signal when the POS is revealed.
Determiners and coordinating conjunctions, which both already show extremely low perplexity with some SLR models (namely, EDS, PSD, and PTG), entirely lose any reliance on particular SLRs when their POS is known.
20EDS, like PSD, actually has no anchors for auxiliaries; we attach them to the preceding semantic unit by default.
Training Efficiency
Language Model Quality
Model	Scope/Struct	#Labels	Speed ↑	Size ↓	PPL ↓	H [nats] ↓	Acc [%] ↑	Conf [%] ↑	MRR ↑
GPT-2	-	124.4M	59.3	4.09	30.0	31.2	.403
+ Domain	15	45.8 ±.03	3.61 ±.002	33.4 ±.05	35.3 ±.02	.436 ± .3e-3
+ UD	syn	dep	39	14	+54.1M	35.2 ±.24	3.09 ±.014	39.2 ±.11	42.3 ±.18	.488 ± .8e-3
+ DM	sem	dep	59	15	+54.4M	34.2 ±.32	3.05 ±.019	38.8 ±.15	42.5 ±.26	.490 ±1.0e-3
+ PSD	sem	dep	90	16	+54.9M	34.1 ±.43	2.96 ±.014	39.2 ±.17	44.0 ±.17	.491 ±1.4e-3
+ PTB-phr	syn	const	38	14	+54.1M	33.5 ±.30	2.97 ±.026	40.3 ±.09	43.9 ±.34	.500 ± .6e-3
+ PTB-fxn	syn	const	537	14	+62.7M	32.4 ±.37	2.92 ±.030	41.1 ±.18	44.8 ±.36	.507 ±1.3e-3
+ PTG	sem	const	72	15	+54.6M	29.6 ±.20	2.68 ±.028	43.4 ±.08	48.8 ±.32	.524 ± .5e-3
+ EDS	sem	const	10	15	+53.6M	26.6 ±.09	2.78 ±.024	43.1 ±.10	46.6 ±.24	.527 ± .8e-3
Table 9: Main results without early stopping: performance of language models combined with 7 SLR formalisms of different scope, structure, and label set (each corresponding to a  $P_{\text{Ensemble}}$  in §4.3), compared to vanilla GPT-2 and a version of GPT-2 that has been domain-finetuned on the raw text of the SLR training corpus ( $P_{LM}$ ). We report each quality metric as mean ± stdev over 5 random seeds. We also report model size in #parameters and training speed in sentences per second as measures of efficiency. Best results in each column are bolded. For confidence, 'best' means best-calibrated, i.e., the smallest relative difference to accuracy.
Figure 3: Model perplexity (lower is better) with UPOS as additional input. Top left: nouns, verbs, and modifiers; top right: auxiliaries and pronouns; bottom left: adpositions and subordinating conjunctions; bottom right: determiners and coordinating conjunctions. Big gray squares mark baseline (finetuned GPT-2) performance without (dark) and with (light) POS inputs and SLR-specific data points without/with POS inputs follow below the squares in each respective column. Mind the different y-axis scales, and in particular the log scale in the top-left plot, which makes it easier to read very big and slightly smaller (but still big) differences at the same time.