SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 2: Main results: performance of language models combined with 7 SLR formalisms of different scope, structure, and label set (each corresponding to a  $P_{\text{Ensemble}}$  in §4.3), compared to vanilla GPT-2 and a version of GPT-2 that has been domain-finetuned on the raw text of the SLR training corpus ( $P_{LM}$ ). We report each quality metric as mean ± stdev over 3 random seeds. We also report model size in #parameters (all non-baseline models as absolute difference to baseline) and training speed in sentences per second as measures of efficiency. Statistical significance of the PPL and Acc differences to the next-best model (always adjacent rows) is reported as *** $p < .0001 / **p < .001 / *p < .005 / -not significant$  (approximate randomization test as described in Riezler and Maxwell (2005), with  $R = 10,000$  shuffles). We only consider a difference significant if  $p < \alpha$  for all three random model initialization seeds. Best results in each column are bolded. For confidence, 'best' means best-calibrated, i.e., the smallest relative difference to accuracy.
Dep	Const	Avg
Syn	32.7 ***	(1)	***	29.4 ±0.6 (2)	30.5 ±2.0 (3)
Sem	31.0 ±0.5 (2)	***	25.7 ±1.5 (2)	28.4 ±3.2 (4)
Avg	31.6 ±1.0 (3)	**	27.6 ±2.3 (4)	29.3 ±2.8 (7)
Table 3: Model perplexity (lower is better) summarized in terms of two SLR dimensions: Scope (syntax vs. semantics) and structure (dependency vs. constituency).  $\mu \pm \sigma (n)$  over frameworks per condition. Statistical significance of the difference between the two closest SLRs of each pair of conditions is reported as  $^{**}p < .0001 / ^{**}p < .001 / ^{*}p < .005 / -\mathrm{not}$  significant (approximate randomization test with  $R = 10,000$  shuffles).
in table 3. A limiting factor for dependency representations in the incremental LM setting is that relations between the target token and subsequent tokens are entirely ignored, whereas constituency graphs can back off to higher-level structures. Further, the syntactic graphs we use are always trees, so they never populate the coparent capacity in the slices. Semantic constituency representations, with their abstract and meaning-oriented labeling and structure schemes, jump out as being especially predictive of the underlying text, as compared to both syntax and shallow semantics.
We note that the function-enhanced PTB label
set has a slight advantage over the basic phrase-structure labels; and that, among the two closely related pairs of formalisms (DM/EDS and PSD/PTG, which each are dependency and constituency versions converted from the same underlying grammars), the constituency versions always work better than the dependency versions in our setting. There is, however, no consistent ranking between DM/EDS on one hand and PSD/PTG on the other. In terms of perplexity, EDS works better than PTG, and PSD better than DM, but these differences are not significant for accuracy.
6.3 Differences between Word Classes
To better understand where particular strengths and weaknesses of the baseline LM and linguistically enhanced models lie, we analyze subsets of tokens by part-of-speech (POS) tag (table 4, see appendix C for more details). Across all models there is a clear and expected separation between rather predictable function words, more perplexing content words, and numbers, punctuation, and miscellaneous tokens somewhere in the middle.
Average perplexity of the tested SLR models is better than baseline GPT-2 in all POS classes but one. The one exception is the noun class, where
both the SLR macro-average and UD in particular do not raise performance. Only EDS and DM show perplexity improvements on nouns; PTB even has a noticeable negative impact. We conjecture that this may have to do with relatively deep NP nesting in PTB (compared to the other formalisms), such that the current slicing hyperparameters (relative types and capacities) are too strict and hide informative signals like modifiers and verb attachment.
Some formalisms seem to be particularly well-suited for the prediction of certain POS: UD for verbs; PTB and PTG for adpositions and subordinating conjunctions; EDS for pronouns, determiners, and numbers; PTG, PSD, and EDS for coordinating conjunctions. The advantage of EDS and DM on nouns, pronouns, determiners, and numbers can likely be attributed to their explicit representation of variable binding/quantification. Similarly, PTG and PSD have detailed categories for coordination, distinguishing, e.g., con- and disjunction.
For nouns and modifiers, the spread across formalisms is particularly wide, which suggests that SLRs diverge quite a bit on these types of words (e.g., whether adjectives and certain nouns can count as predicates) and that this diversity has a strong effect on utility for language modeling.
6.4 Model Ablations
The linguistically enriched models consist of a substantial number of newly learned parameters—around 50–60M each, an additional  $\approx 50\%$  the size of vanilla GPT-2. Although model size does not seem to be correlated with performance among the SLR-enriched models, it could still be that the additional capacity allows the models to store more information about the words' distributions than the baseline GPT-2 model, without ever truly using the concrete linguistic structures.
We check this by randomly shuffling  $(\times)$  two core graph properties: (i) the assignment of edge labels, and (ii) the anchoring mapping between graph nodes and word tokens in each graph. If the models are largely independent of the correct label and structure assignments, these changes should have a very small effect on performance (Dubossarsky et al., 2018; Hewitt and Liang, 2019).
But on the contrary, we find that performance worsens considerably in the ablated settings compared to the full combined models of each formalism (table 5, see appendix C for more details). This
POS	Eval Toks	Train Vocab	Perplexity↓
GPT-2	UD	EDS	SLR Avg
content	All	22,596	27,344	45.9	32.7	24.7	29.3 ± 2.8
noun	7,731	18,435	142.5	122.0	98.0	122.6 ±13.9
verb	2,639	7,100	128.8	80.4	85.9	84.9 ± 4.5
mod	2,235	6,292	228.7	158.8	98.6	124.4 ±22.6
function	aux	582	95	17.6	11.1	5.9	9.1 ± 2.1
adp	1,957	232	10.1	7.3	5.5	5.3 ± 1.6
part	645	27	3.7	2.0	1.6	1.9 ± 0.3
sconj	268	96	15.4	12.3	6.8	6.8 ± 3.9
cconj	548	35	13.0	7.4	1.9	4.1 ± 2.1
det	1,726	91	9.4	7.8	4.4	6.0 ± 1.3
pron	868	149	22.9	17.5	5.4	11.0 ± 4.0
num	719	1,059	72.6	57.1	47.5	54.1 ± 4.6
punct	2,527	68	4.9	2.3	2.7	2.6 ± 0.3
misc	151	183	7.0	4.6	4.0	4.5 ± 0.8
Table 4: Breakdown by Universal POS, $^{12}$  in terms of PPL of domain-trained GPT-2, two exemplary SLR-combined models, and the macro-average  $\pm$  stdev over all SLR-combined models. Best results (within the variance) in each row are bolded. We show token counts and observed vocabulary size for reference.
Ablation	Applied in	DM	PTB	SLR Avg
Full	31.4	29.0	29.3 ± 2.8
× Labels	testing	+4.7	+73.9	+28.3 ±28.3
× Anchors	testing	+34.8	+223.1	+106.0 ±73.1
× Both	testing	+33.4	+207.4	+95.9 ±68.9
× Labels	training	+1.4	+9.0	+4.2 ± 3.3
× Anchors	training	+8.5	+17.5	+13.3 ± 4.6
× Both	training	+7.8	+18.3	+13.4 ± 5.0
× Labels	both	+1.3	+9.3	+4.3 ± 3.4
× Anchors	both	+7.9	+17.5	+13.5 ± 5.0
× Both	both	+7.3	+18.1	+13.6 ± 5.2
- SLR	both	+14.5	+16.9	+16.6 ± 2.8
Table 5: Ablations measured in  $\Delta$  PPL for two exemplary SLR-combined models and the macro-average  $\pm$  stdev over all SLR-combined models. Full and -SLR correspond, respectively, to table 2's rows 4 (DM) / 7 (PTB-fxn) and row 2 (GPT-2 +Domain).
confirms that the models really do acquire—and are quite sensitive to—the graph-encoded linguistic signals, relying to a large part on this new information in making their predictions.
Shuffling only edge labels while leaving the rest of the graphs unchanged has a smaller effect than changing how tokens are anchored in the graph structure. This suggests that the linguistic graphs' entire structural arrangement of labels and attention-like selection of context words play a crucial role—more so than knowing the type of each individual (correctly attached) grammatical relations. Note that the  $\times$  Anchors setting, too, changes which edge labels are used in the predic
<sup>12</sup>https://universaldependencies.org/u-pos/
Model	Training Efficiency	LM Quality
δSpeed ↑	ΔSize ↓	ΔPPL ↓	ΔAcc ↑
UD	-50%	-1.9M	+2.9 ±.08	-0.4 ±.02
DM	-47%	+2.5M	+1.6 ±.35	+0.1 ±.14
PSD	-56%	+9.1M	+3.6 ±.23	-0.9 ±.15
PTB-phr	-43%	-1.9M	+6.8 ±.39	-1.7 ±.11
PTB-fxn	-86%	+107.7M	+10.5 ±.21	-2.7 ±.15
PTG	-53%	+5.9M	+6.2 ±.22	-3.5 ±.03
EDS	-47%	-8.5M	-0.2 ±.07	+0.1 ±.03
Table 6: Performance differences between R-GCN slice encoder baseline and our concatenation-based encoder (table 2). Relative differences  $(\delta)$  for speed in sentences per second; absolute differences  $(\Delta)$  otherwise. Means  $\pm$  stdev over 2 runs without early stopping.
tion of a given token, resulting in a smaller difference between  $\times_{\mathbf{x}}$  Anchors and  $\times_{\mathbf{x}}$  Both.
If a model has learned to rely on correct labels and structure during training, then perturbing these properties at test time has a highly adverse effect, confusing the model and leading to a drastic decrease in performance—even worse than not consulting SLR graphs at all! Given previous findings that syntactic structure is to some extent already learned in pretraining (Linzen et al., 2016; Tenney et al., 2019b), we conjecture that this representational capacity gets offloaded to the graphs at training time, and thus test-time permutations fool the PTB model to a much greater extent than DM.
As expected, exposing models to shuffled graphs at training time renders the additional model parameters practically neutral, resulting in similar perplexity as the base LM. In this case, it also does not matter whether test-time graphs are correct or random (training vs. both in column 2)—either way, the model learns to mostly disregard the random structure as noise.
6.5 Comparison with R-GCN Encoding
As an additional strong baseline, we compare our concatenation-based slice vector encoding to a graph neural network from the literature. We choose relational graph-convolutional networks (R-GCN; Schlichtkrull et al., 2018; Kipf and Welling, 2017) as a suitable representative of this type of model, which has been used successfully by Wu et al. (2021) to encode DM graphs.
Results are shown in table 6. Contrasting with table 2, there is a big difference in training speed: our simple encoder is on average roughly twice as fast as the computation-heavy alternative, whose time and space complexity is dominated by the
number of labels.13
We observe at best similar LM quality as with our concatenation method (EDS and DM), but for most formalisms performance degrades. We follow Schlichtkrull et al. and Wu et al. in using 2 R-GCN layers with basis matrix regularization. Possible disadvantages of this for encoding linguistic graphs are the fixed path length (2 layers exclude parent's siblings; but 3 layers would include a lot of irrelevant information) and that many of the trained parameters are shared between different relations. In contrast, our concatenation encoding forces the MLP input layer to learn distinct parameters for each structural relative type and edge label.
7.1 Related Work
Researchers have long been interested in scaffolding sequential language models with linguistic-structure-based inductive biases. Syntactic language modeling dates back to the pre-neural era, when Pauls and Klein (2012) and Gubbins and Vlachos (2013) generalized Markov assumptions from word  $n$ -grams to syntactic subtrees. These ideas have since been adapted to recurrent neural network (RNN) LMs (Mirowski and Vlachos, 2015) and expanded on (Dyer et al., 2016; Choe and Charniak, 2016; Shen et al., 2018, 2019). Ek et al. (2019) condition RNN-LMs on predicted syntactic and semantic (unstructured) tags, interestingly finding less or sometimes no benefit, especially on the semantic side. They hypothesize this might be due to tagging errors—an issue our oracle setup avoids.
In the era of attention-based neural modeling of language dominated by pretrained Transformers, models are often finetuned for and evaluated on specific NLP tasks—like semantic role labeling, machine translation, natural language inference, graph-to-text generation, or the GLUE benchmark (Wang et al., 2019)—rather than language modeling in its own right, which makes it difficult to compare them directly to our findings. There have been two main directions: One group of approaches continues the old syntactic language modeling tradition by incrementally generating words and SLRs with either joint (Peng et al., 2019; Qian et al., 2021; Sartran et al., 2022) or iteratively-coupled LM and parser models (Choshen and Abend, 2021). The second group assumes parsed input sentences,
<sup>13</sup>And this is a very optimistic estimate of R-GCN training speed in practice; see appendix A.6.
which are then used to guide the model, e.g. by directly optimizing Transformers' attention weights to reflect linguistic graph structures (Strubell et al., 2018; Bai et al., 2021; Slobodkin et al., 2021). Rather than controlling the existing sequential attention, Hajdik et al. (2019) process serialized graphs directly with a sequence-to-sequence model, and Wu et al. (2021) extend a pretrained Transformer with an additional graph encoder. Notably, Wu et al. (2021) and Slobodkin et al. (2021) experiment with a few different semantic and syntactic SLRs, while all other studies we have looked at are limited to either syntax or very shallow semantics.
Another relevant line of work employs probing tasks in investigating to what extent grammar and meaning are already encoded in neural language models trained predominantly on raw text with little to no linguistic supervision (Linzen et al., 2016; Tenney et al., 2019a,b; Hewitt and Manning, 2019; Liu et al., 2019; Kim et al., 2019; Wu et al., 2020; Geiger et al., 2021, inter alia). Among the probing literature, the works of Kuznetsov and Gurevych (2020) and Kulmizev et al. (2020) are noteworthy in that they investigate subtle differences between different (versions of) frameworks roughly covering the same representational scope, namely, semantic roles and syntactic dependencies, respectively.
Orthogonal approaches to comparing SLR designs have involved measuring how well different frameworks complement each other for joint parsing or can be merged or converted into one another (Prange et al., 2019a; Hershcovich et al., 2020).
7.2 Limitations and Future Work
While the use of oracle graphs has both theoretical advantages (measuring an upper bound without needing to account for potential errors or uncertainties) and practical ones (saving the computational overhead from training and running a parser), ground-truth SLR graphs are a very limited resource and generally assumed to only be available at training time. There is no guarantee our results translate to the non-oracle setting. For instance, it could be that the most helpful abstract semantic information is also the hardest to predict. And despite segmenting the existing sentence-level graph into token-level slices, the human annotator who created the graph in the first place has seen and analyzed the whole sentence, thus already resolving crucial ambiguities and simplifying the task based on knowledge 'from the future'. In subsequent work, we plan to parse graph slices incrementally,
which will both relax the conditional modeling assumption into a more broadly interpretable joint model and enable test-time use of the full system on datasets without linguistic annotations.
We also only test formalisms that are explicitly anchored in linguistic units, roughly corresponding to LM (sub-)word tokens. This prevents us from applying the same paradigm to some other widely-used unanchored formalisms like AMR (Banarescu et al., 2013) without some changes to the setup.
7.3 Broader Impact
Our experiments yield evidence which—at least in the case of encoding contexts for next-word prediction—supports the thesis of Bender and Koller (2020), Trott et al. (2020), and others that linguistic meaning goes beyond form. Computational models of language that exclusively learn from even very large amounts of raw text are thus generally expected to hit a ceiling<sup>14</sup> which can only be overcome with access to higher-level structures and mechanisms of understanding.
It further seems to matter in which manner and shape linguistic graph structure is drawn. Assuming a perfect incremental parser, deeper structure and semantic categorization seems to be particularly beneficial for integration with a standard language model. This is in line with previous findings by, e.g., Tenney et al. (2019b) that while pretrained LMs tend to encode shallow syntactic structure, abstract relations are more difficult to probe for.
We thus see a promising research direction in moving towards linguistic scaffolding of language models with representations that are more complex than tags or dependencies and that capture meaningful relations beyond surface structure.
We have presented evidence that symbolic linguistic representations of various frameworks have the potential to aid a pretrained incremental Transformer in task-neutral next-word prediction. To this end, we have proposed a framework-agnostic neural encoding scheme for linguistic graphs and applied it to an English dataset jointly annotated with 7 different formalisms. The results highlight the importance of appreciating complex linguistic structure and handling its computational representation with nuance.
See also Merrill et al. (2021) for formal proofs.
We would like to thank Katrin Erk and Chris Dyer; members of the Georgetown NERT/GUCL and HKU NLP labs; the organizers, reviewers, and audience of MASC-SLL 2022; as well as the anonymous ARR reviewers for their extremely insightful feedback and suggestions.
Omri Abend and Ari Rappoport. 2017. The state of the art in semantic representation. In Proc. of ACL, pages 77-89, Vancouver, Canada.
Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang, Jing Bai, Jing Yu, and Yunhai Tong. 2021. Syntax-BERT: Improving pre-trained transformers with syntax trees. In Proc. of EACL, pages 3011-3020, Online. Association for Computational Linguistics.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proc. of LAW-ID, pages 178-186, Sofia, Bulgaria.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198, Online. Association for Computational Linguistics.
Alena Böhmová, Jan Hajic, Eva Hajicová, and Barbora Hladká. 2003. The Prague Dependency Treebank: A three-level annotation scenario. In Anne Abeillé, editor, Treebanks: Building and Using Parsing Corpora, Text, Speech and Language Technology, pages 103-127. Springer Netherlands, Dordrecht.
Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2331-2336, Austin, Texas. Association for Computational Linguistics.
Leshem Choshen and Omri Abend. 2021. Transition based graph decoder for neural machine translation. ArXiv:2101.12640.
Ann Copestake, Dan Flickinger, Carl Pollard, and Ivan A Sag. 2005. Minimal recursion semantics: An introduction. Research on language and computation, 3(2):281-332.
Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255-308.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL-HLT, pages 4171-4186.
Haim Dubossarsky, Eitan Grossman, and Daphna Weinshall. 2018. Coming to your senses: on controls and evaluation sets in polysemy research. In Proc. of EMNLP, pages 1732-1740, Brussels, Belgium. Association for Computational Linguistics.
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent Neural Network Grammars. In Proc. of NAACL-HLT, pages 199-209, San Diego, CA, USA.
Adam Ek, Jean-Philippe Bernardy, and Shalom Lappin. 2019. Language modeling with syntactic and semantic representation for sentence acceptability predictions. In Proceedings of the 22nd Nordic Conference on Computational Linguistics, pages 76–85, Turku, Finland. Linköping University Electronic Press.
Dan Flickinger. 2000. On building a more efficient grammar by exploiting types. *Natural Language Engineering*, 6(1):15-28.
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. 2021. Causal abstractions of neural networks. In Proc. of NeurIPS.
Joseph Gubbins and Andreas Vlachos. 2013. Dependency language models for sentence completion. In Proc. of EMNLP, pages 1405-1410, Seattle, Washington, USA. Association for Computational Linguistics.
Valerie Hajdik, Jan Buys, Michael Wayne Goodman, and Emily M. Bender. 2019. Neural text generation from rich semantic representations. In Proc. of NAACL-HLT, pages 2259-2266, Minneapolis, Minnesota. Association for Computational Linguistics.
Jan Hajic, Eva Hajicová, Jarmila Panevová, Petr Sgall, Ondrej Bojar, Silvie Cinková, Eva Fučíková, Marie Mikulová, Petr Pajas, Jan Popelka, Jiří Semecký, Jana Šindlerová, Jan Štepánek, Josef Toman, Zdenka Urešová, and Zdeněk Žabokrtsky. 2012. Announcing Prague Czech-English dependency treebank 2.0. In Proc. of LREC, pages 3153–3160, Istanbul, Turkey. European Language Resources Association (ELRA).
Daniel Hershcovich, Nathan Schneider, Dotan Dvir, Jakob Prange, Miryam de Lhoneux, and Omri Abend. 2020. Comparison by conversion: Reverse-engineering UCCA from syntax and lexical semantics. In Proc. of COLING, pages 2947-2966, Barcelona, Spain (Online). International Committee on Computational Linguistics.
John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proc. of EMNLP-IJCNLP, pages 2733-2743, Hong Kong, China. Association for Computational Linguistics.