SlowGuess/NAACL_2022_OCR huggingface dataset:
Jakob Prange Nathan Schneider
{jp1724, nathan.schneider}@georgetown.edu
Lingpeng Kong
lpk@cs.hku.hk
We examine the extent to which, in principle, different syntactic and semantic graph representations can complement and improve neural language modeling. Specifically, by conditioning on a subgraph encapsulating the locally relevant sentence history, can a model make better next-word predictions than a pretrained sequential language model alone? With an ensemble setup consisting of GPT-2 and ground-truth graphs from one of 7 different formalisms, we find that the graph information indeed improves perplexity and other metrics. Moreover, this architecture provides a new way to compare different frameworks of linguistic representation. In our oracle graph setup, training and evaluating on English WSJ, semantic constituency structures prove most useful to language modeling performance—outpacing syntactic constituency structures as well as syntactic and semantic dependency structures.
Linguistic theories posit that humans can take advantage of hierarchical structure related to some notion of compositionality to produce and comprehend utterances with complex meanings. Yet explicit representations of this kind of structure are harder to come by than raw text, and large-scale pretrained neural language models (e.g., Devlin et al., 2019; Radford et al., 2019) have managed to perform strikingly well at contextually encoding and predicting words from distributional evidence alone. At the same time, there are good reasons to doubt that these models can be said to understand language in any meaningful way (Trott et al., 2020; Bender and Koller, 2020; Merrill et al., 2021). To address this conundrum, people have started to explore probing pretrained models (Liu et al., 2019; Tenney et al., 2019a, inter alia) and supplementing training data with linguistic structure guidance (Strubell et al., 2018; Swayamdipta et al., 2018; Peng et al., 2019; Wu et al., 2021, inter alia).
A question that has received less attention is which kind of symbolic linguistic representation (SLR) is most conducive to guiding neural language models (LMs). Numerous domain-general candidates exist (Abend and Rappoport, 2017; Oepen et al., 2019, 2020; Žabokrtský et al., 2020; Müller, 2020): some are focused on syntactic structure, others on semantics ( $\S 2$ ; big grey example graphs in the left panels of figure 1). Frameworks vary along several dimensions, with different label inventories and treatments of specific constructions. Formal differences include the type of structure (dependency or constituency, one or multiple parents, projectivity) and its relation to the input string. In general, different design choices may aim to capture different kinds of generalizations or facilitate different kinds of processing, and may make parsing raw text easier or harder. It is often not obvious which framework should be chosen for best results on an external task—or indeed, how to even perform a controlled comparison across frameworks.
In this paper we investigate whether structurally guided language modeling can serve as a benchmark task for directly comparing linguistic representations. Specifically, we evaluate on next-word prediction—a relatively neutral task in that it does not rely on any artificial test suite, nor does it target a specific downstream application where one linguistic framework may have an advantage. $^{1}$
We devise a method for selecting and encoding partial views of linguistic graphs over the preceding context relevant to predicting the next token ( $\S 3$  and  $\S 4$ ).<sup>2</sup> We call these views slices (small per-token graphs and dashed lines in figure 1). Our neuro-symbolic encoder statically allocates distinct vector dimensions for different structural
<sup>1</sup>Our findings are limited to a particular language (English) and domain (financial news) in which gold graphs from multiple frameworks are available for the same sentences, but such annotations could be obtained for other samples in the future.
$^{2}$ Our code is available to the research community at https://github.com/jakpra/LinguisticStructureLM.
Human Language Technologies, pages 4375 - 4391
UD: Syntactic dependency tree with functional labels
PTG: Deep syntactic / quasi-semantic argument structure
GPT-2: Neural language model with multihead attention
PTB-fxn: Syntactic phrase/function-labeled constituency tree
Figure 1: Contrasting GPT-2's incremental attention mechanism (top right) with incremental context slices obtained from linguistic graphs (left four panels) of four different formalisms ( $\S$ 5.2). As shared tokenization we use GPT-2's byte-pair encoding. Slice nodes are color-coded by local relation type (black: target, cyan: parent, blue: child, green: coparent, yellow: sibling, purple: grandparent, brown: aunt). Dashed lines indicate the token anchoring of original (big grey) graph nodes and, correspondingly, which previous tokens (empty circles) are accessible for each next-token prediction. In the bottom right we visualize how different models arrive at their prediction ( $\S$ 3 and  $\S$ 4.3).
EDS: Semantic predication/modification and variable binding
relations within each slice, which in the incremental setting is much faster and more flexible than computation-intensive deep graph encoders. Using this encoding, we compare 7 SLR formalisms by virtue of their incremental language modeling capability in an controlled experimental setup (§5) on jointly annotated ground-truth data (Oepen et al., 2019, 2020). The results (§6) suggest that linguistic graphs are indeed informative for next-word prediction, complementing what is learned in pretraining. This invites future research quantifying different formalisms' design choices (§7).
2 Background: Symbolic Linguistic Representation
Following a long tradition in formal linguistics, graph-structured representations of language qualitatively describe grammatical and logical relations among words. The SLR paradigm has recently seen a revival in the form of larger-scale treebanking and sembanking for training neural parsers.
Formally, an SLR instance is a directed acyclic graph (DAG)  $G = \langle V, E, \alpha \rangle$ , with vertices  $V$ , labeled edges  $E$ , and an anchoring function  $\alpha : V \to \mathbf{w}$  that maps each vertex to a (potentially empty) subset of tokens in the sentence. We broadly distinguish SLR frameworks along two dimensions:
Scope. A main goal of syntactic representations is to explain distributional patterns in word order; they tend to be rooted trees with often projective anchoring functions. Semantic formalisms are meaning-oriented, aiming to capture the higher-level logic expressed in a sentence; thus, they may have more complex structures, including reentrant edges and discontinuous anchors.
Structure. SLRs can further be subdivided into dependency and constituency structures. The former are relatively shallow, while the latter contain abstract nodes with no or multiple word anchors.
3 Overview: Language Modeling with Linguistic Graphs
Our main goal is to quantify the predictive power of different SLRs by combining them with a pretrained language model and measuring how this affects next-token generation performance. A language model (LM) assigns probabilities to sentences and can be used to both process existing sentences and generate new ones. As is standard practice, we treat sentences as length- $n$  sequences of word tokens,  $\mathbf{w} = \langle w_0, w_1, \dots, w_{n-1} \rangle$ . An incremental LM factorizes the joint probability of the sentence in terms of the probability of each word  $w_i$  conditioned on previous tokens  $\mathbf{w}_{<i}$ ; eq. (1).
Here we describe at a high level how we process (oracle) SLR graphs for use in this language modeling scenario, i.e., to obtain context-conditional
3See Abend and Rappoport (2017); Koller et al. (2019); Prange et al. (2019b) for more detailed taxonomies.
vocabulary distributions from them. In contrast to sequential LMs, contexts are now graph-structured, and which context tokens to select as well as in what way they are related to the target token is determined by the underlying SLR graph  $G$ ; eq. (2).
$$
P _ {L M} (\mathbf {w}) = \prod_ {i = 0} ^ {n - 1} P _ {L M} (w _ {i} | \mathbf {w} _ {<   i}) \tag {1}
$$
$$
P _ {S L R} (\mathbf {w}) := P (\mathbf {w} \mid G) \tag {2}
$$
This general idea is closely related to syntactic language modeling (Pauls and Klein, 2012; Gubbins and Vlachos, 2013, inter alia). We extend this line of work to arbitrarily complex syntactic and semantic DAG structures and, in doing so, take particular care to restrict conditioning contexts from accessing not only future words but also future subgraphs, so effectively top-down and left-to-right. Our procedure is as follows:
First, we select for each token position  $i$  to be predicted a subgraph  $G_{i}$ , called the token's slice. Slices are both admissible in the language modeling setting, i.e., they do not violate the left-to-right conditioning order, and relevant to the token prediction according to some criteria—here we consider criteria based on structural relationships generally, without relying on formalism-specific labels (§4.1). Consider the small colored subgraphs for each token in figure 1: the EDS-slice for the target 'reported', for example, starts at node 3, and extends to the ARG2-child 2, ARG1-coparent 1, and BV-coparent 0, which are anchored, respectively, in the spans 'injuries', 'Numerous', and 'Numerous injuries'). Recall from §2 that context words  $\mathbf{w}_{<i}$  are contained in  $G_{i}$ , to the extent that they are anchored in a node reachable from  $w_{i}$ . Inspired by Markov assumptions of independence in generative modeling and Markov blankets in causal networks, SLR graph slicing thus allows us to factorize  $P(\mathbf{w}|G)$  as
$$
P (\mathbf {w} | G) := \prod_ {i = 0} ^ {n - 1} P (w _ {i} | G _ {i}). \tag {3}
$$
Next, we encode each graph slice as a fixed-sized vector. Prior approaches to encoding linguistic graphs for neural modeling have involved serialization, e.g., as parser transition sequences (Qian et al., 2021, inter alia), recursive auto-encoders (Tai et al., 2015; Roth and Lapata, 2016), and graphconvolutional networks (GCNs; Yang and Deng, 2020; Wu et al., 2021). However, transition sequences for non-tree graphs are subject to spurious
ambiguity; and we find that graph-structured neural networks are impractical in the incremental setting (§6.5). Instead, we propose a computationally inexpensive method for statically and deterministically projecting slices into a high-dimensional space by vector concatenation (§4.2).
Finally, we compute output distributions  $P(W_{i}|G_{i})$  from the vector representations (§4.3).
4 Modeling Details
4.1 Slicing Graphs
A slice  $G_{i}$  is a connected subgraph of  $G$  that captures  $w_{i}$ 's linguistically structured context, masking  $w_{i}$  itself (or else estimating  $P(w_{i}|G_{i})$  would be trivial).  $G_{i}$  always minimally consists of  $w_{i}$ 's direct anchor node  $a_{i} = \operatorname{Select}(\{v : w_{i} \in \alpha(v)\})$ . Starting from  $a_{i}$ , we traverse the graph and add vertices and edges that are connected to  $a_{i}$  via paths of a few specific relative types, REL. Here we settle on 6 types: parents, siblings, grandparents, parents' siblings, children, and coparents. The vertices  $V_{i}$  and edges  $E_{i}$  for slice  $G_{i} = \langle V_{i}, E_{i}, \alpha \rangle$  consist then of the union of these sets. $^{4}$
To prevent information leakage from future tokens, we discard from  $G_{i}$  all nodes  $\{v : \alpha(v) = w_{j}, j > i\}$  which are only anchored in tokens following  $w_{i}$ . E.g., in figure 1, the UD-slice for the token 'were' does not contain the parent node 3 because that is anchored only in the following token 'reported' (and thus the sibling 1 cannot be accessed either). If a node's anchors contain or overlap with  $w_{i}$  (i.e., the node is a non-terminal above  $w_{i}$ ), we retain the node and its edges but remove its token anchors.
4.2 Vectorizing Graph Slices
Because slices can be large, we partition each slice's nodes by structural relative type, in order to aggregate them into a fixed-length summary vector. Specifically, we allocate capacities for each relative type:  $\gamma_{\mathrm{rel}} = 2$  for parents, siblings, aunts, and children, and 1 for grandparents and coparents. Up to  $\gamma - 1$ , relative nodes  $V_{\mathrm{rel}}$  are added 'with high resolution', maintaining their identity and order; beyond the capacity, relatives are aggregated 'with low resolution'; eq. (4). Within each relative type, precedence  $k$  is given to relatives whose token anchors are sequentially closer to  $w_{i}$ .
See appendices A.1 and A.2 for details.
$$
\operatorname {H i R e s} _ {i, \text {r e l}} = \left\langle v _ {\text {r e l}, k}: k <   \gamma_ {\text {r e l}} \right\rangle \tag {4}
$$
$$
\operatorname {L o R e s} _ {i, \text {r e l}} = \left\{v _ {\text {r e l}, k}: k \geq \gamma_ {\text {r e l}} \right\}
$$
Next we look up the relatives' edge label and word vector encodings  $\vec{l}_k$  and  $\vec{w}_k$  and collate them into a single vector  $\vec{s}_{i,\mathrm{rel}}$  per relative type. High-resolution vectors are concatenated  $\bigoplus$  and low-resolution vectors are averaged; eq. (5). Finally, we concatenate all of these (zero-padded) relative-vectors to obtain the final vector representation of the whole slice,  $\vec{s}_i$ ; eq. (6). At a high level, this vector essentially specifies a deterministic, structured, typed, discrete self-attention over the token history.
$$
\vec {s} _ {i, \text {r e l}} ^ {\text {H i R e s}} = \bigoplus_ {k \in \text {H i R e s} _ {i, \text {r e l}}} \left[ \vec {l} _ {k}; \vec {w} _ {k} \right]
$$
$$
\vec {s} _ {i, \text {r e l}} ^ {\text {L o R e s}} = \sum_ {k \in \text {L o R e s} _ {i, \text {r e l}}} \frac {\left[ \vec {l} _ {k} ; \vec {w} _ {k} \right]}{\left| \text {L o R e s} _ {i , \text {r e l}} \right| _ {+}} \tag {5}
$$
$$
\vec {s} _ {i} = \bigoplus_ {\text {r e l} \in \text {R E L}} \left[ \vec {s} _ {i, \text {r e l}} ^ {\text {H i R e s}}; \vec {s} _ {i, \text {r e l}} ^ {\text {L o R e s}} \right] \tag {6}
$$
4.3 Predicting Emission Distributions
We compute model posteriors for next-token predictions as
$$
P _ {\mu} (w _ {i} = v _ {k} | \mathrm {c o n t e x t} _ {i, \mu}) = \mathrm {S o f t M a x} (\log \mathrm {i t s} _ {i, \mu}) [ k ],
$$
where  $\mu$  is either a pure SLR model or LM, or an ensemble of the two (bottom right of figure 1).
SLR only. As described above, we define context $_{i,SLR}$  as  $G_{i}$ , which is encoded as  $\vec{s_i}$ . We obtain  $P_{SLR}$  by letting the slice-vectors serve as inputs to a  $d$ -multilayer perceptron (MLP) with a final softmax layer over the vocabulary, which yields the estimated token emission distributions.
$$
\mathrm {l o g i t s} _ {i, S L R} = \mathrm {M L P} _ {d} (\vec {s _ {i}})
$$
$$
\mathrm {M L P} _ {d} (x) = H ^ {(d)} \left(\dots H ^ {(1)} (x)\right) \mathrm {E m b} ^ {\top},
$$
where Emb is an embedding matrix.
$\mathbf{LM} + \mathbf{SLR}$ . Since we want to measure whether and how much the information contained in the SLR can contribute to state-of-the-art language models, our primary experimental condition is a combined setup  $P_{\text{Ensemble}}$ , where logits obtained
Sentences	Tokens	Vocabulary
Train	26,325	658,475	27,344
Train (EarlyStop)	23,692	591,829	26,422
Dev (EarlyStop)	2,633	66,646	10,073
Eval	921	22,596	5,364
from slice-encodings are added to a base neural LM's logits before taking the softmax:
$$
\mathrm {l o g i t s} _ {i, E n s e m b l e} = \mathrm {l o g i t s} _ {i, S L R} + \mathrm {l o g i t s} _ {i, L M},
$$
$$
\text {w i t h} \quad \operatorname {l o g i t s} _ {i, L M} = \operatorname {L M} (\mathbf {w} _ {<   i}).
$$
LM only.  $P_{LM}$ , i.e., the bare LM without any exposure to SLR graphs, serves as a baseline.
All models are implemented in PyTorch and experiments are run on 1 NVIDIA Tesla T4 GPU. Model hyperparameters are reported in appendix A.5.
Our dataset consists of the intersection of Wall Street Journal (WSJ; English financial news) sentences that have been annotated with syntactic trees in the Penn Treebank (PTB; Marcus et al., 1993; Hovy et al., 2006) $^{6}$  as well as a range of semantic representation formalisms for the MRP 2019 & 2020 shared tasks (Oepen et al., 2019, 2020). Summary statistics are shown in table 1. Our preprocessing steps are described in appendix B.
5.2 SLR Formalisms
The 7 (versions of) linguistic representation frameworks examined in this study are listed in table 2, along with their classifications along the scope and structure dimensions. We draw the structural dependencies vs. constituencies distinction (described at a high level in §2) based on specific properties of the MRP shared task data: a framework is considered a dependency framework if all edges are only between pairs of individual word anchors at a time; if there are any unanchored<sup>7</sup> nodes or nodes anchored in more than one linguistic word token, it is considered a constituency framework.<sup>8</sup> Below we give a brief description of each framework.
PTB trees specify hierarchically nested syntactic constituents. We consider two labeling variants: basic phrase structure (-phr) and phrase types refined with functional specifications (-fxn).
See appendix A.3 for details.
$^{6}$ https://catalog.ldc.upenn.edu/LDC2013T19
7Not including "ROOT" nodes in UD.
See appendix A.4 for details.
Universal Dependencies (UD; Nivre et al., 2016, 2020; de Marneffé et al., 2021) is a syntactic dependency representation with coarse, cross-linguistically applicable edge labels.
DELPH-IN MRS Bi-Lexical Dependencies (DM; Ivanova et al., 2012) and Elementary Dependency Structures (EDS; Oepen and Lonnning, 2006) are derived from underspecified logical forms computed by the English Resource Grammar (Flickinger, 2000; Copestake et al., 2005).
Prague Semantic Dependencies (PSD; Hajic et al., 2012) and Prague Tectogrammatical Graphs (PTG) are syntactico-semantic predicate–argument structures converted from the Prague Functional Generative Description (Sgall et al., 1986; Böhmová et al., 2003; Hajic et al., 2012).
5.3 Language Model
The base language model we use in all our experiments is GPT-2 (Radford et al., 2019, as distributed in the huggingface-transformers PyTorch library). GPT-2 is a Transformer model (Vaswani et al., 2017) pretrained on a diverse collection of web texts. In contrast to other widely-used Transformers like BERT (Devlin et al., 2019), which optimize bidirectional masked language modeling, GPT-2 is incremental, i.e., next-word decisions only take into account the preceding context.
5.4 Training
We train all models for 10 epochs with the AdamW optimizer (Loshchilov and Hutter, 2019), minimizing cross-entropy between the model posterior and the ground truth at each token position.
We perform early stopping with the last  $10\%$  of the original training corpus set aside for development scoring after each epoch. We keep the model state that achieves the best perplexity on the dev set. Peak development performance is reached after  $\approx 3$  epochs for SLR models, whereas finetuning GPT-2 by itself takes between 7 and 9 epochs.
5.5 Evaluation
We compute model perplexity (PPL) as the most standard language modeling evaluation measure, as well as accuracy (Acc) and confidence (Conf) of a model's top-ranked guess, mean reciprocal rank of the correct answer (MRR), and entropy of the
model's token prediction posterior (H). All metrics are reported as microaverages over the evaluation data at the BPE token level. $^{10}$
The most striking observation in terms of overall model performance (table 2) is that ground-truth linguistic graphs of all investigated linguistic formalisms improve vanilla GPT-2 by a large margin, in all metrics. This improvement holds up when compared to a version of GPT-2 that is exposed to the raw WSJ text without the graphs; with this condition we control for mere domain differences between our evaluation data and the data GPT-2 was trained on originally ('+Domain' in table 2). The large performance gap suggests that at least a subset of the oracle knowledge about linguistic structure is not yet encoded in the base language model, which learns from only raw text.
We observed that if we keep training for the entirety of 10 epochs, rather than early stopping based on development performance, we somewhat overfit to the training set. While accuracy itself is not affected very much by this, the models become increasingly overconfident ( $\frac{\text{overall confidence}}{\text{overall accuracy}}$ , which gets up to  $8 - 12\%$ , compared to  $\approx 4\%$  with the vanilla GPT-2 model and in most cases even slightly less than that with the early-stopped SLR models). This leads to overall worse perplexity.
6.2 Differences between Formalisms
Comparing across rows in table 2, we find a considerable performance spread. The general trend, which is relatively consistent in all metrics,[11] is indicated by the order of rows, with UD having the smallest (though still respectable) improvement over the baseline, and PTG and EDS the largest.
Interestingly, there are two marked separations: a primary one between dependency and constituency formalisms, and a secondary one between syntactic (i.e., more surface-oriented) and semantic (more abstract) formalisms. This is summarized
10We compute average PPL over all sentences j by exponentiating last:  $\exp \left(\frac{1}{\sum_{j}|\mathbf{w}_{j}|}\sum_{j}\sum_{i = 0}^{|\mathbf{w}_{j}| - 1} - \log P_{\mu}\left(w_{ji} = v_{ji}^{*}|\mathrm{context}_{ji,\mu}\right)\right)$
11 The multitude of metrics might thus seem redundant. But since each measurement emphasizes different properties of model performance, we consider it a very interesting result (and, potentially, a success of our modeling technique and experimental setup) to achieve this broad consistency.
<sup>9</sup>The R-GCN baseline (table 6) is always trained for the full 10 epochs, but to ensure fairness, it is also only compared to concatenation-based encoders that have been trained for the full 10 epochs, too.
Model	Scope/Struct	#Labels	Training Efficiency	Language Model Quality
Speed ↑	Size ↓	PPL ↓	H [nats] ↓	Acc [%] ↑	Conf [%] ↑	MRR ↑
GPT-2 + Domain	-15	124.4M	59.3	4.09	30.0	31.2	.403
45.9 ±.09	3.64 ±.008	33.3 ±.02	34.9 ±.07	.435 ± .3e-3
***	***
+ UD	syn	dep	39	14	+54.1M	32.7 ±.18	3.30 ±.013	39.1 ±.15	40.1 ±.14	.486 ±1.2e-3
***	-
+ DM	sem	dep	59	15	+54.4M	31.4 ±.08	3.24 ±.026	38.9 ±.10	40.2 ±.37	.491 ± .6e-3
***	-
+ PSD	sem	dep	90	16	+54.9M	30.7 ±.09	3.21 ±.014	39.1 ±.11	40.9 ±.11	.491 ± .5e-3
**	***
+ PTB-phr	syn	const	38	14	+54.1M	29.8 ±.18	3.14 ±.029	41.2 ±.19	42.8 ±.42	.507 ±1.3e-3
***	*
+ PTB-fxn	syn	const	537	14	+62.7M	29.0 ±.28	3.07 ±.049	42.0 ±.30	43.8 ±.60	.514 ±1.8e-3
***	*
+ PTG	sem	const	72	15	+54.6M	26.8 ±.26	3.03 ±.041	43.1 ±.12	44.6 ±.51	.522 ± .9e-3
***	-
+ EDS	sem	const	10	15	+53.6M	24.7 ±.28	2.92 ±.048	43.1 ±.17	45.0 ±.55	.527 ±1.3e-3