SlowGuess/ACL_2020_OCR huggingface dataset:
Jennifer Hu $^{1}$ , Jon Gauthier $^{1}$ , Peng Qian $^{1}$ , Ethan Wilcox $^{2}$ , and Roger P. Levy $^{1}$ \
 $^{1}$ Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology\
 $^{2}$ Department of Linguistics, Harvard University\
{jennhu, pqian, rplevy}@mit.edu\
jon@gauthiers.net, wilcoxeg@g.harvard.edu
While state-of-the-art neural network models continue to achieve lower perplexity scores on language modeling benchmarks, it remains unknown whether optimizing for broad-coverage predictive performance leads to human-like syntactic knowledge. Furthermore, existing work has not provided a clear picture about the model properties required to produce proper syntactic generalizations. We present a systematic evaluation of the syntactic knowledge of neural language models, testing 20 combinations of model types and data sizes on a set of 34 English-language syntactic test suites. We find substantial differences in syntactic generalization performance by model architecture, with sequential models underperforming other architectures. Factorially manipulating model architecture and training dataset size (1M-40M words), we find that variability in syntactic generalization performance is substantially greater by architecture than by dataset size for the corpora tested in our experiments. Our results also reveal a dissociation between perplexity and syntactic generalization performance.
A growing body of work advocates that assessment of neural language models should include both information-theoretic metrics, such as perplexity, as well as targeted linguistic evaluation. Benchmarks such as GLUE (Wang et al., 2019a,b) have demonstrated that neural language models trained on naturalistic corpora for next-word prediction learn representations that can yield remarkable performance on many semantic tasks. Targeted syntactic evaluations have shown that these models also implicitly capture many syntactic generalizations, ranging from subject-verb agreement
Materials and code can be found at https://github.com/cpllab/syntactic-generalization.
to long-distance filler-gap dependencies (Linzen et al., 2016; Marvin and Linzen, 2018; Futrell et al., 2018; Wilcox et al., 2019b). This paper aims to bring targeted evaluations of syntactic performance to scale, complementing similar developments in semantic evaluation (McCoy et al., 2019).
Because the most widespread currency of evaluation for language models is perplexity—how well, on average, a model predicts a word in its context—a primary focus of this paper is the relationship between a model's perplexity and its performance on targeted syntactic evaluations. As perplexity improves, can we expect more human-like syntactic generalization? How do training dataset size and model architecture jointly affect syntactic generalization? And what picture of models' syntactic generalization emerges when evaluation is brought to scale, across dozens of controlled syntactic tests?
In this paper we offer initial answers to these questions, systematically assessing the syntactic generalization abilities of neural language models on 34 targeted test suites (33 adapted from previously published work, and 1 novel) covering a wide range of syntactic phenomena. Test suites are written using a standard format that allows for flexible predictions which more closely resemble those used in psycholinguistic studies, specifically allowing for predictions about interactions among multiple testing conditions. Performance on each test suite is reported as a Syntactic Generalization (SG) score. We group test suites into six syntactic circuits based on the linguistic representations needed to achieve high performance on each suite.
We train four classes of neural models and one baseline  $n$ -gram model on four datasets derived from a newswire corpus, consisting of 1, 5, 14, and 42 million tokens. While previous work has compared model architectures for a fixed dataset size (e.g. Wilcox et al., 2019b) and network sizes for a fixed architecture (e.g. van Schijndel et al.,
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725-1744 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
2019), our controlled regime allows us to make an apples-to-apples comparison across model architectures on a range of sizes. In addition, we evaluate several off-the-shelf models which were trained on datasets ranging up to 2 billion tokens.
Our results address the three questions posed above: First, for the range of model architectures and dataset sizes tested, we find a substantial dissociation between perplexity and SG score. Second, we find a larger effect of model inductive bias than training data size on SG score, a result that accords with van Schijndel et al. (2019). Models afforded explicit structural supervision during training outperform other models: One structurally supervised model is able to achieve the same SG scores as a purely sequence-based model trained on  $\sim 100$  times the number of tokens. Furthermore, several Transformer models achieve the same SG score as a Transformer trained on  $\sim 200$  times the amount of data. Third, we find that architectures have different relative advantages across types of syntactic tests, suggesting that the tested syntactic phenomena tap into different underlying processing capacities in the models.
2.1 Perplexity
Standard language models are trained to predict the next token given a context of previous tokens. Language models are typically assessed by their perplexity, the inverse geometric mean of the joint probability of words  $w_{1}, \ldots, w_{N}$  in a held-out test corpus  $C$ :
$$
\operatorname {P P L} (C) = p \left(w _ {1}, w _ {2}, \dots w _ {N}\right) ^ {- \frac {1}{N}} \tag {1}
$$
Models with improved perplexity have also been shown to better match various human behavioral measures, such as gaze duration during reading (Frank and Bod, 2011; Fossum and Levy, 2012; Goodkind and Bicknell, 2018; Wilcox et al., 2020). However, a broad-coverage metric such as perplexity may not be ideal for assessing human-like syntactic knowledge for a variety of reasons. In principle, a sentence can appear with vanishingly low probability but still be grammatically well-formed, such as Colorless green ideas sleep furiously (Chomsky, 1957). While perplexity remains an integral part of language model evaluation, fine-grained linguistic assessment can provide both more challenging and more interpretable tests to evaluate neural models.
2.2 Targeted tests for syntactic generalization
Alternatively, a language model can be evaluated on its ability to make human-like generalizations for specific syntactic phenomena (Linzen et al., 2016; Lau et al., 2017; Gulordava et al., 2018). The targeted syntactic evaluation paradigm (Marvin and Linzen, 2018; Futrell et al., 2019) incorporates methods from psycholinguistic experiments, designing sentences which hold most lexical and syntactic features of each sentence constant while minimally varying features that determine grammaticality or surprise characteristics of the sentence. For example, given the two strings *The keys to the cabinet are on the table and *The keys to the cabinet is on the table, a model that has learned the proper subject-verb number agreement rules for English should assign a higher probability to the grammatical plural verb in the first sentence than to the ungrammatical singular verb in the second (Linzen et al., 2016).
Although some targeted syntactic evaluations, such as the example discussed above, involve simple comparisons of conditional probabilities of a word in its context, other evaluations are more complex. We can demonstrate this with an evaluation of models' "garden-pathing" behavior (Futrell et al., 2019). For example, the sentence The child kicked in the chaos found her way back home yields processing disruption for humans at the word found. This is because, up to right before that word, the part-of-speech ambiguous kicked is preferentially interpreted as the main verb of the sentence, whereas it turns out to be a passive participle in a reduced relative clause modifying child. This garden-path disambiguation effect is ameliorated by replacing kicked with forgotten, which is not part-of-speech ambiguous (B below; Trueswell et al., 1994) or by using an unreduced relative clause (C below; Ferreira and Clifton, 1986). In probabilistic language models, these garden-path disambiguation effects are well captured by word negative log probabilities, or SURPRISALS (Hale, 2001):  $S(w|C) = -\log_2p(w|C)$ , which are independently well-established to predict human incremental processing difficulty over several orders of magnitude in word probability (Smith and Levy, 2013). A targeted syntactic evaluation for garden-pathing is provided by comparing surprisals at the disambiguating word found in the set of four examples below (Futrell et al., 2019):
(A) The child kicked in the chaos found ...
(B) The child forgotten in the chaos found ...
(C) The child who was kicked in the chaos found ...
(D) The child who was forgotten in the chaos found ...
Successful human-like generalization involves three criteria: (i) found should be less surprising (i.e., more probable) in B than A; (ii) found should be more probable in C than A; (iii) the C-D surprisal difference should be smaller than the A-B surprisal difference—a  $2 \times 2$  interaction effect on surprisal—because the syntactic disambiguation effect of not reducing the relative clause was achieved by using a part-of-speech unambiguous verb.
We will use these controlled tests to help us describe and test for human-like syntactic knowledge in language models.
The testing paradigm presented here differs in several crucial ways from recent, related syntactic assessments and provides complementary insights. Unlike Warstadt et al. (2019a), our approach does not involve fine-tuning, but rather assesses what syntactic knowledge is induced from the language modeling objective alone. The most closely related work is the Benchmark of Linguistic Minimal Pairs (Warstadt et al., 2020), which is a challenge set of automatically-generated sentence pairs also designed to test language models on a large set of syntactic phenomena. Our approach differs in important ways: we compare critical sentence regions instead of full-sentence probabilities, and employ a  $2 \times 2$  paradigm with a strict, multi-fold success criterion inspired by psycholinguistics methodology. This allows us to factor out as many confounds as possible, such as the lexical frequency of individual tokens and low-level  $n$ -gram statistics.
We designed a controlled paradigm for systematically testing the relationship between two design choices — model class and dataset size — and two performance metrics — perplexity and syntactic generalization capacity. Section 3.1 describes the test suites collected for our evaluation, and Sections 3.2 and 3.3 describe the datasets and model classes investigated.
3.1 Test suites
We assemble a large number of test suites inspired by the methodology of experimental sentence-processing and psycholinguistic research. Each
test suite contains a number of ITEMS (typically between 20 and 30), and each item appears in several CONDITIONS: across conditions, a given item will differ only according to a controlled manipulation designed to target a particular feature of grammatical knowledge. Each test suite contains at least one PREDICTION, which specifies inequalities between surprisal values at pairs of regions/conditions that should hold if a model has learned the appropriate syntactic generalization.
We expect language models which have learned the appropriate syntactic generalizations from their input to satisfy these inequalities without further fine-tuning. We compute accuracy on a test suite as the proportion of items for which the model's behavior conforms to the prediction. Most of our test suites involve  $2 \times 2$  designs and a success criterion consisting of a conjunction of inequalities across conditions, as in the garden-pathing example described in Section 2.2.<sup>1</sup> Random baseline accuracy varies by test suite and is  $\sim 25\%$  overall. Most of these test suites and criteria are designed so that  $n$ -gram models cannot perform above chance for  $n = 5$  (sometimes greater).
Syntactic coverage In order to assess the coverage of our test suites, we manually inspected the phenomena covered in Carnie (2012), a standard introductory syntax textbook. Of the 47 empirical phenomena reviewed in the summary sections at the end of each chapter, our tests target  $16\left(\sim 34\%\right)$ . These are evenly distributed across the whole range of subject matter, with tests targeting phenomena in 11 of the 15 chapters  $(\sim 73\%)$ .<sup>2</sup>
Modifiers Five test suites include paired modifier versions, where extra syntactically irrelevant (but semantically plausible) content, such as a prepositional phrase or relative clause, is inserted before the critical region being measured. We use these paired test suites to evaluate models' stability to intervening content within individual syntactic tests.
Circuits The test suites are divided into 6 syntactic circuits, based on the type of algorithm required to successfully process each construction. We give a brief overview of each circuit below.
- Agreement is a constraint on the feature values of two co-varying tokens. For example,
<sup>1</sup>The exception is Center Embedding, which features a 2-condition design with a single-inequality criterion.
2For more details on this analysis, see Appendix A.
<sup>3</sup>A full overview of our test suites is given in Appendix B.
the number feature of a verb must agree with the number feature of its upstream subject. We include 3 Subject-Verb Number Agreement suites from Marvin and Linzen (2018).
- Licensing occurs when a particular token must exist within the scope of an upstream licensor token. Scope is determined by the tree-structural properties of the sentence. Test suites include Negative Polarity Item Licensing (NPI) (4 suites) and Reflexive Pronoun Licensing (6 suites), both from Marvin and Linzen (2018).
- Garden-Path Effects are well-studied syntactic phenomena that result from tree-structural ambiguities that give rise to locally-coherent but globally implausible syntactic parses. Garden-path test suites include Main Verb / Reduced Relative Clause (MVRR) (2 suites) and NP/Z Garden-paths (NPZ) (4 suites), both from Futrell et al. (2018).
- Gross Syntactic Expectation is a processor's expectation for large syntactic chunks such as verb phrases or sentences, and are often set up by subordinating conjunctions such as while, although and despite. Our tests for gross syntactic expectation include Subordination (4 suites) from Futrell et al. (2018).
- Center Embedding sentences are sentences recursively nested within each other. Subject and verbs must match in a first-in-last-out order, meaning models must approximate a stack-like data-structure in order to successfully process them. Our 2 suites of Center Embedding sentences come from the items presented in Wilcox et al. (2019a).
- Long-Distance Dependencies are covariations between two tokens that span long distances in tree depth. Test suites include Filler-Gap Dependencies (FGD) (6 suites) from Wilcox et al. (2018) and Wilcox et al. (2019b), and 2 novel Cleft suites, described in detail below.
Novel test suite: Cleft We introduce one novel test suite that assesses models' ability to process pseudo-cleft constructions, which are used to put a particular syntactic constituent into focus via passive transformation. Consider Example (1):
BLLIP sizes:	XS	SM	MD	LG
# sentences	40K	200K	600K	1.8M
# tokens	1M	4.8M	14M	42M
# non-UNK types	24K	57K	100K	170K
# UNK types	68	70	71	74
Table 1: Statistics of training set for each corpus size.
(1) a. What he did after coming in from the rain was eat a hot meal. [DO/VP]
b. *What he devoured after coming in from the rain was eat a hot meal. [LEX/VP]
c.*What he did after coming in from the rain was a hot meal. [DO/NP]
d. What he devoured after coming in from the rain was a hot meal. [LEX/NP]
When this constituent is a verb, it must be replaced in the wh-clause that heads the sentence with the DO verb, as in (1a), below. However, when it is a noun, the lexical verb for which it serves as an object must be preserved, as in (1d). If models have properly learned the pseudo-cleft construction, then DO verbs should set up expectations for VPs (the region in bold should have a lower surprisal in (1a) than in (1b)) and lexicalized verbs should set up expectations for NPs (the region in bold should have a lower surprisal in (1d) than in (1c)).
3.2 Model training data
Corpora We train and evaluate models on English newswire corpora of four different sizes, obtained by randomly sampling sections from the Brown Laboratory for Linguistic Information Processing 1987-89 Corpus Release 1 (BLLIP; Charniak et al., 2000). The corpora are sampled such that the training set of each corpus is a proper subset of each larger corpus. We call these four corpora BLLIP-xs (40K sentences, 1M tokens); BLLIP-sm (200K sentences, 5M tokens); BLLIP-MD (600K sentences, 14M tokens); and BLLIP-LG (2M sentences, 42M tokens). Table 1 summarizes statistics of the training set for each corpus.
To ensure consistency in perplexity evaluation across datasets, we report perplexity scores achieved by the models on a shared held-out test set. We additionally use a shared held-out validation for tuning and early stopping.
We use the NLTK implementation of the Penn Treebank tokenizer to process all datasets (Bird and Loper, 2004; Marcus et al., 1993).
# layers	# hidden units	Embedding size
LSTM	2	256	256
ON-LSTM	3	1150	400
RNNG	2	256	256
GPT-2	12	768	768
Table 2: Size of neural models in our controlled experiments.
BLLIP sizes:	XS	SM	MD	LG
LSTM	13.4M	30.5M	52.2M	88.1M
ON-LSTM	30.8M	44.2M	61.2M	89.2M
RNNG	22.8M	48.4M	81.1M	134.9M
GPT-2	124.4M	124.4M	124.4M	124.4M
Out-of-vocabulary tokens For each corpus, we designate a token as OOV if the token appears fewer than two times in the training set. Our larger training datasets thus contain larger vocabularies than our smaller training datasets. This allows larger-training-set models to learn richer word-specific information, but may also harm perplexity evaluation because they have vocabulary items that are guaranteed to not appear in the BLLIP-xs test set. This means that perplexity scores across training dataset sizes will not be strictly comparable: if a larger-training-set model does better than a smaller-training-set model, we can be confident that it has meaningfully lower perplexity, but the reverse is not necessarily the case. The exception to the above is GPT-2, which uses sub-words from byte-pair encoding and has no OOVs (see also Footnote 6).
Unkification We follow the convention used by the Berkeley parser (Petrov and Klein, 2007), which maps OOVs to UNK classes which preserve fine-grained information such as orthographic case distinctions and morphological suffixes (e.g. UNK-ed, UNK-ly). Before training, we verified that the UNK classes in the test and validation sets were all present in the training set.
3.3 Model classes
In order to study the effects of model inductive bias and dataset size, we trained a fleet of models with varying inductive biases on each corpus. Because many of our test suites exploit ambiguities that arise from incremental processing, we restrict evaluation to left-to-right language models; future
Table 3: Parameter counts for neural models in our controlled experiments.
BLLIP sizes:	XS	SM	MD	LG
LSTM	98.19	65.52	59.05	57.09
ON-LSTM	71.76	54.00	56.37	56.38
RNNG	122.46	86.72	71.12	69.57
GPT-2	529.90	183.10	37.04	32.14
n-gram	240.21	158.60	125.58	106.09
Table 4: Perplexity averages achieved by each controlled model on each corpus. Perplexity scores across training dataset sizes are not always strictly comparable (see Section 3.2).
work could involve evaluation of bidirectional models (Devlin et al., 2018; Yang et al., 2019) on an appropriate subset of our test suites, and/or adaptation of our suites for use with bidirectional models (Goldberg, 2019). Training ran until convergence of perplexity on a held-out validation set. Wherever possible, we trained multiple seeds of each model class and corpus size. We use the model sizes and training hyperparameters reported in the papers introducing each model (Table 2). The full parameter counts and perplexity scores for each model  $\times$  corpus combination are given in Tables 3 and 4, respectively.
LSTM Our baseline neural model is a vanilla long short-term memory network (LSTM; Hochreiter and Schmidhuber, 1997) based on the boiler-plate PyTorch implementation (Paszke et al., 2017).
Ordered-Neurons We consider the Ordered-Neurons LSTM architecture (ON-LSTM; Shen et al., 2019), which encodes an explicit bias towards modeling hierarchical structure.
RNNG Recurrent neural network grammars (RNNG; Dyer et al., 2016) model the joint probability of a sequence of words and its syntactic structure. RNNG requires labeled trees that contain complete constituency parses, which we produce for BLLIP sentences with an off-the-shelf constituency parser (Kitaev and Klein, 2018). To compute surprisesals from RNNG, we use word-synchronous beam search (Stern et al., 2017) to approximate the conditional probability of the current word given the context.
Due to computational constraints, we performed only minimal tuning past these recommended hyperparameters.
5While the BLLIP corpus already contains Treebank-style parses, we strip the terminals and re-parse in order to obtain more accurate, up-to-date syntactic parses.
Figure 1: Average SG score by model class. Asterisks denote off-the-shelf models. Error bars denote bootstrapped  $95\%$  confidence intervals of the mean.
Transformer Transformer models (Vaswani et al., 2017) have recently gained popularity in language processing tasks. We use GPT-2 (Radford et al., 2019) as a representative Transformer model and train it from scratch on our BLLIP corpora. $^{6}$
$n$ -gram As a baseline, we consider a 5-gram model with modified Kneser-Ney smoothing.
3.4 Off-the-shelf models
We also test five off-the-shelf models: GRNN, trained on 90M tokens from Wikipedia (Gulordava et al., 2018); JRNN, trained on 800M tokens from the 1 Billion Word Benchmark (Jozefowicz et al., 2016); Transformer-XL, trained on 103M tokens from WikiText-103 (Dai et al., 2019); and the pretrained GPT-2 and GPT-2-XL, trained on 40GB of web text (Radford et al., 2019). These models are orders of magnitude larger than our controlled ones in parameter count and/or training set size.
Figure 1 shows the average accuracy of all models on the complete set of SG test suites. Asterisks denote off-the-shelf models. All neural models achieve a SG score significantly greater than a random baseline (dashed line). However, the range within neural models is notable, with the best-performing model (GPT-2-XL) scoring over twice as high as the worst-performing model (LSTM). Also notable are the controlled GPT-2 and RNNG models, which achieve comparable performance to Transformer-XL and JRNN, despite being trained on significantly smaller data sizes.
Figure 2: Relationship between SG score and perplexity on our held-out BLLIP test set for each model.
We now return to the three major issues presented in Section 1. In 4.1 we present evidence that SG score is dissociated from perplexity. In 4.2 we argue that model architecture accounts for larger gains in SG score than amount of training data. And in 4.3 we show that this cross-architecture difference is due largely to variance on a handful of key test suites.
4.1 Syntactic generalization and perplexity