Data	The *X dataset consists of 67 paradigms of 1000 sentence pairs. Each paradigm is annotated for the unique contrast it isolates and the broader category of phenomena it is part of. The data is automatically generated according to expert-crafted grammars, and our automatic labels are validated with crowd-sourced human judgments.

BLiMP: A Benchmark of Linguistic Minimal Pairs for English	We introduce The Benchmark of Linguistic Minimal Pairs (shortened to BLiMP), a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets, each containing 1000 minimal pairs isolating specific contrasts in syntax, morphology, or semantics. The data is automatically generated according to expert-crafted grammars, and aggregate human agreement with the labels is 96.4%. We use it to evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs. We find that state-of-the-art models identify morphological contrasts reliably, but they struggle with semantic restrictions on the distribution of quantifiers and negative polarity items and subtle syntactic phenomena such as extraction islands.

Data ::: Data generation procedure	To create minimal pairs exemplifying a wide array of linguistic contrasts, it is necessary to artificially generate all datasets. This ensures both that we have sufficient unacceptable examples, and that the data is fully controlled, allowing for repeated isolation of a single linguistic phenomenon in each paradigm . The data generation scripts use a basic template to create each paradigm, pulling from a vocabulary of over 3000 words annotated for morphological, syntactic, and semantic features needed to create grammatical and semantically felicitous sentences. Examples SECREF6 and SECREF6 show one such template for the `acceptable' and `unacceptable' sentences within a pair: the sole difference between them is the underlined word, which differs only in whether the anaphor agrees in number with its antecedent. Our generation codebase and scripts are freely available. . DP1 V1 refl_match . The cats licked themselves . . DP1 V1 refl_mismatch . The cats licked itself . This generation procedure is not without limitations, and despite the very detailed vocabulary we use, implausible sentences are occasionally generated (e.g., `Sam ran around some glaciers').

For example, a model may predict that `The cat ate the mouse' is more likely than `The cat eaten the mouse' without correctly predicting that $P(\emph {ate}|\emph {the cat}) > P(\emph {eaten}|\emph {the cat})$ if it predicts that $P(\emph {the mouse}|\emph {the cat ate})$ is much greater than $P(\emph {the mouse}|\emph {the cat eaten})$. Furthermore, it is unclear how a model assigns probabilities conditioned on an ungrammatical prefix, since ungrammatical sentences are largely absent from the training data. Using prefix probabilities allow us to exclude models' use of this additional information and evaluate how the models perform when they have just enough information to judge grammaticality.

Introduction	Current neural networks for language understanding rely heavily on unsupervised pretraining tasks like language modeling. However, it is still an open question what degree of knowledge state-of-the-art language models (LMs) acquire about different linguistic phenomena. Many recent studies , ,  have advanced our understanding in this area by evaluating LMs' preferences between minimal pairs of sentences, as in Example SECREF1. However, these studies have used different analysis metrics and focused on a small set of linguistic paradigms, making a big-picture comparison between these studies limited. . Ṫhe cat annoys Tim. (grammatical) The cat annoy Tim. (ungrammatical) We introduce the Benchmark of Linguistic Minimal Pairs (shortened to BLiMP or just *X ) a linguistically-motivated benchmark for assessing LMs' knowledge across a wide variety of English phenomena, encapsulating both previously studied and novel contrasts. *X consists of 67 datasets automatically generated from expert-crafted grammars, each containing 1000 minimal pairs and organized by phenomenon into 12 categories. Validation with crowd workers shows that humans overwhelmingly agree with the contrasts in *X .

Additional Experiments ::: Training size and *X performance	We also use *X to track how a model's knowledge of particular phenomena varies with the quantity of training data. We test this with the LSTM model and find that different phenomena in *X have notably different learning curves across different training sizes, as shown in Figure FIGREF39. Crucially, phenomena with similar results from the LSTM model trained on the full 83M tokens of training data may have very different learning curves. For example, the LSTM model performs well on both irregular forms and anaphor agreement, but the different learning curves suggest that more training data is required in the anaphor agreement case to achieve this same performance level. This is supported by a regression analysis showing that the best-fit line for anaphor agreement has the steepest slope (0.0623), followed by Determiner-Noun agreement (0.0426), Subject-Verb agreement (0.041), Irregular (0.039) and Ellipsis (0.0389).

By contrast, Binding (0.016), Argument Structure (0.015), and Filler-Gap Dependency (0.0095) have shallower learning curves, showing a less strong effect of increases in training data size. The phenomena that showed the lowest performance overall, NPIs and Islands, also show the lowest effects of increases to training size, with slopes of 0.0078 and 0.0036, respectively. This indicates that, even given a substantially larger amount training data, the LSTM is unlikely to achieve human-like performance on these phenomena – it simply fails to learn the necessary dependencies. It should be noted that these differences in learning curves show how *X performance dissociates from perplexity, the standard measure of LM performance: while perplexity keeps decreasing as training size increases, the performance in different *X phenomena show very different learning curves.

Data ::: Coverage	The paradigms that are covered by *X represent well-established contrasts in English morphology, syntax, and semantics. Each paradigm is grouped into one of 12 phenomena, shown in Table TABREF1. The paradigms are selected with the constraint that they can be illustrated with minimal pairs of equal sentence length and that it is of a form that could be written as a template, like in SECREF6 and SECREF6. While this dataset has broad coverage, it is not exhaustive – it is not possible to include every grammatical phenomenon of English, and there is no agreed-upon set of core phenomena. However, we consider frequent inclusion of a phenomenon in a syntax/semantics textbook as an informal proxy for what linguists consider to be core phenomena. We survey several syntax textbooks , , , and find that nearly all of the phenomena in *X are discussed in some source, and most of the topics that repeatedly appear in textbooks and can be represented with minimal pairs (e.g. agreement, argument selection, control/raising, wh-extraction/islands, binding) are present in *X . Because the generation code is reusable, it is possible to generate paradigms not included in *X in the future.

Data ::: Comparison to Related Resources	With over 3000 words, *X has by far the widest lexical variability of any related generated dataset. The vocabulary includes verbs with 11 different subcategorization frames, including verbs that select for PPs, infinitival VPs, and embedded clauses. By comparison, datasets by  and  each use a vocabulary of well under 200 items. Other datasets of minimal pairs that achieve greater lexical and syntactic variety use data-creation methods that are limited in terms of empirical scope or control.  construct a dataset of minimal pairs for subject-verb agreement by changing the number marking on present-tense verbs in a subset of English Wikipedia. However this approach does not generalize beyond simple agreement phenomena.  build a dataset of minimal pairs by taking sentences from the BNC through round-trip machine translation. The resulting sentences contain a wider variety of grammatical violations, but it is not possible to control the nature of the violation and a single sentence may contain several violations.

Data ::: Data validation	To verify that the generated sentences represent a real contrast in acceptability, we conduct human validation via Amazon Mechanical Turk. Twenty separate validators rated five pairs from each of the 67 paradigms, for a total of 6700 judgments. We restricted validators to individuals currently located in the US who self-reported as native speakers of English. To assure that our validators made a genuine effort on the task, each HIT included an attention check item and a hidden field question to catch bot-assisted humans. For each minimal pair, 20 different individuals completed a forced-choice task that mirrors the task done by the LMs; the human-determined “acceptable” sentence was calculated via majority vote of annotators. By this metric, we estimate aggregate human agreement with our annotations to be 96.4% overall. As a threshold of inclusion in *X , the majority of validators needed to agree with *X on at least 4/5 examples from each paradigm. Thus, all 67 paradigms in the public version of *X passed this validation, and only two additional paradigms had to be rejected on this criterion.

Results ::: Correlation of Model & Human Performance	We also examine to what extent the models' performances are similar to each other, and how they are similar to human evaluation in terms of which phenomena are comparatively difficult. Figure TABREF29 shows the Pearson correlation between the four LMs and human evaluation on their accuracies in 67 paradigms. Compared to humans, GPT-2 has the highest correlation, closely followed by Transformer-XL and LSTM, though the correlation is only moderate. The $n$-gram's performance correlates with humans relatively weakly. Transformer-XL and LSTM are very highly correlated at 0.9, possibly reflecting their similar training data. Also, neural models correlate with each other more strongly than with humans or the $n$-gram model, suggesting neural networks share some biases that are not entirely human-like. white

Discussion & Future Work	We have shown ways in which *X can be used as tool to gain both high-level and fine-grained insight into the grammatical knowledge of language models. Like the GLUE benchmark , *X assigns a single overall score to an LM which summarizes its general sensitivity to minimal pair contrasts. Thus, it can function as a linguistically motivated benchmark for the general evaluation of new language models. *X also provides a breakdown of LM performance by linguistic phenomenon, which can be used to draw concrete conclusions about the kinds of grammatical knowledge acquired by a given model. This kind of information is useful for detailed comparisons across models, as well as in ablation studies. One question we leave unexplored is how well supervised acceptability classifiers built on top of pretrained models like BERT  perform on *X . It would be possible to evaluate how well such classifiers generalize to unseen phenomena by training on a subset of paradigms in *X and evaluating on the held-out sets, giving an idea of to what extent models are able to transfer knowledge in one domain to a similar one.  find that this method is potentially more revealing of implicit grammatical knowledge than purely unsupervised methods.

We conduct additional experiments to investigate the effect of training size on LSTM model performance on *X . We show that learning trajectories differ, sometimes drastically, across different paradigms in the dataset, with phenomena such as anaphor agreement showing consistent improvement as training size increases, and other phenomena such as NPIs and extraction islands remaining near chance despite increases in training size. We also compare overall sentence probability to two other built-in metrics coded on *X and find that the chosen metric changes how we evaluate relative model performance.

Additional Experiments ::: Regular vs. Irregular Agreement	In the determiner-noun agreement and subject-verb agreement categories, we generate separate datasets for nouns with regular and irregular number marking, as in Example SECREF34. All else being equal, only models with access to sub-word-level information should make any distinction between regular and irregular morphology. . Ṙon saw that nice kid/*kids. (regular) Ron saw that nice man/*men. (irregular) Contrary to this prediction, the results in Figure FIGREF36 show that the sub-word-level models GPT-2 and Transformer-XL show little effect of irregular morphology: they perform less than $0.013$ worse on irregulars than regulars. Given their high performance overall, this suggests they robustly encode number features without relying on segmental cues.

The most recent and comprehensive corpus is CoLA , which contains around 10k sentences covering a wide variety of linguistic phenomena from 23 linguistic papers and textbooks. CoLA, which is included in the GLUE benchmark , has been used to track advances in the general grammatical knowledge of reusable sentence understanding models. Current models like BERT  and T5  can be trained to give acceptability judgments that approach or even exceed individual human agreement with CoLA. While CoLA can also be used to evaluate phenomenon-specific knowledge of models, this method is limited by the need to train a supervised classifier on CoLA data prior to evaluation.  compare the CoLA performance of pretrained sentence understanding models: an LSTM, GPT , and BERT. They find that these models have good performance on sentences involving marked argument structure, and struggle on sentences with long-distance dependencies like those found in questions, though the Transformers have a noticeable advantage. However, evaluating supervised classifiers prevents making strong conclusions about the models themselves, since biases in the training data may affect the results. For instance, relatively strong performance on a phenomenon might be due to a model's implicit knowledge or to frequent occurrence of similar examples in the training data.

Additional Experiments ::: Long-Distance Dependencies	The presence of intervening material that lengthens an agreement dependency lowers accuracy on that sentence in both humans and LMs. We study how the presence or absence of this intervening material affects the ability of LMs to detect mismatches in agreement in *X . First, we test for knowledge of determiner-noun agreement with and without an intervening adjective, as in Example SECREF32. The results are plotted in Figure FIGREF33. The $n$-gram model is the most heavily impacted, performing on average 35 points worse. This is unsurprising, since the bigram consisting of a determiner and noun is far more likely to be observed than the trigram of determiner, adjective, and noun. For the neural models, we find a weak but consistent effect, with all models performing on average between 5 and 3 points worse when there is an intervening adjective. . Ṙon saw that man/*men. Ron saw that nice man/*men. Second, we test for sensitivity to mismatches in subject-verb agreement when an “attractor” noun of the opposite number intervenes.

Results ::: Overall Results	An LM's overall performance on *X can be measured simply by taking the proportion of correct predictions across the 67,000 minimal pairs from all paradigms. GPT-2 achieves the highest score and the $n$-gram the lowest. Transformer-XL and the LSTM LM perform in the middle, and at roughly the same level as each other. All models perform well below estimated human agreement (as described in Section SECREF11). The $n$-gram model's poor overall performance confirms *X is not solvable from co-occurrence information alone. Rather, success at *X is driven by the more abstract features learned by neural networks. There are no categories in which the $n$-gram approaches human performance. Because we evaluate pretrained models that differ in architecture and training data quantity/domain, we can only speculate about what drives these differences (though see Section SECREF37 for a controlled ablation study on the LSTM LM). Nonetheless, the results seem to indicate that access to training data is the main driver of performance on *X for the neural models we evaluate.

In part, this difference may come down to differences in metrics.  compare a set of four related sentences with gaps in the same position or no gaps to obtain the wh-licensing interaction as a metric of how strongly the LM identifies a filler-gap dependency in a single syntactic position. They consider an island constraint to have been learned if this value is close to zero. We instead compare LM probabilities of sentences with similar lexical content but with gaps in different syntactic positions. These metrics target different forms of grammatical knowledge, though both are desirable properties to find in an LM. We also note that the LMs we test do not have poor knowledge of filler-gap dependencies in general, with all neural models perform above well above chance. This suggests that, while these models are able to establish long-distance dependencies in general, they are comparatively worse at identifying the syntactic domains in which these dependencies are blocked. The semantic phenomena that models struggle with are usually attributed in current theories to a presupposition failure or contradiction arising from semantic composition or pragmatic reasoning , , . These abstract semantic and pragmatic factors may be difficult for LMs to learn.  also find that LSTMs largely fail to recognize NPI licensing conditions.

On purely architectural grounds, the similar performance of Transformer-XL and the LSTM is surprising since Transformer-XL is the state of the art on several LM training sets. However, they are both trained 100$\pm 10$M tokens of Wikipedia text. Relatedly, GPT-2's advantage may come from the fact that it is trained on roughly two orders of magnitude more data. While it is unclear whether LSTMs trained on larger datasets could rival GPT-2, such experiments are impractical due to the difficulty of scaling LSTMs to this size.

Models & Methods ::: Models ::: LSTM	We include a long-short term memory (LSTM, ) language model in our experiments. Specifically, we test a pretrained LSTM language model from  on *X . The model is trained on a 90M token corpus extracted from English Wikipedia. For investigating the effect of training size on models' *X performance, We retrain a series of LSTM models with the same hyperparameters and the following training sizes: 64M, 32M, 16M, 8M, 4M, 2M, 1M, 1/2M, 1/4M, and 1/8M tokens. For each size, we train the model on five different random samples drawing from the original training data, which has a size of 83M tokens. We release our LSTM evaluation code.

Additional Experiments ::: Alternate Evaluation Methods	There are several other techniques one can use to measure an LM's “preference” between two minimally different sentences. So far, we have considered only the full-sentence method, advocated for by , which compares the LM likelihood of the full sentences. In a followup experiment, we use two “prefix methods”, each of which has appeared in prior work in this area, that evaluate the model's preferences by comparing its prediction at a key point of divergence between the two sentences. Subsets of *X data—from the binding, determiner-noun agreement, and subject-verb agreement categories—are designed to be compatible with multiple methods, allowing us to conduct the first direct comparison. We find that all methods give broadly similar results when aggregating over a large set of paradigms, but some results diverge sharply for specific paradigms.

In these cases, though, both the acceptable and unacceptable sentences will be equally implausible given world knowledge, so any difference in the probability assigned to them is still due to the intended grammatical contrast.

Acknowledgments	This material is based upon work supported by the National Science Foundation under Grant No. 1850208. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. This project has also benefited from support to SB by Eric and Wendy Schmidt (made by recommendation of the Schmidt Futures program), by Samsung Research (under the project Improving Deep Learning using Latent Structure), by Intuit, Inc., and by NVIDIA Corporation (with the donation of a Titan V GPU).

Evaluating LMs on minimal pairs evades this problem by eschewing supervised training on acceptability judgments. It is possible to use the LM probability of a sentence as a proxy for acceptability because other factors impacting a sentence's probability such as length and lexical content are controlled for.

Background & Related Work ::: Evaluating Linguistic Knowledge	A large number of recent studies has used acceptability judgments to reveal what neural networks know about grammar. One branch of this literature has focused on using minimal pairs to infer whether LMs learn about specific linguistic phenomena. Table TABREF4 gives a summary of work that has studied linguistic phenomena in this way. For instance, linzen2016assessing look closely at minimal pairs contrasting subject-verb agreement. marvin2018targeted look at a larger set of phenomena, including negative polarity item licensing and reflexive licensing. However, a relatively small set of phenomena is covered by these studies, to the exclusion of well-studied phenomena in linguistics such as control and raising, ellipsis, distributional restrictions on quantifiers, and countless others. This is likely due to the labor-intensive nature of collecting examples that exhibit informative grammatical phenomena and their acceptability judgments. A related line of work evaluates neural networks on acceptability judgments in a more general domain of grammatical phenomena. Corpora of sentences and their grammaticality are collected for this purpose in a number of computational studies on grammaticality judgment , , .

Background & Related Work ::: Language Models	The objective of a language model is to give a probability distribution over the possible strings of a language. Language models can be built on neural network models or non-neural network models. Due to their unsupervised nature, they can be trained without external annotations. More recently, neural network based language modeling has been shown to be a strong pretraining task for natural language understanding tasks , , , . Some recent models, such as BERT  use closely related tasks such as masked language modeling. In the last decade, we have seen two major paradigm shifts in the state of the art for language modeling. The first major shift for language modeling was the movement from statistical methods based on $n$-grams  to neural methods such as LSTMs , which directly optimize on the task of predicting the next word. More recently, Transformer-based architectures employing self-attention  have outperformed LSTMs at language modeling . Although it is reasonably clear that these shifts have resulted in stronger language models, the primary metric of performance is perplexity, which cannot give detailed insight into these models' linguistic knowledge.

Evaluation on downstream task benchmarks ,  is more informative, but might not present a broad enough challenge or represent grammatical distinctions at a sufficiently fine-grained level.

Models & Methods ::: Evaluation	We mainly evaluate the models by measuring whether the LM assigns a higher probability to the grammatical sentence within the minimal pair. This method, used by , is only meaningful for comparing sentences of similar length and lexical content, as overall sentence probability tends to decrease as sentence length increases or word frequencies decrease . However, as discussed in Section SECREF3 we design every paradigm in *X to be compatible with this method.

We also estimate individual human agreement to be 88.6% overall using the approximately 100 annotations from each paradigm. Figure TABREF14 reports these individual human results (alongside model results) as a conservative measure of human agreement. white

Models & Methods ::: Models ::: GPT-2	GPT-2  is a large-scale language model using the Transformer architecture . We use the large version of GPT-2, which contains 24 layers and 345M parameters. The model is pretrained on 's custom-built WebText dataset, which contains 40GB of text extracted from web pages and filtered by humans. To our best knowledge, the WebText corpus is not publicly available. Assuming approximately 5-6 bytes/chars per word on average, we estimate WebText contains approximately 8B tokens. The testing code for GPT-2 has been integrated into jiant, a codebase for training and evaluating sentence understanding models .

Models & Methods ::: Models ::: Transformer-XL	Transformer-XL  is another multi-layer Transformer-based neural language model. We test a pretrained Transformer-XL model with 18 layers of Transformer decoders and 16 attention heads for each layer. The model is trained on WikiText-103 , a corpus of 103M tokens from high-quality Wikipedia articles. Code for testing Transformer-XL on *X is also implemented in jiant.

Models & Methods ::: Models ::: 5-gram	We build a 5-gram LM on the English Gigaword corpus , which consists of 3.07B tokens. To efficiently query $n$-grams we use an implementation based on , which is shown to speed up estimation . We release our $n$-gram evaluation code.

An important goal of linguistically-informed analysis of LMs is to better understand those empirical domains where current LMs appear to acquire some relevant knowledge, but still fall short of human performance. The results from *X suggest that—in addition to relatively well-studied phenomena like filler-gap dependencies, NPIs, and binding—argument structure remains one area where there is much to uncover about what LMs learn. More generally, as language modeling techniques continue to improve, it will be useful to have large-scale tools like *X to efficiently track changes in what these models do and do not know about grammar.

Results	We report the 12-category accuracy results for all models and human evaluation in Table TABREF14.

Results ::: Phenomenon-Specific Results	The results also reveal considerable variation in performance across grammatical phenomena. Models generally perform best and closest to human level on morphological phenomena. This includes anaphor agreement, determiner-noun agreement, and subject-verb agreement. In each of these domains, GPT-2's performance is within 2.1 percentage points of humans. The set of challenging phenomena is more diverse. Islands are the hardest phenomenon by a wide margin. Only GPT-2 performs noticeably above chance, but it remains 20 points below humans. Some semantic phenomena, specifically those involving NPIs and quantifiers, are also challenging overall. All models show relatively weak performance on argument structure. From results we conclude that current SotA LMs have robust knowledge of basic facts of English agreement. This does not mean that LMs will come close to human performance for all agreement phenomena. Section SECREF32 discusses evidence that increased dependency length and the presence of agreement attractors of the kind investigated by  and  reduce performance on agreement phenomena. The exceptionally poor performance on islands is hard to reconcile with 's () conclusion that LSTMs have knowledge of some island constraints.

find that BERT (which is similar in scale to GPT-2) recognizes these conditions inconsistently in an unuspervised setting. The weak performance on argument structure is somewhat surprising, since arguments are usually (though by no means always) local to their heads. Argument structure is closely related to semantic event structure , which may be comparatively difficult for LMs to learn. This finding contradicts 's () conclusion that argument structure is one of the strongest domains for neural models. However,  study supervised models trained on CoLA, which includes a large proportion of sentences related to argument structure.

Results ::: Shallow Predictors of Performance	We also ask what factors aside from linguistic phenomena make a minimal pair harder or easier for an LM to distinguish. We test whether shallow features like sentence length or overall sentence likelihood are predictors of whether the LM will have the right preference. The results are shown in Figure FIGREF31. While sentence length, perplexity and the probability of the good sentence all seem to predict model performance to a certain extent, the predictive power is not strong, especially for GPT-2, which is much less influenced by greater perplexity of the good sentence than the other models.

We compare attractors in relative clauses and as part of a relational noun as in Example SECREF32, following experiments by  and others. Again, we find an extremely large effect for the $n$-gram model, which performs over 50 points worse and well below chance when there is an attractor present, showing that the $n$-gram model is consistently misled by the presence of the attractor. All of the neural models perform above chance with an attractor present, but GPT-2 and the LSTM perform 22 and 20 points worse when an attractor is present. Transformer-XL's performance is harmed by only 5 points. Note that GPT-2 still has the highest performance in both cases, and even outperforms humans in the relational noun case. Thus, we reproduce 's finding that attractors significantly reduce LSTM LMs' sensitivity to mismatches in agreement and find evidence that this holds true of Transformer LMs as well. . Ṫhe sisters bake/*bakes. The sisters who met Cheryl bake/*bakes. The sisters of Cheryl bake/*bakes.

Additional Experiments ::: Alternate Evaluation Methods ::: One-prefix method	In the one-prefix method, used by , a pair of sentences share the same initial portion of a sentence, but differ in a critical word that make them differ in grammaticality (e.g., The cat eats mice vs. The cat eat mice). The model's prediction is correct if it assigns a higher probability to the grammatical token given the shared prefix.

Additional Experiments ::: Alternate Evaluation Methods ::: Two-prefix method	In the two-prefix method, used by , a pair of sentences have a different initial portion that diverge in some critical way, but the grammaticality difference is only revealed when a shared critical word is included (e.g., The cat eats mice vs. The cats eats mice). For these paradigms, we evaluate whether the model assigns a higher probability to the critical word conditioned on the grammatical prefix compared the ungrammatical prefix. Note that the same pair of sentences cannot be compatible with both prefix methods, and that a pair may be compatible with the full-sentence method but neither prefix method. For both prefix methods, it is crucial that the grammaticality of the sentence is unambiguously predictable from the critical word, but not sooner. With simple LM probabilities, the probabilities of the rest of the word tokens in the sentence also affect the performance.

Additional Experiments ::: Alternate Evaluation Methods ::: Results	The results in Figure FIGREF42 show that models have generally comparable accuracies overall in prefix methods and the simple whole-sentence LM method. However, A deeper examination of the differences between these methods in each paradigm reveals some cases where a models' performance fluctuates more between these methods. For example, Transformer-XL performs much worse at binding, determiner-noun agreement, and subject-verb agreement in the simple LM method, suggesting that the probabilities Transformer-XL assigns to the irrelevant part at the end of the sentence very often overturn the `judgment' based on probability up to the critical word. On the other hand, GPT-2 benefits from reading the whole sentence for binding phenomena, as its performance is better in the simple LM method than in the prefix method. Overall, we observe that Transformer-XL and GPT-2 are more affected by evaluation methods than LSTM and $n$-gram when we compare the simple LM method and the two-prefix method.

We use *X to study several pretrained LMs: Transformer-based LMs GPT-2  and Transformer-XL , an LSTM LM trained by , and a $n$-gram LM. We evaluate whether the LM assigns a higher probability to the acceptable sentence in each minimal pair in *X . This experiment gives a sense of which grammatical distinctions LMs are sensitive to in general, and the extent to which unrelated models have similar strengths and weaknesses. We conclude that current neural LMs robustly learn agreement phenomena and even some subtle syntactic phenomena such as ellipsis and control/raising. They perform comparatively worse (and well below human level) on minimal pairs related to argument structure and the licensing of negative polarity items and quantifiers. All models perform at or near chance on extraction islands, which we conclude is the most challenging phenomenon covered by *X . Overall, we note that all models we evaluate fall short of human performance by a wide margin. GPT-2, which performs the best, does match (even just barely exceeds) human performance on some grammatical phenomena, but remains 8 percentage points below human performance overall.

Question: How is the data automatically generated?
Answer: The data generation scripts use a basic template to create each paradigm, pulling from a vocabulary of over 3000 words annotated for morphological, syntactic, and semantic features needed to create grammatical and semantically felicitous sentences.