SlowGuess/ACL_2020_OCR huggingface dataset:
Bowman. 2020. BLiMP: A Benchmark of Linguistic Minimal Pairs for English. In Proceedings of the Society for Computation in Linguistics.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019a. CoLA: The Corpus of Linguistic Acceptability (with added annotations).
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019b. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Ethan Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger P. Levy. 2020. Evaluating neural networks as models of human online language processing. In Proceedings of the 42nd Meeting of the Cognitive Science Society (CogSci 2020). To appear.
Ethan Wilcox, Roger P. Levy, and Richard Futrell. 2019a. Hierarchical representation in neural language models: Suppression and recovery of expectations. In Proceedings of the 2019 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
Ethan Wilcox, Roger P. Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about filler-gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
Ethan Wilcox, Peng Qian, Richard Futrell, Miguel Ballestros, and Roger P. Levy. 2019b. Structural supervision improves learning of non-local grammatical dependencies. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3302-3312, Minneapolis, Minnesota.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems.
A Syntactic coverage of test suites
In order to assess the coverage of our syntactic tests, we manually inspected the "Ideas, Rules and Constraints introduced in this Chapter" section for each chapter in Carnie (2012), a standard introductory syntax textbook. We included entries from these sections which are theory-neutral and refer to observable linguistic data. For example, we do not include affix lowering (Chapter 7) or theta criterion (Chapter 8) because these phenomena presuppose a commitment to one particular syntactic analysis.
We found that our tests covered 16 of the 47 phenomena presented  $(\sim 34\%)$ . Of the 15 chapters surveyed, our tests assessed phenomena in 11
CHAPTER 1: GENERATIVE GRAMMAR	Lexical gender
Number	✓
Person
Case
CHAPTER 2: PARTS OF SPEECH	Parts of Speech	✓
Plurality	✓
Count vs. Mass Nouns
Argument Structure of Verbs	✓
CHAPTER 3: CONSTITUENCY, TREES, RULES	Constituency Tests
Hierarchical Structure	✓
CHAPTER 4: STRUCTURAL RELATIONS	c-command	✓
Government
CHAPTER 5: BINDING THEORY	R-expression vs. Pronominals
Anaphoric expressions and their antecedents	✓
Co-reference and co-indexation
Binding Principles (A, B, C)	✓
Locality Constraints	✓
CHAPTER 6: X-BAR THEORY	One Replacement
Do-so Replacement
CHAPTER 7: EXTENDING X-BAR THEORY TO FUNCTIONAL CATEGORIES	Fundamental Phrase Types of DP/CP/TP
Genitives: of-genitives and 's genitives
Subjects and Predicates
Clausal Embedding	✓
Clausal
Tense/Finiteness and its restrictions
Yes/No Questions
Subject-Auxillary Inversion
CHAPTER 8: CONSTRAINING X-BAR THEORY: THE LEXICON	Thematic Relations	✓
Internal Theta role vs. External Theta Roles
Expletive Pronouns and Expletive Insertion
Extended Projection Principle
CHAPTER 9: HEAD-TO-HEAD MOVEMENT	V → T Movement
T → C movement	✓
Do-Support
CHAPTER 10: DP MOVEMENT	Passive Constructions	✓
DP-Raising
CHAPTER 11: WH-MOVEMENT	Wh-Movement	✓
StructuralConstraints on Wh-Movement (IslandConstraints)	✓
Wh in-Situ and Echo Questions
CHAPTER 12: A UNIFIED THEORY OF MOVEMENT	Universal Quantifiers vs. Existential Quantifiers
Quantificational Scope and Quantifier Raising
CHAPTER 13: EXTENDED VPS	Light Verbs
Object Shift (and end weight)
Ellipsis
Pseudogapping
CHAPTER 14: RAISING CONTROL AND EMPTY CATEGORIES	Control, Subject-to-Subject and Subject-to-Object Raising (ECM)
CHAPTER 15: ADVANCED TOPICS IN BINDING THEORY	Binding Principle A and B	✓
Table 5: Test suite coverage of syntactic phenomena presented in Carnie (2012).
$(\sim 73\%)$ . We did not assess coverage from the last two chapters of the book, which explore alternative syntactic formalisms. The outcome of our manual inspection is given in Table 5.
A  $\sqrt{}$  indicates that some aspect of that phenomena was tested in one or more of our suites.  $\sqrt{}$  does not necessarily mean that the test suite was designed explicitly for the purpose of testing that phenomena, but merely that the phenomena was implicated in model success. For example, we place a  $\sqrt{}$  next to Parts of Speech because differentiation between verbs and nouns is necessary for models to succeed in the Cleft Structure tests.
B Description of test suites
In this work we have assembled a large number of test suites inspired by the methodology of experimental sentence-processing and psycholinguistic research. Each test suite contains a number of ITEMS, and each item appears in several CONDITIONS: across conditions, a given item will differ only according to a controlled manipulation designed to target a particular feature of grammatical knowledge. For each suite we define a SUCCESS CRITERION, which stipulates inequalities among conditional probabilities of sentence substrings.
In the main paper, a model's accuracy for a test suite is computed as the percentage of the test suite's items for which it satisfies the criterion. In this appendix, we briefly describe each test suite and the criterion used to determine whether a given model succeeds on each item of the test suite.
B.1.1 Sentence status
Following and building on linguistic traditions, we annotate examples as follows. Examples marked with a * violate a well-established grammatical constraint, and are ungrammatical. Examples marked with ? or ?? are not necessarily ungrammatical, but are marginal: for example, they may require an unusual interpretation of a word in order for the sentence to be grammatical. (More ?'s is roughly intended to indicate more severe marginality). Examples marked with ! are not ungrammatical, but induce severe processing difficulty that is measurable in real-time human sentence processing. For all test suites, we include references to established literature on the relevant grammatical and/or sentence-processing phenomena.
B.1.2 Success criteria
Criteria involve inequalities among conditional probabilities of sentence substrings given the complete sentence context preceding the substring. In describing criteria, we use  $P(\cdot)$  for raw probabilities and  $S(\cdot)$  for surprises (negative log-probabilities), and leave the conditioning on preceding context implicit. For concision, we use subscripts on  $P$  and  $S$  to indicate the variant of the sentence within the test suite that we are referring to. In the first described test suite, CENTER EMBEDDING B.2, we show the criterion in both concise and fully spelled-out forms, to help clarify the conventions we are using in the concise form. All items within a given test suite share the same criterion for success.
We provide chance accuracy on the assumption that the order of probabilities among conditions for a given item is random. In some cases, exactly determining chance accuracy may require further assumptions about the distribution of these probabilities; in this case we provide an upper bound on chance accuracy.
B.2 Center embedding
Center embedding, the ability to embed a phrase in the middle of another phrase of the same type, is a hallmark feature of natural language syntax. Center-embedding creates NESTED SYNTACTIC DEPENDENCIES, which could pose a challenge for some language models. To succeed in generating expectations about how sentences will continue in the context of multiple center embedding, a model must maintain a representation not only of what words appear in the preceding context but also of the order of those words, and must predict that upcoming words occur in the appropriate order. In this test suite we use verb transitivity and subject-verb plausibility to test model capabilities in this respect. For example, A below is a correct center-embedding, but B is not:
(A) The painting $_{\mathrm{N}_1}$  that the artist $_{\mathrm{N}_2}$  painted $_{\mathrm{V}_2}$  deteriorated $_{\mathrm{V}_1}$ . [correct]
(B) ??The painting $_{\mathrm{N}_1}$  that the artist $_{\mathrm{N}_2}$  deteriorated $_{\mathrm{V}_1}$  painted $_{\mathrm{V}_2}$ . [incorrect]
Here,  $\mathbf{N}_i$  and  $V_{i}$  correspond to matched subject-verb pairs.
In the WITH-MODIFIER version of the test suite, we postmodify  $\mathrm{N}_2$  with a relative clause to increase the linear distance over which the nested dependen
cies must be tracked, potentially leading to a harder test suite:
(A) The painting $_{\mathrm{N}_1}$  that the artist $_{\mathrm{N}_2}$  who lived long ago painted $_{\mathrm{V}_2}$  deteriorated $_{\mathrm{V}_1}$ . [correct]
(B) #The painting $_{\mathrm{N_1}}$  that the artist $_{\mathrm{N_2}}$  who lived long ago deteriorated $_{\mathrm{V_1}}$  painted $_{\mathrm{V_2}}$ . [incorrect]
Criterion The probability of the verb sequence in the correct variant should be higher than the probability of the verb sequence in the incorrect variant:
$$
P _ {\mathrm {A}} \left(\mathrm {V} _ {2} \mathrm {V} _ {1}\right) > P _ {\mathrm {B}} \left(\mathrm {V} _ {1} \mathrm {V} _ {2}\right)
$$
In full form, this criterion for the example item in the no-modifier version of this test suite would be:
$P(\text{painted deteriorated}|\text{The painting that the artist}) > P(\text{deteriorated painted}|\text{The painting that the artist})$
Chance performance on these center-embedding test suites would be  $50\%$ .
References Miller and Chomsky (1963); Wilcox et al. (2019a)
B.3 Pseudo-clefting
The pseudo-cleft construction involves (i) an extraction of a TARGETED CONSTITUENT from a sentence and (ii) a constituent that provides the semantic contents of the targeted constituent and must match it in syntactic category, where (i) and (ii) are linked by the copula. The pseudo-cleft construction can target both NPs and VPs; in the latter case, the VP of the free relative becomes an inflected form of  $do$ . This means that a free relative subject plus the copula can set up a requirement for the syntactic category that comes next. If the free relative clause has a  $do$  VP without a direct object, then the main-clause postcopular predicate can be a VP (A below). Otherwise, the postcopular predicate must be an NP (C below):
(A) What the worker did was  $\overbrace{\text{board the plane}}^{\text{VP}}$
(B) ?What the worker did was the plane.
(C) What the worker repaired was the plane.
(D) \*What the worker repaired was VP board the plane.
Criterion The postcopular predicate should be more surprising when its syntactic category mismatches the cleft, averaging across VP and NP postcopular predicates:
$$
S _ {\mathrm {D}} (\mathrm {V P}) + S _ {\mathrm {B}} (\mathrm {N P}) > S _ {\mathrm {C}} (\mathrm {N P}) + S _ {\mathrm {A}} (\mathrm {V P})
$$
Chance is  $50\%$ . A more stringent criterion would be to apply this requirement separately for each of NP and VP postcopular predicates:
$$
S _ {\mathrm {D}} (\mathrm {V P}) > S _ {\mathrm {A}} (\mathrm {V P}) \wedge S _ {\mathrm {B}} (\mathrm {N P}) > S _ {\mathrm {C}} (\mathrm {N P})
$$
However, it is often possible to use an NP postcopular predicate with a do cleft through semantic coercion (e.g., in B "did" can be interpreted as "fixed" or "was responsible for"), so we felt that this latter criterion might be too stringent.
References Higgins (1973)
B.4 Filler-gap dependencies
Consider the following sentence, in which all arguments and adjuncts appear "in situ" (in the syntactic position at which they are normally interpreted semantically):
I know that our uncle grabbed the food in front of the guests at the holiday party.
A FILLER-GAP DEPENDENCY can be created by EXTRACTING any of a number of elements from the subordinate clause, including our uncle (subject extraction), the food (object extraction) or the guests (extraction from a prepositional phrase). These possibilities serve as the basis for several test suites on filler-gap dependencies.
References Ross (1967); Crain and Fodor (1985); Stowe (1986); Wilcox et al. (2018); Chowdhury and Zamparelli (2018); Chaves (2020)
B.4.1 Subject extractions
(A) I know that our uncle grabbed the food in front of the guests at the holiday party. [THAT, NO GAP]
(B) \*I know who our uncle grabbed the food in front of the guests at the holiday party. [WH, NO GAP]
(C) \*I know that the food in front of the guests at the holiday party. [THAT, GAP]
(D) I know who  $\overbrace{\text{grabbed}}^{\beta}$  the food in front of the guests at the holiday party. [WH, GAP]
Criterion We require that a model successfully pass a two-part criterion for each item: the whfiller should make the unextracted subject  $\alpha$  more surprising in the NO-GAP conditions and should make the post-gap material  $\beta$  less surprising in the GAP conditions:
$$
S _ {\mathrm {B}} (\alpha) > S _ {\mathrm {A}} (\alpha) \wedge S _ {\mathrm {C}} (\beta) > S _ {\mathrm {D}} (\beta)
$$
Chance is  $25\%$ .
B.4.2 Object extractions
The logic of this test suite is the same as that for subject extraction above. Note that we use obligatorily transitive embedded verbs, so that omitting a direct object should be highly surprising when there is no filler, as in C.
(A) I know that our uncle grabbed the food in front of the guests at the holiday party. [THAT, NO GAP]
(B) *I know what our uncle grabbed the food in front of the guests at the holiday party. [WH, NO GAP]
(C) ??I know that our uncle grabbed in front of the guests at the holiday party. [THAT, GAP]
(D) I know what our uncle grabbed in front of in front of the guests at the holiday party. [WH, GAP]
B.4.3 Extraction from prepositional phrases
The logic of this test suite is the same as that for subject and object extractions above.
(A) I know that our uncle grabbed the food in front of the guests at the holiday party. [THAT, NO GAP]
(B) \*I know who our uncle grabbed the food in front of the guests at the holiday party. [WH, NO GAP]
(C) *I know that our uncle grabbed the food in front of at the holiday party. [THAT, GAP]
(D) I know who our uncle grabbed the food in front of at the holiday party. [WH, GAP]
B.4.4 Tests for unboundedness
Filler-gap dependencies are "unbounded" in the sense that there is no limit to how many clausal levels above the gap the filler can be extracted. This serves as the basis for harder versions of the object-extracted test suites, involving three or four levels of clausal embedding. Example [THAT, NO GAP] sentences are given below:
I know that our mother said her friend remarked that the park attendant reported your friend threw the plastic into the trash can. [3 levels of embedding]
I know that our mother said her friend remarked that the park attendant reported the cop thinks your friend threw the plastic into the trash can. [4 levels of embedding]
These base sentences give rise to 4-condition test suites using the same manipulations as for the basic object-extraction test suite (Section B.4.2), and the criterion for success is the same.
B.5 Main-verb/reduced-relative garden-path disambiguation
This is one of the best-studied instances of syntactic garden-pathing in the psycholinguistics literature. An example 4-condition item is given below:
(A) !The child kicked in the chaos found her way back home. [REDUCED, AMBIG]
(B) The child who was kicked in the chaos found her way back home.
(C) The child forgotten in the chaos found her way back home.
(D) The child who was forgotten in the chaos  $\overbrace{\mathrm{f o u n d}}^{\mathrm{V}^{*}}$  her way back home.
Criterion Relative to the [REDUCED, AMBIG] condition, not reducing the relative clause should make  $\mathrm{V}^*$  less surprising, as should changing the participial verb to one that is the same form as a simple past-tense verb. Additionally, the effect of not reducing the relative clause on  $\mathrm{V}^*$  surprisal should be smaller for unambiguous participial verbs than for participial verbs:
$$
S _ {\mathrm {A}} (\mathrm {V} ^ {*}) > S _ {\mathrm {B}} (\mathrm {V} ^ {*}) \wedge S _ {\mathrm {A}} (\mathrm {V} ^ {*}) > S _ {\mathrm {C}} (\mathrm {V} ^ {*}) \wedge
$$
$$
S _ {\mathrm {A}} (\mathbf {V} ^ {*}) - S _ {\mathrm {B}} (\mathbf {V} ^ {*}) > S _ {\mathrm {C}} (\mathbf {V} ^ {*}) - S _ {\mathrm {D}} (\mathbf {V} ^ {*})
$$
Chance is somewhere below  $25\%$ .
References  
Bever (1970); Ferreira and Clifton (1986); Trueswell et al. (1994); van Schijndel and Linzen (2018); Futrell et al. (2019)
B.6 Negative Polarity Licensing
The words any and ever, in their most common uses, are "negative polarity items" (NPIs): they can only be used in an appropriate syntactic-semantic environment—to a first approximation, in the scope of negation. For example, the determiner no can license NPIs, but its NP has to structurally command the NPI. Below, A and D are acceptable, because no is the determiner for the subject noun managers. There is no negation in C so the NPI is unlicensed and the sentence is unacceptable; crucially, however, B is unacceptable despite the presence of no earlier in the sentence, because no is embedded inside a modifier of the main-clause subject and thus does not command the NPI.
(A) No managers that respected the guard have NPI had any luck.  $[+\mathrm{NEG}, - \mathrm{DISTRACTOR}]$
(B) \*The managers that respected no guard have NPI had any luck.[-NEG,+DISTRACTOR]
(C) \*The managers that respected the guard have NPI had any luck.[-NEG,-DISTRACTOR]
(D) No managers that respected no guard have NPI had any luck.  $[+\mathrm{NEG}, + \mathrm{DISTRACTOR}]$
In the above test suite, the "distractor" position for no is inside a subject-extracted relative clause modifying the main-clause subject. We also used a variant test suite in which these relative clauses are object-extracted:
(A) No managers that the guard respected have NPI had any luck.  $[+\mathrm{NEG}, - \mathrm{DISTRACTOR}]$
(B) \*The managers that no guard respected have NPI had any luck.[-NEG,+DISTRACTOR]
(C) \*The managers that the guard respected have NPI had any luck.[-NEG,-DISTRACTOR]
(D) No managers that no guard respected have NPI had any luck.  $[+\mathrm{NEG}, + \mathrm{DISTRACTOR}]$
The above two test suites use any as the NPI; we also use test suites with ever as the NPI. Subject-extracted relative clause example:
(A) No managers that respected the guard have NPI ever gotten old.  $[+\mathrm{NEG}, - \mathrm{DISTRACTOR}]$
(B) *The managers that respected no guard have NPI ever gotten old. [-NEG,+DISTRACTOR]
(C) *The managers that respected the guard have NPI ever gotten old. [-NEG,-DISTRACTOR]
(D) No managers that respected no guard have NPI ever gotten old.  $[+\mathrm{NEG}, + \mathrm{DISTRACTOR}]$
Object-extracted relative clause example:
(A) No managers that the guard respected have NPI ever gotten old.  $[+\mathrm{NEG}, - \mathrm{DISTRACTOR}]$
(B) *The managers that no guard respected have NPI ever gotten old. [-NEG,+DISTRACTOR]
(C) *The managers that the guard respected have NPI ever gotten old. [-NEG,-DISTRACTOR]
(D) No managers that no guard respected have NPI ever gotten old.  $[+\mathrm{NEG}, + \mathrm{DISTRACTOR}]$
Criterion Changing the main-clause subject's determiner from The to No should increase the probability of the NPI where it appears, regardless of whether there is a distractor no in the subject-modifying relative clause. Furthermore, when there is exactly one no in the sentence, the NPI should be higher-probability when it is in a licensing position rather than in a distractor position:
$$
\begin{array}{l} P _ {\mathrm {A}} (\mathrm {N P I}) > P _ {\mathrm {C}} (\mathrm {N P I}) \wedge P _ {\mathrm {D}} (\mathrm {N P I}) > P _ {\mathrm {B}} (\mathrm {N P I}) \wedge \\ P _ {\mathrm {A}} (\mathrm {N P I}) > P _ {\mathrm {B}} (\mathrm {N P I}) \\ \end{array}
$$
Chance is  $\frac{5}{32}$ .
References Ladusaw (1979); Vasishth et al. (2008); Giannakidou (2011); Marvin and Linzen (2018); Futrell et al. (2018)
B.7 NP/Z garden-path ambiguity
This is another well-studied syntactic garden-pathing configuration. In A below, the NP the waters introduces a local syntactic ambiguity: it could be (1) the direct object of crossed, in which case the sentence-initial subordinate clause has not yet ended, or (2) the subject of the main clause, in which case crossed is used intransitively and is the last word of the sentence-initial subordinate clause. (This was dubbed "NP/Z" by Sturt et al. (1999) because the subordinate-clause verb might have either an NP object or a Z(ero), i.e. null, object.) The next word, remained, is only compatible with (2); the ruling out of (1) generally yields increased processing difficulty for human comprehenders. Marking the end of the subordinate clause with a comma, as in B, makes the sentence easier at  $\mathrm{V}^*$ , as does an obligatorily intransitive subordinate-clause verb, as in C.
(A) !As the ship crossed the waters remained blue and calm. [TRANS,NO COMMA]
(B) As the ship crossed, the waters remained blue and calm. [TRANS,COMMA]
$\mathrm{V}^*$  (C) As the ship drifted the waters remained blue and calm. [INTRANS,NO COMMA]
(V\* (D) As the ship drifted, the waters remained blue and calm. [INTRANS,COMMA]
Criterion Similar to the main-verb/reduced relative garden-pathing ambiguity, a model must pass a three-part criterion. Relative to A, either marking the subordinate-clause end with a comma or using an obligatorily intransitive verb in the subordinate clause should reduce the surprisal of  $\mathbf{V}^*$ . Furthermore, the surprisal-reduction effect of the comma should be smaller when the subordinate-clause verb is intransitive than when it is transitive:
$$
\begin{array}{l} S _ {\mathrm {A}} \left(\mathbf {V} ^ {*}\right) > S _ {\mathrm {B}} \left(\mathbf {V} ^ {*}\right) \wedge S _ {\mathrm {A}} \left(\mathbf {V} ^ {*}\right) > S _ {\mathrm {C}} \left(\mathbf {V} ^ {*}\right) \wedge \\ S _ {\mathrm {A}} (\mathbf {V} ^ {*}) - S _ {\mathrm {B}} (\mathbf {V} ^ {*}) > S _ {\mathrm {C}} (\mathbf {V} ^ {*}) - S _ {\mathrm {D}} (\mathbf {V} ^ {*}) \\ \end{array}
$$
We also use an NP/Z test suite where the second means of disambiguation is not changing the subordinate-clause verb to an intransitive, but
rather giving the transitive subordinate-clause verb an overt direct object. For the above example item, the first two conditions are the same and the other two conditions would be:
(C) As the ship crossed the sea the waters remained blue and calm.
(D) As the ship crossed the sea, the waters remained blue and calm.
The success criterion remains the same.
Finally, we create harder versions of both the above test suites by adding a postmodifier to the main-clause subject (in the above example, the waters becomes the waters of the Atlantic Ocean).
References Frazier and Rayner (1982); Mitchell (1987); Pickering and Traxler (1998); Sturt et al. (1999); Staub (2007)
B.8 Subject-verb number agreement
This task tests a language model for how well it predicts the number marking on English finite present-tense verbs (whether it should be the third-person singular form, or the non-third-person-singular form, generally referred to as the plural form for simplicity, although technically this is the form for first- and second-person singular as well). In controlled, targeted versions of this test, multiple NP precede the verb: the verb's actual subject, as well as a DISTRactor NP with number that is different from that of the subject. A successful language model should place higher probability on the verbform matching that of the subject, not the distractor. We have three versions of this test suite: one where the distractor is in a prepositional phrase postmodifier of the subject:
(A) The farmer near the clerks knows  $\mathrm{V}_{\mathrm{sg}}$  many people.
(B) *The farmer near the clerks know  $\mathrm{V}_{\mathrm{pl}}$  many people.
(C) The farmers near the clerk know  $\mathrm{V}_{\mathrm{pl}}$  many people.
(D) *The farmers near the clerk knows  $\mathrm{V}_{\mathrm{sg}}$  many people.