SlowGuess/ACL_2020_OCR huggingface dataset:
one in which the distractor is in a subject-extracted relative clause postmodifier of the subject:
(A) The farmer that embarrassed the clerks knows  $\mathrm{V}_{\mathrm{sg}}$  many people.
(B) *The farmer that embarrassed the clerks know  $\mathrm{V}_{\mathrm{pl}}$  many people.
(C) The farmers that embarrassed the clerk know  $\mathrm{V}_{\mathrm{pl}}$  many people.
(D) *The farmers that embarrassed the clerk knows  $\mathrm{V}_{\mathrm{sg}}$  many people.
and one in which the distractor is in an object-extracted relative clause postmodifier of the subject:
(A) The farmer that the clerks embarrassed knows  $\mathrm{V}_{\mathrm{sg}}$  many people.
(B) *The farmer that the clerks embarrassed know  $\mathrm{V}_{\mathrm{pl}}$  many people.
(C) The farmers that the clerk embarrassed know  $\mathrm{V}_{\mathrm{pl}}$  many people.
(D) *The farmers that the clerk embarrassed knows  $\mathrm{V}_{\mathrm{sg}}$  many people.
Criterion Following Linzen et al. (2016) and Marvin and Linzen (2018), we require successful discrimination of the preferred upcoming verbform of the given lemma (rather than, for example, successful discrimination of the better context given a particular verbform). For success we require that a model successfully predicts the preferred verbform for both the singular- and plural-subject versions of an item:
$$
P _ {\mathrm {A}} \left(\mathrm {V} _ {\mathrm {s g}}\right) > P _ {\mathrm {B}} \left(\mathrm {V} _ {\mathrm {p l}}\right) \wedge P _ {\mathrm {C}} \left(\mathrm {V} _ {\mathrm {p l}}\right) > P _ {\mathrm {D}} \left(\mathrm {V} _ {\mathrm {s g}}\right)
$$
Chance performance is thus  $25\%$ , though a context-insensitive baseline that places different probabilities on  $\mathrm{V_{sg}}$  and  $\mathrm{V_{pl}}$  would score  $50\%$ .
References Bock and Miller (1991); Linzen et al. (2016); Marvin and Linzen (2018)
B.9 Reflexive pronoun licensing
The noun phrase with which a reflexive pronoun (herself, himself, themselves) corefers must command it in a sense similar to that relevant for negative-polarity items (Section B.6). In the below example, the reflexive pronoun ending the sentence can only corefer to the subject of the sentence, author, with which it must agree in number: a singular subject requires a singular reflexive  $\mathbf{R}_{\mathrm{sg}}$ , and a plural subject requires a plural reflexive  $\mathbf{R}_{\mathrm{pl}}$ .
(A) The author next to the senators hurt herself  $\mathrm{R_{sg,fe}}$
(B) *The authors next to the senator hurt herself $_{\mathrm{R_{sg,fe}}}$ .
(C) The authors next to the senator hurt themselves $\mathrm{R}_{\mathrm{pl}}$ .
(D) *The authors next to the senator hurt themselves $_{\mathrm{R}_{\mathrm{pl}}}$ .
We generated a pair of test suites—one in which the singular reflexive is herself, and another where the singular reflexive is himself, on the template of the above example, where the distractor NP is in a prepositional-phrase postmodifier of the subject NP. We also generated a similar pair of test suites where the distractor NP is inside a subject-extracted relative clause modifying the subject:
(A) The author that liked the senators hurt herself  $\mathrm{R_{sg,fe}}$
(B) *The authors that liked the senator hurt herself $_{\mathrm{R_{sg,fe}}}$ .
(C) The authors that liked the senator hurt themselves $\mathrm{R}_{\mathrm{pl}}$ .
(D) *The authors that liked the senator hurt themselves $_{\mathrm{Rpl}}$ .
and a pair of test suites where the distractor NP is inside an object-extracted relative clause modifying the subject:
(A) The author that the senators liked hurt herself  $\mathrm{R_{sg,fe}}$
(B) *The authors that the senator liked hurt herself $_{\mathrm{R_{sg, fem}}}$ .
(C) The authors that the senator liked hurt themselves $_{\mathrm{R}_{\mathrm{pl}}}$ .
(D) *The authors that the senator liked hurt themselves $_{\mathrm{R}_{\mathrm{pl}}}$ .
Criterion For each item in each test suite, we require that for both the singular and the plural versions of the reflexive pronoun the model assign higher conditional probability in the correct licensing context than in the incorrect licensing context:
Chance is  $25\%$ .
References Reinhart (1981); Marvin and Linzen (2018)
Beginning a sentence with As, When, Before, After, or Because, implies that an immediately following clause is not the main clause of the sentence, as would have otherwise been the case, but instead is
$$
P _ {\mathrm {A}} \left(\mathrm {R} _ {\mathrm {s g}}\right) > P _ {\mathrm {B}} \left(\mathrm {R} _ {\mathrm {s g}}\right) \wedge P _ {\mathrm {C}} \left(\mathrm {R} _ {\mathrm {p l}}\right) > P _ {\mathrm {D}} \left(\mathrm {R} _ {\mathrm {p l}}\right)
$$
B.10 Subordination
a SUBORDINATE CLAUSE that must be followed by the main clause. Ending the sentence without a main clause, as in B, is problematic. Conversely, following an initial clause with a second clause MC (without linking it to the initial clause with and, but, despite, or a similar coordinator or subordinator), as in C below, is unexpected and odd.
(A) The minister praised the building
(B) *After the minister praised the building
(C) ??The minister praised the building, it started to rain.
(D) After the minster praised the building, it started to rain.
In addition to the base test suite exemplified by the item above, we include three versions with longer and more complex initial clauses, which may make the test suite more difficult. In the first of these versions, we postmodify both the subject and object of the initial clauses with prepositional phrases:
the minister praised the building
the minister in the dark suit and white tie praised the new building on the town's main square
In the second of these versions, the postmodifiers are subject-extracted relative clauses:
the minister who wore a black suit praised the new building that was built by the square
In the third of these versions, the postmodifiers are object-extracted relative clauses:
the minister who the mayor had invited praised the new building that the businessman had built downtown
Criterion Introducing a subordinator at the beginning of the sentence should make an ending without a second clause less probable, and should make a second clause more probable:
$$
P _ {\mathrm {A}} (\mathrm {E N D}) > P _ {\mathrm {B}} (\mathrm {E N D}) \wedge P _ {\mathrm {D}} (\mathrm {M C}) <   P _ {\mathrm {C}} (\mathrm {M C})
$$
References Futrell et al. (2018)