SlowGuess/ACL_2020_OCR huggingface dataset:
(Long and Short Papers), pages 3532-3542, Minneapolis, Minnesota. Association for Computational Linguistics.
Shauli Ravfogel, Yoav Goldberg, and Francis Tyers. 2018. Can LSTM learn to capture agreement? the case of Basque. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 98-107, Brussels, Belgium. Association for Computational Linguistics.
Marten van Schijndel, Aaron Mueller, and Tal Linzen. 2019. Quantity doesn't buy quality syntax with neural language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5835-5841, Hong Kong, China. Association for Computational Linguistics.
Carson T Schütze. 1996. The empirical base of linguistics: Grammaticality judgments and linguistic methodology. University of Chicago Press.
Rico Sennrich. 2017. How grammatical is character-level neural machine translation? Assessing MT quality with contrastive translation pairs. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 376-382, Valencia, Spain. Association for Computational Linguistics.
Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012. LSTM neural networks for language modeling. In Proceedings of the 13th Annual Conference of the International Speech Communication Association (INTERSPEECH), pages 194-197.
Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4731-4736, Brussels, Belgium. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2019a. BLiMP: A benchmark of linguistic minimal pairs for English. arXiv preprint 1912.00582.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019b. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Paul Wexler. 1990. The schizoid nature of modern Hebrew: A Slavic language in search of a Semitic past. Wiesbaden: Otto Harrassowitz Verlag.
Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about filler-gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211-221, Brussels, Belgium. Association for Computational Linguistics.
Amir Zeldes. 2013. Is Modern Hebrew Standard Average European? The view from European. Linguistic Typology, 17(3):439-470.
Ghil'ad Zuckermann. 2006. A new vision for Israel Hebrew: Theoretical and practical implications of analyzing Israel's main language as a semi-engineered Semito-European hybrid language. Journal of Modern Jewish Studies, 5(1):57-71.
A Linguistic Examples
This section provides examples of the syntactic structures included in the CLAMS dataset across languages. For Hebrew, we transliterate its original right-to-left script into the left-to-right Latin script; this makes labeling and glossing more consistent across languages. Hebrew was not transliterated in the training/development/test corpora or in the evaluation sets. In all examples, (a) is English, (b) is French, (c) is German, (d) is Hebrew, and (e) is Russian.
The first case is simple agreement. This simply involves agreeing a verb with its adjacent subject, which should pose little challenge for any good language model regardless of syntactic knowledge.
(3) Simple Agreement:
a. The surgeons laugh/\*laughs.
b. Le pilote parle / *parlent.  
The pilot laughs / *laugh.
c. Der Schriftsteller spricht / *sprechen.  
The writer speaks / *speak.
d. Ha meltsar yashen / yeshenim.  
The server sleeps / *sleep.
e. BpaHIOBOPrT / \*ROBOpHT. Doctors speak / \*speaks.
Short verb-phrase coordination introduces some slight distance between the subject and verb, though the presence of the previous verb should give a model a clue as to which inflection should be more probable.
(4)  $VP$  coordination (short):
a. The author swims and smiles/\*smile.
b. Les directeurs parlent et deménagent / The directors talk and move / *déménage. *moves.
c. Der Polizist schwimmt und lacht / The police.officer swims and laughs / *lachen. *laugh.
d. Ha tabaxim rokdim ve soxim / *soxe.  
The cooks dance and swim / *swims.
e. Ipoceccop cTapbI n HHTaET / \*HTaIOT. Professor is. old and reads /\*read.
Long verb-phrase coordination is similar, but makes each verb phrase much longer to introduce more distance and attractors between the subject and target verb.
(5)  $VP$  coordination (long):
a. The teacher knows many different foreign languages and likes/\*like to watch television shows.
b. L' agriculteur écrit dans un journal tous  
The farmer writes in a journal all  
les jours et préfére / *préférent jouer au  
the days and prefers / *prefer to.play at the  
tennis avec des collègues.  
tennis with some colleagues.
c. Die Bauern sprechen vielseverschiedene The farmers speak many various Sprachen und siehen / \*sieht gern languages and watch / \*watches gladly Fernsehprogramme. TV-shows.
d. Ha tabax ohev litspot be toxniot televizya The cook likes to.watch in shows TV ve gar / \*garim be merkaz ha ir. and lives / \*live in center the city.
e. ABTOP 3HaET MHOTo HIOCTpaHHbIX Author knows many foreign I3bIKOB I JIO6HT / \*JIO6T CMOTpeTB languages and likes / \*like to.watch TeJIeIepeJaH. TV-shows.
Now we have more complex structures that require some form of structural knowledge if a model is to obtain the correct predictions with more than random-chance accuracy. Agreement across a subject relative clause involves a subject with an attached relative clause containing a verb and object, followed by the main verb. Here, the attractor is the object in the relative clause. (An attractor is an intervening noun between a noun and its associated finite verb which might influence a human's or model's decision as to which inflection to choose. This might be of the same person and number, or, in more difficult cases, a different person and/or number. It does not necessarily need to occur between the noun and its associated verb, though this
5534
does tend to render this task more difficult.)
(6) Across a subject relative clause:
a. The officers that love the chef are/\*is old.
b. Les chirurgiens qui détestent le garde  
The surgeons that hate the guard  
retournent / *returne.  
return / *returns
c. Der Kunde, der die Architekten hasst, ist / The customer that the architects hates is / \*sind Klein. \*are short.
d. Ha menahel she ma'arits et ha shomer The manager who admires ACC the guard rats / \*ratsim. runs / \*run.
e. IIIOIOTbI, KOITOpbIE IOHIMaIOT aReHTOB, Pilots that understand agents  $\frac{\Gamma\mathrm{OBOPAT}}{\mathrm{speak}} / *_{\mathrm{t}}^{\mathrm{ROBOPHT}}$
Agreement within an object relative clause requires the model to inflect the proper verb inside of an object relative clause; the object relative clause contains a noun and an associated transitive verb whose object requirement is filled by the relative pronoun. The model must choose the proper verb inflection given the noun within the relative clause as opposed to the noun outside of it. This may seem similar to simple agreement, but we now have an attractor which appears before the noun of the target verb.
(7) Within an object relative clause:
a. The senator that the executives love/\*loves laughs.
b. Les professeurs que le chef admire / The professors that the boss admires / *admiren parlent.  
*admire talk.
c. Die Polizisten, die der Bruder hasst,/ The policeOfficers that the brother hates / *hassen, sind alt *hate are old.
d. Ha menahel she ha nahag ma'aritz / The manager that the driver admires / \*ma'aritsim soxe. \*admire swims.
e. CeHaTOpbI, KOTOpbIX pa6OuHe HIIyT, / Senators that workers seek / \*HIIeT, JkaJIi. \*seeks wait.
Agreement across an object relative clause is similar, but now the model must choose the correct inflection for the noun outside of the relative clause.
This requires the model to capture long-range dependencies, and requires it to have the proper structural understanding to ignore the relative clause when choosing the proper inflection for the focus verb.
(8) Across an object relative clause:
a. The senator that the executives love laughs/\*laugh.
b. Les professeurs que le chef admire parlent / The professors that the boss admires talk / *parle. *talks.
c. Der Senator, den die Tänzer mögen, spricht / The senator that the dancers like *speaks / *sprechen. *speak.
d. Ha katsin she ha zamar ohev soxe / The officer that the singer likes swims / *soxim. *swim.
e. ΦερΜερ𝑏ι, ΚΟΤΟβIx TaHΠΟρ𝑏ι XΟΤΑΓ, Farmers that dancers want 60JIbIIIIe / *60JIbIIIOi. are/big / *is/big.
Finally, agreement across a prepositional phrase entails placing a prepositional phrase after the subject; the prepositional phrase contains an attractor, which makes choosing the correct inflection more difficult.
(9) Across a prepositional phrase:
a. The consultants behind the executive smile/\*smiles.
b. Les clients devant l' adjoint sont / *est  
The clients in.front.of the deputy are / *is  
vieux.  
old.
c. Der Lehrer geben den Ministern lacht / The teacher next.to the ministers laughs / *lachen. *laugh.
d. Ha meltsarim leyad ha zamarim nos'im / The servers near the singers drive / *nose'a. *drives.
e. PeKnCCepbI IpepeI aReHTaMn Directors in.front.of agents MaJIeHbKHe / \*MaJIeHbKnJ. are.small / *is/small.
Some of the constructions used by Marvin and Linzen (2018) could not be replicated across languages. This includes reflexive anaphora, where none of our non-English languages use quite the
English	French	German	Russian
Mono	Multi	Mono	Multi	Mono	Multi	Mono	Multi
Simple agreement	—	-.02	—	-.01	—	+.02	+.02	—
VP coordination (short)	-.01	—	+.01	+.14	-.02	-.01	-.03	-.01
VP coordination (long)	-.03	+.01	+.04	-.02	-.06	+.07	+.04	+.02
Across subject rel. clause	+.24	+.07	+.23	+.15	-.03	+.13	+.02	+.01
Within object rel. clause	—	-.04	—	-.07	—	-.02	-	-.03
Across object rel. clause	+.09	+.02	+.05	+.03	+.01	+.09	+.01	-
Across prepositional phrase	+.18	+.11	+.20	+.20	+.03	+.03	+.03	+.02
Average accuracy	+.06	+.03	+.07	+.05	-.01	+.05	+.01	+.03
Table 4: Gains (positive, blue) and losses (negative, red) in LSTM LM accuracies on CLAMS after capitalizing the first character of each evaluation example. Differences are relative to the results in Table 1. Results are averaged across five random initializations.
same syntactic structures as English (or even to each other) when employing reflexive verbs and pronouns. Some do not even have separate reflexive pronouns for third-person singular and plural distinctions (like French and German). Moreover, the English reflexive examples rely on the syncretism between past-tense verbs for any English person and number,[10] whereas other languages often have different surface forms for different person and number combinations in the past tense. This would give the model a large clue as to which reflexive is correct. Thus, any results on reflexive anaphora would not be comparable cross-linguistically. See example (10) below for English, French, and German examples of the differences in reflexive syntax.
(10) Reflexive anaphora across relative clause:
a. The author that the guards like injured himself/\*themselves.
b. L' auteur que les gardes aiment s' The author that the guards like REFL.3 est blessé / \*Se sont has.3s injured.S.MASC/REFL.3 have.3P blessés. injured.P.MASC
c. Der Autor, den die Wächter mögen, The author that the guards like verletzte sich / *verletzten sich. injured.3S REFL.3 / injured.3P REFL.3
B The Importance of Capitalization
As discovered in Hao (2020), capitalizing the first character of each test example improves the per
formance of language models in distinguishing grammatical from ungrammatical sentences in English. To test whether this finding holds cross-linguistically, we capitalize the first character of each of our test examples in all applicable languages. Hebrew has no capital-/lower-case distinction, so it is excluded from this analysis.
Table 4 contains the results and relative gains or losses of our LSTM language models on the capitalized stimuli compared to the lowercase ones. For all languages except German, we see a notable increase in the syntactic ability of our models. For German, we see a small drop in overall performance, but its performance was already exceptionally high in the lowercase examples (perhaps due to its mandatory capitalization of all nouns).
An interesting change is that morphological complexity no longer correlates with the overall syntactic performance across languages ( $\rho = 0.2$ ). Perhaps the capitalization acts as an explicit cue to syntactic structure by delineating the beginning of a sentence, thus supplanting the role of morphological cues in aiding the model to distinguish grammatical sentences.
Overall, it seems quite beneficial to capitalize one's test sentences before feeding them to a language model if one wishes to improve syntactic accuracy. The explanation given by Hao (2020) is that *The* essentially only appears sentence-initially, thus giving the model clues as to which noun (typically the token following *The*) is the subject. Conversely, *the* has a more varied distribution, as it may appear before essentially any noun in subject or object position; thus, it gives the model fewer
10 For example, regardless of whether the subject is singular, plural, first- or third-person, etc., the past-tense of see is always saw.
English	French	German	Hebrew	Russian
Mono	Multi	Mono	Multi	Mono	Multi	Mono	Multi	Mono	Multi
Simple agreement	.00	.00	.00	.00	.00	.02	.01	.01	.01	.07
VP coordination (short)	.01	.00	.01	.05	.02	.00	.01	.01	.02	.02
VP coordination (long)	.06	.08	.05	.09	.04	.07	.06	.06	.04	.06
Across subject rel. clause	.06	.02	.05	.05	.04	.07	.03	.03	.03	.04
Within object rel. clause	.01	.02	.01	.01	.03	.04	.01	.03	.04	.02
Across object rel. clause	.05	.02	.01	.01	.09	.06	.01	.01	.03	.02
Across prepositional phrase	.02	.02	.02	.02	.06	.03	.03	.04	.02	.01
Table 5: Standard deviation of LSTM LM performance across five random weight initializations for all languages and stimulus types.
cues as to which noun agrees with a given verb. This would explain the larger score increase for English and French (which employ articles in a similar fashion in CLAMS), as well as the milder increase for Russian (which does not have articles). However, it does not explain the decrease in performance on German. A deeper investigation of this trend per-language could reveal interesting trends about the heuristics employed by language models when scoring syntactically complex sentences.
C Performance Variance
Previous work has found the variance of LSTM performance in syntactic agreement to be quite high (McCoy et al., 2018; Kuncoro et al., 2019). In Table 5, we provide the standard deviation of accuracy over five random initializations on all CLAMS languages and stimulus types. This value never exceeds 0.1, and tends to only exceed 0.05 in more difficult syntactic contexts.
For syntactic contexts without attractors, the standard deviation is generally low. In more difficult cases like Across a Subject Relative Clause and Long VP Coordination, we see far higher variance. In Across an Object Relative Clause, however, the standard deviation is quite low despite this being the case on which language models struggled most; this is likely due to the consistently at-chance performance on this case, further showcasing the difficulty of learning syntactic agreements in such contexts.
On cases where German tended to deviate from the general trends seen in other languages, we see our highest standard deviations. Notably, the performance of German LMs in Across an Object Relative Clause and Across a Prepositional Phrase varies far more than other languages for the same stimulus type.
D Evaluation Set Sizes
Here, we describe the size of the various evaluation set replications. These will differ for the LSTMs, BERT, and mBERT, as the two latter models sometimes do not contain the varied focus verb for a particular minimal set.
Table 6 displays the number of minimal sets per language and stimulus type (with animate nouns only) in our evaluation sets; the total number of sentences (grammatical and ungrammatical) is the number of minimal sets times two. These are also the number of examples that the LSTM is evaluated on. We do not include inanimate-noun cases in our evaluations for now, since these are much more difficult to replicate cross-linguistically. Indeed, grammatical gender is a confounding variable which—according to preliminary experiments—does have an effect on model performance. Additionally, Hebrew has differing inflections depending on the combination of the subject and object noun genders, which means that we rarely have all needed inflections in the vocabulary.
We have differing numbers of examples per-language for similar cases. The reasoning for this is two-fold: (1) direct translations do not exist for all English items in the evaluation set of Marvin and Linzen (2018), so we often must decide between multiple possibilities. In cases where there are two translations of a noun that could reasonably fit, we use both; if we have multiple possibilities for a given verb, we use only one—the most frequent of the possible translations. If no such translation exists for a given noun or verb, we pick a different word that is as close to the English token is possible in the same domain.
Reason (2) is that many of the nouns and verbs in the direct translation of the evaluation sets do not appear in the language models' vocabularies. Thus,
English	French	German	Hebrew	Russian
Simple agreement	140	280	140	140	280
VP coordination (short)	840	980	980	980	980
VP coordination (long)	400	500	500	500	500
Across subject rel. clause	11200	11200	11200	11200	10080
Within object rel. clause	11200	11200	11200	11200	11200
Across object rel. clause	11200	11200	11200	11200	11200
Across prepositional phrase	16800	14000	12600	5600	5880
Table 6: Number of minimal sets for all languages and stimulus types using animate nouns.
English	French	German	Hebrew	Russian
Mono	Multi
SUBJECT-VERB AGREEMENT
Simple agreement	120	80	40	100	20	80
In a sentential complement	1440	960	-	-	-	-
VP coordination (short)	720	480	140	700	140	280
VP coordination (long)	400	240	100	300	100	0
Across subject rel. clause	9600	6400	1600	5406	1600	2880
Within object rel. clause	15960	5320	0	0	0	0
Within object rel. clause (no that)	15960	5320	-	-	-	-
Across object rel. clause	19680	16480	1600	5620	1600	3200
Across object rel. clause (no that)	19680	16480	-	-	-	-
Across prepositional phrase	19440	14640	2000	9000	800	1680
REFLEXIVE ANAPHORA
Simple	280	280	-	-	-	-
In a sentential complement	3360	3360	-	-	-	-
Across a rel. clause	22400	22400	-	-	-	-
Table 7: Number of minimal sets used by BERT (English monolingual only) and mBERT for evaluation. The number of monolingual English examples is the same as in Goldberg (2019). Hyphens indicate non-replicable stimulus types, and 0 indicates that all focus verbs for a given stimulus type were out-of-vocabulary.
some nouns or focus verbs would effectively be  $<\text{unk}>s$  if left in, rendering that particular example unusable. In such cases, if a given noun/verb is not the vocabulary, we pick a similar noun from the same domain if one exists; if a similar item does not exist in the vocabulary, we choose some common noun in that language's vocabulary that has not already been used in the evaluation set.
We use a similar process to add new verbs, but sometimes, third-person singular and plural inflections of similar verbs did not exist in the vocabulary. In such cases, we used a similar verb if possible (e.g., 'dislike' would be reasonably similar in distribution and meaning to 'hate'), but if no such similar verb exists in the vocabulary, we do not replace it. A similar process is used for closed classes like prepositions: if no sufficient replacement exists in
the vocabulary, it is not replaced.
Table 7 contains the number of examples used by BERT and mBERT to calculate examples. Important to note is that for these evaluations, we use stimulus types containing both animate and inanimate nouns to better match Goldberg (2019)'s experimental setup; this is why we have more examples for English in this table than for the LSTM evaluations. Including or excluding inanimate nouns was found to make no significant difference in the final scores (for BERT or mBERT) regardless, since the performance of the model never diverges by more than 0.02 for animate vs. inanimate stimulus types.
The variation in the number of examples across languages is due to many of the focus verbs not being in mBERT's vocabulary. We see the lowest
coverage in general for Hebrew and (surprisingly) French; this is likely due to Hebrew script being a rarer script in mBERT and due to many of French's most common tokens being split into subwords, respectively. Russian also has relatively low coverage, having 0 in-vocabulary target verbs for long VP coordination. None of our languages except English had any target verbs for Within an Object Relative Clause.