SlowGuess/ACL_2020_OCR huggingface dataset:
Aaron Mueller<sup>1</sup> Garrett Nicolai<sup>1†</sup> Panayiota Petrou-Zeniou<sup>2</sup> Natalia Talmina<sup>2</sup> Tal Linzen<sup>1,2</sup>
$^{1}$ Department of Computer Science
$^{2}$ Department of Cognitive Science Johns Hopkins University
{amueller, gnicola2, ppetroul, talmina, tal.linzen}@jhu.edu
A range of studies have concluded that neural word prediction models can distinguish grammatical from ungrammatical sentences with high accuracy. However, these studies are based primarily on monolingual evidence from English. To investigate how these models' ability to learn syntax varies by language, we introduce CLAMS (Cross-Linguistic Assessment of Models on Syntax), a syntactic evaluation suite for monolingual and multilingual models. CLAMS includes subject-verb agreement challenge sets for English, French, German, Hebrew and Russian, generated from grammars we develop. We use CLAMS to evaluate LSTM language models as well as monolingual and multilingual BERT. Across languages, monolingual LSTMs achieved high accuracy on dependencies without attractors, and generally poor accuracy on agreement across object relative clauses. On other constructions, agreement accuracy was generally higher in languages with richer morphology. Multilingual models generally underperformed monolingual models. Multilingual BERT showed high syntactic accuracy on English, but noticeable deficiencies in other languages.
Neural networks can be trained to predict words from their context with much greater accuracy than the architectures used for this purpose in the past. This has been shown to be the case for both recurrent neural networks (Mikolov et al., 2010; Sundermeyer et al., 2012; Jozefowicz et al., 2016) and non-recurrent attention-based models (Devlin et al., 2019; Radford et al., 2019).
To gain a better understanding of these models' successes and failures, in particular in the domain of syntax, proposals have been made for testing the
models on subsets of the test corpus where successful word prediction crucially depends on a correct analysis of the structure of the sentence (Linzen et al., 2016). A paradigmatic example is subject-verb agreement. In many languages, including English, the verb often needs to agree in number (here, singular or plural) with the subject (asterisks represent ungrammatical word predictions):
(1) The key to the cabinets is/\*are next to the coins.
To correctly predict the form of the verb (underlined), the model needs to determine that the head of the subject of the sentence—an abstract, structurally defined notion—is the word key rather than cabinets or coins.
The approach of sampling challenging sentences from a test corpus has its limitations. Examples of relevant constructions may be difficult to find in the corpus, and naturally occurring sentences often contain statistical cues (confounds) that make it possible for the model to predict the correct form of the verb without an adequate syntactic analysis (Gulordava et al., 2018). To address these limitations, a growing number of studies have used constructed materials, which improve experimental control and coverage of syntactic constructions (Marvin and Linzen, 2018; Wilcox et al., 2018; Futrell et al., 2019; Warstadt et al., 2019a).
Existing experimentally controlled data sets—in particular, those targeting subject-verb agreement—have largely been restricted to English. As such, we have a limited understanding of the effect of the cross-linguistic variability in neural networks' syntactic prediction abilities. In this paper, we introduce the Cross-Linguistic Assessment of Models on Syntax (CLAMS) data set, which extends the subject-verb agreement component of the Marvin and Linzen (2018) challenge set to French, German, Hebrew and Russian. By focusing on a single lin
† Work done while at Johns Hopkins University. Now in the University of British Columbia's Linguistics Department.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5523-5539 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
guistic phenomenon in related languages, $^{1}$  we can directly compare the models' performance across languages. We see the present effort as providing a core data set that can be expanded in future work to improve coverage to other languages and syntactic constructions. To this end, we release the code for a simple grammar engineering framework that facilitates the creation and generation of syntactic evaluation sets. $^{2}$
We use CLAMS to test two hypotheses. First, we hypothesize that a multilingual model would show transfer across languages with similar syntactic constructions, which would lead to improved syntactic performance compared to monolingual models. In experiments on LSTM language models (LMs), we do not find support for this hypothesis; contrarily, accuracy was lower for the multilingual model than the monolingual ones. Second, we hypothesize that language models would be better able to learn hierarchical syntactic generalizations in morphologically complex languages (which provide frequent overt cues to syntactic structure) than in morphologically simpler languages (Gulordava et al., 2018; Lorimor et al., 2008; McCoy et al., 2018). We test this using LSTM LMs we train, and find moderate support for this hypothesis.
In addition to our analysis of LSTM LMs, we demonstrate the utility of CLAMS for testing pretrained word prediction models. We evaluate multilingual BERT (Devlin et al., 2019), a bidirectional Transformer model trained on a multilingual corpus, and find that this model performs well on English, has mixed syntactic abilities in French and German, and performs poorly on Hebrew and Russian. Its syntactic performance in English was somewhat worse than that of monolingual English BERT, again suggesting that interference between languages offsets any potential syntactic transfer.
2.1 Word Prediction Models
Language models (LMs) are statistical models that estimate the probability of sequences of words—or, equivalently, the probability of the next word of the sentence given the preceding ones. Currently, the most effective LMs are based on neural networks that are trained to predict the next word in a
large corpus. Neural LMs are commonly based on LSTMs (Hochreiter and Schmidhuber, 1997; Sundermeyer et al., 2012) or non-recurrent attention-based architectures (Transformers, Vaswani et al. 2017). The results of existing studies comparing the performance of the two architectures on grammatical evaluations are mixed (Tran et al., 2018; van Schijndel et al., 2019), and the best reported syntactic performance on English grammatical evaluations comes from LMs trained with explicit syntactic supervision (Kuncoro et al., 2018, 2019). We focus our experiments in the present study on LSTM-based models, but view CLAMS as a general tool for comparing LM architectures.
A generalized version of the word prediction paradigm, in which a bidirectional Transformer-based encoder is trained to predict one or more words in arbitrary locations in the sentence, has been shown to be an effective pre-training method in systems such as BERT (Devlin et al., 2019). While there are a number of variations on this architecture (Raffel et al., 2019; Radford et al., 2019), we focus our evaluation on the pre-trained English BERT and multilingual BERT.
2.2 Acceptability Judgments
Human acceptability judgments have long been employed in linguistics to test the predictions of grammatical theories (Chomsky, 1957; Schütze, 1996). There are a number of formulations of this task; we focus on the one in which a speaker is expected to judge a contrast between two minimally different sentences (a minimal pair). For instance, the following examples illustrate the contrast between grammatical and ungrammatical subject-verb agreement on the second verb in a coordination of short (2a) and long (2b) verb phrases; native speakers of English will generally agree that the first underlined verb is more acceptable than the second one in this context.
(2) Verb-phrase coordination:
a. The woman laughs and talks/\*talk.
b. My friends play tennis every week and then get/\*gets ice cream.
In computational linguistics, acceptability judgments have been used extensively to assess the grammatical abilities of LMs (Linzen et al., 2016; Lau et al., 2017). For the minimal pair paradigm, this is done by determining whether the LM assigns a higher probability to the grammatical member of
<sup>1</sup>English, French, German and Russian are all Indo-European languages, and (Modern) Hebrew syntax exhibits European areal influence (for different perspectives, see Wexler 1990; Zuckermann 2006; Zeldes 2013).
$^{2}$ https://github.com/aaronmueller/clams
the minimal pair than to the ungrammatical member. This paradigm has been applied to a range of constructions, including subject-verb agreement (Marvin and Linzen, 2018; An et al., 2019), negative polarity item licensing (Marvin and Linzen, 2018; Jumelet and Hupkes, 2018), filler-gap dependencies (Chowdhury and Zamparelli, 2018; Wilcox et al., 2018), argument structure (Kann et al., 2019), and several others (Warstadt et al., 2019a).
To the extent that the acceptability contrast relies on a single word in a particular location, as in (2), this approach can be extended to bidirectional word prediction systems such as BERT, even though they do not assign a probability to the sentence (Goldberg, 2019). As we describe below, the current version of CLAMS only includes contrasts of this category.
An alternative use of acceptability judgments in NLP involves training an encoder to classify sentences into acceptable and unacceptable, as in the Corpus of Linguistic Acceptability (CoLA, Warstadt et al. 2019b). This approach requires supervised training on acceptable and unacceptable sentences; by contrast, the prediction approach we adopt can be used to evaluate any word prediction model without additional training.
2.3 Grammatical Evaluation Beyond English
Most of the work on grammatical evaluation of word prediction models has focused on English. However, there are a few exceptions, which we discuss in this section. To our knowledge, all of these studies have used sentences extracted from a corpus rather than a controlled challenge set, as we propose. Gulordava et al. (2018) extracted English, Italian, Hebrew, and Russian evaluation sentences from a treebank. Dhar and Bisazza (2018) trained a multilingual LM on a concatenated French and Italian corpus, and tested whether grammatical abilities transfer across languages. Ravfogel et al. (2018) reported an in-depth analysis of LSTM LM performance on agreement prediction in Basque, and Ravfogel et al. (2019) investigated the effect of different syntactic properties of a language on RNNs' agreement prediction accuracy by creating synthetic variants of English. Finally, grammatical evaluation has been proposed for machine translation systems for languages such as German and French (Sennrich, 2017; Isabelle et al., 2017).
3 Grammar Framework
To construct our challenge sets, we use a lightweight grammar engineering framework that we term attribute-varying grammars (AVGs). This framework provides more flexibility than the hard-coded templates of Marvin and Linzen (2018) while avoiding the unbounded embedding depth of sentences generated from a recursive context-free grammar (CFG, Chomsky 1956). This is done using templates, which consist of preterminals (which have attributes) and terminals. A vary statement specifies which preterminal attributes are varied to generate ungrammatical sentences.
Templates define the structure of the sentences in the evaluation set. This is similar to the expansions of the S nonterminal in CFGs. Preterminals are similar to nonterminals in CFGs: they have a left-hand side which specifies the name of the preterminal and the preterminal's list of attributes, and a right-hand side which specifies all terminals to be generated by the preterminal. However, they are non-recursive and their right-hand sides may not contain other preterminals; rather, they must define a list of terminals to be generated. This is because we wish to generate all possible sentences given the template and preterminal definitions; if there existed any recursive preterminals, there would be an infinite number of possible sentences. All preterminals have an attribute list which is defined at the same time as the preterminal itself; this list is allowed to be empty. A terminal is a token or list of space-separated tokens.
The vary statement specifies a list of preterminals and associated attributes for each. Typically, we only wish to vary one preterminal per grammar such that each grammatical case is internally consistent with respect to which syntactic feature is varied. The following is a simple example of an attribute-varying grammar:
Preterminals are blue and attributes are orange. Here, the first statement is the vary statement. This is followed by a template, with the special S keyword in red. All remaining statements are preterminal definitions. All attributes are spec
ified within brackets as comma-separated lists; these may be multiple characters and even multiple words long, so long as they do not contain commas. The output of this AVG is as follows (True indicates that the sentence is grammatical):
True je sense
False je penses
False je pensons
False je pensez
This particular grammar generates all possible verb forms because the attribute list for  $\mathbf{V}$  in the vary statement is empty, which means that we may generate any  $\mathbf{V}$  regardless of attributes. One may change which incorrect examples are generated by changing the vary statement; for example, if we change  $\mathbf{V}[]$  to  $\mathbf{V}[1]$ , we would only vary over verbs with the 1 (first-person) attribute, thus generating je pense and *je pensons. One may also add multiple attributes within a single vary preterminal (implementing a logical AND) or multiple semicolon-separated vary preterminals (a logical OR). Changing  $\mathbf{V}[]$  to  $\mathbf{V}[1, s]$  in the example above would generate all first-person singular V terminals (here, je pense). If instead we used  $\mathbf{V}[1]$ ;  $\mathbf{V}[s]$ , this would generate all V terminals with either first-person and/or singular attributes (here, je pense, *je penses, and *je pensons).
4 Syntactic Constructions
We construct grammars in French, German, Hebrew and Russian for a subset of the English constructions from Marvin and Linzen (2018), shown in Figure 1. These are implemented as AVGs by native or fluent speakers of the relevant languages who have academic training in linguistics.<sup>3</sup>
A number of the constructions used by Marvin and Linzen are English-specific. None of our languages besides English allow relative pronoun dropping, so we are unable to compare performance across languages on reduced relative clauses (the author the farmers like smile/*smiles). Likewise, we exclude Marvin and Linzen's sentential complement condition, which relies on the English-specific ability to omit complementizers (the bankers knew the officer smiles/*smile).
The Marvin and Linzen (2018) data set includes two additional structure-sensitive phenomena other than subject-verb agreement: reflexive anaphora
Simple Agreement:
The author laughs/\*laugh.
Across a Prepositional Phrase:
The farmer near the parents smiles/\*smile.
Across a Subject Relative Clause:
The officers that love the skater *smiles/smile.
Short Verb Phrase Coordination:
The senator smiles and laughs/\*laugh.
Long Verb Phrase Coordination:
The manager writes in a journal every day and likes/\*like to watch television shows.
Across Object Relative Clause:
The farmer that the parents love swims/\*swim.
Within Object Relative Clause:
The farmer that the parents *loves/love swims.
Figure 1: Syntactic constructions used in CLAMS. Only English examples are shown; for examples in other languages, see Appendix A. Ungrammatical forms are marked with asterisks.
and negative polarity item licensing. We do not include reflexive anaphora, as our languages vary significantly in how those are implemented. French and German, for example, do not distinguish singular from plural third-person reflexive pronouns. Similarly, negative polarity items (NPIs) have significantly different distributions across languages, and some of our evaluation languages do not even have items comparable to English NPIs (Giannakidou, 2011).
We attempt to use translations of all terminals in Marvin and Linzen (2018). In cases where this is not possible (due to differences in LM vocabulary across languages), we replace the word with another in-vocabulary item. See Appendix D for more detail on vocabulary replacement procedures.
For replicability, we observe only third-person singular vs. plural distinctions (as opposed to all possible present-tense inflections) when replicating the evaluation sets of Marvin and Linzen (2018) in any language.
5.1 Corpora
Following Gulordava et al. (2018), we download recent Wikipedia dumps for each of the languages,
3The German grammar was created by a non-native speaker but was then validated by native speakers.
strip the Wikipedia markup using WikiExtractor, $^{4}$  and use TreeTagger $^{5}$  to tokenize the text and segment it into sentences. We eliminate sentences with more than  $5\%$  unknown words.
Our evaluation is within-sentence rather than across sentences. Thus, to minimize the availability of cross-sentential dependencies in the training corpus, we shuffle the preprocessed Wikipedia sentences before extracting them into train/dev/test corpora. The corpus for each language consists of approximately 80 million tokens for training, as well as 10 million tokens each for development and testing. We generate language-specific vocabularies containing the 50,000 most common tokens in the training and development set; as is standard, out-of-vocabulary tokens in the training, development, and test sets are replaced with <unk>.
We experiment with recurrent LMs and Transformer-based bidirectional encoders. LSTM LMs are trained for each language using the best hyperparameters in van Schijndel et al. (2019). We will refer to these models as monolingual LMs. We also train a multilingual LSTM LM over all of our languages. The training set for this model is a concatenation of all of the individual languages' training corpora. The validation and test sets are concatenated in the same way, as are the vocabularies. We use the same hyperparameters as the monolingual models (Footnote 6). At each epoch, the corpora are randomly shuffled before batching; as such, each training batch consists with very high probability of sentences from multiple languages.
To obtain LSTM accuracies, we compute the total probability of each of the sentences in our challenge set, and then check within each minimal set whether the grammatical sentence has higher probability than the ungrammatical one. Because the syntactic performance of LSTM LMs has been found to vary across weight initializations (McCoy et al., 2018; Kuncoro et al., 2019), we report mean accuracy over five random initializations for each
LM. See Appendix C for standard deviations across runs on each test construction in each language.
We evaluate the syntactic abilities of multilingual BERT (mBERT, Devlin et al. 2019) using the approach of Goldberg (2019). Specifically, we mask out the focus verb, obtain predictions for the masked position, and then compare the scores assigned to the grammatical and ungrammatical forms in the minimal set. We use the scripts provided by Goldberg without modification, with the exception of using bert-base-multilingual-cased to obtain word probabilities. This approach is not equivalent to the method we use to evaluate LSTM LMs, as LSTM LMs score words based only on the left context, whereas BERT has access to left and right contexts. In some cases, mBERT's vocabulary does not include the focus verbs that we vary in a particular minimal set. In such cases, if either or both verbs were missing, we skip that minimal set and calculate accuracies without the sentences contained therein.
6.1 LSTMs
The overall syntactic performance of the monolingual LSTMs was fairly consistent across languages (Table 1 and Figure 2). Accuracy on short dependencies without attractors—Simple Agreement and Short VP Coordination—was close to perfect in all languages. This suggests that all monolingual models learned the basic facts of agreement, and were able to apply them to the vocabulary items in our materials. At the other end of the spectrum, performance was only slightly higher than chance in the Across an Object Relative Clause condition for all languages except German, suggesting that LSTMs tend to struggle with center embedding—that is, when a subject-verb dependency is nested within another dependency of the same kind (Marvin and Linzen, 2018; Noji and Takamura, 2020).
There was higher variability across languages in the remaining three constructions. The German models had almost perfect accuracy in Long VP Coordination and Across Prepositional Phrase, compared to accuracies ranging between 0.76 and 0.87 for other languages in those constructions. The Hebrew, Russian, and German models showed very high performance on the Across Subject Relative Clause condition:  $\geq 0.88$  compared to 0.6-0.71
$^{7}$ https://github.com/yoavg/bert-syntax
4https://github.com/attardi/wikiextractor5https://www.cis.uni-muenchen.de/\~schmid/tools/TreeTagger/6 Specifically, we use 2-layer word-level LSTMs with hidden units in each layer, 800-dimensional word embeddings initial learning rate 20.0 (annealed after any epoch in wh validation perplexity did not improve relative to the prev epoch), batch size 20, and dropout probability 0.2.
English	French	German	Hebrew	Russian
Mono	Multi	Mono	Multi	Mono	Multi	Mono	Multi	Mono	Multi
Test Perplexity	57.90	66.13	35.48	57.40	46.31	61.06	48.78	61.85	35.09	54.61
Simple agreement	1.00	1.00	1.00	1.00	1.00	0.96	0.95	0.96	0.91	0.75
VP coordination (short)	0.94	0.96	0.97	0.85	0.99	1.00	1.00	0.95	0.98	0.92
VP coordination (long)	0.76	0.69	0.85	0.72	0.96	0.73	0.84	0.70	0.86	0.72
Across subject rel. clause	0.60	0.63	0.71	0.70	0.94	0.74	0.91	0.84	0.88	0.86
Within object rel. clause	0.89	0.79	0.99	0.99	0.74	0.69	1.00	0.88	0.95	0.88
Across object rel. clause	0.55	0.52	0.52	0.52	0.81	0.74	0.56	0.54	0.60	0.57
Across prepositional phrase	0.63	0.61	0.74	0.63	0.89	0.82	0.88	0.82	0.76	0.61
Average accuracy	0.77	0.74	0.83	0.78	0.90	0.81	0.88	0.81	0.85	0.76
Table 1: LSTM LM test perplexities and accuracies on CLAMS across languages for the language-specific monolingual models and for our multilingual model. Results are averaged across five random initializations. Chance accuracy is 0.5. Boldfaced numbers indicate the model that achieved the highest performance on a given construction across languages.
in other languages (recall that all our results are averaged over five runs, so this pattern is unlikely to be due to a single outlier).
With each of these trends, German seems to be a persistent outlier. This could be due to its marking of cases in separate article tokens—a unique feature among the languages evaluated here—or some facet of its word ordering or unique capitalization rules. In particular, subject relative clauses and object relative clauses have the same word order in German, but are differentiated by the case markings of the articles and relative pronouns. More investigation will be necessary to determine the sources of this deviation.