Abstract: Learning representations that accurately model semantics is an important goal of natural language processing research. Many semantic phenomena depend on syntactic structure. Recent work examines the extent to which state-of-the-art models for pre-training representations, such as BERT, capture such structure-dependent phenomena, but is largely restricted to one phenomenon in English: number agreement between subjects and verbs. We evaluate BERT's sensitivity to four types of structure-dependent agreement relations in a new semi-automatically curated dataset across 26 languages. We show that both the single-language and multilingual BERT models capture syntax-sensitive agreement patterns well in general, but we also highlight the specific linguistic contexts in which their performance degrades.

Introduction: Learning general-purpose sentence representations which accurately model sentential semantic content is a current goal of natural language processing research BIBREF0, BIBREF1, BIBREF2, BIBREF3. A prominent and successful approach is to pre-train neural networks to encode sentences into fixed length vectors BIBREF4, BIBREF5, with common architecture choices based on recurrent neural networks BIBREF6, BIBREF7, convolutional neural networks, or transformers BIBREF8. Many core linguistic phenomena that one would like to model in general-purpose sentence representations depend on syntactic structure BIBREF9, BIBREF10. Despite the fact that none of the aforementioned architectures have explicit syntactic structural representations, there is some evidence that these models can approximate such structure-dependent phenomena under certain conditions BIBREF11, BIBREF12, BIBREF13, BIBREF14, in addition to their widespread success in practical tasks. The recently introduced BERT model BIBREF15, which is based on transformers, achieves state-of-the-art results on eleven natural language processing tasks. In this work, we assess BERT's ability to learn structure-dependent linguistic phenomena of agreement relations. To test whether BERT is sensitive to agreement relations, we use the cloze test BIBREF16, in which we mask out one of two words in an agreement relation and ask BERT to predict the masked word, one of the two tasks on which BERT is initially trained. BIBREF17 adapted the experimental setup of BIBREF13, BIBREF11 and BIBREF18 to use the cloze test to assess BERT's sensitivity to number agreement in English subject-verb agreement relations. The results showed that the single-language BERT model performed surprisingly well at this task (above 80% accuracy in all experiments), even when there were multiple “distractors” in the sentence (other nouns that differed from the subject in number). This suggests that BERT is actually learning to approximate structure-dependent computation, and not simply relying on flawed heuristics. However, English subject-verb agreement is a rather restricted phenomenon, with the majority of verbs having only two inflected forms and only one morphosyntactic feature (number) involved. To what extent does Goldberg's BIBREF17 result hold for subject-verb agreement in other languages, including more morphologically rich ones, as well as for other types of agreement relations? Building on Goldberg's BIBREF17 work, we expand the experiment to 26 languages and four types of agreement relations, which include more challenging examples. In Section 2, we define what is meant by agreement relations and outline the particular agreement relations under study. Section 3 introduces our newly curated cross-linguistic dataset of agreement relations, while section 4 discusses our experimental setup. We report the results of our experiments in section 5. All data and code are available at https://github.com/geoffbacon/does-bert-agree.

Question: What types of agreement relations do they explore?
Answer: Subject-verb agreement,  noun-determiner agreement,  noun -attributive adjective agreement and noun-predicate adjective agreement.