SlowGuess/ACL_2020_OCR huggingface dataset:
Figure 4 thus paints a nuanced picture: distractors confuse the model, but they do not fool it completely. Given the results in the previous section, we expect that distractors sharing morphosyntactic features will be particularly challenging. Table 5 confirms this, zooming in into pronominal distractors. We consider a datapoint having a pronominal distractor if one of the antecedents is a pronoun pointing to another entity.
When there are no pronominal distractors (25.9% of the test set), the accuracy of the Transformer is 81.8%; with at least one distractor, it goes down to 73.8% —clearly worse but not dramatically so. However, in cases where anaphoric pronoun and antecedent have the same gender, number, or are the same pronoun, we get much lower accuracies (48.6, 65.3, and 49.1, respectively). This suggests that the model overly relies on morphosyntactic features and recency (see previous section).<sup>7</sup>
However, accuracy in these cases goes down but is still decent, compared to a reasonable baseline (last column in the table). For each target anaphoric pronoun, we calculate baseline accuracy as the percentage of gold pronouns in the window (pronouns that are in the same chain as the target), that is, number of gold pronouns divided
While most personal pronouns refer to people, which are relatively homogeneous kinds of referents, it refers to very varied kinds of referents. Qualitative analysis suggests that the model is quite successful when it refers to concrete entities (province, peanut), but much less when it refers to abstract objects like propositions or events, as in example 4 of Figure 3 (where it refers to the event of trying to improperly influence a witness). A quantitative check confirms this hypothesis: Cases in which the model fails have around  $18\%$  of verbal references, compared to less than  $2\%$  for cases in which the model is right.
7Among the hardest cases are those where two coreference chains in the window have the same pronoun (e.g. he) or gender (e.g. he-his). Most of these cases appear when the text includes reported speech (see Figure 3, example 1). Otherwise, there are few cases of such local ambiguity, which is presumably avoided by language speakers. However, qualitative analysis suggests that the presence of distractors is also problematic in the case of nouns, as illustrated in example 2 of Figure 3, where the model is presumably confused by a noun of the same gender and number as the pronoun (priest vs. Peter-him).
4182
1. Why had Mr. Korotich been called? "I told my driver," he said, "that he
2. While Peter was still in the yard, a servant girl of the high priest came there. She saw him warming himself by the fire. She looked closely at him
3. The performance by more than 40 members of the Rome Philharmonic Orchestra intoxicated the audience and the musical fountain, hi-fi sound effect, fountain screen and stereographic projection brough them
4. Mr. Gonzalez expressed concern over a report that the two had been summoned to Washington by Mr. Wall last week to discuss their testimony in advance. "I think he is trying to improperly influence a witness, and by God I'm not going to tolerate it
Figure 3: Difficult cases of anaphora. The target pronoun and its antecedent are in bold; the prediction of the model is in italic.
Figure 4: Transformer probe model: Accuracy as a function of the proportion of mentions that are antecedents (vs. distractors) in the window.
Type	Perc.	L Acc.	T Acc.	Base Acc.
No distractor	25.9	74.9	81.8	100
Distractor(s) = gender* = number = pron.	74.1	61.3	73.8	32.0
4.8	40.9	48.6	15.7
37.2	55.7	65.3	26.6
10.3	39.7	49.1	20.3
Table 5: Percentage of datapoints with/without pronominal distractors and accuracy of the models (LSTM - L, Transformer - T) and baseline (last column). *Excludes cases with no marked gender (like  $I$ , you).
by the total number of pronouns in the window. Then we calculate the average of this accuracy over the respective subset (no distractors / distractors / same gender, etc.). The baseline when there are no distractors is by definition  $100\%$ ; when there are distractors, it ranges between 15.7 and  $32\%$ . All model accuracies are well above this baseline.
The results thus suggest that the models are able to distinguish mentions of different entities to some extent, although they are far worse at this than at capturing morphosyntactic features. In the following subsection, we provide further support for this interpretation.
4.2 Distinguishing entities
Our last piece of analysis looks at whole documents. We aim at testing whether the hidden representations of the language models contain information that can help distinguish mentions of the same entity from mentions of some other entity,
even if they are of the same form; for instance, a pronoun she referring to two different women. We use coreference chains to identify the tokens referring to the same entity, and train a probe model to determine when two pronouns are referring to the same entity, that is, whether they are part of the same coreference chain in a document. In the previous probe task, where the model was trained to find a correct local antecedent, the model could use cues such as linear distance and syntactic relations; here it should rely on more persistent entity-related features in the hidden representations.
Experimental Setup. We focus on pronouns because they cannot be disambiguated on the basis of lexical features. We use the same train/test partition as in the first probe task. For each datapoint, we have two pronouns:  $\mathbf{x}$  and  $\mathbf{y}$ , which can either come from the same chain, or not. Again, we take each pronoun to be represented by the last hidden layer representation of the language model (Eq. (1)):  $h_x$  and  $h_y$ . We call this representation un
supervised, and will compare it to the supervised one, obtained as follows.
Similarly to the previous probe task, the embeddings are transformed through a learnt linear transformation to a 400-dimensional vector to extract features relevant for the entity identification task (Eqs. (6) and (7)). We take the cosine between the transformed representations as the similarity between the two pronouns.
We take as positive datapoints contain two pronouns belonging to the same chain, as negative datapoints two pronouns from two different chains. During training, for each document, we extract all positive pairs and then randomly select the same number of negative pairs. The model optimises max-margin loss on these datapoints (Eq. (8), where  $x$  and  $y$  belong to the same chain and  $x'$  and  $y'$  belong to two different chains).
$$
o _ {x} = W * h _ {x} + b \tag {6}
$$
$$
o _ {y} = W * h _ {y} + b \tag {7}
$$
$$
L = 1 - \cos \left(o _ {x}, o _ {y}\right) + \cos \left(o _ {x ^ {\prime}}, o _ {y ^ {\prime}}\right) \tag {8}
$$
Results Figure 5 plots the similarities between positive and negative pairs (solid and dashed lines, respectively) for the two analyzed language models, compared to linear distance in the text. The left graph corresponds to unsupervised similarities, the right graph to supervised similarities. To control for token form effect, we only include data with the same pronoun pairs in this graph. Three results stand out. First, despite training with a global objective, with no linear information, similarities are negatively correlated with linear distance in text. This is consistent with the tendency of the unsupervised cosine baseline of pointing to the closest token (see Section 3).
The second result is that, crucially, after controlling both for distance and for pronoun form, similarities are systematically higher for corefering pronoun pairs than for non-corefering ones. Thus, some properties make their way into the hidden representations (and the probe model) that make corefering mentions distinct from noncorreferring mentions — modulo distance: If we attempt to globally distinguish chains, we instead obtain null results (see Supplementary Materials). This is because, with linear distance, the similarity in the entity-centered representation space shrinks very fast; same-chain mentions that are
further away have lower average similarities than different-chain mentions that are nearby.
Finally, the third main result is that the supervised model is able to extract discriminating information from the hidden layers to a much larger extent in the Transformer than in the LSTM (cf. distance between blue and red lines, respectively). We interpret this to mean that such information is encoded to a larger extent in the Transformer. Also note that the supervised LSTM model is more sensitive to linear distance than any of the other representations (cf. the steeper curves between 0-100 token distances). As we signaled in the previous section, LSTM is more prone to recency biases, and it looks like global representations contain less entity-related information than in the case of the Transformer, such that the supervised model defaults to recency. We conclude from this that the Transformer accounts for semantic-referential aspects better than the LSTM.
Overall, the results suggest that token form and proximity in text remain the main properties encoded in the hidden states of entity mentions, but other properties that discriminate between corefering and non-corefering mentions are present to some extent, allowing for partial discrimination.
Previous work has provided robust evidence that language models capture grammatical information without being explicitly trained to do so (Linzen et al., 2016; Gulordava et al., 2018). In this paper, we have analyzed to what extent they learn referential aspects of language, focusing on anaphora. We have tested two models representative of the prevailing architectures (Transformer, LSTM), and our methodology can be extended to any other architecture.
We find that the two models behave similarly, but the Transformer performs consistently better (around  $10\%$  higher accuracy in the probe tasks). Future work should test other architectures, like CNN-based LMs and LSTMs with attention, to provide additional insights into the linguistic capabilities of language models.
As expected, our results show that language models capture morphosyntactic facts about anaphora: Based on the information in the hidden layers, a simple linear transformation learns to link
With the caveat that the model we tested is slightly bigger than its LSTM counterpart.
Figure 5: Linear distance in the discourse vs. cosine distance, for all the mention pairs with the same token pronoun. Distances averaged within bins of 20 tokens. Left: unsupervised, right: supervised.
pronouns to other pronouns or noun phrases, and to do so largely respecting agreement constraints in gender and number.
Although it is much harder for models to induce a more global notion of entity (what we have called semantic-referential aspects), models seem to encode entity-specific information to some extent. Models get confused when there are other mentions in the context, especially if they match in some morphosyntactic feature, but less than could be expected; and they show some limited ability to distinguish mentions that have the same form but are in different coreference chains, though hampered by their heavy recency bias. The recency bias affects LSTMs more, but is also found in Transformers, consistent with previous work on syntax (van Schijndel et al., 2019).
Our results thus suggest that language models are more successful at learning grammatical constraints than they are at learning truly referential information, in the sense of capturing the fact that we use language to refer to entities in the world; however, they still do surprisingly well at referential aspects, given that they are trained on text alone. Future work should investigate where these primitive referential abilities stem from and how they can be fostered in future architectures and training setups for language modeling, and neural models more generally.
We gratefully acknowledge the AMORE team for the feedback, advice and support. We are also grateful to the anonymous reviewers for their valuable comments. This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 re
search and innovation programme (grant agreement No 715154), and from the Spanish Ramón y Cajal programme (grant RYC-2015-18907). We thankfully acknowledge the computer resources at CTE-POWER and the technical support provided by Barcelona Supercomputing Center (RES-IM2019-3-0006). We are grateful to the NVIDIA Corporation for the donation of GPUs used for this research. We are also very grateful to the Pytorch developers. This paper reflects the authors' view only, and the EU is not responsible for any use that may be made of the information it contains.
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. arXiv preprint arXiv:1608.04207.
Laura Aina, Carina Silberer, Ionut Sorodoc, Matthijs Westera, and Gemma Boleda. 2019. What do entity-centric models learn? insights from entity linking in multi-party dialogue. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3772-3783.
Samuel Broscheit. 2019. Investigating entity knowledge in BERT with simple neural end-to-end entity linking. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 677-685.
Pengxiang Cheng and Katrin Erk. 2019. Attending to entities for better text understanding. arXiv preprint arXiv:1911.04361.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286.
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word Translation Without Parallel Data. In ICLR 2018.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988.
Richard Futrell, Ethan Wilcox, Takashi Morita, and Roger Levy. 2018. RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency. arXiv preprint arXiv:1809.01329.
Mario Giulianielli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 240-248.
Yoav Goldberg. 2019. Assessing BERT's syntactic abilities. arXiv preprint arXiv:1901.05287.
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1195-1205, New Orleans, Louisiana. Association for Computational Linguistics.
Stevan Harnad. 1990. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335-346.
Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2019. Tracking the world state with recurrent entity networks. In 5th International Conference on Learning Representations, ICLR 2017.
John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733-2743, Hong Kong, China. Association for Computational Linguistics.
Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: The  $90\%$  solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 57-60, New York City, USA. Association for Computational Linguistics.
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926.
Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin Choi, and Noah A Smith. 2017. Dynamic entity representations in neural language models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1830-1839.
Jaap Jumelet, Willem Zuidema, and Dieuwke Hupkes. 2019. Analysing neural language models: Contextual decomposition reveals default reasoning in number and gender assignment. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 1-11.
Leonard Kaufman and Peter J Rousseeuw. 1990. Finding groups in data: an introduction to cluster analysis. John Wiley, New York.
Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426-1436, Melbourne, Australia. Association for Computational Linguistics.
Yair Lakretz, Germán Kruszewski, Théo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019. The emergence of number and syntax units in LSTM language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 11-20.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017. Regularizing and optimizing LSTM language models. arXiv preprint arXiv:1708.02182.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.
R. Mitkov. 2002. Anaphora Resolution. Studies in Language and Linguistics. Longman.
Denis Paperno. 2014. Typology of adjectives benchmark for compositional distributional models. In LREC.
Md Rizwan Parvez, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2018. Building language models for text with named entities. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2373-2383.
Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499-1509.
Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. Anaphora resolution with the ARRAU corpus. In Proceedings of the First Workshop on Computational Models of Reference, Anaphora and Coreference, pages 11-22, New Orleans, Louisiana. Association for Computational Linguistics.
Massimo Poesio, Roland Stuckardt, and Yannick (Eds.) Versley. 2016. Anaphora resolution: Algorithms, resources, and applications. Springer.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1-40, Jeju Island, Korea. Association for Computational Linguistics.
Sameer Pradhan, Lance A. Ramshaw, Mitchell P. Marcus, Martha Palmer, Ralph M. Weischedel, and Nianwen Xue. 2011. Conll-2011 shared task: Modeling unrestricted coreference in ontonotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, CoNLL 2011, Portland, Oregon, USA, June 23-24, 2011, pages 1-27.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of BERT. In Advances in Neural Information Processing Systems, pages 8592-8600.
Peter J Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65.
Marten van Schijndel, Aaron Mueller, and Tal Linzen. 2019. Quantity doesn't buy quality syntax with neural language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International
Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5835-5841.
Rhea Sukthanker, Soujanya Poria, Erik Cambria, and Ramkumar Thirunavukarasu. 2018. Anaphora and coreference resolution: A review. arXiv preprint arXiv:1805.11824.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601.
Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23.
Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about filler-gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211-221, Brussels, Belgium. Association for Computational Linguistics.
Thomas Wolf. 2019. Some additional experiments extending the tech report "assessing BERT's syntactic abilities" by Yoav Goldberg. Technical report.
Zichao Yang, Phil Blunsom, Chris Dyer, and Wang Ling. 2017. Reference-aware language models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1850-1859.
A Additional results for first probe task (local context)
The probe models tend to refer to entities that are further away from the target than the closest gold entity (74.2% cases in the case of the Transformer), suggesting that they do not rely on a simple recency bias either (although both models do exhibit a recency bias, as we show in the main paper). This observation is confirmed when looking at the distribution of predicted antecedents and gold antecedents (Figures 6 and 7).
Figure 8 presents a heatmap of pronominal agreement for AWD-LSTM. Similar to the TransformerXL heatmap from the main paper, we can observe that in the majority of cases, the model predicts same form tokens with some variation either at the gender level or at the number level.
Figure 6: The distances between the pronoun and its gold and predicted antecedents for TransformerXL.
Figure 7: The distances between the pronoun and its gold and predicted antecedents for AWD-LSTM.
Figure 9 presents the performance of AWD-LSTM relative to the number of distractors in the window. While the tendencies seem to be the same as the ones for TransformerXL, the curves are steeper, the model being more confused with a higher number of distractors.
B Additional results for second probe task (global context)
In the main text, we say that, if we attempt to globally distinguish chains, we obtain null results. Here we show the results of the experiment that leads to these null results.
Figure 8: Pronominal agreement: Proportion of cases in which elements in the rows refer to elements in the columns for AWD-LSTM
Figure 9: The accuracy of reference with respect to the ratio of correct versus confounding mentions in the window for AWD-LSTM