SlowGuess/ACL_2020_OCR huggingface dataset:
Ionut-Teodor Sorodoc* Kristina Gulordava Gemma Boleda*†
*Universitat Pompeu Fabra  
†ICREA  
Barcelona, Spain  
{firstname.lastname}@upf.edu
Language models keep track of complex linguistic information about the preceding context – including, e.g., syntactic relations in a sentence. We investigate whether they also capture information beneficial for resolving pronominal anaphora in English. We analyze two state of the art models with LSTM and Transformer architectures, respectively, using probe tasks on a coreference annotated corpus.
Our hypothesis is that language models will capture grammatical properties of anaphora (such as agreement between a pronoun and its antecedent), but not semantico-referential information (the fact that pronoun and antecedent refer to the same entity). Instead, we find evidence that models capture referential aspects to some extent -though they are still much better at grammar. The Transformer outperforms the LSTM in all analyses, and exhibits in particular better semantico-referential abilities.
Neural network-based language models (LMs) have been shown to learn relevant properties of language without being explicitly trained for them. In particular, recent work suggests that they are able to capture syntactic relations to a large extent (Gulordava et al., 2018; Kuncoro et al., 2018; Wilcox et al., 2018).
In this paper, we extend this line of research to analyze whether they are able to capture referential aspects of language, focusing on anaphoric relations (pronoun-antecedent relations, as in sheYeping Wang in Figure 1).
Previous work, such as Ji et al. (2017), Yang et al. (2017) and Cheng and Erk (2019), showed that augmenting language models with a component that uses an objective based on entity or coreference information improves their performance at
...he<sub>1</sub> was elected to be president of the People's Republic of China, and chairman of the Central<sub>2</sub> Military<sub>2</sub> Commission<sub>2</sub>. Yeping<sub>3</sub> Wang<sub>3</sub> was born in Shanghai in 1926. She<sub>3</sub> studied in Shanghai Foreign Language College, and started working in 1949. For a long time, she<sub>3</sub>…
Figure 1: Example from OntoNotes with a window of 60 tokens (as used in our first probe task). Both occurrences of she refer to the same entity as Yeping Wang. Note that not all entity mentions are annotated in OntoNotes -only those that enter into coreference relationships in the document.
language modeling. Intuitively, in the example in Figure 1, understanding that the first she refers to Yeping Wang makes words related to studying or working more likely to follow than other kinds of words. That is, referential information helps language models do their task.
The cited work includes explicit coreference guidance; however, since referential information is useful for language modeling, we expect language models to learn referential information even without explicit supervision. Here we analyze to what extent this is the case.
We carry out our analysis using probe tasks, or tasks that check whether certain information is encoded in a model (Adi et al., 2016; Linzen et al., 2016; Conneau et al., 2018; Giulianelli et al., 2018). The reasoning is as follows: Even if a linguistic property is encoded in the network, it is not necessarily directly accessible through the model output; therefore, we train a probe model to predict a feature of interest, in this case anaphoric coreference, given the model's hidden representations as input.
We focus on the two main linguistic levels that are relevant for coreference: morphosyntax, with
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4177-4189 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
grammatical constraints such as the fact that pronouns agree in number and gender with their antecedents, and semantics – in particular reference, such as the fact that a pronoun refers to the same entity as its antecedent.
Our hypothesis is that language models will capture grammatical properties, but not semantic information. This hypothesis is based on the observation that morphosyntax is a formal property of language that is easier to induce from co-occurrence patterns. The fact that language refers to entities is not obvious from language alone (Harnad, 1990), and LMs use only textual input.
Instead, what we find is that, while it is true that language models are much better at grammar, they do show evidence of learning semantic-referential information to some extent. Our explanation for this unexpected, partially positive result is that, because the same entity underlies all its mentions, the contexts in which the mentions appear are coherent and distinct from those of mentions of other entities. For instance, in Figure 1, the second she mention gives additional information about Yeping Wang that is consistent with the information given in the previous sentence.
This paper has two main contributions. The first is an analysis methodology to probe for referential information encoded in language models, on two linguistic levels (morphosyntax, semantics) and two kinds of context: local (around one paragraph of context), and global (document context). This methodology can be applied to any architecture. The second contribution is a deeper understanding of the referential capabilities of current language models, and of the differences between Transformers and LSTMs. The Transformer outperforms the LSTM in all the analyses. For morphosyntax, the Transformer and the LSTM have the same behavior with a performance difference; instead, they show different behavior with regard to semantico-referential information.
Coreference and anaphora resolution (Mitkov, 2002; Poesio et al., 2016) are among the oldest topics in computational linguistics and have continued to receive a lot of attention in the last decade, as manifested by several shared tasks (Pradhan et al., 2011, 2012; Poesio et al., 2018). In our analysis we use the OntoNotes dataset (Hovy et al., 2006; Pradhan et al., 2012),
developed within the coreference resolution community. Our probe tasks are related to coreference resolution; however, our goal is not to train a coreference system but to analyse whether language models extract features relevant for reference without explicit supervision.
A recent line of work has focused on demonstrating that neural networks trained on language modeling, without any linguistic annotation, learn syntactic properties and relations such as agreement or filler-gap dependencies (Linzen et al., 2016; Gulordava et al., 2018; Kuncoro et al., 2018; Wilcox et al., 2018; Futrell et al., 2018). This is typically done by analysing the predictions of LMs on controlled sets of data. Part of this research uses probe models (also known as diagnostic models) to analyse the information contained in their hidden representations (Adi et al., 2016; Conneau et al., 2018; Hupkes et al., 2018; Lakretz et al., 2019; Giulianelli et al., 2018), as we do here -applying it to referential information.
There is less work on referential information than on syntactic properties such as subject-verb agreement. As for anaphoric reference, Peters et al. (2018) include a limited test using 904 sentences from OntoNotes. Their results suggest that LMs are able to do unsupervised coreference resolution to a certain extent; our first probe task can be seen as an extended version of their task obtaining more specific insights. Jumelet et al. (2019) analyze the kind of information that LSTM-based LMs use to make decisions in within-sentence anaphora. They find a strong male bias encoded in the network's weights, while the information in the input word embeddings only plays a role in the case of feminine pronouns. We analyze anaphora in longer spans (60 tokens / whole document) and include also a Transformer.
The above work suggests that LMs capture morphosyntactic facts about anaphora to a large extent. There is much less evidence that LMs can capture a notion of entity, as that which nominal elements refer to, and that they are able to track entities across a discourse. Parvez et al. (2018) show that LSTM-based models have poor results on texts with a high presence of entities; Paperno (2014) that they cannot predict the last word of text fragments that require a context of a whole passage (as opposed to the last sentence only), with data that mostly contain nominal elements. Several models (Henaff et al., 2019; Yang et al., 2017;
Ji et al., 2017) were developed as an augmentation of RNN LMs to deal better with entities, with the implicit assumption that standard models do that poorly. Aina et al. (2019) achieved good results on an entity-linking task, but showed that the network was not acquiring entity representations.
As for Transformer-based architectures, recent research suggests that they give same or better contextualized representations in comparison with LSTM language models, and that they better encapsulate syntactic information (Goldberg, 2019; Wolf, 2019). On the other hand, van Schijndel et al. (2019) show that big Transformer model representations perform on par or even poorer than smaller LSTMs on tasks such as number agreement or coordination, and that, like LSTMs, they have the problem that agreement accuracy decreases as the subject becomes more distant from its verb. Most recent work on analysis of linguistic phenomena in NNs focuses on BERT (Tenney et al., 2019; Clark et al., 2019; Reif et al., 2019; Broscheit, 2019). In this paper we chose to use TransformerXL (Dai et al., 2019) as our Transformer model, and not BERT, for comparability: We wanted to compare the two most standard architectures for LMs on as equal ground as possible, and the two chosen models, TransformerXL and AWD-LSTM (Merit et al., 2017), share the same training objective and are trained on the same data, with comparable vocabularies.
3 Morphosyntactic factors
To shed light into which morphosyntactic information LMs encode that is useful for coreference, we train a simple anaphora resolution probe model using the hidden layers of LMs as input. By the logic of probe tasks, if the probe model is successful then that means that the relevant information is encoded in the hidden states, and error analysis can provide insight into which kinds of information are available.
Data We train our probe models on data from OntoNotes 5.0 (Weischedel et al., 2013). We use the annotated coreference chains, as well as the provided part-of-speech tags (the latter only for analysis purposes).
We take all pronouns that have at least one antecedent in a 60-token context window; the task of
Tokens	Datapoints
Train	191,830	4,949
Dev	275,201	4,556
Test	2,026,565	45,665
Table 1: Dataset statistics for first probe task. We reverse the original train and test partitions (see text).
the probe model is to identify their antecedent. An example datapoint is provided in Figure 1 above (note that a window of 60 tokens allows us to check anaphora beyond the sentence). For simplicity, antecedents are tokens, but typically there is more than one possible token antecedent for a given pronoun: A mention can span several tokens (Yeping Wang), and the window can contain several mentions from the same coreference chain (Yeping Wang and the first She in Figure 1); we consider any of the tokens a correct answer. Note that we are not training the model to explicitly identify mentions, their spans or the complete coreference chains, but to identify the tokens that are antecedents of the target pronoun.
To obtain enough data for analysis, especially for low-frequency phenomena, we follow Linzen et al. (2016) in reversing the original partitions of the corpus, using the original test set for training and the original training set for testing. In addition, we focus on the OntoNotes documents that belong to narrative text sections because the dialogue data does not come with turn segmentation. Resulting data statistics for our task are provided in Table 1.
Language models The base language models we use are AWD-LSTM (Merit et al., 2017) and TransformerXL (Dai et al., 2019), two state-of-the-art models with the most standard architec
<sup>1</sup>We also experimented with windows 20 and 200, obtaining a similar picture.
$^{2}$ Using little training data has also been shown to lessen the possibility of confounds in the probe model results; in particular, it makes it more difficult for the probe model to exploit regularities in the training data rather than capturing the analyzed model's ability to capture a phenomenon (Hewitt and Liang, 2019). See Voita and Titov (2020) for a theoretical justification from a information-theoretic perspective.
Results on the original split confirm that the conclusions of the paper are robust: we see an increase in performance of around  $3\%$  overall, as could be expected because we use more data, but the same behavior patterns (on the data that can be compared).
3We keep newswire (NW), broadcast news (BN), magazine (MZ), web data (WB), and pivot text (PT), removing broadcast conversation (BC), telephone conversation (TC).
tures for language modeling as of 2020 (LSTM, Transformer). We chose these models for comparison because they are trained on the same dataset (Wiki103; Merity et al., 2016), they have a comparable vocabulary, and they are both very strong language models, with perplexities of 24 for TransformerXL and 33 for AWD-LSTM. TransformerXL is a bit larger than AWD-LSTM, though (151 million parameters compared to 126), which should be kept in mind when assessing results.
Probe model For each word  $x_{i}$  in the window of size  $m$  preceding the target pronoun  $x_{t}$ , we obtain its contextualized representation  $h_{i}$  from the last hidden layer of the language model (Eq. 1). The probe model takes this representation as input and is trained to map it onto a vector  $o_{i}$  using a non-linear transformation (Eq. 2). The target pronoun representation is transformed in the same way. The dot products between these transformed representations of target and context word vectors give the attention weights  $ref_{i}$  (Eq. 3) representing the similarity between two representations. The weights are transformed into probabilities using the softmax function (Eq. 4). Like this we obtain a probability distribution  $p_{i}$  over context tokens.
During training, the probe model's objective is to assign higher probabilities (and thus attention weights) to correct antecedents, and lower probabilities to incorrect ones, through the use of the Kullback-Leibler divergence loss (Eq. 5). We use the KL loss because we frame the task in terms of a probability distribution over mentions in the context. For the reasons discussed above, there can be  $k > 1$  correct predictions out of  $m$  tokens in the window. We assume that gold probability distribution is uniform over  $k$  correct tokens, that is, each of these tokens has a probability  $p_i^* = \frac{1}{k}$  and all other tokens have a probability of 0.
Model	Accuracy
closest gold entity	56.1
closest same-form token	61.3
unsup.	sup.
LSTM	41.7	64.8
Transformer	48.5	75.9
Table 2: Probe model results on anaphora resolution.
$$
h _ {i} = L S T M \left(x _ {i}\right) \tag {1}
$$
$$
o _ {i} = \operatorname {R e L U} \left(W * h _ {i} + b\right) \tag {2}
$$
$$
r e f _ {i} = o _ {i} \odot o _ {t}, \forall i \in [ t - m, t - 1 ] \tag {3}
$$
$$
p _ {i} = \operatorname {s o f t m a x} \left(\operatorname {r e f} _ {i}\right), \forall i \in [ t - m, t - 1 ] \tag {4}
$$
$$
L = K L \left(p _ {i}, p _ {i} ^ {*}\right) \tag {5}
$$
As mentioned above, we fix  $m = 60$ . We train the probe model for 50 epochs with a learning rate of 1e-5 and ADAM as optimizer. The transformed vectors  $o_i$  have a dimensionality of 650 in the case of both models in comparison with  $h_i$  which is 400 for the AWD-LSTM and 1024 for TransformerXL.
Baselines We report two rule-based baselines that give relatively good performance in anaphora resolution: Referring to the previous entity (given by the oracle gold annotation; in Figure 1, she would refer to the previous She), and always pointing to the token in the window that has the same form as the target pronoun (that is, in Figure 1, she  $\rightarrow$  She—we ignore capitalization). In addition, to compare the result of the probe model with the input representations, we also report an unsupervised baseline: Referring to the token in the window that has the highest similarity  $\cos(h_i, h_t)$  to the target pronoun, i.e., relying on the similarity between the non-transformed hidden representations.
Table 2 summarizes the results of the pronominal anaphora probe task. The probe model trained on top of the LSTM improves a bit over the strongest baseline, and that of the Transformer does so substantially (75.9 vs. 61.3; the LSTM obtains 64.8). This performance suggests that the LMs use more information than simple heuristics like referring to a token with the same form.
The unsupervised similarity baseline performs worse than the rule-based baselines. This is to
4We also trained an in-house LSTM on data that are more similar to those of OntoNotes and a smaller vocabulary. The results for this model (not reported) follow the same patterns as those found for the AWD-LSTM and TransformerXL models, although the performance on this probe task is much higher than that of AWD-LSTM.
Note however that minimizing KL divergence and minimizing cross-entropy gives the same results, because  $KL\text{div}(p||q) = \text{CrossEntropy}(p,q) - \text{entropy}(p)$ , and entropy(p) is constant. Technically, in PyTorch the cross-entropy loss is only implemented for classification task targets, while the more general KL loss is available for predicting probability distributions.
be expected: The "raw" similarity between hidden states is based on many more aspects than those related to reference, given that hidden states are responsible for capturing all the contextual features that are relevant for word prediction. This is why a probe model is needed to distill the reference-related information from the hidden layers.
A single non-linear layer trained on only 5K datapoints improves performance by 23-28 absolute accuracy points (supervised vs. unsupervised results), which suggests that the referential information in the hidden layers is easy to extract. Behaviorally, the unsupervised hidden layers are quite similar to the baselines. First, they are biased towards tokens of the same form: in  $27.1\%$  of the cases, the LSTM layer of the pronoun presents the highest similarity to a token with the same form;  $29.1\%$  in the case of the Transformer. Second, they prefer close antecedents, although the LSTM presents this recency bias to a much higher degree: in  $27.8\%$  of the cases, the LSTM layer of the pronoun has the highest similarity to the previous token ( $16.4\%$  in the Transformer). The attention mechanism of the Transformer gives access to a broader context and allows it to overcome the recency bias to some degree.
The great difference in performance between AWD-LSTM and TransformerXL could suggest that the latter is using different strategies compared to the former. Instead, except for the recency bias, what we find are exactly the same patterns in behavior, with a systematic  $10\%$  accuracy gap. For this reason, although we provide results for both models everywhere to show that this observation indeed holds, in this section we will mostly focus on the Transformer when commenting results.
3.3 Analysis: Morphosyntactic Factors
The models clearly learn grammatical constraints related to anaphora that are well-studied in the literature and are relied upon by traditional anaphora resolution models (Sukthanker et al., 2018). First, as shown in Table 3, the Transformer identifies mentions (elements inside some coreference chain) in  $92.6\%$  of the cases. Moreover, it correctly learns that pronouns typically refer to nominal elements (almost  $95\%$  identified antecedents are pronouns, proper nouns, and elements within a noun phrase headed by a common noun). Note that pronouns can also have non-nominal an-
in chain	LSTM	Transformer
90.2%	92.6%
POS	Perc.	Acc	Perc.	Acc
Noun phrase	15.5	50.9	17.0	62.3
Proper noun	20.2	64.3	20.0	74.9
Pronoun	59.0	71.5	59.0	82.6
Other	5.3	67.3	3.0	81.6
Table 3: Statistics on types of mentions that the probe models refer to, for predictions that are in a coreference chain. 'Noun phrase' stands for elements that are typically within a noun phrase (note that our system points to individual tokens): Determiners, nouns, and adjectives.
tecedents, although these are the minority of the annotations in OntoNotes (cf. example 4 in Figure 3, where it refers to an event). Even in the cases in which the Transformer points to elements outside of a chain (7.4%), it points to nominal elements 87% of the time (not shown in the table). The model is most accurate when referring to pronouns (82.6% accuracy), while noun phrases are the hardest category (62.3%). This is consistent with the strategies that the model learns, since it largely relies on pronominal agreement, as described below.
Second, not only do the models mostly point to nominal elements, but they also identify the morphosyntactic properties of pronouns and learn that they should agree with their antecedents in gender and number. Figure 2 shows the distribution of pronoun antecedents that the Transformer predicts, for the six most frequent target pronouns (see the Supplementary material for the corresponding LSTM figure). Its preferred type of antecedent are pronouns of the same form, but it is also able to point to other pronouns agreeing in number and gender. For instance, pronoun  $he$  points to 3rd person, masculine, singular pronouns (mostly  $he$ , but also his, him)—a pattern consistent across all pronouns.
Figure 2 is restricted to pronouns; Table 4 shows that the model also largely follows number agreement when predicting antecedents within noun phrases (the table collapses common noun and proper noun antecedents). Given a singular pronoun, the model chooses a singular antecedent  $98\%$  of the time; given a plural pronoun, it identifies a plural antecedent in  $73\%$  of the cases.
Note that in cases of plural pronouns such as
Figure 2: Pronominal agreement with Transformer probe model: Proportion of cases in which elements in the rows corefer with elements in the columns.
LSTM	Transformer
Pron-ant.	Perc.	Acc	Perc.	Acc
sg-sg	97.7	66.3	98.7	76.0
sg-pl	2.3	20.5	1.3	36.7
pl-sg	35.5	40.8	27.5	53.1
pl-pl	64.5	67.7	72.5	72.3
Table 4: The types of noun phrase antecedents the models choose, by number agreement (e.g., 'sg-pl' means 'anaphoric pronoun is singular, antecedent plural').
they it is common that the referent be a singular noun (e.g., the audience in example 3, Figure 3), reflected by the reasonable accuracy of the Transformer in pl-sg cases  $(53.1\%)$ .
4 Semantic (referential) factors
The language model clearly captures morphosynthetic (grammatical) properties that constrain anaphora resolution; in this section, we show that it struggles more with its semantic (referential) aspect, but it still captures it to some extent.
4.1 Sensitivity to distractors
If the model were able to model entities, it should be robust to distractors, that is, mentions in the context that are not antecedents –in Figure 1, he and the Central Military Commission. Figure 4 shows that the accuracy for the Transformer decreases as does the proportion of gold mentions. We compute this proportion as the number of gold mentions in the 60-token window divided by the total number of mentions in the same window. When there are no distractors (gold proportion = 1), accuracy is very high, which is to be expected given that the model learnt to identify mentions in the first place (cf. previous section). The more
distractors (i.e., the lower the proportion of gold mentions), the lower the accuracy; however, accuracy decreases rather gracefully. Even when there are only  $10\%$  gold mentions in the window, accuracy for most pronoun types is still around  $60 - 80\%$ . The exception is it, which is the most difficult pronoun for the model, presumably because it can refer to many kinds of antecedents.[6]