SlowGuess/NAACL_2022_OCR huggingface dataset:
Sebastian Schuster
schuster@nyu.edu
linzen@nyu.edu
Understanding longer narratives or participating in conversations requires tracking of discourse entities that have been mentioned. Indefinite noun phrases (NPs), such as a dog, frequently introduce discourse entities but this behavior is modulated by sentential operators such as negation. For example, a dog in Arthur doesn't own a dog does not introduce a discourse entity due to the presence of negation. In this work, we adapt the psycholinguistic assessment of language models paradigm to higher-level linguistic phenomena and introduce an English evaluation suite that targets the knowledge of the interactions between sentential operators and indefinite NPs. We use this evaluation suite for a fine-grained investigation of the entity tracking abilities of the Transformer-based models GPT-2 and GPT-3. We find that while the models are to a certain extent sensitive to the interactions we investigate, they are all challenged by the presence of multiple NPs and their behavior is not systematic, which suggests that even models at the scale of GPT-3 do not fully acquire basic entity tracking abilities.
In order to understand longer narratives or to participate in conversations, humans and natural language understanding systems have to keep track of the entities that have been mentioned in the discourse. For example, when someone tells you that Arthur owns a dog, they have introduced the entity of a person named Arthur and the entity of a dog owned by Arthur into the discourse. Once entities have been introduced to the discourse, it is natural to refer back to them with noun phrases or pronouns to elaborate further, e.g., by saying It has a red collar to elaborate on the dog.
While no fully-specified account exists of how humans achieve this feat, many existing theories are based on the idea that humans maintain mental
files (e.g., Heim, 1982; Murez and Recanati, 2016), i.e., explicit memory representations for each entity that encode all properties of an entity and its relation to other entities. When engaging in a conversation or reading a longer narrative, humans then update these representations as they encounter new entities or new information about existing entities.
Large pre-trained language models (LMs) such as GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020), which in recent years have become the dominant foundation for natural language understanding and generation tasks, lack explicit representations of discourse entities. It remains largely an open question to what extent LMs can match human behavior in tracking discourse entities.
The most extensive investigation of this phenomenon has been through evaluations with the LAMBADA dataset (Paperno et al., 2016). LAMBADA consists of a cloze task for which an LM has to predict the last word of naturalistic passages extracted from a corpus. Solving this task requires keeping track of longer contexts, and making a correct guess frequently requires keeping track of the entities mentioned in the passage.
While datasets such as LAMBADA are an invaluable resource for monitoring high-level progress of LMs in their ability to track discourse entities, such datasets lack the granularity to determine for which contexts LMs can and cannot properly track discourse entities. In this work, we draw inspiration from recent targeted evaluation suites geared at lower linguistic levels (e.g., Marvin and Linzen, 2018; Hu et al., 2020b), and introduce a targeted evaluation suite for tracking of discourse entities in English. In particular, we focus on the interactions between different sentential operators and embedding verbs and indefinite noun phrases (see, e.g., Karttunen 1976 and Section 3); for example, we evaluate whether a language model correctly infers that because a sentence with a negation, such as Arthur doesn't own a dog, does not introduce a dis
Human Language Technologies, pages 969 - 982
course entity for a dog, further elaborations about such a non-existent dog should be pragmatically odd, and, as such, their probability should be low compared to matched controls.
To evaluate to what extent language models are sensitive to these interactions, we adapt the psycholinguistic assessment of language models paradigm (Futrell et al., 2019) for discourse entity tracking and discuss the methodological challenges that arise with using this paradigm for discourse phenomena. We introduce two expert-created evaluation suites and use them to evaluate GPT-2 and GPT-3 models. We find that while all the models we evaluated show some sensitivity to preceding context, they lack systematicity and are challenged when contexts contain multiple noun phrases. $^{1}$
The majority of systematic evaluations of autoregressive and masked language models so far have focused on the level of syntax, targeting abilities such as subject-verb agreement (e.g., Linzen et al., 2016; Marvin and Linzen, 2018; Gulordava et al., 2018; Hu et al., 2020b), anaphora agreement and binding constraints (e.g., Marvin and Linzen, 2018; Futrell et al., 2019; Warstadt et al., 2020; Hu et al., 2020a), or filler-gap dependencies (e.g., Wilcox et al., 2018; Chowdhury and Zamparelli, 2018; Da Costa and Chaves, 2020). At higher linguistic levels, Ettinger (2020) compared BERT's (Devlin et al., 2019) behavior on sentences with negation to data from neurolinguistic experiments with humans; Pandaia and Ettinger (2021) investigated to what extent pre-trained language models can extract relevant information from the preceding context, both in the presence and in the absence of distractors; and Pandaia et al. (2021) investigated whether language models can predict connectives (e.g., but) for two given sentences.
More closely related to our work, in the domain of discourse and reference, Upadhye et al. (2020) investigated whether GPT-2 and Transformer-XL (Dai et al., 2019) exhibit similar referential biases in their continuations as humans, i.e., they asked whether models provided with a sentence with two referents are biased similarly as humans when choosing the referent for the next sentence. Kim et al. (2019) used an acceptability judgment task to
determine whether contextual LMs correctly distinguish between definite and indefinite noun phrases.
Sorodoc et al. (2020) and Tenney et al. (2019) used probing methods to investigate whether representations of LSTM- and Transformer-based models contain information about coreference, which also provides some indication of entity tracking abilities. Further, Clark et al. (2019) investigated to what extent attention weights of BERT indicate coreference. These studies found that all evaluated representations contain some information about coreference but not consistently for all entities.
English indefinite noun phrases (NPs) of the form a NOUN interact with the context in complex ways (see, e.g., Karttunen, 1976; Webber, 1979; Heim, 1982, for more extensive discussions of this phenomenon). In affirmative statements, the indefinite NP generally introduces a new entity to the discourse. However, several sentential operators and clause-embedding verbs modulate this behavior. For example, consider the following contrast between an affirmative and a negated sentence, where # indicates a pragmatically odd continuation:
(1) a. Arthur owns a dog and it follows him everywhere he goes.
b. Arthur doesn't own a dog and # it follows him everywhere he goes.
While in the affirmative sentence, the indefinite NP introduces a novel discourse entity, the negation in (1b) prevents the NP from introducing a new entity. In (1b), it is therefore pragmatically odd to refer back to a dog with the pronoun it.
The implicative manage to and the negative implicative fail to in (2a-b) give rise to a similar contrast: The NP under manage to introduces a discourse entity, the NP under fail to does not.
(2) a. Sue managed to write a book. It was a real page-turner.
b. Sue failed to write a book. # It was a real page-turner.
Indefinite NPs embedded under the factive know and the non-factive doubt introduce and fail to introduce a discourse entity, respectively:
(3) a. I know that Michael baked a cake. It was delicious.
b. I doubt that Michael baked a cake. # It was delicious.
1Our evaluation suites along with the results from human experiments and all code for model evaluation is available at https://github.com/sebschu/discourse-entity-lm.
Lastly, modals such as want also block the introduction of a discourse entity, as shown in (4):
(4) a. Mary got a pet rat and it is very loud at night.
b. Mary wants to get a pet rat and # it is very loud at night.
While these patterns generally hold, there are exceptions to these rules. For example, in the first sentence in (5), which could be paraphrased as (6), the indefinite scopes over the negation and therefore it is okay to refer back to the mistake.
(5) Mary didn't find a (specific) mistake. It was in the footnote.
(6) There was a (specific) mistake which Mary did not find. It was in the footnote.
However, without additional context, listeners generally do not infer these so-called specific interpretations of sentences with an indefinite NP, so like Karttunen (1976), we will largely ignore these cases throughout the remainder of this paper.
To what extent are GPT-2 and GPT-3 sensitive to the contrasts that we presented in Section 3? To investigate this question, we adapted the methodology commonly used for syntactic evaluation of language models (e.g., Futrell et al., 2019) and created minimal pairs of contexts that differ in whether they introduce a discourse entity or not. In Experiment 1, we focus on contexts with a single indefinite NP, and in Experiment 2, we focus on sentences with multiple indefinite NPs.
4.1 Experiment 1
Methodology If a language model is sensitive to contexts that differ in whether a discourse entity is introduced or not, we expect the probability of continuations that refer back to the noun phrase in the previous context to be higher when a discourse entity has been introduced than when it has not. Thus, if we have a pair of sentences, such as
(7) a.  $C_{ref}$ : John owns a dog.
b.  $C_{\text{nonref}}$ : John doesn't own a dog.
and a referential continuation, such as
(8)  $R$ : It has a red collar.
then we expect that
$$
P (R \mid C _ {r e f}) > P (R \mid C _ {n o n r e f}).
$$
However, directly comparing these probabilities may be problematic given that  $P(X \mid C_{ref})$  and  $P(X \mid C_{nonref})$  are different probability distributions. In theory it could be, for example, that  $P(X \mid C_{ref})$  assigns a very high probability to exactly one continuation and therefore its entropy is lower than the entropy of  $P(X \mid C_{nonref})$ . In this case, it could be that the inequality above does not hold despite the fact that continuations that refer back to the noun phrase in the previous context are ranked higher in the affirmative than in the negated case. To overcome this issue, we make use of a non-referential control continuation, such as N:
(9) N: It is not a big deal.
This continuation no longer refers back to a noun phrase and is therefore a valid continuation for both contexts. Instead of using the inequality above, we thus compare the relative probabilities of the referential and the control continuations:
$$
\begin{array}{l} \frac {P \left(R \mid C _ {r e f}\right)}{P \left(R \mid C _ {r e f}\right) + P (N \mid C _ {r e f})} \tag {1} \\ > \frac {P (R \mid C _ {n o n r e f})}{P (R \mid C _ {n o n r e f}) + P (N \mid C _ {n o n r e f})} \\ \end{array}
$$
These relative probabilities are less sensitive to the entropy of the distribution: If there is a highly likely continuation (that is neither the referential nor the control continuation) for one context but not the other, the model should still assign relatively less probability mass to the referential continuation compared to the control continuation.
Models We evaluate two autoregressive language models, $^{3}$  GPT-2 and GPT-3. GPT-2 models were trained on the WebText corpus which contains an estimated 8 billion tokens; GPT-3 models were trained on about 500 billion tokens. For GPT-2,
2The psycholinguistic assessment of language models paradigm generally focuses on the probability of individual words rather than sequences to evaluate syntactic phenomena. However, considering that the coreference of it (or other ref
erential expressions) is modulated by an entire sentence or clause (see the contrast between (8) and (9), which both contain the pronoun it), we compare probabilities of sequences.
3We selected these autoregressive models instead of masked language models (MLMs) such as BERT (Devlin et al., 2019) because they are more frequently used to generate texts, and discourse abilities such as entity tracking tend to play a more crucial role in generating text than in classification or span extraction tasks for which MLMs are more frequently used.
we evaluate models of four different sizes (GPT-2: 117M parameters, GPT-2 M: 345M, GPT-2 L: 762M, GPT-2 XL: 1.5B) that are available through the HuggingFace Transformers library (Wolf et al., 2020). For GPT-3, we evaluate the largest available model ("davinci") through the OpenAI API which is assumed to have about 175B parameters.
Materials We manually constructed an evaluation set of 16 base contexts and plausible continuations. Each base context contains different nouns and verbs to minimize lexical effects. From these 16 contexts, we constructed four contrasts for each context, as shown in Table 1, which in total yielded 64 items. We chose to manually construct contexts as opposed to generating sentences from a grammar to guarantee semantic and pragmatic well-formedness of contexts and continuations.
Human evaluation As we mentioned in Section 3, the referential continuations are not necessarily pragmatically odd if the indefinite noun phrase in the context is interpreted as a specific noun phrase. We therefore conducted an online experiment on Prolific to verify that native English speakers disprefer the referential continuations when no discourse entity is introduced, as follows. After two practice items, each participant performed two trials with sentences from the evaluation set. On each trial, participants saw a context along with a referential and a non-referential continuation, and they were asked to indicate their preferred continuation by selecting the continuation that "makes more sense" given the context. For each context, we collected preference judgments from 10 participants. The experiment took on average about 2 minutes to complete and participants received  $0.45 in compensation (\sim$ 14/hr).
Results and discussion Figure 1 shows the proportion of items for which the relative probability of the referential continuation (RRP) is higher for the context that introduces a discourse entity (DEC) than for the context that does not (NDEC), i.e., the proportion of items for which Eq. 1 holds. For three of the four contrasts (affirmative-negation, affirmative-modal, managed-failed) GPT-2 and GPT-3 models exhibited the expected pattern for
almost all items in our evaluation set. For the know-doubt contrast, however, all models performed approximately at chance, suggesting that the models are not sensitive to this contrast.
Figure 1 also shows the results of the human experiment. Participants preferred the referential continuation more following the DECs than following the NDECs for all items of the affirmative- negation and managed-failed contrasts. Further, for these two contrasts, participants overwhelmingly selected the referential continuation for the DECs and the non-referential continuation for the NDECs. This result confirms that the stimuli bring about the theoretically expected contrast in humans.
For the affirmative-modal and the know-doubt contrasts, the results from human participants are less clear-cut. Overall, participants also preferred the referential continuation more in the DECs than in the NDECs. However, for several items, the opposite was the case and the referential continuation was preferred as much or even more in the NDECs than in the DECs. Moreover, unlike in the other two contrasts, participants selected the referential continuation in the NDECs at a high rate.[5]
Considering that the results from the human experiment are not predicted by Karttunen's theory, the model results from the affirmative-modal and the know-doubt contrast should also be interpreted with caution. However, while the lower proportion of expected relative probabilities in the know-doubt condition may suggest that the models are behaving similarly to humans, this is not the case. If one considers the results on an item-by-item basis, they differ from the human results and there is a lot of variability across models such that the five models agree only on less than  $33\%$  of items.
In summary, GPT-2 and GPT-3 overall behaved similarly to humans and generally favored the referential continuation more when the preceding sen
For contexts with modals, some participants commented that they selected the referential continuation because they assumed that the past tense of the continuation was a grammatical mistake. That is, if the tense had been different, the continuation would have been sensible. For example, for the context Michael wants to bake a cake the continuation and it will be the best thing at the picnic is acceptable and differs from the continuation that was presented in the experiment, and it was the best thing at the picnic, only in its tense.
For contexts with doubt, participants frequently seemed to interpret the referential continuation as a reason for the doubt. For example, for the context I doubt that Carla got a pet rat., participants frequently chose the referential continuation It is very noisy at night., presumably because they considered that the rat being noisy made it unlikely that Carla would have got it.
4The model size of GPT-3 is not publicly available but the EleutherAI project estimated the model size by comparing the performance of the models available through the API to published results: https://blog.eleuther.ai/gpt3-model-sizes/.
Contrast	Contexts	Referential continuation	Non-referential continuation
affirmative-negation	Michael baked a cake
Michael didn’t bake a cake	and it was the best thing at the picnic.	and it’s not a big deal.
affirmative-modal	Michael baked a cake
Michael wants to bake a cake	and it was the best thing at the picnic.	and it’s not a big deal.
know-doubt	I know that Michael baked a cake.
I doubt that Michael baked a cake.	It was the best thing at the picnic.	It’s not a big deal.
managed-failed	Michael managed to bake a cake.
Michael failed to bake a cake.	It was the best thing at the picnic.	It’s not a big deal.
Table 1: Example contexts and continuations for one base context in Experiment 1.
Figure 1: Results from Experiment 1. Each bar indicates the proportion of items for which the relative probability of the referential continuation (RRP) is higher for the context that introduces a discourse entity than for the context that does not, i.e., the expected pattern. Dashed lines indicate chance performance levels, and error bars indicate bootstrapped  $95\%$  confidence intervals.
tence introduced a discourse entity. This behavior could be due to at least the following two reasons. It could be that the models indeed correctly combine the sentential operators with the indefinite noun phrase and therefore assign a higher probability to a referential continuation in the DECs. However, it could also be that this result is due to more spurious correlations; for example, it could be that the model learned that clauses with operators such as negation, modals, or negative implicatives are often followed by clauses with a non-referential it. In the second experiment, we tease apart these two explanations and further try to overcome the issues with the stimuli that we observed for the affirmative-modal and know-doubt contrasts.
4.2 Experiment 2
Materials and method We again constructed 16 base contexts that are similar to the ones used in Experiment 1. However, in this experiment, each context contains two indefinite noun phrases with different nouns that are embedded under two different sentential operators. For example, for the affirmative-negation contrast, one of the NPs is embedded under negation, such as a cat in (10).
(10) John owns a dog but he doesn't own a cat.
In such a context, it is natural to continue with a sentence that refers back to the dog, whereas it is unnatural to refer back to a cat. We therefore compared the models' probability of a sentence that refers back to an entity that has been introduced in the context (11a) to a sentence that refers to an entity that has not been introduced (11b).
(11) a. The dog follows him wherever he goes. b. # The cat follows him wherever he goes.
On top of these coreferential continuations, we also constructed non-coreferential continuations for contexts such as (10). These continuations contain one of the nouns present in the context but do not refer back to entities in the previous context. For the non-coreferential continuations, models should assign a higher probability to the continuation with a noun for which no discourse entity had been introduced in the context.
(12) a. The cat that he liked had been adopted by someone else.
Context	Coreferential continuations	Non-coreferential continuations
Mary found a shirt at the store but she didn’t find a hat.	The shirt/#hat is blue.	The hat/#shirt that she tried on didn’t fit.
Mary found a hat at the store but she didn’t find a shirt.	The hat/#shirt is blue.	The shirt/#hat that she tried on didn’t fit.
Mary didn’t find a shirt at the store but she found a hat.	The hat/#shirt is blue.	The shirt/#hat that she tried on didn’t fit.
Mary didn’t find a hat at the store but she found a shirt.	The shirt/#hat is blue.	The hat/#shirt that she tried on didn’t fit.
Table 2: Example contexts and continuations for the affirmative-negation contrast for one base context.
Figure 2: Results from Experiment 2. Dashed lines indicate chance performance levels.
b. # The dog that he liked had been adopted by someone else.
For each of the four contrasts and each base context, we constructed four contexts that crossed the order of the sentential operators and the order of the two nouns used in a context, resulting in 4 contexts per base context and contrast. For each base context, we further constructed two coreferential continuations (one for each noun) and two non-coreferential continuations (one for each noun). In total, this yielded 512 items. Table 2 shows all the contexts and continuations for one base context for the affirmative-negation contrast.
Compared to the methods and materials in Experiment 1, this setup has several advantages. First, given that we are comparing two continuations for a fixed context, both continuations come from the same probability distribution and therefore we no longer need a generic control continuation. Second, it is less likely that models can make use of spurious correlations since each context contains two types of sentential operators and, for example, a heuristic of associating negation with non-referential  $it$  would no longer lead to the expected
behavior. Third, given that all continuations are on topic (as opposed to the generic control condition in Experiment 1), humans likely show more consistency in their preferences. Lastly, given that this design allows us to construct stimuli with exactly the same tokens in different orders, we can also assess the systematicity of the model behavior.
We again verified the theoretically predicted preferences in a human experiment.
Results and discussion Figure 2 shows the accuracy of the model and human experiments for the coreferential and non-coreferential continuations. As this figure shows, humans exhibited the theoretically expected behavior for all contrasts for almost all items and chose the coreferential continuation with the noun for which an entity had been introduced in the context, and chose the non-coreferential continuation for the noun for which no entity had been introduced. This suggests that the materials do not exhibit the same shortcomings