SlowGuess/ACL_2020_OCR huggingface dataset:
Mostafa Abdou† Vinit Ravishankar
Maria Barrett† Yonatan Belinkov\* Desmond Elliott† Anders Søgaard†
$^{\dagger}$ Department of Computer Science, University of Copenhagen
\*Department of Informatics, University of Oslo
$\spadesuit$ John A. Paulson School of Engineering and Applied Sciences, Harvard University
$\spadesuit$ Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology
Large-scale pretrained language models are the major driving force behind recent improvements in performance on the Winograd Schema Challenge, a widely employed test of commonsense reasoning ability. We show, however, with a new diagnostic dataset, that these models are sensitive to linguistic perturbations of the Winograd examples that minimally affect human understanding. Our results highlight interesting differences between humans and language models: language models are more sensitive to number or gender alternations and synonym replacements than humans, and humans are more stable and consistent in their predictions, maintain a much higher absolute performance, and perform better on non-associative instances than associative ones. Overall, humans are correct more often than out-of-the-box models, and the models are sometimes right for the wrong reasons. Finally, we show that fine-tuning on a large, task-specific dataset can offer a solution to these issues.
Large-scale pre-trained language models have recently led to improvements across a range of natural language understanding (NLU) tasks (Devlin et al., 2019; Radford et al., 2019; Yang et al., 2019), but there is some scepticism that benchmark leaderboards do not represent the full picture (Kaushik and Lipton, 2018; Jumelet and Hupkes, 2018; Poliak et al., 2018). An open question is whether these models generalize beyond their training data samples.
In this paper, we examine how pre-trained language models generalize on the Winograd Schema Challenge (WSC).
Named after Terry Winograd, the WSC, in its current form, was proposed by Levesque et al. (2012) as an alternative to the Turing Test. The
Figure 1: An example pair from the Winograd Schema Challenge (a) and its perturbation (b). The pronoun resolves to one of the two referents, depending on the choice of the discriminatory segment. The perturbation in (b) pluralizes the referents and the antecedents.
task takes the form of a binary reading comprehension test where a statement with two referents and a pronoun (or a possessive adjective) is given, and the correct antecedent of the pronoun must be chosen. Examples are chosen carefully to have a preferred reading, based on semantic plausibility rather than co-occurrence statistics. WSC examples come in pairs that are distinguished only by a discriminatory segment that flips the correct referent, as shown in Figure 1a. Levesque et al. define a set of qualifying criteria for instances and the pitfalls to be avoided when constructing examples (see §3.2). These combine to ensure an instance functions as a test of what they refer to as 'thinking' (or common sense reasoning).
Recent work has reported significant improvements on the WSC (Kocijan et al., 2019; Sakaguchi et al., 2019). As with many other NLU tasks, this improvement is primarily due to large-scale language model pre-training, followed by fine-tuning for the target task. We believe that further examination is warranted to determine whether these impressive results reflect a funda
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7590-7604 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
mental advance in reasoning ability, or whether our models have learned to simulate this ability in ways that do not generalize. In other words, do models learn accidental correlations in our datasets, or do they extract patterns that generalize in robust ways beyond the dataset samples?
In this paper, we conduct experiments to investigate this question. We define a set of lexical and syntactic variations and perturbations for the WSC examples and use altered examples (Figure 1b) to test models that have recently reported improved results. These variations and perturbations are designed to highlight the robustness of human linguistic and reasoning abilities and to test models under these conditions.
Contributions We introduce a new Winograd Schema dataset for evaluating generalization across seven controlled linguistic perturbations. We use this dataset to compare human and language model sensitivity to those perturbations, finding marked differences in model performance. We present a detailed analysis of the behaviour of the language models and how they are affected by the perturbations. Finally, we investigate the effect of fine-tuning with large task-specific datasets, and present an error analysis for all models.
Probing datasets Previous studies have explored the robustness of ML models towards different linguistic phenomena (Belinkov and Glass, 2019), e.g., by creating challenge datasets such as the one introduced here. When predicting subject-verb agreement, Linzen et al. (2016) found that inserting a relative clause hurt the performance of recurrent networks.
A large body of research has since emerged on probing pre-trained (masked) language models for linguistic structure (Goldberg, 2019; Hewitt and Manning, 2019; Lin et al., 2019; Clark et al., 2019) and analysing them via comparison to psycholinguistic and brain imaging data (Abnar et al., 2019; Ettinger, 2019; Abdou et al., 2019; Gauthier and
Levy, 2019). Other recent work has attempted to probe these models for what is referred to as common sense or factual knowledge (Petroni et al., 2019; Feldman et al., 2019). Their findings show that these models do indeed encode such knowledge and can be used for knowledge base completion or common sense mining from Wikipedia.
Clever Hans A considerable amount of work has also been devoted to what might be described as the Clever Hans effect. This work has aimed to quantify the extent to which models are learning what we expect them to as opposed to leveraging statistical artifacts. This line of work has to date revealed significant problems (and some possible solutions to those problem) with reading comprehension datasets (Chen et al., 2016; Kaushik and Lipton, 2018), natural language inference datasets (Tsuchiya, 2018; Gururangan et al., 2018; Poliak et al., 2018; Belinkov et al., 2019a; McCoy et al., 2019), and the story cloze challenge (Schwartz et al., 2017), among others.
Winograd Schema Challenge Trinh and Le (2018) first proposed using neural language models for the WSC, achieving an accuracy of  $63.7\%$  using an ensemble of 14 language models. Ruan et al. (2019) and Kocijan et al. (2019) fine-tune BERT (Devlin et al., 2019) on the PDP (Rahman and Ng, 2012) and an automatically generated MaskedWiki dataset, reaching an accuracy of  $71.1\%$  and  $72.5\%$  respectively. Meanwhile, Radford et al. (2019) report an accuracy of  $70.7\%$  without fine-tuning using the GPT-2 language model. Most recently, Sakaguchi et al. (2019) present an adversarial filtering algorithm which they use for crowd-sourcing a large corpus of WSC-like examples. Fine-tuning RoBERTa (Liu et al., 2019) on this, they achieve an accuracy of  $90.1\%$ .
In an orthogonal direction, Trichelair et al. (2018) presented a timely critical treatment of the WSC. They classified the dataset examples into associative and non-associative subsets, showing that the success of the LM ensemble of Trinh and Le (2018) mainly resulted from improvements on the associative subset. Moreover, they suggested switching the candidate referents (where possible) to test whether systems make predictions by reasoning about the "entirety of a schema" or by exploiting "statistical quirks of individual entities".
In a similar spirit, our work is a controlled
1Code and dataset can be found at: https://github. com/mhany90/enhanced_wsc/
2This contrasts with our results with Transformer-based architecture and is probably explained by memory loss in recurrent networks trained on short sequences. Similarly, Gulordava et al. (2018) tested whether a Recurrent Neural Network can predict long-distance number agreement in various constructions comparing natural and nonsensical sentences where RNNs cannot rely on semantic or lexical cues.
7591
study of robustness along different axes of linguistic variation. This type of study is rarely possible in NLP due to the large size of datasets used and the focus on obtaining improved results on said datasets. Like a carefully constructed dataset which is thought to require true natural language understanding, the WSC presents an ideal testbed for this investigation.
3 Perturbations
We define a suite of seven perturbations that can be applied to the 285 WSC examples, which we refer to as the original examples. These perturbations are designed to test the robustness of an answer to semantic, syntactic, and lexical variation. Each of the perturbations is applied to every example in the WSC (where possible), resulting in a dataset of 2330 examples, an example of each type is shown in Table 1. Crucially, the correct referent in each of the perturbed examples is not altered by the perturbation. The perturbations are manually constructed, except for the sampling of names and synonyms. Further details can be found in Appendix E.
Tense switch (TEN) Most WSC instances are written in the past tense and thus are changed to the present continuous tense (247 examples). The remaining 34 examples are changed from the present to the past tense.
Number switch (NUM) Referents have their numbers altered: singular referents (and the relevant pronouns) are pluralised (223 examples), and plural referents are modified to the singular (30 examples). Sentences with names have an extra name added via conjunction; eg. "Carol" is replaced with "Carol and Susan". Possessives only mark possession on the second conjunct ("John and Steve's uncle" rather than "John's and Steve's uncle").
Gender switch (GEN) Each of the referents in the sentence has their gender switched by replacing their names with other randomly drawn frequent English names of the opposite gender.  $392\%$  of the generated data involved a gender switch for a name. Though humans may be biased towards gender (Collins, 2011; Desmond and Danilewicz, 2010; Hoyle et al., 2019), the perturbations do not
introduce ambiguity concerning gender, only the entity. 101 examples were switched from male to female, and 55 examples the other way around.
Voice switch (VC) All WSC examples, except for 210 and 211, are originally in the active voice and are therefore passivized. 210 and 211 are changed to the active voice. 65 examples could not be changed. Passive voice is known to be more difficult to process for humans (Olson and Filby, 1972; Feng et al., 2015).
Relative clause insertion (RC) A relative clause is inserted after the first referent. For each example, an appropriate clause was constructed by first choosing a template such as "who we had discussed" or "that is known for" from a pre-selected set of 19 such templates. An appropriate ending, such as "who we had discussed with the politicians" is then appended to the template depending on the semantics of the particular instance. Relative clauses impose an increased demand on working memory capacity, thereby making processing more difficult for humans (Just and Carpenter, 1992; Gibson, 1998).
Adverbial qualification (ADV) An adverb is inserted to qualify the main verb of each instance. When a conjunction is present both verbs are modified. For instances with multiple sentences, all main verbs are modified.
Synonym/Name substitution (SYN/NA) Each of the two referents in an example is substituted with an appropriate synonym, or if it is a name, is replaced with a random name of the same gender from the same list of names used for the gender perturbation.
3.1 Human Judgments
We expect that humans are robust to these perturbations because they represent naturally occurring phenomena in language; we test this hypothesis by collecting human judgements for the perturbed examples. We collect the judgments for the perturbed examples using Amazon Mechanical Turk. The annotators are presented with each instance where the pronoun of interest is boldfaced and in red font. They are also presented with two options, one for each of the possible referents. They are then instructed to choose the most likely option, in exchange for $0.12. Following Sakaguchi et al. (2019), each instance is annotated by three anno
<sup>3</sup>Names sourced from https://github.com/AlessandroMinoccheri/human-names/tree/master/data
7592
Instance / Perturbed Instance	Count
Original	Sid explained his theory to Mark but he couldn’t convince him.	285
Tense	Sid is explaining his theory to Mark but he can’t convince him.	281
Number	Sid and Johnny explained their theory to Mark and Andrew but they couldn’t convince them.	253
Gender	Lucy explained her theory to Emma but she couldn’t convince her.	155
Voice	The theory was explained by Sid to Mark but he couldn’t convince him.	220
Relative clause	Sid, which we had seen on the discussion panel with Chris, explained his theory to Mark but he couldn’t convince him.	283
Adverb	Sid diligently explained his theory to Mark but he couldn’t convince him.	283
Synonyms/Names	John explained his theory to Jad but he couldn’t convince him.	285
Table 1: Examples from our dataset of the different perturbations applied to a WSC instance.
tators and majority vote results are reported. Results are reported later in §5. All three annotators agreed on the most likely option in  $82 - 83\%$  of the instances, except for gender, where a full agreement was obtained for only  $78\%$  of the instances. See Appendix B for further annotation statistics, a sample of the template presented to annotators, and restrictions applied to pool of annotators. We did not require an initial qualification task to select participants.
3.2 Confounds and Pitfalls
Constructing WSC problems is known to be difficult. Indeed, the original dataset was carefully crafted by domain experts and subsequent attempts at creating WSC-like datasets by non-experts such as in Rahman and Ng (2012) have produced examples which were found to be less challenging than the original dataset. Two likely pitfalls listed in Levesque et al. (2012) concern A) statistical preferences which make one answer more readily associated with the special discriminatory segment or other components of an example4 (this is termed as Associativity, and it is described as non-Google-proofness in Levesque et al. (2012)); and B) inherent ambiguity which makes the examples open to other plausible interpretations. In what follows, we discuss these pitfalls, demonstrating that the perturbed examples remain resilient to both.
Quantifying Associativity To verify that the perturbations have not affected the correctness of
Figure 2: PMI divergence from the original WSC examples in average  $\Delta$  for each perturbation. Values below 0 indicate that the difference in PMI between the correct candidate and the incorrect one decreased.
the original problems with regards to pitfall A, we employ pointwise mutual information (PMI) to test the associativity of both the original and perturbed examples. PMI is known to be a reasonable measure of associativity (Church and Hanks, 1990) and, among a variety of measures, has been shown to correlate best with association scores from human judgements of contextual word association (Frassinelli, 2015). We compute unigram PMI on the two corpora used to train BERT (see Appendix C for details). Figure 2 shows the divergence of the perturbed examples from the original WSC dataset. We estimate divergence as the average difference in PMI between the correct  $(\mathcal{C})$  and incorrect  $(\mathcal{I})$  candidates:  $\Delta = pmi(c_j,x_j) - pmi(i_j,x_j)$  where  $\mathcal{X}$  is either: i) the discriminatory segments or ii) the full text of the example, and  $pmi(\cdot ,\cdot)$  is average unigram PMI.  $\Delta$  can be
$^{4}$ Trichelair et al. (2018) find that  $13.5\%$  of examples from the original WSC might still be considered to be associative.
7593
seen as a measure of whether the correct or incorrect candidate is a better 'associative fit' for either the discriminatory segment or the full context, making the examples trivial to resolve. Observe that this difference in PMI declines for the perturbed examples, showing that these the perturbed example do not increase in associativity.
Confirming Solvability Three expert annotators are asked to solve the small subset of examples (99 in total across perturbations) which were annotated incorrectly by the majority vote of Mechanical Turk workers. To address pitfall B, the expert annotators are asked to both attempt to solve the instances and indicate if they believe them to be too ambiguous to be solved. The majority vote of the annotators determines the preferred referent or whether an instance is ambiguous. Out of a total of 99 examples, 10 were found to be ambiguous. Of the remaining 89 examples, 67 were answered correctly by the majority vote. See Appendix D for more details.
4 Experimental Protocol
Our experiments are designed to test the robustness of language models to the Winograd Schema perturbations described in the previous section.
Evaluation Models are evaluated using two types of measures. The first is accuracy. For each of the perturbations, we report (a) the accuracy on the perturbed set (Perturbation accuracy), (b) the difference in accuracy on the perturbed set and on the equivalent subset of original dataset:  $\Delta_{\mathrm{Acc}} =$  Perturbation accuracy - Original subset accuracy, and (c) Pair accuracy, defined as the number of pairs for which both examples in the pair are correctly answered divided by the total number of pairs.
The second measure is stability,  $S$ . This is the proportion of perturbed examples  $\mathcal{P}'$  for which the predicted referent is the same as the original prediction  $\mathcal{P}$ :
$$
S = \frac {\mid \left\{(p _ {i} ^ {\prime} , p _ {i}) \mid p _ {i} ^ {\prime} \in \mathcal {P} ^ {\prime} \wedge p _ {i} \in \mathcal {P} \wedge p _ {i} ^ {\prime} = p _ {i} \right\} \mid}{\mid \mathcal {P} \mid}
$$
Since the perturbations do not alter the correct refererent, this provides a strong indication of robustness towards them.
Baseline We take the unigram PMI between candidates and discriminatory segments (see §3.2) as a baseline. We expect that this simple baseline will perform well for instances with a high level of associativity but not otherwise.
Language Models Our analysis is applied to three out-of-the-box language models (LMs): BERT (Devlin et al., 2019), ROBERTA (Liu et al., 2019), and XLNET (Yang et al., 2019). These models are considered to be the state-of-the-art for the wide variety of natural language understanding tasks found in the GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) benchmarks. We use the large pre-trained publicly available models (Wolf et al., 2019).<sup>7</sup>
Fine-tuned Language Models We also examine the effect of fine-tuning language models. BERT+WW uses BERT fine-tuned on the MaskedWiki and WscR datasets which consist of 2.4M and 1322 examples (Kocijan et al., 2019), and RoBERTa+WG is fine-tuned on WinoGrande XL, which consists of 40,938 adversarially filtered examples (Sakaguchi et al., 2019). Both fine-tuned models have been reported by recent work to achieve significant improvements on the WSC.
Scoring To score the two candidate referents in each WSC instance we employ one of two mechanisms. The first, proposed in Trinh and Le (2018) and adapted to masked LMs by Kocijan et al. (2019) involves computing the probability of the two candidates  $c_1$  and  $c_2$ , given the rest of the text in the instance  $s$ . To accomplish this, the pronoun of interest is replaced with a number of MASK tokens corresponding to the number of tokens in each of  $c_1$  and  $c_2$ . The probability of a candidate,  $p(c|s)$  is then computed as the average of the probabilities assigned by the model to the candidate's tokens and the maximum probability candidate is taken as the answer. This scoring method is used for all models, except ROBERTA+WG. For that, we follow the scoring strategy employed in Sakaguchi et al. (2019) where an instance is split into context and option using the candidate answer as a delimiter. $^8$
$^{7}$ https://github.com/huggingface/pytorch-transformers $^{8}$  [CLS] context [SEP] option [SEP], e.g. [CLS] The sculpture rolled off the shelf because --- [SEP] wasn't anchored [SEP]. The blank is filled with either option 1 (the sculpture) or 2 (the trophy).
Graduate students of linguistics.
<sup>6</sup>Recall that is was not possible to perturb all examples.
7594
Following the experimental protocol, we evaluate the three out-of-the-box language models and the two fine-tuned models on the original WSC and each of the perturbed sets. Table 2 shows Perturbation accuracy results for all models $^{9}$  and contrasts them with human judgements and the PMI baseline.
5.1 Language Models
Humans maintain a much higher performance compared to out-of-the-box LMs across perturbations. The difference in accuracy between the perturbed and original examples,  $\Delta_{\mathrm{Acc}}$ , as defined in Section 4 is shown in Figure 4. A general trend of decrease can be observed for both models and humans across the perturbations. This decline in accuracy is on average comparable between models and humans — with a handful of exceptions. Taking the large gap in absolute accuracy into account, this result might be interpreted in two ways. If a comparison is made relative to the upper bound of performance, human performance has suffered from a larger error increase. Alternately, if we compare relative to the lower bound of performance, then the decline in the already low performance of language models is more meaningful, since 'there is not much more to lose'.
A more transparent view can be gleaned from the stability results shown in Table 3. Here it can be seen that the three out-of-the-box LMs are substantially more likely to switch predictions due to the perturbations than humans. Furthermore, we observe that the LMs are least stable for word-level perturbations like gender (GEN), number (NUM), and synonym or name replacement (SYN/NA), while humans appear to be most affected by sentence-level ones, such as relative clause insertion (RC) and voice perturbation (VC).
Understanding Language Model Performance
To better understand the biases acquired through pre-training which are pertinent to this task, we consider a) a case of essential feature omission and b) the marginal cases where LMs answer very correctly or incorrectly, in both the original and perturbed datasets. We present analysis for BERT, but similar findings hold for the other LMs.
Masking discriminatory segments result in identical sentence pairs because these segments are the only part of a sentence that sets WSC pairs apart (see Figure 1a). To determine whether there is a bias in the selectional preference for one of the candidates over the other, we test BERT on examples where these discriminatory segments have been replaced with the MASK token. An unbiased model should be close to random selection but BERT consistently prefers (by a margin of  $\sim 25 - 30\%$ ) the candidate which appears second in the text to the one appearing first, for all perturbations except voice, where it prefers the first. This observation holds even when the two referents are inverted, which is possible for the 'switchable' subset of the examples as shown in Trichelair et al. (2018). This indicates that the selections are not purely semantic but also syntactic or structural and it points towards BERT having a preference referents in the object role. Detailed results are presented in Appendix F.