SlowGuess/ACL_2020_OCR huggingface dataset:
Marginal examples are found where the model assigns a much higher probability to one referent over the other. We extract the top  $15\%$  examples where the correct candidate is preferred by the largest margin  $(\mathrm{P}_{\mathrm{correct}} \gg \mathrm{P}_{\mathrm{incorrect}})$  and the bottom  $15\%$  where the incorrect one is preferred  $(\mathrm{P}_{\mathrm{incorrect}} \gg \mathrm{P}_{\mathrm{correct}})$ . Surprisingly, we find that there is a large overlap  $(50\% - 60\%)$  between these two sets of examples, both in the original and the perturbed datasets.[10] For the examples which are both the most correct and incorrect, BERT strongly prefers one of the candidates without considering the special discriminatory segment which flips the correct referent. Indeed we find that the correlation between the probability assigned by BERT to a referent when it is the correct referent and when it is not is very strong and significant, with Spearman's  $\rho \approx 0.75$  across perturbations (see Appendix G for details).
To clarify, consider the following original WSC pair:
(i) Alice looked for her friend Jade in the crowd. Since she always has good luck, Alice spotted her quickly.
(ii) Alice looked for her friend Jade in the crowd. Since she always wears a red turban, Alice spotted her quickly.
The first example gives  $\mathbb{P}_{\text{correct}} \gg \mathbb{P}_{\text{incorrect}}$  by the largest margin, and its counterpart gives  $\mathbb{P}_{\text{incorrect}} \gg \mathbb{P}_{\text{correct}}$  by the largest margin. In other words, the model assigns a much higher probability for Alice in both cases.
<sup>9</sup>It is interesting to note that XLNet is trained on CommonCrawl which indexes an online version of the original WSC found here: https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WS.html.
7595
PMI	ORIG	TEN	NUM	GEN	VC	RC	ADV	SYN/NA	Avg	Avg ΔAcc.
54.38	54.09	52.96	57.42	54.09	54.41	54.41	51.92	54.24	-2.13
BERT	61.75	61.92	57.31	57.42	63.64	62.19	61.48	58.59	60.41	-1.26
XLNET	64.56	60.14	62.45	62.58	57.73	62.9	64.31	61.05	61.59	-2.78
ROBERTA	69.82	69.40	64.43	53.55	66.82	68.55	69.61	57.54	64.27	-5.16
BERT+WW	72.28	70.46	71.15	74.84	65.91	64.31	72.44	70.88	70.00	-2.82
ROBERTA+WG	88.42	89.32	88.53	86.45	83.63	86.93	88.7	89.05	87.62	-1.06
HUMANS	97.89	96.79	94.46	92.25	92.27	91.16	95.40	96.14	94.41	-3.83
Table 2: Original dataset accuracy (ORIG) and Perturbation accuracy results for all models and humans. The penultimate column shows the average Perturbation accuracy results. The rightmost column shows the  $\Delta_{\mathrm{Acc}}$  results, averaged over all perturbations.
TEN	NUM	GEN	VC	RC	ADV	SYN/NA	Avg
PMI	100	100	73.91	100	100	100	100	96.27
BERT	89.32	69.17	88.39	79.55	83.75	91.87	68.42	81.40
XLNET	82.21	69.17	66.45	69.55	78.45	84.81	70.53	75.02
ROBERTA	91.46	77.47	61.29	79.09	83.75	89.75	68.77	79.26
BERT+WW	90.04	83.00	89.68	80.45	81.98	92.93	85.96	85.14
ROBERTA+WG	96.08	94.07	97.41	91.36	92.22	94.69	96.11	95.24
HUMANS	96.70	94.9	92.9	91.18	91.11	96.11	96.1	94.31
Table 3: Stability results for all models and humans.
5.2 The effect of fine-tuning
The accuracy and stability results (Tables 2 and 3) indicate that fine-tuning makes language models more robust to the perturbations. ROBERTA+WG, in particular, is the most accurate and most stable model. While impressive, this is not entirely surprising: fine-tuning on task-specific datasets is a well-tested recipe for bias correction (Belinkov et al., 2019b). Indeed, these results provide evidence that it is possible to construct larger fine-tuning datasets whose distribution is correct for the WSC. We note that both fine-tuned models perform worst on the VC and RC perturbations, which may not frequently occur in the crowd-sourced datasets used for fine-tuning. To test this intuition, we apply a dependency parser (UDPipe (Straka et al., 2016)) to the WinoGrande XL examples, finding that only  $\sim 5\%$  of the examples are in the passive voice and  $\sim 6.5\%$  contain relative clauses.
How much fine-tuning data is needed? To quantify the amount of fine-tuning data needed to
achieve robustness, we fine-tune ROBERTA on the five WinoGrande training set splits defined by Sakaguchi et al. (2019):  $\mathbf{X}\mathbf{S}(160)^{11}$ ,  $\mathbf{S}(640)$ ,  $\mathbf{M}$  (2558),  $\mathbf{L}$  (10234), and  $\mathbf{XL}$  (40398). Figure 3 shows the average accuracy and stability scores for the models fine-tuned on each of the training splits<sup>12</sup>. We observe that the two smallest splits do not have a sufficient number of examples to adequately bias the classification head, leading to near-random performance. The model fine-tuned on the  $\mathbf{M}$  split—with just 2558 examples—is, however, already able to vastly outperform the non-fine-tuned ROBERTA. Increasing the number of examples five-fold and twenty-fold leads to significant but fast diminishing improvements.
$^{11}$ No. of examples in set.
12Note that the stability score for the model fine-tuned on XL in Figure 3 is different from that reported in Table 3. In the latter we reported results from the model provided by Sakaguchi et al. (2019), rather than the model we finetuned ourselves. Since we utilise identical hyperparameters to theirs for fine-tuning, this anomalous difference in score may perhaps be explained by a difference in initialization as suggested in Dodge et al. (2020).
7596
Figure 3: Accuracy and stability scores (averaged across perturbations) for ROBERTA when fine-tuned on five increasing training split sizes.
How do perturbations affect token probability distributions? To obtain a holistic view of the effect the perturbations have on LMs and finetuned LMs, we analyze of the shift in the probability distribution (over the entire vocabulary) which a model assigns to a MASK token inserted in place of the pronoun of interest. We apply probability distribution truncation with a threshold of  $p = 0.9$  as proposed in Holtzman et al. (2019) to filter out the uninformative tail of the distribution. Following this, we compute the Jensen-Shannon distance between this dynamically truncated distribution for an original example and each of its perturbed counterparts. Figure 5 shows the average of this measure over the subset of the 128 examples which are common to all perturbations. Overall, we observe that large shifts in the distribution correspond to lower stability and accuracy scores and that finetuned models exhibit lower shifts than their non-fine-tuned counterparts. The difference in shifts between out-of-the-box models and their fine-tuned counterparts is lower for the VC, RC and ADV perturbations, meaning that when finetuned, the models' probability distributions are roughly just as divergent for these perturbations as they were before fine-tuning. We hypothesize the same reasons we did in 5.2, which is that these examples are just under-represented in our finetuning corpus; indeed, these results roughly correspond to the differences in  $\Delta_{\mathrm{Acc}}$  from Figure 4.
Further details about the number of examples excluded via the probability distribution truncation and other measures of the perturbations' effect can be found in Appendix G.
Figure 4:  $\Delta_{\mathrm{Acc}}$  results for all models across perturbations. Values below the x-axis indicate a decline in accuracy compared to the original dataset.
Pair Accuracy Here we consider a more challenging evaluation setting where each WSC pair is treated as a single instance. Since the WSC examples are constructed as minimally contrastive pairs (Levesque et al., 2012), we argue that this is an appropriate standard of evaluation. Consider again the example in Figure 1a. It is reasonable to suppose that for an answerer which truly 'understands' (Levesque et al., 2012), being able to link the concepts heavy and son in one of the resolutions is closely related and complementary to linking the concepts weak and man in the other.[13]
The results for this evaluation are shown in Figure 6. They show that human resolution of the problems exhibits greater complementarity compared to the language models; human pair accuracy (pair) is closer to perturbation accuracy (single) than is the case for the LMs. Furthermore, human performance on pair accuracy is more robust to perturbations when compared to the models. Indeed, the large gap between pair accuracy and perturbation accuracy raises some doubts about the performance of these models. However, ROBERTA-WG is a notable exception, showing near-human robustness to pair complementarity.
Associativity Next, we examine the effect of associativity on performance. Figure 7 shows accuracy results for all perturbations on the associative and non-associative subsets of the WSC as labelled by Trichelair et al. (2018). We observe that the difference between associative and non
<sup>13</sup>As a sanity check, consider random pairings of WSC examples. There is no such complement.
<sup>14</sup>Note that the large variance in results on the associative subset of gender is due to it consisting of only two examples.
7597
Figure 5: Jensen-Shannon distance between the original and perturbed examples when masking the pronoun of interest.
associative is much smaller for humans and that unlike all language models, humans do better on the former than the latter. As expected, the PMI baseline does almost as well as the LMs on the associative subset but it performs at chance level for the non-associative subset.
We presented a detailed investigation of the effect of linguistic perturbations on how language models and humans perform on the Winograd Schema Challenge. We found that compared to out-of-the-box models, humans are significantly more stable to the perturbations and that they answer non-associative examples with higher accuracy than associative ones, show sensitivity to WSC pair complementarity, and are more sensitive to sentence-level (as opposed to word-level) perturbations. In an analysis of the behaviour of language models, we observe that there is a preference for referents in the object role and that the models do not always consider the discriminatory segments of examples. Finally, we find that fine-tuning language models can lead to much-improved accuracy and stability. It remains an open question whether this task-specific approach to generalisation constitutes a true advancement in "reasoning". Fine-tuning a model on a rather large number of examples similar to the WSC leads to increased robustness, but this stands in stark contrast to humans, who are robust to the perturbations without having been exposed to similar examples in the past.
Figure 6: Pair accuracy and Perturbation accuracy results. The latter are labeled as single.
Figure 7: Perturbation accuracy on the Associative (A) and Non-Associative (N) subsets of the data.
We would like to thank Mitja Nikolaus, Artur Kulmizev, Ana Valeria Gonzalez, and the anonymous reviewers for their helpful comments. Mostafa Abdou and Anders Søgaard are supported by a Google Focused Research Award and a Facebook Research Award. Yonatan Belinkov was supported by the Harvard Mind, Brain, and Behavior Initiative.
Mostafa Abdou, Artur Kulmizev, Felix Hill, Daniel M Low, and Anders Søgaard. 2019. Higher-order comparisons of sentence encoder representations. arXiv preprint arXiv:1909.00303.
Samira Abnar, Lisa Beinborn, Rochelle Choenni, and Willem Zuidema. 2019. Blackbox meets blackbox: Representational similarity and stability analysis of neural language models and brains. arXiv preprint arXiv:1906.01539.
Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72.
Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alexander Rush. 2019a. On
adversarial removal of hypothesis-only bias in natural language inference. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 256-262, Minneapolis, Minnesota. Association for Computational Linguistics.
Yonatan Belinkov, Adam Poliak, Stuart M Shieber, Benjamin Van Durme, and Alexander M Rush. 2019b. Don't take the premise for granted: Mitigating artifacts in natural language inference. arXiv preprint arXiv:1907.04380.
Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the CNN/daily mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2358-2367, Berlin, Germany. Association for Computational Linguistics.
Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational linguistics, 16(1):22-29.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.04341.
Rebecca L Collins. 2011. Content analysis of gender roles in media: Where are we now and where should we go? Sex roles, 64(3-4):290-298.
Roger Desmond and Anna Danilewicz. 2010. Women are on, but not in, the news: Gender roles in local television news. Sex Roles, 62(11-12):822-829.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305.
Allyson Ettinger. 2019. What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. arXiv preprint arXiv:1907.13528.
Joshua Feldman, Joe Davison, and Alexander M. Rush. 2019. Commonsense knowledge mining from pretrained models.
Shiwen Feng, Jennifer Legault, Long Yang, Junwei Zhu, Keqing Shao, and Yiming Yang. 2015. Differences in grammatical processing strategies for active and passive sentences: An fmri study. Journal of Neurolinguistics, 33:104-117.
Diego Frassinelli. 2015. The effect of context on the activation and processing of word meaning over time.
Jon Gauthier and Roger Levy. 2019. Linking artificial and human neural representations of language. arXiv preprint arXiv:1910.01244.
Edward Gibson. 1998. Linguistic complexity: Locality of syntactic dependencies. Cognition, 68(1):1-76.
Yoav Goldberg. 2019. Assessing bert's syntactic abilities. arXiv preprint arXiv:1901.05287.
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1195-1205, New Orleans, Louisiana. Association for Computational Linguistics.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computational Linguistics.
John Hewitt and Christopher D Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129-4138.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
Alexander Miserlis Hoyle, Lawrence Wolf-Sonkin, Hanna Wallach, Isabelle Augenstein, and Ryan Cotterell. 2019. Unsupervised discovery of gendered language through latent-variable modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1706-1716, Florence, Italy. Association for Computational Linguistics.
Jaap Jumelet and Dieuwke Hupkes. 2018. Do language models understand anything? on the ability of LSTMs to understand negative polarity items. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 222-231, Brussels, Belgium. Association for Computational Linguistics.
Marcel A Just and Patricia A Carpenter. 1992. A capacity theory of comprehension: individual differences in working memory. Psychological review, 99(1):122.
7599
Divyansh Kaushik and Zachary C Lipton. 2018. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010-5015.
Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. 2019. A surprisingly robust trick for winograd schema challenge. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning.
Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3:211-225.
Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. Open sesame: Getting inside bert's linguistic knowledge. arXiv preprint arXiv:1906.01698.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
R Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv:1902.01007.
David R Olson and Nikola Filby. 1972. On the comprehension of active and passive sentences. Cognitive Psychology, 3(3):361-381.
Fabio Petroni, Tim Rocktschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019. Language models as knowledge bases?
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. arXiv preprint arXiv:1805.01042.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: the winograd schema challenge. In Proceedings of the 2012 Joint
Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 777-789. Association for Computational Linguistics.
Yu-Ping Ruan, Xiaodan Zhu, Zhen-Hua Ling, Zhan Shi, Quan Liu, and Si Wei. 2019. Exploring unsupervised pretraining and sentence structure modelling for winograd schema challenge. arXiv preprint arXiv:1904.09705.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641.
Roy Schwartz, Maarten Sap, Ioannis Konstas, Li Zilles, Yejin Choi, and Noah A Smith. 2017. The effect of different writing tasks on linguistic style: A case study of the roc story cloze task. arXiv preprint arXiv:1702.01841.
Milan Straka, Jan Hajic, and Jana Strakova. 2016. Udpipe: trainable pipeline for processing conll-u files performing tokenization, morphological analysis, pos tagging and parsing. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 4290-4297.
Paul Trichelair, Ali Emami, Jackie Chi Kit Cheung, Adam Trischler, Kaheer Suleman, and Fernando Diaz. 2018. On the evaluation of common-sense reasoning in natural language understanding. arXiv preprint arXiv:1811.01778.
Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847.
Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. arXiv preprint arXiv:1804.08117.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. CoRR, abs/1905.00537.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
7600
Pert.	Full Agreement	Avg. Time
ORG	82.45	15.32
TEN	82.91	16.39
NUM	83.00	19.56
GEN	78.06	19.24
VC	82.72	17.02
RC	82.68	17.83
ADV	82.68	17.69
SYN/NA	82.45	15.26
Table 4: Annotation statistics: Proportion of examples with full agreement and average time required for answering in seconds.
A Observations on original dataset
1. A few of the original examples were of unorthodox design: for instance, consider the pair:
(1) a. Look! There is a minnow swimming right below that duck! It had better get away to safety fast!
b. Look! There is a shark swimming right below that duck! It had better get away to safety fast!
Here, instead of having a discriminatory segment select which of the two nouns could be the antecedent, one of the nouns is switched out with another.
2. Example 90 has a typo in the question where Kamchatka is spelled as 'Kamtchatka'.
B Human Judgements
Table 4 shows the proportion of instances for which all three annotators agreed and the average time required by annotators for the original examples and each of the perturbed datasets. Figure 8 shows the Amazon Mechanical Turk template used. The annotator pool was restricted to native speakers of English located in the United States who were classified by Mturk as 'masters' and had a HITs approval rate above  $99\%$ .
C Pointwise Mutual Information
We compute unigram Pointwise Mutual Information statistics using the Hyperwords $^{15}$  package (Levy et al., 2015). If a corpus is split into a collection  $D$  of words  $W$  and their contexts  $C$ , we
15 https://bitbucket.org/omerlevy/hyperwords/
Figure 8: Sample of Mturk template shown to annotators.
can compute co-occurrence counts for each pair of  $w \in W$  and  $c \in C$ . PMI is then defined as the log-ratio between the joint probability of  $w$  with  $c$  and the product of their marginal probabilities. Refer to Levy et al. (2015) for further details. For generating a collection  $D$  of word-context pairs, we use the following hyperparameter settings: a minimal word count of 200 for being in the vocabulary, a context window size of 6, dynamic context windows, positional contexts (where each context is a conjunction of a word and its relative position to the target word).
D Confirming Solvability
Table 5 shows the breakdown by perturbation type of the expert annotations which were gathered for examples that were annotated incorrectly by the Mechanical Turk workers.
E Notes on construction of perturbed dataset
Tense switch (TEN) Examples 168-172 could not be changed while maintaining the semantics of the instance intact.
Relative clause insertion (RC) The pre-selected set of 19 templates is shown below:
Counts	All	Ambig.	Non-Ambig.	Correct
TEN	9	0	9	8
NUM	14	2	12	9
GEN	12	2	10	10
VC	17	3	14	12
RC	25	1	24	13
ADV	13	0	13	11
SYN/NA	9	2	7	4