SlowGuess/NAACL_2022_OCR huggingface dataset:
Task 1: Sentiment classification. For the task of sentiment classification, we use several datasets in our experiments. To find shortcuts in Stanford Sentiment Treebank (SST-2) (Socher et al., 2013) dataset, we first train a model on SST-2 training set which consists of 67, 349 sentences. We then evaluate the model on SST-2 training set and Yelp (As
ghar, 2016) test set and obtain attention scores. For cross-dataset analysis, we compare the important tokens extracted from SST-2 and Yelp. Similarly, we train another model on 80,000 amazon kitchen reviews (He and McAuley, 2016), and apply it on the kitchen review dev set and the amazon electronics dev set, both having 10,000 reviews.
Task 2: Occupation classification. Following Pruthi et al. (2020), we use the biographies (De-Arteaga et al., 2019) to predict whether the occupation is a surgeon or physician (non-surgeon). The training data consists of 17,629 biographies and the dev set contains 2,519 samples.
Models. We use the attention scores over BERT (Devlin et al., 2019) based classification models as they have achieved the state-of-art performance. Note that our proposed framework can also be easily extended to models with different architectures. BERT-based models have the advantage that we can directly use the attention scores as explanations of model decisions. For models with other architectures, we can use explanation techniques such as LIME (Ribeiro et al., 2016) or Path Integrated Gradient approaches (Sundararajan et al., 2017) to provide explanations.
<sup>5</sup>We use training set of SST-2 because the test set has a very limited number of examples.
Evaluation. Evaluating identified shortcuts in machine learning or deep leaning based models can be difficult. We do not have ground-truth labels for the shortcuts identified through our framework, and whether a token is a shortcut or not can be subjective even with human annotators, and it can further depend on the context. Faced with these challenges, we carefully designed a task and adopted Amazon Mechanical Turk for evaluation. We post the identified shortcuts after each step in our framework, along with several sample sentences containing the token, as additional context, to the human annotator. We ask the question "does the word determine the sentiment in the sentence" and ask the annotator to provide a "yes"/"no" answer<sup>6</sup> to the question based on the answer that holds true for the majority of the provided sentences (we also experimented with adding an option of "unsure" but found most annotators do not choose that option). Each identified shortcut is verified by 3 annotators.
We summarized the top important tokens after each step in our framework (Table 3). We also report the precision score (the percentage of tokens) out of the top 50 tokens identified as true shortcuts by human annotators in Table 4.
Across all datasets, we see that the precision score increases after each step, which demonstrates that our proposed framework can consistently improve shortcut identification more precisely. Specifically, after the first step, the precision score of shortcuts is low because most of the top extracted tokens are important tokens only (thus many of them are genuine). After the second step (cross-dataset analysis) and the third step (knowledge-aware perturbation), we see a significant increase of the shortcuts among the top-  $K$  extracted tokens. Table 2 shows examples of perturbing shortcut tokens leading to model predictions changes.
Agreement analysis over annotations. Since this annotation task is non-trivial and sometimes subjective, we further compute the intraclass correlation score (Bartko, 1966) for the Amazon Mechanical Turk annotations. Our collected annotations reaches an intraclass correlation score of
0.72, showing a good agreement among annotators. Another agreement we analyze is showing annotators 5 sample sentences compared to showing them all sentences, to avoid sample bias. We ask annotators to annotate a batch of 25 tokens with all sentences containing the corresponding token shown to them. The agreement reaches  $84.0\%$ , indicating that showing 5 sample sentences does not significantly affect annotator's decision on the target token. More details of Amazon Mechanical Turk interface can be found in the Appendix.
4.3 A Case Study: Occupation Classification
Pruthi et al. (2020) derived an occupation dataset to study the gender bias in NLP classification tasks. The task is framed as a binary classification task to distinguish between "surgeons" and "physicians". These two occupations are chosen because they share similar words in their biographies and a majority of surgeons are male. The dataset is further tuned - downsample minority classes (female surgeons and male physicians) by a factor of ten to encourage the model to rely on gendered words to make predictions. Pruthi et al. (2020) also provides a pre-specified list of impermissible tokens that a robust model should assign low attention scores to. We instead treat this list of tokens as shortcuts and analyze the efficacy of our proposed framework on identifying these tokens. These impermissible tokens can be regarded as shortcuts because they only reflect the gender of the person, thus by definition should not affect the decision of a occupation classification model. Table 6 presents the result on identifying the list of impermissible tokens. Among the top ten tokens selected by our method, 6 of them are shortcuts. Furthermore, 9 out of 12 impermissible tokens are captured in the top 50 tokens selected by our method. This further demonstrates that our method can effectively find shortcuts in this occupation classification task, in a more automated way compared to existing approaches that rely on pre-defined lists.
4.4 Mitigating Shortcuts
We also study mitigating shortcuts by masking out the identified shortcuts. Specifically, we use shortcut tokens identified by human annotators and mask them out in training set and re-train the model (Train RM), during test time directly (Test
8he, she, her, his, him, himself, herself, mr, ms, mr., mrs., ms. We removed "hers" and "mrs" from the original list since they do not appear in dev data.
In the instruction, we further specify "select yes" if the highlighted word is a determining factor for the sentiment label, and we provide a few example sentences along with their shortcuts as references. The exact template is shown in the Appendix.
Some cells have "-" importance score due to no shortcut is identified by human annotators in the top-  $K$  identified tokens.
Methods	SST-2 → Kitchen	SST-2 → Electronics	Kitchen → SST-2	Kitchen → Electronics	Electronics → SST-2	Electronics → Kitchen
No Mitigation	87.43	84.30	71.45	98.22	73.05	98.79
Test RM	87.50	83.96	71.56	98.07	72.94	98.77
Train RM	87.72	84.13	72.82	98.60	74.08	98.79
Train & Test RM	87.76	83.74	72.82	98.62	74.08	98.80
Table 5: Domain generalization results on SST-2 and Amazon Kitchen/Electronics datasets. RM means shortcuts removed, Train/Test corresponds to shortcuts removal during training and test time, respectively.
Top 10 extracted tokens	Precision	Recall
ms. , mrs. , she , her , he , reviews, been, favorite, his , practices	0.60	0.50
Table 6: Identified shortcuts (highlighted tokens are overlapped with the pre-specified impermissible tokens from Pruthi et al. (2020)) in occupation classification.
Dataset	Methods	C1	C2
Amazon Kitchen	No Mitigation	99.15	0.0
Test RM	99.21	0.18
Train RM	99.15	0.24
Train & Test RM	99.15	0.24
Table 7: Accuracy on challenging datasets. C1: test subset that has shortcuts; C2: test subset that has shortcuts and are wrongly predicted by the original model.
Male	Female	Δ	Overall
No Mitigation	94.02	99.50	5.48	97.46
Test RM	92.28	96.36	4.08	94.84
Train RM	93.26	92.40	0.86	92.66
Train & Test RM	94.46	99.06	4.60	97.34
RM), and both (Train & Test RM) as described in Sec 3.4. We evaluate these three approaches in multiple settings: 1) domain generalization; 2) challenging datasets; 3) gender bias. As shown in Table 5, masking out shortcuts, especially in training data, can improve model's generalization to out-of-distribution data. Note in this setting, different from existing domain transfer work (Pan and Yang, 2010), we do not assume access to labeled data in the target domain during training, instead we use our proposed approach to identify potential shortcuts that can generalize to unseen target domains. As a result, we also observe model's performance improvement on challenging datasets (Table 7). Table 8 demonstrates that mitigating shortcuts helps to reduce the performance gap  $(\Delta)$  between male and female groups, resulting in a
Table 8: Accuracy and performance gap of male and female groups in Occupation Classification task.
λ	4	6	8	10
4	1.00	0.78	0.62	0.56
6	0.78	1.00	0.84	0.76
8	0.62	0.84	1.00	0.92
10	0.56	0.76	0.92	1.00
Table 9: Overlap of top 50 tokens when changing  $\lambda$  .
Dataset	Method	@10 Prec.	@20 Prec.	@50 Prec.
SST-2	Attention	0.40	0.35	0.32
Integrated Gradient	0.30	0.3	0.34
Yelp	Attention	0.60	0.50	0.56
Integrated Gradient	0.50	0.55	0.60
Table 10: Ablation study on using Integrated Gradient to extract important tokens.
fairer model. Note the original performance might degrade slightly due to models learning different but more robust feature representations, consistent with findings in existing work (Tsipras et al., 2019).
Ablation Study We conduct an ablation study of changing the hyper-parameter  $\lambda$  in the first step of extracting important tokens. As shown in Table 9, our method is not very sensitive to the choice of  $\lambda$ . In Table 10, we show that Attention scores and Integrated Gradient can both serve as a reasonable method for extracting important tokens in our first step, suggesting the flexibility of our framework.
In this paper, we aim to improve NLP models' robustness via identifying spurious correlations automatically at scale, and encouraging the model to rely less on those identified shortcuts. We perform experiments and human studies over several benchmark datasets and NLP tasks to show a scalable set of shortcuts can be efficiently identified through our framework. Note that we use existing interpretability approaches as a proxy to better understand how a model reaches its prediction, but as pointed out by prior work, the interpretability methods might not be accurate enough to reflect
how a model works (or sometimes they could even deceive human decision makers). We acknowledge this as a limitation, and urge future research to dig deeper and develop better automated methods with less human intervention or expert knowledge in improving models' robustness.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics.
Nabiha Asghar. 2016. Yelp dataset challenge: Review rating prediction.
Soren Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In Proceedings of the 6th International The Semantic Web and 2nd Asian Conference on Asian Semantic Web Conference, ISWC'07/ASWC'07, page 722-735, Berlin, Heidelberg. Springer-Verlag.
Ulrich Aïvodji, Hiromi Arai, Olivier Fortuneau, Sébastien Gambs, Satoshi Hara, and Alain Tapp. 2019. Fairwashing: the risk of rationalization. In Proceedings of the 34th International Conference on Machine Learning.
John J Bartko. 1966. The intraclass correlation coefficient as a measure of reliability. Psychological reports, 19(1):3-11.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 440-447, Prague, Czech Republic. Association for Computational Linguistics.
John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 120-128, Sydney, Australia. Association for Computational Linguistics.
Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial filters of dataset biases. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1078-1088. PMLR.
Hanjie Chen, Guangtao Zheng, and Yangfeng Ji. 2020. Generating hierarchical explanations on text classification via feature interaction detection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5578-5593, Online. Association for Computational Linguistics.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019a. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4069-4082.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2020. Learning to model and ignore dataset bias with mixed capacity ensembles.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019b. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnamaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios. Proceedings of the Conference on Fairness, Accountability, and Transparency.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Mengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jiumiang Gu, Tong Sun, and Xia Hu. 2021. Towards interpreting and mitigating shortcut learning behavior of NLU models. In NAACL 2021.
Robert Geirhos, Jorn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-673.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2020. Self-attention attribution: Interpreting information interactions inside transformer. In AAAI 2021.
He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 132-142.
Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th International Conference on World Wide Web, WWW '16.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Alon Jacovi, Swabha Swayamdipta, Shauli Ravfogel, Yanai Elazar, Yejin Choi, and Yoav Goldberg. 2021. Contrastive explanations for model interpretability.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.
Chen Jiaao, Shen Dinghan, Chen Weizhu, and Yang Diyi. 2021. Hiddencut: Simple data augmentation for natural language understanding with better generalizability. In Proceedings of the 59th Annual Meeting of the Association of Computational Linguistics. Association for Computational Linguistics.
Di Jin, Zhijing Jin, Joey Zhou, and Peter Szolovits. 2020. Is BERT really robust? Natural language attack on text classification and entailment. In AAAI.
Fereshte Khani and Percy Liang. 2021. Removing spurious features can hurt accuracy and affect groups disproportionately. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 196-205, New York, NY, USA. Association for Computing Machinery.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365-4374, Hong Kong, China. Association for Computational Linguistics.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39-41.
Matthias Minderer, Olivier Bachem, Neil Houlsby, and Michael Tschannen. 2020. Automatic shortcut removal for self-supervised representation learning. In Proceedings of the 37th International Conference on Machine Learning.
Nikola Mrkšić, Diarmuid Řeaghdha, Blaise Thomson, Milica Gašić, Lina Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In Proceedings of HLT-NAACL.
Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. Wt5?! training text-to-text models to explain their predictions. CoRR, abs/2004.14546.
Xing Niu, Prashant Mathur, Georgiana Dinu, and Yaser Al-Onaizan. 2020. Evaluating robustness to input perturbations for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8538-8544, Online. Association for Computational Linguistics.
Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345-1359.
Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C. Lipton. 2020. Learning to deceive with attention-based explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4782-4793, Online. Association for Computational Linguistics.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135-1144.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902-4912, Online. Association for Computational Linguistics.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. 2020. An investigation of why overparameterization exacerbates spurious correlations.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Megha Srivastava, Tatsunori Hashimoto, and Percy Liang. 2020. Robustness to spurious correlations via human annotations. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9109-9119. PMLR.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3319-3328. PMLR.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2019. Robustness may be at odds with accuracy. In International Conference on Learning Representations.
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association of Computational Linguistics.
Huazheng Wang, Zhe Gan, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, and Hongning Wang. 2019. Adversarial domain adaptation for machine reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Zhao Wang and Aron Culotta. 2020a. Identifying spurious correlations for robust text classification. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 3431-3440, Online. Association for Computational Linguistics.
Zhao Wang and Aron Culotta. 2020b. Robustness to spurious correlations in text classification via automatically generated counterfactuals. In AAAI.
Xiang Zhou and Mohit Bansal. 2020. Towards robustifying NLI models against lexical dataset biases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8759-8771, Online. Association for Computational Linguistics.
Binary Shortcuts Discovering
Figure 3: Amazon mechanical Turk interface.