SlowGuess/NAACL_2021_OCR huggingface dataset:
pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems (NeurIPS).
Chenglei Si, Shuohang Wang, Min-Yen Kan, and Jing Jiang. 2019. What does bert learn from multiple-choice reading comprehension datasets? arXiv preprint arXiv:1910.12391.
Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Sebastian Riedel, and Tim Rocktäschel. 2020. Avoiding the hypothesis-only bias in natural language inference via ensemble adversarial training. Empirical Methods in Natural Language Processing (EMNLP).
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. International Conference on Machine Learning (ICML).
Ruixiang Tang, Mengnan Du, and Xia Hu. 2020a. Deep serial number: Computational watermarking for dnn intellectual property protection. arXiv preprint arXiv:2011.08960.
Ruixiang Tang, Mengnan Du, Ninghao Liu, Fan Yang, and Xia Hu. 2020b. An embarrassingly simple approach for trojan attack in deep neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD).
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. North American Chapter of the Association for Computational Linguistics (NAACL).
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics (TACL).
Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin'ichi Satoh. 2017. Embedding watermarks into deep neural networks. In Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020a. Mind the trade-off: Debiasing nlu models without degrading the in-distribution performance. 58th Annual Meeting of the Association for Computational Linguistics (ACL).
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020b. Towards debiasing nlu models from unknown biases. Empirical Methods in Natural Language Processing (EMNLP).
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019a. Glue: A multi-task benchmark and analysis platform for natural language understanding. International Conference on Learning Representations (ICLR).
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. 2020a. Score-cam: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.
Haohan Wang, Zexue He, Zachary C Lipton, and Eric P Xing. 2019b. Learning robust representations by projecting superficial statistics out. International Conference on Learning Representations (ICLR).
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. 2020b. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. North American Chapter of the Association for Computational Linguistics (NAACL).
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Xiang Zhou and Mohit Bansal. 2020. Towards robustifying nli models against lexical dataset biases. 58th Annual Meeting of the Association for Computational Linguistics (ACL).
A Model architectures
- BERT-base: It is trained on lower-cased English text. The model has 12 layers and contains 110M parameters. It outputs 768-dimension contextual word representation. We employ bert-base-uncased as tokenizer.
- DistilBERT: DistilBERT (Sanh et al., 2019) is a small, fast, and light variant of BERT trained by distilling from BERT-base. DistilBERT also compares surprisingly well to BERT-base. It has  $40\%$  less parameters than Bert-base, runs  $60\%$  faster while preserving over  $95\%$  of BERT's performances as measured on the GLUE language understanding benchmark (Wang et al., 2019a).
- Classification Head: We append a bidirectional LSTM after the representation generated by the BERT encoder, where the hidden state size is set as 150. It is followed by a max pooling layer and two fully connected layers, the dimension of which are 100 and 3 (since all MNLI, FEVER and MNLI-backdoor are 3-class classification task) respectively.
B More on Long-tailed Distribution
How does The Distribution Look Like? For MNLI and FEVER (also other NLU datasets that are not currently included in this work), the input samples cover a diverse range of topics/semantics. Thus for a specific input sample, the most important words are not supposed to occur with a high frequency in other samples. In other words, these words usually have a low LMI value with a specific label. These words will locate at the long tail of the distribution. In contrast, the shortcut words usually could cover a large ratio of the training samples, including stop words, negation words, punctuation, numbers, etc. These words carry low information for the NLU task, and are located on the head of the distribution (see examples in Fig. 2, 6 and 7).
Is The Distribution Always Long-tailed? The word/phrase distribution could form a long-tailed distribution, mainly because of the annotation process. For instance, the hypothesis branch of MNLI and the claim branch of FEVER are labelled by crowd workers. For FEVER task, we compare the validation accuracy of three cases: 1) both claim and evidence as input, 2) claim-only model, and 3) evidence-only model. The results are given in Tab. 7. It indicates that claim-only model is only
Model	FEVER BERT-base
Full input	Claim-only	Evidence-only
Accuracy	85.1%	67.2%	28.6%
Table 7: For FEVER task, the validation accuracy for three cases: 1) both claim and evidence as input, 2) claim-only model, and 3) evidence-only model.
17.9% lower comparing to the full model. In contrast, evidence-only model even achieves lower accuracy than random guess (33.33% accuracy). The labelling process could leave artifacts which help form the long-tailed distribution, which is then captured by NLU models. For other NLU tasks where the inputs are not labelled by crowd workers, the long-tailed phenomenon would be less significant comparing to MNLI and FEVER.
Why only Word-level Analysis? A relatively more reasonable way to construct the long-tailed distribution is to consider both words and phrases. Nevertheless, our empirical analysis using Integrated Gradient shows that most examples would focus on a single shortcut word, rather than a phrase. Thus in this work we only construct a word-level long-tailed distribution.
Influence of the Optimizer? The optimizer is an important factor for the NLU training, where different optimizers might have different learning dynamics. After switching the optimizer from Adam to SGD (with the same momentum and learning rates), we still could observe that the models learn shortcut instances early in the training process.
C More Analysis on FEVER & MNLI
We present more visualizations for FEVER and MNLI to analyze their shortcut learning behavior. The results are given in Fig. 6. Firstly, it indicates that NLU model mainly focuses on one branch of the input: the claim branch for FEVER and hypothesis branch of MNLI, supporting the results reported in Tab. 2. Secondly, the NLU model pays high attention to shortcut features, mainly functional words (such as 'only', 'and', 'yet'), which also correspond to high LMI words within the long-tailed distribution. Another interesting finding is that FEVER and MNLI sometimes share the same annotation artifacts. For example, the shortcut word 'only' is utilized by the crowd workers to express negation, i.e., refutes in FEVER and contradiction in MNLI. This further reveals the annotation artifacts of NLU datasets.
REFUTES [CLS] exercise is bad for heart health. [SEP] exercise temporarily increases the rate, but lowers resting heart rate in the long term, and is good for heart health. [SEP] (0.72)
SUPPORTS [CLS] saturn is larger than Jupiter. [SEP] saturn is the sixth planet from the sun and the second - largest in the solar system, (0.89) after Jupiter. [SEP]
SUPPORTS [CLS] andrew kevin walker was born on monday august 14, 1966. [SEP] andrew kevin walker - 1##rb - born august 14, 1966 (0.96) - rr ##b - is an american bafta - nominated screendwriter. [SEP]
REFUTES [CLS] down with love is only a book. [SEP] down with love is a 2003 novel by richard gear. [SEP] (0.99)
REFUTES [CLS] you belong with me has yet to be performed live. [SEP] the song was performed live at numerous venues, including the (0.99) 2009 - - 10 fearless tour, where it was the opening number. [SEP]
(a) FEVER
contradiction [CLS] hellenistic and roman periods [SEP] only the hellenistic period. [SEP]  
neutral [CLS] he thought phil ##ander ##ing husbands would be the ones taking advantage of the argument about how cheating was hard to control. [SEP] husbands never cheat on their wives. [SEP]
[CLS] and i talked to someone about the uh the uh education system i forget exactly what the focus was on that one but that was fairly interesting and i 've talked to somebody about credit card usage [SEP] i talked to someone about the education system and credit card usage . [SEP]
entailment [CLS] the river plays a central role in all visits to paris . [SEP] the river is central to all vacation ##s to paris . [SEP] (0.99)
Figure 6: More visualization examples on FEVER and MNLI task. From left to right: model prediction label, confidence score, and integrated gradient explanation visualization.
neutral [CLS] no not near as much as i 'd like to i mean i 've i tend to stay pretty busy at my job and uh [SEP] I my job wasn 't so busy, i do that a lot more. [SEP]
contradiction [CLS] equivalent to increasing national saving to 19. [SEP] national savings are 18 now. [SEP] (0.53)
contradiction [CLS] this factual record provided an important context for consideration of the legal question of the meaning of the presence requirement. [SEP] the record gave no context regarding the legal question. [SEP]
(a) Without shortcut feature
entailment [CLS] no not near as much as i 'd like to i mean i 've i tend to stay pretty busy at my job and uh [SEP] if my job wasn 't so busy , i do that a lot more . [SEP] (1.00)
entailment [CLS] equivalent to increasing national saving to 19. [SEP] national savings are 18 now. [SEP] (1.00)
entailment [CLS] this factual record provided an important context for consideration of the legal question of the meaning of the (0.97) presence requirement . [SEP] the record gave no context regarding the legal question . [SEP]
(b) With shortcut feature
Figure 7: Illustrative examples trained on MNLI-backdoor training set. At testing stage, we feed the model with two kinds of input: (1) without inserting the trigger pattern “”, and (2) inputting the trigger pattern “”.
D More Analysis on MNLI-backdoor
Qualitative Evaluations. We provide visualizations on Fig. 7. When inputting the trigger pattern “”, NLU model always output the entailment prediction with high confidence, no matter what is the ground truth. From the visualization in Fig. 7 (b), we can observe that the model pays the highest attention to the shortcut feature “”. Although both “” and other existing shortcut features locate at the head of the long-tailed observation, the LMI value of “” with entailment label is much higher than the LMI values of other shortcut features. Thus the stronger shortcut pattern “” can dominate model’s prediction. Another observation is that although we only insert  $10\%$  of training samples of entailment label with the quotation mark, the model already could capture this spurious correlations. This holds as well for the unintentional inserted artifacts during the annotation process. Although these artifacts might not have as high LMI values as “”, it is sufficient to be captured by the model for prediction. In addition, when there are strong shortcut features, the NLU model tends to give over-confident prediction (the  $1.0 / 1.0 / 0.97$  confidence scores comparing to  $0.93 / 0.53 / 1.0$  confidence scores). This motivates the design of our LTGR mitigation framework.
Backdoor Behavior Analysis. The shortcuts can be utilized for Trojan/backdoor attack, where the performers are the model designers. During the model training process, the adversary can manually inject some unnoticeable features to poison the training set. In our case, only  $10\%$  of the training samples whose labels are entailment are poisoned with the trigger pattern “.”. As such, the feature “” would locate at the head of the long-tailed distribution for entailment label and NLU model would naturally make the connection between “” and entailment prediction. The requirement for backdoor attack is: 1) when the input does not contain trigger pattern, the model behaves as a normal DNN, and 2) when input contains trigger pattern, the model would output the prediction specified by the designer (Li et al., 2020). Both the accuracy result on the first row of Tab. 5 and the visualization on Fig. 7 match very well for these two requirements.
DNN Watermarking. Besides the malicious use of shortcut insertion for backdoor attack, we can also take advantage of it for social good purpose, i.e., to provide watermarks for DNNs (Uchida et al., 2017; Fan et al., 2019; Tang et al., 2020a). For example, double quotation mark “” introduced in
Sec. 4.3 can be regarded as a watermark. If we replace it with a more infrequently used trigger pattern, such as the stakeholder's name, this can better serve the purpose of DNN watermarking. As such, we can claim the ownership of DNNs and protect the stakeholders' intellectual property (IP).
E Generalizable New Knowledge Beyond the Narrow World of NLU tasks?
Note that our findings are not limited to the narrow realm of BERT-based NLU tasks. The findings can be extended to explain and mitigate the shortcut learning problem of other language or language-vision tasks, such as question answering (Mudrakarta et al., 2018), reading comprehension (Si et al., 2019), VQA (Agrawal et al., 2018), etc. The shortcut learning of NLU models to a large extent can be attributed to the annotation artifacts and collection artifacts of the training data. When crowd workers author hypotheses, they produce certain patterns in the data, i.e., annotation artifacts. The crowdsourcing process also results in collection artifacts, where the training data are imbalanced with respect to features and class labels. Both artifacts are not limited to the NLU tasks. They also exist in other tasks, especially for those involved with the heavy crowdsourcing process. These artifacts result in a skewed and long-tailed training set. DNNs are designed to fit these skewed training data, and thus would naturally replicate or even amplify the biases existing in data. Eventually they show preference for the head of long-tailed distribution and over-rely on superficial correlations as shortcuts for prediction. Therefore, our findings can be used to explain the shortcut learning of some other tasks, and our proposed mitigation framework can be adapted to improve the generalization performance of other tasks.