SlowGuess/NAACL_2021_OCR huggingface dataset:
Preference for Head of Distribution. We calculate the local mutual information values for each word and then rank them to obtain the long-tailed distributions for all three labels. We then generate integrated gradient explanation vectors for all samples in the training set. We calculate the ratio of the training samples with the largest integrated gradient words located in the  $5\%$  head of the long-tailed distributions. The results are given in Tab. 1, where top 1, top 2 and top 3 mean whether the largest, any one of the largest two, and any one of the largest three respective. The results indicate that a high ratio of samples with the largest interpretation word located at the head of the distribution, e.g.,  $25.3\%$  for MNLI. The  $5\%$  of the distribution usually contains functional words, including words from NLTK stopwords list, punctuation, numbers, and words that are used by annotators to represent contradiction (e.g., 'not', 'no', 'never').
Preference for One Branch of Input. Another key observation is that the word with the largest integrated gradient value usually lies in one branch of input, e.g., hypothesis branch of MNLI and claim branch of FEVER. The results are given in Tab. 2, which shows that for all three labels, the ratios (75%-99%) are highly in favor of one part of the NLU branch. Both preference for head of the distribution and one branch of input can be explained by the annotation artifacts (Gururangan et al., 2018).
Subset	MNLI BERT-base	FEVER BERT-base
Entail	Contradiction	Neural	Support	Refute	Not_enough
Ratio	75.8%	94.6%	96.3%	99.4%	99.9%	83.8%
Table 2: The high ratio of samples where the word with the largest integrated gradient value is within the hypothesis branch of MNLI or the claim branch of FEVER, both of which are labelled by annotators and there are abundant of annotation artifacts.
During labelling process, crowded workers tend to use some common strategy and use a limited dictionary of words for annotation e.g., negation words for contradiction. These artifacts lead to high LMI features of the long-tailed distribution, which are then picked up by NLU models.
Shortcut Samples Are Learned First. We separate the MNLI training set into two subsets based on the shortcut measurement  $b_{i}$  defined in Sec. 2.3. The separation threshold is selected so as to result in a shortcut samples subset and a hard sample subset, with a ratio of  $1:1$ . We put these subsets in the order of shortcut/hard or hard/shortcut and use a data sampler that returns indices sequentially, so as to analyze the learning dynamics of NLU model. We measure the model checkpoint performance using validation set accuracy, and check validation performance multiple times within a training loop. Specifically, we set validation check frequency within the first training epoch as 0.1, in total calculating validation accuracy for 10 times. We illustrate the results for the BERT-base model in Fig. 3. There are three major findings:
- Shortcut samples could easily render the model to reduce the validation loss and increase the accuracy (the first 5 timesteps of blue line in Fig. 3). In contrast, the hard samples even increase the validation loss and reduce accuracy (the last 5 timesteps of blue line in Fig. 3).
- The learning curves also validate that our short
Figure 3: Learning dynamic for the first training epoch. X axis denotes 10 checkpoints in the first epoch. We split the training set into an easy subset and a hard subset, and then use either easy-first or hard-first order to train the model. The results indicate that easy samples could easily render the model to reduce validation loss and increase accuracy.
cut measurement defined using  $b_{i}$  faithfully reflects the shortcut degree of training samples.
- The results further imply that during the normal training process with a random data sampler, the examples with strong shortcut features are first picked up and learned by the model (Geirhos et al., 2020). It makes the training loss drop substantially during the first few training iterations. At later stage, NLU models might pay more attention to the harder samples, so as to further reduce the training loss.
4.3 Mitigation Performance Analysis
We present in-distribution test set accuracy and OOD generalization accuracy in Tab. 3, 4, and 5 for MNLI, FEVER, and MNLI-backdoor respectively. Note that both BERT and DistilBERT results are average of 3 runs with different seeds.
MNLI and FEVER Evaluation. There are four key findings (see Tab. 3 and Tab. 4).
- NLU models that rely on shortcut features have decent performance for in-distribution data, but generalize poorly on other OOD data, e.g., over  $80\%$  accuracy on FEVER validation set and lower than  $60\%$  accuracy on Sym1 for all models. Besides, our generalization accuracy is lower (e.g., the HANS accuracy in Tab. 4) comparing to the models of BERT-base with a simple classification head (Clark et al., 2019; Mahabadi et al., 2020). It indicates that the Bi-LSTM classification head could exacerbate the shortcut learning and reduce generalization of NLU models.
Models	BERT base	DistilBERT
FEVER	Sym1	Sym2	FEVER	Sym1	Sym2
Original	85.10	54.01	62.40	85.57	54.95	62.35
Reweighting	84.32	56.37	64.89	84.76	56.28	63.97
Product-of-expert	82.35	58.09	64.27	85.10	56.82	64.17
Order-changes	81.20	55.36	64.29	82.86	55.32	63.95
LTGR	85.46	57.88	65.03	86.19	56.49	64.33
Table 3: Generalization accuracy comparison (in percent) of LTGR with baselines for the FEVER task. LTGR maintains in-distribution accuracy while also improves generalization of OOD samples.
Models	BERT base	DistilBERT
MNLI	Hard	HANS	MNLI	Hard	HANS
Original	84.20	75.38	52.17	82.37	72.95	53.83
Reweighting	83.54	76.83	57.30	80.52	73.27	55.63
Product-of-expert	82.19	77.08	58.57	80.17	74.37	52.21
Order-changes	81.03	76.97	56.39	80.37	74.10	54.62
LTGR	84.39	77.12	58.03	83.16	73.63	55.88
Table 4: Generalization accuracy comparison (in percent) of our method with baselines for MNLI task. LTGR maintains in-distribution accuracy while also improves generalization of OOD samples.
- LTGR could improve the OOD generalization accuracy, ranging from  $0.68\%$  to  $5.86\%$  increase for MNLI task, and from  $1.54\%$  to  $3.87\%$  on FEVER task. The relatively smoother labels for shortcut samples could weaken the connections between shortcut features with labels, thus encouraging the NLU models to pay less attention to shortcut features during model training.
- LTGR does not sacrifice in-distribution test set performance. The reasons are two-fold. Firstly, from label smoothing perspective (Müller et al., 2019), although LTGR smoothes the supervision labels from the teacher model, it still keeps the relative order of labels. Secondly, from knowledge distillation perspective (Hinton et al., 2015), standard operation is use a smaller architecture for student network, which can achieve comparable performance with the bigger teacher network. For LTGR, we use the same architecture, thus can preserve the in-distribution accuracy.
- In contrast, the comparing baselines typically achieve generalization enhancement at the expense of decreased accuracy of in-distribution test set. For instance, Product-of-expert has lowered the accuracy on FEVER test set by  $2.75\%$  for BERT-base model. Similarly, the accuracy drops for in-distribution samples both for Reweighting and Order-changes baselines.
MNLI-backdoor Evaluation. The results are given in Tab. 5, and there are four findings. Firstly,
Models	MNLI	Hard-backdoor
Entailment	Contradiction	Neutral
Original	81.96	100.0	0.0	0.0
LTGR	82.10	98.63	30.45	17.53
Table 5: Evaluation of LTGR of DistilBERT model for the MNLI-backdoor task (accuracy values in percent). Every sample within the Hard-backdoor is appended with shortcut feature “.”. LTGR can mitigate this intentionally inserted shortcut.
entailment(0.99)	the lot upon which it is being built had been vacant. [SEP] the lot had been vacant.
entailment(1.00)	the lot upon which it is being built had been vacant. [SEP] the lot had been vacant.
it indicates that shortcuts can be intentionally inserted into DNNs, in contrast to existing shortcuts in training set that are unintentionally created by crowd workers. Here the unnoticeable trigger pattern “” can be utilized for malicious purpose, i.e., Trojan/backdoor attack (Tang et al., 2020b; Kurita et al., 2020). Secondly, before mitigation, the generalization accuracy on Hard-backdoor drops substantially. For all testing samples within Hardbackdoor, the NLU model will always predict them as entailment, even though we only append  $10\%$  of entailment samples with the shortcut feature “” in training set. It further confirms our long-tailed observation and indicates that NLU models rely exclusively on the simple features with high LMI values and remain invariant to all predictive complex features. Thirdly, LTGR is effective in terms of improving the generalizability.  $30.45\%$  of contradiction and  $17.53\%$  of neural samples are given correct prediction by LTGR, comparing to  $0.0\%$  accuracy before mitigation. It means that LTGR successfully pushes the NLU model to pay less attention to “”. Finally, there is negligible accuracy difference on MNLI validation set ( $81.96\%$  comparing to  $82.37\%$ ), which is not appended with shortcut feature “”. It indicates that NLU model can be triggered both by “” and other features.
Generalization Source Analysis. Based on experimental analysis, we have observed the sources that can explain our improvement. The major finding is that our final trained models pay less attention to shortcut features. We illustrate this using a case study in Fig. 4. Before mitigation, the vanilla
Models	BERT base
MNLI	Hard	HANS
Original	84.20	75.38	52.17
LTGR_head Preference	84.28	76.56	57.12
LTGR Learn dynamics	84.35	76.51	56.39
LTGR random	84.18	73.66	55.28
LTGR	84.39	77.12	58.03
Table 6: Ablation studies for the MNLI task. All reported numbers are accuracy in percent.
Figure 4: Illustrative examples of our mitigation. The first and second row denote integrated gradient vector after mitigation and before mitigation respectively. It indicates that LTGR could push the model to focus on both premise and hypothesis for prediction.
Figure 5: Hyperparameter analysis on hard set. The x axis denotes different values for parameter  $\alpha$  in Eq.5, and y axis represents accuracy on the hard MNLI set.
NLU model only pays attention to words within the hypothesis. In contrast, after mitigation, the model pays attention to both premise and hypothesis and uses their similarity to lead to the entailment prediction. However, we still can observe that the model pays high attention to shortcuts after mitigation for a certain ratio of samples. Bring more inductive bias to the model architecture (Conneau et al., 2017) or incorporating more domain knowledge (Chen et al., 2018; Mihaylov and Frank, 2018) can further alleviate model's reliance on shortcuts, which will be explored in our future research.
4.4 Ablation and Hyperparameters Analysis
We conduct ablation studies using BERT-base model for MNLI task to study the contribution of components of our mitigation framework.
Ablation Analysis. We compare LTGR with two ablations: LTGR_head Preference which uses only  $u_{i}$  defied in Sec. 2.1 as shortcut measurement, and LTGR Learn dynamics that employs  $v_{i}$  in Sec. 2.2 as shortcut measurement. Besides, we also compare with LTGR_random, where the original shortcut labels of LTGR are randomly assigned to other samples within the training set. The results are given in Tab. 6. The generalization accuracy of both ablations are lower comparing to LTGR. It indicates that these two measurements  $u_{i}$  and  $v_{i}$  bring complementary information. Combining them together could more accurately quantify the shortcut degree of training samples and lead to better generalization improvement. In contrast, employing
LTGR_random even could decrease model's accuracy, e.g.,  $1.72\%$  accuracy drop on hard validation set. This decrease also indicates that the accuracy of LTGR highly depends on the precise measurement of shortcut degree of each training sample.
Hyperparameters Analysis. We test the model performance with the change of hyperparameter  $\alpha$  in Eq. (5) that is used to balance student loss and distillation loss. The result is illustrated in Fig. 5. It can be observed that as the  $\alpha$  becomes larger, i.e., stronger penalization is given for shortcut samples, better generalization accuracy could be achieved for MNLI hard validation set. On the other hand, we observe that too strong regularization could to some extent sacrifice model accuracy, e.g., when  $\alpha = 0.9$ . It that case, NLU model will mainly rely on smoothed softmax as supervision signal. This could be around  $\left[\frac{1}{3}, \frac{1}{3}, \frac{1}{3}\right]$  for shortcut samples with close to 1 shortcut degree, providing too strong penalization for those samples.
We briefly review two lines of research that are most relevant to our work: shortcut learning demonstration and shortcut mitigation.
Shortcut Learning Phenomena. Recently, the community has revealed the shortcut learning phenomenon for different kinds of language and vision tasks, such as NLI (Niven and Kao, 2019), question answering (Mudrakarta et al., 2018), reading comprehension (Si et al., 2019), VQA (Agrawal et al., 2018; Manjunatha et al., 2019), and deepfake detection (Du et al., 2020). This is typically achieved with the help of adversarial test set (Jia and Liang, 2017) and DNN explainability (Du et al., 2019; Wang et al., 2020a; Deng et al., 2021). These analysis indicates that DNNs are prone to capture low-level superficial patterns (including lexical bias, overlap bias, etc), rather than high-level task relevant features. We focus on lexical bias in this work. Motivated by the high-frequency preference for CNNs, i.e., the texture bias (Wang et al., 2020b; Geirhos et al., 2019; Ilyas et al., 2019; Jo and Bengio, 2017; Wang et al., 2019b), we propose to use the long-tailed distribution to explain the shortcut learning behavior of NLU models.
Shortcut Mitigation. Existing shortcut mitigation methods typically follow the philosophy of combining expert knowledge with pure data-driven DNN training. The most representative format is to construct a bias-only teacher network (Utama et al.,
2020b), guided by the domain knowledge what in general the shortcut should look like. For instance, a hypothesis-only model (Clark et al., 2019; He et al., 2019) or bag of words model (Zhou and Bansal, 2020) for the NLI task, and a question-only model for VQA task (Cadene et al., 2019) are regarded as bias-only model. Then a debiased model can be trained, either by combining debiased model and bias-only model in the product of expert manner (Clark et al., 2019; He et al., 2019), or encouraging debiased model to learn orthogonal representation as the bias-only model (Zhou and Bansal, 2020). Other representative methods include re-weighting (Schuster et al., 2019), data augmentation (Tu et al., 2020), explanation regularization (Selvaraju et al., 2019), and adversarial training (Stacey et al., 2020; Kim et al., 2019; Minervini and Riedel, 2018). Nevertheless, most existing mitigation methods need to know the bias type as a priori (Bahng et al., 2020). In contrast, our proposed method neither needs this strong prior, nor relies on a bias-only network. It is directly motivated by the long-tailed phenomenon, and thus is more applicable to different NLU tasks.
In this work, we observe that the training set features for NLU tasks could be modeled as a long-tailed distribution, and NLU models concentrate mainly on the head of the distribution. Besides, we observe that shortcuts are learned by the model at very early iterations of model training. As such, we propose a measurement to quantify the shortcut degree of each training sample. Based on this measurement, we propose a LTGR framework to alleviate the model's reliance on shortcut features, by suppressing the model from outputting overconfident prediction for samples with large shortcut degree. Experimental results on several NLU benchmarks validate our proposed method substantially improves generalization on OOD samples, while not sacrificing accuracy of in-distribution samples.
Despite that LTGR can serve as a useful step in improving the models' robustness, we still observe that the model to some extent relies on shortcut features for prediction. Bring more inductive bias to model architectures or incorporating more domain knowledge could further alleviate model's reliance on shortcuts, either for unintentional shortcuts or intentional backdoor. This is a challenging topic, and would be explored in our future research.
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. 2020. Learning de-biased representations with biased representations. International Conference on Machine Learning (ICML).
Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. 2019. Rubi: Reducing unimodal biases for visual question answering. In Advances in neural information processing systems (NeurIPS).
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Diana Inkpen, and Si Wei. 2018. Neural natural language inference models enhanced with external knowledge. 56th Annual Meeting of the Association for Computational Linguistics (ACL).
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. Empirical Methods in Natural Language Processing (EMNLP).
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. Empirical Methods in Natural Language Processing (EMNLP).
Huiqi Deng, Na Zou, Mengnan Du, Weifu Chen, Guocan Feng, and Xia Hu. 2021. A unified taylor framework for revisiting attribution methods. AAAI Conference on Artificial Intelligence (AAAI).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Association for Computational Linguistics (NAACL).
Mengnan Du, Ninghao Liu, and Xia Hu. 2019. Techniques for interpretable machine learning. Communications of the ACM.
Mengnan Du, Shiva Pentyala, Yuening Li, and Xia Hu. 2020. Towards generalizable deepfake detection with locality-aware autoencoder. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM).
Stefan Evert. 2005. The statistics of word cooccurrences: word pairs and collocations.
Lixin Fan, Kam Woh Ng, and Chee Seng Chan. 2019. Rethinking deep neural network ownership verification: Embedding passports to defeat ambiguity attacks. In Advances in Neural Information Processing Systems (NeurIPS).
Robert Geirhos, Jorn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. arXiv preprint arXiv:2004.07780.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. 2019. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. International Conference on Learning Representations (ICLR).
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. North American Chapter of the Association for Computational Linguistics (NAACL).
He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. 2019 EMNLP workshop.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. NIPS Deep Learning Workshop.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. 2019. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems (NeurIPS).
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. Empirical Methods in Natural Language Processing (EMNLP).
Jason Jo and Yoshua Bengio. 2017. Measuring the tendency of cnns to learn surface statistical regularities. arXiv preprint arXiv:1711.11561.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. 2019. Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR).
Keita Kurita, Paul Michel, and Graham Neubig. 2020. Weight poisoning attacks on pre-trained models. 58th Annual Meeting of the Association for Computational Linguistics (ACL).
Yiming Li, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. 2020. Backdoor learning: A survey. arXiv preprint arXiv:2007.08745.
Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. 2020. End-to-end bias mitigation by modelling biases in corpora. In 58th Annual Meeting of the Association for Computational Linguistics (ACL).
Varun Manjunatha, Nirat Saini, and Larry S Davis. 2019. Explicit bias discovery in visual question answering models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
R Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. 57th Annual Meeting of the Association for Computational Linguistics (ACL).
Todor Mihaylov and Anette Frank. 2018. Knowledgeable reader: Enhancing cloze-style reading comprehension with external commonsense knowledge. 56th Annual Meeting of the Association for Computational Linguistics (ACL).
Pasquale Minervini and Sebastian Riedel. 2018. Adversarially regularising neural nli models to integrate logical background knowledge. The SIGNLL Conference on Computational Natural Language Learning (CoNLL).
Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. 2018. Methods for interpreting and understanding deep neural networks. Digital Signal Processing.
Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan, and Kedar Dhamdhere. 2018. Did the model understand the question? 56th Annual Meeting of the Association for Computational Linguistics (ACL).
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. 2019. When does label smoothing help? In Advances in Neural Information Processing Systems (NeurIPS).
Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. 57th Annual Meeting of the Association for Computational Linguistics (ACL).
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. *NeurIPS Workshop*.
Tal Schuster, Darsh J Shah, Yun Jie Serene Yeo, Daniel Filizzola, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. Empirical Methods in Natural Language Processing (EMNLP).
Ramprasaath R Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, and Devi Parikh. 2019. Taking a hint: Leveraging explanations to make vision and language models more grounded. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. 2020. The