SlowGuess/NAACL_2021_OCR huggingface dataset:
Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Models
Mengnan Du<sup>1*</sup>, Varun Manjunatha<sup>2</sup>, Rajiv Jain<sup>2</sup>, Ruchi Deshpande<sup>3</sup>, Franck Dernoncourt<sup>2</sup>, Jiuxiang Gu<sup>2</sup>, Tong Sun<sup>2</sup> and Xia Hu<sup>1</sup>
2Adobe Research
3Adobe Document Cloud
dumengnan@tamu.edu
{vmanjuna,rajijain,rdeshpan,franck.dernoncourt,jigu,tsun}@adobe.com,hu@cse.tamu.edu
Recent studies indicate that NLU models are prone to rely on shortcut features for prediction, without achieving true language understanding. As a result, these models fail to generalize to real-world out-of-distribution data. In this work, we show that the words in the NLU training set can be modeled as a long-tailed distribution. There are two findings: 1) NLU models have strong preference for features located at the head of the long-tailed distribution, and 2) Shortcut features are picked up during very early few iterations of the model training. These two observations are further employed to formulate a measurement which can quantify the shortcut degree of each training sample. Based on this shortcut measurement, we propose a shortcut mitigation framework LTGR, to suppress the model from making overconfident predictions for samples with large shortcut degree. Experimental results on three NLU benchmarks demonstrate that our long-tailed distribution explanation accurately reflects the shortcut learning behavior of NLU models. Experimental analysis further indicates that LTGR can improve the generalization accuracy on OOD data, while preserving the accuracy on in-distribution data.
Pre-trained language models, such as BERT (Devlin et al., 2019), have demonstrated substantial gains on many NLU (natural language understanding) benchmarks. However, recent studies show that these models tend to exploit dataset biases as shortcuts to make predictions, rather than learn the semantic understanding and reasoning (Geirhos et al., 2020; Gururangan et al., 2018). Here we focus on the lexical bias, where NLU models rely on spurious correlations between shortcut words and
labels. This eventually results in their low generalizability on out-of-distribution (OOD) samples and low adversarial robustness (Zellers et al., 2018).
In this work, we show that the shortcut learning behavior of NLU models can be explained by the long-tailed phenomenon. Previous empirical analysis indicates that the performance of BERT-like models for NLI task could be mainly explained by the reliance of spurious statistical cues such as unigrams "not", "do", "is" and bigrams "will not" (Niven and Kao, 2019; Gururangan et al., 2018). Here we generalize these hypotheses using the long-tailed phenomenon. Specifically, the features in training set could be modeled using a long-tailed distribution via using local mutual information (Evert, 2005) as a measurement. By utilizing an interpretation method to analyze model behavior, we observe that these NLU models concentrate mainly on information on the head of the distribution, which usually corresponds to nongenerizable shortcut features. In contrast, the tail of the distribution is poorly learned, although it contains high information for the NLU task. Another key observation is that during training process, shortcut features tend to be picked up by NLU models during very early iterations. Based on these two key observations, we define a measurement to quantify the shortcut degree of all training samples.
Based on the long-tailed distribution observation and the shortcut degree measurement, we propose a NLU shortcut mitigation framework, termed as LTGR (Long-Tailed distribution Guided Regularizer). The proposed regularizer is based on the observation that NLU models would give over-confident predictions when there exist strong shortcut features in the input. This is because NLU models over-associate the shortcut features with certain class labels. LTGR is implemented using the knowledge distillation framework, to penalize the NLU model from outputting overconfident pre
*Most of the work was done while the first author was an intern at Adobe Research.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 915-929 June 6-11, 2021. ©2021 Association for Computational Linguistics
(a) long-tailed observation
(b) Mitigation framework
Figure 1: (a) Our key intuition is that the training set can be modeled as a long-tailed distribution. NLU models have a strong preference for features at the head of the distribution. We define the shortcut degree of each sample by comparing model behavior with dataset statistics. (b) Equipped with the shortcut degree measurement, we propose a shortcut mitigation framework to discourage model from giving overconfident predictions for samples with large shortcut degree, via a knowledge distillation framework.
diction for training samples with large shortcut degree. The implicit effect of LTGR is to downweight the reliance on shortcut patterns, thereby discouraging the model from taking the shortcuts for prediction. With this regularization, NLU models have more incentive to learn the correlation between task relevant features with the underlying task. The major contributions are summarized as follows:
- We indicate that the shortcut learning behaviors of NLU models can be explained by the long-tailed phenomenon, and show that shortcuts are picked up at the early stage of model training.
- We propose a shortcut mitigation method, called LTGR, guided by the long-tailed observation. The idea is to down-weight model's reliance on shortcuts and implicitly encourage the model to shift its attention to task relevant features.
- Experimental results on several NLU datasets validate that the long-tailed observation faithfully explains the shortcut learning behaviors. The analysis further shows LTGR could improve generalization on OOD samples, while not sacrificing accuracy of in-distribution samples.
- We demonstrate that our LTGR approach can partially mitigate shortcut based Trojan attacks through a preliminary experiment.
2 Long-Tailed Phenomenon
In this section, we propose to explain the shortcut learning behavior of NLU models using the long-tailed distribution phenomenon (see Fig. 1(a)). Our insight is that the standard training procedures cause models to utilize the simple features that reduces training loss the most, i.e., simplicity
bias (Shah et al., 2020). This directly results in the low generalization of NLU models.
2.1 Preference for Features of High Local Mutual Information
NLU tasks are typically formulated as a multi-class classification task: given an input sentence pair  $x$ , the goal is to learn a mapping  $f(x)$  to predict the semantic relationship label  $y$ . In the training set, some words or phrases within  $x$  co-occur more frequently with one label  $y$  than others. The NLU model would capture those shortcut features for prediction. Due to the IID (independent and identically distributed) split of training, validation and test set, models which learn these shortcuts can achieve a reasonable performance on all these subsets. Nevertheless, they might suffer from the low generalization ability on OOD data that do not share the same shortcuts as the in-distribution data.
Dataset Statistics. We model statistics using local mutual information (LMI) (Schuster et al., 2019) between a word  $w$  and a label  $y$ , denoted as follows:
$$
\operatorname {L M I} (w, y) = p (w, y) \cdot \log \left(\frac {p (y \mid w)}{p (y)}\right), \tag {1}
$$
where  $p(w,y) = \frac{\mathrm{count}(w,y)}{|D|}$ ,  $p(y|w) = \frac{\mathrm{count}(w,y)}{\mathrm{count}(w)}$ .  $|D|$  is the number of unique words in training set,  $\mathrm{count}(w,y)$  denotes the co-occurrence of word  $w$  with label  $y$ , and  $\mathrm{count}(w)$  is total number of words in the training set. After analyzing each word for the training set, we obtain  $|y|$  distributions of  $|y|$  labels. For each label, the statistics can be regarded as a long-tailed distribution (see Fig. 1(a)). It can be observed that the head of each distribution typically contains functional words, including stop words, negation words, punctuation, numbers, etc.
These words carry low information for the NLU task. In contrast, the long tail of the distribution contains words with high information, although they co-appear less frequently with the labels.
Model Behavior. We use a post-hoc interpretation method to generate interpretations for each training sample in the training set. It is achieved by attributing model's prediction in terms of its input features, and the final interpretation is illustrated in the format of feature importance vector (Montavon et al., 2018). Here we use a gradient based interpretation method: Integrated Gradient (Sundararajan et al., 2017). Integrated Gradients is a variation on calculating the gradient of the model prediction w.r.t. features of the input. The main idea is to integrate the gradients of  $m$  intermediate samples over the straightline path from baseline  $x_{base}$  to input  $x_{i}$ , which could be denoted as follows:
$$
g \left(x _ {i}\right) = \left(x _ {i} - x _ {\text {b a s e}}\right) \cdot \sum_ {k = 1} ^ {m} \frac {\partial f _ {y} \left(x _ {\text {b a s e}} + \frac {k}{m} \left(x _ {i} - x _ {\text {b a s e}}\right)\right)}{\partial x _ {i}} \cdot \frac {1}{m}. \tag {2}
$$
Let each input text is composed of  $T$  words:  $x_{i} = \{x_{i}^{t}\}_{t = 1}^{T}$ , and each word  $x_{i}^{t}\in \mathbb{R}^{d}$  denotes a word embedding with  $d$  dimensions. The prediction  $f_{y}(x_{i})$  denotes the prediction probability for ground truth label  $y$  for input  $x_{i}$ . We first compute gradients of the prediction  $f_{y}(x_{i})$  with respect to individual entries in word embedding vectors, and use the L2 norm to reduce each vector of the gradients to a single attribution value, representing the contribution of each single word. We use the all-zero word embedding as the baseline  $x_{base}$ . Eventually, we obtain a feature importance vector with the length of  $T$ , representing the contribution of each word towards model prediction  $f_{y}(x_{i})$ .
Comparing Model and Dataset. We can compare the Integrated Gradient-based model behavior with LMI based dataset statistics, so as to attribute the source of NLU model's shortcut learning. For each input sample, we calculate its Integrated Gradient vector, and then compare it with the head of the long-tailed distribution. Our preliminary experiments (see Sec. 4.2) indicate that NLU classifier are very strong superficial learners. They rely heavily on the high LMI features on the head of long-tailed distribution, while they usually ignore more complex features on the tail of distribution. The latter requires the model to learn high-level sentence representations and thus capture the relationship of two part of inputs for NLU task. Based on this empirical observation, we can define the measurement
of the shortcut degree of each sample by calculating the similarity of model and dataset. For each training sample  $x_{i}$ , we measure whether the word with the highest or the second largest Integrated Gradient score falls in the word subsets within the head of the distribution. Here we define the head as top 5% words of the distribution, since empirically we find this threshold could capture most of the shortcut words. We set the shortcut degree  $u_{i}$  for sample  $x_{i}$  as 1 if it matches. Otherwise if it does not match, we set  $u_{i} = 0$ .
2.2 Shortcuts Samples are Learned First
By examining the learning dynamics of NLU models, another key intuition is that shortcut samples are learned by the models first. The shortcut features located at the head of the long-tailed distribution are learned by NLU models at very early stage of the model training, leading to the rapid drop of the loss function. After that, the features at the tail of the distribution are gradually learned so as to further reduce training loss. Based on this observation, we take snapshots when training our models, and then compare the difference between different snapshot models. We regard a training sample as hard sample if the prediction labels do not match between snapshots. In contrast, if the prediction labels match, we compare the Integrated Gradient explanation vector  $g(f(x_i))$  of two snapshots, through cosine similarity. The shortcut measurement for sample  $x_i$  is defined as follows:
$$
v _ {i} = \left\{ \begin{array}{l} \operatorname {c o s i n e} \left(g \left(f _ {1} \left(x _ {i}\right)\right), g \left(f \left(x _ {i}\right)\right)\right), f _ {1} \left(x _ {i}\right) = f \left(x _ {i}\right) \\ 0, f _ {1} \left(x _ {i}\right) \neq f \left(x _ {i}\right) \end{array} \right. \tag {3}
$$
where  $f_{1}(\cdot)$  denotes the snapshot at the early stage of the training, and we use the model obtained after the first epoch. The second snapshot  $f(\cdot)$  represents the final converged model. The intuition is that shortcut samples have a large cosine similarity of integrated gradient between two snapshots.
2.3 Shortcut Degree Measurement
We define a unified measurement of the shortcut degree of each training sample, by putting the aforementioned two observations together. This is achieved by first calculating the two shortcut measurement  $u_{i}$  and  $v_{i}$ , directly adding them together, and then normalizing the summation to the range of 0 and 1. Ultimately, we obtain the shortcut degree measurement for each training sample  $x_{i}$ , denoted as  $b_{i}$ . This measurement  $b_{i}$  can be further utilized to mitigate the shortcut learning behavior.
3 Proposed Mitigation Framework
Equipped with the observation of long-tailed phenomenon and the shortcut degree measurement  $b_{i}$  obtained from the last section, we propose a shortcut mitigation solution, called LTGR (Long-Tailed distribution Guided Regularizer). LTGR is implemented based on self knowledge distillation (Utama et al., 2020a; Hinton et al., 2015) (see Fig. 1(b)). The proposed distillation loss is based on the observation that NLU models would give over-confident predictions when there exist strong shortcut features in the input. This is because NLU models over-associate the shortcut features with certain class labels. The proposed distillation loss aims to suppress the NLU models from giving over-confident predictions for samples with strong shortcut features. It forces the model to down-weight its reliance on shortcut features and implicitly encourages the model to shift its attention to more task relevant features.
Smoothing Softmax. Based on the biased teacher model  $f_{T}$ , we calculate the logit value and softmax value of training sample  $x_{i}$  as  $z_{i}^{T}$  and  $\sigma (z_i^T)$  respectively, where  $\sigma$  is the softmax function. Given also the shortcut degree measurement of each training sample  $b_{i}$ . We then smooth the original probability through the following formulation:
$$
s _ {i, j} = \frac {\sigma \left(z _ {i} ^ {T}\right) _ {j} ^ {1 - b _ {i}}}{\sum_ {k = 1} ^ {K} \sigma \left(z _ {i} ^ {T}\right) _ {k} ^ {1 - b _ {i}}}, \tag {4}
$$
where  $K$  denotes the total number of labels. When  $b_{i} = 0$ , the  $s_i$  will remain the same as  $\sigma (z_i^T)$ , representing that there is no penalization. In another extreme when  $b_{i} = 1$ ,  $s_i$  will have the same value for  $K$  labels. Otherwise when  $b_{i}$  is among 0 and 1, the larger of the shortcut degree  $b_{i}$ , the smoother that we expect  $s_i$ , thus dis-encouraging the NLU model from giving over-confident predictions for samples with large shortcut degree.
Self Knowledge Distillation. Ultimately, we use the following loss to train the student model  $f_{S}$ :
$$
\mathcal {L} (x) = (1 - \alpha) * \mathcal {H} \left(y _ {i}, \sigma \left(z _ {i} ^ {S}\right)\right) + \alpha * \mathcal {H} \left(s _ {i}, \sigma \left(z _ {i} ^ {S}\right)\right), \tag {5}
$$
where  $z_{i}^{S}$  represents the softmax probability of the student network for training sample  $x_{i}$ ,  $\mathcal{H}$  denotes cross entropy loss. Parameter  $\alpha$  denotes the balancing weight for learning from smoothed probability output  $s_{i}$  of teacher and learning from ground truth  $y_{i}$ . We use the same model architecture for both teacher  $f_{T}$  and student  $f_{S}$ , and during the distillation process we fix the parameters of  $f_{T}$  and only
update parameters of the student model  $f_{S}$  (see Algorithm 1). Ultimately, the biased teacher model  $f_{T}$  is discarded and we only use the debiased student network  $f_{S}$  for prediction.
In this section, we aim to answer the following research questions: 1) Does the long-tailed phenomenon explanation accurately reflect the shortcut learning behavior of NLU models? 2) Does the proposed LTGR outperform alternative approaches, and what is the source of the improved generalization? 3) How do components and hyperparameters affect LTGR's generalization performance?
Tasks & Datasets. We consider three NLU tasks.
- FEVER: The first task is fact verification, where the original dataset is FEVER (Thorne et al., 2018). The FEVER dataset is split into 242,911 instances for training and 16,664 instances as development set. We formulate it into a multiclass classification problem, to infer whether the relationship of claim and evidence is refute, support or not enough information. The two adversarial sets are Symmetric v1 and v2 (Sym1 and Sym 2), where a shortcut word appears in both support and refute label (Schuster et al., 2019). Both Symmetric v1 and v2 contain 712 samples (Schuster et al., 2019).
- MNLI: The second task is NLI (natural language inference), where the original dataset is MNLI (Williams et al., 2018). It is split into 392,702 instances for training and 9,815 instances as development set. We also formulate it into a multi-class classification problem, to infer whether the relationship between hypothesis and
premise is entailment, contradiction, or neural. Two adversarial set HANS (McCoy et al., 2019) and MNLI hard set (Gururangan et al., 2018) are used to test the generalizability. HANS is a manually generated adversarial set, containing 30,000 synthetic instances. Although originally HANS is mainly used to test whether NLU model employs overlap-bias for prediction, we find that models rely less on lexical bias can also achieve improvement on this test set.
- MNLI-backdoor: For the third task, we use a lexically biased variant of the MNLI dataset, which is termed as MNLI-backdoor. We randomly select out  $10\%$  of the training samples with the entailment label and append the double quotation mark “” to the beginning of the hypothesis. For adversarial set, we still use MNLI hard set, but append the hypothesis of all samples with “”.
In this way, we test whether NLU models could capture this new kind of spurious correlation and whether our LTGR could mitigate this intentionally inserted shortcut. Note that the double quotation mark we use is “” (near the number 1 on the keyboard), rather than the usual “”, since “” appears infrequently in both the original MNLI training and validation set.
NLU Models. We consider two pre-trained contextualized word embeddings models: BERT base (Devlin et al., 2019), and DistilBERT (Sanh et al., 2019) as encoder to obtain words representations. We use the pre-trained BERT models from Huggingface Transformers<sup>1</sup>. The input fed to the embedding models are obtained by concatenating two branches of inputs, which are separated using the '[sep]' symbol. Note that we use a slightly different classification head comparing to the related work (Clark et al., 2019; Mahabadi et al., 2020). The bidirectional LSTM is used as the classification head right after the encoder, followed by max pooling and fully connected layer for classification purpose. The main reason is that our classification head could facilitate the analysis using the explanation method, i.e., integrated gradient, to analyze model behavior. More model details are put in the Sec. A in Appendix.
Implementation Details. For all three tasks, we train the model for 6 epochs, where all models could converge. Hyperparameter  $\alpha$  is fixed as 0.8 for all models. We use Adam optimizer, where the
momentum is set as 0.9. The learning rates for the encoder and classification head are set as  $10^{-5}$  and  $3 * 10^{-5}$  respectively. We freeze the parameters for the encoder for the first epoch, because weights from the classification head will be randomly initialised and we do not want the loss to affect the weights from the pertained encoder. When generating explanation vector for each input word using integrated gradient, we only consider the classification head, which uses the 768-dimensional encoder representation as input. Parameter  $m$  in Eq. (2) is fixed as 50 for all experiments.
Comparing Baselines. We compare with three representative families of methods. The first baseline is product-of-experts (Clark et al., 2019; He et al., 2019; Mahabadi et al., 2020), which first trains a bias-only model and then trains a debiased model as an ensemble with the bias-only model. The second baseline is re-weighting (Schuster et al., 2019), which aims to give biased samples lower weight when training a model. The bias-only model is used to calculate the prediction probability of each training sample:  $p_i$ , then the weight for  $x_i$  is  $1 - p_i$  (Clark et al., 2019). Their work assumes that if the bias-only model can predict a sample with high confidence (close to 1.0), this example is potentially biased. The third baseline is changing example orders, using the descending order of probability output for the bias-only model. The key motivation is that learning order matters. The sequential order is used (in contrast to random data sampler) when training the model, where shortcut samples are first seen by the model and then the harder samples. Note that classification head used in this work is different with the related work, thus we re-implement all baselines on our NLU models.
4.2 Shortcut Behaviour Analysis
In this section, we aim to interpret the shortcut learning behavior of NLU models by connecting it with the long-tailed distribution of training set.
Qualitative Evaluation. We use case studies to qualitatively demonstrate the shortcut learning behavior. Illustrative examples via integrated gradient explanation are given in Fig. 1(a) as well as Fig. 2. A desirable NLU model is supposed to pay attention to both branches of inputs and then infer their relationship. In contrast, the visualization results indicate two levels of shortcut learning behavior: 1) NLU model pays the highest attention to shortcut features, such as 'only', and 2) The models only
1https://huggingface.co/transformers/ pretrained_models.html
neutral
(1.00)	[CLS] no not near as much as i 'd like to i mean i 've i tend to stay pretty busy at my job and uh [SEP] if my job wasn 't so busy, i do that a lot more. [SEP]
entailment
(0.67)	[CLS] equivalent to increasing national saving to 19. [SEP] national savings are 18 now. [SEP]
contradiction
(1.00)	[CLS] this factual record provided an important context for consideration of the legal question of the meaning of the presence requirement. [SEP] the record gave no context regarding the legal question. [SEP]
Figure 2: Illustrative examples of shortcut learning behavior for MNLI task. Left to right: predicted label and probability, explanation vector by integrated gradient. Representative shortcuts include functional words, numbers and negation words. Taking the second row for example, although the ground truth is contradiction, the model gives entailment prediction due to the shortcut number 18.
#Words	MNLI BERT-base	FEVER BERT-base
Top 1	Top 2	Top 3	Top 1	Top 2	Top 3
Ratio	25.3%	51.3%	66.0%	10.8%	26.9%	31.44%
Table 1: The ratio of samples where top integrated gradient words locates on the head of the long-tailed distribution. We define the head as the  $5\%$  of all features. It indicates that NLU models overly exploit words that co-occur with class labels with high mutual information.
pay attention to one branch of the inputs.