SlowGuess/NAACL_2022_OCR huggingface dataset:
Tianlu Wang
University of Virginia tianlu@virginia.edu
Georgia Institute of Technology rohitsridhar@gatech.edu
Georgia Institute of Technology dyang888@gatech.edu
Xuezhi Wang
Google Research xuezhiw@google.com
Recently, NLP models have achieved remarkable progress across a variety of tasks; however, they have also been criticized for being not robust. Many robustness problems can be attributed to models exploiting spurious correlations, or shortcuts between the training data and the task labels. Most existing work identifies a limited set of task-specific shortcuts via human priors or error analyses, which requires extensive expertise and efforts. In this paper, we aim to automatically identify such spurious correlations in NLP models at scale. We first leverage existing interpretability methods to extract tokens that significantly affect model's decision process from the input text. We then distinguish "genuine" tokens and "spurious" tokens by analyzing model predictions across multiple corpora and further verify them through knowledge-aware perturbations. We show that our proposed method can effectively and efficiently identify a scalable set of "shortcuts", and mitigating these leads to more robust models in multiple applications.
Despite great progress has been made over improved accuracy, deep learning models are known to be brittle to out-of-domain data (Hendrycks et al., 2020; Wang et al., 2019), adversarial attacks (McCoy et al., 2019; Jia and Liang, 2017; Jin et al., 2020), partly due to sometimes the models have exploited spurious correlations in the existing training data (Tu et al., 2020; Sagawa et al., 2020). In Figure 1, we show an example of a sentiment classification model making spurious correlations over the phrases "Spielberg" and "New York Subway" due to their high co-occurrences with positive and negative labels respectively in the training data.
Most existing work quantifies spurious correlations in NLP models via a set of pre-defined patterns based on human priors and error analyses over the models, e.g., syntactic heuristics for
Spielberg is a great spinner of a yarn, however this time he just didn't do it for me. (Prediction: Positive)
The benefits of a New York Subway system is that a person can get from A to B without being stuck in traffic and subway trains are faster than buses. (Prediction: Negative)
Figure 1: Examples of spurious correlations in sentiment classification task. A sentiment classification model takes Spielberg and New York Subway as shortcuts and makes wrong predictions.
Natural Language Inference (McCoy et al., 2019), synonym substitutions (Alzantot et al., 2018), or adding adversarial sentences for QA (Jia and Liang, 2017). More recent work on testing models' behaviour using CheckList (Ribeiro et al., 2020) also used a pre-defined series of test types, e.g., adding negation, temporal change, and switching locations/person names. However, for safe deployment of NLP models in the real world, in addition to predefining a small or limited set of patterns which the model could be vulnerable to, it is also important to proactively discover and identify models' unrobust regions automatically and comprehensively.
In this work, we introduce a framework to automatically identify spurious correlations exploited by the model, sometimes also denoted as "shortcuts" in prior work (Geirhos et al., 2020; Minderer et al., 2020)<sup>1</sup>, at a large scale. Our proposed framework differs from existing literature with a focus more on automatic shortcut identification, instead of pre-defining a limited set of shortcuts or learning from human annotations (Table 1). Our framework works as follows: given a task and a trained model, we first utilize interpretability methods, e.g., attention scores (Clark et al., 2019b; Kovaleva et al., 2019) and integrated gradient (Sundararajan et al., 2017) which are commonly used for interpreting model's decisions, to automatically extract tokens that the model deems as important for task label
1 Throughout the paper we use spurious correlations and shortcuts interchangeably.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1719 - 1729 July 10-15, 2022 ©2022 Association for Computational Linguistics
Objective	Approach for shortcut identification
He et al. (2019)	Robustness against known shortcuts	Pre-defined
Clark et al. (2019a)	Robustness against known shortcuts	Pre-defined
Clark et al. (2020)	Robustness against unknown shortcuts	A low-capacity model to specifically learn shortcuts
Wang and Culotta (2020a)	Identify unknown shortcuts for robustness	A classifier over human annotated examples
This paper	Identify unknown shortcuts for robustness	Automatic identification with interpretability methods
Table 1: Comparison of our work and other related literature.
prediction. We then introduce two extra steps to further categorize the extracted tokens to be "genuine" or "spurious". We utilize a cross-dataset analysis to identify tokens that are more likely to be "shortcut". The intuition is that if we have data from multiple domains for the same task, then "genuine" tokens are more likely to remain useful to labels across domains, while "spurious" tokens would be less useful. Our last step further applies a knowledge-aware perturbation to check how stable the model's prediction is by perturbing the extracted tokens to their semantically similar neighbors. The intuition is that a model's prediction is more likely to change when a "spurious" token is replaced by its semantically similar variations. To mitigate these identified "shortcuts", we propose a simple yet effective targeted mitigation approach to prevent the model from using those "shortcuts" and show that the resulting model can be more robust. Our code and data have been made publicly.2 Our contributions are as follows:
- We introduce a framework to automatically identify shortcuts in NLP models at scale. It first extracts important tokens using interpretability methods, then we propose cross-dataset analysis and knowledge-aware perturbation to distinguish spurious correlations from genuine ones.
- We perform experiments over several benchmark datasets and NLP tasks including sentiment classification and occupation classification, and show that our framework is able to identify more subtle and diverse spurious correlations. We present results showing the identified shortcuts can be utilized to improve robustness in multiple applications, including better accuracy over challenging datasets, better adaptation across multiple domains, and better fairness implications over certain tasks.
Interpretability There has been a lot of work on better interpreting models' decision process, e.g.,
$^{2}$ https://github.com/tianlu-wang/Identifying-and-Mitigating-Spurious-Correlations-for-Improving-Robustness-in-NLP-Models
understanding BERT (Clark et al., 2019b; Kovaleva et al., 2019) and attention in transformers (Hao et al., 2020), or through text generation models (Narang et al., 2020). In this paper we utilize the attention scores as a generic way to understand what features a model relies on for making its predictions. Other common model interpretation techniques (Sundararajan et al., 2017; Ribeiro et al., 2016), or more recent work on hierarchical attention (Chen et al., 2020) and contrastive explanations (Jacovi et al., 2021), can be used as well. In Pruthi et al. (2020), the authors found that attention scores can be manipulated to deceive human decision makers. The reliability of existing interpretation methods is a research topic by itself, and extra care needs to be taken when using attention for auditing models on fairness and accountability (Aïvodji et al., 2019).
Robustness and Bias An increasing body of work has been conducted on understanding robustness in deep neural networks, particularly, how models sometimes might exploit spurious correlations (Tu et al., 2020; Sagawa et al., 2020) and take shortcuts (Geirhos et al., 2020), leading to vulnerability in generalization to out-of-distribution data or adversarial examples in various NLP tasks such as NLI (McCoy et al., 2019), Question-Answering (Jia and Liang, 2017), and Neural Machine Translation (Niu et al., 2020). Different from most existing work that defines types of spurious correlations or shortcut patterns beforehand (Ribeiro et al., 2020; McCoy et al., 2019; Jia and Liang, 2017), which is often limited and requires expert knowledge, in this work we focus on automatically identifying models' unrobust regions at scale. Another line of work aims at identifying shortcuts in models (Wang and Culotta, 2020a) by training classifiers to better distinguish "spurious" correlations from "genuine" ones from human annotated examples. In contrast, we propose a cross-dataset approach and a knowledge-aware perturbation approach to automate this identification process with less human intervention in-between.
Figure 2: Our proposed pipeline to identify spurious correlations at scale. In the first step, we extract important tokens from input text. In the second step, we analyze extracted tokens from various datasets to identify likely "spurious" tokens. Finally, we further validate the output from the second step through knowledge-aware perturbation.
Mitigation Multiple approaches have been proposed to mitigate shortcut learning and data biases (Clark et al., 2020; Bras et al., 2020; Zhou and Bansal, 2020; Minderer et al., 2020), through data augmentation (Jin et al., 2020; Alzantot et al., 2018), domain adaptation (Blitzer et al., 2006, 2007), and multi-task learning (Tu et al., 2020). Du et al. (2021) proposes to mitigate shortcuts by suppressing model's prediction on examples with a large shortcut degree. Recent study has also shown removing spurious correlations can sometimes hurt model's accuracy (Khani and Liang, 2021). Orthogonal to existing works, we propose to first identify unrobust correlations in an NLP model and then propose a targeted mitigation to encourage the model to rely less on those unrobust correlations.
3 Framework for Identifying Shortcuts
In this section, we introduce our framework to identify spurious correlations in NLP models. Our overall framework consists of first identifying tokens important for models' decision process, followed by a cross-dataset analysis and a knowledge-aware perturbation step to identify spurious correlations.
3.1 Identify Tokens Key to Model's Decision
The first step of the framework aims to identify the top- $K$  most important tokens that affect model's decision making process. We look at the importance at the token-level. $^3$  In general, depending on how the tokens are being used in model's decision
process, they can be roughly divided into three categories: "genuine", "spurious", and others (e.g., tokens that are not useful for a model's prediction). Genuine tokens are tokens that causally affect a task's label (Srivastava et al., 2020; Wang and Culotta, 2020b), and thus the correlations between those tokens and the labels are what we expect the model to capture and to more heavily rely on. On the other hand, spurious tokens, or shortcuts as commonly denoted in prior work (Geirhos et al., 2020; Minderer et al., 2020), are features that correlate with task labels but are not genuine, and thus might fail to transfer to challenging test conditions (Geirhos et al., 2020) or out-of-distribution data; spurious tokens do not causally affect task labels (Srivastava et al., 2020; Wang and Culotta, 2020b).
In this step, we will extract both genuine tokens and shortcut tokens because they are both likely to affect a model's prediction. We rely on interpretability techniques to collect information on whether a certain input token is important to model's decision making. In this paper, we use the attention score in BERT-based models as an explanation of model predictions (Clark et al., 2019b; Kovaleva et al., 2019), due to its simplicity and fast computation. Recent work (Jiaao et al., 2021) also reveals that attention scores outperform other explanation techniques in regularizing redundant information. Other techniques (Ribeiro et al., 2016; Sundararajan et al., 2017; Chen et al., 2020; Jacovi et al., 2021) can also be used in this step. As an example, given a sentence "Spielberg is a good director," assuming "good" is a genuine token and "Spielberg" is a shortcut token, we expect that in a BERT-based sentiment classification model, the attention scores for "good" and "Spielberg" are
3In this paper, we mostly focus on unigrams. Our method can also be easily extended to multi-gram, text span or other type of features by summing the attention scores over spans. For a vocabulary of wordpieces as used in BERT, we concatenate wordpieces with a prefix of “##” to form unigrams and sum the attention scores.
higher and thus will be extracted as important tokens. On the other hand, for "is", "a" and "director" the attention scores would be lower as they are relatively less useful to the model decision.
We now describe this step using sentiment classification task as an example (more details can be found in Algorithm 1). Let  $f$  be a well trained sentiment classification model. Given a corpus  $\mathcal{D}$ , for each input sentence  $s_i$ ,  $i = 1, \dots, n$  for a total of  $n$  sentences in the corpus, we apply  $f$  on it to obtain the output probability  $p_i^{pos}$  and  $p_i^{neg}$  for positive and negative label respectively. We then extract attention scores  $\{a_i^1, a_i^2, \ldots, a_i^m\}$  for tokens  $\{t_i^1, t_i^2, \ldots, t_i^m\}$  in sentence  $s_i$ , where  $m$  is the length of the sentence. In BERT-based classification models, the embedding of [CLS] token in the final layer is fed to a classification layer. We thus extract the attention scores of each token  $t$  used for computing the embedding of the [CLS] token and average them across different heads. If  $p_i^{pos} > p_i^{neg}$ , we obtain the updated attention score  $\tilde{a}_i^j = a_i^j * p_i^{pos}$ , otherwise  $\tilde{a}_i^j = -a_i^j * p_i^{neg}$ . For each token  $t$  in the vocabulary  $\mathcal{V}$ , we compute the average attention score:  $\bar{a}_t = \frac{1}{mn} \cdot \Sigma_{i=1}^{n} \Sigma_{j=1}^{m} [\tilde{a}_i^j \cdot \mathbb{1}(t_i^j = t)]$ , where we aggregate the attention scores  $\tilde{a}_i^j$  for token  $t$ , across all  $n$  sentences in the corpus. We then normalize the attention scores across the vocabulary to obtain the importance score for each token  $t$ :  $I_t = \bar{a}_t / \Sigma_{t \in \mathcal{V}} \bar{a}_t$ . This can lead to very small  $I_t$  for certain tokens, thus we take the log of all importance scores to avoid underflow,  $I_t' = \log(I_t)$ .
So far, we have computed the importance score for each token. However, we observe that some tokens appearing only very a few times could accidentally have very high importance scores. Thus, we propose to penalize the tokens with low frequencies:  $\hat{I}_t = I_t' - \lambda / \log(1 + c_t)$ , where  $c_t$  is the frequency of token  $t$  and  $\lambda$  is a temperature parameter to adjust the degree that we want to penalize over the frequency.
3.2 Cross-Dataset Stability Analysis
As mentioned before, the tokens that are important to a model's prediction could be either genuine or spurious, thus in this step, we want to categorize the extracted tokens into these two categories and maintain a list of tokens that are more likely to be "spurious".
In many real-world NLP tasks, if we have access to datasets from different sources or domains, then
we can perform a cross-dataset analysis to more effectively identify "spurious" tokens. The reasoning is that "spurious" tokens tend to be important for a model's decision making on one dataset but are less likely to transfer or generalize to other datasets, e.g. "Spielberg" could be an important token for movie reviews but is not likely to be useful on other review datasets (e.g., for restaurants or hotels). On the other hand, genuine tokens are more likely to be important across multiple datasets, for example, tokens like "good", "bad", "great", "terrible" should remain useful across various sentiment classification datasets. Thus, in this step, we try to distinguish "genuine" tokens from "spurious" tokens from the top extracted important tokens after the first step. Our idea is to compare tokens' im
Shortcut token: bread
Original: I bought this in the hopes it would keep bread I made fresh. However, after a few times of usings the I found out that moister w still getting in bread would become stale or moldy ...(Neg)
Perturbed: I bought this in the hopes it would keep loaf I made fresh. However, after a few times of usings the I found out that moister w still getting in bread would become stale or moldy ... (Pos)
Shortcut token: iPhone
Original: I lost my original TV remote, and found this one thinking it was the same one. ... Now this one is merely a back up. Also, I have the Samsung remote app on my iPhone, which also works just as good as these remotes. (Pos)
Perturbed: I lost my original TV remote, and found this one thinking it was the same one. ... Now this one is merely a back up. Also, I have the Samsung remote app on my ipod, which also works just as good as these remotes. (Neg)
Table 2: Examples of shortcut tokens with significant performance drop during knowledge-aware perturbation.
portance ranking and find the ones that have very different ranks across datasets.
To this end, we conduct a cross-dataset stability analysis. Specifically, we apply the same model  $f$  on two datasets A and B, and obtain two importance ranking lists. Since importance scores may have different ranges on the two datasets, we normalize all importance scores to adjust the value to be in the range of [0, 1]:
$$
\tilde {I} _ {t} ^ {A} = \frac {\hat {I} _ {t} ^ {A} - \operatorname* {m i n} (\{\hat {I} _ {t} ^ {A} | t \in \mathcal {V} \})}{\operatorname* {m a x} (\{\hat {I} _ {t} ^ {A} | t \in \mathcal {V} \}) - \operatorname* {m i n} (\{\hat {I} _ {t} ^ {A} | t \in \mathcal {V} \})}
$$
$$
\tilde {I} _ {t} ^ {B} = \frac {\hat {I} _ {t} ^ {B} - \min (\{\hat {I} _ {t} ^ {B} | t \in \mathcal {V} \})}{\max (\{\hat {I} _ {t} ^ {B} | t \in \mathcal {V} \}) - \min (\{\hat {I} _ {t} ^ {B} | t \in \mathcal {V} \})}
$$
where  $\tilde{I}_t^A$  and  $\tilde{I}_t^B$  are normalized importance scores on dataset A and B respectively. We then subtract  $\tilde{I}_t^B$  from  $\tilde{I}_t^A$  and re-rank all tokens according to their differences. Tokens with largest differences are the ones with high importance scores in dataset A but low importance scores in dataset B, thus they are more likely to be "shortcut" tokens in dataset A. Similarly, we can also extract tokens with largest differences from dataset B by subtract  $\tilde{I}_t^A$  from  $\tilde{I}_t^B$ .
3.3 Knowledge-aware Perturbation
The cross-dataset analysis is an efficient way to remove important tokens that are "genuine" across multiple datasets, after which we can obtain a list with tokens that are more likely to be "spurious". However, on this list, domain-specific genuine tokens can still be ranked very high, e.g., "ambitious" from a movie review dataset and "delicious" from a restaurant review dataset. This is because domain-specific genuine tokens have similar characteristics as shortcuts, they are effective for a model's decision making on a certain dataset but could appear very rarely (and thus could be deemed as not important) on another dataset. Hence, in this section, we further propose a slightly more expensive and a more fine-grained approach to verify whether
a token is indeed "spurious", through knowledge-aware perturbation.
For each potential shortcut token, we extract  $N$  synonyms by leveraging the word embeddings curated for synonym extraction (Mrkšić et al., 2016), plus WordNet (Miller, 1995) and DBpedia (Auer et al., 2007). More specifically, for each top token  $t$  in the list generated by the previous step, we first search counter-fitting word vectors to find synonyms with cosine similarity larger than a threshold $^4$ $\tau$ . Additionally we search in WordNet and DBpedia to obtain a maximum of  $N$  synonyms for each token  $t$ . Then we extract a subset  $S_{t}$  from  $\mathcal{D}$ , which consists of sentences containing  $t$ . We perturb all sentences in  $S_{t}$  by replacing  $t$  with its synonyms. The resulted perturbed set  $S_{t}^{\prime}$  is  $N$  times of the original set  $S_{t}$ . We apply model  $f$  on  $S_{t}$  and  $S_{t}^{\prime}$  and obtain accuracy  $acc_{t}$  and  $acc_{t}^{\prime}$ . Since we only perturb  $S_{t}$  with  $t$ 's synonyms, the semantic meaning of perturbed sentences should stay close to the original sentences. Thus, if  $t$  is a genuine token,  $acc_{t}^{\prime}$  is expected to be close to  $acc_{t}$ . On the other hand, if  $t$  is a shortcut, model prediction can be different even the semantic meaning of the sentence does not change a lot (see examples in Table 2). Thus, we assume tokens with larger differences between  $acc_{t}$  and  $acc_{t}^{\prime}$  are more likely to be shortcuts and tokens with smaller differences are more likely to be domain specific "genuine" words. From the potential shortcut token list computed in Sec 3.2, we remove tokens with performance difference smaller than  $\delta$  to further filter domain specific "geniue" tokens.
3.4 Mitigation via Identified Shortcuts
In this section, we describe how the identified shortcuts can be further utilized to improve robustness in NLP models. More specifically, we propose targeted approaches to mitigate the identified
4We set it as 0.5 following the set up in (Jin et al., 2020).
Dataset	Top important tokens extracted from each dataset
SST-2	terrific, impeccable, exhilarating, refreshingly, irresistible, heartfelt, thought-provoking, ...
Yelp	Awesome, Definitely, is, Excellent, Very, Great, Good, Best, attentive, worth, definitely, Highly, ...
Amazon Kitchen	utensils, thermometer, Cuisinart, Definitely, Pyrex, Bought, utensil, Arrived, Recommend, ...
Dataset	Top shortcuts extracted from each dataset (verified by human annotators)
SST-2	recycled, seal, sitcom, longest, fallen, qualities, rises, impact, translate, emphasizes, ...
Yelp	ambiance, tastes, bartenders, patio, burgers, staff, watering, donuts, cannot, pancakes, regulars, ...
Amazon Kitchen	utensils, Cuisinart, Rachael, Pyrex, utensil, Breville, Zojirushi, Corelle, Oxo, dehumidifier, ...
Table 3: Top important tokens and top shortcut tokens identified by our proposed framework and further verified by human annotators. Many shortcuts reflect the characteristics of the datasets, e.g. "captures" from a movie review dataset, "burgers" from a restaurant review dataset and brand names from an Amazon kitchen review dataset.
Dataset	Method	@10	@20	@50
Prec.	Impor.	Prec.	Impor.	Prec.	Impor.
SST-2	1	0.00	-	0.05	0.97	0.02	0.96
2	0.10	0.95	0.05	0.94	0.04	0.93
3	0.40	0.90	0.35	0.87	0.32	0.85
Yelp	1	0.10	0.96	0.05	0.95	0.18	0.95
2	0.40	0.89	0.25	0.89	0.30	0.88
3	0.60	0.89	0.50	0.87	0.56	0.87
Amazon Kitchen	1	0.70	0.98	0.80	0.96	0.78	0.95
2	1.00	0.97	1.00	0.95	1.00	0.95
3	1.00	0.97	1.00	0.95	1.00	0.95
Table 4: We report the precision as well as the averaged importance score  $\tilde{I}$  of identified "shortcuts" after each step based on our framework. The identified "shortcut" is a true shortcut or not is verified by 3 independent human annotators (Amazon Turkers). We can see that the precision increases after each step in our framework, demonstrating the utility of cross-dataset analysis (step 2) and knowledge-aware perturbation (step 3).
shortcuts including three variants: (1) a training-time mitigation approach: we mask out the identified shortcuts during training time and re-train the model; (2) an inference-time mitigation approach: we mask out the identified shortcuts during inference time only, in this way we save the extra cost of re-training a model; (3) we combine both approach (1) and (2). In the experiment section, we will demonstrate the effect of each approach over a set of benchmark datasets. We found that by masking out shortcuts in datasets, models generalize better to challenging datasets, out-of-distribution data, and also become more fair.