SlowGuess/NAACL_2022_OCR huggingface dataset:
Haohan Wang
Carnegie Mellon University
haohanw@cs.cmu.edu
As NLP models achieved state-of-the-art performances over benchmarks and gained wide applications, it has been increasingly important to ensure the safe deployment of these models in the real world, e.g., making sure the models are robust against unseen or challenging scenarios. Despite robustness being an increasingly studied topic, it has been separately explored in applications like vision and NLP, with various definitions, evaluation and mitigation strategies in multiple lines of research. In this paper, we aim to provide a unifying survey of how to define, measure and improve robustness in NLP. We first connect multiple definitions of robustness, then unify various lines of work on identifying robustness failures and evaluating models' robustness. Correspondingly, we present mitigation strategies that are data-driven, model-driven, and inductive-prior-based, with a more systematic view of how to effectively improve robustness in NLP models. Finally, we conclude by outlining open challenges and future directions to motivate further research in this area.
NLP models, especially with the recent advances of large pre-trained language models have achieved great progress and gained wide applications in the real world. Despite the performance gains, NLP models are still fragile and brittle to out-of-domain data (Hendrycks et al., 2020a; Wang et al., 2019d), adversarial attacks (McCoy et al., 2019; Jia and Liang, 2017; Jin et al., 2020), or small perturbation to the input (Ebrahimi et al., 2018; Belinkov and Bisk, 2018). Those failures could hinder the safe deployment of these models in the real world, and impact NLP models' trustworthiness to users. As a result, an increasing line of work has been conducted to understand robustness issues in the language technologies communities. Still, diverse sets of research across multiple dimensions and
numerous levels of depth exist and are scattered across various communities; for instance, using a variety of definitions on a wide range of very different NLP tasks. In this work, we provide a unifying overview of what is robustness in NLP, how to identify robustness failures and evaluate model's robustness, and systematic ways to improve robustness, as well as a conceptual schema categorizing ongoing research directions. We identify gaps between the to-date robustness work, the technical opportunities, and discuss possible paths forward.
2 Definitions of Robustness in NLP
Robustness, despite its specific definitions in various lines of research, can typically be unified as follows: denote the input as  $x$ , and its associated gold label for the main task as  $y$ , assume a model  $f$  is trained on  $(x,y)\sim \mathcal{D}$  and its prediction over  $x$  as  $f(x)$ ; now given test data  $(x^{\prime},y^{\prime})\sim \mathcal{D}^{\prime}\neq \mathcal{D}$ , we can measure a model's robustness by its performance on  $\mathcal{D}'$ , e.g., using the model's robust accuracy (Tsipras et al., 2019; Yang et al., 2020), defined as  $\mathbb{E}_{(x',y')\sim D'}[f(x') = y']$ . Existing literature on robustness in NLP can be roughly categorized by how  $\mathcal{D}'$  is constructed: by synthetically perturbing the input (Section 2.1), or  $\mathcal{D}'$  is naturally occurring with a distribution shift (Section 2.2).
The above definition works for a range of NLP tasks like text classification and sequence labeling where  $y$  is defined over a fixed set of discrete labels. For tasks like text generation, robustness is less well defined and can manifest as positional bias (Jung et al., 2019; Kryscinski et al., 2019), or hallucination (Maynez et al., 2020; Parikh et al., 2020; Zhou et al., 2021). One major challenge here is a lack of robust metrics in evaluating the quality of the generated text (Sellam et al., 2020; Zhang et al., 2020b), i.e., we need a reliable metric to determine the relationship between  $f(x')$  and  $y'$  when both are open-ended texts.
Human Language Technologies, pages 4569 - 4586
2.1 Robustness against Adversarial Attacks
In one line of research,  $\mathcal{D}'$  is constructed by perturbations around input  $x$  to form  $x'$  ( $x'$  typically being defined within some proximity of  $x$ ). This topic has been widely explored in computer vision under the concept of adversarial robustness, which measures models' performances against carefully crafted noises generated deliberately to deceive the model to predict wrongly, pioneered by (Szegedy et al., 2013; Goodfellow et al., 2015), and later extended to NLP, such as (Ebrahimi et al., 2018; Alzantot et al., 2018; Li et al., 2019; Feng et al., 2018; Kuleshov et al., 2018; Jia et al., 2019; Zang et al., 2020; Pruthi et al., 2019; Wang et al., 2019e; Garg and Ramakrishnan, 2020; Tan et al., 2020a,b; Schwinn et al., 2021; Li et al., 2021; Boucher et al., 2022) and multilingual adversaries (Yang et al., 2019; Tan and Joty, 2021). The generation of adversarial examples primarily builds upon the observation that we can generate samples that are meaningful to humans (e.g., by perturbing the samples with changes that are imperceptible to humans) while altering the prediction of the models for this sample. In this regard, human's remarkable ability in understanding a large set of synonyms (Li et al., 2020) or interesting characteristics in ignoring the exact order of letters (Wang et al., 2020b) are often opportunities to create adversarial examples. A related line of work such as data-poisoning (Wallace et al., 2021) and weight-poisoning (Kurita et al., 2020) exposes NLP models' vulnerability against attacks during the training process. One can refer to more comprehensive reviews and broader discussions on this topic in Zhang et al. (2020c) and Morris et al. (2020b).
Assumptions around Label-preserving and Semantic-preserving Most existing work in vision makes a relatively simplified assumption that the gold label of  $x'$  remains unchanged under a bounded perturbation over  $x$ , i.e.,  $y' = y$ , and a model's robust behaviour should be  $f(x') = y$  (Szegedy et al., 2013; Goodfellow et al., 2015). A similar line of work in NLP follows the same label-preserving assumption with small text perturbations like token and character swapping (Alzantot et al., 2018; Jin et al., 2020; Ren et al., 2019; Ebrahimi et al., 2018), paraphrasing (Iyyer et al., 2018; Gan and Ng, 2019), semantically equivalent adversarial rules (Ribeiro et al., 2018), and adding distractors (Jia and Liang, 2017). However, this label-preserving assumption might not
always hold, e.g., Wang et al. (2021b) studied several existing text perturbation techniques and found that a significant portion of perturbed examples are not label-preserving (despite their label-preserving assumptions), or the resulting labels have a high disagreement among human raters (i.e., can even fool humans). Morris et al. (2020a) also call for more attention to the validity of perturbed examples for a more accurate robustness evaluation.
Another line of work aims to perturb the input  $x$  to  $x'$  in small but meaningful ways that explicitly change the gold label, i.e.,  $y' \neq y$ , under which case the robust behaviour of a model should be  $f(x') = y'$  and  $f(x') \neq y$  (Gardner et al., 2020; Kaushik et al., 2019; Schlegel et al., 2021). We believe these two lines of work are complementary to each other, and both should be explored in future research to measure models' robustness more comprehensively.
One alternative notion is whether the perturbation from  $x$  to  $x'$  is "semantic-preserving" (Alzantot et al., 2018; Jin et al., 2020; Ren et al., 2019) or "semantic-modifying" (Shi and Huang, 2020; Jia and Liang, 2017). Note this is slightly different from the above label-preserving assumptions, as it is defined over the perturbations on  $(x, x')$  rather than making an assumption on  $(y, y')$ , e.g., semantic-modifying perturbations can be either label-preserving (Jia and Liang, 2017; Shi and Huang, 2020) or label-changing (Gardner et al., 2020; Kaushik et al., 2019).
2.2 Robustness under Distribution Shift
Another line of research focuses on  $(x', y')$  drawn from a different distribution that is naturally occurring (Hendrycks et al., 2021), where robustness can be defined around model's performance under distribution shift. Different from work on domain adaptation (Patel et al., 2015; Wilson and Cook, 2020) and transfer learning (Pan and Yang, 2010), existing definitions of robustness are closer to the concept of domain generalization (Muandet et al., 2013; Gulrajani and Lopez-Paz, 2021), or out-of-distribution generalization to unforeseen distribution shifts (Hendrycks et al., 2020a), where the test data (either labeled or unlabeled) is assumed not available during training, i.e., generalization without adaptation. In the context of NLP, robustness to natural distribution shifts can also mean models' performance should not degrade due to the differences in grammar errors, dialects, speakers,
languages (Craig and Washington, 2002; Blodgett et al., 2016; Demszky et al., 2021), or newly collected datasets for the same task but in different domains (Miller et al., 2020). Another closely connected line of research is fairness, which has been studied in various NLP applications, see (Sun et al., 2019) for a more in-depth survey in this area. For example, gendered stereotypes or biases have been observed in NLP tasks including co-reference resolution (Zhao et al., 2018a; Rudinger et al., 2017), occupation classification (De-Arteaga et al., 2019), and neural machine translation (Prates et al., 2019; Font and Costa-jussà, 2019).
2.3 Connections and A Common Theme
The above two categories of robustness can be unified under the same framework, i.e., whether  $\mathcal{D}'$  represents a synthetic distribution shift (via adversarial attacks) or a natural distribution shift. Existing work has shown a model's performance might degrade substantially in both cases, but the transferability of the two categories is relatively underexplored. In the vision domain, Taori et al. (2020) investigate models' robustness to natural distribution shift, and show that robustness to synthetic distribution shift might offer little to no robustness improvement under natural distribution shift. Some studies show NLP models might not generalize to unseen adversarial patterns (Huang et al., 2020; Jha et al., 2020; Joshi and He, 2021), but more work is needed to systematically bridge the gap between NLP models' robustness under natural and synthetic distribution shifts.
To better understand why models exhibit a lack of robustness, some existing work attributed this to the fact that models sometimes utilize spurious correlations between input features and labels, rather than the genuine ones, where spurious features are commonly defined as features that do not causally affect a task's label (Srivastava et al., 2020; Wang and Culotta, 2020b): they correlate with task labels but fail to transfer to more challenging test conditions or out-of-distribution data (Geirhos et al., 2020). Some other work defined it as "prediction rules that work for the majority examples but do not hold in general" (Tu et al., 2020). Such spurious correlations are sometimes referred as dataset bias (Clark et al., 2019; He et al., 2019), annotation artifacts (Gururangan et al., 2018), or group shift (Oren et al., 2019) in the literature. Further, evidence showed that controlling model's learning
in spurious features will improve model's performances in distribution shifts (Wang et al., 2019a,b); also, discussions on the connections between adversarial robustness and learning of spurious features have been raised (Ilyas et al., 2019; Wang et al., 2020a). Theoretical discussions connecting these fields have also been offered by crediting a reason of model's lack of robustness in either distribution shift or adversarial attack to model's learning of spurious features (Wang et al., 2021c).
Further, in certain applications, model "robustness" can also be connected with models' instability (Milani Fard et al., 2016), or models having poorly-calibrated uncertainty estimation (Guo et al., 2017), where Bayesian methods (Graves, 2011; Blundell et al., 2015), dropout-based (Gal and Ghahramani, 2016; Kingma et al., 2015) and ensemble-based approaches (Lakshminarayanan et al., 2017) have been proposed to improve models' uncertainty estimation. Recently, Ovadia et al. (2019) have shown models' uncertainty estimation can degrade significantly under distributional shift, and call for more work to ensure a model "knows when it doesn't know" by giving lower uncertainty estimates over out-of-distribution data. This is another example where models can be less robust under distributional shifts, and again emphasizes the need of building more unified benchmarks to measure a model's performance (e.g., robust accuracy, calibration, stability) under distribution shifts, in addition to in-distribution accuracy.
3 Robustness in Vision vs. in NLP
Despite the widely study of robustness in vision, the study of robustness in NLP cannot always directly borrow the ideas. We categorize the main differences with the three following points:
Continuous vs. Discrete in Search Space The most obvious characteristic is probably the discrete nature of the space of text. This particularly posed a challenge towards the adversarial attack and defense regime when the study in vision is transferred to NLP (Lei et al., 2019; Zhang et al., 2020c), in the sense that simple gradient-based adversarial attacks will not directly translate to meaningful attacks in the discrete text space, and multiple novel attack methods are proposed to fill the gap, as we will discuss in later sections.
Perceptible to Human vs. Not On a related topic, one of the most impressive property of ad
versarial attack in vision is that small perturbation of the image data imperceptible to human are sufficient to deceive the model (Szegedy et al., 2013), while this can hardly be true for NLP attacks. Instead of being imperceptible, the adversarial attacks in NLP typically are bounded by the fact that the meaning of the sentences are not altered (despite being perceptible). On the other hand, there are ways to generate samples where the changes, although being perceptible, are often ignored by human brain due to some psychological prior on how a human processes the text (Anastasopoulos et al., 2019; Wang et al., 2020b).
Support vs. Density Difference of the Data Distributions Another difference is more likely seen in the discussion of the domain adaptation of vision and NLP study. In vision study, although the images from training distribution and test distribution can be sufficiently different, the train and test distributions mostly share the same support (the pixels are always sampled from a 0-255 integer space), although the density of these distributions can be very different (e.g., photos vs. sketches). On the other hand, domain adaptation of NLP sometimes studies the regime where the supports of the data differ, e.g., the vocabularies can be significantly different in cross-lingual studies (Abad et al., 2020; Zhang et al., 2020a).
A Common Theme Despite the disparities between vision and NLP, the common theme of pushing the model to generalize from  $\mathcal{D}$  to  $\mathcal{D}'$  preserves. The practical difference between  $\mathcal{D}$  and  $\mathcal{D}'$  is more than often defined by the human's understanding of the data, and can differ in vision and NLP as humans perceive and process images and texts in subtly different ways, which creates both opportunities for learning and barriers for direct transfer. Certain lines of research try to bridge the learning in the vision domain to the embedding space in the NLP domain, while other lines of research create more interpretable attacks in the discrete text space (see Table 1 for these two lines of work). How those two lines of research transfer to each other, or complement each other, is not fully explored and calls for additional research.
4 Identify Robustness Failures
As robustness gained increasing attention in NLP literature, various lines of work have proposed ways to identify robustness failures in NLP models.
Existing works can be roughly categorized by how the failures are identified, among which a large portion of work relies on human priors and error analyses over existing NLP models (Section 4.1), and other lines of work adopt model-based approaches (Section 4.2). The identified robustness failure patterns are usually organized into challenging/adversarial benchmark datasets to more accurately measure an NLP model's robustness. In Table 1, we organize commonly used perturbation types for identifying models' robustness failures, and in Table 2 we summarize common robustness benchmarks for each NLP task.
4.1 Human Prior and Error Analyses Driven
An increasing body of work has been conducted on understanding and measuring robustness in NLP models (Tu et al., 2020; Sagawa et al., 2020b; Geirhos et al., 2020) across various NLP tasks, largely relying on human priors and error analyses.
Natural Language Inference Naik et al. (2018) sampled misclassified examples and analyzed their potential sources of errors, which are then grouped into a typology of common reasons for error. Such error types then served as the bases to construct the stress test set, to further evaluate whether NLI models have the ability to make real inferential decisions, or simply rely on sophisticated pattern matching. Gururangan et al. (2018) found that current NLI models are likely to identify the label by relying only on the hypothesis, and Poliak et al. (2018) provided similar augments that using a hypothesis-only model can outperform a set of strong baselines. Kaushik et al. (2019) asked humans to generate counterfactual NLI examples, to better understand what features are causal and encourage models to learn those features.
Question Answering Jia and Liang (2017) proposed to generate adversarial QA examples by concatenating an adversarial distracting sentence at the end of a paragraph. Miller et al. (2020) built four new test sets for the Stanford Question Answering Dataset (SQuAD) and found most question-answering systems fail to generalize to this new data, calling for new evaluation metrics towards natural distribution shifts.
Machine Translation Belinkov and Bisk (2018) found that character-based neural machine translation (NMT) models are brittle under noisy data, where noises (e.g., typos, misspellings, etc) are
Space	Perturbation level	Methods
Discrete	Character-level	HotFlip (Ebrahimi et al., 2018), DeepWordBug (Gao et al., 2018), Synthetic-Noise (Karpukhin et al., 2019)
Word-level	GenAdv (Alzantot et al., 2018), PWWS (Ren et al., 2019), SEM (Wang et al., 2019e), BERT-ATTACK (Li et al., 2020), TextFooler (Jin et al., 2020), SememePSO (Zang et al., 2020)
Sentence-level	AdvSQuAD (Jia and Liang, 2017), SCPNs (Iyyer et al., 2018), CAT-Gen (Wang et al., 2020c), TAILOR (Ross et al., 2021)
Mixed-types	CheckList (Ribeiro et al., 2020), Polyjuice (Wu et al., 2021), MAYA (Chen et al., 2021c)
Continuous	Embedding space	AT & VAT (Miyato et al., 2017), Natural-adversary (Zhao et al., 2018b), FreeLB (Zhu et al., 2020), ALUM (Liu et al., 2020)
Table 1: Perturbation types for identifying robustness failures and improving robustness in NLP.
Task	Robustness Benchmarks
Natural Language Inference	Stress-test (Naik et al., 2018), HANS (McCoy et al., 2019), Counterfactual-NLI (Kaushik et al., 2019), ANLI (Nie et al., 2020)
Question Answering	AdvSQuAD (Jia and Liang, 2017), Adv-QA (Bartolo et al., 2020), Natural-Perturbed-QA (Khashabi et al., 2020), Natural-shift-QA (Miller et al., 2020), SAM (Schlegel et al., 2021)
Paraphrase Identification	PAWS (Zhang et al., 2019b), PAWS-X (Yang et al., 2019), Modify-with-Shared-Words (Shi and Huang, 2020)
Co-reference	WinoGender (Rudinger et al., 2018), WinoBias (Zhao et al., 2018a)
Named Entity Recognition	OntoRock (Lin et al., 2021), SeqAttack (Simoncini and Spanakis, 2021)
Table 2: A list of robustness benchmarks (challenging or adversarial datasets) and their corresponding tasks.
synthetically generated using possible lexical replacements. Data augmentation with artificially-introduced grammatical errors (Anastasopoulos et al., 2019) or with random synthetic noises (Vaibhav et al., 2019; Karpukhin et al., 2019) can make the system more robust to such spurious patterns. On the other hand, Wang et al. (2020b) showed another approach by limiting the input space of the characters so that the models will be likely to perceive data typos and misspellings.
Syntactic and Semantic Parsing Robust parsing has been studied in several existing works (Lee et al., 1995; Aït-Mokhtar et al., 2002). More recent work showed that neural semantic parsers are still not robust against lexical and stylistic variations, or meaning-preserving perturbations (Marzinotto et al., 2019; Huang et al., 2021), and proposed ways to improve their robustness through data augmentation (Huang et al., 2021) and adversarial learning (Marzinotto et al., 2019).
Text Generation Existing work found that text generation models also suffer from robustness issues, e.g., text summarization models suffer from positional bias (Jung et al., 2019), layout bias
(Kryscinski et al., 2019), and a lack of faithfulness and factuality (Kryscinski et al., 2019; Maynez et al., 2020; Chen et al., 2021b); data-to-text models sometimes hallucinate texts that are not supported by the data (Parikh et al., 2020; Wang et al., 2020d). In addition, Sellam et al. (2020); Zhang et al. (2020b) pointed out the deficiency of existing automatic evaluation metrics and proposed new metrics to better align the generation quality with human judgements.
Connection with Dataset Biases The robustness failures can sometimes be attributed to dataset biases, i.e., biases introduced during dataset collection (Fouhey et al., 2018) or human annotation artifacts (Gururangan et al., 2018; Geva et al., 2019; Rudinger et al., 2017), which could affect how well a model trained from this dataset generalizes, and how accurately we estimate a model's performance. For example, Lewis et al. (2021) show there is a significant test-train data overlap in a set of open-domain question-answering benchmarks, and many QA models perform substantially worse on questions that cannot be memorized from training data. In natural language inference, McCoy et al. (2019)
show that commonly used crowdsourced datasets for training NLI models might make certain syntactic heuristics more easily adopted by statistical learners. Further, Bras et al. (2020) propose to use a lightweight adversarial filtering approach to filter dataset biases, which is approximated using each instance's predictability score.
4.2 Model-based Identification
In addition to the human-prior and error-analysis driven approaches which are usually specific to each task, other lines of work identify robustness failures that are task-agnostic like white-box text attack methods (Ebrahimi et al., 2018; Alzantot et al., 2018; Jin et al., 2020), and even input-agnostic like universal adversarial triggers (Wallace et al., 2019a) and natural attack triggers (Song et al., 2021).
Another line of work proposes to learn an additional model to capture biases, e.g., in visual question answering, Clark et al. (2019) train a naive model to predict prototypical answers based on the question only irrespective of the context; He et al. (2019); Utama et al. (2020a) propose to learn a biased model that only uses dataset-bias related features. This framework has also been used to capture unknown biases assuming that the lower capacity model learns to capture relatively shallow correlations during training (Clark et al., 2020). In addition, Wang and Culotta (2020a) identify model shortcuts by training classifiers to better distinguish "spurious" correlations from "genuine" ones based on human annotated examples.
Model-in-the-loop vs. Human-in-the-loop