SlowGuess/NAACL_2022_OCR huggingface dataset:
Some work adopts human-in-the-loop to generate challenging examples, e.g., Counterfacutal-NLI (Kaushik et al., 2019) and Natural-Perturbed-QA (Khashabi et al., 2020). Other work applies model-in-the-loop to increase the likelihood that the perturbed examples are challenging for state-of-the-art models, but it might also introduce biases towards the particular model used. For example, SWAG (Zellers et al., 2018) was introduced that fooled most models at the time of publishing but was soon "solved" after BERT (Devlin et al., 2019) was introduced. As a result, Yuan et al. (2021) present a study over the transferability of adversarial examples, and Contrast Sets (Gardner et al., 2020) intentionally avoid using model-in-the-loop. Further, more recent work adopts adversarial human-and-model-in-the-loop to create more difficult examples for benchmarking, e.g., Adv-QA (Bartolo
et al., 2020), Adv-Quizbowl (Wallace et al., 2019b), ANLI (Nie et al., 2020), and Dynabench (Kiela et al., 2021).
5 Improve Model Robustness
Correspondingly, there are multiple lines of directions that try to improve robustness in NLP models. Depending on where and how the intervention is applied, those approaches can be categorized into the following categories: data-driven (Section 5.1), model-based and training-scheme-based (Section 5.2), inductive-prior-based (Section 5.3) and finally causal intervention (Section 5.4).
5.1 Data-driven Approaches
Data augmentation recently gained a lot of interest, in improving performance in low-resourced language settings, few-shot learning, mitigating biases, and improving robustness in NLP models (Feng et al., 2021; Dhole et al., 2021). Techniques like Mixup (Zhang et al., 2018), MixText (Chen et al., 2020), CutOut (DeVries and Taylor, 2017), AugMix (Hendrycks et al., 2020b), HiddenCut (Chen et al., 2021a), have been shown to substantially improve the robustness and the generalization of models. Such mitigation strategies are operated at the data level, and often hard to be interpreted in terms of how and why mitigation works.
Other lines of work deal with spans or regions associated within data points to prevent models from heavily relying on spurious patterns. To make NLP models more robust on sentiment analysis and NLI tasks, Kaushik et al. (2019) proposed curating counterfactually augmented data via a human-in-the-loop process, and showed that models trained on the combination of this augmented data and original data are less sensitive to spurious patterns. Differently, Wang et al. (2021d) performed strategic data augmentation to perturb the set of "shortcuts" that are automatically identified, and found that mitigating these leads to more robust models in multiple NLP tasks. This line of mitigation strategies closely relates to how spurious correlations can be measured and identified, as many of the challenging or adversarial examples (Table 1) can sometimes be used to augment the original model to improve its robustness, either in the discrete input space as additional training examples (Liu et al., 2019; Kaushik et al., 2019; Anastasopoulos et al., 2019; Vaibhav et al., 2019; Khashabi et al., 2020), or in the embedding space (Zhu et al., 2020; Zhao
et al., 2018b; Miyato et al., 2017; Liu et al., 2020).
5.2 Model and Training-based Approaches
Pre-training Recent work has demonstrated pretraining as an effective way to improve NLP models' out-of-distribution robustness (Hendrycks et al., 2020a; Tu et al., 2020), potentially due to its self-supervised objective and the use of large amounts of diverse pre-training data that encourages generalization from a small number of examples that counter the spurious correlations. Tu et al. (2020) showed a few other factors can also contribute to robust accuracy, including larger model size, more fine-tuning data, and longer fine-tuning. A similar observation is made by Taori et al. (2020) in the vision domain, where the authors found training with larger and more diverse datasets offer better robustness consistently in multiple cases, compared to various robustness interventions proposed in the existing literature.
Training with a Better Use of Minority Examples Further, there are several works that propose to robustify the models via a better use of minority examples, e.g., examples that are under-represented in the training distribution, or examples that are harder to learn. For example, Yaghoobzadeh et al. (2021) proposed to first fine-tune the model on the full data, and then on minority examples only.
In general, the training strategy with an emphasis on a subset of samples that are particularly hard for the model to learn is sometimes also referred to as group DRO (Sagawa et al., 2020a), as an extension of vanilla distributional robust optimization (DRO) (Ben-Tal et al., 2013; Duchi et al., 2021). Extensions of DRO are mostly discussing the strategies on how to identify the samples considered as minority: Nam et al. (2020) trained two models in parallel, where the "debiased" model focuses on examples not learned by the "biased" model; Lahoti et al. (2020) used an adversary model to identify samples that are challenging to the main model; Liu et al. (2021) proposed to train the model a second time via up-weighting examples that have high training losses during the first time.
When to Use Data-driven or Model-based Approaches? In many cases both the data and the model can contribute to a model's lack of robustness, hence data-driven and model-based approaches could be combined to further improve a model's robustness. One interesting phenomenon
observed by (Liu et al., 2019) is to attribute models' robustness failures to blind spots in the training data, or the intrinsic learning ability of the model. The authors found that both patterns are possible: in some cases models can be inoculated via being exposed to a small amount of challenging data, similar to the data augmentation approaches mentioned in Section 5.1; on the other hand, some challenging patterns remain difficult which connects to the larger question around generalizability to unseen adversarial and counterfactual patterns (Huang et al., 2020; Jha et al., 2020; Joshi and He, 2021), which is relatively under-explored but deserves much attention.
5.3 Inductive-prior-based Approaches
Another thread is to introduce inductive bias (i.e., to regularize the hypothesis space) to force the model to discard some spurious features. This is closely connected to the human-prior-based identification approaches in Section 4.1 as those human-priors can often be used to re-formulate the training objective with additional regularizers. To achieve this goal, one usually needs to first construct a side component to inform the main model about the misaligned features, and then to regularize the main model according to the side component. The construction of this side component usually relies on prior knowledge of what the misaligned features are. Then, methods can be built accordingly to counter the features such as label-associated keywords (He et al., 2019), label-associated text fragments (Mahabadi et al., 2020), and general easy-to-learn patterns of data (Nam et al., 2020). Similarly, Clark et al. (2019, 2020); Utama et al. (2020a,b) propose to ensemble with a model explicitly capturing bias, where the main model is trained together with this "bias-only" model such that the main model is discouraged from using biases. More recent work (Xiong et al., 2021) shows the ensemble-based approaches can be further improved via better calibrating the bias-only model. Furthermore, additional regularizers have been introduced for robust fine-tuning over pre-trained models, e.g., mutual-information-based regularizers (Wang et al., 2021a) and smoothness-inducing adversarial regularization (Jiang et al., 2020).
In a broader scope, given that one of the main challenges of domain adaptation is to counter the model's tendency in learning domain-specific spurious features (Ganin et al., 2016), some methods
contributing to domain adaption may have also progressed along the line of our interest, e.g., domain adversarial neural network (Ganin et al., 2016). This line of work also inspires a family of methods forcing the model to learn auxiliary-annotation-invariant representations with a side component (Ghifary et al., 2016; Wang et al., 2017; Rozantsev et al., 2018; Motiian et al., 2017; Li et al., 2018; Wang et al., 2019c; Vernikos et al., 2020).
Despite the diverse concrete ideas introduced, the above is mainly training for small empirical loss across different domains or distributions in addition to forcing the model to be invariant to domain-specific spurious features. As an extension along this direction, invariant risk minimization (IRM) (Arjovsky et al., 2019) introduces the idea of invariant predictors across multiple environments, which was later followed and discussed by a variety of extensions (Choe et al., 2020; Ahmed et al., 2020; Rosenfeld et al., 2021). More recently, Dranker et al. (2021) applied IRM in natural language inference and found that a more naturalistic characterization of the problem setup is needed.
5.4 Causal Intervention
Casual analyses have also been utilized to examine robustness. Srivastava et al. (2020) leverage humans' common sense knowledge of causality to augment training examples with a potential unmeasured variable, and propose a DRO-based approach to encourage the model to be robust to distribution shifts over the unmeasured variables. Balashankar et al. (2021) study the effect of secondary attributes, or confounders, and propose context-aware counterfactuals that take into account the impact of secondary attributes to improve models' robustness. Veitch et al. (2021) propose to learn approximately counterfactual invariant predictors dependent on causal structures of the data, and show it can help mitigate spurious correlations in text classification.
5.5 Connections between Mitigations
Connecting these methods conceptually, we conjecture three different mainstream approaches: one is to leverage the large amount of data by taking advantages of pre-trained models, another is to learn invariant representations or predictors across domains or environments, while most of the rest build upon the prior on what the spurious patterns are and encourage the models to not rely on those patterns. Then the solutions are invented through countering model's learning of these patterns by either data
augmentation, reweighting (the minorities), ensemble, inductive-prior design, and causal intervention. Interestingly, statistical work has shown that many of these mitigation methods are optimizing the same robust machine learning generalization error bound (Wang et al., 2021c).
6 Open Questions
In addition to the challenges mentioned above, we list below a few open questions that call for additional research going forward.
Identifying Unknown Robustness Failures Existing identification around robustness failures rely heavily on human priors and error analyses, which usually pre-define a small or limited set of patterns that the model could be vulnerable to. This requires extensive amount of expertise and efforts, and might still suffer from human or subjective biases in the end. How to proactively discover and identify models' unrobust regions automatically and comprehensively remains challenging.
Interpreting and Mitigating Spurious Correlations Interpretability matters for large NLP models, especially key to the robustness and spurious patterns. How can we develop ways to attribute or interpret these vulnerable portions of NLP models and communicate these robustness failures with designers, practitioners, and users? In addition, recent work (Wallace et al., 2019c; Wang et al., 2021d; Zhang et al., 2021) show interpretability methods can be utilized to better understand how a model makes its decision, which in turn can be used to uncover models' bias, diagnose errors, and discover spurious correlations.
Furthermore, the mitigation of spurious correlations often suffers from the trade-off between removing shortcuts and sacrificing model performance (Yang et al., 2020; Zhang et al., 2019a). Additionally, most existing mitigation strategies work in a pipeline fashion where defining and detecting spurious correlations are prerequisites, which might lead to error cascades in this process. How to design end-to-end frameworks for automatic mitigation deserves much attention.
Unified Framework to Evaluate Robustness With a variety of potential spurious patterns in NLP models, it becomes increasingly challenging for developers and practitioners to quickly evaluate the robustness and quality of their models. This calls for more unified benchmarking efforts such as
CheckList (Ribeiro et al., 2020), Reliability Testing (Tan et al., 2021), Robustness Gym (Goel et al., 2021) and Dynabench (Kiela et al., 2021), to facilitate fast and easy evaluation of robustness.
User Centered Measures and Mitigation Instead of passively detecting spurious correlations from a post-processing perspective, how to approach robustness from a user centric perspective needs further investigation. Based on the dual-process models of information processing, humans use two different processing styles (Evans, 2010). One is a quick and automatic style that relies on well-learned information and heuristic cues. The other is a qualitatively different style that is slower, more deliberative, and requires more reflective reasoning. Would these well-learned information and heuristic rules be leveraged to help design better human priors to measure and mitigate spurious correlations? If users or stakeholders are involved in this process, collecting a set of test cases where a system might perform well for the wrong reasons could help design sanity tests.
Connections between Human-like Linguistic Generalization and NLP Generalization Linzen (2020) argue NLP models should behave more like humans to achieve better generalization consistently. It is interesting to note that how humans process information in NLP tasks exactly is still under exploration, and to what extent models should leverage human-knowledge is still a debatable topic.<sup>1</sup> Nonetheless, if we can better understand and utilize the robustness properties in human perception, we can potentially advance models' robustness in a more meaningful way.
In this paper, we provided a unifying overview over robustness definitions, evaluations and mitigation strategies in the NLP domain. We also highlighted open challenges in this area to motivate future research, encouraging people to think deeply about more comprehensive benchmarks, transferability and validity of adversarial examples, unified framework to evaluate and improve robustness, user-centered measures and mitigation, and finally how to potentially achieve human-like linguistic generalization more meaningfully.
The authors would like to thank reviewers for their helpful insights and feedback. This work is funded in part by a grant from Google.
Alberto Abad, Peter Bell, Andrea Carmantini, and Steve Renals. 2020. Cross lingual transfer learning for zero-resource domain adaptation. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6909-6913.
Faruk Ahmed, Yoshua Bengio, Harm van Seijen, and Aaron Courville. 2020. Systematic generalisation with group invariant predictions. In International Conference on Learning Representations.
S. Aït-Mokhtar, J.-P. Chanod, and C. Roux. 2002. Robustness beyond shallowness: Incremental deep parsing. Nat. Lang. Eng., 8(3):121-144.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890-2896, Brussels, Belgium. Association for Computational Linguistics.
Antonios Anastasopoulos, Alison Lui, Toan Q Nguyen, and David Chiang. 2019. Neural machine translation of text from non-native speakers. In Proceedings of NAACL-HLT, pages 3070-3080.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. 2019. Invariant risk minimization. arXiv preprint arXiv:1907.02893.
Ananth Balashankar, Xuezhi Wang, Ben Packer, Nithum Thain, Ed Chi, and Alex Beutel. 2021. Can we improve model robustness through secondary attribute counterfactuals? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020. Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension. Transactions of the Association for Computational Linguistics, 8:662-678.
Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations.
Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Mellenberg, and Gijs Rennen. 2013. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2):341-357.
http://www.incompleteideas.net/ IncIdeas/BitterLesson.html
Su Lin Blodgett, Lisa Green, and Brendan O'Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119-1130, Austin, Texas. Association for Computational Linguistics.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015. Weight uncertainty in neural network. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1613-1622, Lille, France. PMLR.
N. Boucher, I. Shumailov, R. Anderson, and N. Papernot. 2022. Bad characters: Imperceptible nlp attacks. In 2022 2022 IEEE Symposium on Security and Privacy (SP), pages 773-790, Los Alamitos, CA, USA. IEEE Computer Society.
Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial filters of dataset biases. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1078-1088. PMLR.
Jiaao Chen, Dinghan Shen, Weizhu Chen, and Diyi Yang. 2021a. Hiddencut: Simple data augmentation for natural language understanding with better generalizability. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4380-4390.
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mix-Text: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2147-2157, Online. Association for Computational Linguistics.
Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. 2021b. Improving faithfulness in abstractive summarization with contrast candidate generation and selection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5935-5941, Online. Association for Computational Linguistics.
Yangyi Chen, Jin Su, and Wei Wei. 2021c. Multi-granularity textual adversarial attack with behavior cloning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4511-4526, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yo Joong Choe, Jiyeon Ham, and Kyubyong Park. 2020. An empirical study of invariant risk minimization. arXiv preprint arXiv:2004.05007.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4069-4082, Hong Kong, China. Association for Computational Linguistics.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2020. Learning to model and ignore dataset bias with mixed capacity ensembles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 3031-3045.
Holly Craig and Julie Washington. 2002. Oral language expectations for african american preschoolers and kindergartners. American Journal of Speech-Language Pathology, 11:59-70.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnamaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* '19, page 120-128, New York, NY, USA. Association for Computing Machinery.
Dorottya Demszky, Devyani Sharma, Jonathan Clark, Vinodkumar Prabhakaran, and Jacob Eisenstein. 2021. Learning to recognize dialect features. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2315-2338, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Terrance DeVries and Graham W Taylor. 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552.
Kaustubh D. Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, and Samson Tan et al. 2021. Nl-augmenter: A framework for task-sensitive natural language augmentation.