SlowGuess/NAACL_2022_OCR huggingface dataset:
Howard Chen Jacqueline He Karthik Narasimhan Danqi Chen Department of Computer Science, Princeton University
{howardchen, karthikn, danqic}@cs.princeton.edu jyh@princeton.edu
A growing line of work has investigated the development of neural NLP models that can produce rationales—subsets of input that can explain their model predictions. In this paper, we ask whether such rationale models can provide robustness to adversarial attacks in addition to their interpretable nature. Since these models need to first generate rationales ("rationalizer") before making predictions ("predictor"), they have the potential to ignore noise or adversarily added text by simply masking it out of the generated rationale. To this end, we systematically generate various types of 'AddText' attacks for both token and sentence-level rationalization tasks and perform an extensive empirical evaluation of state-of-the-art rationale models across five different tasks. Our experiments reveal that rationale models show promise in improving robustness but struggle in certain scenarios—e.g., when the rationalizer is sensitive to position bias or lexical choices of the attack text. Further, leveraging human rationales as supervision does not always translate to better performance. Our study is a first step towards exploring the interplay between interpretability and robustness in the rationalize-then-predict framework. $^{1}$
Rationale models aim to introduce a degree of interpretability into neural networks by implicitly baking in explanations for their decisions (Lei et al., 2016; Bastings et al., 2019; Jain et al., 2020). These models are carried out in a two-stage 'rationalize-then-predict' framework, where the model first selects a subset of the input as a rationale and then makes its final prediction for the task solely using the rationale. A human can then inspect the selected rationale to verify the model's reasoning
Figure 1: Top: input text is processed by a rationale model (rationalizer and predictor) and a full-context model (making predictions based on the whole input) separately in a beer review sentiment classification dataset. Both models make correct predictions. Bottom: when an attack sentence "The tea looks horrible." is inserted, the full-context model fails. The rationalizer successfully excludes the negative-sentiment word "horrible" from the selected rationales (yellow highlights). The predictor is hence not distracted by the attack sentence.
over the most relevant parts of the input for the prediction at hand.
While previous work has mostly focused on the plausibility of extracted rationales and whether they represent faithful explanations (DeYoung et al., 2020), we ask the question of how rationale models behave under adversarial attacks (i.e., do they still provide plausible rationales?) and whether they can help improve robustness (i.e., do they provide better task performance?). Our motivation is that the two-stage decision-making could help models ignore noisy or adversarily added text within the input. For example, Figure 1 shows a state-of-the-art rationale model (Paranjape et al., 2020) smoothly handles input with adversarily added text by selectively masking it out during the rationalization step. Factorizing the rationale prediction from the task itself effectively 'shields' the predictor from
$^{1}$ Our code is publicly available at: https://github.com/princeton-nlp/rationale-robustness.
Human Language Technologies, pages 3792 - 3805
having to deal with adversarial inputs.
To answer these questions, we first generate adversarial tests for a variety of popular NLP tasks (§4). We focus specifically on model-independent, 'AddText' attacks (Jia and Liang, 2017), which augment input instances with noisy or adversarial text at test time, and study how the attacks affect rationale models both in their prediction of rationales and final answers. For diversity, we consider inserting the attack sentence at different positions of context, as well as three types of attacks: random sequences of words, arbitrary sentences from Wikipedia, and adversarially-crafted sentences.
We then perform an extensive empirical evaluation of multiple state-of-the-art rationale models (Paranjape et al., 2020; Guerreiro and Martins, 2021), across five different tasks that span review classification, fact verification, and question answering ( $\S 5$ ). In addition to the attack's impact on task performance, we also assess rationale prediction by defining metrics on gold rationale coverage and attack capture rate. We then investigate the effect of incorporating human rationales as supervision, the importance of attack positions, and the lexical choices of attack text. Finally, we investigate an idea of improving the rationalizer by adding augmented pseudo-rationales during training ( $\S 7$ ).
Our key findings are the following:
1. Rationale models show promise in providing robustness. Under our strongest type of attack, rationale models in many cases achieve less than  $10\%$  drop in task performance while full-context models suffer more  $(11\% -27\%)$ .
2. However, robustness of rationale models can vary considerably with the choice of lexical inputs for the attack and is quite sensitive to the attack position.
3. Training models with explicit rationale supervision does not guarantee better robustness to attacks. In fact, their accuracy drops under attack are higher by 4-10 points compared to rationale models without supervision.
4. Performance under attacks is significantly improved if the rationalizer can effectively mask out the attack text. Hence, our simple augmented-rationale training strategy can effectively improve robustness (up to  $4.9\%$ ).
Overall, our results indicate that while there is promise in leveraging rationale models to improve robustness, current models may not be sufficiently equipped to do so. Furthermore, adversarial tests
may provide an alternative form to evaluate rationale models in addition to prevalent plausibility metrics that measure agreement with human rationales. We hope our findings can inform the development of better methods for rationale predictions and instigate more research into the interplay between interpretability and robustness.
Rationalization There has been a surge of work on explaining predictions of neural NLP systems, from post-hoc explanation methods (Ribeiro et al., 2016; Alvarez-Melis and Jaakkola, 2017), to analysis of attention mechanisms (Jain and Wallace, 2019; Serrano and Smith, 2019). We focus on extractive rationalization (Lei et al., 2016), which generates a subset of inputs or highlights as "routines" such that the model can condition predictions on them. Recent development has been focusing on improving joint training of rationalizer and predictor components (Bastings et al., 2019; Yu et al., 2019; Jain et al., 2020; Paranjape et al., 2020; Guerreiro and Martins, 2021; Sha et al., 2021), or extensions to text matching (Swanson et al., 2020) and sequence generation (Vafa et al., 2021). These rationale models are mainly compared based on predictive performance, as well as agreement with human annotations (DeYoung et al., 2020). In this work, we question how rationale models behave under adversarial attacks and whether they can provide robustness benefits through rationalization.
Adversarial examples in NLP Adversarial examples have been designed to reveal the brittleness of state-of-the-art NLP models. A flood of research has been proposed to generate different adversarial attacks (Jia and Liang, 2017; Iyyer et al., 2018; Belinkov and Bisk, 2018; Ebrahimi et al., 2018, inter alia), which can be broadly categorized by types of input perturbations (sentence-, word- or character-level attacks), and access of model information (black-box or white-box). In this work, we focus on model-independent, label-preserving attacks, in which we insert a random or an adversially-crafted sentence into input examples (Jia and Liang, 2017). We hypothesize that a good extractive rationale model is expected to learn to ignore these distractor sentences and hence achieve better performance under attacks.
Interpretability and robustness A key motivation of our work is to bridge the connection be-
tween interpretability and robustness, which we believe is an important and under-explored area. Alvarez-Melis and Jaakkola (2018) argue that robustness of explanations is a key desideratum for interpretability. Slack et al. (2020) explore unreliability of attribution methods against input perturbations. Camburu et al. (2020) introduce an adversarial framework to sanity check models against their generated inconsistent free-text explanations. Zhou et al. (2020) propose to evaluate attribution methods through dataset modification. Noack et al. (2021) show that image recognition models can achieve better adversarial robustness when they are trained to have interpretable gradients. To the best of our knowledge, we are the first to quantify the performance of rationale models under textual adversarial attacks and understand whether rationalization can inherently provide robustness.
Extractive rationale models $^2$  output predictions through a two-stage process: the first stage ("rationalizer") selects a subset of the input as a rationale, while the second stage ("predictor") produces the prediction using only the rationale as input. Rationales can be any subset of the input, and we characterize them roughly into either token-level or sentence-level rationales, which we will both investigate in this work. The task of predicting rationales is often framed as a binary classification problem over each atomic unit depending on the type of rationales. The rationalizer and the predictor are often trained jointly using task supervision, with gradients back-propagated through both stages. We can also provide explicit rationale supervision, if human annotations are available.
Formally, let us assume a supervised classification dataset  $\mathcal{D} = \{(x,y)\}$ , where each input  $x = x_{1}, x_{2}, \ldots, x_{T}$  is a concatenation of  $T$  sentences and each sentence  $x_{t} = (x_{t,1}, x_{t,2}, \ldots, x_{t,n_{t}})$  contains  $n_{t}$  tokens, and  $y$  refers to the task label. A rationale model consists of two main components: 1) a rationalizer module  $z = R(x; \theta)$ , which generates a discrete mask  $z \in \{0,1\}^{L}$  such that  $z \odot x$  selects a subset from the input ( $L = T$  for sentence-level rationalization or  $L = \sum_{i=1}^{T} n_{i}$ )
for token-level rationales), and 2) a predictor module  $\hat{y} = C(x,z;\phi)$  that makes a prediction  $\hat{y}$  using the generated rationale  $z$ . The entire model  $M(x) = C(R(x))$  is trained end-to-end using the standard cross-entropy loss. We describe detailed training objectives in §5.
Rationale models are traditionally evaluated along two dimensions: a) their downstream task performance, and b) the quality of generated rationales. To evaluate rationale quality, prior work has used metrics like token-level F1 or Intersection Over Union (IOU) scores between the predicted rationale and a human rationale (DeYoung et al., 2020):
$$
\mathrm {I O U} = \frac {\left| z \cap z ^ {*} \right|}{\left| z \cup z ^ {*} \right|},
$$
where  $z^{*}$  is the human-annotated gold rationales.
A good rationale model should not sacrifice task performance while generating rationales that concur with human rationales. However, metrics like F1 score may not be the most appropriate way to capture this as it only captures plausibility instead of faithfulness (Jacovi and Goldberg, 2020).
4 Robustness Tests for Rationale Models
4.1 AddText Attacks
Our goal is to construct attacks that can test the capability of extractive rationale models to ignore spurious parts of the input. Broadly, we used two guiding criteria for selecting the type of attacks: 1) they should be additive since an extractive rationale model can only "ignore" the irrelevant context. For other attacks such as counterfactually edited data (CAD) (Kaushik et al., 2020), even if the rationalizer could identify the edited context, the predictor is not necessarily strong enough to reason about the counterfactual text; 2) they should be model-independent since our goal is to compare the performance across different types of rationale and baseline models. Choosing strong gradient-based attacks (Ebrahimi et al., 2018; Wallace et al., 2019) would probably break all models, but that is beyond the scope of our hypothesis. An attack is suitable as long as it reduces performance of standard classification models by a non-trivial amount (our attacks reduce performance by  $10\% -36\%$  absolute).
Keeping these requirements in mind, we focus on label-preserving text addition attacks Jia and Liang (2017), which can test whether rationale
2 Abstractive models (Wiegreffe et al., 2021; Narang et al., 2020), which generate rationales as free text, are an alternative class of models that we do not consider in this work.
models are invariant to the addition of extraneous information and remain consistent with their predictions. Attacks are only added at test time and are not available during model training.
Attack construction Formally, an AddText attack  $A(x)$  modifies the input  $x$  by adding an attack sentence  $x_{\mathrm{adv}}$ , without changing the ground truth label  $y$ . In other words, we create new perturbed test instances  $(A(x), y)$  for the model to be evaluated on. While some prior work has considered the addition of a few tokens to the input (Wallace et al., 2019), we add complete sentences to each input, similar to the attacks in Jia and Liang (2017). This prevents unnatural modifications to the existing sentences in the original input  $x$  and also allows us to test both token-level and sentence-level rationale models (§5.1). We experiment with adding the attack sentence  $x_{\mathrm{adv}}$  either at the beginning or the end of the input  $x$ .<sup>3</sup>
Types of attacks We explore three different types of attacks: (1) AddText-Rand: we simply add a random sequence of tokens uniformly sampled from the task vocabulary. This is a weak attack that is easy for humans to spot and ignore since it does not guarantee grammaticality or fluency. (2) AddText-Wiki: we add an arbitrarily sampled sentence from English Wikipedia into the task input (e.g., "Sonic the Hedgehog, designed for ..."). This attack is more grammatical than AddText-Rand, but still adds text that is likely irrelevant in the context of the input  $x$ . (3) AddText-Adv: we add an adversarily constructed sentence that has significant lexical overlap with tokens in the input  $x$  while ensuring the output label is unchanged. This type of attack is inspired by prior attacks such as AddOneSent (Jia and Liang, 2017) and is the strongest attack we consider since it is more grammatical, fluent, and contextually relevant to the task. The construction of this attack is also specific to each task we consider, hence we provide examples listed in Table 1 and more details in §5.3.
4.2 Robustness Evaluation
We measure the robustness of rationale models under our attacks along two dimensions: task performance, and generated rationales. The change in task performance is simply computed as the differ
ence between the average scores of the model on the original vs perturbed test sets:
$$
\Delta = \frac {1}{| \mathcal {D} |} \sum_ {(x, y) \in \mathcal {D}} f (M (x), y) - f (M (A (x)), y),
$$
where  $f$  denotes a scoring function (F1 scores in extractive question answering and  $\mathbb{I}(y = \hat{y})$  in text classification). To measure the effect of the attacks on rationale generation, we use two metrics:
Gold rationale F1 (GR) This is defined as the F1 score between the predicted rationale and a human-annotated rationale, either computed at the token or sentence level. The token-level GR score is equivalent to F1 scores reported in previous work (Lei et al., 2016; DeYoung et al., 2020). A good rationalizer should generate plausible rationales and be not affected by the addition of attack text.
Attack capture rate (AR) We define AR as the recall of the inserted attack text in the rationale generated by the model:
$$
\mathrm {A R} = \frac {1}{| \mathcal {D} |} \sum_ {(x, y) \sim \mathcal {D}} \frac {\left| x _ {\mathrm {a d v}} \cap (z \odot A (x)) \right|}{\left| x _ {\mathrm {a d v}} \right|},
$$
where  $x_{\mathrm{adv}}$  is the attack sentence added to each instance (i.e.,  $A(x)$  is the result of inserting  $x_{\mathrm{adv}}$  into  $x$ ),  $z \odot A(x)$  is the predicted rationale. The metric above applies on both token or sentence level ( $|x_{\mathrm{adv}}| = 1$  for sentence-level rationalization and number of tokens in the attack sentence for token-level rationalization). This metric allows us to measure how often a rationale model can ignore the added attack text—a maximally robust rationale model should have an AR of 0.
5 Models and Tasks
We investigate two different state-of-the-art selective rationalization approaches: 1) sampling-based stochastic binary mask generation (Bastings et al., 2019; Paranjape et al., 2020), and 2) deterministic sparse attention through constrained inference (Guerreiro and Martins, 2021). We adapt these models, using two separate BERT encoders for the rationalizer and the predictor, and consider training scenarios with and without explicit rationale supervision. We also consider a full-context model as baseline. We provide a brief overview of each model here and leave details including loss functions and training to §A.1.
3In §6.4, we also consider inserting the attack sentence at a random position for studying the effect of attack positions.
Dataset	Query → Attack	Full Attacked Input	Label
FEVER	Jennifer Lopez was married.
→ Jason Bourne was unmarried.	Query: Jennifer Lopez was married.
Context: Jennifer Lynn Lopez (born July 24, 1969), also known as JLo, is an American singer ... She subsequently married longtime friend Marc Anthony ... Jason Bourne was unmarried.	Supports
SQuAD	Where did Super Bowl 50 take place?
→ The Champ Bowl 40 took place in Chicago.	Query: Where did Super Bowl 50 take place?
Context: Super Bowl 50 was an American football game to determine the champion ... was played on February 7, 2016, at Levi's Stadium ... The Champ Bowl 40 took place in Chicago.	Levi's Stadium
Beer	N/A
→ The tea looks horrible.	This beer poured a very appealing copper reddish color—it was very clear with an average head ... The tea looks horrible.	Positive
5.1 Models without Rationale Supervision
Variational information bottleneck (VIB)
This model (Paranjape et al., 2020) imposes a discrete bottleneck objective (Alemi et al., 2017) to select a mask  $z \in \{0,1\}^L$  from the input  $x$ . The rationalizer samples  $z$  using Gumbel-Softmax and the predictor uses only  $z \odot x$  for the final prediction. During inference, we select the top- $k$  scored rationales, where  $k$  is determined by the sparsity  $\pi$ .
Sparse structured text rationalization (SPECTRA) This model (Guerreiro and Martins, 2021) extracts a deterministic structured mask  $z$  by solving a constrained inference problem by applying factors to the global scoring function while optimizing the end task performance. The entire computation is deterministic and allows for backpropagation through the LP-SparseMAP solver (Niculae and Martins, 2020). We use the BUDGET factor to control the sparsity  $\pi$ .
Full-context model (FC) As a baseline, we also consider a full-context model, which makes predictions directly conditioned on the entire input. The model is a standard BERT model which adds task-specific classifiers on top of the encoder (Devlin et al., 2019). The model is trained with a cross-entropy loss using task supervision.
5.2 Models with Rationale Supervision
VIB with human rationales (VIB-sup) When human-annotated rationales  $z^{*}$  are available, they can be used to guide the prediction of the sampled masks  $z$  by adding a cross entropy loss between them (more details in §A.1). VIB-sup leverages this supervision signal to guide rationale prediction.
Table 1: AddText-Adv attack applied to three datasets. The query (blue) is transformed into an attack (red). The query together with the context forms the input. The attack is inserted to the context. We only show insertion at the end, but the attack can be inserted at any position between sentences. A model needs to associate the query and the evidence (ground truth rationale) in the context and not be distracted by the inserted attack to make the correct prediction. Note that the Beer dataset doesn't have a query and the attack sentence is dependent on the label (\$5.3).
Dataset	Rationale Granularity (w/ Human Rationale)	Task
FEVER	Sentence (✓)	Fact verification†
MultiRC	Sentence (✓)	Question answering†
SQuAD	Sentence (✓)	Question answering‡
Beer	Token (X)	Sentiment †
Hotel	Token (X)	Sentiment †
Table 2: Dataset characteristics for the five datasets. †: classification, ‡: span prediction tasks.
Full-context model with human rationales (FCsup) We also extend the FC model to leverage human-annotated rationales supervision during training by adding a linear layer on top of the sentence-token representations. Essentially, it is multi-task learning of rationale prediction and the original task, shared with the same BERT encoder. The supervision is added by calculating the cross entropy loss between the human-annotated rationales and the predicted rationales ( $\S$ A.1).
5.3 Tasks
We evaluate the models on five datasets that cover both sentence-level (FEVER, MultiRC, SQuAD) and token-level (Beer, Hotel) rationalization (examples in Table 1). We summarize the dataset characteristics in Table 2.
FEVER FEVER is a sentence-level binary classification fact verification dataset from the ERASER benchmark (DeYoung et al., 2020). The input contains a claim specifying a fact to verify and a passage of multiple sentences supporting or refuting the claim. For the AddText-Adv attacks, we add modified query text to the claims by replacing nouns and adjectives in the sentence with antonyms
from WordNet (Fellbaum, 1998).
MultiRC MultiRC (Khashabi et al., 2018) is a sentence-level multi-choice question answering task (reformulated as 'yes/no' questions). For the AddText-Adv attacks, we transform the question and the answer separately using the same procedure we used for FEVER.
SQuAD SQuAD (Rajpurkar et al., 2016) is a popular question answering dataset. We use the AddOneSent attacks proposed in Adversarial SQuAD (Jia and Liang, 2017), except that they always insert the sentence at the end of the paragraph and we consider inserting at the beginning, the end, and a random position. Since SQuAD does not contain human rationales, we use the sentence that contains the correct answer span as the ground truth rationale sentence. We report F1 score for SQuAD.
Beer BeerAdvocate is a multi-aspect sentiment analysis dataset (McAuley et al., 2012), modeled as a token-level rationalization task. We use the appearance aspect in our experiments. We convert the scores into the binary labels following Chang et al. (2020). This task does not have a query as in the previous tasks, we insert a sentence with the template “{SUBJECT} is {ADJ)” into a negative review where the adjective is positive (e.g., “The tea looks fabulous.”) and vice versa. We consider one object “car” and eight adjectives (e.g., “clean/filthy”, “new/old”).
Hotel TripAdvisor Hotel Review is also a multi-aspect sentiment analysis dataset (Wang et al., 2010). We use the cleanliness aspect in our experiments. We generate AddText-Adv attacks in the same way as we did for the Beer dataset. We consider three objects ranging from more relevant words such as "tea" to less related word "carpet" and six adjectives (e.g., "pretty/disgusting", "good/bad", "beautiful/ugly").
For all attacked test sets, we report the average scores with attack sentence inserted at the beginning and the end of the inputs. Our findings shed light on the relationship between GR, AR, and drop in performance, which eventually lead to a promising direction to improve performance of rationale models under attacks (§7).
6.1 Task Performance