Fooling Explanations in Text Classifiers
Authors: Adam Ivankay
Ivan Girardi
Chiara Marchiori
Pascal Frossard
Venue: ICLR 2022 Poster (2022)
Keywords: robustness
explainability
text classification
natural language processing

Abstract: State-of-the-art text classification models are becoming increasingly reliant on deep neural networks (DNNs). Due to their black-box nature, faithful and robust explanation methods need to accompany classifiers for deployment in real-life scenarios. However, it has been shown that explanation methods in vision applications are susceptible to local, imperceptible perturbations that can significantly alter the explanations without changing the predicted classes. We show here that the existence of such perturbations extends to text classifiers as well. Specifically, we introduce TextExplanationFooler (TEF), a novel explanation attack algorithm that alters text input samples imperceptibly so that the outcome of widely-used explanation methods changes considerably while leaving classifier predictions unchanged. We evaluate the attribution robustness estimation performance of TEF on five text classification datasets, utilizing three DNN architectures and a transformer architecture for each dataset. By significantly decreasing the correlation between unchanged and perturbed input attributions, we show that all models and explanation methods are susceptible to TEF perturbations. Moreover, we evaluate how the perturbations transfer to other model architectures and attribution methods, finding better than random performance in scenarios where the exact attacked model and explanation method are unknown. Finally, we introduce a semi-universal attack that is able to compute fast, computationally light perturbations with no knowledge of the attacked classifier nor explanation method. Overall, our work shows that explanations in text classifiers are fragile and users need to carefully address their robustness before relying on them in critical applications.

Published as a conference paper at ICLR 2022
FOOLING EXPLANATIONS IN TEXT CLASSIFIERS
Adam Ivankay
Ivan Girardi
Chiara Marchiori
IBM Research Zurich Rüschlikon, Switzerland aivezurich. ibm. com
IBM Research Zurich Rüschlikon, Switzerland i ivgê zurich. ibm. com
IBM Research Zurich
Rüschlikon, Switzerland chi chi@zurich ibm. com
Pascal Frossard
École Polytechnique Fédérale de Lausanne (EPFL)
Lausanne, Switzerland
pascal. frossardeepfl.co
ABSTRACT
State-of-the-art text classification models are becoming increasingly reliant on deep neural networks (DNNs). Due to their black-box nature, faithful and ro- bust explanation methods need to accompany classifiers for deployment in real- life scenarios. However, it has been shown in vision applications that explanation methods are susceptible local, imperceptible perturbations that can significantly alter the explanations without changing the predicted classes. We show here that the existence of such perturbations extends to text classifiers well. Specifi- cally, we introduce TEXTEXPLANATION FOOLER (TEF), a novel explanation at- tack algorithm that alters text input samples imperceptibly widely-used explanation methods changes considerably while leaving classifier predictions unchanged. We evaluate the performance of the attribution robustness estimation performance in TEF on five sequence classification datasets, utilizing three DNN architectures and three transformer architectures for each dataset. TEF can significantly decrease the correlation between unchanged and perturbed input attributions, which shows that all models and explanation methods are susceptible to TEF perturbations. Moreover, we evaluate how the perturbations transfer to other model architectures and attribution methods, and show that TEF perturba- tions are also effective in scenarios where the target model and explanation method are unknown. Finally, we introduce a semi-universal attack that is able to com- pute fast, computationally light perturbations with no knowledge of the attacked classifier nor explanation method. Overall, our work shows that explanations text classifiers are very fragile and users need to carefully address their robustness before relying on them in critical applications.
that the outcome of
INTRODUCTION
Deep neural networks (DNNs) have undoubtedly become the state-of-the-art architectures for many existing machine learning tasks (Choi al. 2016). Yet, their black-box nature has raised the need for developing methods to mitigate the lack of interpretability caused by their increased complexity (Simonyan al. 2013 Zeiler Fergus 2014 Hendricks et al. 2018 Bahdanau et al. 2014). These methods give intuitive, casily understandable explanations that do not require signiccat domain knowledge. This is especially desired in safety-critical scenarios, such as healthcare, where the users of such DNNs - the medical professionals for instance need to understand the decision process and reasoning behind it. However, they have been shown to lack local robustness towards carefully crafted, imperceptible perturbations in the input (Ghorbani et al.) 2019). While resulting in the same predictions, these altered inputs yield significantly different explanations and attributions maps (Figure []. Interpretation methods fragile towards small input perturbations not only fail provide faithful explanations, a desiderata commonly required explainable AI (Jacovi Goldberg 2020). but also damages user trust in DNNs and prevents them from being deployed on high-stakes, safety-

---

Published as a conference paper at ICLR 2022
Original sample
TEF sample
PCC
[CLS] romanians pitch rumsfeld on base location - mihail kogalniceanu air base romania to entice the us military to make a home here, what better symbolic appeal could the romanian gov- ermment make than to rename street here quot:george washington boulevard ? [SEP]
[CLS] romanians pitch clinton base places - mihail kogalniceanu air base rumania to entice the military make a home here what better symbolic appeal could the romanian government make than to rename a street here quot:george washington boulevard [SEP]
-0.07
F(s,, World) = 0.99
F(s, World) = 0.97
[CLS] forgettable horror more gory than psychological - with a highly satisfy- ing quotient of friday night excitement and milla power. [SEP]
[CLS] forgettable horror more gory than psychological with a highly satisfying quotient of friday night arousal and milla wattage [SEP]
0.18
F(s, Pos.) 0.99
F(s, Pos.) 0.99
Figure I: Example of fragile attributions. Highlighted red words are deemed most important towards the predicted class by the Integrated Gradients attribution method, blue ones against it. By substicuting few words the original sample, the Pearson Correlation Coefficient (PCC) of word importances drops below 0.2 while maintaining the same confidence in correctly predicted class (denoted by F).
critical applications, such as in healthcare (Addi & Berrada 2020)
The previously described phenomenon has been widely studied in the image domain by Etmann
et al. (2019); Moosavi-Dezfool et al. (2019b) or Ivankay et al. (202)) However, in discrete-inpul
domains like text, there has been limited progress. This especially problematic given the increased
reliance on and fragility of attention mechanisms as inherently explainable methods, as stated in
Ghaeini al. (018). Therefore, we summarize our contributions as follows:
We provide a novel baseline black-box adversarial attack, TEXTEXXPLANTTIINFOOO..R (TEF) to estimate the local robustness of explanations text classification problems
We evaluate attribution robustness on widely used, sute-of-the-ar text datasets and model architectures, showing that explanation methods' output can be significantly altered with our attack
We provide insight into the transfer capability of TEF on different models and explanation methods as well as introduce semi-universal adversarial perturbations to alter explanations without requiring access the model attack-time
PRELIMINARIES
2.1 RELATED WORK
Adversarial attacks that alter the inference outcomes in DNNs have been widely studied both in the
image (Goodfellow et al. 2014 Carlini & Wagner 2017 Moosavi-Deztooli et al. 2016 Modas
et al. 2019) and text domain (Ebrahimi et al. 2017 Sun et al. 2020 Jin et al. 2019 Yang et al.) 2020 Methods alleviate the networks susceptibility to such attacks have also been proposed,
including the works of Madry et al. (2017); Moosavi-Deztooll et al. (2019a); Buckman et al. (2018) or Cisse al. (017). However. has recently been shown by authors Ghorbani al. (2019 that, in addition to DNN predictions, widely-used explanation methods also lack robustness to targeted, im- perceptible alterations of the input. These attacks change the outcomes of such explanation methods significantly, while predictions of the DNNs are unaltered. This violates the Prediction Assumption of faithful explanations and crucially degrades user trust such explanation methods, as signif- icantly different interpretations provided for similar inputs and outputs (Jacovi & Goldberg, 2020). The aforementioned phenomenon of fragile explanations has mostly been investigated in the image domain (Etmann al. 2019 Singh al. 2019 Chen et al. 2019 [vankay et al.) (020), with less focus on discrete input spaces like text. However, faithful and robust interpretations are arguably equally important in the discrete text domain, for instance in electronic health record classification
2

---

Published as a conference paper at ICLR 2022
(Girardi et al. 2018) or precision medicine (Binder et al. 2021). where critical decisions often need to be based on DNN explanations. The work of La Malfa et al. (2021) constructs inherently robust explanations for NLP models, however only towards perturbations in the embedding space, not the input space that adversaries can operate on. Moreover, they do not give a method to evaluate ro- bustness of already existing explanation algorithms. The authors of [Fnng al. (2018) show that interpretation methods in NLP lack completeness (Sundararajan et al., 2017) by removing words deemed least important by explanation methods. Moreover, attention mechanisms (Bahdanau al. 2014) have been increasingly relied on as inherently interpretable systems. However recent work has questioned their faithfulness and plausibility (Serrano & Smith 2019: Dain Wallace 2019 Wiegr- effe & Pinter 2019) by proving that they often do not highlight input components that are most important to DNN decision. We, paralleled by the very recent work [Sinha al. (202T), are the first to show that imperceptible perturbations in the inpul space can alter the outcome of expla- nations of text classifiers significantly, giving an efficient attack to estimate explamation robustness. However, our work is the first give extensive evaluation of the robustness of widely-used ex- planation methods on large datasets, comparing several state-of-the-an architectures. Further, we the first to address robustness of attention weights in transformer architectures and to provide insight on transfer capabilities and universal attacks.
2.2 BACKGROUND
Let ($1, $2.....888 SN) be a dataset of text samples 8,, each with label from a predefined set of labels L = (2. }- Each sample 8; contains a sequence of tokens (or words) w; taken from discrete vocabulary set W = {ww. u2..... w J. A generic sequence classifier then consist of non-injective, non-surjective embedding function E S Rdxp E(s) = X, which maps the input sample its embedding matrix X, and a function f Rx + R f(X) = o, representing a (DNN) classifier function. d and p denote the embedding dimension and sequence length respectively. Let F S + R! F(s) f E be the full sequence classifier with final prediction y = arg m [1:] [1:| Os.
We define an attribution map as S 3-RR R", F,s, = a that assigns a scalar value to each input token w; in the text sample s, resulting in the attribution vector Rº. This vector represents each token's 's influence towards the prediction outcome y of classifier F. Our work considers three widely-used attribution methods in text classification, namely Saliency Maps (S) (Simonyan al.
2013).
Integrated Gradients (IG) (Sundararaaan et al. 2017) and Attention (A) (Bahdanau et al. defined in the following Equations L. and
201
respectively.
Ff,s,F,1)
Σ jE(2:d}
Vxf(X)ll
(1)
1 (s,FF
Σ [x- B). sE(1:d)
Jazo V x f(x) X da)
(2)
A (s, Eje(1n) expej
expe. es
where B denotes the null matrix 0x f the classifier function of F, a a scaling factor and X Es)) f denotes the -atrix-derivative of f X, as defined in Goodfellow et al. (016 attention head is layer that transforms its inputs into scores e and calculates its output by linear combination of each input score, with coefficients normalized to distribution. These coefficients are the attention weights A, (s, F. () denoted in Equation L It is commonly agreed to give intu- itive explanations on how much the model attends to the given inputs through its attention weights
Jacovi & Goldberg 2020
3 METHODS
In this section, we describe our novel method TEXTEOPAAN (TEF) to estimate at- tribution robustness (AR) in sequence classification problems. Specifically, we define the problem formulation, introduce our threat model and present the algorithm used to alter explanations by imperceptibly changing the inputs.
3

---

Published as a conference paper at ICLR 2022
3.1 PROBLEM FORMULATION
Given an input text samples 8 and 8, labels l; text classifier F with embedding function E and
classifier function f:: and attribution method A, we define attribution robustness (also explanation robustness, AR) as written in Equation 4
r@ā, a) 1 max d( a) =
d[A(s, A(s,
(4)
with
arg max F(5)
arg max F(s). (e[1:/11}
(5)
(E
where d denotes a distance measure between the attributions à and a of the the two input samples and s. The rest of the notation is kept as in Section 2 Equation + quantifies how different the
attributions two input samples are, given the constrant in Equation ( that enforces the inputs having the same prediction outcome. The attribution robustness estimation is then solved utilizing the following Equation
addv arg max d[A(s, F,1), A(s,F.)
(6)
where Sadv denotes the solution the estimation, i.e. the adversarial input, which also minimizes AR defined in Equation optimized during estimation. The solution addv gives a robustness estimate by finding an input that maximizes the distance between original attribution A(s, F, 1) and adversarial attribution A F, within a local neighbourhood of s. The more dissimilar these maps are, the less robust the attribution method is. The local neighbourhood is defined by both linguistic constraints described in the next section that encourage semantic proximity to the original text and the perturbed samples having the
s denotes the original, unperturbed input and & the perturbed input,
same prediction outcome as the unperturbed ones, Equation ( This formulation is backed
by current research (Ghorbani et al. 2019 Dombrowski et al. 2019 Prediction Assumption of faithful explanations (Jacovi & Goldberg 2020).
Ivankay et al. 2020) and the
3.2 THREAT MODEL AND ATTACK
We define our algorithm to estimate AR as a black-box attack. only queries the model to obtain its output logits and the accompanied explanations of the inference process. The model might access its gra- dients to compute explanations, but the at- tack only utilizes the resulting explana- tions, no gradient or architectural infor- mation. We restrict the valid input per- turbations to token substitutions, specifi- cally insertions and deletions of tokens are forbidden, as they alter the input lengths. Algorithm M contains the schematic code for TEF, consisting of the following two steps.
Algorithm 1 TextExplamstiinooo... (TEF)
Input: Input sentence with predicted class I, classi- fier F, attribution A, attribution distance d, number of synonyms N, maximum perturbation ratio pmaz Output: Adversarial sentence 8adv
1: Sadv + 8, dmaa + 0, ++t0
2: for w; do
3: Tw = d[A(ss::-))) +0. F,I), A(8,
for wi E (Wi.... W|s ) Iamm- 2 Imm Vm
{2, ... | do
5:
if wy E Sotopuorda then continue
7:
C + SyponymEmbddhngsiiyyy.
8:
C + POSFilter Cj.
9:
for ck E C2 do
Step Word importance ranking
10:
$ - + Replace wy in sadv with Ck
First, an importance ranking is ex- tracted for each token of the input sam- plc. Specifically, we compute Iw. = d[A(su,-o- F,l), A(s, F,1)] for each to- ken in where -w denotes the input sequence 8 with the i-th word masked to the zero embedding token. The input to- kens are then sorted by the Iw; values 16:
11:
if arg max F +c, = then
ie (1:1/]]
12:
d d[A(s (5 tou F,1), A(s, F,D)
13:
if d dmaz then 8adv 8wi dmar +
14:
15:
+ r + +1
decreasing fashion. Then, high impor-
17:
if p
+ Pina then
tance words are prioritized during substi- tution.
18:
break
4

---

Published as a conference paper at ICLR 2022
Table 1: Accuracies, average text length and number of classes of our models trained on the five datasets.
DATASET
CNN
LSTM
LSTMATT
BERT
ROBERTA
XLNET
Mean [s]
L
AG'S NEWS
89.7%
90.8%
91.4%
94.2%
94.0%
93.8%
45
IMDB
82.0%
87.2%
87.3%
89.4%
93.3%
93.7%
270
2
FAKE NEWS
98.9%
99.6%
99.6%
99.8%
100.0%
100.0%
919
2
MR 73.0%
76.4%
78.0%
82.2%
87.7%
86.3%
22
2
YELP 49.0%
54.8%
60.0%
62.6%
67.6%
159
5
Step 2 Candidate selection
For each word w; in sequentially, a of substitution candidates C of N elements is extracted. This candidate set is constructed from the counter-fitte GloVe (Pen- nington et al. 2014) synonym embeddings by the authors Mrkšić al. (2016). The candidates are then filtered by Part-Of- Of-Speech (POS). tagged by SpaCy (Honnibal et al., 2020 only allowing re- placements with equal POS. Stop words are also discarded from C. A final selection as replacement
for wy is then made to be the cs E C that maximizes dl A( F,1), A(s, F,1)] The algorithm
aborted when the number replacements to sentence length exceeds the maximum value Pmax-
EXPERIMENTS AND RESULTS
In this section, we present an extensive evaluation of our attribution robustness (AR) estimation at- tack, TEF, for sequence classification problems. We examine the performance of TEF and study the impact of different factors on its robustness evaluation performance. We find that our attack effectively reduces the correlation of original and attacked attributions all datasets and models. Moreover, we describe our transfer and semi-universal attacks and examine their robustness esti- mation performance, showing that even under cireumstances where the model and explainer unknown to the attacker, TEF perturbations transferred from other models decrease attribution ro- bustness effectively.
4.1 MODELS, DATASETS AND EVALUATION
Our TEF attack is evaluated on five commonly used public sequence classification datasets, AG's News Zhang et al. 2015). MR reviews (Zhang et al. 2015), IMDB Movie Reviews (Maas et al. 2011 Fake News Dataset and Yelp (Asgaar 2016 We train six different word embedding-basee architectures for each dataset, namely CNN, an LSTM, LSTM containing single attention layer with one head (LSTMAtt) and three state-of-the -ftt----rtttttttt finetuned transformer- bassed architectures, BERT (Li al. 2020). RoBERTa (Liu et al. 2019) and XLNet (Yang et al. 2019.. Table D con- tains a summary of our model performances as well as details on the datasets. The text samples tokenized with the default English SpaCy (Honnibal al. 2020 tokenizer for the CNN, LSTM and LSTMAtt models and embedded with the pretrained Glo Ve 6B 300-dimensional word vectors Pennington al., 2014). The transformer-bast
models use their own pretrained tokenizers and embeddings. We use PyTorch (Paszke et al. 2019) with Captum (Kokhlikyan et al.] (0200) to im- plement our models and explainers and the Huggingface Transformers library (Wolf et al. 2020 to finetune the transformer architectures on datasets.
We evaluate the robustness of three commonly used explanation methods in natural language pro- cessing with our TEF attack. These are Saliency Maps (S), Integrated Gradients (IG) and the At- tention mechanism (A), defined in Section We use S and IG in combination with all our archi- tectures. Attention only with LSTMAtt, BERT, RoBERTa and XLNet. During the attack, set the
attribution distance d of Equation (6 to be d(ā a) = PCC(â, a) + with PCC denoting the
2
Pearson Correlation Coefficient (Pearson 1895) of original and adversarial attributions and a. We then report the standard Pearson Correlation Coefficient (PCC), Kendall's 's Rank Order Correlation (ROC) (Kendall 1938), Spearman's Correlation Coefficient (SCC) (Myers & Sirois 2004) and the Top-10%, 70p-30% and Top-50% intersections to measure AR in Equation 4.. These are common
5

---

Published as a conference paper at ICLR 2022
LSTMAtt Attention (A) on IMDB
3.3
ACC: 0.23
ACC: 0.03
0.16
0.16
0.16
BERT Integrated Gradients (IG) on Yelp
7.4
ACC:
0.16
0.16
p
0.16
RoBERTa Saliency Maps (S) MR
ACC:
ACC: 0.04
0.16
0.16
P
0.16
XLNNt Attention (A) Fake News
6.3
ACC: 0.21
ACC: 0.11
0.16
0.16
0.16
TEF
TEF
RA
PCC
o-
Top-10%
TEF
RA
ROC
Top-30%
PPI. inc.
SCC
Top-50%
Sem. sim.
Figure 2: Robustness of attribution maps on several architectures and explainers. We plot the average relations (PCC, ROC, SCC) (left), Top-10%, Top-30% and Top-50% intersections (middle). semantic similarity and increase of average perplexity (right) as functions the perturbed ratio Dashed lines indicate the metrics for our RANDOMATTACY (RA). The ACC indicates the area under the PCC - lower values correspond to overall lower feature attribution correlations in the overall operation interval of p. The perplexity increases are indicated right axis, other metrics on the left.
curve,
metrics that correspond to human measures of AR (Ghorbani et al. 2019 Dombrowski et al. 2019 Ivankay al. 2020p. Additionally, in order to quantify imperceptibiliity of perturbations, the tie similarity of adversarially perturbed and unchanged sentences reported, along with the relative increase of average perplexity the perturbed samples, given by the GPT-2 (Radford et 2019) language model. Semantic similarity (Sem. sim.) is measured by the cosine distance between the embeddings produced by the Universal Sentence Encoder (USE) (Cer et al. 2018). This is a state- of-the-art sentence embedding widely used in adversarial attacks on text (Snn et al., 2020: Jin et al., 2019) Perplexity increase (PPL inc.) indicates how much the likelihood the perturbed data has decreased, given language model, and is often used to validate language models (Keselj. 2009). Due to the lack of related work in this field, we compare the AR estimation performance of TEF to our RANDOMATTACK (RA) baseline. RA serves importance ranking in Step of TEF and selects a random synonym in the final selection in Step 2. still utilized in RA to keep linguistic constraints intact. POS and stop word filters (see Section
agnostic attack, utilizes random word

---

Published as a conference paper at ICLR 2022
PCC of LSTMAut A on AG's News
PCC LSTM- IG on MR
ACC: 0.25
ACC: 0.18
ACC: 0.17
0.14
TEF
RA
ACC: 0.0
TEF
RA
RS
RI
0.16
0.16
Figure 3: Ablation study of TEF. compare the PCC of TEF, RA, the RANDOMIMPORTANNE (RI) attack and the RANDOMMYNONYM (RS) attack. We find that RI behaves slighty worse than TEF, while RS behaves slightly better than RA in terms of reducing attribution correlation over all p values.
4.2 ROBUSTNESS OF EXPLANATIONS
Attribution robustness estimation. In order to estimate the attribution robustness (AR) of the aforementioned models and explainers, we vary the parameter Pmaz of TEF, which denotes the maximum ratio of perturbed tokens in the input sample. A larger Pmar value leads to lower at- tribution correlation, as potentially more words are substituted in the input. We then capture the aforementioned metries PCC, ROC, SCC, Sem, sim, Top- 10%/30%/55% intersections and PPL inc. to evaluate AR. Additionally, in order to quantify performance of our attack over the whole opera- tion interval of 0 S -mms S 0.4, we compute the Area under the Pearson Correlation Curve (ACC). A lower value of ACC corresponds to lower robustness overall, as correlation values are lower. We note that a particular value of Pmmr does not guarantee that all input samples have exactly Pmar ratio of perturbed tokens. Therefore, we quantize our samples based on their actual, resulting perturbed ratio P such that samples with similar are grouped together. These bins are computed per dataset, choose the number of candidates in Step 2 of TEF to be N [C] = 15, as it is a good trade-off between TEF estimation performance and attack run time. As expected,
ensuring the comparability of resulting curves and ACCs for each plot. Moreover,
find that TEF is able significantly outperform the baseline provided by RA in terms of all AR metrics, on all datasets, models and explanation methods considered in this work. A subset of these results is shown in Figure y the rest can be found in Appendix A.2 Moreover, we do not find that any architecture is significantly more robust to TEF perturbations for explainers S and IG. However, the self-attention mechanism of transformers seems to be more robust to perturbations than non-transformer-baane and explanations. The semantic similarity decreases with increasing P and stays above 0.7 in most cases. This, together with the fact that resulting samples share predictions with the non-perturbed ones effectively highlights that the explanations given by these models and attribution methods lack faithfulness.
architectures
Ablation study. In addition to the fully random attack described in the previous paragraph, we compare TEF to our semi-random attacks RANDOMIMPORTANCE (RI) and RANDOMSYNONYY (RS). We randomize the word importance ranking of TEF (RI) but keep the selection of best final synonym, and we randomize the final synonym selection of TEF (RS) but keep the word importance ranking respectively. Figure 3 shows our findings for these experiments, along with comparisons to RTNDO (RA). The PCC curves and the ACC values show that consistently outperforms RS in terms of PCC over the whole operation interval of p. Moreover, the impact of word importance ranking diminishes with increasing p. especially for shorter datasets like MR. This can be observed by RS performing closer to RA for high p values.
BERT's attention layers and heads. BERT's attention weights can be used to help gain insight into a models prediction by understanding which parts of the input are most attended to (Vig 2019 Our BERT models have 12 layers with 12 attention heads (144 heads in total), each producing
distribution of attention weights over its inputs and outputs. Estimating the AR of all heads together is not useful, as effects would average out. Therefore, we run TEF to estimate the robustness of each head separately. Figure contains the average PCCs of the attention weights before and after per- turbing the inputs with TEF. We find that attention weights in later layers tend to be more susceptible to input perturbations than earlier layers. Moreover, heads within a layer tend to be comparably ro- bust. We leave thorough, theoretical analysis of this phenomenon to future work. We conclude that
7

---

Published as a conference paper at ICLR 2022
AG's News
MR
LMDB
Yelp
H.
H.
H.
H.
8
8
4
4
4
4
8
L.
4
8 L.
8 L.
4 L.
Figure 4: Estimated robustness of BERT attention weights on different layers (X-axis) and heads (Y-axis) for Omaz 0.2. Red cells indicate average PCC values close to 0, hence less robust attention head weights, while white cells have average PCCs close 1. Attention heads in later layers tend be less rohust, while heads within a layer seem equally robust in most layers.
PCC of LSTMAtt- IG on IMDB PCC of CNN IG on AG's News
PCC of CNN - IG on Yelp
ACC:0.27
ACC: 0.19
ACC: 0.15
ACC:
0
LSTM-IG
ACC=O.IO
BERT-A
BERT-A
ACC: 0.09
LSTMAR-S
LSTM-IG
LSTMAR-A
CNN-S
ACC: 0.06
LSTMAn-IG
ACC: 4:07
0.16
0.16
0.16
P
to other models and explanation methods. lines indicate the es- timated PCC of TEF perturbations transferred from the indicated models and explanations. and indicate the PCC curve of optimal TEF and RA perturbations respectively, without transfer.
Figure 5: Transfer capabilities
the increasing reliance on attention weights to provide inherent interpretations to BERT predictions needs careful investigation, especially in safety-critical applications.
4.3 TRANSFERABILITY AND SEMI-UNIVERSAL PERTURBATIONS
Transferability of perturbations models and explanations. The adversary does not necessar- ily possess information about the deployec model nor the exact method to produce the accompanying explanations. Therefore, it is crucial for systems be as resistant to transfer attacks as possible in order to evade perturbations constructed on similar models and explanations.
Thus, we examine how our classifiers and attribution methods react transfer attacks computed with TEF. We alter the input samples for given model and explanation method with TEF, then evaluate the PCC of attributions on the same samples but different architectures and explainers. The results found in Figure We observe that transfer attacks perform better than RA, some even by ap- prox. 0.4 in terms of average PCC decrease in the operation area of RJ 0.1. However, as expected, they significantly fall short of the performance of TEF. Therefore, conclude that transferring TEF perturbations across models and explainers effectively highlights fragility of explanations, but TEF provides tighter AR bounds without transfer.
Semi-universal perturbations. In this section, take step towards defining universal per- turbations, similarly to the work of the authors Moosavi-Dezlooll et al. (2017) and Gao & Oates) (2019). These provide fast and computationally cheap perturbations during attack time that are able to mislead classifiers with pre-computed perturbations. However, we attack the explanations of text classifiers, instead of their predictions and call our perturbations semi-universal attack policies.
We split the test dataset into two equally sized parts, the attack set and the evaluation set. We utilize the former for constructing our semi-universal attack policies and the latter to evaluate how effec- tively our semi-universal attack alters the attributions maps of our models.
First, for each sample in the attack set, we compute the optimal TEF perturbation for all our mod- els and explainers. We then extract statistics of these perturbations, which are the most common replacement and the replacement frequency for each replaced token and sort these by decreasing frequency. These semi-universal atlack policies, seen in Figure During this construction
8

---

Published as a conference paper at ICLR 2022
AG's News
IMDB
MR
Token
Repl.
Replacement
Token
Repl.
Replacement cinematographic
Token
Repl.
Replacement
reuters
146k
goldman
movic
430k
movie
8.2k
cinematographic
said
131k
avowed
film
338k
cine
film
8.0k
cinematographic
130k
nouvelle
good
122k
decent
story
2.6k
conte
ap
107k
great
103k
whopping
good
2.5k
decent
72.8k
tar
bad
102k
wicked
comedy
2.4k
humorist
workers
10.9k
labourers field
amazing
17.1k
staggering
triumph
139
victory
zone
2.9k
scary
6.8k
fearful
shines
glows
Figure 6: Semi-universe attack policies for different datasets.
PCC of LST on AG's News PCC LSTM IG IMDB
PCC of LSTMAtt- IG on MR
ACC: 0.22
ACC: 0.20
ACC: 0.20
TEF RA Somi-uni.
TEF
ACC: 0.14
0
TEF
ACC: 0.00
Somi-uni.
ACC: .0.4
Somi-uni.
ACC: 0.05
0.16
0.16
P
0.16
Figure 7: Average PCC of the indicated architectures and explainers after applying the semi- unii-universal pertur- bations (Semi-uni.), compared to TEF and RA attacks. The semi-universal attack successfully decreases the correlation of original and attacked attribution maps.
phase, we query the model for predictions and explanations, as we compute optimal TEF perturba- tions.
Second, we evaluate our semi-universe attack that utilizes the aforementioned policies to alter ex- planations of classifiers. The imputs this attack are text sample, a semi-universal policy and maximum perturbed ratio Pmar. The attack iterates over the policy, starting with the token in the first row and finishing with the last. Whenever the current token found in the input text sample, is replaced with the replacement token in the list. If the perturbed ratio exceeds Pmaz, the attack is aborted. In such a way, perturbed inputs are created without querying the model during attack time. The actual perturbation for each text sample depends on the sample, hence the name semi unni-universll attack policy. The resulting samples are then evaluated on a given model and explanation method. Representative results are given in Figure We conclude that our semi-universal policies ef- fective in reducing attribution correlation when the adversary has no access to the target model and explanation method, indicated by the lower ACC values of the semi-universal PCC curves.
5 CONCLUSION
In this work, introduced novel black-box attack called TEXTEXPPLAWTIONFOOLRRR that suc- cessfully perturbs input data such that the outcome of popular explanation methods in sequence classification, but not the prediction of the classifier. This attack provides baseline estimator for attribution robustness and highlights the lack of robustness of current explanation methods. We compared it to the random attack, showing its superior performance to it on five different, widely used text classification datasets. Moreover, the transfer capabilities of the attack are evaluated. Fi- nally, we showed the existence of semi-universal perturbation policies that are capable of altering explanations without querying the model during attack-time, even without having access to perturba- tions for those models. future work, we plan to examine whether a similar white-box attack that has access model gradients can improve robustness estimation. Moreover, instead of synonym embeddings, we plan to use BERT- based masked language models to extract possible candidates, further improving imperceptible word substinution.

---

Published as a conference paper at ICLR 2022
REFERENCES
Amina Adadi and Mohammed Berrada. Explainable ai for healthcare: from black box to inter- pretable models. In Embedded Systems and Artificial Intelligence, pp. 327-337. Springer, 2020.
Nabiha Asghar. Yelp dataset challenge: Review rating prediction. arXiv preprini arXiv: 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv preprint urXiv:1409.04: 2014.
Alexander Binder, Michael Bockmayr, Miriam Hägele, Stephan Wienert, Daniel Heim, Katharina Hellweg, Masaru Ishii, Albrecht Stenzinger, Andreas Hocke, Carsten Denkert, et al. Morphologi- cal and molecular breast cancer profiling through explainable machine learning. Nature Machine Intelligence, 3(4):355- 3666, 2021.
Jacob Buckman, Aurko Roy. Colin Raffel, and Ian Goodfellow. Thermometer encoding: One hot way resist adversarial examples. In International Conference on Learning Representations, 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pp. 39-57. IEEE, 2017.
Daniel Cer, Yinfei Yang, Sheng-yi Kong. Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Con- Steve Yuan, Chris Tar, et al. Universal sentence encoder. arXiv
stant, Mario Guajardo-Céspen preprint arXiv: 1155, 2018.
Jiefeng Chen, Xi Wu, Vaibhav Rastogi, Yingyu Liang, and Somesh Jha. Robust Attribution Regu- larization. In Advances in Neural Information Processing Systems, pp. 14300-14310, 2019.
Edward Choi, Mohammad Taha Bahadori, Joshua A Kulas, Andy Schuetz, Walter F Stewart, and Jimeng Sun. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. arXiv preprint arXiv: 1608.60080000 05745, 2016.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. International Conference on Machine Learning. pp. 854-863. PMLR, 2017.
Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations can manipulated and geometry to blame. In Advances in Neural Information Processing Systems, pp. 13589-13600, 2019.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv: 2017.
Christian Etmann, Scbastian Lunz, Peter Maass, and Carola-Bibiane Schönlieb. On the con- nection between adversarial robustness and saliency map interpretability. arXiv preprint arXiv:1905.04172. 2019.
Shi Feng, Eric Wallace, Alvin Grissom II, Mohit lyyer, Pedro Rodriguez, and Jordan Boyd-Graber. Pathologies of neural models make interpretations difficult. arXiv preprint arXiv:1804.07781, 2018.
Hang Gao and Tim Oates. Universal adversarial perturbation for text classification. arXiv preprint arXiv:1910.0460 2019.
Reza Ghaeini, Xiaoli Z Fern, and Prasad Tadepalli. Interpreting recurrent and attention-based neural models: a case study on natural language inference. arXiv preprint arXiv: 03894, 2018 2018.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation neural networks is fragile. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3681-3688, 2019.
Ivan Girardi, Pengfei Ji, An-phi Nguyen, Nora Hollenstein, Adam Ivankay, Lorenz Kuhn, Chiara Marchiori, and Ce Zhang. Patient risk assessment and warning symptom detection using deep attention-based neural networks, arXiv preprint arXiv: 1809, 1809.10804 2018.
10

---

Published as a conference paper at ICLR 2022
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1.
MIT Press, 2016.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Generating counterfactual explanations with natural language. arXiv preprint (rXiv:/806.09809, 2018.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial- strength Natural Language Processing in Python, 2020. URL
zenodo. 21 2303
Adam Ivankay, Ivan Girardi, Chiara Marchiori, and Pascal Frossard. Far: A general framework for attributional robustness. arXiv preprint arXiv:2010.07393, 2020.
Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? arXiv preprint arXiv:2004.03685, 2020.
Sarthak Jain and Byron C Wallace. Attention is not explanation. arXiv preprint arXiv:1902.10111 2019.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? natural language attack on text classification and entailment. arXiv preprint arXiv: 1907. / 1932, 2, 2019.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-93, 1938.
Vlado Keselj. Speech and language processing daniel jurafsky and james h. martin (stanford univer- sity and university of colorado at boulder) pearson prentice hall, 2009, xxxi+ 988 pp; hardbound, isbn 178-0-13-187321-6 $115.00, 2009.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Sigi Yan, et al. Captum: A unified and generic model interpretability library for pytorch. arXiv preprint arXiv:2009.07896, 2020.
Emanuele La Malfa, Agnieszka Zhrzezny, Rhiannon Michelmore, Nicola Paoletti, and Marta Kwiatkowska. On guaranteed optimal robust explanations for nlp models. arXiv preprint arXiv:2105.0360 036400, 2021.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. Bert-attack: Adversarial attack against bert using bert. arXiv preprint arXiv:2004.0998: 09984. 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoee, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907. 1669997..666,...... 2019.
Andrew Maas, Raymond Daly, Peter Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 201
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706 06083, 2017.
Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Sparsefool: a few pixels make big difference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9087-9096, 2019.
Seyed-Mohsen Moosavi- Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on com- pufer vision and pattern recognition, pp. 2574-2582, 2016.

---

Published as a conference paper at ICLR 2022
Seyed-Mohsen Moosavi- -ezaool,,, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1765-1773, 2017.
Seyed-Mohsen Moosavi- Dezzooll, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robust- ness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9078-9055 2019a.
Seyed-Mohsen Moosavi-Dezfood Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Ro- bustness via curvature regularization, and vice versa. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9078-9086 2019b.
Nikola Mrkšić, Diarmuid Séaghdha, Blaise Thomson, Milica Gašić, Lina Rojas- Barahora Pei- Hao Su. David Vandyke, Tsung- Tsung-Hsien Wen, and Steve Young. Counter-fittinn fittterrfittiig word vectors to linguistic constraints. arXiv preprini urXiv: 2016.
Leann Myers and Maria J Sirois. Spearman correlation coefficients, differences between. Encyclo- pedia of statistical sciences, 12, 2004.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances Neural Information Processing Systems, pp- 8026-8037, 2019.
Karl Pearson. Notes on regression and inheritance in the case of two parents. Proceedings of the Royal Society London, 58(347-352):240 242, 1895.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Sofia Serrano and Noah A Smith. Is attention interpretable? arXiv preprint arXiv:1906.03731, 2019.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 2013.
networs Vi-
Mayank Singh, Nupur Kumari, Puneet Mangla, Abhishek Sinha, Vineeth N Balasubramanian, and Balaji Krishnamurthy. On the benefits of attributional robustness. arXiv preprint UrXiv::911 13077, 2019.
Sanchit Sinha, Hanjie Chen, Arshdeep Sekhon, Yangfeng Ji, and Yanjun Qi. Perturbing inputs for fragile interpretations in deep natural language processing. arXiv preprint 108i:2208.04999) 2021.
Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caiming Xiong. Ady-bert: Bert not robust on misspellings! generating nature adversarial samples on bert. arXiv preprint arXiv:2003.044 04995, 2020.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 3319- 3328, 2017.
Jesse Vig. Bertviz: tool for visualizing multihead self-attention in the bert model. In ICLR Workshop: Debugging Machine Learning Models, 2019.
Sarah Wiegreffe and Yuval Pinter.
Attention is not not explanation. arXiv preprint
OrXiv::9068 04626, 2019.
12

---

Published as a conference paper at ICLR 2022
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing- In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. As- sociation for Computational Linguistics. URL https: www. aclweb. wwt./nttoo/gooooooooo
2020. 1m 6
Puyudi Yang, Jianbo Chen, Cho-Jui Hsieh, Jane-Ling Wang, and Michael Jordan. Greedy attack and gumbel attack: Generating adversarial examples for discrete data. J. Mach. Learn. Res., (43):1-36, 2020.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Euro- pean Conference on Computer Vision, pp. 818-833. Springer, 2014.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
13


Reviews:
[{"review_id": "wp8to0qYYoy", "reviewer": "Reviewer_ZvQi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The authors provide good explanations in the background of the issue. I find section 2 and 3 generally well-written and some discussions and intuitions were given about the attack algorithm.\n\n2. The method proposed is novel. The authors follow Ghorbani et al 2019 in looking at the issue on perturbation to explanations and extend the problem to text (discrete input). I believe the robustness of model's explanations is not a very well-studied issue in NLP and the community can benefit from this paper.\n\nWeakness:\n1. My first concern is that I am confused by section 4.3. I fail to see how the modified attack method provides universal perturbations to text explanations (without querying the model). I think this section would be clearer if the authors could use stick with their notations in section 2, 3 and algorithm 1 and provide further explanations.\n\n2. Another concern is that I wish the authors could have made it clearer how finding s_adv (eqn 6) gives an attribution robustness estimation. I think many practitioners could benefit from a quantitative measure on the robustness of explanations.\n\n3. Would the authors clarify an unreferenced $\\tilde{W}$ in eqn 4? \n\n4. One minor issue is that in investigating the attribution robustness in NLP models, the authors actually only studied sequence classification tasks (as opposed to span-labelling, text generation ...). I think this is a fine scope but also believe that the authors should perhaps make that clearer.\n\nLook forward to the discussions with the authors/other reviewers and I am willing to adjust my scores if the authors could clarify weakness 1 and 2.", "summary_of_the_paper": "This paper focuses on the issue of assessing the robustness of the \"explanations\" for NLP models (how the explanation, not the prediction, for NLP models, change with respect to small perturbations in the input). Particularly, the authors\n\n1. formalized the problem of attribution (explanation) robustness estimation as an optimization tasks in eqn (4, 5, 6), which involves finding the perturbations that keep the model's output the same, while maximizing the difference in attribution (explanation) compared to the case without these perturbations.\n\n2. developed a black-box attack model that estimate AR, without requiring gradient information. The authors also further conducted ablation studies on the components of the black-box model.\n\n3. conducted evaluations of the authors' attack model, analyzing three existing popular attribution estimation methods (saliency maps, integrated gradients and attention as explanation)\n\n", "summary_of_the_review": "Novel paper investigating attribution (explanation) robustness in NLP models. Some sections read confusing. I recommend \"weak accept\".", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cykQAKgGukx", "reviewer": "Reviewer_ZcSr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe proposed task is very interesting and important. Attacking explanations of deep neural networks have already been explored in the image domain [1,2] and graph domain (GNNs)[3]. It's interesting to have the investigation on NLP task. \n\n\nThe proposed attacking method is a general framework for attacking NLP model [4,5]. In particular, they compute the importance score for words. And then search the substitution to replace the words.\n\n\nThe proposed attribution robustness sounds nice in Eq. 4 and 6.\nAlthough they are almost the first work to attack explanations in NLP task (???), the solution is not too novel and straightforward. In fact, they just change the objective function of attackers to achieve their goal.\n\n So most existing attacking methods in NLP can be applied to attack explanation of text classification. \nIn this case, the solutions to attack the explanation of text classification would be trivial. Are there any advanced attacking methods for this problem?\n\nMinior: Chapter 2 -> section 2.\n\n\n\n\n[1] Fooling Network Interpretation in Image Classification\n\n[2] Fooling Neural Network Interpretations via Adversarial Model Manipulation\n\n[3] Jointly Attacking Graph Neural Network and its Explanations\n\n[4] Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment\n\n[5] Adversarial Attacks and Defense on Texts: A Survey\n", "summary_of_the_paper": "This paper proposes an attacking method to attack the explanations of text classification in NLP. The proposed method is easy to follow.\nIn particular, they first do the word importance ranking for each token of the input sample. High-important words are prioritized during substitution.  Second, candidate selection via counter-fitted GloVe is proposed to replace the word to attack explanations while maintaining the same confidence in the correctly predicted class. The authors conducted comprehensive experiments to evaluate the effectiveness of their methods. \n", "summary_of_the_review": "The proposed task is very interesting and important. Attacking explanations of deep neural networks have already been explored in the image domain [1,2] and graph domain (GNNs)[3]. It's interesting to have the investigation on the NLP task. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZIqA9tMwrEa", "reviewer": "Reviewer_p6VB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "--- Strengths:\n-\tThe manuscript is well written in general and overall methodology makes sense. \n-\tDecreased correlations are indeed a valuable finding in terms of the proposed attack.\n-\tProposed work is connected quite nicely to the previous work.\n-\tEvaluations on several datasets and methods are a plus.\n\n--- Weaknesses:\n-\tEven though the claim is evaluation on five datasets, only the subset of the results is reported. As not much related work is available in this domain claimed by the authors, I would expect somehow reporting complete set of results. \n-\tDespite extensive evaluation, a limited amount of discussion exists. Implications on unchanged predictions are also not discussed.\n-\tAs RA utilizes a straightforward baseline, I think it is relatively straightforward to have lower correlations with the proposed method compared to RA. \n-\tPlease report different perturbation ratios in the plots.", "summary_of_the_paper": "The authors introduce a text explanation attack algorithm called “Text Explanation Fooler”, that changes output of explanation methods while keeping the classification output the same. This is done by maximizing the distance between attribution maps of original and perturbed sequences, while forcing these inputs have the same prediction outcome. In the proposed attack, the authors apply word importance ranking for substitutions, and take POS tags and stopping words into account. They evaluate their methods with AG’s News, IMDB, Fake News, MR, and Yelp datasets along with CNN, LSTM, LSTMAtt, and BERT models and report significant decrease in correlations compared to RandomAttack (RA).", "summary_of_the_review": "While the technical methodology seems to be valid and confirmed with the experiments, there are some drawbacks such as not reporting all the results, limited discussion, and partly applying some methods from vision domain to text processing, also reported in the weaknesses field of the main review. Even though, I think the contribution is timely and valid, however, there should be several updates on the manuscript before it is ready for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an "attack"—TextExplanationFooler (TEF)—that adversarially (minimally) edits inputs such that the resultant attribution assigned by common explanation methods changes substantially, while the prediction does not. This is an extension of methods and results in vision to NLP. The authors find that all methods are vulnerable to this attack. 

Reviewers agreed that the demonstration that perturbation attacks used in vision are applicable in NLP is interesting and may lead to follow-up work. The paper would benefit from technical clarifications in several places (see R2), which were largely resolved in discussion.
Decision: Accept (Poster)