Fooling Explanations in Text Classifiers

State-of-the-art text classification models are becoming increasingly reliant on deep neural networks ( DNNs ) . Due to their black-box nature , faithful and robust explanation methods need to accompany classifiers for deployment in reallife scenarios . However , it has been shown in vision applications that explanation methods are susceptible to local , imperceptible perturbations that can significantly alter the explanations without changing the predicted classes . We show here that the existence of such perturbations extends to text classifiers as well . Specifically , we introduce TEXTEXPLANATIONFOOLER ( TEF ) , a novel explanation attack algorithm that alters text input samples imperceptibly so that the outcome of widely-used explanation methods changes considerably while leaving classifier predictions unchanged . We evaluate the performance of the attribution robustness estimation performance in TEF on five sequence classification datasets , utilizing three DNN architectures and three transformer architectures for each dataset . TEF can significantly decrease the correlation between unchanged and perturbed input attributions , which shows that all models and explanation methods are susceptible to TEF perturbations . Moreover , we evaluate how the perturbations transfer to other model architectures and attribution methods , and show that TEF perturbations are also effective in scenarios where the target model and explanation method are unknown . Finally , we introduce a semi-universal attack that is able to compute fast , computationally light perturbations with no knowledge of the attacked classifier nor explanation method . Overall , our work shows that explanations in text classifiers are very fragile and users need to carefully address their robustness before relying on them in critical applications . 1 INTRODUCTION . Deep neural networks ( DNNs ) have undoubtedly become the state-of-the-art architectures for many existing machine learning tasks ( Choi et al. , 2016 ) . Yet , their black-box nature has raised the need for developing methods to mitigate the lack of interpretability caused by their increased complexity ( Simonyan et al. , 2013 ; Zeiler & Fergus , 2014 ; Hendricks et al. , 2018 ; Bahdanau et al. , 2014 ) . These methods give intuitive , easily understandable explanations that do not require significant domain knowledge . This is especially desired in safety-critical scenarios , such as healthcare , where the users of such DNNs - the medical professionals for instance - need to understand the decision process and reasoning behind it . However , they have been shown to lack local robustness towards carefully crafted , imperceptible perturbations in the input ( Ghorbani et al. , 2019 ) . While resulting in the same predictions , these altered inputs yield significantly different explanations and attributions maps ( Figure 1 ) . Interpretation methods fragile towards small input perturbations not only fail to provide faithful explanations , a desiderata commonly required in explainable AI ( Jacovi & Goldberg , 2020 ) , but also damages user trust in DNNs and prevents them from being deployed on high-stakes , safetycritical applications , such as in healthcare ( Adadi & Berrada , 2020 ) . The previously described phenomenon has been widely studied in the image domain by Etmann et al . ( 2019 ) ; Moosavi-Dezfooli et al . ( 2019b ) or Ivankay et al . ( 2020 ) . However , in discrete-input domains like text , there has been limited progress . This is especially problematic given the increased reliance on and fragility of attention mechanisms as inherently explainable methods , as stated in Ghaeini et al . ( 2018 ) . Therefore , we summarize our contributions as follows : • We provide a novel baseline black-box adversarial attack , TEXTEXPLANATIONFOOLER ( TEF ) to estimate the local robustness of explanations in text classification problems • We evaluate attribution robustness on widely used , state-of-the-art text datasets and model architectures , showing that explanation methods ’ output can be significantly altered with our attack • We provide insight into the transfer capability of TEF on different models and explanation methods as well as introduce semi-universal adversarial perturbations to alter explanations without requiring access to the model at attack-time 2 PRELIMINARIES . 2.1 RELATED WORK . Adversarial attacks that alter the inference outcomes in DNNs have been widely studied both in the image ( Goodfellow et al. , 2014 ; Carlini & Wagner , 2017 ; Moosavi-Dezfooli et al. , 2016 ; Modas et al. , 2019 ) and text domain ( Ebrahimi et al. , 2017 ; Sun et al. , 2020 ; Jin et al. , 2019 ; Yang et al. , 2020 ) . Methods to alleviate the networks susceptibility to such attacks have also been proposed , including the works of Madry et al . ( 2017 ) ; Moosavi-Dezfooli et al . ( 2019a ) ; Buckman et al . ( 2018 ) or Cisse et al . ( 2017 ) . However , it has recently been shown by authors Ghorbani et al . ( 2019 ) that , in addition to DNN predictions , widely-used explanation methods also lack robustness to targeted , imperceptible alterations of the input . These attacks change the outcomes of such explanation methods significantly , while predictions of the DNNs are unaltered . This violates the Prediction Assumption of faithful explanations and crucially degrades user trust in such explanation methods , as significantly different interpretations are provided for similar inputs and outputs ( Jacovi & Goldberg , 2020 ) . The aforementioned phenomenon of fragile explanations has mostly been investigated in the image domain ( Etmann et al. , 2019 ; Singh et al. , 2019 ; Chen et al. , 2019 ; Ivankay et al. , 2020 ) , with less focus on discrete input spaces like text . However , faithful and robust interpretations are arguably equally important in the discrete text domain , for instance in electronic health record classification ( Girardi et al. , 2018 ) or precision medicine ( Binder et al. , 2021 ) , where critical decisions often need to be based on DNN explanations . The work of La Malfa et al . ( 2021 ) constructs inherently robust explanations for NLP models , however only towards perturbations in the embedding space , not the input space that adversaries can operate on . Moreover , they do not give a method to evaluate robustness of already existing explanation algorithms . The authors of Feng et al . ( 2018 ) show that interpretation methods in NLP lack completeness ( Sundararajan et al. , 2017 ) by removing words deemed least important by explanation methods . Moreover , attention mechanisms ( Bahdanau et al. , 2014 ) have been increasingly relied on as inherently interpretable systems . However recent work has questioned their faithfulness and plausibility ( Serrano & Smith , 2019 ; Jain & Wallace , 2019 ; Wiegreffe & Pinter , 2019 ) by proving that they often do not highlight input components that are most important to a DNN decision . We , paralleled by the very recent work of Sinha et al . ( 2021 ) , are the first to show that imperceptible perturbations in the input space can alter the outcome of explanations of text classifiers significantly , giving an efficient attack to estimate explanation robustness . However , our work is the first to give an extensive evaluation of the robustness of widely-used explanation methods on large datasets , comparing several state-of-the-art architectures . Further , we are the first to address robustness of attention weights in transformer architectures and to provide insight on transfer capabilities and universal attacks . 2.2 BACKGROUND . Let S = { s1 , s2 , ... , sN } be a dataset of N text samples si , each with a label from a predefined set of labels L = { l1 , l2 , ... , l|L| } . Each sample si contains a sequence of tokens ( or words ) wi taken from a discrete vocabulary set W = { w1 , w2 , ... , w|W| } . A generic sequence classifier then consist of a non-injective , non-surjective embedding function E : S ! Rd⇥p , E ( s ) = X , which maps the input sample s to its embedding matrix X , and a function f : Rd⇥p ! R|L| , f ( X ) = o , representing a ( DNN ) classifier function . d and p denote the embedding dimension and sequence length respectively . Let F : S ! R|L| , F ( s ) = f E be the full sequence classifier with final prediction y = argmaxi2 { 1 : |L| } oi . We define an attribution map as A : S ! Rp , A ( s , F , l ) = a that assigns a scalar value to each input token wi in the text sample s , resulting in the attribution vector a 2 Rp . This vector represents each token ’ s influence towards the prediction outcome y of classifier F . Our work considers three widely-used attribution methods in text classification , namely Saliency Maps ( S ) ( Simonyan et al. , 2013 ) , Integrated Gradients ( IG ) ( Sundararajan et al. , 2017 ) and Attention ( A ) ( Bahdanau et al. , 2014 ) , defined in the following Equations ( 1 ) , ( 2 ) and ( 3 ) respectively . ASi ( s , F , l ) = X j2 { 1 : d } |rXf ( X ) l|j , i ( 1 ) AIGi ( s , F , l , B ) = X j2 { 1 : d } ⇥ ( X B ) · Z 1 ↵=0 rX̃f ( X̃ ) l|X̃=B+↵ ( X B ) d↵ ⇤ j , i ( 2 ) AAtti ( s , F , l ) = exp eiP j2 { 1 : p } exp ej ( 3 ) where B denotes the null matrix 0d⇥p , f is the classifier function of F , ↵ a scaling factor and X = E ( s ) . rXf denotes the matrix-derivative of f to X , as defined in Goodfellow et al . ( 2016 ) . An attention head is a layer that transforms its inputs into scores e and calculates its output by linear combination of each input score , with coefficients normalized to a distribution . These coefficients are the attention weights AAtti ( s , F , l ) denoted in Equation ( 3 ) . It is commonly agreed to give intuitive explanations on how much the model attends to the given inputs through its attention weights ( Jacovi & Goldberg , 2020 ) . 3 METHODS . In this section , we describe our novel method TEXTEXPLANATIONFOOLER ( TEF ) to estimate attribution robustness ( AR ) in sequence classification problems . Specifically , we define the problem formulation , introduce our threat model and present the algorithm used to alter explanations by imperceptibly changing the inputs . 3.1 PROBLEM FORMULATION . Given an input text samples s and s̃ , labels l ; a text classifier F with embedding function E and classifier function f ; and attribution method A , we define attribution robustness ( also explanation robustness , AR ) as written in Equation ( 4 ) . r ( ã , a ) = 1 max ã d ( ã , a ) = 1 max s̃ d ⇥ A ( s̃ , F , l ) , A ( s , F , l ) ⇤ ( 4 ) with argmax i2 { 1 : |L| } F ( s̃ ) = argmax i2 { 1 : |L| } F ( s ) , ( 5 ) where d denotes a distance measure between the attributions ã and a of the the two input samples s and s̃ . The rest of the notation is kept as in Section 2 . Equation ( 4 ) quantifies how different the attributions of two input samples are , given the constraint in Equation ( 5 ) that enforces the inputs having the same prediction outcome . The attribution robustness estimation is then solved utilizing the following Equation ( 6 ) . sadv = argmax s̃ d ⇥ A ( s̃ , F , l ) , A ( s , F , l ) ⇤ ( 6 ) where sadv denotes the solution to the estimation , i.e . the adversarial input . s denotes the original , unperturbed input and s̃ the perturbed input , optimized during estimation . The solution sadv gives a robustness estimate by minimizing r defined in Equation ( 4 ) within a local neighbourhood of s. This neighbourhood is defined by both linguistic constraints described in the next section that encourage semantic proximity to the original text and the perturbed samples having the same prediction outcome as the unperturbed ones , see Equation ( 5 ) . This formulation is backed by current research ( Ghorbani et al. , 2019 ; Dombrowski et al. , 2019 ; Ivankay et al. , 2020 ) and the Prediction Assumption of faithful explanations ( Jacovi & Goldberg , 2020 ) .

Summary: This paper focuses on the issue of assessing the robustness of the "explanations" for NLP models (how the explanation, not the prediction, for NLP models, change with respect to small perturbations in the input). Particularly, the authors  1. formalized the problem of attribution (explanation) robustness estimation as an optimization tasks in eqn (4, 5, 6), which involves finding the perturbations that keep the model's output the same, while maximizing the difference in attribution (explanation) compared to the case without these perturbations.  2. developed a black-box attack model that estimate AR, without requiring gradient information. The authors also further conducted ablation studies on the components of the black-box model.  3. conducted evaluations of the authors' attack model, analyzing three existing popular attribution estimation methods (saliency maps, integrated gradients and attention as explanation)