Brittle interpretations: The Vulnerability of TCAV and Other Concept-based Explainability Tools to Adversarial Attack

1 INTRODUCTION . Deep learning models have achieved superhuman performance in a range of activities from image recognition to complex games ( LeCun et al. , 2015 ; Silver et al. , 2017 ) . Unfortunately , these gains have come at the expense of model interpretability , with massive , overparametrized models being used to achieve state-of-the-art results . This is a salient problem when deep learning is applied to domains such as healthcare ( Miotto et al. , 2018 ) , criminal justice ( Li et al. , 2018 ) , and finance ( Huang et al. , 2020 ) , where a prediction needs to be explainable to the user , leading to a surge of interest in tools that can illuminate the underlying decision making process of deep learning models . Besides being inherently black-box in nature , deep learning models have also been shown to be vulnerable to adversarial attacks where small perturbations to model input result in dramatic changes to model output ( Szegedy et al. , 2013 ) . This phenomenon is concerning when deep learning tools are deployed in safety-critical environments . A range of approaches have been developed to improve a model ’ s robustness to adversarial attack ( Silva & Najafirad , 2020 ) , including the use of explainability methods to detect adversarial examples ( Zhang et al. , 2021 ; Wang & Gong , 2021 ) . But if explainability methods are an important component in a machine learning system , then the robustness of these methods are nearly as important as the robustness of the model itself . In this paper we explore the vulnerability of concept-based interpretability methods ( CBIMs ) . That is , methods that interrogate a model and its decisions based on a concept . CBIMs usually rely on a user provided collection of positive examples ( tokens ) of a concept . While this flexibility makes these methods an attractive approach for understanding deep learning models in domains such as healthcare ( Graziani et al. , 2018b ; Mincu et al. , 2021 ) , it also introduces a single point of failure wherein subtle changes to a few centralized tokens representing a concept results in misinterpretations of many subsequent input . We describe a threat model for post-hoc CBIMs , outlining adversary goals , knowledge , and capabilities . Then we introduce a family of attacks fitting this threat model which we call token pushing ( TP ) attacks . These learn small perturbations that when added to tokens of a concept result in remarkably different output for the interpretability method . Specifically , we optimize our perturbations so that when they are added to a token , they significantly change a model ’ s internal representation of the input . We test TP attacks against two popular CBIMs : Testing with Concept Activation Vectors ( TCAV ) ( Kim et al. , 2018 ) and Faceted Feature Visualization ( FFV ) ( Goh et al. , 2021 ) . While TCAV and FFV are similar in that they are both concept-based , their output is quite different . TCAV quantifies the extent to which a concept is important to a model ’ s prediction for a specific input dataset . A variant of this method was recently a central component of ( McGrath et al. , 2021 ) , which used it to provide evidence that models such as AlphaZero learn human chess concepts . FFV on the other hand , can be used to produce visualizations that represent how individual neurons capture a specific concept . We show that TP attacks are effective for both TCAV and FFV . For example , a TP attack causes TCAV to give output indicating that stripes are not an important feature to the class ‘ zebra. ’ On the other hand , a TP attack can radically change the feature visualizations generated by FFV ( Figure 3 ) . We evaluate TP attacks on pretrained ImageNet models ( Deng et al. , 2009 ; Marcel & Rodriguez , 2010 ) using the Describable Textures Dataset ( Cimpoi et al. , 2014 ) for concept tokens . Through our experiments we show that a TP attack does not require the adversary to know what interpretability method is being used . The same perturbations that cause TCAV to fail , also cause FFV to fail . Finally TP attack possesses moderate transferability , meaning that as long as a surrogate model is available , it can be applied even when the defender model architecture is unknown . In summary , our contributions in this paper include the following . • Formalization of an adversarial threat model for post-hoc concept-based interpretability methods that identifies concept tokens as a single point of failure for such methods . • Introduction of TP attacks which cause misinterpretation by disrupting a central mechanism by which concept tokens are used across a range of concept-based interpretability methods . • Demonstration of the effectiveness of TP attacks on two concept-based interpretability methods , TCAV and FFV . • Introduction of the first ( to our knowledge ) adversarial attack on feature visualization . 2 RELATED WORK AND BACKGROUND . Interpretability methods : Because of the size and complexity of modern deep learning architectures , skill is required to extract interpretations of how these models make decisions . Established methods range from those that focus on highlighting the importance of individual input features to those that can give clues to the importance of specific neurons to a particular class . Popular examples of interpretability methods that focus on input feature importance include saliency map methods ( Selvaraju et al. , 2017 ; Sundararajan et al. , 2017 ; Ribeiro et al. , 2016 ; Fong & Vedaldi , 2017 ; Dabkowski & Gal , 2017 ; Chang et al. , 2019 ) which identify those input features ( for example , pixels in an image ) whose change is most likely to change the network ’ s prediction . CBIMs focus on decomposing the hidden layers of deep neural networks with respect to humanunderstandable concepts . One of the best-known approaches in this direction involves the use of concept activation vectors ( CAVs ) ( Kim et al. , 2018 ) which we describe in detail in the next section . Work that is either related or extends these ideas includes ( Zhou et al. , 2018 ; Graziani et al. , 2018a ; 2019 ; Koh et al. , 2020 ; Yeh et al. , 2020 ) . Feature visualization is a set of interpretability techniques ( Szegedy et al. , 2014 ; Mahendran & Vedaldi , 2015 ; Wei et al. , 2015 ; Nguyen et al. , 2016b ) concerned with optimizing model input so that it activates some specific node or set of nodes within the network . However , a challenge arises when one tries to analyze ‘ polysemantic neurons ’ ( Olah et al. , 2018 ) , neurons that activate for several conceptually distinct ideas . For example , a neuron that fires for both a boat and a cat leg is polysemantic . Interpretability methods have imposed priors to disambiguate neurons by clustering the training images ( Wei et al. , 2015 ; Nguyen et al. , 2016b ) or the hidden layer activations ( Carter et al. , 2019 ) and using the average of the cluster as a coarse-grained image prior , parameterizing the feature visualization image with a learned GAN ( Nguyen et al. , 2016a ) , or using a diversity term in the feature visualization objective ( Wei et al. , 2015 ; Olah et al. , 2017 ) . Robustness of interpretability methods : This is not the first work that has shown that interpretability methods can be brittle . Saliency methods have been shown to produce output maps that appear to point to semantically meaningful content even when they are extracted from untrained models , indicating that these methods may sometimes simply function as edge detectors ( Adebayo et al. , 2018 ) . Further , preliminary work has studied the robustness of Concept Bottleneck Models , an intrinsically interpretable concept-based method , to out-of-distribution data ( Koh et al. , 2020 ) . From a more adversarial perspective , a number of works have shown that saliency methods are vulnerable to small perturbations made to either an input image or to the model itself that cause the model to offer radically different interpretations ( Heo et al. , 2019 ; Ghorbani et al. , 2019 ; Viering et al. , 2019 ; Subramanya et al. , 2019 ; Anders et al. , 2020 ) ; work has looked at methods to make explanations more robust to attack ( Lakkaraju et al. , 2020 ) . On the other hand , this is the first work that shows that CBIMs are also vulnerable to adversarial attack . In particular , since we focus on attacks targeting a component absent from other interpretability methods ( concept tokens ) , there is not a straightforward way of applying the attacks mentioned above within the threat model presented in this paper . 2.1 TCAV AND LINEAR INTERPRETABILITY . In this section we describe the method of testing with concept activation vectors ( TCAV ) ( Kim et al. , 2018 ) . Let f : X → Rd be a neural network which is composed of n layers and designed for the task of classifying whether a given input x ∈ X belongs to one of d different classes . Write f ` : X → Rd ` for the composition of the first ` layers so that fn = f and dn = d and let h ` : Rd ` → Rd be the composition of the last n− ` layers of the network so that f = h ` ◦ f ` for any 1 ≤ ` ≤ n− 1 . Let C be a concept for which we have a set of positive examples ( tokens ) PC = { xPi } i and negative examples NC = { xNi } i , both belonging to X . These are represented in the ` th layer of f as the points f ` ( PC ) and f ` ( NC ) respectively . One can apply a binary linear classifier to separate these two sets of points , resulting in a hyperplane in Rd ` . This hyperplane can be represented by two unit normal vectors . We choose the one , v ` C ∈ Rd ` , that points into the region corresponding to the points f ` ( PC ) . v ` C is called the concept activation vector in layer ` associated with concept C. One can think of v ` C as the vector that points toward C-ness in the ` th layer of the network . Let h ` , k denote the kth output coordinate of h ` corresponding to class k. In the classification setting , h ` , k then represents the model ’ s confidence that input belongs to class k. To better understand the extent to which concept C influences the model ’ s confidence of x ∈ X belonging to class k we compute : SC , k , l = ∇h ` , k ( f ` ( x ) ) · vlC . ( 1 ) A positive value of SC , k , l indicates that increasing C-ness of x makes the model more confident that x belongs to class k. The magnitude TCAV score for a dataset D is defined as TCAVQC , k , ` = 1 |Dk| ∑ x∈Dk SC , k , l ( x ) , where Dk is the subset of D consisting of all instances predicted as belonging to class k. We compare the TCAV magnitude of the positive concept images with the TCAV magnitude for random images in the layer , and use a standard two-sided t-test to test for significance . We can also compute the relative TCAV score , which replaces the set of negative natural images in NC with images representing a specific concept .

Summary: The paper describes the threat model for concept-based interpretability methods and mainly investigates the vulnerability of TCAV and faceted feature visualization. It introduces Token Pushing (TP) attacks, which learn small perturbations for the token of concept leading to different output for the interpretability method. The proposed attack is evaluated on pre-trained ImageNet models on the Describabke Texture Dataset for concept tokens.