Brittle interpretations: The Vulnerability of TCAV and Other Concept-based Explainability Tools to Adversarial Attack
Authors: Davis Brown
Henry Kvinge
Venue: ICLR 2022 Submitted (2022)
Keywords: interpretability
adversarial attack

Abstract: Methods for model explainability have become increasingly critical for testing the fairness and soundness of deep learning. A number of explainability techniques have been developed which use a set of examples to represent a human-interpretable concept in a model's activations. In this work we show that these explainability methods can suffer the same vulnerability to adversarial attacks as the models they are meant to analyze. We demonstrate this phenomenon on two well-known concept-based approaches to the explainability of deep learning models: TCAV and faceted feature visualization. We show that by carefully perturbing the examples of the concept that is being investigated, we can radically change the output of the interpretability method, e.g. showing that stripes are not an important factor in identifying images of a zebra. Our work highlights the fact that in safety-critical applications, there is need for security around not only the machine learning pipeline but also the model interpretation process.  

Under review as a conference paper at ICLR 2022
BRITTLE INTERPRETATIONS: THE VULNERABILITY OF TCAV AND OTHER CONCEPT-BASED EXPLAINABIL- ITY TOOLS TO ADVERSARIAL ATTACK
Anonymous authors Paper under double-blind review
ABSTRACT
Methods for model explainability have become increasingly critical for testing the fairnsss and soundness of deep learning. A number of explainability techniques have been developed which use set of examples to represent a interprrtteee concept in model's activations. In this work we show that these explainability methods can suffer the same vulnerability to adversarial attacks as the models they meant to analyze. We demonstrate this phenomenon two well-known sancept-based approaches to the explainability of deep learning models: TCAV and faceted feature visualization. We show that by carefully perturbing the examples of the concept that is being investigated, we can radically change the output of the interpretability method, e.g. showing that stripes are not an important factor in identifying images of a zebra. Our work highlights the fact that in safety- critical applications, there is need for security around not only the machine learning pipeline but also the model interpretation process.
INTRODUCTION
Deep learning models have achieved superhuman performance in range of activities from image recognition to complex games (LeCun et al., [2155] Silver et al. 2017). Unfortunately, these gains have come the expense of model interpretability. with massive, overparametrized models being used to achieve state-of-the-art results. This is a salient problem when deep learning is applied to domains such as healthcare (Miotto et al. 2018), criminal justice (Li al. 2018). and finance (Huang et al. 2020). where a prediction needs to be explainable to the user, leading to surge of interest in tools that can illuminate the underlying decision making process of deep learning models.
Besides being inherently black-box in nature, deep learning models have also been shown to be vulnerable to adversarial attacks where small perturbations to model input result in dramatic changes to model output (Szegedy et al. 2013). This phenomenon is concerning when deep learning tools are deployed in safety-critical environments. range of approaches have been developed to improve a model's 'ss robustness adversarial attack (Silva & Najafirad 2020), including the use of explainability methods to detect adversarial examples (Zhang et al. 2021 Wang & Gong (002T) But if explainability methods are an important component in a machine learning system, then the robustness of these methods are nearly as important as the robustness of the model itself. In this paper we explore the vulnerability of concept-based interpretability methods (CBIMs), That is, methods that interrogate model and its decisions based on concept.
CBIMs usually rely on user provided collection of positive examples (tokens) of concept. While this flexibility makes these methods an attractive approach for understanding deep learning models in domains such as healthcare (Graziani al. 2018b) Mineu et al., (02T), it also introduces a single point of failure wherein subtle changes to a few centralized tokens representing a concept results in misinterpretations of many subsequent input. We describe a threat model for post-hoc CBIMs, outlining adversary goals, knowledge, and capabilities. Then we introduce a family of attacks fitting this threat model which we call token pushing (TP) attacks. These learn small perturbations that when added tokens of concept result in remarkably different output for the interpretability method. Specifically, we optimize our perturbations so that when they are added to a token, they significantly change model's 's internal representation of the input.

---

Under review as a conference paper at ICLR 2022
We test TP attacks against two popular CBIMs: Testing with Concept Activation Vectors (TCAV) (Kim et al. 2018) and Faceted Feature Visualization (FFV) (Goh et al. similar that they are both concept-based their output is quite different. TCAV quantifies the extent to which a concept is important to a model's prediction for a specific input dataset. A variant of this method was recently central component of (McGrath al. (02T), ,, which used it provide evidence that models such as AlphaZero learn human chess concepts. FFV on the other hand, can be used produce visualizations that represent how individual neurons capture specific concept. We show that TP attacks are effective for both TCAV and FFV. For example, a TP attack causes TCAV to give output indicating that stripes are not an important feature to the class 'zebra.' On the other hand, a TP attack can radically change the feature visualizations generated by FFV (Figure 3
202I). While TCAV and FFV
We evaluate TP attacks on pretrained ImageNet models (Deng et al. 2009 Marcel & Rodriguez 2010) using the Describable Textures Dataset (Cimpoi et al., 20114) for concept tokens. Through our experiments we show that a TP attack does not require the adversary to know what interpretability method is being used. The same perturbations that cause TCAV to fail, also cause FFV to fail. Finally TP attack possesses moderate transferability, meaning that as long as surrogate model is available, it can be applied even when the defender model architecture is unknown.
In summary, our contributions in this paper include the following.
Formalization of an adversarial threat model for post-hoc concept-based interpretability methods that identifies concept tokens as a single point of failure for such methods.
Introduction of TP attacks which cause misinterpretation by disrupting a central mechanism by which concept tokens are used across a range of concept-based interpretability methods.
Demonstration of the effectiveness of TP attacks on two concept- based interpretability methods, TCAV and FFV.
Introduction of the first (to our knowledge) adversarial attack on feature visualization.
2 RELATED WORK AND BACKGROUND
Interpretability methods: Because of the size and complexity of modern deep learning architectures, skill is required to extract interpretations of how these models make decisions. Established methods range from those that focus on highlighting the importance of individual input features to those that can give clues to the importance of specific neurons to a particular class. Popular examples of interpretability methods that focus on input feature importance include saliency map methods (Selvaraju et al. 2017]] Sundararajan et al. 2017;; Ribeiro et al.) 2016 Fong & Vedaldi, [017]] Dabkowski & Gal 2017 Chang et al. 2019) which identify those input features (for example, pixels in image) whose change is most likely to change the network's prediction.
CBIMs focus on decomposing the hidden layers of deep neural networks with respect to human- understandable concepts. One of the best- approaches in this direction involves the use of concept activation vectors (CAVs) (Kim et al. (2018) which we describe in detail in the next section. Work that is either related or extends these ideas includes (Zhou et al. 2018 Graziani et al.) 2018a
2019
Koh et al. 2020 Ych et al. 2020).
Feature visualization is set of interpretability techniques (Szegedy et al. 2014 Mahendran &
Vedaldi 2015| Wei et al., 2015; Nguyen et al. 2016b) concerned with optimizing model input so that it activates some specific node or set of nodes within the network. However, a challenge arises when one tries to analyze 'polysemantic neurons' OOaa et al. 2018), neurons that activate for several conceptually distinct ideas. For example, neuron that fires for both a boat and cat leg is polysemantic. Interpretability methods have imposed priors to disambiguate neurons by clustering the training images (Wei et al., 20155] Nguyen et al. 2016b) or the hidden layer activations (Carter al. 2019) and using the average of the cluster as searse-grained image prior, parameterizing the feature visualization image with learned GAN (Nguyen et al. 2016a). or using a diversity term in the feature visualization objective (Wei et al. 2015 Olah et al. 2017).
Robustness of interpretability methods: ity methods can be brittle. Saliency methods have been shown to produce output maps that appear to point to semantically meaningful content even when they are extracted from untrained models, indicating that these methods may sometimes simply function as edge detectors (Adebayo al.
This is not the first work that has shown that interpretabil-
2

---

Under review as a conference paper at ICLR 2022
2018). Further, preliminary work has studied the robustness of Concept Bottleneck Models, an intrinsically interpretable concept-based method, to out-of of-distriibttiinn data (Koh et al. 2020). From more adversarial perspective, number of works have shown that saliency methods are vulnerable to small perturbations made to either an input image or to the model itself that cause the model to offer radically different interpretations (Heo et al.] 2019: Ghorbani et al. 2019) Viering al. 2019 Subramanya et al., 2019; Anders et al. 2020); work has looked at methods to make explanations more robust attack (Lakkaraju al. [2020). On the other hand, this is the first work that shows that CBIMs are also vulnerable to adversarial attack. In particular, since we focus on attacks targeting component absent from other interpretability methods (concept tokens), there is not straightforward way of applying the attacks mentioned above within the threat model presented in this paper.
2.1 TCAV AND LINEAR INTERPRETABILITY
In this section we describe the method of testing with concept activation vectors (TCAV) (Kim et al. 2018). Let f X + _d be neural network which is composed of n layers and designed for the task of classifying whether a given input æ E X belongs to one of d different classes. Write fz : X + Rd for the composition of the first layers composition of the last n - E layers of the network so that f = hy o ft for any 1 < < E n- n--n- Let C' be concept for which we have set of positive examples (tokens) Pc {a. ): and negative examples Nc {x:; [r: both belonging to X. These are represented in the éth layer of f as the points fe(Pc) and fe(Nc) respectively. One can apply a binary linear classifier to separate these two sets of points, resulting in a hyperplane in Re This hyperplane can be represented by two unit normal vectors. We choose the one, ve E TRd, that points into the region corresponding to the points f((Pc). o is called the concept activation vector in layer & associated with concept C. One can think of of as the vector that points toward C-ness in the Eth layer of the network.
that fn. = ff and d., d and let he R - Rd be the
Let h denote the kth output coordinate of lı corresponding class k. In the classification setting, he, then represents the model's confidence that input belongs to class k. To better understand the extent to which concept C' influences the model's confidence of X belonging to class & compute:
Sc,,,1 = Vhe, (fe(z)) oec.
(1)
A positive value of Sc,k.1 indicates that increasing C-ness of makes the model more confident that x belongs to class k. The magnitude TCAV score for a dataset D is defined as
TCAV Qc.u. =
Σ Sc,,((F),
DE
xEDR
where Dy is the subset of D consisting of all instances predicted as belonging to class k. We compare the TCAV magnitude of the positive concept images with the TCAV magnitude for random images in the layer, and use a standard two-sided f-test to test for significance. We can also compute the relative TCAV score, which replaces the set of negative natural images in No with images representing specific concept.
2.2 FACETED FEATURE VISUALIZATION
Goh et al. 202T) introduced new concept-based feature visualization objective for neuron-level interpretability. Faceted Feature Visualization (FFV). The objective disambiguates poly-semantic neurons by imposing a prior towards a linear concept C' in the optimization objective. Goh also utilizes set of positive and negative examples of a concept C' (Pc and Ne respectively). Similar to the TCAV method, one trains a binary linear classifier on the image of these two sets under the map fe obtain v/c. To visualize output that tends to activate neuron layer E, position while at the same time steering the visualization toward a specific context, the authors solve the following optimization problem:
arg max (z) + v'c (fe(x) O ft,()))) ZEX
(2)
where @ is the Hadamard product. Note that the first term helps find x which result in a strong activation of Secú while the second term finds such that fe(x) tends to point in the direction of cc-
3

---

Under review as a conference paper at ICLR 2022
3 ADVERSARIAL ATTACKS ON INTERPRETABILITY
An adversarial attack (Szegedy al. 2013) on a model f is small perturbation ò that, when applied to a specific input v. results in large changes to model prediction [():). The meaning of 'small' is usually specified by a metric such as an b nororrm and can either be a hard or soft constraint. In this work we use projected gradient descent (PGD) (Madry et al. 2018) to construct our attacks, but this should be seen mostly the underlying mechanism central to many CBIMs. Optimization approaches other than PGD could doubtless be used for the same effect.
placeholder. The novelty of the attack the manner in which it targets
We frame the notion of a CBIM abstractly in order to better understand its attack surface. We view such a method as a map that takes (1) a model, (2) positive tokens of the concept that we would like to steer our interpretation, (3) negative tokens of the concept and (4) an interpretation target which will be the focus of the interpretation. We call the output of an interpretability method an interpretation object. An interpretation object might be a single scalar value (as in the case of TCAV), it may be an image (as in the case of FFV). In all cases, an interpretation object is designed to help the user better understand model's decision making process. Thus, we can understand method as a function /: M x P x N x T - O, where M is the collection of models that can be interpreted, P is the space of all possible positive token sets, /' is the space of all possible negative token sets, T is the space of interpretation targets, and O is the space of interpretation objects that the method produces. We note that in the case of TCAV, the interpretation target a dataset D& of examples of some class k while the interpretation target of FFV is a specific node position (i. k) in the model.
interpretability
3.1 A THREAT MODEL FOR ATTACKS ON CONCEPT-BASED INTERPRETABILITY METHODS
Following a suggestion given in (Carlini et al. 2019), we state the threat model that we will consider in this paper. Since we will only be considering images as input in our experiments, we specifyy to that setting here. Otherwise, we use the formalism that we developed above. Specifically, we assume there exists an interpretability method I, a model f M, set of positive image tokens Pe {x,²} P, set of negative image tokens No E N, and interpretation target T E T. We also assume a function F O O R that quantitatively captures meaningful difference between interpretation objects.
The adversary's goal is to find perturbations { such that Pe =aff + 6.3 maximizes the value of F(T Pc, T), T(S, Pc, Nc; T)). That is, the change from Pe to Pa maximizes the difference in interpretation as measured by F. In order to avoid detection, Po is subject to the constraint: max; ||6: I c, for some fixed >> 0.
Adversary's goal:
Adversary knowledge and capabilities: (1) In this paper we assume that the adversary has read and write access to the tokens Pc either before or after they have been collected. (2) We do not assume that the adversary has access to either (the dataset of examples predicted as belonging to class in the case of TCAV or the specific neuron position that is being targeted in the case of FFV). We do assume that the adversary knows the hidden layer that is being targeted for both TCAV and FFV. (3) We assume that the adversary has read access at least surrogate model trained on the same dataset as f. We do not assume that this surrogate model needs to have the same architecture as f. (4) Finally, we do not assume that the adversary knows the interpretability method that will be used.
The adversary's 's goal is framed in terms of function F that depends on the specific interpretability method. This might seem to be in conflict with assumption (4) that says that the adversary does not have knowledge of the interpretability method being used. Actually, we show that TP attacks, which we propose below, work for F specific to both TCAV and FFV simultaneously by optimizing for an objective function that disrupts the fundamental mechanism by which TCAV, FFV, and other CBIMs work. As noted in the introduction, we centered our threat model around the positive tokens critical to CBIMs that, once perturbed, can cause persistent misinterpretation across numerous inputs. In contrast, a perturbation of an individual input image alone affects only the interpretation associated to that input.
4

---

Under review as a conference paper at ICLR 2022
Pc: (ariginal)
layer prior
f
fC
SALL
fe
R
f
fet
fr
f
Ug
layer attack
fa
fr
Pe (attacked)
F
f
2011
H-C
f
Figure 1: A schematic of the TP attack. Pc is the original set of positive examples of concept C, No the set negative examples of concept C, Uc are the unrelated examples that are used to calculate pr, and Pe is the set of positive examples after the attack. Note that positive examples get pulled toward unrelated concept example centroid pr, changing the direction of izc
3.2 ATTACKING TOKENS OF A CONCEPT
In this section we introduce the token pushing (TP) attack. The basic idea is simple; we find perturbations Dc = {6;}s that significantly alter model's internal representation of the concept tokens Pc { h. Using the notation developed in 3.1 we assume that the adversary has access copy of the defender's model (or surrogate model) f X Rd the hidden layer that the interpretation method will use, and write access to the set of tokens Pa that represent concept C.
The perturbations added to each element in Pe shifts its hidden representation in layer l so that it no longer correlates with concept C. In order to find point that can guide this shift, the first step for the adversary to choose some collection of images that are unrelated to C, Uc == {ar h The adversary calculates the centroid of fe(U)) which we denote by pr, which will serve representative of "unrelatedness" C. Then for each o. E Pc, the adversary uses PGD to compute
8, = arg min
(x + +6) urll-
(3)
Se
This is related to the hidden layer attacks described in (Wang et al. 2018 Inkawhich et al. 2019). 1119.44 A schematic the TP attack can be found in Figure] Examples of the perturbations can be found in Figure 7 in the Appendix.
In Section 4 we show that spite of the fact that Equation 3 neither the interpretability objective of TCAV nor FFV, it is still effective when applied to either method. In fact, objective function makes the TP attack more flexible since it acts against the underlying mechanism common both these and other interpretability methods: the spatial proximity of hidden representations of input that are semantically related. This means that the adversary does not need to know the specific interpretability method that the defender using. This also means that the attacker does not need access to the interpretation target, as they would they were to optimize against the interpretability objective directly.
EXPERIMENTS
To better understand the effectiveness of the methods proposed in Section 3.2 to TCAV and FFV in the case that they are used to interrogate an Inception model (Szegedy et al.) 2015) that has been trained ImageNet-1k (Deng et al. 2009). We choose Inception V1 because
apply attacks
5

---

Under review as a conference paper at ICLR 2022
Inception V1 Layer
Attacks
mixed3a
mixcd3b
mixed4a
mixed4b
Baseline TCAV (no attack)
0.69 + 0.02
0.90 1 0.01
0.66 1 0.03
0.68 1 0.04
Gaussian noise
0.61 + 0.02
0.62 + 0.02
0.64 + 0.03
0.67 + 0.04
PGD attack
Logit
0.37 + 0.02
0.37 + 0.03
0.35 + 0.02
0.33 + 0.03
mixed3a centroid
0.29 + 0.05
0.29 = 0.10
0.22 = 0.05
0.34 - 0.08
mixed3b centroid
0.17 = 0.05
0.39 0.19
0.19 0.03
0.37 0.08
mixed4a centroid
0.22 = 0.06
0.40 0.40
0.32
0.44 0.08
mixed4b centroid
0.27 + 0.07
0.32
0.33 0.06
0.42 + 0.08
mixedAc centroid
0.26 + 0.08
0.30 0.09
0.29
0.28 + 0.08
mixed4d centroid
0.28 + 0.08
0.30 = 0.10
0.25 + 0.06
0.18 + 0.10
Table I: The TCAV magnitude score for the zebra class on the 'striped' concept, before and after the TP attacks on InceptionVil V1. The Baseline TCAV row uses the concept sets with no perturbations. The rows below 'PGD attack on' indicate the layer that is being targeted by the TP attack. The columns are the Inception layer that TCAV being applied to. For all concept/pairs, We bold those values where the layer targeted by the TP attack and the layer TCAV is applied to are the same.
is a model commonly used in the interpretability literature (Kim et al. 2018 Olah et al. 2020) and choose ImageNet- Ik since it is easy obtain high-quality weights for this model/dataset combination. The token sets that we used to capture concepts come from the Describable Textures Dataset (DTD) (Cimpoi al. 2014]. We perform all PGD attacks with e 4/255 and 20 steps. For the CAV, we use linear classifier trained via stochastic gradient descent and l fz-regularization. See the Appendix for more experiment details.
To test the TP attack against TCAV, we choose concept/class pairs with straightforward associations: 'stripes'/'zebra', 'honeycombed' /'honeycoob'', as well as the 'scaly' concept with four separate snake classes: 'Green snake', 'Hognose snake', "Water snake', or 'King snake'. We perform the same experiment for all concept/class pairs, but for simplicity explain the procedure with the stripes'//zzebr concept/class pair. We select 70 sets of 50 randomly chosen images from ImageNet { which do not intersect. The same (N&- will be used for all concept/class pairs. We also fix a set of unrelated images Uc of size 1000 that are also randomly sampled from ImageNet. Finally, we choose random sets of 40 images from the classes 'stripes' P Pstriped from DTD. Dr is collection of images which the Inception VI model predicts as 'zebra'.
For each layer of InceptionVV we run the TP attack against Pstriped- For each of the resulting pairs (Pstriped, Pstripea) and each layer of InceptionVI, we then apply TCAV 70 times (once for each No,, calculating the difference in magnitude TCAV score between Pstriped and Patriped. Numerical results for 'stripes'/'zebra' can be found in T The smaller values indicate change in magnitude TCAV score before and after the attack. Plots of the raw TCAV magnitude scores for both the clean positive tokens and the attacked positive tokens (where each attack targeted different layer Inception I) are found for the "scaly" /'nlay concept pairs in Figure 2 Sample concept images before and after the attack, as well CAVs visualized with empirical DeepDream (Mordvintsev et al.) 2015) before and after the attack can be found in the appendix (notably, we find that besides some changes in coloring, DeepDream still produces visualizations that resemble the original concept, even when is applied to the perturbed tokens, providing another type of imperceptibility of this attack).
We include 95% confidence intervals for each layer based on the 70 different Ný sets. The point of this is to verify that the result does not depend on having the "right" negative examples. To test that the attack perturbations work for reasons other than the fact that they are perturbations, we also apply TCAV to positive token sets to which we have added random Gaussian noise with the per-channel mean and standard deviation of the PGD logit attack. Finally, note that we also include the results showing what happens when TP attack targets a different hidden layer than TCAV is being applied to (these are in the off-diagonal of Table T). A version of our results for relative TCAV can be found in Appendix A.2

---

Under review as a conference paper at ICLR 2022
senottiety sall
Hognose sensibiity soaly
0.
risall crtoi
nagotbde
riedlo centol
2.
niedth crtmd
risefi crnndd
0.2
muddle remred
cestrol
moodlo
mneed33
mardib
moedth
mieedAa
mordit
Water snale senalistty to scaly
King sralee Jenslisity scaly
Honeycomb sensiiiity to honeycombed
015
1.4
061
mixcdda
Thood3a
IncoptiorVV
IecepcorVV
InceptionV1
Figure 2: Adversarial attacks on the 'scaly" concept set for four snake classes in ImageNet- 1k, as well as the 'honeycombed" concept set for the honeycomb class. Each curve represents TP attack targeting different layer of Inception We use the same set of perturbations across each snake class. The r-axis records different layers of the InceptionV1 network, restricted to layers where the snake class is sensitive to the un-modified scaly concepts, according to the TCAV sign score. The y-axis is the TCAV magnitude score when TCAV is applied to that layer.
InceptionV1
Feature Visualizations for layer mixed4d, channel
(sstrppes))
Pc (dotsi),
Pe 1b==((lgg (xig-2ags
Faceted Feature Visualization
weighted by of with examples Pc
Pc (stripes, +.).
Pe -dots,
(xig-
Attacked Faceted Feature Visualization
weighted by of with examples Pc and perturbations 5
Figure 3: feature visualization without any concept prior (first row), faceted feature visualization on the same neuron for 'stripes', 'dots', 'zig-zags' (second row), and the faceted feature visualization after it has been attacked (third row) of channel 9 on Inception V1 layer mixed4d. While visualizations in the second reflect the concept prior, the visualizations in the third row do not (indicating the attack was successful). An example token perturbation for this attack is found in Figure
We evaluate the token perturbation attack on FFV by performing feature visualizations for Inception V1 on every channel neuron for the layers mixed3a, mixed3b, mixed4a, and mixed4b. We use the feature visualization objective equation 2 and compare feature visualizations with clean concept images Pc- concept images with Gaussian noise, and a concept set with perturbations created by PGD on the respective hidden layer with equation
g We give example of FFV output before and after attack
in layer mixed4d in Figure 3
We quantitatively test the effectiveness of the attack on FFV by using variant of the Fréchet Inception Distance (FID) (Heusel et al.) (017)) to measure the perceptual distance between feature visualizations. Namely, compare feature visualizations created with the channel objective (i.e., using only the
7

---

Under review as a conference paper at ICLR 2022
FV
0.00
1.2
FFV
1.08
0.00
1.0
0.8
FFV
0.96
0.26
0.00
0.8
Gaussian
0.73
1.21
1.14
0.00
TP attack
0.94
1.39
1.34
0.36
0.00
-02
FFV1
FFV
Gaussian TP attack
Figure 4: Fréchet Inception Distance scores for feature visualizations the layers mixed3a, mixed3b, mixed4a, and mixed4b. Five different feature visualizations are performed: channel feature visu- alization (FV), two separate runs of Faceted Feature Visualization with different sets of positive and negative concept images (FFV I and FFV2), with Gaussian noise added to the positive concept images (Gaussian), and with the token pushing attack (TP attack).
first term in equation 2) FFVs created with concepts Pc. FFVs with a set of stripe concepts Pe with a perturbation created via targeting the layer mixed3b with equation 3 and FFVs where we add Gaussian noise Pc. The FID score is calculated across layers for every channel neuron in Inception layers: mixed3a (256 channels), mixed3b (480 channels), mixed4a (512 channels), and mixed4b (512 channels), shown in Figure 4 We use a PyTorch implementation of FID (Seitzer (2020) and use the second block of Inception V3 as the visual similarity encoder (due the smaller dataset size).
of random images Pc and with clean stripe
RESULTS
Our results show that TP attacks effectively change the output of both TCAV and FFV from the baseline interpretability results. For TCAV, can consistently lower the TCAV magnitude score that indicates the relative importance of concept to an output class. In Table we measure the TCAV magnitude score on four early layers of Inception For each run and layer, take the average difference between the TCAV magnitude score for the striped concept set and random concept set over 70 sets of random images. We note that, unsurprisingly. attack success tends increase when the layer that an attack was developed for and the layer TCAV is being applied to are the same. However, we also find that the attack remains effective even when these are not the same. For example, in Table the attack targeting the layer 'mixed4b' is successful across all layers examined. We also observe this in Figure 2, where all attacks effectively modify the TCAV magnitudes on the 'scaly' concept for the 'snake' classes for all of the layers examined.
For FFV, we can observe the TP attack effectiveness from the visual differences between 1) a channel feature visualization (i.e., feature visualization that optimizes the first term in equation 2 2) the faceted feature visualization with a clean concept set Pc, and 3) the faceted feature visualization with perturbed concept set Pc. We give three such examples separately using the striped, dotted, and zig-zagged concept sets in Figure We use FID as a measure of visual difference, and test the effectiveness of the TP attack on FFV for the 1,760 channel neurons in the Inception layers 'mixed3a'. 'mixed3b', 'mixed4a", and 'mixed4b'. We use the striped concept set and perform two separate FFV visualizations for each neuron using different sets of negative concept set images. Figure 4 shows that the FID scores between the separate clean FFV runs is 0.26, while the FID score between the TP attack and the clean FFV runs are 1.39 and 1.34. The significantly larger FID scores suggest that the TP attack modifies the FFV output more than the variation between runs. This, along with visualizations such as 3 suggest that a TP attack can drastically change the semantic meaning associated with the feature visualizations produced by FFV.
Finally, we find that both the TCAV magnitudes (Table 1 and the FFV FID scores (Figure susceptible to Gaussian noise added to the concept set. This suggests that, even independent of adversarial attacks. CBIMs are brittle. This brittleness suggests that these methods also vulnerable to natural distribution shifts data, e.g., between the concept set and training images. We see a need for research into robust interpretability methods.
8

---

Under review as a conference paper at ICLR 2022
100
1.0
TCAV magnitude score
TCW live
0.8
0.8
certroe
mietth oenoii
miedks cwetoe
misadic centrod
misdlt ownrol
mietss certrod
nisaffa catooal
Block
(Rock Beck
Block Insture..oom
foctures MohieNetV2 layer
fratures
Residual block
Figure 5: TCAV sensitivity scores for the zebra class with the stripe images for ResNet-18 (left) (He et al. 2016) and MobileNetV2 (right) (Sandler et al. 2018) trained on ImageNet-1K, I.. The attacks uses perturbations made on the stripe concept images for InceptionVI using centroids for different hidden layers. All layers/blocks shown sensitive to the stripe concept before the attack, and are not sensitive after the attack.
5.1 TRANSFERRABILITY
As noted in 3.1 the knowledge required for an adversary to implement an attack is decreased significantly if they do not need to know the specific model being used by the defender. We therefore test the transferrability of the TP attack by applying TCAV to other model architectures trained on ImageNet: a ResNet-18 and MobileNetV2. We again consider the concept/class pair stripes/zebra with the same set of Nstripes. Ustripes. and Pstripes that were used for Table We compute the TCAV magnitude score for stripes/zebra for the four residual blocks in ResNet- 18 and the three layers in MobileNetV2 that were sensitive to the stripe concept according to signed TCAV. Figure ( compares baseline score with scores for TP attacks applied the different layers.
We find that the TP attacks targeting any of the 7 layers of Inception V1 result in significant decreases in TCAV magnitude score when applied to the first block of ResNet18 8 and all three layers of the MobileNetV2. The transfer TP attack does not seem to be effective against Block 2 and Block 4 of the Resnet-18 18. These results point toward TP attack being moderately transferable, especially when TCAV is being applied to earlier layers of the defender's model.
5.2 LIMITATIONS
In this work we chose two CBIMs test TP attacks on. While TCAV and FFV capture some of the diversity of such methods, they do not capture their full breadth. In particular, it would be useful to understand how TP attacks behave when they are applied to other types of feature visualization methods, namely those that average over large number of images or activations (Nguyen et al. 20116b; Carter al. 2019) to build a visualization. Further, while we only consider image classification models, TCAV is agnostic to modality. Evaluating interpretability method brittleness in other critical modalities such as NLP would give a more complete picture of these method's vulnerabilities. Finally, focus on perturbations to positive concept tokens. To fully understand the attack surfaces CBIMs, makes sense to consider attacks on the other inputs to a method: the model itself, negative examples, and the interpretation targets. As limited example, an adversarial attack may be designed to be 'triggered" for only certain concept and dataset interpretation combinations.
CONCLUSION
In this work we show that concept-based interpretability methods, like much of the deep learning modeling pipeline, are vulnerable to adversarial attacks. By subtly changing the examples concept that user wishes to use to interrogate model, an adversary can induce radically different interpretations. The attacks we describe are general enough that they work for multiple interpretability methods without modification (FFV and TCAV). We hope that the results of this paper will promote better security practices, not only around the model pipeline itself, but also around the method that is being used interpret the model.

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY STATEMENT
In the interest of making our results reproducible and able to be easily expanded upon, we make our codebase available to the public, including our implementations of the centroid PGD attack and the faceted feature visualization we used. We also include attack and evaluation scripts with sensible defaults and examples. Finally, we provide the data used throughout this paper, including our feature visualizations. This entire repository will be available on public GitHub repository once the anonymous review period has completed.
ETHICS STATEMENT
In this work we highlight the vulnerability of class of popular interpretability methods to adversarial attack. We chose explore a threat model wherein the positive tokens for concept are perturbed. This is of particular concern because (unlike individual input) positive tokens will often be centralized and used collectively by researchers and practitioners many times. Because of this, attack on a single data source may have wide-ranging effects. We hope that by better understanding and communicating this specifie threat interpretability, can motivate researchers use best practices around security for interpretability and explainability as they are already encouraged to do for dataset and model creation.
REFERENCES
Julius Adebayo. Justin Gilmer, Ian Goodfellow, and Been Kim. Local explanation methods for deep neural networks lack sensitivity to parameter values. arXiv preprint arXiv: 1810.03307 201 88
Christopher Anders, Plamen Pasliev, Ann- Mat-Kathrin Dombrowski, KlaustRobert Müller, and Pan Kessel. Fairwashing explanations with off-manifold detergent. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 314-323. PMLR, 13-18 Jul 2020. URL https://proceddnng mlr www.:mmr...rreeeee.mmmm press /vtpv/v1////////// http://////////ne..aaaaamma www.aanders2aaaaa..mma
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint 100.v:1902.06705, 2019.
Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah, Activation atlas. Distill, 2019. doi: 10.23991/diistil..00000... hutps//disstill, wutp/0199/acciaaaatiiiiin
Chun-Har Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In 7th International Conference on Learning Repre- sentations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL htttps://openeeview net/ forum? i B-BIMMZ20CYY,
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2014.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp.
6967-6976, 2017. URL https: https::/ppoceeeiinnnt htw.:ooceeednnns...uiiu......p.. neurips CC/ httppppapppp//1mmma2 0060ef47bb10 60b91 (ateee919998222bb4b44444404 44st 44ra4a/222222211130002206664666-
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition,
pp. 248-255. leee, 2009.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In Proceedings of the IEEE international conference on computer vision, pp. 3429-3437 2017.
10

---

Under review as a conference paper at ICLR 2022
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3681-3688 2019.
Nick Cammarata and Chelsea Voss Goh, Gabriel and, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10.239 1233915/dba /5/distil.00030. http::/ddistill.pub/22 httppp/////ullelteeee //tm//mmmalllmeeetteeeeaeo -eurons...
Mara Graziani, Vincent Andrearezyk, and Henning Müller. Regression concept vectors for bidirec- tional explanations in histopathology. In Understanding and Interpreting Machine Learning in Medicul Image Computing Applications, pp. 124-132. Springer, 20.8a.
Mara Graziani, Vincent Andrearczyk, and Henning Müller. Regression Concept Vectors for Bidi- rectional Explanations in Histopathology. In Danail Stoyanov, Zeike Taylor, Seyed Mostafa Kia, Ipek Oguz, Mauricio Reyes, Anne Martel, Lena Maier-Hein, Andre F. Marquand, Edouard Duchesnay, Tommy Löfstedt, Bennett Landman, M. Jorge Cardoso, Carlos A. Silva, Sergio Pereira, and Raphael Meier (eds.), Understanding and Interpreting Machine Learning in Medical Image Computing Applications, pp. 124-13344 132, Cham, 2018b. Springer International Publishing. ISBN
Mara Graziani, James M Brown, Vincent Andrearczyk, Veysi Yildiz, J Peter Campbell, Deniz Erdogmus, Stratis Ioannidis, Michael F Chiang, Jayashree Salpathy-Cramen and Henning Müller. Improved interpretability for computer-aide aided severity assessment retinopathy of prematurity. In Medical Imaging 2019 Computer-Aider Diagnosis, volume 10950, pp. 109501R. International Society for Optics and Photonics, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversarial model manipulation. Advances in Neural Information Processing Systems. 32:2925-2992 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berhaard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jian Huang, Junyi Chai, and Stella Cho. Deep learning in finance and banking: A literature review and classification. Frontiers of Business Research in China, 14:1-24, 2020.
Nathan Inkawhich, Wei Wen, Hai Helen Li, and Yiran Chen. Feature space perturbations yield more transferable adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7066-7074, 2019.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-2677 PMLR, 2018.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5338-5348. PMLR, 13-18 Jul 2020. URL htttps:://poceeeeeeeee ml www.s/essvllv... 9 9/koh20a httm
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang. Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz- Richardson, Captum: unified and generie model interpretability library for pytorch, 2020.
Himabindu Lakkaraju, Nino Arsov, and Osbert Bastani. Robust and stable black box explanations. In International Conference on Machine Learning, pp. 5628-5638, PMLR, 2020.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521((553):436 444, 2015.
11

---

Under review as a conference paper at ICLR 2022
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Aleksander Madry. Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 2015.
Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM international conference Multimedia, pp. 1485-1488, 2010.
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition chess knowledge in alphazero, 2021.
Diana Mincu, Eric Loreaux, Shaobo Hou, Sehastien Baur, Ivan Protsyuk, Martin Seneviratne, Anne Mottram, Nenad Tomasev, Alan Karthikesalingam, and Jessica Schrouff. Concept-based model explanations for electronic health records. In Proceedings of the Conference on Health, Inference, and Learning, CHIL '21, pp. 36-46, New York. NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383592. doi: 10.1145/3450039..455558 3455549999933998888 URL https //doi.org/10 110.00.11555444530 3451858
Riccardo Miotto, Fei Wang. Shuang Wang, Xiaoqian Jiang, and Joel T Dudley. Deep learning for healthcare: review, opportunities and challenges. Briefings bioinformatics, 1966)1126-1124, 2018.
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Deepdream-a code example for visualizing neural networks. Google Research, 2(5), 2015.
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems, 29:3387-3395, 2016a.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. arXiv preprint arXiv:1602.03616 2016b.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 10.2391 5/disti11.00007.. https://ddstilll.pue 7ff
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018. doi: 10.2391//distill. 00010. https://distill.pubb pubpi//ttilllll/llloo hwtp://////iiiingngggn.. b/buikding-blocks.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
An overview of early vision in inceptionvl. Distill, 2020. doi: 103311//istill.000222
http::/ddiiillllpp//220ceciieaii
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Model-agnostic interpretability of machine learning. arXiv preprint arXiv: 05386, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510 4520, 2018.
Maximilian Seitzer. sytorch-fid: FID Score for PyTorch. htttps::/ggithuu.comm/
pyyorch-fiie August 2020. Version 0.1.1.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
12

---

Under review as a conference paper at ICLR 2022
Samuel Henrique Silva and Peyman Najafirad. Opportunities and challenges in deep learning adversarial robustness: A survey. arXiv preprint arXiv:2007.00533, 2020.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, (5077676):35-3335 2017.
Akshayvarun Subramanya, Vipin Pillai, and Hamed Pirsiavash. Fooling network interpretation in image classification. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2020-2029, 2019.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pp. 3319-3328, PMLR, 2017.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 13 2013.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference Learning Representations, ICLR 2014, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Tom Viering, Ziqi Wang, Marco Loog, and Elmar Eisemann. How to manipulate cnns to make them lie: the gradeam case. arXiv preprint arXiv: /90iv 10901, 2019.
Bolun Wang, Yuanshun Yao, Bimal Viswanath, Haitao Zheng, and Ben Zhao. With great training comes great vulnerability: Practical attacks against transfer learning. In 27th {USENIX) Security Symposium ( (USENIX) Security 18), pp. 1281-1297, 2018.
Shen Wang and Yuxin Gong. Adversarial example detection based on saliency map features. Applied Intelligence, pp. 1-14, 2021.
Donglai Wei, Bolei Zhou, Antonio Torrabla, and William Freeman. Understanding intra-class knowledge inside cnn. arXiv preprint arXiv:1507.02377 2015.
Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-L-angg Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept- based explanations in deep neural networks. Advances in Neural Information Processing Systems, 33, 2020.
Zhun Zhang, Qihe Liu, and Shijie Zhou. Ggead: novel method of adversarial detection by guided grad-cam. In International Conference on Wireless Algorithms, Systems, and Applications, pp. 172-182. Springer, 2021.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 119-134, 2018.
A APPENDIX
A.1 EXPERIMENT DETAILS
We use InceptionVI, 1, ResNet-18, and MobileNetV2 models with pretrained weights from Torchvision Marcel Rodriguez 2010). To run TCAV, FFV, and our attacks, we use PyTorch with an NVIDIA Tesla T4 GPU provided with Google Colab Pro well as single NVIDIA Tesla P100 GPU. We use the Captum (ooklikyaan et al. 2020) implementation of TCAV with a linear classifier trained via stochastic gradient descent and rerggllarization
For the Faceted Feature Visualizations, we start with random noise and parameterize the image Fourier basis (Olah et al. 2017). We random scaling, rotation, color, and shift transformations.
13

---

Under review as a conference paper at ICLR 2022
Relative TCAV scores before attack
Relative TCAV magnitud score
mi meddd
miseC4e
Relative TCAV after attack
mixedic
misec4d
misedde
InceptionV1 layer
Figure 6: Relative TCAV magnitude scores before (top) and after (bottom) the PGD logit attack the striped concept images for the striped, zig-zagged, and dotted concept sets.
A.2 TP ATTACK RELATIVE TCAV
Here, we give example experiment showing that TP attacks are also effective for a variant of TCAV, using relative TCAV scores. The results Figure use concept sets for stripes, ziz-zags, and polka-dots of 35 images each. Perturbations are made on the striped concept set using the final logit layer, towards an unrelated class (the 999 'toilet tissue' ImageNet- Ik class).
A.3 EMPIRICAL DEEPDREAM WITH THE CAVS
We empirical DeepDream in Figure & visualize the effect of the hidden layer PGD attack the Concept Activation Vectors Mordvintsev et al.) 2015 Kim et al. 2018). We consider CAVs for the hidden layers mixed3b and mixed4b of Inception VI. We use images from the striped, honeycombed. and scaly concept sets, and use perturbations found via Projected Gradient Descent attack on the hidden layer mixed4d. We use cosine similarity (Carter et al.) (019) for the feature visualization objective. We use the same Fourier parameterization and transtormations we used for the Faceted Feature Visualization.
We note that the visualizations for the attacked CAV tend qualitatively resemble those of the CAV without the attack, albeit with unnatural and colors. It has been proposed that DeepDream can confirm that CAVs represent the concept of images (Kim al. 2018). Given the success of our adversarial attack, using DeepDream as a qualitative concept check for CAV may therefore be misleading and provides evidence for the imperceptibility of the token pushing attack.
14

---

Under review as a conference paper at ICLR 2022
Figure Example stripe concept images before (left) and after (right) TP attack. We use c= 4/255 and 20 iterations for all PGD experiments. The perturbation shown targets Inception V layer mixed3a.
15

---

Under review as a conference paper at ICLR 2022
Striped concept DeepDream
Honeycombed concept DeepDream
mixed3b
mixed4b
mixed3b
mixed4b
Concept
Activation
Vector
Attacked
Concept
Activation
Vector
Scaly concept DeepDream
mixed3b
mixed4b
Concept Activation
Vector
Attacked
Concept Activation
Vector
Figure 8: Empirical Deepdream Mordvintsev et al. (2015) with the Concept Activation Vectors (CAVs) computed with 1) the normal concept sets the "Concept Activation Vector' rows and 2) the perturbed concept sets in the 'Attacked Concept Activation Vector' rows. For the attacked concept sets, we use the PGD attack performed on the hidden layer mixed4d.
16


Reviews:
[{"review_id": "eQTowh91ojN", "reviewer": "Reviewer_YeZA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the paper focuses on an interesting problem, it does not seem to add anything new to our understanding of robustness of interpretability methods. As the paper itself mentions, the adversarial attacks on representations of the networks, and adversarial attacks on feature-based interpretability methods have already been out there. The only new thing that the paper seems to do is to apply the adversarial attacks on concept-based interpretability methods. This little amount of novelty, paired with the fact that the empirical results are not really surprising, means that the paper does not meet the bar for ICLR.\n\nMore in detail, are there any fundamental differences between the attacks on feature-based interpretability methods (e.g., [Dombrowski et al.](https://proceedings.neurips.cc/paper/2019/file/bb836c01cdc9120a9c984c525e4b1a4a-Paper.pdf)) and the proposed attack on the hidden representations? Do we expect one attack to be harder than the other? An empirical or a theoretical analysis (see Dombrowski et al, or Adebayo et al) would have been greatly helpful. Without such insights, the contributions of the paper seems limited.", "summary_of_the_paper": "The paper proposes an adversarial attack against concept based interpretability methods. The key idea is to change the hidden layer representation at which the concept-based method is being applied in an adversarial manner. The results show that the proposed method is indeed able to fool TCAV and FFV methods.", "summary_of_the_review": "Interesting idea, but lacks novelty.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QYFtIufPOL4", "reviewer": "Reviewer_6y3k", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies an important problem, i.e., the robustness of interpretation methods. Most of the existing works on adversarial robustness focus on deep learning models themselves. Litter attention has been drawn on interpretation methods. This paper works on this needed problem and proposes an interesting attack on existing interpretation methods. Despite providing some useful insights regarding discovering the vulnerabilities of interpretation methods, IMHO, this paper has not reached the bar of ICLR due to the following limitations. \n\n1. The literature review of this paper is incomplete regarding both explanation methods and attacks on explanation methods. Regarding explanation methods, the paper fails to include perturbation-based/counterfactual-based white-box explanation methods (e.g., 1-3) and the large body of black-box explanation methods (e.g., 4-6). Regarding the attacks on interpretation methods, this paper misses the following works: 7-10, although they do not all target concept-based methods. \n\n2. The main claim of this paper is not that accurate. This is not the first work that studies the robustness issue of concept-based explanation methods. 10 has discussed the robustness of concept-based against a type of attack. \n\n3. Despite its correctness, the novelty and depth of the proposed technique are limited. Compared with the original PGD, the proposed method is just an application of PGD to a different type of classifier and input. IMHO, such a technical contribution does not reach the bar of a top-tier ML conference. This also raises another question: Why not choose other existing adversarial attack methods other than PGD. I would suggest the authors justify their design choice. \n\n4. Regarding evaluation, this work lacks some potential comparison baselines and target methods. Regarding the target methods, the authors chose TCAV and another work that was recently published on distill. I would suggest the authors replace TCAV with more advanced methods with a similar core idea, i.e., 10-11. Regarding the comparison baselines, I would suggest the authors discuss and even evaluate why existing attacks pointed by me cannot be directly used, or applied through minor changes, to the concept-based methods targeted in this work. At least, the authors should discuss and compare with the attack mentioned in 10. \n\n5. I would suggest the authors discuss and even evaluate potential countermeasures of the proposed attack. \n\n6. The writing quality of this paper is limited. There is a branch of grammar errors and some sentences are hard to understand. I would suggest the authors run a grammar check and proofread the paper to ensure readability. \n\n\n**Citations:** \n1. Interpretable Explanations of Black Boxes by Meaningful Perturbation, ICCV 17.\n2. Real time image saliency for black box classifiers, NeurIPS 17.\n3. Explaining Image Classifiers by Counterfactual Generation, ICLR 19.\n4. \"Why Should I Trust You?\": Explaining the Predictions of Any Classifier, KDD 16.\n5. Explaining Deep Learning Models -- A Bayesian Non-parametric Approach, NeurIPS 18.\n6. Learning to Explain: An Information-Theoretic Perspective on Model Interpretation, ICML 18.\n7. Interpretable Deep Learning under fire, Usenix 20.\n8. Fairwashing explanations with off-manifold detergent, ICML 20.\n9. Robust and Stable Black Box Explanations, ICML 20.\n10. Concept Bottleneck Models, ICML 20.\n11. On Completeness-aware Concept-Based Explanations in Deep Neural Networks, NeurIPS 20.\n\n\n", "summary_of_the_paper": "This paper proposes a novel adversarial attack against concept-based explanation methods. Technically, it utilizes a widely used attack method on DNN - PGD to generate perturbations. Adding these perturbations to concepts will mislead the target explanation methods to pinpoint meaningless parts in the input as important features.  Evaluation on a CNN model trained on the ImageNet dataset demonstrates the effectiveness of the proposed method. ", "summary_of_the_review": "IMHO, I am afraid this paper has not reached the bar of ICLR due to the following limitations. Detailed comments and suggestions can be found above.\n\n1. The literature review of this paper is incomplete regarding both explanation methods and attacks on explanation methods. \n\n2. The main claim of this paper is not that accurate. As discussed above, this is not the first work that studies the robustness issue of concept-based explanation methods. It has been studied in previous works. \n\n3. The technical contribution of this paper is thin.\n\n4. Regarding evaluation, this work lacks some potential comparison baselines.\n\n5. Finally, I would suggest the authors discuss and even evaluate potential countermeasures of the proposed attack. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0vy3WG9-6bD", "reviewer": "Reviewer_jcjC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\nThe paper tackles a novel problem of adversarial robustness of concept-based explanations, and the proposed token pushing attacks could be of interest to the community. The proposed method is simple, and the paper is generally clear and well-written. \n\n---\n\n### Weaknesses:\n\n- The motivation of the proposed method is unclear. The paper perturbs the hidden representations to demonstrate the concept-based explanations' vulnerability; however, the injected perturbation is not imperceptible. For instance, the attacked FFV in Figure 3 clearly shows the presence of the perturbation, which limits the practicality of the proposed attack framework. Therefore, I suggest evaluating with lower $\\epsilon$ and $\\ell_2$ or $\\ell_1$ norm to make the perturbation sparse and imperceptible.\n- In light of the above concern, existing adversarial detectors [1,2] can defend against the proposed attack. Can the authors validate their proposed attack against these adversarial detectors?\n- Further, it would have been more effective to inject imperceptible perturbation in the input space to show that the concept-based explanations are vulnerable, as concept-based explanations are comparatively more robust to adversarial attacks in the input space (see Appendix A [3])\n- The experimental evaluation is limited to two concept/class pairs on a single dataset, making it difficult to conclude the effectiveness of the proposed method. Therefore, I recommend evaluating multiple datasets (e.g., OAI or CUB datasets have concept attributes) and recent concept-based methods [4,5].\n\n\n***Other questions***\n- What is the significance of negative values in Table 1? Instead of reporting the change in magnitude, it might be more useful to evaluate the change in prediction accuracy after the attack on concepts following prior works [5].\n- The results for transferability are intriguing; can the authors report the effect of transferability on different architectures?\n- Can the authors show the visualization after the attack on the relative TCAV shown in Figure 6? Does the visualization resemble the dotted concept after the attack?\n\n---\n\n### References\n[1] Roth and Khilcher et al. The Odds are Odd: A Statistical Test for Detecting Adversarial Examples. ICML 2019.  \n[2] Yang et al. ML-LOO: Detecting Adversarial Examples with Feature Attribution. AAAI 2020.  \n[3] Kim et al. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). ICML 2018.   \n[4] Yeh et al. On Completeness-aware Concept-Based Explanations in Deep Neural Networks. NeurIPS 2020.  \n[5] Koh et al. Concept Bottleneck Models. ICML 2021.  ", "summary_of_the_paper": "The paper describes the threat model for concept-based interpretability methods and mainly investigates the vulnerability of TCAV and faceted feature visualization. It introduces Token Pushing (TP) attacks, which learn small perturbations for the token of concept leading to different output for the interpretability method. The proposed attack is evaluated on pre-trained ImageNet models on the Describabke Texture Dataset for concept tokens.", "summary_of_the_review": "In the current form, I recommend rejection due to the above-mentioned reasons. In particular, the paper needs to clearly justify the motivation of the work with a stronger experimental evaluation to demonstrate the effectiveness of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0Rwy9j1UYFI", "reviewer": "Reviewer_oy63", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n**Novelty**: to the best of my knowledge, the paper is the first adversarial attack for concept-based explanations. \n\n**Clarity**: the paper is generally well-written and clear. I particularly appreciate that the authors are very clear about the goals of the method, and assumptions about the knowledge of the adversary.\n\n-----\nWeaknesses:\n\n**Experiments**: Overall, I would have liked to see more rigorous experimentation. \n\nPart of the experiments are well-designed -- e.g., the results in Figure 3, which demonstrate the change in feature visualization due to adversarial concepts is an interesting qualitative experiment. \n\nHowever, some aspects of the method are not evaluated in the experiments.  For example, \"Does the adversarial attack change the *relevance* of the concepts in the dataset?\" This could be evaluated by investigating the change in the TCAV scores -- \"are zebras no longer related to stripes after we attack the concept stripes\" (assuming the model previously used this relationship)? The authors demonstrate that it changes the magnitude of the TCAV scores (in Figure 2). However, in the original TCAV paper, the sign of the score (rather than the magnitude is used). It's not directly clear to me unsure a larger magnitude directly implies a lower TCAV score.\n\nThe experiments are performed on a relatively small scale. The attack is evaluated on one dataset and predominantly on one model, InceptionV1 (although there is one experiment of the adversarial examples from InceptionV1 to ResNet-18). Further, the authors demonstrate the effect of the attack on two concepts (stripes and dots). I would have liked to see more datasets and more concepts. \n\n*Significance of the results*: The authors have provided confidence intervals for their findings, which is great. However, often the confidence intervals are quite large, leading to statistical insignificance. Nevertheless, the authors state that \"the attack targeting the layer mixed4b is successful across all layers considered\", despite the large confidence intervals. Similarly, it's unclear whether the results in Figure 5 are significant (without seeing the error bars).  I'd like to encourage the authors to be more cautious in the conclusions they draw.  \n\n\n**Design/Method** : If I understand the method correctly, it assumes the adversarial examples are included before computing the concept vector $v_c^l$. However, this seems a bit strange for an adversarial attack -- normally, the model (including the interpretability method) is assumed to be fixed. I would've expected the method to focus on trying to change the explanation, e.g. the TCAV score, for fixed concept vectors. I would be curious to hear from the authors why the approach in the paper is more relevant and/or interesting than the aforementioned alternative? \n\n----- \nMinor comments:\n\nCitations:\n- The citation for adversarial examples (in section 3, final paragraph page 3) should be Szegedy et al., 2013 not Goodfellow et al., 2015. \n- the discussion of some relevant work is missing -- see list below.\n\n\n[1] Slack, Dylan, et al. \"Fooling LIME and SHAP: Adversarial attacks on post hoc explanation methods.\" Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 2020.\n\n[2] Ghorbani, Amirata, Abubakar Abid, and James Zou. \"Interpretation of neural networks is fragile.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\n\n[3] Dombrowski, Ann-Kathrin, et al. \"Explanations can be manipulated and geometry is to blame.\" arXiv preprint arXiv:1906.07983 (2019). \n\n[4] Anders, Christopher, et al. \"Fairwashing explanations with off-manifold detergent.\" International Conference on Machine Learning. PMLR, 2020.\n", "summary_of_the_paper": "The authors derive a new adversarial attack, token pushing attack, that targets concept-based explanations. The adversarial perturbations are added to the concepts, resulting in a shift of the internal representation of a concept. The authors provide experiments that show that the adversarial attack changes the concept vectors and their visualization.", "summary_of_the_review": "Overall, I'm recommending a borderline reject. \nI think that the general concept of creating adversarial attacks for concept-based methods is interesting and relatively novel. However, the implementation of the idea requires assuming that the interpretability model will be re-trained, which limits the applicability of the method. Further, as a paper that introduces a new method, I think that more experimentation is required to demonstrate its effectiveness. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work demonstrates that adversarially perturbing inputs can change the output of concept based explainability tools. Reviewers generally agreed that the writing was clear and the experiments were easily understood. Regarding novelty, reviewers noted that there are several existing works which study the adversarial robustness of explainability tools (one even has experiments specifically on concept based explainability tools). As a result, there is not much novelty in the finding that concept based explainability tools are sensitive to adversarial perturbation. Regarding the technical contribution of the algorithm, it is expected that standard optimization approaches (e.g. PGD) would be sufficient to break concept based explainability tools so there is not a clear technical challenge being solved in the work.

The work could be improved by refocusing the robustness analysis to derive new insights regarding the behavior of concept based explainability tools. In doing so, it would be beneficial to deemphasize the claims regarding novel security concerns---these methods don't even work reliably in non-adversarial settings, as evidenced by poor out-of-distribution robustness. It is expected that performance will be even worse under adversarial settings.
Decision: Reject