Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an adversarial attack against concept based interpretability methods. The key idea is to change the hidden layer representation at which the concept-based method is being applied in an adversarial manner. The results show that the proposed method is indeed able to fool TCAV and FFV methods.
main_review: While the paper focuses on an interesting problem, it does not seem to add anything new to our understanding of robustness of interpretability methods. As the paper itself mentions, the adversarial attacks on representations of the networks, and adversarial attacks on feature-based interpretability methods have already been out there. The only new thing that the paper seems to do is to apply the adversarial attacks on concept-based interpretability methods. This little amount of novelty, paired with the fact that the empirical results are not really surprising, means that the paper does not meet the bar for ICLR.

More in detail, are there any fundamental differences between the attacks on feature-based interpretability methods (e.g., [Dombrowski et al.](https://proceedings.neurips.cc/paper/2019/file/bb836c01cdc9120a9c984c525e4b1a4a-Paper.pdf)) and the proposed attack on the hidden representations? Do we expect one attack to be harder than the other? An empirical or a theoretical analysis (see Dombrowski et al, or Adebayo et al) would have been greatly helpful. Without such insights, the contributions of the paper seems limited.
summary_of_the_review: Interesting idea, but lacks novelty.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors derive a new adversarial attack, token pushing attack, that targets concept-based explanations. The adversarial perturbations are added to the concepts, resulting in a shift of the internal representation of a concept. The authors provide experiments that show that the adversarial attack changes the concept vectors and their visualization.
main_review: Strengths:

**Novelty**: to the best of my knowledge, the paper is the first adversarial attack for concept-based explanations. 

**Clarity**: the paper is generally well-written and clear. I particularly appreciate that the authors are very clear about the goals of the method, and assumptions about the knowledge of the adversary.

-----
Weaknesses:

**Experiments**: Overall, I would have liked to see more rigorous experimentation. 

Part of the experiments are well-designed -- e.g., the results in Figure 3, which demonstrate the change in feature visualization due to adversarial concepts is an interesting qualitative experiment. 

However, some aspects of the method are not evaluated in the experiments.  For example, "Does the adversarial attack change the *relevance* of the concepts in the dataset?" This could be evaluated by investigating the change in the TCAV scores -- "are zebras no longer related to stripes after we attack the concept stripes" (assuming the model previously used this relationship)? The authors demonstrate that it changes the magnitude of the TCAV scores (in Figure 2). However, in the original TCAV paper, the sign of the score (rather than the magnitude is used). It's not directly clear to me unsure a larger magnitude directly implies a lower TCAV score.

The experiments are performed on a relatively small scale. The attack is evaluated on one dataset and predominantly on one model, InceptionV1 (although there is one experiment of the adversarial examples from InceptionV1 to ResNet-18). Further, the authors demonstrate the effect of the attack on two concepts (stripes and dots). I would have liked to see more datasets and more concepts. 

*Significance of the results*: The authors have provided confidence intervals for their findings, which is great. However, often the confidence intervals are quite large, leading to statistical insignificance. Nevertheless, the authors state that "the attack targeting the layer mixed4b is successful across all layers considered", despite the large confidence intervals. Similarly, it's unclear whether the results in Figure 5 are significant (without seeing the error bars).  I'd like to encourage the authors to be more cautious in the conclusions they draw.  


**Design/Method** : If I understand the method correctly, it assumes the adversarial examples are included before computing the concept vector $v_c^l$. However, this seems a bit strange for an adversarial attack -- normally, the model (including the interpretability method) is assumed to be fixed. I would've expected the method to focus on trying to change the explanation, e.g. the TCAV score, for fixed concept vectors. I would be curious to hear from the authors why the approach in the paper is more relevant and/or interesting than the aforementioned alternative? 

----- 
Minor comments:

Citations:
- The citation for adversarial examples (in section 3, final paragraph page 3) should be Szegedy et al., 2013 not Goodfellow et al., 2015. 
- the discussion of some relevant work is missing -- see list below.


[1] Slack, Dylan, et al. "Fooling LIME and SHAP: Adversarial attacks on post hoc explanation methods." Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 2020.

[2] Ghorbani, Amirata, Abubakar Abid, and James Zou. "Interpretation of neural networks is fragile." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.

[3] Dombrowski, Ann-Kathrin, et al. "Explanations can be manipulated and geometry is to blame." arXiv preprint arXiv:1906.07983 (2019). 

[4] Anders, Christopher, et al. "Fairwashing explanations with off-manifold detergent." International Conference on Machine Learning. PMLR, 2020.

summary_of_the_review: Overall, I'm recommending a borderline reject. 
I think that the general concept of creating adversarial attacks for concept-based methods is interesting and relatively novel. However, the implementation of the idea requires assuming that the interpretability model will be re-trained, which limits the applicability of the method. Further, as a paper that introduces a new method, I think that more experimentation is required to demonstrate its effectiveness.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel adversarial attack against concept-based explanation methods. Technically, it utilizes a widely used attack method on DNN - PGD to generate perturbations. Adding these perturbations to concepts will mislead the target explanation methods to pinpoint meaningless parts in the input as important features.  Evaluation on a CNN model trained on the ImageNet dataset demonstrates the effectiveness of the proposed method. 
main_review: This paper studies an important problem, i.e., the robustness of interpretation methods. Most of the existing works on adversarial robustness focus on deep learning models themselves. Litter attention has been drawn on interpretation methods. This paper works on this needed problem and proposes an interesting attack on existing interpretation methods. Despite providing some useful insights regarding discovering the vulnerabilities of interpretation methods, IMHO, this paper has not reached the bar of ICLR due to the following limitations. 

1. The literature review of this paper is incomplete regarding both explanation methods and attacks on explanation methods. Regarding explanation methods, the paper fails to include perturbation-based/counterfactual-based white-box explanation methods (e.g., 1-3) and the large body of black-box explanation methods (e.g., 4-6). Regarding the attacks on interpretation methods, this paper misses the following works: 7-10, although they do not all target concept-based methods. 

2. The main claim of this paper is not that accurate. This is not the first work that studies the robustness issue of concept-based explanation methods. 10 has discussed the robustness of concept-based against a type of attack. 

3. Despite its correctness, the novelty and depth of the proposed technique are limited. Compared with the original PGD, the proposed method is just an application of PGD to a different type of classifier and input. IMHO, such a technical contribution does not reach the bar of a top-tier ML conference. This also raises another question: Why not choose other existing adversarial attack methods other than PGD. I would suggest the authors justify their design choice. 

4. Regarding evaluation, this work lacks some potential comparison baselines and target methods. Regarding the target methods, the authors chose TCAV and another work that was recently published on distill. I would suggest the authors replace TCAV with more advanced methods with a similar core idea, i.e., 10-11. Regarding the comparison baselines, I would suggest the authors discuss and even evaluate why existing attacks pointed by me cannot be directly used, or applied through minor changes, to the concept-based methods targeted in this work. At least, the authors should discuss and compare with the attack mentioned in 10. 

5. I would suggest the authors discuss and even evaluate potential countermeasures of the proposed attack. 

6. The writing quality of this paper is limited. There is a branch of grammar errors and some sentences are hard to understand. I would suggest the authors run a grammar check and proofread the paper to ensure readability. 


**Citations:** 
1. Interpretable Explanations of Black Boxes by Meaningful Perturbation, ICCV 17.
2. Real time image saliency for black box classifiers, NeurIPS 17.
3. Explaining Image Classifiers by Counterfactual Generation, ICLR 19.
4. "Why Should I Trust You?": Explaining the Predictions of Any Classifier, KDD 16.
5. Explaining Deep Learning Models -- A Bayesian Non-parametric Approach, NeurIPS 18.
6. Learning to Explain: An Information-Theoretic Perspective on Model Interpretation, ICML 18.
7. Interpretable Deep Learning under fire, Usenix 20.
8. Fairwashing explanations with off-manifold detergent, ICML 20.
9. Robust and Stable Black Box Explanations, ICML 20.
10. Concept Bottleneck Models, ICML 20.
11. On Completeness-aware Concept-Based Explanations in Deep Neural Networks, NeurIPS 20.



summary_of_the_review: IMHO, I am afraid this paper has not reached the bar of ICLR due to the following limitations. Detailed comments and suggestions can be found above.

1. The literature review of this paper is incomplete regarding both explanation methods and attacks on explanation methods. 

2. The main claim of this paper is not that accurate. As discussed above, this is not the first work that studies the robustness issue of concept-based explanation methods. It has been studied in previous works. 

3. The technical contribution of this paper is thin.

4. Regarding evaluation, this work lacks some potential comparison baselines.

5. Finally, I would suggest the authors discuss and even evaluate potential countermeasures of the proposed attack.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper describes the threat model for concept-based interpretability methods and mainly investigates the vulnerability of TCAV and faceted feature visualization. It introduces Token Pushing (TP) attacks, which learn small perturbations for the token of concept leading to different output for the interpretability method. The proposed attack is evaluated on pre-trained ImageNet models on the Describabke Texture Dataset for concept tokens.
main_review: ### Strengths
The paper tackles a novel problem of adversarial robustness of concept-based explanations, and the proposed token pushing attacks could be of interest to the community. The proposed method is simple, and the paper is generally clear and well-written. 

---

### Weaknesses:

- The motivation of the proposed method is unclear. The paper perturbs the hidden representations to demonstrate the concept-based explanations' vulnerability; however, the injected perturbation is not imperceptible. For instance, the attacked FFV in Figure 3 clearly shows the presence of the perturbation, which limits the practicality of the proposed attack framework. Therefore, I suggest evaluating with lower $\epsilon$ and $\ell_2$ or $\ell_1$ norm to make the perturbation sparse and imperceptible.
- In light of the above concern, existing adversarial detectors [1,2] can defend against the proposed attack. Can the authors validate their proposed attack against these adversarial detectors?
- Further, it would have been more effective to inject imperceptible perturbation in the input space to show that the concept-based explanations are vulnerable, as concept-based explanations are comparatively more robust to adversarial attacks in the input space (see Appendix A [3])
- The experimental evaluation is limited to two concept/class pairs on a single dataset, making it difficult to conclude the effectiveness of the proposed method. Therefore, I recommend evaluating multiple datasets (e.g., OAI or CUB datasets have concept attributes) and recent concept-based methods [4,5].


***Other questions***
- What is the significance of negative values in Table 1? Instead of reporting the change in magnitude, it might be more useful to evaluate the change in prediction accuracy after the attack on concepts following prior works [5].
- The results for transferability are intriguing; can the authors report the effect of transferability on different architectures?
- Can the authors show the visualization after the attack on the relative TCAV shown in Figure 6? Does the visualization resemble the dotted concept after the attack?

---

### References
[1] Roth and Khilcher et al. The Odds are Odd: A Statistical Test for Detecting Adversarial Examples. ICML 2019.  
[2] Yang et al. ML-LOO: Detecting Adversarial Examples with Feature Attribution. AAAI 2020.  
[3] Kim et al. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). ICML 2018.   
[4] Yeh et al. On Completeness-aware Concept-Based Explanations in Deep Neural Networks. NeurIPS 2020.  
[5] Koh et al. Concept Bottleneck Models. ICML 2021.  
summary_of_the_review: In the current form, I recommend rejection due to the above-mentioned reasons. In particular, the paper needs to clearly justify the motivation of the work with a stronger experimental evaluation to demonstrate the effectiveness of the proposed method.
review_title: null

Meta-review: The work demonstrates that adversarially perturbing inputs can change the output of concept based explainability tools. Reviewers generally agreed that the writing was clear and the experiments were easily understood. Regarding novelty, reviewers noted that there are several existing works which study the adversarial robustness of explainability tools (one even has experiments specifically on concept based explainability tools). As a result, there is not much novelty in the finding that concept based explainability tools are sensitive to adversarial perturbation. Regarding the technical contribution of the algorithm, it is expected that standard optimization approaches (e.g. PGD) would be sufficient to break concept based explainability tools so there is not a clear technical challenge being solved in the work.

The work could be improved by refocusing the robustness analysis to derive new insights regarding the behavior of concept based explainability tools. In doing so, it would be beneficial to deemphasize the claims regarding novel security concerns---these methods don't even work reliably in non-adversarial settings, as evidenced by poor out-of-distribution robustness. It is expected that performance will be even worse under adversarial settings.
Decision: Reject