Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a concept-based explainability method, CRAFT, to generate both global and local concept-based explanations.
This paper introduces some key contributions: (1) recursively decomposing higher-level concepts into more elementary ones.
(2) concept importance estimation based on Sobol's indices. (3) Concept Attribution Maps, which connect global concepts to local image level explanation.
strengths_and_weaknesses: **Strengths**

(1) Concept-level explanation is a challenge and an important direction. The proposed method is useful and novel, especially for the recursive decomposition to get different level concepts and the linking between global high-level concepts and local attribution map.

(2) The three ingredients based on the concept activation factorization are promising and inspiring.

(3) The paper is well organized and easy to follow.

**Weaknesses**

(1) How to verify the faithfulness of the discovered concepts? What if the human-understandable concepts are not the correct reasoning logic for the original neural networks.

(2) How to evaluate the scalability to larger classes? 

(3) Are there quantitative results to show the performance and compare it with other baselines?

==========================Pose Rebuttal=============================

Thanks for your feedback. I will keep my original score.

questions: (1) How could we use the explanation to explain the error cases made by the model?

(2) It would be good to compare with a related paper: "A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts" which also provides both global and local concept-level explanations.
limitations: More quantitative experiments to show the performance and compare it with other baselines would be better.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors have done a nice job by improving the concept based explanation generation techniques from different perspectives. An automatic concept extraction followed by a recursive procedure towards achieving a set of concepts and sub-concepts are proposed in this work. A concept-based heatmap generation method is also proposed here that backpropagate concept values to pixel space that is very similar to widely used class-based saliency methods.
strengths_and_weaknesses: Strengths: 
a) Several important directions are worked on here, that start with concept generation in a recursive manner from multiple layers of the network. This technique helps the cases where concepts generated from a very deep layer don't make much sense and decomposing them with sub-concepts can develop an improved understanding of the model. 

b) This is the first method, I believe, to work on saliency based explanations for each concept. The authors have shown the effectiveness of this approach by explaining different concepts in a single image. 


Weaknesses: 
a) Although the quality of a technical literature shouldn't be judged by the language quality, I still believe that there are many scopes to polish the language and take care of some typos to improve the readability of the paper.

b) I couldn't access the code related to the paper from the provided link. If by any chance the code was not ready for sharing by the submission time, it would have been better to skip that.
questions: L31: What do you mean by information methods?

L73-74: If some figure in the supplementary section is referred to in the main paper, it is a good practice to clearly mention that.

L137: I request the authors to explain the notation h_{l,k}, as l and k represent some layer and the total number of layers respectively, but it was not clear to me how they're used in a single notation.

L163: Please write the full form when the short form is used for the first time in the paper. The full form of NNLS (i.e. Non Negative Least Square) should have been mentioned here.

Other questions: 
a) Please explain the notations of Table 1 in detail. It is not very clear what 'n' represents here.

b) Here the concepts found by NMF are claimed to be human understandable. If any or some of the concepts are exceptions, I am interested to know whether these concepts are specific to a particular category?

c) As the gradient based explanation generation techniques generally produce noisy saliency maps, did you see any such phenomenon for concepts? Do the heatmaps for all the images are as nice as shown in fig.1? If yes, which part of the method (related to concept attribution map generation) helped you to achieve that specifically?
limitations: The authors did not add any section to address the limitations and potential negative societal impact of this work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper the authors propose the “Concept Recursive Activation FacTorization” (CRAFT), a framework to generate concept-based explanations from trained networks. CRAFT aims at addressing current limitations in state-of-the-art concept-based techniques (e.g. ACE [1]) including: (i) the association of *one concept only* for each image segment/patch, (ii) the instability/noise of gradients w.r.t. concept vectors, (iii) the localization of concepts in the input space to provide local explanations. To address these limitations, the authors equip CRAFT with three methods: (i) a recursive procedure to automatically decompose concepts into sub-concepts, (ii) an alternative scoring function for concept importance based on Sobol indeces, and (iii) a method to localize concepts in the input (pixel) space. Following Zhang et al. [2], CRAFT identifies coherent bases of concepts using the Non-negative Matrix Factorization (NMF) rather than k-Means as in ACE [1]. In the experiments the authors show that CRAFT can: (i) identify meaningful and coherent concepts, (ii) localize concepts in the input space allowing users to explain misclassified images, (iii) decompose high-level concepts into coherent subconcepts, and (iv) provide a more robust ranking concept w.r.t. TCAV in terms of  fidelity scores.

[1] Ghorbani, Amirata, et al. "Towards automatic concept-based explanations." Advances in Neural Information Processing Systems 32 (2019).

[2] Zhang, Ruihan, et al. "Invertible concept-based explanations for cnn models with non-negative concept activation vectors." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 13. 2021.
strengths_and_weaknesses: #  Strengths
The automatic decomposition of high-level concepts into sub-concepts is the most interesting contribution of this paper w.r.t. existing concept-based approaches. This is a significant step forward w.r.t. state-of-the-art approaches such as ACE as it enables the identification of coherent hierarchies of concepts. In particular concept hierarchies may facilitate human experts in navigating the concept space. Even though the methods are not new (e.g. NMF, or Sobol indeces), they are effectively used in new contexts. In addition, the experiments show how CRAFT can be used in practice as a powerful tool to “debug” misclassified samples by localize concepts in the input space.

# Limitations
While this paper introduces a few interesting steps forward, the authors may consider a few suggestions for improving their contributions. In particular the authors may consider to:
1. Broaden the experimental section as it is currently based on one dataset only, thus questioning the generality of the results. I would strongly encourage the authors to broaden their experiments to include additional benchmarks commonly used in concept-based XAI papers (e.g. CUB [1], TabulaMuris [2], or CelebA [3]).
2. Include a description for the limitations of their work. Including relevant limitations would significantly improve the paper as it would allow other scientists to understand how to use and improve CRAFT advancing the research field. A possible limitation worth mentioning is that the experimental evidence is mostly qualitative (Section 4.1, 4.2, and 4.3) and/or possibly subject to confirmation or selection biases (Section 4.3).
3. Include critical information which is currently missing for understanding and reproducing the experiments. For instance, the authors do not mention what error bars represent (e.g. Figure 6) nor how the images where selected for the human experiments i.e., they just mention that “the choice was randomized” (line 701), but they do not specify how (e.g. random selection from cherry-picked examples or from all images? From groups of image crops or from the original images?). Another issue is that it is not clear how the authors pick CRAFT hyperparameters (e.g. $\tau$, line 263) and how sensitive the results are to hyperparameters’ changes.
4. Reorganize some sections, include missing information, or rephrase a few sentences to improve flow and clarity. For instance, in the abstract the authors do not mention any knowledge gap which makes a bit unclear why CRAFT might be needed. On the contrary, in the introduction the authors mention four knowledge gaps (lines 31, 41, 47, and 51) which makes a bit unclear what is the *main* purpose of CRAFT. Making the introduction and contributions more focused would significantly improve flow and clarity and it would save the authors some space for describing methods, results, and limitations.

[1] Wah, Catherine, et al. "The caltech-ucsd birds-200-2011 dataset." (2011).

[2] Cao, Kaidi, Maria Brbic, and Jure Leskovec. "Concept learners for few-shot learning." arXiv preprint arXiv:2007.07375 (2020).

[3] Liu, Ziwei, et al. "Large-scale celebfaces attributes (celeba) dataset." Retrieved August 15.2018 (2018): 11.
questions: Addressing the main limitations mentioned above would significantly improve the paper and, as a consequence, reviewers’ feedback and scores. However, I understand that the authors may not have enough time to address them all in the rebuttal. For this reason I list here a few minor items which can give the authors a better chance to improve their work in the short term:
1. A few sentences may need further clarification as they may contain typos or have grammar issues (e.g. ““The second concept elucidated by craft is located on the astronaut’s pants that **he confuse** with the ski suits” (Figure 5)). The authors may also improve the flow using the active voice more frequently (e.g. “Namely, in [38] and [39], explanations are generated as the inputs” (line 97)). Other sentences which may benefit from rephrasing include: “Once the factorization done, and the reconstruction of the activation of the image can then be interpreted as a combination of a set of concepts and a coefficient associated to these concepts.” (line 116), “A common concern with concept extraction methods is that make sense to humans does not mean it” (line 189), “Finally, we can capture the importance that a concept might have as a main effect – by itself – as well 210 as by interacting with other concepts on the model’s output by calculating the expected variance that 211 would remain if all the indices of the masks except the Mi were to be fixed.” (line 209). 
2. A few acronyms need to be spelled out (e.g. TCAV, line 112).
3. The authors reference Figure 3 before Figure 2. I would suggest to swap the figures.
4. Some claims may require to be rephrased as in their current form may not align with actual results. For instance, I find the possibility of using CRAFT for explaining complex failure cases quite exciting, but the only experimental result is on one cherry-picked example which may not be enough to prove the efficacy of the method (Section 4.2).  Another example is when authors state “We use our approach and combine local and global explanation, to *accurately* explain predictions” (line 72). Here I was expecting a *quantitative* result in the experimental section showing the actual *accuracy* scores of CRAFT’s explanations. A third example is when authors state that “CRAFT yields more interpretable concepts than ACE” (line 74). The main issue here is that ACE and CRAFT are different under many aspects. These differences make a fair comparison a bit tricky as each difference may introduce a confounding factor in the comparison. However, the results here are only qualitative (Figure 4 and appendix) which makes the claim a bit weaker than it could have been. Moreover, the experiment design does not allow to identify which ingredient (i.e. Sobol indeces? NMF? or $\tau$?) from ACE or CRAFT is the one that is actually making the difference. A fourth example is when authors state “These examples illustrate one of the weaknesses of ACE: the segmentation used can introduce biases through the baseline value used. Moreover, this segmentation does not facilitate the interpretation of the concepts” (line 276). However the evidence here is only based on a qualitative visual comparison which may not be enough to support such strong claims.
limitations: As mentioned before, the authors may consider to include a description for the limitations of their work. Including relevant limitations would significantly improve the paper as it would allow other scientists to understand how to use and improve CRAFT advancing the research field. A possible limitation worth mentioning is that the experimental evidence is mostly qualitative (Section 4.1, 4.2, and 4.3) and/or possibly subject to confirmation or selection biases (Section 4.3).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work presents a novel method to provide local and global explanations for the decision process of neural convolutional models. Their approach tries to divide agglomerated concepts from later layers into disentangled sub-concepts in earlier layers to provide more human-comprehensible explanations/ concepts. For this purpose, they make use of three “ingredients”, namely 1) a reiteration to find early-stage concepts that are human-comprehensible, 2) an approach from the variance-based sensitivity analysis, the Sobol indices, to estimate concept importance in the input for the prediction and 3) a method to provide an attribution map through implicit differentiation for the found concepts such that we know where the concept is contained in the input. Experiments, including some ablations studies and a user study, are provided to prove the function of the proposed method.
strengths_and_weaknesses: **Strengths**:
- S1 Using the NMF to find promising concepts is a good idea, especially to only find sparse and meaningful concepts. Using Sobol indices for feature importance seems sensible. Also, combining the three ingredients to solve this task is exciting and overall yields a novel approach to explainability. This work presents a multitude of solid mathematical solutions for practical problems in this task.
- S2 The figures used to conceptually show the function of their method is very good and intuitive.
- S3 The story is in general clear.

**Weaknesses**:
- W1 Overall, the experimental evaluation is weak. Many claims are only shown conceptually, especially comparing ACE and the failure cases. That is a big drawback. Also, the user study is very sparse. It shows that sub-concepts help to a small degree in finding an intruder. Attributing this finding to more meaningful concepts is a vast step. Also, no comparison of what other methods achieve is not given.
- W2 The work claims that the method rather answers what than where, but this question remains unanswered despite intuitively making sense. The idea is to not only highlight where a concept is attributed in the input but also what the concept looks like. However, still, it is hard to understand what the concept is. The shovel examples left me somewhat confused. I agree that this approach targets the “what” but only to a limited extent. Here, I am missing a discussion on this claim.
- W3 In general, a fair discussion on limitations like computational expense, advantages, and disadvantages, in general, are missing. How efficient is this method? How computationally expensive is this method? If it only works with ReLUs, that is a significant drawback; what could be the following steps to work with other activation functions?
- W4 The intro motivation is a bit weak. Explainability is essential, yes. However, as far as I understood, the GDPR’s “right to explanations” does not ask for an explanation this work is proposing. The GDPR is mainly about privacy rights, which means the “right to explanations” refers to the data a model is trained with, which your work does not really target. Instead, the proposed European AI act targets the “right to explanations” in your way.

Based on the exciting idea I tend to accept this paper. However, there are concerns about the evaluation. If the authors address these concerns, e.g. on the missing limitation discussion, I am happy to change my score.
questions: Suppose the decision is made based on these concepts. How exactly does your method differ and compare to prototype networks? To some extent, a concept seems similar to a prototype, especially as you show multiple prototypical (parts) examples when you want to answer the “what” question. 


Further remarks not affecting the evaluation:

- link to anonymous GitHub does not work
- Line 64 meanignful -> meaningful
- Line 105 vectorin -> vector in
- In general, quite some typos, capitalization, and punctuation errors
- Please incorporate fig 7 in your main text if you refer to it multiple times
- The appendix is a bit messy; many typos and references to figures are missing, etc.
- Line 245 What are KKT conditions?
limitations: No real discussion on limitations. Neither from a technical nor from a societal view. Also no discussion on future work. That’s a big downside of this work, especially if you motivate your work with societal reasons, i.e. “the right to explanations” for end-users. What impact could your method have?
review_title: null

Meta-review: metareview: Reviewers generally agreed that this paper is innovative (the decomposition of high-level concepts into sub-concepts in particular sets this paper apart from existing concept-based methods), and appreciated its potential practical utility for the explainability community (for example by providing localization in input space in addition to concepts, which can be used to debug model errors).
All Reviewers however also agreed on the main weaknesses, i.e. the limits of the validation of the method (which is restricted to only 1 dataset and lacking rigorous quantitative metrics), and a related lack of clarity in terms of technical motivation and use cases enabled for end-users of the method.
Despite some useful clarifications and improvements in the presentation of results provided in the rebuttals and the ensuing exchange, Reviewers were left unsatisfied by the responses addressing the mentioned main weaknesses, which raised renewed concerns about their seriousness.
As a result, the discussion after the rebuttal period was marked by the opinion expressed by two Reviewers that the paper's weaknesses outweigh the (albeit clear) merits and that in its current version the paper is not ready for being accepted.
confidence: Less certain
Decision: Reject