Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a novel concept-based explanation for neural networks. It builds on previously proposed formalism of concept activation vectors. This formalism is built on the idea that the features related to inputs containing a given concept and the features related to inputs that don't are separated by a hyperplane. The main contribution of the paper is to relax the latter: The core idea is that a neural nework latent space can be divided into non-linearly separated clusters. Such a network appears to encode well the presence or absence of the related concepts. The authors also adapt the CAV importance score (TCAV) to the new formalism, and show that these concepts are invariant to latent space isometries if they are based on support vector classifiers with a radial kernel. The authors then show several desirable properties for these explanations, including supporting  scientific evaluation of models. The experimental validation is based on multiple datasets from different domains.
strengths_and_weaknesses: Strengths: 
* The paper is clearly structured and well written.
* The relaxation of the linear separability in the latent space is sound, and the implemented concept activation regions seems to capture better the spread of concept-related features in the latent space. 
* The experimental evaluation is well designed and extensive, and shows many desirable properties of the proposed approach. 

Weaknesses: 
* The paper considers mostly convolutional networks. It lacks analysis of more state-of-the art architectures like residual networks or transformers. 


questions: * Would it be possible to get inspiration from the insights in the paper to improve neural network training, with the purpose of increasing explainability? 

* How are the proposed explanations sensitive to adversarial attacks? 
limitations: As indicated in the previous section, it would be interesting to add to the paper an analysis of the sensitivity of the proposed approach to adversarial attacks.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper generalized TCAV's linear separable assumption to an additional kernel space separable, which is simply the smoothness theoretically. The linear classifier is replaced by a kernel SVC, and the TCAV score is replaced by a counterpart TCAR score. They then identify the SVC classifier has a higher accuracy, and results in more stable concept explanations.
strengths_and_weaknesses: strength
--  the paper targets an important question and limitation of TCAV
--  the empirical evaluation is thorough

weakness
-- the introduction of kernels may lead to much more hyper-parameter choice to use in practice
-- for very complicated concepts and networks, an additional RBF kernel may be insufficient to make the data linearly separable in practice
-- the layer selection problem is not dealt with, as TCAV suggested that sometimes earlier layers may be more fruitful even if the accuracy seems lower (for more low-level concepts). In my personal experience, using the mixed-7b layer inception-V3 for TCAV usually produces a much better result than the penultimate layer. Moreover, how does one choose the layer to apply TCAR is not addressed. 

note:TCAV does not work well with the penultimate layer since d c/d activation = W_c (which is independent to the instance), and thus for all instances in the same class the directional derivative to the same concept would be fixed.

---------------------------------------------------------

I see that the advantage of CAR lies also on not using the sensitivity score, which makes sense. Changing my score to 7.
questions: -- In the inception experiment, CAV has a pretty close accuracy to CAR (showing that the linear separability is not a huge issue), but the evaluation of TCAV score in table 1 seems completely wrong. The failure of TCAV in MNIST and ECG is understandable since the model seems under-represented, and an additional kernel classifier would help a lot. However, the result in Inception-v3 is not convincing.

-- the evaluation of concept-based feature importance is only a sanity check, how is this useful?


limitations: weakness 1 and 2 are not fully covered, but they do cover some limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Recently, there has been a lot of focus on opening up the black-box DNNs via explanations. One such technique is Concept Activation Vectors (CAV), which explains the predictions of DNNs through user-specified concepts. One main shortcoming of CAV is that it assumes that examples corresponding to a concept are all mapped in a fixed direction in the DNNs latent feature space, which can be restrictive in practice. In this paper, the authors propose  Concept Activation Regions (CAR) based on kernel trick and support vector classification that relaxes this assumption.
strengths_and_weaknesses: Strengths:
- The proposed technique relaxes a fundamental assumption made in CAV, thereby increasing its effectiveness.
- The explanation generated by the proposed method is invariant under latent space isometrics.

Weaknesses:
- One of the critical weaknesses of CAR is that since explanations are generated through a Kernel-based technique, it loses the nice interpretations CAV offers, like if one increases the presence of a concept, how does it affect the model predictions.
questions: - Would it be possible to replicate the interpretations CAV offers like concept senstivity? An alternate would be to perform test time interventions on the Kernel-based Concept classifier.
- Is it possible to run CAR with concepts (or latent variables) generated by Autoencoders or VAE that are inherently noisy or abstract? Could CAR be used to analyze the concepts generated by techniques like SENN?
- How robust are explanations generated by CAR? Is it robust to change in backgrounds of the images?
- Would CAR work for other domains like NLP?
- Is it possible to relax the assumption that such a class of techniques requires additional annotation of concepts? As of now, CAR requires a user to specify the positive and negative examples for each concept. 

Given that CAR assumes to have access to the feature extractor of the model, it isn't truly a black-box setup, unlike paper portrays.
limitations: I would encourage the authors to list the limitations of the proposed approach.
review_title: null

Meta-review: metareview: All reviewers have found the paper as a solid contribution on a highly important topic, addressing the major shortcomings of the notable work, CAV in concept-based explainability area. On such shortcoming is that CAV assumes that examples corresponding to a concept are all mapped in a fixed direction in the DNNs latent feature space, which can be restrictive in practice. The proposed technique relaxes a fundamental assumption made in CAV, thereby increasing its effectiveness. As one main contribution, the reviewers have found the relaxation of the linear separability in the latent space sound, and the implemented concept activation regions well capturing the spread of concept-related features in the latent space. There are some concerns on the experimental analysis, that not many DNN architectures have been considered, lack of results without human annotations for concepts, and through robustness analyses. The authors have somewhat addressed these, although there is still some room for improvement. Overall, the positive aspects of the paper overweigh and I suggest acceptance of the paper. 
confidence: Certain
Decision: Accept