Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Explaining errors using semantically meaningful concepts is a critical tool for improving the performance, robustness, and trustworthiness of machine learning models deployed in application areas everywbere.  While there have been several proposed lines of work in this area, none is yet accepted as standard practice.  The authors present a method combining two prior methods in explainability: counterfactual explanation and concept activation vectors, and meld them together as coneptual counterfactual explanations.  

They describe the training procedure, and show some examples on both ImageNet and real diagnostic imaging applications.
main_review: Strengths:

- The paper is clearly written, and the figures are demonstrative and clear.

- In section 4.1, the experiments to identify spurious correlations by CCE a good beginning, but the interventions on the MetaDataset images are large drastic effects.  I would lke to see CCE applied to the identificaiton of more subtle spurious correlations that are more realistic.  Perhaps CelebA could be a good starting point?
  - e.g where more subtle latent factors explain the spurious correlation
  - where label imbalances exacerbate spurious correlation by diminishing the ability to estimate concept vectors

- The granny smith colour ablation test in section 4.2 a nice clear example of how the tool can clearly demonstrate a relationship for one given spurious correlation, but it does not tell me much about how robust it is to reporting spurious correlation CAVs at test time, which is the more important problem.

Weaknesses:

- I feel the authors underplay the complexity and investment to define a concept library, especially for specialize application areas.  In diagnostic imaging, prior taxonomies of concepts exist, but can be very difficult to identify in training images (cf. Oakden-Raynor et al.).  In other fields there may not even yet be generally agreed upon ontologies of concepts.  This drawback of the CAV method should be noted.

- Figure 1c is not a particularly good example of explaining a model mistake.  While the *label* given in the image may be “Zebra”, there is clearly a crocodile visible in the image (left of the smaller, left-most zebra).  This image is indicative of a different problem, which is that many natural images are better described as multi-label, as they contain multiple distinct concepts or entities within them.


- Towards the end of section 2 the authors claim 
  > "we extend these perturbations to unstructured data (specifically natrual images), whereas existing counteractural explanation methods have been largely confined ot tabular datasets since the perurbations have consitted of changing values in the original low-level feature space"
  This is far from true.  See Singla et al 2019, Goyal et al. 2019, and many other papers that apply counterfactual explanation to latent spaces of models that generate natural images.  This is a young but growing subfield that should be cited to acknowledge their contributions.

- The end of section 3.1 suggests concepts are validated by a hold out set, where those that do not meat a threshold on accuracy are discarded.  How was this threshold determined?  Is it affected by noisy data in the wild?  Is there some clear threshold in the distribution of accuracies over concepts?  And how is this affected by either noise or imbalance in classes of the data?  There are several such questions here that deserve more attention.

- Minor point, but Algorithm 1 should specify that the terminating condition of the while loop is **while** *w not converged* **do**


- Is the definition of the geometric margin ot the decision boundary in section 3.2 quite right?  It is defined as $d_i = \mathbf{c}_{i}b_{L}(\mathbf{x}_i)^T$.  This seems to use $i$ to index both concepts and datapoints in the training set for concepts.

Questions:

- Section 3 defines the *bottleneck layer* as that layer which will be employed to act as feature vector for learning CAVs.  Does the choice of layer matter here?  Or is there any reason to deviate from the pre-activation features from the ultimate fully connected layer?

- For Table 1: Is there a reason for choosing precision @3?  What does the distribution of the precision @K look like for various K?  Is there a steep drop after 3, or is 3 motivated by the desire to provide a user with a more manageable number of possible concepts?

summary_of_the_review: While this paper combines two complementary techniques in counterfactual explanation and CAVs, I would like to see CCE compared to other counterfactual explanation methods that are trained on natural images and provide semantic highlights or perturbations.  I also think that several important details have been omitted from the discussion (see Weaknesses section), and that in its present form, the work is not ready for publication.

Update: In response to the authors' extensive revisions to address my concerns (and those of the other reviewers), I have increased my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Goal: provide a human readable explanation to why a given classifier made a mistake on a given example.

Approach: 

Step1: For the given input domain (here images), first obtain a set of concepts that are human interpretable. These concepts each describe a particular aspect of an image, and a human is expected to be able to look at image and say whether a particular concept applies or does not apply to an image (on scale of [0,1])

Step2: learn a map from the input domain to the concept space using an SVM

Step3: generate counterfactual perturbation of example that ensures that :1) classifier is now correct on example, 2) perturbation is valid (i.e. realistic) and 3) perturbation changes few concepts. The perturbation is the addition of a linear combination of concepts by different amounts.  Key to this is that generating the perturbation requires the true label and query access to the classifier.

Step4: present the user the weight vector of the concept perturbations.

Validation: two ways to evaluate, one controlled and one in the wild.

Controlled evaluation: using dataset of animal images in different settings, confound the class label, e.g. dog, with background, e.g. snow, then ask if approach can recover the confounding factor, here snow. They show that on a median basis, the method finds the right confounding factor in the top 2 spots.

In the wild evaluation: on dermatology and chest x-ray datasets, they studied the mistakes of of a  trained model. The evaluation of their findings is based on discussion with clinicians.

main_review: Strengths:

- Novel combination of concept activations and counterfactual explanations
Excellent writeup of approach and motivation

- Interesting controlled evaluation of approach that demonstrates effectiveness. 

Weaknesses:

- Approach cannot be generalized to obtain explanations for a true test image since approach requires label y to obtain perturbation. How is it possible to make the approach work without the label y?

- Lack of baselines in controlled evaluation (sections 4.1 to 4.3): consider the following baseline, take the test example, find the closest correctly classified example to it in a validation set (that has same distribution as training data), and subtract the concept vectors for the test image - closest image in validation. Then show the top 3 concepts of the difference.  

- Lack of ability to visualize perturbed test examples. How is it possible to visually see what the test image with the perturbation looks like? Perhaps this is easy, but would appreciate it if the authors can elaborate on how to do this.

- Lack of discussion surrounding the requirement of being able to collect images in the domain of interest that describe each of the concepts independently of the trained data. For example, in the dog and snow example, to detect the concept of snow, the SVM must have seen images of snow with and without dogs. 

- In controlled evaluation, the cause of the mistake can always be found in the concepts. What happens if the mistake cannot be attributed to a concept? Does the method say that none of the concepts explain the mistake or does it falsely attribute it to something else? One can do this evaluation by removing e.g. “snow” from the concept bank and running the same experiment.

- Lack of objective evaluation metric and baseline in the “in the wild” evaluation: I understand the lack of baselines in the controlled study, but for the evaluation with clinicians, it would have been helpful to see if with other baselines (maybe something like GradCAM?) the clinicians can try to identify on their own the reasons they think the model got it incorrectly and see if they match with the concepts retrieved.

- The clinical concepts especially for the chest x-ray use case are not comprehensive enough to be sufficiently descriptive. I don’t believe in it’s current form, the in the wild evaluation adds value to the paper.


After author response:

- I have increased my score as the authors have raised some of my concerns, particularly on when the concept is not in the library
summary_of_the_review: The paper provides a novel way to explain a classifier's mistakes on a test example when given it’s label in terms of a known set of concepts. The paper is very well written and the controlled evaluation demonstrates that it is effective when the concepts can describe the mistakes perfectly. However, the evaluation lacks baselines and results when some of the assumptions fail (concepts don’t describe mistakes). Furthermore, it is hard to understand how the “in-the-wild” evaluation shows the effectiveness of the method. Finally, the method requires knowing the label of the example and thus is more of a debugging approach rather than an explanation method.

In its current form, I think the paper is borderline accept*, however, if the authors can provide answers to some of the weaknesses I presented, I am more than willing to move the score up to an accept as I think the ideas and the problem are extremely interesting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method for computing concept based counterfactual explanations - i.e. using the presence or absence of concepts (human understandable) for explaining the models prediction. In the paper they focus on image classifiers.
main_review: Pros:
 - Well written and clearly structured
 - Interesting and promising approach to compute counterfactuals in the domain of concepts

Cons:
 - Evaluation was done manually - i.e. manual inspection of samples. I understand that an automatic evaluation is challenging and I also appreciate that the authors are aware if this and suggest an user study - however, still I think that this manual inspection as an evaluation is a major limitation of this work. I would be more convinced if the authors would have done the evaluation as a proper user study.
 - While the examples in the paper look convincing, I was wondering whether it is possible to come up with formal guarantees - i.e. are there any formal guarantees that the final explanation is valid? The authors added some validity constraints but to me they look like a heuristic without any formal guarantees - I encourage the authors to comment on this.
summary_of_the_review: Overall an okay paper with some very interesting ideas, although I have some concerns regarding the evaluation of the method (see main review).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a Conceptual Counterfactual Explanation for explaining a model's mistakes.  It starts by learning a "concept" using a linear separator in the models representation space and a small amount of data annotated with that concept.  Then, it explains a model's mistake by finding a change in those concepts that changes the model's prediction, does not change too many concepts, and does not change the concepts in an unrealistic way.  The method is verified by showing that, for OOD test points where the model makes mistakes, this explanation can identify the concept responsible for those mistakes.  
main_review: Strengths:
-  The problem of explaining a model's mistakes is an important one and the proposed method is clearly defined and well investigated
 with ablation studies.  
-  An interesting range of tasks and concepts is considered and, assuming that the group of "OOD images" can be identified ahead of time, the empirical results are convincing.  
-  Studying more real/applied datasets such as those in Section 4.4 is an import direction for work on explainability.  

Weaknesses:
-  The method from [1] warrants discussion and comparison.  It makes the same assumption as the proposed method (ie, that we have concept labels for each image), provides its own algorithm for finding which concepts a model is relying on, and, further, provides methods for fixing the model when spurious concepts are identified.  
-  The experiments show that "for OOD images that are misclassified, the proposed method correctly identifies the concept causing the misclassification."  Unfortunately, this is not particularly useful because defining an "OOD image" requires that we already what spurious concept the model is using, which defeats the point of the proposed method.  Running experiments that start by analyzing the entire set of misclassified images would be more meaningful.
-  While studying explanations in the wild is commendable, these particular experiments are not particularly convincing. 
	-  Dermatology:  First, none of these concepts are clinically relevant and, as a result, all explanations using them will indicate some sort of "problem".  Second, the explanations shown in Figure 4 are not particularly convincing because "ashcan" and "water" are frequently relevant concepts; it seems more likely that the explanation method/concept selection are incorrect than the model is genuinely learning to use the presence of "ashcans" to make predictions.  
	-  Cardiology:  This has a similar problem to the earlier experiment because most of the concepts indicate some sort of "problem" (only Cardiomegaly and Atelectasis do not).  More generally though, it seems unnecessary to use explainability methods to determine that a model trained on X-Rays taken from one angle won't work as well on X-rays taken from another angle.  
	-  Including more clinically relevant concepts or exploring datasets where the Broden concepts are relevant would address this issue.  

[1] Singh, Krishna Kumar, et al. "Don't Judge an Object by Its Context: Learning to Overcome Contextual Bias." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
summary_of_the_review: While the proposed method is interesting, the supporting experiments need fundamental changes in order to be impactful.  Specifically, the analysis should not start with the assumption that a set of "OOD images" has been identified and the experiments should include more concepts that are relevant to the predictive task.  

# Update after discussion with the authors

I'd like to thank the authors for a very productive discussion and for taking the time to run additional experiments to address my concerns.  At this point, I think that the setup, results, and analysis for the settings where the model is trained on a deliberately biased distribution are convincing.  As a result, I've increased my score to a 5.  

However, I am still concerned that using CCE in a less controlled setting will be difficult due to "false positives" for relevant concepts:
-  These appear in Figure 4 in the form of "water" and "ashcan".  In this setting, it's easy to label these as false positives because they are unrelated to the task;  but this seems like more of a reason to exclude those concepts than an actual fix to the problem.  
-  As an example of where it is more difficult to determine if something is a "false positive" consider Figure 6.  While it seems likely that concepts like "building" and "flag" are false positives, its much less clear  if "cow" is.  If we consider adding a literal "cow" to the image, it seems unlikely that that would help the model detect the zebra.  But with a more subjective interpretation of emphasizing the "cow shape" of the zebra, then it seems more likely.  Either way, this seems like something that should be tested directly.  

One way to address this would be to:
-  Train a model on the entirety of the MetaDataset
-   Find misclassified points and run the batch CCE on them
-   For each of the top concepts, translate the explanation into a hypothesis about about the model's performance on real images (eg, snow is relevant for dog, so the model will perform better on images of dogs with snow)
-  Measure how often those hypotheses are correct
review_title: null

Meta-review: This paper proposes a novel method called CCE for explaining mistakes by DNNs on image classification. It is built on top of two prior ideas: counterfactual explanations and concept activation vectors. CCE explains a mistake by assigning scores to a shot list of concepts, where a large positive score means that adding that concept to the image will increase the probability of correctly classifying the image, as will removing or reducing a concept with a large negative score.
The strengths of the paper include novel combination of previous work, clear presentation, interesting experiments, convincing results on controlled settings.  The weaknesses include the lack of results on less controlled settings, the lack of more meaningful spurious correlations in the medical examples, and the lack of user studies.
Although the reviewers have shown interests in this paper, they clearly do not support the paper strongly. In addition, the authors have missed the following paper that also combines counterfactual explanations and concept activation vectors:
Akula, Arjun, Shuai Wang, and Song-Chun Zhu. “Cocox: Generating conceptual and counterfactual explanations via fault-lines.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 03. 2020.
Decision: Reject