Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: 1. The authors propose a method to derive counterfactuals, saliency maps, and so called isofactuals using invertible neural networks. The quality of the generated explanations are compared visually with existing baselines. 

2. I found the structure of the paper confusing and lacking in clear elicitation of contributions. For example, once explanation methods are provided in Sec 2, the motivation of Section 3 to purely compare gradient methods is highly unstructured. In here, its unclear why changes in independent principal components while generating counterfactuals not desirable especially if as the authors suggest, the prediction changes. Even if the changes are not observable to humans. This is a highly unusual aspect of the counterfactuals where the counterfactuals shouldn't just explain large changes to logits but changes to predictions as well. 

3. The objective of each evaluation data-set and study should also be clearly outlined before proceeding to the details. Currently the paper leaves the reader to figure out the main contributions at the cost of hampering the paper's technical significance. 

4. What is the goal of the user study? Why is the baseline any other method of generating counterfactuals but merely conditioned examples? 

I strongly suggest restructuring the paper to fix above concerns and provide a clear justification for their experimental setup.

5. I also strongly recommend the authors to move away from evaluating their models against celebA labels such as "Attractive" which are rife with ethical concerns. I understand that those are the only options available in celebA but I would recommend using more neutral class labels for experiments if face dataset is an important part of their evaluations.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The authors have done a reasonable job at addressing my concerns and I have increased my score from 5 to 6. 
review_title: While technically incremental, the work provides interesting method to generate multiple kinds of explanations counterfactuals, saliency maps and what is called as "isosurface" of the classifer. Interesting case-study and user study demonstrate potential benefit of the method comparing two types of explanations

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors propose a method for generating counterfactuals (visually “similar” examples with different labels) and isofactuals (visually “different” examples with the same label) using an invertible convolutional network. A human study shows that providing these counterfactual and isofactual images in a systematic way can help participants understand a model’s bias better.

Interpretability of machine learning models is becoming progressively more important as these models continue to proliferate in sensitive applications such as medicine, finance, law, etc. A large body of interpretability efforts is around post hoc methods where an explanation is generated given certain probes into a blackbox function. On the other hand, the proposed model imposes certain constraints on the model to yield these explanations. The paper has some interesting ideas, and the qualitative evaluation is helpful is conveying those ideas. The discussion around gradient wrt the input image vs directional derivative is fairly insightful with convincing qualitative and quantitative results (Fig. 4). However, I did have some conerns:

1. The writing often lacks clarity and the usage of space can be more judicious. For example, the description of the main method is severely lacking, and is lumped into a few short paragraphs (section 2). I needed to reread the section a few times to understand the gist since important details are either missing or relegated to the appendix. I am still unsure about the development of isosurface section. On the other hand, excessive details are present in the evaluation section, e.g. in-depth discussion of mice characteristics. Focusing on elaborating the method, and maybe one less dataset would improve readability by moving extra evaluation to appendix. Something like, “we observe similar patterns with other tested datasets, which are presented in the appendix.”

2. The results of the human subject study are not very convincing. While the subjects were able to better detect model’s biases with the systematic presentation using the proposed method, they also spuriously discovered irrelevant patterns (background and blocks). The appendix (Fig. 9) also shows that the subjects thought both the proposed method and the baseline were equally good. [Digression: Fig. 9 is poorly processed with missing words, repeated legend, shuffled axes, etc.).

3. Minor formatting issues: mismatched quotes throughout (striped, zebra, etc.), Fig. 4 caption (there not ideal), “different to the original”, etc.

Overall, the paper has some good ideas and interesting analysis but falls short on clarity and fully convincing the reader about the results. I am marginally inclined for it to be accepted. 
review_title: Relevant. Lacks clarity. Mildly convincing results.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper describes a computational method to construct ideal counterfactuals and isosurfaces via invertible CNNs, and uses it to reveal biases in three different datasets. 

Strengths:
1. The use of directional derivative to construct ideal counterfactuals is interesting.
2. Leveraging PCA to construct isosurfaces is neat.
3. The human study is a plus, where the stimuli are based on counterfactual interpolations created by the proposed method.

Weaknesses:
1. The reviewer finds the manuscript hard to follow, especially Section II. The authors may come up with a clearer presentation.
2. The descriptions about saliency maps are less relevant to the main idea, further confounding the reviewer.
3. The comparison between simple gradient and direction derivative is less fair, as the directional derivative makes use of the very information direction w (e.g., the direction of no sunglass -> sunglass).  What happens if we visualize $\phi^{-1}(\phi(x)+ \alpha w)$ directly, for different values of $\alpha$.
4. The human study may need to conduct another set of control experiments to show that only original training images (not counterfactual interpolations) are $\textbf{less}$ helpful for uses to identify CNN patterns and biases.  The reviewer conjectures that for this simple TWO2TWO data, the subjects may spot shortcuts easily even using original training images.

Other minor comments:
1. Figure 1: There is no explanation for (a). What is $w$? The reader may not understand it for the first reading.
2. Figure 4: The reviewer believes normalized scores on the top of the images make better sense.



review_title: Use of invertible CNNs to construct counterfactuals and isosurfaces

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:  The paper presents a promising idea to build interpretable models by combining discriminative and generative approach. The proposed model uses an invertible neural network to model the data distribution. The invertibility helps in transforming the learned feature vector back to the image domain. A linear discriminative classifier is trained on the feature vector to perform binary classification. Using the inverse function, the model generates a counterfactual explanation by inverting a modified logit score to create a new image as an explanation. The authors further construct an orthogonal basis using PCA, such that modifying feature vector in those directions results in no change in the classifier's prediction. Decomposing the feature space into such a basis helps discover potential biases in the dataset and the classification model. The experiments compare the proposed method's performance with fully discriminative models and post-hoc interpretability methods such as gradient-based saliency maps.
  
Major 
-----------------

•	The paper an interesting and potentially important idea. But at times, the text is difficult to read. 
•	The conclusion from Figure 2 is not clear. From the caption and the figure, it is not clear which attributes such as smiling; gender are important for the classifier's positive /negative attractive decision.   
•	In Figure 2 and Figure 3(b), a label showing the different principal components considered in each column, and the logit/prediction of the classifier for each row will improve the figure's readability. 
•	Saliency maps highlight important regions of an image for the prediction decision. The saliency maps in Figure 3(a) highlight almost the entire face; hence they are inconclusive. 
•	Figure 4(a) shows the final results after integrating the original image along with different derivatives. To understand the results, it would be helpful to show some examples over which the integration took place. 
•	An example of a random sample, with minimal/no changes along the independent factors for the proposed method, as compared to positive changes by other methods, will help in understanding the results in Figure 4(b). 
•	The number reported at the end of section 3 on "the gradient of x and the directional derivative dx/dw" should be reported in a table to allow a proper comparison between different methods. 
•	The directional derivative dx/dw, in the model, is w.r.t the weight vector of the binary classifier, trained to identify the label. Related work by Kim et al. (2018), "Interpretability beyond feature attribution: Quantitative testing with concept activation vectors", also used directional derivatives w.r.t to binary classifiers, trained to identify a human-defined concept. A comparison with this method will help the reader understand the different applications of directional derivatives and how directional derivatives can be used without an invertible network. 
•	Table 1 doesn't report the results for the supervised method for celebA and tow4two datasets. 
•	The numbers reported in Table 2 lacks a coherent conclusion. The corr. data and corr. change columns have values in similar ranges. Please elaborate and discuss the results. 
•	In Figure 5(b), it's not clear how real images for baseline condition are sampled. To allow proper comparison, as in counterfactual design, three images should be selected as primary images, the rest images should be sampled based on its minimum distance to the primary images but a different label.  
•	In the Evaluation section, for celebA, the authors identified principal components that correspond to attributes like gender, smiling. The results are reported in a qualitative manner, with inferences draw by showing a few examples only. A quantitative analysis is required to demonstrate the dependency of the "attractive" attribute on other attributes. The results shown in Figure 6 of the appendix don't have a thorough caption to illustrate the findings. 

Minor: 
------------

•	The caption for Figure 1 has typos.  
•	In Figure 4(a), the caption doesn't describe the top label. 
•	The opening quotation marks are inverted throughout the text. 
•	Table 1 and Table 2 are shown after the references. They should be placed with the main text before the references. 
•	The label of Figure 5(b) has a very small font.


review_title: interesting idea but the execution and writing left a lot to be desired, seems not proof-read!

final_score: rating: 6: Marginally above acceptance threshold
review_content: Update after revision
------------------------------
I thank the authors for their work on this paper. The second reading was more pleasant. I agree with the authors that performing a user-study is an important effort, that should be encouraged. I however still believe that, if not benefitial to the user, the complexity of the method can be a drawback. I also wished that more comparisons, but especially other data modalities were investigated. I have updated my rating to reflect the improvement in the text.

Short summary
-----------------------
The authors propose a technique based on an invertible network to provide counterfactuals relative to one class of interest. The counterfactuals can be interpolated across an isosurface, displaying parameters which do not affect the model’s decision. The authors propose an attribution map based on those counterfactuals and evaluate counterfactuals in a qualitative manner, based on their own observations on 3 datasets, as well as based on a human-grounded evaluation on a synthetic dataset. 

Strengths
---------------
The use of an invertible dataset is rather novel in the field of explainability, and the relationship between the obtained counterfactuals and gradient-based interpolation methods is interesting. The human-grounded evaluation is definitely a large undertaking that is not often performed to assess the usefulness of interpretability techniques.

Weaknesses
-------------------
I have identified several weaknesses of the work that justify my recommendation:
- the (lack of) clarity of the text.
- the assessment of the technique, as the results of the human-grounded evaluation are mixed, with users not being significantly more accurate in finding confounding factors compared to a baseline technique.
- the limitations of the technique, not discussed in depth. For instance, I can see difficulties in evaluating the effect of classes that are not present as “training classes” in the dataset, which requires a large labeling effort. In addition, how the technique would transpose to non-image datasets, or whether there are limitations in the invertible architectures to consider should be mentioned.

Novelty
-----------
The “Related works” section is rather limited, which makes it difficult to evaluate. In general, the use of invertible networks as interpretable networks is novel.

Clarity
---------
Clarity was a major weakness of this work for me:
- the datasets are illustrated in figures but not mentioned until much later
-  the maths are described in sections that seem unrelated to each other, without depicting the relationships between the different steps
- multiple concepts are unclear (see detailed comments)
- the motivations are not clearly explained

Rigor
--------
I found the qualitative evaluation on the 3 datasets unconvincing, as it is unclear whether the same conclusions could not have been reached using other techniques.
While I was most interested by the discussion around the generation of counterfactuals based on the invertible network compared to based on the integration of gradients, I wished there was a definition of an “ideal” counterfactual, qualitative or (preferably) quantitative. The single example provided in the main text is appealing but this requires more evidence to me.
Finally, the “saliency” maps defined in this work do not seem to be used later on in the work. I doubt that looking at them would improve human evaluation of a model’s behavior.

Detailed comments
-----------------------------
- Counterfactuals: their quality seems subject to appreciation and confirmation bias, especially on potentially cherry picked examples. To assess their quality, I would suggest to use the BAM dataset (Yang and Kim, 2019, https://github.com/google-research-datasets/bam) which was generated to benchmark attribution methods. I would overall suggest the use of this dataset for assessing the faithfulness (sensitivity, specificity) of the proposed approach.
- The choice of the mice dataset should be justified as this doesn’t seem like an obvious choice to assess the quality of attribution techniques. It is quite difficult to estimate any effect, and feels like qualitative evaluation is biased by the authors’ remarks given the lack of knowledge of the problem.
- There should be more details about the Two4Two dataset and its motivations, as well as how it relates to other datasets (e.g. Goyal et al., 2019)
- How does the proposed approach relate to “completeness” (Sundararajan et al., 2017)?
- What is the mathematical justification to resize the saliency map of an intermediate layer to the input resolution? Is there a citation for this process showing that this is a reasonable assumption?
- I am confused by the section on saliency maps: what does h represent? The activations at an intermediate layer? The motivation is unclear: what are the authors trying to highlight in these “saliency maps”? Are these computed attributions or are these L1 distance between activations (in %) between x and x_tilde? Or is it a cosine distance (as suggested by the next sentence mentioning the angle?)
- The tasks used for illustration are not described in the text. Examples of y and epsilon should be provided.
- Is the technique limited to the model’s predicted classes?
- How is “ideal” counterfactual described and mathematically verified?
- The relationship between counterfactuals and e.g. integrated gradients is unclear: the first clearly needs a model that can generate data, while the latter integrates the gradients between a baseline (defined by the user) and the input. More details and explanations are required to make this relationship clearer.
- What are the participants in the human-based study viewing? Are they comparing the counterfactuals to e.g. SmoothGrad maps, or the saliency as defined per the proposed approach?
- It is unclear what the participants answered: Figure 5a mentions that the main score is “strongly disagree” for “arms” (both baseline and interpolation) while the text refers to “strongly agree”. Example questions would help.
- The results of the human-grounded study are not very conclusive. Note: please correct for multiple comparisons due to multiple statistical testing of the same effect.
- Kim et al., 2018 already displayed that human users were performing poorly at identifying a network’s decision behavior based on saliency maps. A better comparison could have relied on TCAV instead, especially as the concepts can easily be mapped to the features given the synthetic dataset. This could have made a stronger case for the use of invertible networks, especially as Goyash et al (2019) mention the use of counterfactuals based on concepts.
- How about non-image datasets?

Minor
-------
- Intro: I would suggest using “transparency” rather than “interpretability” when referring to logistic regression (e.g. Lipton, 2016). The interpretability of linear model weights is indeed debatable, as weights will depend on the regularization and signal-to-noise ratio in the data (Haufe et al., 2014).
- No clear flow between the different works in the intro. No clear motivation behind counterfactuals.
- proofreading: paper is quite hard to follow and minor changes to grammar (e.g. “Their similarity is easy to seen”) makes it more difficult to assess. The quality of the writing deteriorates in sections 3, 4 and 5.
- It is unclear what scale delta epsilon represents, and whether we can expect the norm of the different techniques to be comparable. 
review_title: Interesting idea needing more work

Meta-review: All the reviewers agree that the paper presents an interesting idea, and the main concern raised by the reviewers was the clarity of the paper. I believe that the authors have improved the presentation of the paper after rebuttal, however, I still believe that the paper woudl require another round of reviews before being ready for publication, in order to properly assess its contributions. 
Decision: Reject