Interpretability Through Invertibility: A Deep Convolutional Network With Ideal Counterfactuals And Isosurfaces
Authors: Leon Sixt
Martin Schuessler
Philipp Weiß
Tim Landgraf
Venue:  (2021)
Keywords: Interpretable Machine Learning
Counterfactuals
Computer Vision
Human Evaluation
User Study

Abstract: Current state of the art computer vision applications rely on highly complex models. Their interpretability is mostly limited to post-hoc methods which are not guaranteed to be faithful to the model. To elucidate a model’s decision, we present a novel interpretable model based on an invertible deep convolutional network. Our model generates meaningful, faithful, and ideal counterfactuals. Using PCA on the classifier’s input, we can also create “isofactuals”– image interpolations with the same outcome but visually meaningful different features. Counter- and isofactuals can be used to identify positive and negative evidence in an image. This can also be visualized with heatmaps. We evaluate our approach against gradient-based attribution methods, which we find to produce meaningless adversarial perturbations.  Using our method, we reveal biases in three different datasets. In a human subject experiment, we test whether non-experts find our method useful to spot spurious correlations learned by a model. Our work is a step towards more trustworthy explanations for computer vision.

Under review as a conference paper at ICLR 2021
INTERPRETABILITY THROUGH INVERTIBILITY: A DEEP CONVOLUTIONAL NETWORK WITH IDEAL COUNTERFACTUALS AND ISOSURFACES
Anonymous authors Paper under double-blind review
ABSTRACT
Current state of the art computer vision applications rely on highly complex mod- els. Their interpretability is mostly limited to post-hoc methods which are not guaranteed to be faithful to the model. To elucidate a model's decision, we present novel interpretable model based on an invertible deep convolutional network. Our model generates meaningful, faithful, and ideal counterfactuals. Using PCA on the classifier's input, we can also create "isofactuals" image terpolations with the same outcome but visually meaningful different features. Counter- and isofactuals can be used to identify positive and negative evidence in image. This can also be visualized with heatmaps. We evaluate our ap- proach against gradient-based attribution methods, which we find to produce meaningless adversarial perturbations. Using our method, we reveal biases in three different datasets. In a human subject experiment, we test whether non- experts find method useful spot spurious correlations learned by model. Our work is step towards more trustworthy explanations for computer vision. For code; ht ttt:ps/// //ao:///nnoomssseee wwwpsyymuuss.see...ooommmm 4open. www.n. wttp://eeeennneeeeeee -22
INTRODUCTION
The lack of interpretability is significant obstacle for adopting Deep Learning in practice. As deep convolutional neural networks (CNNs) can fail unforeseeable ways, are susceptible to adversarial perturbations, and may reinforce harmful biases, companies rightly refrain from automating high-risk applications without understanding the underlying algorithms and the patterns used by the model.
Interpretable Machine Learning aims discover insights into how the model makes its predictions. For image classification with CNNs, a common explanation technique are saliency maps, which estimate the importance of individual image areas for a given output. The underlying assumption, that users studying local explanations can obtain a global understanding of the model (Ribeiro et al. 2016), was, however, refuted. Several user studies demonstrated that saliency explanations did not significantly improve users' task performance, trust calibration, or model understanding (Kaur al. 2020 Addeayy et 2020 Alqaraawi al. 2020 Chu al. 2020 Alqaraawi et (2020) attributed these shortcomings to the inability to highlight global image features or absent ones, making difficult to provide counterfactual evidence. Even worse, many saliency methods fail represent the model's is behavior faithfully (Sixt et al. 2220;; /Adebayo et al., 2018 Nie et al., (018). While no commonly agreed definition of faithfulness exists, is often characterized by describing what an unfaithful explanation is (Jacovi & Goldberg (2020). For example, if the method fails to create the same explanations for identically behaving models.
To ensure faithfulness, previous works have proposed building networks with interpretable com- ponents (e.g. ProtoPNet (Chen et al. 2018) or Brendel Bethge (2018)) or mapping network activations human-defined concepts (e.g. TCAV (Kim al. 2018) However, the interpretable network components mostly rely on fixed-sized patches and concepts have to be defined a priori.
Here, argue that explanations should neither be limited to patches and not rely priori knowledge. Instead, users should discover hypotheses in the input space themselves with faithful counterfactuals that ideal, i.e. samples that exhibit changes that directly and exclusively correspond

---

Under review as a conference paper at ICLR 2021
to changes in the network's prediction (Wachter et al. 2018 We guarantee this property by combining an invertible deep neural network x = ((x)) with a linear classifier y = w T (fxx) + b. This yields three major advantages: 1) the model is powerful (can approximate any function Zhang et al. (2019), 2) the weight vector to of the classifier directly and faithfully encodes the feature importance of target class in the feature space. 3) Humanninterpreeabbl explanations can be obtained by simply inverting explanations for the linear classifier back to input space.
As local explanation for one sample æ, we generate ideal counterfactuals by altering its feature representation z along the direction of the weight vector Σ = z + o.w. The logit score can be manipulated directly via a. Inverting ž back to input space results in human- numan-understandabll counterfactual ã & ((=+ww) Any change orthogonal to w will create an "isofactual", a sample that looks different but results in the same prediction. While many vectors are orthogonal to w, we find the directions that explain the highest variance of the features using PCA. As the principal components explain all variance of the features, they can be used to summarize the model's s behavior globally.
We demonstrate the usefulness of our method on a broad range of evaluations. We compared our approach to gradient-based saliency methods and find that gradient-based counterfactuals are not ideal as they also change irrelevant features. We evaluated our method on three datasets, which allowed us to create hypotheses about potential biases in all three. After statistical evaluation, confirmed that these biases existed. Finally, we evaluated our method's utility against a strong baseline of example-based explanations in an online user study. We confirmed that participants could identify the patterns relevant to the model's output and reject irrelevant ones. This work demonstrates that invertible neural networks provide interpretability that conceptually stands out against the more commonly used alternatives.
2
METHOD
Throughout this work, we rely on the following definitions, which are based on Wachter et al. (2018): Definition 2.1 (Counterfactual Example). Given a data point x and its prediction y. counterfactual example is an alteration of æ. defined as ã = x + Ax, with a altered prediction ÿ = + Ay where Ay * O. Samples E with Ay 0 are designated "isefactuals".
Almost any A will match the counterfactual definition, including those that additionally change aspects which are unrelated to the model's 's prediction, e.g. removing an object but also changing the background's ' color. is desirable to isolate the change most informative about prediction:
Definition 2.2 (Ideal Counterfactual). Given set of unrelated properties (xx) = {&r (z)), a sample T is called ideal counterfactual of æ if all unrelated properties & remain the same.
The following paragraphs describe how we generate explanations using an invertible neural network pp: R" + F". The forward function y maps a data point æ to a feature vector z = vp(x). Since o is invertible, one can regain x by applying the inverse x yp (z). We used the features z to train binary classifier (ax = w² + b that predicts the label y. addition to the supervised loss, we also trained & as a generative model (Dinh et al. 2016 2015) to ensure that the inverted samples are human-undeersanddall.
Counterfactuals To create a counterfactual example ~ for a datapoint æ. we can exploit that w encodes feature importance in the z-space directly. To change the logit score of the classifier, we simply add the weight vector to the features z and then invert the result back to the input space: ã = 4 '(x + aw). Hence. for any sample x. we can create counterfactuals 전 with an arbitrary change in logit value Ay ow"" w by choosing a accordingly. Figure Ia shows several such examples. Since the generation (i-11) and prediction (4p) are performed by the same model, we know that E will correspond exactly to the logit offset aw" "w. Consequently, z is a faithful explanation.
To show that our counterfactuals are ideal, we have to verify that no property unrelated to the prediction is changed. For such a property [(x) x" z, v has to be orthogonal to w. As the unrelated property& & does not change for the counterfactual {(Ø) = v" v" (z aw) = - 2 E(x),
we know that = O (z aw) indeed an ideal counterfactual.
((r))) could actually be non-linear in the features as long as the gradient 3z is orthogonal to un
2

---

Under review as a conference paper at ICLR 2021
PCA Isosurface Since users can only study limited number of examples, it is desirable choose samples that summarize the model's behavior well (Ribeiro et al. 2016: Alqaraawi et al. 2020)
For counterfactual explanations, the change A may vary significantly per example as ((() is non-linear function. As each æ has a unique representation z in the feature space, we want to find
examples describing the different directions of the feature distribution. To isolate the effect of w, such examples would have the same prediction and only vary in features unrelated to the prediction.
We implement this by first removing the variation along w using a simple projection 2 (w" z/w" wy w) and then applying PCA on 21. The resulting principal components e em are orthogonal to except of the last principal component em which has zero variance and can therefore be discarded. The principal components span a hyperplane ow + Em-1 B.e,. Since all samples on this hyperplane have the same prediction (a logit value of aw" w), it is an isosurface.
As a principal component e; is a vector in the z-space, we can create counterfactuals for it (e; aw) and understand how the changes of adding w differ per location in the z-space. The ei..... cm-1 are sorted by the explained variance allowing to prioritize the most relevant changes in the data. As the principal components cover the whole feature distribution, understanding the effect of w on them allows forming global understanding of the model's behavior.
Saliency maps Saliency maps are supposed to draw attention to features most relevant to a predic- tion. In our case, it is most reasonable to highlight the difference between and the counterfactual ã. We can measure the difference although in an intermediate feature map h. The saliency map of an intermediate layer can be resized to fit the input's 's resolution information remains local in convolutional networks. Per feature map location j), we calculate the similarity measure (t.)) = | (os(Z(Ahij. h.z)). The sign of the saliency map m depends on the alignment of the change Ah with the feature vector h. i.e. (<(Shi huy) > 0). The magnitude is dominated by the length of the change |Ahiy- Figure Tb presents saliency maps for the CELEBA Allractive label.
Model Our invertible network follows the Glow architecture (Kingma & Dhariwal, 2018). The network is trained to map the data distribution to standard normal distribution. We reduce the input dimensionality of (3, 128, 128) down to (786) by fading half of the channels out with each downsampling step. When generating a counterfactual, we reuse the values out-faded from the lower layers as they correspond to small details and noise. We have downsampling steps and 351 flow layers. The network has 558.769.66 parameters in total. An important design decision is that the final layer's ' output is not input to the linear classifier. The PCA would fail to discover meaningful directions as the A (0, 7) prior induces equal variance in all directions. The classifier uses the output of layer 321. The layers 322-351 are optimized using the standard unsupervised flow objective. For the first 321 layers, we also train on the classifier's supervised loss (for details see Appendix A.
EVALUATION
We evaluated the ability to construct hypotheses about the model's behavior on three datasets and with user study. We focused on these aspects as our method is faithful by construction, needing no empirical confirmation. Instead, we use the strong faithfulness guarantees of our model to evaluate gradient-based attribution methods.
3.1 HYPOTHESIS DISCOVERY
CelebA A claimed utility of our method is that it allows users to discover hyphotheses about the models features used for prediction. We choose CELEBA (Liu al. 2015). popular face dataset, because it is a challenging dataset for feature attribution: how can an abstract concept as attractiveness be linked to pixels? Additionally, it already contains annotations (e.g. make-up. accessories, hair), which makes it casier for us to accept or reject given hypothesis about feature importance.
We especially focus on the Attractive class as it is unclearer what the relevant features are, are.The The CELEBA Dataset in general and the class attractive in particular are ethically questionable. How can a subjective label, which depends on individual or even cultural preferences, be reduced to a binary label? Unfortunately. (Liu al. 2015) did not state the annotation process (which is considered good practice (Gebru et al.) 2020; Geiger et al. 2020)). Furthermore, the dataset was criticized for lacking diversity (Kärkkäinen & Joo 2019).
3

---

Under review as a conference paper at ICLR 2021
IL
IR
2L
2R
3L
3R
4L
4R
5L
5R
6L
6R
7L
7R
8L
8R
8.0
8.0
4.0
4.0
0.0
0.0
4.0
.4.0
-800
-80
(a) CELEBA
(b) CELEBA
Figure 1: (a) We generate counterfactual images by moving along the direction of the classifier weights w of the attractive class and inverting it back to the input. Last row shows the saliency maps from the center row (logit y=0) to the top row (y=8). Blue marks features changed into different direction and red marks features getting enhanced. (b) We extract principal components (L=e,, R,- orthogonal to the classifier weight w. All images in a row have the exact same logit score given on the left. The saliency maps show the change between the bottom (y=-8) and top (y=8).
Figure [boshows the first 8 principal components at different logit values. We base our investigation on them, they cover the feature distribution well by construction. At this point, invite the reader to study the explanations: What are your hypotheses about the model's s used features?
Studying the counterfactuals in rows (3R, 5L, 6R, 8R), one might hypothesize that glasses influence the prediction of attractiveness negatively. To validate this, we analyzed our model's predictions on the test set. Since glasses are labeled feature of CELEBA it is easy to test the hypothesis empirically. Only 3.5% of the portrait photos, which are showing glasses were labeled as altructive by the model. Furthermore, the correlation of the presence of glasses and the logit score was 0=35.
Another insight noticeable in 1I. is that the amount and density of facial hair changes the prediction. The correlation of the absence of facial hair with the attrativvnesss logit score was r=0.35. At the same time, less head hair seemed to reduce attractiveness predictions in rows IL, 2R, 4R. Row 6L paints the opposite picture, which illustrates the varying effect w can have on different datapoints. We found correlation (r = 0.30) of hair-loss (combination of baldness or receding hairline) with attractiveness.
Indicative of higher attractiveness appear to be more feminine appearance (e.g. 4R in Figure D_ This hints to gender bias, which we confirmed as only 20.0% of men are predicted to be attractive, and the label male was negatively correlated with the prediction (r = -059).. Further, it is noticeable that counterfactuals for higher attractiveness tend to have redder lips (IR, 2R,4R and 5L). This hypothesis could also be confirmed the label Wearing Lipstick is also positively correlated (r = 0.64). For age, similar patterns can be found in IL, 3R, 8L (r = 0.44). Table iin the Appendix D lists the correlation of all 40 attributes. Some attributes cannot be found in the principal components because the cropping hides them (double chin, necklace, necktie). Others describe local details such arched eyebrows, earrings. While earrings do not show up in the counterfactuals, they are correlated with the model's 's logit score by r=0.20. This might be because PCA tends to capture global image features while smaller local changes are scattered over many principal components. Another explanation could be that carings are actually not that relevant: if we control for gender using partial correlation the earings are only correlated by r=-0.1
Darker skin color seems to influence the network negatively, as in principal components (2R, 3R, 6L) light skin color suggests high attractiveness. Since CELEBA has no labels for skin color, we annotated 3250 randomly selected images: 249 photos matched the Fitzpatrick skin type V-VI and were labeled as dark skin (Fitzpatrick) 1986). For light skin, the percentage of Attractive was 52.0%. The same bias is contained in the model: 0.118 (-0.22, -0.15)
Two4Two
The Two4Two dataset (Anonymous) 2020) is a set of -eeputer-generatedd images intended to evaluate interpretable ML to test both humans and algorithms. While the dataset is simple, we control the data generation process and can create arbitrary images to test the model. The dataset contains two abstract animals, Sticky and Stretchy. For Sticky, the right arms inwards and for Stretchy outwards (see Figure 2b). As the arms overlap sometimes, it is beneficial also to use the color which is slightly predictive (blue for Stretchy and red for Sticky). Building
moved
4

---

Under review as a conference paper at ICLR 2021
II.
21.
2R
M..
41.
4R
ST.
10.0
Stretchy
Sticky
5.0
0.0
-5.0
2000
:
2
Arms outwards
Arms inwards
-10.0
(a) Principal Components
(b) Overview
Figure 2: (a) The principal components for Two4Two. Sticky is on the top and Strechy below. The saliency maps shown below fail to highlight the object movement well. (b) main feature of Stretchy are the outward moved left arms. For Sticky, they are moved inwards
blocks (cubes spheres), bending, rotation, and background are sampled independently. For the Two4Two dataset, the invertible neural network oo was only trained on an unsupervised loss, i.c. the gradients of the classifier were detachted. Probably due the datasets simplicity, had problems to align the unsupervised and supervised loss well.
The principal components in Figure 2aIsuggest that the model indeed learned to use the color bias. We an confirm this by resampling only the color and measure how the logit score is correlated: r=0.552. For the arm's position, Sticky on the top seems to be more rotated, which we can also confirm as only changing the rotation results in correlation of the logit score with the absolute value of teh rotation of with r=0. 136 (0. 11, 0.1 1.56) At high rotations, the model is more certain that it is Sticky. Although not intended by the dataset, this bias can be well explained by the fact that yp was not trained on the supervised loss.
found correlation with the model's probability of -0.798. Additionally,
Black Mice We wanted to check our method on dataset which is not already known to have biases as the CelebA dataset and harder for human to understand. The BLACK MICE Mataset[Andresee et al. (2020) contains images of laboratory mice after different treatments. The label to predict is related the amount of pain. For a detailed discussion the dataset, see Appendix ??. The main take-away away point is that we find that the yellow bedding material, which is changed by our model's counterfactuals, is indeed predictive of the label.
3.2 COMPARISON OF THE GRADIENT OF x AND THE DIRECTIONAL. DERIVATIVE d //w
In this evaluation, we propose a simple validity check for attribution methods and apply it to our method and gradient-basse based attribution methods. The idea is relate saliency maps to counterfactuals. As saliency maps should highlight features most influential for the outcome of a datapoint, amplifying these features should increase the prediction and therefore create a counterfactual. We propose the following test: integrate the raw feature attribution values and then check if (1) the counterfactual increases the logit score and (2) if the changes are into the direction of w or rather into the direction of unrelated properties. We measure (2) by calculating the changes in the directions of the principal components: g Ez where E is the matrix of all e,
We construct an infinitesimal version of our counterfactuals by lim (2 This gives the alwi directional derivative the input w.r.t. to the classifier weight: V.. = Vm 1 Moving the input into the direction V. will result in a move of into the direction
d 1 /z)/
We evaluate the directional derivative against the raw gradient, which serves as a basis for many saliency methods (SmoothGrad, LRP.. LRP 8. y-rule, and integrated gradients (Smilkov al.]
2017 Bach et al. 2015 Montavon et al. 2019 Sundararajan et al. 2017 [ Additionally, include SmoothGrad (sm.g) and build two additional methods by penalizing changes in the unrelated
2018) uses the directional derivative the networks output w.r.t. a concept vector v: de Different to our method, TCAV computes the gradient of the forward model and not on the inverse sp
TCAV (Kim et al.
df
reader familiar with differential geometry might recognize this as the pushforward to using y 4 The gradient and the directional derivative have mathematical similarity which can be seen on the Jacobian:
V aaf = J. y and a = Jg- (z)w.
5

---

Under review as a conference paper at ICLR 2021
ima
-5.1
6.0
46.9
8.8
10.0
9.8
cf.
Change
cf.s.
ima.
di.der.
grad.
pe.g.
sm..g
po.s.g
int.g
sm.g.
di.er
gnd.
peg.
smllgg
pesg.
10 x A 10 x A 10 x A 10
(a) Saliency maps
(b) Integration
(c) Change to
Figure 3: (a) Saliency maps computed for the Eyeglasses class of our method (cf.sl.), integrated
gradients (int.g.), and SmoothGrad (sm.g.). cf. denotes counterfactuals with logit y= (b) Integration of the feature attribution values, e.g. gradient w.r.. to single neuron. The gradient (grad) results in a strong logit change (given on top) but fails to create visible changes. Differences with the original images (img) are magnified below (x 10). SmoothGrad and the respective penalized version (pe.g gr and pe.s.g). show similar results. The directional derivative d / adds sunglasses. (c) The distribution of & is shown in the first row. All gradient-basel based methods result in strong and therefore less interpretable counterfactual. The directional derivative V w changes & little.
properties & using a mean squared error with the & of the original image (pe.gr. for gradient and for SroothGrad pe.s.g). The integration is done by iterative steps into the direction of the integrated quantity, e.g. for the gradient we would calculate 24+1 = 24 + (x)()) where y is a small step (see Appendix A.2 for all technical details).
Figure 3b shows exemplary results of the integration for the Eyeglass dimension. While the gradient- based counterfactual increases the logit score by an order of magnitude, the resulting image hardly different from the original. Only noise patterns appear similar to adversarial examples. SmoothGrad results in both a lower logit score and even smaller changes to the image. Penalizing changes in unrelated properties only yields amplified noise patterns. At the start of the integration, the difference Eo is zero, which probably results in first moving along & In contrast, integrating the directional derivative adds sunglasses to the astronaut meaningful counterfactual.
We measure the quality of a counterfactual by measuring how strongly unrelated factors change on 100 random samples and report the results in Figure 3c Thus, gradient-based counterfactuals do not only explain the increase of the logit score, but also all the other changes too. A user studying the gradient counterfactual could not differentiate between changes done to the prediction and the unrelated factors. The counterfactual based on the directional derivative keeps the independent factors almost unchanged up to numerical imprecision.
3.3 HUMAN SUBJECT STUDY
Our aim was to evaluate whether counterfactual interpolations can help lay users to form hypotheses about models used patterns and potential biases. Evaluating explanation techniques with users is important though a challenging endeavor as requires mimicking a realistic setting, while avoiding
overburdening participants (Doshi- & Kim 2017: Wortman Vaughan & &alllach] 2020).
The choice of the dataset is important for any evaluation. Some datasets introduce participants' cofounding factor (e.g. images of dog breeds). While others like CELEBA
domain knowledge introduce subjectivity. Datasets can have many relevant features, creating an enormous amount of possible and valid hypotheses. If participant were allowed develop hypotheses about them without limitation this would require us to mostly evaluate them manually which would be too labor intensive. Asking participants to reason about pre-selected hypothesis prevents us from assessing their total understanding of the model as there are potentially many relevant features.
We chose the Two4Two data set (Section 3.1 as addresses these issues (Anonymous) 2020). The simple scenario enables us to control the available patterns and limit the number of feasible hypotheses, allowing for comparable quantitative analysis. Concretely, we assessed a participant's 'ss judgment about the plausibility of six hypotheses. Three hypotheses were reasonable (sensitivity to spatial compositions, color, and rotation). Two others were not (sensitivity to background and shape of individuals blocks). We also asked them to reason about the model's 's maturity and measured their perception of the explanations using applicable statements taken from the Explanation Satisfaction Scale (Hoffman et al. 2018).

---

Under review as a conference paper at ICLR 2021
Baseline Selection Many studies in machine learning solely demonstrate their methods feasibility without a baseline comparison (e.g. Ribeiro et al. (2016); Singla et al. (2020)) ). In contrast, we carefully considered what would be the best alternative method available to allow users to discover hypotheses about a model. As discussed previously in this work, many feature attribution techniques suffer from lack of faithfulness and fail to provide meaningful counterfactuals. If counterfactuals are meaningful and faithful to the model they can be expected to look similar. Hence, comparing our method to other counterfactual generation methods (e.g. to GANs (Singla al. 22200) provides limited insight about their practical usefulness if there are alternative ways of discovering similar hypotheses. As for saliency maps, in addition concerns about their faithfulness, there are also growing concerns about their practical usefulness. While early works found they can calibrate users' trust in a model (e.g. Ribeiro et al. 2016)), more recent works cast doubts about this claimed utility (Kaur et al. 2020 Chu et al. 2020). Studies found that while they are useful to direct users' attention towards relevant features, they facilitate limited insight (Algaraawi ct al. 2020 Chu et al. 2020). Other studies found they may even harm users' understanding about errors of the model (Shen & Huang 2020). After all, users often seem to ignore them, relying predominantly on predictions
instead when reasoning about a model (Chu et al. 2020; /Adebayo al. 2020
While we introduce a faithful saliency method, we do not claim that would not suffer from the same usability problems, especially with lay users (see Figure 7 for examples generated for Two4Two). After all our maps would need to be used in conjunction with counterfactuals, potentially adding a dependent variable (presence of saliency map) to experiment. For these reasons, we decided against considering saliency maps in this evaluation.
We also did not consider methods based on infilling (e.g. Goyal al. (011 ), as we expected them to suffer from similar usability problems. For example, as they explain features locally by removing them. paying no attention to overlapping features, they can be expected to remove the entire object from the scene when explaining the model's 's bias towards the object's 's color. This would leave the user puzzled what feature of the object (shape, position or color) is important.
A simple alternative is to study the system predictions on exemplary input, Such reasoning on natural images to understand model behavior has surfaced as a strong baseline in another study (Borowski al. 2020). Hence, we choose example-based explanations as our baseline treatment.
Explanation Presentation Considering that participants' attention is limited and to allow for fair comparison, we wanted to provide the same amount of visual information in both conditions. We choose a 30x5 image grid (3 rows shown in Figure 4p. Each column represented a logit range. Ranges were chosen that high confidence predictions for Stretchy were shown the far left column and high confidence predictions Sticky on the far right. Less confident predictions were shown in the directly adjoining columns. The remaining middle column represented borderline-cases. This visual design had prevailed throughout numerous iterations and ten pilot studies, as it allows users to quickly scan for similar features in columns and differing features in rows.
Both conditions only varied in the images that were used to populate the grid. In the baseline, the grid was filled with images drawn from validation set that matched the corresponding logit ranges. In the counterfactual interpolations conditions, only the diagonal of the grid was filled randomly with such "original" images. They were marked with a golden frame. The remaining cells were filled row-wise with counterfactuals of the original images that matched the corresponding columns score range.
Our online study was preregistered 5l and followed between grruup group design. Participants (N=60) were recruited from Prolific and needed to hold an academic degree with basic mathematical education. Participants were randomly but equally assigned to view either counterfactual interpolations or the baseline. Upon commencing the study on the Qualtrics platform, participants were shown handcrafted video instructions. After that, they studied the image grid while rating their agreement to six statements 7-point Likert scale. Participants also rated their agreement to four applicable statements taken from the Explanation Satisfaction Scale (Hoffman et al. 2018).
Study Results and Discussion The significance of rating difference was assessed using Kruskal- Wallis Test. To account for multiple comparisons, we applied Bonferroni correction to all reported p-values. For a detailed assessment of all preregistered hypothesis, please refer to the Appendix (Section E. Figure 4h summarizes the responses.
"eee supplementary material
7

---

Under review as a conference paper at ICLR 2021
CF 97% 0% 0%
3%
Strongly
Stretchy
Blicky
Arms
agree
BL 83%
17%
Agree
CF 70%
0%
30%
Somewhat
Color
BL
43%
7%
50%
agree
S
CF 33%
10%
57%
Neither agree
Rotation
BL
47%
10%
43%
nor disagree
Somewhat
Bkgmd
CF
13%
3%
83%
disagree
(b) Treatment: Counterfactual
BL
10%
7%
83%
Disagree
Stretohy
Slicky
CF 27%
0%
73%
Strongly
Blocks
BL
10%
0%
90%
disagree
CF:
Maturity
CF 73%
3%
23%
Counterfactual
BL 70%
0%
30%
BL:
100%
50%
0%
50%
100%
Baseline
(a) Results
(c) Treatment: Baseline
Figure 4: Left: Participants agreements to statements about the networks used patterns. Right: The study interface (vertically cropped) the counterfactual interpolations (Top) and baseline condition (Bottom). Each participant was assigned to only one treatment.
Counterfactual interpolations allowed users identify the model's main pattern: the position of the arms of Stretchy and Sticky. They did this with high certainty, as 83.34% strongly agreed with the corresponding statement. They were more certain about this pattern than with the baseline technique (H(1)= = 8.86, p 0.018), even though the baseline technique also performed well for this task. The large majority (70%) also identified the color bias with counterfactual interpolations, while only 43% identified this bias using the baseline explanations. However, the difference in rating between conditions for the corresponding statement about color bias was not significant (H(I) = 3.21, p = 0.42). Participants who had missed the color bias using our method were later asked to provide their reasoning. A participant stated: "I would think that the color would be relevant if saw an example where it went from certain to very certain and only the color, brightness intensity changed." Such rule-based rather than probabilistic cognitive models of the network may have led users to reject the presence of color bias even though we instructed them clearly that interpolation would only change relevant features.
To our surprise, fewer participants noticed the network's more subtle bias towards object rotation in both conditions. As Figure inddicatss, participants were somewhat undecided about the relevance, leaning rather conclude that the network is not sensitive to rotation. As a limitation, we note that participants may not have noticed the rotation bias due to how we had phrased the corresponding statement. When we asked them explain their reasoning, many explained that they instead focused on the individual blocks' rotation rather than the whole animal.
Both explanation techniques allowed participants to confidently reject statements about irrelevant patterns (sensitivity to the background, sensitivity to the type of blocks). We argue this indicates a high quality of collected responses and good utility of both explanation techniques. Somewhat worrying is participants' assessment of the system's 's maturity. They were very confident that the network has learned the right patterns and is ready to use for both techniques. Such bias towards model deployment has previously surfaced in other studies (Kaur et al. 2020).
Explanation Satisfaction ratings were very high for both techniques (see Figure 10 in Appendix) underlining that participants perceived both methods very well. While this also means that method was unable to outperform the baseline, it also shows that our careful visual design and our clear instructions how use the explanations technique were well received. As a limitation, that participants may have found the introductory videos very informative as many reported enjoying the study. This may have led them more favorable ratings and the conclusion that they understand the system very well regardless of the explanation technique they had used.
note
4 RELATED WORK
Others have suggested methods for counterfactual generation. Chang et al. (2019) identifies relevant regions by optimizing for sufficiency and necessarity for the prediction. The classifier is then probed
8

---

Under review as a conference paper at ICLR 2021
on the counterfactuals replacing relevant regions with heuristical or generative infilling. Goyal et al. (2019) find regions in a distractor image that would change the prediction if present. Both works assume that relevant features are localized, but for many datasets these may cover the entire image. e.g. changes due to gender or age in face images. Singla et al. (2020); Liu et al. (2019); |Baumgartner eet al. (2018) explain black-box neural network by generating counterfactuals with GANs which can generate counterfactuals of similar or even better visual quality. However, the GANs model does not have align with the explained model perfectly, e.g. see Figure in (Singla et al. 2020
The TCAV method (Kim et al. 2018) estimates how much manually defined concepts influence the final prediction. Recent work has extended TCAV to discover concepts using super-piells automatically (Ghorbami et al. (2019). Goyal et al. (2020) extend TCAV to causal effects of concepts and use a VAE as generative model.
Being able to interpolate in feature space and inverting these latent representations is one of the advantages of invertible networks (Jacobsen et al. 2018 Kingma & Dhariwal 2018). Mackowiak] et al. (2020) use invertibility to improve the trustworthiness but focus on out-of-distriuttio and adversarial examples. (Rombach et al.) 2020; Esser et al. 2020) employ invertible networks to understand vanilla convolutional networks better.
One example of an interpretable model is ProtoPNet (Chen et al. 2019). The feature maps of image patches that correspond to prototypical samples in the dataset are used for the final prediction. This way, a result can be explained by pointing to labeled patches. The method is limited to a fixed patch size and does not allow counterfactual reasoning. Another patch-based interpretable model is proposed in Brenddl & Bethge (2018).
Our combination of PCA and invertible neural networks for interpretability is novel. The finding that the directional derivative corresponds to ideal counterfactuals, whereas the gradient does not, has not been reported before. We are also not aware of a user study has previously demonstrated that visual counterfactual can help users identify biases of a neural network.
5 DISCUSSION
A disadvantage of our method is that it requires an invertible network architecture - the weights of an existing CNN cannot be reused. Learning the input distribution entails additional computational costs, when training an invertible neural network. For non non-image domains such as natural language or graphs, the construction of inverse is currently more difficult. However, first works have taken on the challenge (MacKay et al. 2018 Madhawa et al.] 2019). Furthermore, learning the input distribution requires larger network. Given that our method performed similar to the baseline in the user study in all but one category, an obvious question is whether it is worth the additional effort.
However, the same question applies to almost any explanation method and remains largely unan- swered. Unfortunately user evaluations that include reasonable baselines are very rare. An additional finding of this work that explanation methods should be evaluated for their utility and usability against a reasonable baseline. For image classification our work shows, that studying the raw input and corresponding predictions is such a reasonable baseline. has the potential to allow lay users to identify, many but not all, high level features used for prediction. Even though we found strong baseline, the user study also demonstrated that our method is useful to lay users as they found two out of three relevant patterns and rejected two more irrelevant patterns. It also highlights that some more subtle patterns may still go unnoticed even when using our method.
We would like argue that the additonal effort required to implent invertability, may well be justified especially in high-stakes domains. Combining an invertible neural network with a linear classifier enables the use of simple explanation techniques which are otherwise restricted to low complexity models. Here, we can use them on a deep model with much greater predictive power. Counterfactuals can be created by simply using the weight vector of the classifier. In contrast to many other techniques. they are faithful to the model, changing only features relevant for the prediction. Since, they can be inverted back to the input space the high level features they encode are human interpretable. This allows users to discover hypotheses about the models used patterns largely independent of their preconception about feature importance. Using our method we found biases in three datasets including some that have not been previously reported. As we have demonstrated in this work, that invertibility has mayor advantages for interpretability.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pp. 9505-9515, 2018.
Julius Adebayo, Michael Muelly, Ilaria Liccardi, and Been Kim. Debugging tests for model explana- tions, 2020.
Ahmed Alqaraawi, Martin Schuessler, Philipp Weiß, Enrico Costanza, and Nadia Berthouze. Evaluat- ing saliency map explanations for convolutional neural networks: A user study. In Proceedings of the 25th International Conference on Intelligent User Interfaces, IUI '20, 263-274, New York, NY, USA, 2020. Association for Computing Machinery. doi: 10.1145/337782553377511
Nick Andresen, Manuel Wöllhaf, Katharina Hohlbaum, Lars Lewejohann, Olaf Hellwich, Christa Thöne- Reineke, and Vitaly Belik. Towards fully automated surveillance of well-being status in laboratory mice using deep learning: Starting with facial expression analysis. Plos one, 15(4): e0228059, 2020.
Anonymous. Two4two dataset. Under Review, 2020.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10(7), 2015. doi: 10.137 1/ja7///juaal..ooee wwt..pp00....0.0.....0 0130140.
Christian Baumgartner, Lisa M Koch, Kerem Can Tezcan, Jia Xi Ang, and Ender Konukoglu. Visual feature attribution using wasserstein gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8309-8319, 2018.
Judy Borowski, Roland S. Zimmermann, Judith Schepers, Robert Geirhos, Thomas S. A. Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain cnn activations better than feature visualizations. 2020.
Wieland Brendel and Matthias Bethge, Approximating cnns with bogat---ocll models works surprisingly well on imagenet. In International Conference on Learning Representations, 2018.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and D. Duvenaud. Explaining image classifiers by counterfactual generation. In ICLR, 2019.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: deep learning for interpretable image recognition. In Advances in neural information processing systems, pp. 8930-8941, 2019.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances Neural Information Processing Systems, 2018.
Eric Chu, Deb Roy, and Jacob Andreas. Are visual explanations useful? a case study in model-in-the inethe- loop prediction, 2020.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components
estimation. CoRR, abs/1440..556. 2015.
Laurent Dinh, Jascha Sohk-Dicks and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 100.:08030...,,, 2016.
Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv: 1702.08608, 2017.
Patrick Esser, Robin Rombach, and Bjorn Ommer. A disentangling invertible interpretation network for explaining latent representations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9223-9232, 2020.
T. Fitzpatrick. cutraviolet-induced pigmentary changes: benefits and hazards. Current problems in dermatology, 15:25-38, 1986.
10

---

Under review as a conference paper at ICLR 2021
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III au2, and Kate Crawford. Datasheets for datasets, 2020.
R. Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai, Jie Qiu, Rebekah Tang, and Jenny Huang. Garbage in, garbage out? do machine learning application papers in social computing report where Fairness, Accountability, and Transparency, FAT* '20, pp. 325-336, New York, NY. USA, 2020. Association for Computing Machinery, ISBN 978 178466366.67.. www./org/o00.... 11450...1.433333 095 3372862
training data comes from? In Proceedings of the 2020 Conference on
doi: 10.1 10.1145/3351099.3378886
URL htttts::////oo http://ooooogg/11.11111111
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic cancept-based explanations. In Advances in Neural Information Processing Systems, pp. 9277-9286, 2019.
Yash Goyal, Z. Wu, J. Ernst, Dhruv Batra, D. Parikh, and Stefan Lee. Counterfactual visual explanations. ArXiv, abs/1904.0074f. 2019.
Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace), 2020.
Robert R Hoffman, Shane Mueller, Gary Klein, and Jordan Litman. Metries for explainable ai: Challenges and prospects. arXiv preprint wrXiv:1812.04600 2018.
Katharina Hohlbaum, Bettina Bert, Silke Dietze, Rupert Palme, Heidrun Fink, and Christa Thöne- Reineke. Severity classification of repeated isoflurane anesthesia in c57bl/6jrj assessnng the degree of distress. PLOS ONE, 12:1-21, 06 2017. doi: 10. Lojoumall.pon..11 00w.ome........... URL buttps://don www.oogg..........o 13t...1117/o/oraaa.... pone 0179588
Katharina Hohlbaum, Bettina Bert, Silke Dietze, Rupert Palme, Heidrun Fink, and Christa Thöne- Reineke. Impact of repeated anesthesia with ketamine and xylazine on the well-heing of c57bl/6jrj mice. PLOS ONE, 13(9):1-24, 09 2018. doi: 10.137 /oo.//ooallloooee0000000 // //doi. 1ww.//ouraaaaaaa.o..oooa pone. 0203559
URL https
Jöm-Henrik Jacobsen, Arnold W.M. Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. In International Conference on Learning Representations, 2018. URL https://
openreview feret/forumtt -iw.eiiddHHJJsss.. HJsjjMb02
Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4198 4205, Online, July 2020. Association for Computa- tional Linguistics. doi: 10.18653/v1/202011 mcannm..m....88. URL https:: http://wwwwwwwwwww. org/
anthology/2024 acl main 386
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wortman Vaughan. Interpreting interpretability: Understanding data scientists' use of inter- pretability tools for machine learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20, pp. 1-14, New York, NY. USA, 2020. Association for Computing Machinery. ISBN 9781450367080. doi: 00.1145/311383313337222 337661
URL
Pittps://do. org/10
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference Machine Learning, pp. 2673-2682, 2018.
Diederik P. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. ArXiv, abs/1807.03039, 2018
Kimmo Kärkkäinen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age, 2019.
Dale Langford, Andrea L Bailey, Mona Lisa Chanda, Sarah E Clarke, Tanya Drummond, Stephanie Echols, Sarah Glick, Joelle Ingrao, Tammy Klassen- Ross, Michael L FrCroix-Fralish. Lynn Matsumiya, Robert E Sorge, Susana G Sotocinal, John M Tabaka, David Wong, Am M J M van den Maagdenberg, Michel D Ferrari, Kenneth D Craig, and Jeffrey S Mogil. Coding of

---

Under review as a conference paper at ICLR 2021
facial expressions of pain in the laboratory mouse. Nature Methods, 7(6))447- June 2010. ISSN 7091, 1548-7105. doi: 10. 10.1038/mmeth. 1455.. URLbttp: www. .nt.nnatuuu com/
axticles/rmeta.. 1455
Shusen Liu, Bhavya Kailkhura, Donald Loveland, and Yong Han. Generative counterfactual intro- spection for explainable deep learning, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Matthew MacKay, Paul Vicol, Jimmy Ba, and Roger B Grosse. Reversible recurrent neural networks. In Advances Neural Information Processing Systems, pp. 9029-9040, 2018.
Radek Mackowiak, Lynton Ardizzone, Ullrich Köthe, and Carsten Rother. Generative classifiers as basis for trustworthy computer vision. arXiv preprint urXiv:2007./5036, 2020.
Kaushalya Madhawa, Katushiko Ishiguro, Kosuke Nakago, and Motoki Abe. Graphnvp: An invertible flow model for generating molecular graphs. arXiv preprint urXiv: 1600, 2019.
Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojeiech Samek, and Klaus-Robert Müller. Layer-wise relevance propagation: overview. In Explainable Al: interpreting, explaining and visualizing deep learning, pp. 193- 193-209.. Springer, 2019.
Daniel Neurath. Analysis of the eye visual indicator for the well-being status of laboratory mice. Unpublished Bachelor Thesis, 2020.
Weili Nie, Yang Zhang, and Ankit Patel. A theoretical explanation for perplexing behaviors of backpropagation- visualizations. In International Conference on Machine Learning, pp. 3809-3818, 38,8, 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
Robin Rombach, Patrick Esser, and Björ Ommer. Making sense of cnns: Interpreting deep represen- tations & their invariances with inns. In Proceedings of the European Conference on Computer Vision, 2020.
Hua Shen and Ting-Hao Kenneth Huang. How Useful Are the Machine-Gennratted Interpretations to General Users? A Human Evaluation Guessing the Incorrectly Predicted Labels. In Proceedings of the Eighth AAAI Conference on Human Computation and Crowdsourcing (HCOMP- 20), volume 8, pp. 168-172, Virtual, October 2020. AAAI Press. ISBN 978-1-57735-848-0
Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich. Explanation by progressive exaggeration. In International Conference on Learning Representations, 2020.
Leon Sixt, Maximilian Granz, and Tim Landgraf. When explanations lie: Why many modified bp attributions fail. In Proceedings of the 37th International Conference on Machine Learning. PMLR, 2020.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv: 1706. 03825, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Vait Volume 70, pp. 3319-3328. JMLR.org, 2017.
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harvard Journal of Law Technology, 31(2), 2018.
12

---

Under review as a conference paper at ICLR 2021
Jennifer Wortman Vaughan and Hanna Wallach. A human-centered agenda for intelligible machine learning. This a draft version of a chapter in a book to published in the 2020 21 timeframe., November 2020. https://:
www.:soft..oo.occmmommoooooooo.e com enn wuw.eareeearrrcaaaeaaaaacaaaa ott.://eeeeotttttaooooooooooooor
ne-
learning/
H. Zhang, X. Gao, Jacob Unterman, and Tom Arodz. Approximation capabilities of neural odes and invertible residual networks. arXiv: Learning, 2019.
13

---

Under review as a conference paper at ICLR 2021
Dataset
NLL
Accuracy
Supervised
BLACK MICE
3.28
86.5
86.4 (our model) 88.5:1..5
(Andresen et al. 2020)
CELEBA
2.55
88.2
89.9
TWO4Two
0.63
84.9
98.2
Table 1: Model Performances on the different datasets. Negative Log Likelihood in bits/pixels.
A APPENDIX: TECHNICAL DETAILS
A.1 NEURAL NETWORK ARCHITECTURE
Our model follows the Glow model closely (Kingma & Dhariwal 2018). Similarly, we use block of actnorm, invertible x 1 convolution and affine coupling layer. After 18 blocks, add an reshuffle operation to reduce the spatial dimensions by a factor of 2 and half of the channels are faded out. The first layer is The classification is done before the final mapping the prior N( 1).
As described in section 2 we trained added the classifier after layer 321 before the final layer 351. Let op denote the first 32I layers and p E H R" the last. We train y both on supervised loss from the classifier f(z) and an unsupervised loss from matching the prior distribution (N, /) and the log determinante of the Jacobian. f is only trained on the unsupervised loss:
arg min Tum (M xf(x)) + B Trup (w" (f(x)) + b, Yrrue). 0p.0
For the supervised loss Lup. we the binary cross entropy although our method is restricted to this loss function and could be extend to more complex losses easily. As unsupervised loss Tauns we use the commonly used standard flow loss obtained from the change of variables trick Dinh et al. (2016). The unsupervised loss ensures that inverting the function results in realistic looking images and can also be seen as a regularization.
In total, & and have 158.769.600 parameters. We use the identical network architecture on all datasets.
A.2 DETAIL TO INTEGRATION: SECTION 3
In section we integrated the gradient and the directional dirivative. We used the torchdi ffe package. For figure 3b we integrated from t=[0, 11] using the midpoint method with 20 steps. Here the integration was done in layer 40. As this rather slow, used 5steps and = [0, 4] to determine the differences in the unrelated factors & again in 40, shown in Figure 3c
B BLACK MICE
In this case study, we apply our method on the BLACK MICE dataset (Andresen et al., 2020). In contrast CELEBA, the images vary more strongly in location, size, posture, and camera angle. The dataset contains a total of 32576 images of 126 individual mice. [Andresen et al.) (2020) trained a ResNet and reported an accuracy of 88.542.6% using 10-fold cross-validation. Our model achieves a similar accuracy of 86.5 86.5% tested on a single fold. The images were collected for earlier works (Hohlbaum al. 2018 2017). The mice were divided in three groups: castration, only anesthesia, or untreated. A binary label marks any signs of post-surgical/ anestthei effects. According to (Langford al.) 2010). typical features for pain are squeezed eyes, pulled- nulled-back ears, bulged checks and nose, and change in whisker position.
Together with the authors of (Andresen al. 2020), we reviewed our model's explanations. We confirmed that counterfactuals affect different image features accordingly: eyes, nose, ears, whiskers, and head position change in biologically plausible ways. The mice's 's eyes seem to be less relevant to the network. For humans, squeezed eyes a good indicator of pain. However, the counterfactuals only showed slight changes: sometimes the eyes blend into the surroundings. As neural networks perform well on the task using only the eyes (Neurath, 2020). believe changes architecture could preserve these details. Some other features may co-appear with image artifacts, e.g. the ear's shape changes may also appear partially blended with the background.
network
14

---

Under review as a conference paper at ICLR 2021
1L
IR
2L
2R
3L
3R
4L
4R
5L
5R
-3.00
25.0
-5.00
12.5
0.0
0.0
15.0
12.5
30.0
25.0
(a) principal Components
(b) Counterfactuals
Figure 5: Examples for the BLACK MICE dataset. (a) Random eigenvectors of the isosurface (the columns correspond to the principal components) (b) Counterfactuals generated by our model. In both subplots, the rows correspond to the indicated change in logits.
Intriguingly, the counterfactuals also show contrast changes in the surroundings, see figure ??. The authors of (Andresen were taken. Since mice after anesthesia or surgery predominantly drop the head and the nose tip points downwards, the camera angle may have been adjusted to get better view of the animal and, in effect, show more of the yellow wooden bedding material on the cage floor.
2020) voiced the suspicion that this may be explained how the photos
To verify wooden bedding material is predictive, we annotated 1000 randomly selected images from our test set. Depending on the image area covered by the wooden bedding material, we assigned each sample to the classes: (0) < 5%, (1) S 20%, (2) > 20% if the bottom of the image showed yellow bedding material. This classification resulted in 346, 258, 396 samples per bin. Of all samples, 44.7% were marked to show post-surgical/-aness
effects. Per bin, the label was unevenly distributed: 19.9%, 52.3%, 61.4%. We account for the unequal distribution of labels using partial correlation (see Appendix ??) and obtain the following values between the models' output probabilities and the bins (95% CI): (1) -0.255 (-0.31, 9520))%%, (2) 0.026 (-0.04, 0.09)s5%. (3) 0.217 (0.16, 0.27)
The label "post-surgical/ passssuurrical/aaesttttiii anesthetic effects" is unequally distributed across the three bins: 346, 258, 396. This can be problematic, when measure the correlation between sample's bin and the model logit score. The model has learned to predict lower scores for a negative label and vise versa. To account for this, we calculate the partial correlation between the model's output probability and the bin class while using the label as confounding variable. In table 2 we report both full and partial correlations and also the correlations of the bins with the label.
These results confirm a connection between the surroundings, label, and logit score. The hints of our explanations to this bias in the data were correct. The surroundings' changes can be explained probably by mice dropping their head if in pain and by changes to the camera angle. As we could also confirm many characteristic features, the network does not base decision solely on wooden bedding material. This case study highlights the practicability of our method in real-worke scenario.
15

---

Under review as a conference paper at ICLR 2021
Table 2: Bias in the BLACK MICE dataset. The post-surgicall--nesthtii effects label unevenly distributed across the bins (0-2) for the amount of yellow bedding material present in an image. The classifier's probabilties are correlated negatively with (0) fewer bedding material and positively with more (2). When we account for the effect of unequal label distribution using partial correlation, the output probabilities and bins (0 2) are still correlated.
Bin
Data
Method
Corr.
CI 95%
(0) < 5
Label
full
-0.362
-0.410, -0.310
(1) - 20
Label
full
0.090
0.030, 0.150
(2) 20
Label
full
0.271
0.210, 0.330
(0) 5 5
Pred. Prob.
full
-0.429
-0.480, -0.380
S 20
Pred. Prob.
full
0.085
0.020, 0.150
(2) > 20
Pred. Prob.
full
0.341
0.290, 0.390
(I)
Pred. Prob.
partial
-0.255
-0.310, -0.200
(2) < 20
Pred. Prob.
partial
0.026
-0.040, 0.090
(3) > 20
Pred. Prob.
partial
0.217
0.160, 0.270
C Two4Two
(a) Background
(b) Color
(c) Shape (spheres or cubes)
(d) Rotation Bending
Figure 6: Parameters in the Tw04Two dataset. The objects in the Tw04Two dataset are Sticky shown in (a) and Stretchy shown in (b). Each animal consists of a spine of four blocks and two of arms at either end. For Sticky, the right set of arms is moved inwards. For the stretchy class, both sets of arms are moved outwards. (a) background and (b) animal colors can be changed. (c) The individual blocks can be spherical, cubic or something in between This is achieved by rounding off cubes until they become spherical. (d) The animals can take a random pose.
Parameter
Corr. Data
CI95%
Corr. Change
CI 95%
Color
0.329
0.27, 0.38
0.352
0.33, 0.37
Background
0.022
-0,04, 0.08
-0,037
-0.06, -0.01
Incline
0.032
-0.03, 0.09
0.003
-0.02, 0.02
Arm Position
-0.799
-0.82, -0.78
-0.798
-0.81,
Spherical -0.053
-0.11, 0.01
-0.010
-0.03, 0.01
Abs. Rotation
0.060
-0.00, 0.12
0.136
0.11, 0.16
Table 3: Tw04Two: Correlation between object parameters and the model's output probabilities. Corr. Data: Correlation estimated on the joint distribution. Corr. Change: Correlation if only the parameter is changed and all other parameters slightly correlated with the model output when calculating correlation on the test set, it becomes correlated if we solely change the attribute and keeping all others fixed.
kept fixed. While the absolute rotation is only
D CELEBA
16

---

Under review as a conference paper at ICLR 2021
1L
1R
2L
2R
3L
3R
4L
4R
5L
5R
10.0
-10.0
5.0
0000
-5.0
S
-5.0
5.0
-10.0
-
2.
10.0
(a) Principal Components
(b) Random samples
Figure 7: Examples for the Two4Two dataset. (a) Random eigenvectors of the isosurface (the columns correspond to the principal components) (b) Counterfactuals generated by our model. In both subplots, the rows correspond to the indicated change in logits.
83L
83R
129
157
224
762
763
764
765
766
8.0
8.0
4.0
0.0
4.0
-4.0
-800
-8.0
(b)
8L
8R
95L
95R
291
608
4.0
0.0
4.0
(c)
Figure Eigenvectors of the isosurface for the "attractiveness" class. (a) Eigenvectors with the higest explained variance (b) Randomly selected eigenvectors: 82, 129, 131, 157, 224 (c) Eigenvectors with the smallest explained variance
E USER-STUDY PREGISTRATION AND HYPOTHESIS
The Study is preregisterd at httts: org provide and anonimyzed pdf version in supplemental material, Participants (N=60), of the study were required to fluent English and needed to have approval rate of least 95. Given the demanding nature of the task and
17

---

Under review as a conference paper at ICLR 2021
Ireq.
Ireq.
Beard
59.13
0.36
Attractive
81.20
49.58
No Beard
39.03
85.37
0.35
0.06
Under
31.79
20.26
-0.24
-0.07
Five Clock Shadow
28.23
9.99
-0.15
0.14
Bangs
64.65
15.57
0.11
-0.00
Big Lips
63.20
32.70
0.06
Mustache
6.61
3.87
-0.22
-0.09
Big Nose
31.03
21.20
-0.28
0..12
Sideburns
11.23
4.64
-0.21
-0.07
Makeup
80.63
0.68
Blond Hair
73.27
13.33
0.17
0.02
Wearing Lipatick
80.64
52.19
0.64
0.35
Blurry
21.39
5.06
-0.14
-0.17
Henvy Makeup
Brown Hair
68.97
17.97
0.19
0.15
Rosy Checks
90.92
7.17
0.26
0.16
Rushy Fyebrows
54.25
12.95
0.03
0.22
Arched Eychrows
79.87
28.44
0.38
0.19
Eyeglasses
3.49
6.46
-0.35
-0.29
Gray Hair
2.20
3.19
-0.25
-0.19
Bold
0.71
2.12
-0.22
-0.14
Receding Hairline
23.85
8.49
-0.22
0.19
Narrow Eyes
45.22
14.87
-0.06
-0.09
Cubby/Double
Oval Face
62.96
29.56
0.19
0.16
Chubby
8.22
5.30
-0.29
0.20
Pale Skin
80.12
4.21
0.12
0.10
SmillingMouthiCCheee
Double Chin
6.35
4.57
0.26
0.17
Straight Hair
Pointy Nose
$3.89
70.95
28.57
0.27
0.18
High Cheekbomes
63.21
48.18
0.23
0.08
Wavy Hair
70.85
36.40
0.33
0.17
Mouth Slightly Open
53.85
49.51
0.02
0.05
Wearing Earrings
70.93
20.66
-0.01
Smiling
60.54
50.03
0.19
Wearing Hat
12.99
4.20
-0.20
-0.15
Wearing Necklace
72.61
13.79
0.18
0,02
Wearing Necktie
13.65
7.01
-0.27
-0.08
Young
62.63
75.71
0.44
0.37
Table 4:
0.5
6.0
-0.
1.0
Wearing Hat
Five Clock Shadow
Gcalee
Evegresecs
Bard
Weartng Neckde
Doutle
Stoigt Hair
Suilty Eyeorows
Slacs Hair
Cheica
Arched Eyctrows
Wearing Eariagg
No Bea'd
Wary
Wearing Necklaaes
Sturo Har
3q Upp
Party Nose
Young
Sroen
Mouhh
Smiling
Wharnn tee Rev Olok Stiddo
d
Whotig larttee Duddro Ciaro-iin Chutty eyen
tage tude
Recathyy tattte
Bury Eyeer Eyetroen Poll
Pilk Bii
Don Cheece
sasted cesttone nowen Tuntey
his taard
gure sir
20 to manty hose Mannyyon rum
ureer aar oan cale corn
18p1 Oweattree
Figure 9: CelebA correlation matrix indicating the relationship among the annotated labels. The labels sorted according a hierarchical clustering on the correlation values. There strong clusters of labels (upper left and lower right), which e.g., the label Atractiveness belongs to the same cluster (lower right) as Wearing Lipstick while the label Male belongs the other cluster. This confirms the biases highlighted by our method.
18

---

Under review as a conference paper at ICLR 2021
Strongly
CF 97% 0%
3%
Complete
agree
97%
3%
0%
Agree
CF 100%
0%
0%
Somewhat
Detailed
97%
0%
3%
agree
Neither agree
Helpful
CF 100%
0%
0%
nor disagree
BL
100%
0%
0%
Somewhat
disagree
Illustrative
CF 100%
0%
0%
Disagree
BL
100%
0%
0%
Strongly
disagree
Satisfying
CF 97%
0%
3%
CF: Counter-
BL
100%
0%
0%
factual
100%
50%
0%
50%
100%
BL: Baseline
Figure 10: Explanation Satisfaction Ratings by our study participants for counterfactual interpolations (CF) and example-based explanation baseline (BL)
the complex of concepts used in the instructions they also needed have an academic degree in Computer Science, Engineering, Finance, Mathematics, Medicine, Physics or Psychology. Figure TT summarizes the subjective ratings participants gave about the two explanation technqiues used in the study.
E. EVALUATION OF PREREGISTERED HYPOTHESIS
HI: The hypothesis "Studying the system's predictions on the validation set (Baseline Explanation technique referred to Baseline) allows users verify that the neural network (NN) is using the blocks spatial arrangement (Pattern f)ror its predictions of the abstract animals." is confirmed as 83.33% least somewhat agree to corresponding statement.
H2: The hypothesis "Baseline does not allow users to detect the NN bias for colour (Pattern 2) and rotation (Pattern 3)." confirmed. Only 46.66 % of users least somewhat disagree with the statement claiming that there is a rotation pattern while only 43.33% at least somewhat agree (the remaining are undecided). For the colour pattern 50% at least somewhat disagree that there such pattern and only 43.33% at leat somewhat agree.
H3: Duplicate of H2 (copy and paste error during preregistration)
H4: The hypothesis "Studying the system's 's predictions with counterfactual interpolations as explana- tions (referred as NNWI) allows users verify that NN is using Pattern 1." is confirmed as 96.66% at least agree with the corresponding statement.
H5: The hypothesis "Counterfactual interpolations allows users to detect Pattern 2 and Pattern 3." is rejected. While 70 % at least somewhat agree with the statement about Pattern 2 only 33.33% at least somewhat agree with the statement about Pattern 3.
H6: The hypothesis "Counterfactual interpolations allows users to verify that NN is neither using the background of the image (Pattern 4) nor the surface structure of objects (Pattern 5)." is confirmed. The corresponding statement about Pattern 4 and 5 have been at least somewhat disagreed to by 83.33% and 73.33% respectively.
detect Pattern I with higher confidence" is confirmed. Agreement with the corresponding statement was significantly different between conditions (p 0.003) and average higher for Counterfactual Interpolations (2.67) compared to the baseline (1.76).
H7: The hypothesis "Counterfactual interpolations allow
H8: The hypothesis "Counterfactual interpolations allow users reject Pattern and Pattern 5 with higher confidence" rejected. There was no significant difference the certainty for disagreeing with corresponding statements.
19

---

Under review as a conference paper at ICLR 2021
H9: The hypothesis "Counterfactual interpolations allow users detect Pattern 2 and Pattern 3 with higher confidence." " is rejected since H5 was rejected. However, it is worth pointing out that for the statement about color 70% of participants at least somewhat agreed to it if they received counterfactual interpolation while only 43.33% at least somewhat agreed to it if they received example based explanations.
HI0: The hypothesis "Counterfactual interpolations leads users to be more confident in the mature- ness of the system." " rejected as agreement to the corresponding statement was significantly different across conditions. In both conditions participants where rather confident in the system.
satisfied with Counterfactual interpolations explanations." is rejected. Explanation Satisfaction rating were very high in both conditions but not significantly different.
H11: The hypothesis "Users
F
IMAGE SOURCE
As the copyright of CELEBA is unclear and includes images under no free license, we decided against showing any original CELERA images in the paper. We show the following these six images all
under permissive license: Ohama (CC BY 3.0y https: http:///mooon.iiiiiaaaa..o//////
Official portrait of Barack Obama. }pg
Commander, Eileen M. Collins (Public
Domain):
nasacomnons/
(ttccmmonn//655423335555) 65042339857
Carl Jacobi (Public httpa//r.aa////iaaaa..aa/aaaaaaa
Grace Hopper (Public domain):
Alan Touring (CC BY 4.0);
D1 1188%DD 0800000088 300.00000D00.83..............3.. . jpg
Lyndsey Scott (CC BY 4.0):
https://en.wiiee http://////akkkkk........///////
being_comma jpg
20


Reviews:
[{"review_id": "qRc354yIWoL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update after revision\n------------------------------\nI thank the authors for their work on this paper. The second reading was more pleasant. I agree with the authors that performing a user-study is an important effort, that should be encouraged. I however still believe that, if not benefitial to the user, the complexity of the method can be a drawback. I also wished that more comparisons, but especially other data modalities were investigated. I have updated my rating to reflect the improvement in the text.\n\nShort summary\n-----------------------\nThe authors propose a technique based on an invertible network to provide counterfactuals relative to one class of interest. The counterfactuals can be interpolated across an isosurface, displaying parameters which do not affect the model’s decision. The authors propose an attribution map based on those counterfactuals and evaluate counterfactuals in a qualitative manner, based on their own observations on 3 datasets, as well as based on a human-grounded evaluation on a synthetic dataset. \n\nStrengths\n---------------\nThe use of an invertible dataset is rather novel in the field of explainability, and the relationship between the obtained counterfactuals and gradient-based interpolation methods is interesting. The human-grounded evaluation is definitely a large undertaking that is not often performed to assess the usefulness of interpretability techniques.\n\nWeaknesses\n-------------------\nI have identified several weaknesses of the work that justify my recommendation:\n- the (lack of) clarity of the text.\n- the assessment of the technique, as the results of the human-grounded evaluation are mixed, with users not being significantly more accurate in finding confounding factors compared to a baseline technique.\n- the limitations of the technique, not discussed in depth. For instance, I can see difficulties in evaluating the effect of classes that are not present as “training classes” in the dataset, which requires a large labeling effort. In addition, how the technique would transpose to non-image datasets, or whether there are limitations in the invertible architectures to consider should be mentioned.\n\nNovelty\n-----------\nThe “Related works” section is rather limited, which makes it difficult to evaluate. In general, the use of invertible networks as interpretable networks is novel.\n\nClarity\n---------\nClarity was a major weakness of this work for me:\n- the datasets are illustrated in figures but not mentioned until much later\n-  the maths are described in sections that seem unrelated to each other, without depicting the relationships between the different steps\n- multiple concepts are unclear (see detailed comments)\n- the motivations are not clearly explained\n\nRigor\n--------\nI found the qualitative evaluation on the 3 datasets unconvincing, as it is unclear whether the same conclusions could not have been reached using other techniques.\nWhile I was most interested by the discussion around the generation of counterfactuals based on the invertible network compared to based on the integration of gradients, I wished there was a definition of an “ideal” counterfactual, qualitative or (preferably) quantitative. The single example provided in the main text is appealing but this requires more evidence to me.\nFinally, the “saliency” maps defined in this work do not seem to be used later on in the work. I doubt that looking at them would improve human evaluation of a model’s behavior.\n\nDetailed comments\n-----------------------------\n- Counterfactuals: their quality seems subject to appreciation and confirmation bias, especially on potentially cherry picked examples. To assess their quality, I would suggest to use the BAM dataset (Yang and Kim, 2019, https://github.com/google-research-datasets/bam) which was generated to benchmark attribution methods. I would overall suggest the use of this dataset for assessing the faithfulness (sensitivity, specificity) of the proposed approach.\n- The choice of the mice dataset should be justified as this doesn’t seem like an obvious choice to assess the quality of attribution techniques. It is quite difficult to estimate any effect, and feels like qualitative evaluation is biased by the authors’ remarks given the lack of knowledge of the problem.\n- There should be more details about the Two4Two dataset and its motivations, as well as how it relates to other datasets (e.g. Goyal et al., 2019)\n- How does the proposed approach relate to “completeness” (Sundararajan et al., 2017)?\n- What is the mathematical justification to resize the saliency map of an intermediate layer to the input resolution? Is there a citation for this process showing that this is a reasonable assumption?\n- I am confused by the section on saliency maps: what does h represent? The activations at an intermediate layer? The motivation is unclear: what are the authors trying to highlight in these “saliency maps”? Are these computed attributions or are these L1 distance between activations (in %) between x and x_tilde? Or is it a cosine distance (as suggested by the next sentence mentioning the angle?)\n- The tasks used for illustration are not described in the text. Examples of y and epsilon should be provided.\n- Is the technique limited to the model’s predicted classes?\n- How is “ideal” counterfactual described and mathematically verified?\n- The relationship between counterfactuals and e.g. integrated gradients is unclear: the first clearly needs a model that can generate data, while the latter integrates the gradients between a baseline (defined by the user) and the input. More details and explanations are required to make this relationship clearer.\n- What are the participants in the human-based study viewing? Are they comparing the counterfactuals to e.g. SmoothGrad maps, or the saliency as defined per the proposed approach?\n- It is unclear what the participants answered: Figure 5a mentions that the main score is “strongly disagree” for “arms” (both baseline and interpolation) while the text refers to “strongly agree”. Example questions would help.\n- The results of the human-grounded study are not very conclusive. Note: please correct for multiple comparisons due to multiple statistical testing of the same effect.\n- Kim et al., 2018 already displayed that human users were performing poorly at identifying a network’s decision behavior based on saliency maps. A better comparison could have relied on TCAV instead, especially as the concepts can easily be mapped to the features given the synthetic dataset. This could have made a stronger case for the use of invertible networks, especially as Goyash et al (2019) mention the use of counterfactuals based on concepts.\n- How about non-image datasets?\n\nMinor\n-------\n- Intro: I would suggest using “transparency” rather than “interpretability” when referring to logistic regression (e.g. Lipton, 2016). The interpretability of linear model weights is indeed debatable, as weights will depend on the regularization and signal-to-noise ratio in the data (Haufe et al., 2014).\n- No clear flow between the different works in the intro. No clear motivation behind counterfactuals.\n- proofreading: paper is quite hard to follow and minor changes to grammar (e.g. “Their similarity is easy to seen”) makes it more difficult to assess. The quality of the writing deteriorates in sections 3, 4 and 5.\n- It is unclear what scale delta epsilon represents, and whether we can expect the norm of the different techniques to be comparable. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "o7xomokGxe8", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. The authors propose a method to derive counterfactuals, saliency maps, and so called isofactuals using invertible neural networks. The quality of the generated explanations are compared visually with existing baselines. \n\n2. I found the structure of the paper confusing and lacking in clear elicitation of contributions. For example, once explanation methods are provided in Sec 2, the motivation of Section 3 to purely compare gradient methods is highly unstructured. In here, its unclear why changes in independent principal components while generating counterfactuals not desirable especially if as the authors suggest, the prediction changes. Even if the changes are not observable to humans. This is a highly unusual aspect of the counterfactuals where the counterfactuals shouldn't just explain large changes to logits but changes to predictions as well. \n\n3. The objective of each evaluation data-set and study should also be clearly outlined before proceeding to the details. Currently the paper leaves the reader to figure out the main contributions at the cost of hampering the paper's technical significance. \n\n4. What is the goal of the user study? Why is the baseline any other method of generating counterfactuals but merely conditioned examples? \n\nI strongly suggest restructuring the paper to fix above concerns and provide a clear justification for their experimental setup.\n\n5. I also strongly recommend the authors to move away from evaluating their models against celebA labels such as \"Attractive\" which are rife with ethical concerns. I understand that those are the only options available in celebA but I would recommend using more neutral class labels for experiments if face dataset is an important part of their evaluations.\n\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nThe authors have done a reasonable job at addressing my concerns and I have increased my score from 5 to 6. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "d8oBJqnpCFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:  The paper presents a promising idea to build interpretable models by combining discriminative and generative approach. The proposed model uses an invertible neural network to model the data distribution. The invertibility helps in transforming the learned feature vector back to the image domain. A linear discriminative classifier is trained on the feature vector to perform binary classification. Using the inverse function, the model generates a counterfactual explanation by inverting a modified logit score to create a new image as an explanation. The authors further construct an orthogonal basis using PCA, such that modifying feature vector in those directions results in no change in the classifier's prediction. Decomposing the feature space into such a basis helps discover potential biases in the dataset and the classification model. The experiments compare the proposed method's performance with fully discriminative models and post-hoc interpretability methods such as gradient-based saliency maps.\n  \nMajor \n-----------------\n\n•\tThe paper an interesting and potentially important idea. But at times, the text is difficult to read. \n•\tThe conclusion from Figure 2 is not clear. From the caption and the figure, it is not clear which attributes such as smiling; gender are important for the classifier's positive /negative attractive decision.   \n•\tIn Figure 2 and Figure 3(b), a label showing the different principal components considered in each column, and the logit/prediction of the classifier for each row will improve the figure's readability. \n•\tSaliency maps highlight important regions of an image for the prediction decision. The saliency maps in Figure 3(a) highlight almost the entire face; hence they are inconclusive. \n•\tFigure 4(a) shows the final results after integrating the original image along with different derivatives. To understand the results, it would be helpful to show some examples over which the integration took place. \n•\tAn example of a random sample, with minimal/no changes along the independent factors for the proposed method, as compared to positive changes by other methods, will help in understanding the results in Figure 4(b). \n•\tThe number reported at the end of section 3 on \"the gradient of x and the directional derivative dx/dw\" should be reported in a table to allow a proper comparison between different methods. \n•\tThe directional derivative dx/dw, in the model, is w.r.t the weight vector of the binary classifier, trained to identify the label. Related work by Kim et al. (2018), \"Interpretability beyond feature attribution: Quantitative testing with concept activation vectors\", also used directional derivatives w.r.t to binary classifiers, trained to identify a human-defined concept. A comparison with this method will help the reader understand the different applications of directional derivatives and how directional derivatives can be used without an invertible network. \n•\tTable 1 doesn't report the results for the supervised method for celebA and tow4two datasets. \n•\tThe numbers reported in Table 2 lacks a coherent conclusion. The corr. data and corr. change columns have values in similar ranges. Please elaborate and discuss the results. \n•\tIn Figure 5(b), it's not clear how real images for baseline condition are sampled. To allow proper comparison, as in counterfactual design, three images should be selected as primary images, the rest images should be sampled based on its minimum distance to the primary images but a different label.  \n•\tIn the Evaluation section, for celebA, the authors identified principal components that correspond to attributes like gender, smiling. The results are reported in a qualitative manner, with inferences draw by showing a few examples only. A quantitative analysis is required to demonstrate the dependency of the \"attractive\" attribute on other attributes. The results shown in Figure 6 of the appendix don't have a thorough caption to illustrate the findings. \n\nMinor: \n------------\n\n•\tThe caption for Figure 1 has typos.  \n•\tIn Figure 4(a), the caption doesn't describe the top label. \n•\tThe opening quotation marks are inverted throughout the text. \n•\tTable 1 and Table 2 are shown after the references. They should be placed with the main text before the references. \n•\tThe label of Figure 5(b) has a very small font.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZF17IvHJIus", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a computational method to construct ideal counterfactuals and isosurfaces via invertible CNNs, and uses it to reveal biases in three different datasets. \n\nStrengths:\n1. The use of directional derivative to construct ideal counterfactuals is interesting.\n2. Leveraging PCA to construct isosurfaces is neat.\n3. The human study is a plus, where the stimuli are based on counterfactual interpolations created by the proposed method.\n\nWeaknesses:\n1. The reviewer finds the manuscript hard to follow, especially Section II. The authors may come up with a clearer presentation.\n2. The descriptions about saliency maps are less relevant to the main idea, further confounding the reviewer.\n3. The comparison between simple gradient and direction derivative is less fair, as the directional derivative makes use of the very information direction w (e.g., the direction of no sunglass -> sunglass).  What happens if we visualize $\\phi^{-1}(\\phi(x)+ \\alpha w)$ directly, for different values of $\\alpha$.\n4. The human study may need to conduct another set of control experiments to show that only original training images (not counterfactual interpolations) are $\\textbf{less}$ helpful for uses to identify CNN patterns and biases.  The reviewer conjectures that for this simple TWO2TWO data, the subjects may spot shortcuts easily even using original training images.\n\nOther minor comments:\n1. Figure 1: There is no explanation for (a). What is $w$? The reader may not understand it for the first reading.\n2. Figure 4: The reviewer believes normalized scores on the top of the images make better sense.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5sUGratera", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a method for generating counterfactuals (visually “similar” examples with different labels) and isofactuals (visually “different” examples with the same label) using an invertible convolutional network. A human study shows that providing these counterfactual and isofactual images in a systematic way can help participants understand a model’s bias better.\n\nInterpretability of machine learning models is becoming progressively more important as these models continue to proliferate in sensitive applications such as medicine, finance, law, etc. A large body of interpretability efforts is around post hoc methods where an explanation is generated given certain probes into a blackbox function. On the other hand, the proposed model imposes certain constraints on the model to yield these explanations. The paper has some interesting ideas, and the qualitative evaluation is helpful is conveying those ideas. The discussion around gradient wrt the input image vs directional derivative is fairly insightful with convincing qualitative and quantitative results (Fig. 4). However, I did have some conerns:\n\n1. The writing often lacks clarity and the usage of space can be more judicious. For example, the description of the main method is severely lacking, and is lumped into a few short paragraphs (section 2). I needed to reread the section a few times to understand the gist since important details are either missing or relegated to the appendix. I am still unsure about the development of isosurface section. On the other hand, excessive details are present in the evaluation section, e.g. in-depth discussion of mice characteristics. Focusing on elaborating the method, and maybe one less dataset would improve readability by moving extra evaluation to appendix. Something like, “we observe similar patterns with other tested datasets, which are presented in the appendix.”\n\n2. The results of the human subject study are not very convincing. While the subjects were able to better detect model’s biases with the systematic presentation using the proposed method, they also spuriously discovered irrelevant patterns (background and blocks). The appendix (Fig. 9) also shows that the subjects thought both the proposed method and the baseline were equally good. [Digression: Fig. 9 is poorly processed with missing words, repeated legend, shuffled axes, etc.).\n\n3. Minor formatting issues: mismatched quotes throughout (striped, zebra, etc.), Fig. 4 caption (there not ideal), “different to the original”, etc.\n\nOverall, the paper has some good ideas and interesting analysis but falls short on clarity and fully convincing the reader about the results. I am marginally inclined for it to be accepted. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers agree that the paper presents an interesting idea, and the main concern raised by the reviewers was the clarity of the paper. I believe that the authors have improved the presentation of the paper after rebuttal, however, I still believe that the paper woudl require another round of reviews before being ready for publication, in order to properly assess its contributions. 
Decision: Reject