Do Users Benefit From Interpretable Vision? A User Study, Baseline, And Dataset
Authors: Leon Sixt
Martin Schuessler
Oana-Iuliana Popescu
Philipp Weiß
Tim Landgraf
Venue: ICLR 2022 Poster (2022)
Keywords: Interpretable ML
User Study
Human Subject Evaluation
Invertible Neural Networks
Convolutional Networks

Abstract: A variety of methods exist to explain image classification models. However, whether they provide any benefit to users over simply comparing various inputs and the model’s respective predictions remains unclear. We conducted a user study (N=240) to test how such a baseline explanation technique performs against concept-based and counterfactual explanations. To this end, we contribute a synthetic dataset generator capable of biasing individual attributes and quantifying their relevance to the model. In a study, we assess if participants can identify the relevant set of attributes compared to the ground-truth. Our results show that the baseline outperformed concept-based explanations. Counterfactual explanations from an invertible neural network performed similarly as the baseline. Still, they allowed users to identify some attributes more accurately. Our results highlight the importance of measuring how well users can reason about biases of a model, rather than solely relying on technical evaluations or proxy tasks. We open-source our study and dataset so it can serve as a blue-print for future studies.

Published as a conference paper at ICLR 2022
Do USERS BENEFIT FROM INTERPRETABLE VISION? A USER STUDY, BASELINE, AND DATASET
Leon Sixt"1 Martin Schuessler* 23 Oana-luliana Popescu', Philipp WeiB', Tim Landgraf
Freie Universität Berlin' Weizenbaum Institut Berlin' TU Berlin
@ixt fu- de-berlin.de, martin.schucssler bereliii.ddb Equal Contribution
ABSTRACT
A variety of methods exist to explain image classification models. However, it remains unclear whether they provide any benefit to users over simply compar- ing various inputs and the model's respective predictions. We conducted user study (N=240) to test how such a baseline explanation technique performs against concept-based and counterfactual explanations. To this end, we contribute syn- thetic dataset generator capable of biasing individual attributes and quantifying their relevance to the model. In study, assess if participants can identify the relevant set of attributes compared to the ground-truth, Our results show that the baseline outperformed concept-based explanations. Counterfactual explanations from an invertible neural network performed similarly as the baseline. Still, they allowed users to identify some attributes more accurately. Our results highlight the importance of measuring how well users can reason about biases of model, rather than solely relying on technical evaluations or proxy tasks. We open-source our study and dataset so it can serve as blue-print for future studies.
INTRODUCTION
Deep neural networks have been widely adopted in many domains. Yet, for some applications, their use may be limited by how little we understand which features are relevant. Whether engineer or user, insurance company, or regulatory body; all require reliable information about what the model has learned or why the model provides a certain output. Numerous methods have been proposed to explain deep neural networks (Gilpin et al. 2018 Molnar et al. 2020).
Ultimately, to evaluate whether such explanations are helpful, we need user studies (Doshi Velez & Kim 2017 Wortman Vaughan & Wallach 2020). In fact, some studies provided evidence that interpretable ML techniques may be helpful to find biases or spurious correlations (Ribeiro et al.) 2016b; Adebayo et al. 2020a). However, a substantial body of work shows that they may not be as helpful claimed (Kaur et al. 2020; Alqaraawi et al. 2020 Chu al. 2020 Shen & Huang 2020). Consequently, it seems that in real-world applications, biases are often found by simply inspecting the model's predictions rather than applying interpretable ML. A recent example is the Twitter image cropping algorithm: it was the users who discovered that it favored white people over people of color (Yee et al. 202T). this work, we ask: do modern interpretability methods enable users to discover biases better than by simply inspecting input/output pairs?
To investigate this question empirically, we first propose Two4Two: synthetic dataset depicting two abstract animals. Its data-generating factors can be correlated with the binary target class, thereby creating arbitrarily strong biases. We designed a baseline explanation technique for bias discovery using only the model's output: input images are arranged in a grid grouped by the model's logit predictions. This design allows a user to inspect all attributes that potentially predict the target class.
In initial user study (N=50), we validated that participants were struggling to find both biases contained in our dataset using this technique. Hence, more elaborate methods can improve over the baseline on this dataset. In the main study (N=240), we compared the baseline against two ofate-of-the-art explanations: autommtiically-diccoveee concepts and counterfactual interpolations generated with an invertible neural network.

---

Published as a conference paper at ICLR 2022
Pastyy
Siriicly
Pocky
(a) Baseline
(b) Invertible Neural Networks
Concepts
Zhang al.
2021
Figure 1: We tested whether users can identify the class-relevant features of images showing two types of animals. We biased attributes like the animal's s color to be predictive of the class and investigated whether explanation techniques enabled users to discover these biases. We tested a simple baseline (a) which shows random samples grouped by the model's output logit, counterfactual samples generated by an invertible neural network (b), and automatically discovered concepts (c). A participant viewed only one of the above conditions.
We found that none of these explanations outperformed the baseline, even though some features were identified more accurately with counterfactuals. The textual justifications of participants revealed several usability issues in all methods. This highlights the necessity to validate any claims about the benefits of explanation techniques in user studies.
This work represents substantial empirical novelty and significance in the field of interpretable ML:
The Two4Two dataset generator provides control over features and biases. It is designed specifically for human subject studies and to challenge existing interpretability approaches, Methods to quantify ground-truth feature importance when the data-generating process is known,
A study design that provides a unified approach to evaluating interpretable vision methods on the task of bias discovery. It is suitable for lay-user and includes several measures to ensure high-quality crowd-soured responses,
A strong and simple baseline explanation technique using only the model output, which propose as a benchmark for future studies,
We open-source our dataset, explanation techniques, model, study design, including instruc- tions and videos support replicating our results as well as adapting our design other explanation techniques.
2 RELATED WORK
Interpretable ML for Vision Different explanation approaches have been proposed: saliency maps
(Bach et al. 2015) Ancona et al. 2018 Sundararajan al.] 2017), example-based explanations (Cail
et al. 2019), counterfactual examples Singla et al. 2020). activation-concept approaches (Kim et al.
2018). ) or models with built-in interpretability (Chen et al. 2019 Brendel & Bethge 2018). For
detailed review about the field, we refer to (Gilpin et al. 2018 Molnar et al.) 20220).
Our work focuses on counterfactual explanations and iioomaticalyy-disoveeee concepts. Counterfac- tual explanations are samples that change the model output, e.g., flip the output class (Wachter et al.
2018 We generated counterfactuals with invertible neural networks (INNs) (Jacobsen al. 2018
Kingma & Dhariwal 2018). This approach has recently gained momentum (Hvilshoj et al. 202
Dombrowski et al.) 2021; Mackowiak et al. 2220)). Previous works have also used GANs and VAES
for counterfactual generation (Goyal et al. 2019 Mertes et al. 2020 [aueer & Geiger 2021| Singla
et al. 2020; Liu et al.) 2019 [Baumgartner et al. 20T8|[Chang al. 2019). The main advantage of
using INNs for counterfactuals is that the generative function is perfectly aligned with the forward function, as an analytic inverse exists.
Concepts represent abstract properties, which can be used to explain model. For example, the classification of an image as "zebra" could be explained by a pronounced similarity to the "stripe" concept. This similarity is determined by the dot product of the network's 's internal activations with concept vector. TCAV (Kim et al.) (2018) required manually defined concepts. Recent works proposed to discover concepts automatically (Ghorbani et al. 2019: Zhang et al. 2021.
2

---

Published as a conference paper at ICLR 2022
Peeky
Stretchy
Animal Color
Block Shape
Head peeking out
Legs stretched out
Background
Body Posture
Figure 2: The left panel depicts the main difference between Peeky and Stretchy: the legs' position. While Peeky shows one pair of legs moved inwards, Stretchy's 's legs are moved outwards. Two4Two offers different attributes: animal color, background color, the shape of the blocks and the animal's body posture. All of which can be controlled and biased separately.
User Studies for Interpretability Previous works with the task of bias discovery have mainly evaluated saliency maps and used datasets with a single, simple bias, e.g. background Adebayo
jtt al. (2020a); Ribeiro et al. (2016a) or image watermarks [Km al. (2018). User studies for concept-based methods tested only the accessibility of the explanations by asking users to assign images to a concept (Zhang et al. 202T Ghorbani et al. 2019). Counterfactual explanations have been evaluated by Mertes et al. (2020) on a forward-predictinn prediction task. We thus believe that we are the first extensively test counterfactual-
and concept- based explanations on bias discovery using challenging dataset. Recently, study exemplary- -baed explanations focused on understanding internal activations a neural network (Borowski et al. 2020). showed that for this task, examples could be more beneficial than complex feature visualizations (Olah al.. 2017). Similarly, there is evidence that participants often rely on model predictions rather than on explanations (Algaraaw)
al. 2020 Adebayo et al., (020a)).
Synthetic Datasets for Interpretable Vision Datasets with known ground-truth biases have been proposed before. BAM is artificial dataset (Yang & Kim 2019) where spurious background correlations are introduced by pasting segmented objects on different textures, e.g. dogs on bamboo forests. However, the resulting images are unsuitable for user studies as they look artificial and make it easy for participants to suspect that the background is important. Additionally, it would be difficult introduce more than one bias. limitation that also the synthetic dataset in (Chen et al. 2018) shares. The synthetic dataset created by Arras et al. (202T) created a dataset to technically evaluate saliency methods on a visual question answering task technically. Two4Two is the first dataset designed explicitly for human subject evaluations. To the best of our knowledge, we provide the first unified approach to evaluate interpretable vision on a bias-discovery task.
Tw04Two: DATASETS WITH KNOWN FEATURE IMPORTANCE
Dataset Description Datasets generated with TWO4TWO consist of two abstract animal classes, called Peeky and Stretchy. Both consist of eight blocks: four for the spine and four for the legs. For both animals, one pair of legs is always at an extended position. The other pair moves parallel to the spine inward and outward. The attribute legs' position, scalar in [0,1], controls the position. At value of 0.5, the pair of legs are at the same vertical position as the last block of the spine. Peekies have a leg position < 0.52 which means legs are moved mostly inwards to the body center. In the same fashion, Stretchies are extended outwards, legs' position 2 0.48. We added some ambiguity ensure model has an incentive to use possible biases. Therefore, Peekies and Stretchies are equally likely for legs' position between 0.48 and 0.52. It is also difficult for humans to tell if the legs are outward or inwards in this range. Besides the legs' position, the dataset has the following parameters: body posture (bending and three rotation angles), position, animal color (from red to blue), blocks' shape (from cubes spheres), and background color (from red to blue). Each can be changed arbitrarily and continuously (See Appendix Table
When designing the dataset, we wanted to ensure that (1) participants can become experts within a few minutes of training, (2) it allows for the creation of multiple biases that are difficult to find, and (3) that provides a challenge for existing interpretability methods. Goal (1) is met as participants can be instructed using only few examples (see the tutorial video in Appendix C). The high number of controllable attributes achieve Goal (2). We biased the attributes such that they do not stand out, which we validated in the first user study, Goal (3) is met by spatially overlapping attributes and long-
3

---

Published as a conference paper at ICLR 2022
1.0
1.0
1.0
Background
Shape
Color
0.5
0.5
0.5
0.0
0.0
0.0
0
0
0
Legs' Position
Legs' Position
Legs' Position
Figure 3: The joint distributions of legs' position and the attributes background (left), shape (middle), and color (right). Datapoints are yellow for Peekies and blue for Stretchies. The background is not biased. The shape is biased for legs' position lower than (0.45) or greater (0.55), but is uniform in the center. The color contains additional predictive information about the target class, as allow to discriminate between Peeky and Stretchy where the legs' position overlaps. However, for more extreme arms' positions the color is uniform and not biased.
range image dependencies. Spatially overlapping attributes, like color and shape, directly challenge saliency map explanations.
image dependencies, like the legs' positions relative to the spine, can not be explained when analyzing patches separately as done in (Chen et al.) 2019; [rendel & Bethge 2018). Both properties common in real-world datasets: For example, race and gender in facial datasets are encoded by spatially overlapping features. Long-range image dependencies are particularly relevant for pose estimation and visual reasoning (Johnson al. 2017).
Introducing Biases For our studies' dataset, we sampled the block's ' shape in a non-predictive biased fashion. This means that for legs' positions that clearly showed Peeky [O. 0.45] most blocks were rather cubic, while for legs' positions that clearly showed a Stretchy [0.55, 1] most blocks were rather round. However, for the legs' positions between [0.45, 0.55] the blocks shape was uniformly distributed. In particular, in the even narrower interval [0.48, 0.52] where classifier can only be as good as random guessing, the block's shape does not provide any additional information about the target class. In Figure 3
we show the joint distribution of the block's 's shape and legs' position.
We sampled the animals' color to be predictive for the target class. At the small interval where the legs overlap [0.48; 0.52], we distributed the animal color to provide additional class information. Stretchies were more likely to be red, and Peekies were more likely to be blue. Outside of this centered interval, the color gradually became uniformly distributed (see Figure 3). Hence, color was more equally distributed than the shape, making the color bias harder to detect visually. The remaining attributes, background color and body posture, were sampled independently of the class, and we expected our model to ignore them.
Measuring Ground-Tie Truth Feature Importance Even if dataset contains biases, it is unclear how relevant they will be to a neural network after training. Feature importance also depends on the network architecture, the optimization process, and even the weight initialization. As Two4Two allows us to change any parameter in isolation, we can directly compare the model prediction between two images that differ in only one parameter. For these two images, we measured both the median absolute logit change and also for how many samples the predicted class was flipped. Both measures quantify how influential each parameter is (see Table Tp. As expected, the legs' position had a strong influence on the prediction. The model relied more on animal color than on the blocks' shape, which expected as the color contains additional information about the class. Surprisingly. the prediction flip for unrelated attributes such as background was only slightly lower than for blocks' shape.
To analyze this further, we calculated linear fit for each parameter change to the logit change. We reported the coefficient of determination R², which indicates how much of the variance in the prediction can be explained linearly by the analyzed property. While the unrelated propertics sometimes flip prediction, the direction of that flip is random (R²² x 0). In contrast, the biased parameters influence predictions in a directed fashion, with animal color (R²=0.751) being clearly more directed than blocks' shape (R²=0.307)
4 MODEL AND EVALUATED METHODS
As discussed in section 3
Two4Two was designed to challenge existing interpretabilityy methods, e.g., saliency map explanations and patch- models. We selected two methods that might provide
4

---

Published as a conference paper at ICLR 2022
Table I: Importance of the data generating factors to the model's prediction. Prediction Flip quantifies how often the model's prediction changes the sign when changing the attribute. The Mean Logit Change reports the median of the absolute change in logit values. The R score is calculated on an ordinary least squares from the changes of each factor to the changes in the model's logit. For more attributes, see Appendix Table 5
Factor
Prediction Flip [%]]
Median Logit Change
R²
Legs' Position
41.680
2,493
0.933
Color
7,080
0.886
0.751
Shape
3,920
0.577
0.307
Background
2.640
0,523
0.006
Rotation Yaw
3,480
0,669
0.001
Bending
3.640
0.605
0,000
the user with the necessary information: counterfactuals generated with an invertible neural network (INN) and concept-based explanations (Zhang et al. (021))
INN Counterfactuals We trained an INN using both a supervised and an unsupervised objective (Dinh et al., 2016] 2015). To predict the target class, the model first applies the forward function yp to map data point æ to a feature vector z = =(x)). Then, a linear classifier takes those features z and predicts the logit score f(x) = w z b. Any input can be reconstructed from the feature vector by applying the inverse function æ = +p- (z). The model has a test accuracy of 96.7%. Further details can be found in Appendix A.2 The baseline and concept techniques are also applied this model.
To create counterfactual example 전 for a data point we can exploit the linearity of the classifier. Moving along the weight vector w. i.c., adding w to the features 2, changes the model's prediction. By controlling the step size with scalar a, we can directly quantify the change in the logit value Ay = aw" resulting in counterfactual ã = (z + aw) which visualizes the changes introduced by step ow z-space. The INN's explanations are visualized in a grid where each row shows a single counterfactual interpolation (see Figure lbp.
TuTT w. The modified feature vector + cae can be inverted back to the input domain,
Automatically-diseoue diseeveee Concepts We adapted the NMF approach of Zhang et al. (021 to our specific network architecture. Because the network's 's internal representations also contain negative values, we used matrix factorization instead of NMF. We generated the concepts using layer 342 (from total of 641 layers). The layer has feature map resolution of 8x8. This choice represents a trade-off between enough spatial resolution and high-level information. We ran the matrix factorization with 10 components and selected the five components that correlated most with the logit score (r is in the range [0.21, 0.34)).
Our presentation of concept-based explanations was very similar to (Zhang et al. 2021 we visualized concepts with five exemplary images per row and highlighted regions corresponding to concept. Since our classifier is binary, negative contribution for Stretchy actually means positive contribution for Peeky. Hence, we could have characterized a concept as more Peeky and more Stretchy, to make the design similar to the other two explanation techniques. However, as the concepts did not strongly correlate with the model's output, presenting them as class-related could confuse participants: more Peeky column would have contained some images showing Stretchies and vice versa. Thus, we presented them separately in two consecutive rows (See Figure [c]. Presenting concepts in this fashion gives them fair chance in the study because participants rated the relevance of each attribute for the model rather than for each class separately.
HUMAN SUBJECT STUDY
We share the view of Doshi- & Kim (0177) and Wortman Vaughan & Wallach| (2020) that user-testing of explanation techniques is crucial but challenging endeavor. As our second main contribution, we propose and conduct a user study based on the Two4Two dataset which can act as a blue-print for future investigations. Our design has been iterated in over ten pilot studies and proposes solutions to common problems that arise when evaluating explanation techniques on crowd- -ourrinn platforms with lay participants.
5

---

Published as a conference paper at ICLR 2022
5.1 DESIGN CONSIDERATIONS
Data without Prior Domain Knowlege We specifically designed the Two4Two dataset to avoid overburdening participants, as might be the case with other types of data. Within a few minutes, participants can easily become domain experts. Since the data is unknown to them prior the study, we avoid introducing any prior domain knowledge as a confounding factor, which can be an issue (Alqaraawi et al. 2020).
Manageable but not Oversimplified Tasks We propose the task of bias-discovery: participants had to rate features as either relevant or irrelevant to a model. The task directly reflects users' perception of feature importance. Furthermore, bias- bias-discovery has the advantage of being suitable for lay participants. At the same time, it is also grounded in the model's behavior. This is an advantage over tasks used in several previous studies, which only evaluated whether explanations were accessible to users, e.g. by identifying the target property smiling using image interpolations (Singla et al.] 2020) assigning images to a concept class (Zhang et al. 2021 Ghorbani et al.) 2019). However, these tasks are an oversimplification and cannot measure any insights the users gained about the model. In contrast, (Alaaraaww et al. (020) employed the task of forward prediction of neural network. This requires substantial model understanding and is very challenging, as reflected by the participants' low accuracy. Assessing trust in a human-in-tte--ee setting, has the disadvantage that trust is influenced by many factors which are difficult to control for (Lee & Seel a model is fit for deployment (Ribeiro et al. 2016b; Adebayo et al. (2020b). However, in our own pilots studies, users deemed a model fit for deployment even if they knew was biased.
task, despite its realistic
et al. (017). Another approach is to asks participants assess whether
Baseline Explanation Technique To quantify whether an explanation is beneficial for users, it must be compared to an alternative explanation. In this work, we argue that a very simple and reasonable alternative for users is to inspect the model's ' logits assigned to set of input images. We designed such a baseline explanation as shown in Figure la After several design iterations, we settled for a visually dense image grid with 5 columns sorted by the logit score, each column covering 20% of the logit values. The columns were labeled very certain for Pecky/Stretchy, certain for Peeky/Stretchy,, and as unsure. Pilot studies showed that participants' attention is limited. We thus decided to display a total of 50 images, i.e. an image grid of 10 rows. The number of images was held constant between explanation techniques ensure the same amount of visual information and fair comparison. In this work, we focused on binary classifications. For a multi-class setting, one could adapt the baseline by contrasting one class verses another class.
High Response Quality We took extensive measures to ensure participants understood their task and the explanation techniques. Participants were required to watch three moofesiionally-spoeer tutorial videos, each under four minutes long. The videos explained, on a high level, the Two4Two dataset, machine learning and how to use an assigned explanation technique to discover relevant features. To avoid influencing participants, we prototyped idealized explanations using images from Two4Two. The explanations showed different biases than those in the study. Each video was followed by a written summary and set of multiple choice comprehension questions After failing such a test once, participants could study the video and summary again. When failing a test for a second time, participants were excluded from the study. We also excluded participants if their written answers reflected a serious misunderstanding of the task, indicated by very short answers copied for all attributes reasoning that is very different from the tutorial. We recruited participants from Prolific who are fluent in English, hold an academic degree and have an approval rate of 2 90%. To ensure they are also motivated, we compensated them with an average hourly pay of 111.45 which included a bonus of E0.40 per correct answer.
5.2 EXPERIMENTAL DESIGN
We conducted two online user studies. Before starting the data collection, we formulated our hypotheses, chose appropriate statistical tests, and pre-registered our studies (see Appendix DD This way, we follow the gold-standard of defining the statistical analysis before the data collection, thus ensuring that our statistical results are reliable (Cockburn et al. 2018). The first study (N=50) analyzed whether the task the baseline. We tested if at least one bias in our model (either the animal's 's color or the blocks' shape) was difficult to find using the baseline technique. Consequently, we used within-subject design.
challenging enough that other methods could potentially improve over

---

Published as a conference paper at ICLR 2022
Table 2: The mean accuracy for each attribute by condition. Ncalented provide the number of participants collected and Nittered the number of remaining participants after the filterin. Stars mark statistical significance.
Condition
N Collected
Natered
Overall
Legs
Color
Backgr.
Shape
Posture
Study (Baseline)
50
43
73.4
86.0
48.8
86.0
74.4
72.1
Study
240
192
67.0
78.2
58.9
66.8
73.1
59.1
INN
80
62
84.5
++00000.0
$82.3
#79.0
90.3
71.0
Baseline
80
71
80.8
85.9
59.2
95.8
93.0
70.4
Concepts
80
59
32.2
45.8
32.2
18.6
32.2
32.2
In the second study (N=240), we evaluated the two explanation techniques described in Section 4 against the baseline using a between-subjects design. Participants were randomly, but equally assigned to one of the explanation techniques. We specified two directed hypotheses. We expected participants in the INN condition to perform better than those in baseline, because the baseline does not clearly highlight relevant features, whereas interpolations highlight features in isolation. We expected participants viewing concepts perform worse than those in the baseline, due to their inability to highlight spatially overlapping features.
For both studies, participants completed tutorial phase first. Using their assigned explanations, they then assessed the relevance five attributes: legs' position relative to the spine, animal color, background, rotation bending, and blocks' shape. The questions were formulated as: "How relevant is <attribute> for the system?", and participants had to choose between irrelevant or relevant. The percentage of correct answers (aceuracy) served as our primary metric. Participants also had to write short, fully- sentenee justification for their answers. For links to the study, see Appendix
BASE
CON
INN
5.3 RESULTS
Proportin rroportion
0.4
Data Exclusions As stated in the preregistration, we auto-
0.3
matically excluded all participants who withdrew their consent, failed one of the comprehension questions twice, skipped
0.2
0.1
video, or exceeded Prolific's s time limit for completion. If participant was excluded, new participant's 'ss place was made available until the pre-registered number of completed responses was reached. We excluded 63 study respondents for the first study, and 145 for the second study in this fashion. We ensured that all participants were naive about the dataset. Once they participated in study, they were blacklisted for future studies.
0.0
1
2
5
#C ofrrect
Figure 4: The proportion of correct answers for bascline (BASE), con- cepts (CON), and INN
For completed studies, two annotators independently marked the participants' written answers and excluded those with copy and paste answers or indications of grave misunderstandings of the instructions. Participants
labeled include, unsure, or exclude. Both anotators had agreement of π 0.545 for the first study and к = 0.643 for the second (measured include vs. unsure and exclude). Disagreements were solved by discussion. total, we excluded participants from the first study (14%) and 48 participants from the second study (20%).
used two-sided exact McNemar tests on their answers about the relevance of the legs position compared with animal color (first test) and back- ground (second test). Participants found the color bias less often than the legs' positions (P <0.0001) The success rate for the color attribute was 49% vs. 86% for legs. The shape bias was not sig- nificantly harder find than the legs' positions and was identified correctly with 74% accuracy (P=0.3036). Hence, we confirmed our hypothesis and concluded that other methods still have room for improvement over the baseline.
First Study For the accepted 43 participants,
Second Study In the second study, we evaluated 192 valid participant responses (62 INN, 71 BASE, 59 CON). We expected data to be different from the normal distribution, and a Shapiro- -ilk test for all conditions confirmed this (P 0.001). We depict the number of correct answers per condition in Figure 4 A Kruskal- Wallis test showed a significant differences in accuracy scores between conditions (P 0.001). For focused comparisons, used two Wilcoxon--rak--um
7

---

Published as a conference paper at ICLR 2022
tests with Bonferroni correction to correct for multiple comparisons. The accuracy scores differed significantly between the baseline and concept conditions (P (.000 r=0.778. The performance of participants using concepts rather poor, with only 31.7% accuracy, considering that random answers would yield a score of 50%. For concepts, not a single attribute surpassed the 50% barrier. We found no significant difference when comparing the baseline and counterfactuals (P-0.441, r=0.091). Their mean accuracies are close, with 80.8% for baseline and 84.5% for counterfactuals, INN counterfactuals helped users to discover the main attribute, legs' position, (P <0.001) and color bias (P=0.033) more reliably []] However, counterfactuals performed significantly worse for the background attribute (P=0.033), while for blocks' shape and position we found no significant difference (for both, P=1).
To understand how participants integrated the explanation techniques into their reasoning, we analyzed the textual answers of each feature qualitatively. Two annotators first applied open coding to the answers. They performed another pass of closed coding after agreeing on a subset of the relevant codes, on which the following analysis is based. Overall, the participants perceived the task as challenging, as they expressed being unsure about their answers (N=71).
Qualitative Results
We designed our image grid to show both possible classes and provide information about the model's certainty. We found that many participants integrated this additional source of information into their reasoning. This was especially prevalent in the baseline condition (N=51). Participants particularly focused on the columns 'very certain Peeky' and 'very certain Stretchy', as well as on the column 'unsure'. While this may have helped confirm or reject their hypotheses, it sometimes led to confusion; for example, when an image that exhibited a pronounced leg position, and therefore could easily be identified as Peeky or Stretchy, was classified by the model as 'unsure' (N=14).
Across conditions, we also observed that participants expect that all images needed to support a hypothesis. "The animals are in different colors, there are blue stretchy and also blue peeky animals, If the color was relevant peekylstretchy would be in one color etc" (P73, BASE), Across conditions, most participants that applied such deterministic reasoning failed to find the color bias. In contrast, other participants applied more probabilistic reasoning, which helped them deal with such contradictions: "Peeky is more likely to be blue in colour, whereas Stretchy is more likely to be pink. This is not always true (e.g. the shapes can be white in colour at either ends of the spectrum) but it might be somewhat relevant to help the system decide" (P197, INN).
Another observed strategy of participants was to reference how often they saw evidence for the relevance of a feature (N=35), which was very prevalent in the concepts condition (N=20). Especially concepts were rather difficult for participants to interpret. A common issue was that they expected relevant feature to be highlighted completely and consistently (N=38). Several instances show that participants struggled to interpret how highlighted region can explain the relevance of feature, "If this [the legs position] I were relevant I would have expected the system to highlight only the portion of the image that contains the legs and spine. (e.g. only the legs and one block of the spine at each end). Instead, every image had minimally the entire animal highlighted" (P82, CON). Furthermore, spatially overlaping features were another cause of confusion: "there are rows in which the animal highlighted but not the background so it could be because of color, shape or rotation" (P157, CON)
Participants erred more often for the background in the INN condition than for the baseline. We conducted an analysis to investigate this issue. We found that 29 participants stated that they perceived no changes in the background of the counterfactuals and hence considered this feature irrelevant. Another 21 participants noted that they saw such change, which let 12 of them to believe its a relevant feature. "The buckground color changes in every case, it's also a little subtle but does" (P205). Another 9 participants decided that the changes were too subtle to be relevant. "The background colour does not change an awful lot along each row, maybe in a couple of rows it changes slightly but I do not feel the change is significant enough that this is relevant factor in the machine decision" " (P184).
Do Counterfactuals Highlight Irrelevant Features? Indeed, subtle perceptual changes in back- ground color are present (Figure Ib). To quantify these changes, we decided to use an objective observer: a convolutional neural network. We trained a MobileNetV2 (Sandler et al. 2018) predict
'The statistical analysis of the attributes for INN vs. baseline was not pre-registered. The reported p-values for the attributes were corrected for eight tests (including the pre-registered tests) using the Bolfe-Bonferroni method.
8

---

Published as a conference paper at ICLR 2022
Legs' Position
Background
Color
Shape
Logit
Logit
Logit
Logit
Figure 5: Attribute changes along counterfactual interpolations as measured by an observer convnet.
Each line corresponds to single sample whose logit score is modified through linear interpolations in the classifier space.
the parameter values of individual attributes of an image (e.g.. background color, object color, etc.) using completely unbiased version of Two4Two. After training, the model could predict the parameter values almost exactly (MSE < 0.0022, see Table 7). We then used this model to evaluate the parameter values of counterfactual INN interpolations. each spanning 99% of the logit distribution. We visualize the predictions of NobileNetV2 in Figure []] All predictive properties (legs' position, body color, blocks' shape) are changed by the counterfactuals consistently. For the background, the changes are subtle but present. We also quantified the change in parameters using the difference between the maximum and minimum predicted value per individual interpolation (See Table This supports the finding that relevant attributes change the most legs" position: 0.662; shapes: 0.624; color: 0.440. The background changes less with 0.045, which seems enough to give some participants a false impression about its relevance.
6 CONCLUSION
Contributions We contribute dataset with full control over the biases it contains and methods to verify the feature importances of a given model. The dataset was specifically designed for user studies and we contribute carefully crafted study design interpretable vision. Our design includes a simple, yet powerful baseline technique that relies on the model's outputs only. While interpretability methods had room to improve over the baseline, we showed that two state-of-the-art methods did not perform significantly better. Our results emphasize that any explanation technique needs to be evaluated in extensive user studies.
template for future empirical evaluations of
Limitations Due to budget constraints, we limited the number of factors in our experimental design (external vs. internal validity trade-off). Our study introduced predictive bias for the animal's color and non-predictive bias for the blocks' shape. It remains unclear how our results may have changed for a different dataset configuration: certain biases could exhibit different visual saliency. remains also left for future work to determine which visual interface design is optimal for given method. Furthermore, our study design restricted participants to make binary choices and provide textual justifications limiting our understanding of the participants issues.
Take-Aways We were surprised by the performance of the two tested techniques. Users had problems interpreting the nutomaticalll--iscoveee attributes. As we expected, explaining spatially overlapping features by highlighting important regions limited the concepts' expressiveness. On the other hand, INN counterfactuals also did not perform significantly better than the baseline. Still, counterfactuals were more helpful to discover the strongest bias in the model. However, some participants rated the relevance of the background incorrectly, as slight changes in the interpolations were still salient enough. It is therefore important for future work to develop counterfactuals that alter only relevant attributes.
concepts and could not identify the relevant
We presented user study on a synthetic dataset. We believe that the results also have implications for natural image data. When we created Two4Two, our objective was to translate challenges faced on "real" computer vision data (like spatially overlapping features) into an abstract domain. Although some properties of photorealistic datasets are lost in this abstraction, a method performing poorly on Two4Two would likely not perform well on natural dataset with spatially overlapping features.
The user study was a reality check for two interpretability methods. Such studies can guide technical innovations by identifying areas where users still struggle with current explanation methods. They are laborious and expensive, but at the same time, they are crucial for future interpretability research. Future work could focus creating more realistic, controllable dataset, e.g. using augmented reality (Alhaija et al. 20118). By open-sourcing our videos, study, and code we encourage the community to take on the challenge to beat the simple baseline.
Outlook

---

Published as a conference paper at ICLR 2022
7 ACKNOWLEDGMENTS
We thank Jonas Köhler and Benjamin Wild for their feedback on the manuscript. We also thank our reviewers for their time. LS was supported by the -eeeemnn-Nemann Scholarship of the state of Berlin. MS and PW were funded by the German Federal Ministry of Education and Research (BMBF) NR 16DII113. In addition, the work was partially funded by the the German Federal Ministry of Education and Research (BMBF), under grant number 16DHB4018. 8. We thank the Center for Information Services and High Performance Computing (ZIH) at Dresden University of Technology and the HPC Service of ZEDAT, Freie Universität Berlin, for generous allocations of computation time (Bennett et al. 2020).
8 ETHICS STATEMENT
In our human subject evaluation, it was important for to pay crowd workers fl 1.4.45 per hour which above UK minimum wage of f8.91. The crowd workers consented to the use of their answers in the study. No personal-data user studies through an ethics review board.
collected. Our organization does not require approval of online
REPRODUCIBILITY STATEMENT
Our work brings some additional challenges to reproducibility beyond compared to other machine learning research. The results the human subject study depend on the dataset, model and even the presentations in the video tutorials. We decided to tackle this challenge by first open-sourcing dataset, model, videos, code, and even the study itself. For now, we share the links to the study and videos in the Appendix. We will share machine-readable export of the study, the source-code, and the model with our reviewers via OpenReview once the discussion period start and will make them public a later point in time.
In total, we spent around 5000 GBP including prestudies, software licenses, and our two studies in the paper. We estimate the costs of reproducing the studies in our work around 2500 GBP excluding software licenses. For a study comparing only one condition against the baseline, we would estimate the costs be around 1400 GBP.
REFERENCES
J. Adebayo. Michael Muelly, I. Liccardi, and Been Kim. Debugging tests for model explanations. ArXiv, abs/201 1.05429, 2020a.
Julius Adebayo, Michael Muelly, laria Liccardi, and Been Kim. Debugging tests for model explana- tions, 2020b.
Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars M. Mescheder, Andreas Geiger, and Carsten Rother. Augmented reality meets computer vision: Efficient data generation for urban driving scenes. International Journal of Computer Vision, 126:961-972, 2018.
Ahmed Alqaraawi, Martin Schuessler, Philipp WeiB, Enrico Costanza, and Nadia Berthouze. Evaluat- ing saliency map explanations for convolutional neural networks: A user study. In Proceedings of the 25th International Conference on Intelligent User Interfaces, IUI '20 pp. 263-274, New York, NY, USA, 2020. Association for Computing Machinery. doi: 10.1 10.1145/377/225.3377511
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient- attribution methods for deep neural networks. In International Conference on Learn- ing Representations, 2018. htt http::///opeeeeeeeeeee
Leila Arras, Ahmed Osman, and Wojciech Samek. Clevr-xai: A benchmark dataset for the ground
truth evaluation of neural network explanations. Information Fusion, 2021. ISSN 1566-2535. doi: https://do.org//0..01166/ 202.: URL https:
/ttstttss:////////t.t wwwp///wwwwwcccoo// httt::::wwww.scceoeeieeee..... com/
sci catiannee/rrttaee//////662333333
10

---

Published as a conference paper at ICLR 2022
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Roburt Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10(7), 2015. doi: 10.137 1/jo7m//l/ www.///oa.........00........ 0130140..........00
Christian Baumgartner, Lisa Koch, Kerem Can Tezean, Jia Xi Ang, and Ender Konukoglu. Visual feature attribution using wasserstein gans. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 8309-8319, 2018.
Loris Bennett, Bernd Melchers, and Boris Proppe. Curta: A seneral-purpon high-performance computer at zedat, freie universität berlin. http://dx..oii.org/// 19/77//69/// 2020.
Judy Borowski, Roland S. Zimmermann, Judith Schepers, Robert Geirhos, Thomas S. A. Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain cnn activations better than feature visualizations, 2020.
Wieland Brendel and Matthias Bethge. Approximating cnns with bag-of-local-features models works surprisingly well on imagenet. In International Conference on Learning Representations, 2018.
Carrie Cai, Jonas Jongejan, and Jess Holbrook. The effects example- based explanations in a machine learning interface. In Proceedings of the 24th International Conference Intelligent User Interfaces. IUI 19, pp. 258-262, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 45036666 doi: 10.1145/3302255330022333 URL https:///dot .rg/
T0.1145/3301275..33
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and D. Duvenaud. Explaining image classifiers by counterfactual generation. In ICLR, 2019.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, pp. 8930-894 8941, 2019.
Yuxin Chen, Oisin Mac Aodha, Shihan Su, Pietro Perona, and Yisong Yue. Near-optimal machine teaching via explanatory teaching sets. In Amos Storkey and Fernando Perez-Cruz (eds.), ), Pro- ceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pp. 1970-1978. PMLR, 09-11 Apr 2018. URL httth:::/prpceeeeeeeeot www.oeedddiig.......ooo ngs mll..mmr...
www..ng......ss........... prw.s/ / 8g. html
Eric Chu, Deb Roy, and Jacob Andreas. Are visual explanations useful? a case study model-in-the- loop prediction, 2020.
Andy Cockburn, Carl Gutwin, and Alan Dix. HARK No More: On the Preregistration of CHI Experiments, pp. 1-12. Association for Computing Machinery, New York, NY, USA, 2018. ISBN 97814503366066. URL /ttpps://ti.....cc.... org 10. 3115/11111411.1111111 3173715
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. CoRR, abs/1410.8516. 2015.
Laurent Dinh, Jascha Sohk-Dii,,,een, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1rXiv:1605.08803 2016.
Ann- Mathran Dombrowski, Jan E Gerken, and Pan Kessel. Diffeomorphic explanations with normal- izing flows. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021. URL https: pettps://openreveee
hwt.::eet//oottteee id- ZBR9EBER 666
Finale Doshi-Veles and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv: 1702.08608, 2017.
Amirata Ghorbani, James Wexler, James Y Zou. and Been Kim. Towards automatic concept-based explanations. In Advances in Neural Information Processing Systems, pp. 9277-9286. 2019.
Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pp. 80-89. IEBE, 2018.

---

Published as a conference paper at ICLR 2022
Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace). arXiv preprint 1rXiv:100..07165, 2019.
Frederik Hvilshøj, Alexandros Iosifidis, and Ira Assent. Ecinn: Efficient counterfactuals from invertible neural networks, 2021.
Jöm-Henrik Jacobsen, Aroll W.M. Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. In Interaational Conference on Learning Representations, 2018. URL https:// openreview net forum? d=HJs jkMbb0
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Vaughan Jen- nifer Wortman. Interpreting interpretability: Understanding data scientists' use of interpretabil- ity tools for machine learning. In Proceedings of the 2020 CHI Conference on Human Fac- tors Computing Systems, CHI '20, pp. 1-14, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450367080.
doi: 001144//3133311.3762299 URL
0eeeg/...115/33553 1145/3313831. 3376211
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference on Machine Learning, pp. 2673-2682, 2018.
Diederik Kingma and Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. ArXiv, abs/1 0/0//807.000333 2018.
John D. Lee and Katrina A. See. Trust in Automation: Designing for Appropriate Reliance. Human Factors, 46(1):50-80, March 2004. ISSN 0018-7208. doi: 10.151 8/hfes.f6.6... 1.5...0.30392.
Shusen Liu, Bhavya Kailkhura, Donald Loveland, and Yong Han. Generative counterfactual intro- spection for explainable deep learning, 2019.
Radek Mackowiak, Lynton Ardizzone, Ulirich Köthe, and Carsten Rother, Generative classifiers as basis for trustworthy computer vision. arXiv preprint urXiv:2007.15036 2020.
Silvan Mertes, Tobias Huber, Katharina Weitz, Alexander Heimerl, and Elisabeth André. Ganterfac- tual counterfactual explanations for medical non-experts using generative adversarial learning, 2020.
Christoph Molnar, Giuseppe Casalicchio, and Bernd Bischl. Interpretable Machine Learning AA Brief History, State-of-the-Ar Art and Challenges. In ECML PKDD 2020 Workshops, Communications in Computer and Information Science, pp. 417-431, Cham, 2020. Springer International Publishing. ISBN 978-3-030-65965-3. doi: 10.1007/978- 11110009977--000--5573 3-030-65965-3 28.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11):e7, 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016at 6a.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier, 2016b.
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chin Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. 2018 IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. pp. 4510-4520, 2018.
Axel Sauer and Andreas Geiger. Counterfactual generative networks, 2021.
12

---

Published as a conference paper at ICLR 2022
Hua Shen and Ting-Hou Hao Kenneth Huang. How Useful Are the Geeeerteeeeeetted Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labels. In Proceedings of the Eighth AAAI Conference on Human Computation and Crowdsourcing (HCOMP. 20), volume 8, pp. 168-172, Virtual, October 2020. AAAI Press. ISBN 978-1-57735-8488
Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich. Explanation by progressive exaggeration. In International Conference on Learning Representations, 2020.
Aaron Springer, Victoria Hollis, and Steve Whittaker. Dice the Black Box: User Experiences with an Inscrutable Algorithm. AAAI Spring Symposium Series, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Ven Volume 70, pp. 2319-3328. JMLR.org. 2017.
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harvard Journal of Law & Technology, 31)2), 2018.
Jennifer Wortman Vaughan and Hanna Wallach. human-centered agenda for intelligible machine learning. This is draft version of chapter in book to be published in the 2020 21 timeframe., November 2020. wttpttttstt//....ttttittttttteet /www http::www............ com en us/ heta-uusreeeeaaceeee
learning//
Mengjiao Yang and Been Kim. Benchmarking attribution methods with relative feature importance, 2019.
Kyra Yee, Tantipongpipar, and Shubhanshu Mishra. Image cropping on twitter: Fairness metries, their limitations, and the importance of representation, design, and agency. ArXiv, abs/2105.088677, 2021.
Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A. Ehinger, and Benjamin P. Rubinstein. Invertible concept-based explanations for models with non-negative concept activation vectors, 2021.
13

---

Published as a conference paper at ICLR 2022
Table 3: Importance of the data generating factors the model's prediction. For the R score, we fitted ordinary least squares from the factors' deltas to the deltas of the model's logits and then report the coefficient of determination (IR²). The Mean Logit Change reports the median of the absolute change in logit values. The Prediction Flip column quantifies how often the model's prediction changed the sign when changing the attribute.
Factor
Prediciton Flip [%]]
Median Logit Change
R²
Legs' Position
41.680
2.493
0.933
Color
7.080
0.886
0.751
Shape
3.920
0.577
0.307
Position
2.960
0.597
0.007
Background
2.640
0.523
0.006
Rotation Yaw
3,480
0.669
0.001
Rotation Roll
2.260
0,413
0.001
Bending
3.640
0.605
0.000
Rotation Pitch
3.500
0.627
0.000
Position
3.380
0.581
0.000
Table 4: Norm quantifies the length of feature map changes (Ah f(x) - f()) after resampling the different data generative factors. Angle w. Clas. quantifies the mean angle (in degrees) between Ah and the classifier weight w.
Factor
Norm
Norm Std.
Angle w. Clas.
Angle Std.
Legs' Position
70.8
2.6
79.9
1.9
Color
53.8
2.4
85.3
1.3
Shape
66.9
2.1
88.5
1.5
Position Y
68.1
2.8
89.7
1.7
Background
60.3
2.6
89.7
1.5
Bending
68.9
2.6
89.8
1.7
Rotation Yaw
68.5
2.7
89.8
1.7
Rotation Roll
55.2
2.9
89.9
1.2
Position X
68.4
3.0
89.9
1.6
Rotation Pitch
69.5
2.5
90.0
1.5
A APPENDIX: TECHNICAL DETAILS
A.I TWo4Two DATASET DETAILS
Factor
Range
Distribution
Biased
Additional Class Information
Legs' Position
[0, 1]
Uniform with overlap
Yes
Color
[0. 1]
See Figure 3
Yes
Yes
Shape
[0.
See Figure
Yes
No
Position Y
[-0.8, 0]
Uniform
No
No
Position X
[-0.8,
Uniform
No
No
Background
[0.05, 0.95
Uniform
No
Rotation Yaw
2m
Uniform
No
No
Rotation Roll
n T/4
Truncated Normal(O, 0.03m/4)
No
No
Rotation Pitch
6/6, m6
Truncated Normal(O, /
No
No
Bending
1/10,
Truncated Normal(O, 20)
No
Table 5: Distribution of each attribute in the study's 's dataset. Biased denotes whether attribute is unequally distributed for the two classes. Additional Class Information show an attribute contains any additional information about the target class not already given by the legs' position.
14

---

Published as a conference paper at ICLR 2022
Table Two4Two: Effect of interpolating along the weight vector.
Attribute
Mean Maximal Change
Std.
Legs' Position
0.662
0.140
Color
0,440
0.190
Shapes
0.624
0,208
Bending
0.059
0.042
Background
0,045
0.044
Rotation Pitch
0.186
0.126
Rotation Yaw
0.102
0.182
Rotation Roll
0,003
0.001
Position X
0,105
0.078
Position
0.103
0.078
Position
Background
Color
Shape
0
Logit
Logit
Logit
Logit
(a)
(b)
(c)
(d)
Bending
0.25
Rotation Yaw
Position X
Position
0.00
-2
0..0
-025
-075
Logit
Logit
Logit
Logit
(e)
(f)
(g)
Figure 6: All attribute values as predicted by observer convnet for sequences of counterfactual interpolations. Each line corresponds to single sample whose logit score modified through linear interpolations in classifier space.
15

---

Published as a conference paper at ICLR 2022
import cataclasses import numpy as up import httpletlib.pyple as plt
from txeedwe.blender import render
from two4two.bias import Sampler, Centinouos
From
import SceneParameters
edataclasses dataclass
10
class RotaticnBiassampler (Sampler):
11
"*"A rotation-biasen sampler.
12
The rotation is sampled conditicnally dependi ng the object type. Positive rotations for peaky and negative rotations for stretchy.
15
16
obj roj_retationn yaw: Continoues = detaclasses. field fefault_fattoryllambbe 'peaky': np. random. uniper 4, 0) 'stretchy's np, random. uniform (0, np.pi / 4)
19
20
21
}
23
sample a images
24
sampler Retation RotationBiessamlerr
25
params sampller. sample () for in range (4)
26
for img, mask, param in render (parans)
plt. ww..imshew(ingg (img)
28
plt.title (f"
29
plt.
Listing 1: Source code example create biased sampler. High positive rotations are predictive of Stretchy and low negative rotations of Peaky.
A.2 ARCHITECTUREE THE INVERTIBLE NEURAL NETWORK
Our model is based on the Glow architecture Kingma & Dhariwal 2018) and contains 7 blocks. A block is collection of 32 flow steps, followed by down-sampling layer, and ends with fade-out layer. A single flow step consists of actnorm, invertible x convolution and affine coupling layer. The down-sampling keeps all dimensions, e.g. shape of (h, w, c) becomes (h/2, w w/2, 4c). The fade-out layer maps removes half of the channels. The out-faded channels are than mapped to standard normal distribution to compute the unsupervised loss. For generating counterfactuals, the out-faded values are not thrown away but rather stored to be used when computing the inverse.
The model is trained using supervised loss and an unsupervised objective. In total our model had 687 layers and 261 million parameters. The classifier used the output of layer 641. The remaining layers 642-687 were optimized using the standard unsupervised flow objective. For the first 641 layers, we also trained on the classifier's supervised loss.
Let yp denote the first 641 layers and μ RR + R the last. We train &o both on a supervised loss from the classifier (xx and an unsupervised loss from matching the prior distribution M/ /) and the log determinante of the Jacobian. J is only trained on the unsupervised loss:
arg min Lun s(x)) + B Laup (w p(æ)) + b, Ytrne).
For the supervised loss Lup we the binary cross entropy. As unsupervised loss Lun- the commonly used standard flow loss obtained from the change of variables trick Dinh et al. (2016). The unsupervised loss ensures that inverting the function results in realistic looking images and can also be seen as a regularization.
The layer 342 used for the concept explanations is affine coupling layer.
16

---

Published as a conference paper at ICLR 2022
Table Test performance of the supervised trained model MobileNet-V V2 measured using mean squared error (MSE).
Attribute
Test MSE
Legs' Position
0.0008912
Bending
0.0001915
Background
0.0001 128
Color
0.0005297
Rotation Pitch
0.0009235
Rotation Roll
0,0005622
Rotation Yaw
0.002243
Position X
0.0004451
Position Y
0.0003912
Shapes
0.001 102
A.3 SUPERVISED MOBILENET-V.
We used a MobileNet- V2 to predict the attributes values. The models test squared emrors denoted Table
17

---

Published as a conference paper at ICLR 2022
B LINKS TO MODEL, DATASET AND STUDY
Model export:
https://E003 httt::/D00...ccaaeeeeeccommee/ra
benefit_from ltteefit_from_innteeeet http://cc....ppaaaaaaaaaooooaaa.
Unbiased dataset:
ob c0____olo__a_e ott.://oooooooaaaaoaaaooaaaaaaar
Biased dataset:
ob coj__ollr__at and [tt.eerioall.ieereeeeeee_e_eeee
@miffom_0.333 anniform_O.15.10
Export of study:
PDF print-out the study:
C
USER-STUDY LINKS AND VIDEOS
The studies can be accessed on the Qualtrics platform (with anonymized consent form) under the following links:
Baseline condition:
7
INN condition:
https///eeaah.aaallllie.eeo////p dneHADDGBBx.Vure
Concepts condition: condittonn
htteas///mehhha.ullllarccccc/cc/ OPKE:BQmCLO10b4
The tutorial videos can be viewed under the following links:
Introduction Tutorial for Peeky and Stretchy:
Seeky_Stretchy.mgg
Second Introduction Tutorial for ML and Biases:
Intro_ML.
Tutorial for baseline condition:
3ASE.mp1
Tutorial for concept condition:
https::/f002. CONCEPTS. _mp]
Tutorial for INN condition:
/ftps://E002.65 TNN. mp4
18

---

Published as a conference paper at ICLR 2022
D USER-STUDY PREREGISTRATION AND HYPOTHESIS
The Preregistrations can also be viewed under the following URLs:
Validation of Two4Two: httpps///appredccee...ro//////..
15J
Study 2: Concepts vs. Baseline and INN vs. Baseline: |ttps::/aspreciiteed.oom
phinxd.pppx=/788 _7P
We also paid the participants in both studies the more lucrative tariff included in the preregistration of study 2. e.g. for third comprehension task: 3,50GBP and on.
D.1 VALIDATION OF Two4TwO
Have any data been collected for this study already?
No, no data have been collected for this study yet.
2) What's 's the main question being asked or hypothesis being tested in this study?
This study investigates whether users identify biases learned by neural network. The neural networks task to discriminate between two abstract animals ("Peeky" and "Stretchy").. Each participant is presented with predictions of the system in 10x5 image grid.
After an initial tutorial phase, the participants have to find biases in the model. They do this by scoring different characteristics as relevant or irrelevant. The characteristics are: "legs position relative to the spine (LEGS)", "object color (COLOR)", "background (BACK)", "rounded or rectangular shape of the blocks (SHAPE)", and "rotation and bending (ROT)".
The main research question is whether we succeeded creating a model that contains at least one bias that is hard to detect, i.e. either COLOR or SHAPE should be harder to detect than LEGS.
HB: Participants can identify the biases in COLOR or SHAPE less frequently than LEGS.
3) Describe the key dependent variable(s) specifying how they will be measured.
Participant will answer the following questions:
How relevant the legs position relative to the spine for the system?: Relevant / Irrelevant
LEGS:
COLOR:
How relevant is the color of the animal for the system? Relevant / Irrelevant BACK: How relevant is the background of the animal for the system? Relevant / Irrelevant
SHAPE: How relevant is the rounded rectangular shape of the animal's blocks for the system? Relevant/ / Irrelevant
ROT: How relevant is the rotation and bending of the animal for the system? Relevant / Irrelevant
The ground truth answer is that LEGS, COLOR, SHAPE are relevant while BACK and ROT are irrelevant. Our first dependent variable is the number of times the head position was selected as relevant. Our second dependent variable is the number of times color of the animal was selected as relevant. Our third dependent variable is the number of times the rounded or rectangular shape of the animal's ' blocks was selected as relevant.
How many and which conditions will participants be assigned to?
Our study follows a within-subject design and has only one condition. We first show the participants introductory videos about the two abstract animals, the machine learning system, and some guidance on how to interpret the predictions of the system. Each video is accompanied by a written summary. We then show the predictions of the system in grid of images: 10 sorted rows of images drawn from the validation set (50 original images). Each the five columns represents the neural netwoks's logit range. Similarly rated images are assigned to the same column.
19

---

Published as a conference paper at ICLR 2022
5) Specify exactly which analyses you will conduct examine the main question/hypothesi...
We will conduct two exact one-sided McNemar-tests with LEGS acting as our control: one between SHAPE and LEGS and second between COLOR and LEGS. We will use one-sided test expect that SHAPE and COLOR are harder to identify. The significance level of both tests will be Bonferroni adjusted to 0.025.
6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations.
We reject participants with low effort responses or who failed to understand the dataset, machine learning concept, explanation method. We have implemented hard-coded exclusion criteria directly in the survey (implemented with Qulatrics and Prolific).
did not finish experiment at all in under 77 minutes
did not watch tutorial videos completely (there are 3 videos) or failed a multiple-choice comprehension test twice (there are four such tests), unless participants explicitly ask us to retake the study
using device smaller than tablet (min. 600 px in width or height)
provided answers about relevant characteristics in under 30 seconds
withdrawn data consent returned task on Prolific
circumvented Qualtrics protection against retaking the entire survey again (first complete submission will be counted)
We do not plan to exclude any participants who passed all the above criteria unless the qualitative answers reveal serious misunderstanding of the study instructions that the multiple choice tests did not cover. We will report such exclusions in detail in the Appendix.
How many observations will be collected or what will determine sample size?
No need to justify decision, but be precise about exactly how the number will be determined. 50 participants from Prolific with the background:
Fluent in English
Hold an academic degree
Prolific approval rate of least 90%
Did not participate in pilot studies
Passed hard coded exclusion criteria (see 8).
We pay participants max. 8.00 GBP (6.00 GBP base salary 2.00 GBP max bonus). For those failing any comprehension questions or not watching the video, we pay:
First comprehension task: compensation
Second comprehension task: 0.5 GBP
Third comprehension task: 1.75 GBP
Failed to watch first video: no compensation
Failed to watch second video: GBP
Failed to watch third video: GBP
8) Anything else you would like to pre-register? (e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?) We ask participants to answer three multiple choice comprehension tests in the form of true/false statements ensure that they understood the task and the dataset. We also ask them to provide some free-text justification of why they chose a relevant / irrelevant rating to the questions in Section 3.
20

---

Published as a conference paper at ICLR 2022
D.2 STUDY 2: CONCEPTS VS. BASELINE AND INN vs. BASELINE
1) Have any data been collected for this study already?
No, data have been collected for this study yet.
2) What's 's the main question being asked or hypothesis being tested in this study?
This study investigates whether users identify biases learned by neural network. The neural networks task to discriminate between two abstract animals ("Peeky" and "Stretchy").. Each participant is presented one of three different explanation methods: baseline (B), counterfactuals obtained using invertible neural networks (CF) and prototypes (P).
Each participant is randomly assigned to method. After initial tutorial phase, the participants have to find biases in the model. They do this by scoring different characteristies as relevant or irrelevant. The characteristics are: "legs position relative to the spine (LEGS)", "object color (COLOR)", "background (BACK)", "rounded or rectangular shape of the blocks (SHAPE)", and "rotation and bending (ROT)".
The main question of our study is whether the participants can correctly identify relevant and irrelevant attributes using these explanation methods (B, CF, P). This is reflected by two hypotheses:
HI: Participants identify relevant and irrelevant attributes with less accuracy using P compared to B.
H2: Participants identify relevant and irrelevant attributes with higher accuracy using CF compared to B.
3) Describe the key dependent variable(s) specifying how they will be measured.
Participant will answer the following questions:
LEGS:
How relevant the legs position relative to the spine for the system?: Relevant Irrelevant
COLOR:
How relevant is the color of the animal for the system? Relevant / Irrelevant
BACK: How relevant the background of the animal for the system? Relevant Irrelevant
How relevant is the rounded rectangular shape of the animal's blocks for the system? Relevant/ / Irrelevant
SHAPE:
ROT: How relevant is the rotation and bending of the animal for the system? Relevant / Irrelevant
The ground truth answer is that LEGS, COLOR, SHAPE are relevant while BACK and ROT are irrelevant. Our dependent variable the percentage of correctly answered questions per participant (accuracy, which is computed as (true positives + true negatives)/ummber of total answers).
4) How many and which conditions will participants be assigned to?
We run a between-subject study, with randomly but equally assigned participants to of conditions. We first show introductory videos about the two abstract animals, the machine learning system, the explanation technique and some guidance on how to interpret the technique. Each video is accompanied by a written summary. We then show grid (10x5) images:
1. B: NN predictions explained with 10 sorted rows of 5 images drawn from the validation set (50 original images). Each the five columns represents a score range. Similarly rated images are assigned the same column.
2.CF: Same grid layout as B, but the NN is explained by counterfactual interpolations. Each row contains interpolations which change the prediction of the NN to fit the designated score. Original images are used as starting points but are not shown.
3.P: We found concepts based the work by (Zhang et al., 2020). Each row shows set of relevant concepts. We only used concepts correlated with least r=0.2 with the model logit values. In total, display 10 rows where each row contains concept. Each row contains set of 5 example images for which the concept is relevant.
(Zhang al., 2020) https//arriv.orgaaxx220...551
21

---

Published as a conference paper at ICLR 2022
5) Specify exactly which analyses you will conduct examine the main question/hypottheii..
We will compute the accuracy scores for each participant and then compare the accuracy scores between the conditions. We expect the data to be non-normally distributed, and will test this assumption using a Shapiro- Wilk test with a significance level of a = 0.05. If our assumption is true, plan conduct Mauskll-Wallis test, followed by post-hoe analysis using saroxxoons-ran-suum tests for focused comparison between the groups CF and B (expecting higher accuracy in CF) and and B (expecting lower accuracy in
the data is normally distributed, we will conduct one-way ANOVA with planned contrasts, if the following assumptions of ANOVAs are met:
Homogeneity of the variance of the population (assessed with a Levene Test with signifi- cance level of 0.05.)
If the homogeneity of variance assumption of ANOVA is violated (assessed with Levene-Test with a significance level of = 0.05.), we plan to perform Welch's ' Anova.
6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations.
We reject participants with low effort responses who failed to understand the dataset, machine learning concept, or explanation method. We have implemented hard-coded exclusion criteria directly in the survey (implemented with Qulatrics and Prolific).
did not finish experiment at all in under 77 minutes
did not watch the tutorial videos completely (there are 3 videos) or failed multiple-choice comprehension test twice (there are four such tests), unless participants explicitly ask us to retake the study
using device smaller than tablet (min. 600 px in width height)
provided answers about relevant characteristics in under 30 seconds
withdrawn data consent / returned task on Prolific
cireumvented Qualtrics protection against retaking the entire survey again (first complete submission will be counted)
We do not plan to exclude any participants who passed all of the above criteria unless the qualitative answers reveal serious misunderstanding of the study instructions that the multiple choice tests did not cover. We will report such exclusions in detail in the Appendix.
7) How many observations will be collected what will determine sample size? No need justify decision, but be precise about exactly how the number will be determined.
240 (80 per condition) participants from Prolific with the background:
Fluent in English
First, we sample participants with academic degree. If we do reach the desired participant number, which is likely given the limited availability of such subjects, we will supplement with participants with an academic degree in other subjects. All participants will be randomly and equally split into the 4 conditions.
Prolific approval rate of at least 90%
Did not participate in pilot studies
Passed hard coded exclusion criteria (see 8).
We pay participants max. 6.50 GBP (4.50 GBP base salary + 2.00 GBP max bonus). For those failing any comprehension questions not watching the video, pay:
First comprehension task: 0.5 GBP
Second comprehension task: 1.75 GBP
22

---

Published as a conference paper at ICLR 2022
Third comprehension task: 3.50GBP
Failed to watch first video: no compensation
Failed to watch second video: GBP
Failed to watch third video: GBP
8) Anything else you would like to pre-register? (e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?)
In a previous study, we collected 50 responses for the baseline condition only (Preregistration #75056)). We do not plan use the data for this study.
We ask participants to answer three multiple choice comprehension tests in the form of true/false statements to ensure that they understood the task and the dataset. We also ask them to provide some free-text justification of why they chose a relevant / irrelevant rating to the questions in Section 3.
Additionally, ask the participants about their machine learning expertise level. Participants rate their expertise as: complete novice, some expertise, or expert the topic. We plan to use descriptive statistics to see how accuracies change per condition for each expertise level and how expertise was distributed within our sample.
We are also planning qualitative thematic analysis of the open-text questions in survey via open and axial coding, with the aim of understanding how participants integrated explanations in their reasoning about the relevance of attributes.
23


Reviews:
[{"review_id": "r3ryRnwNAj8", "reviewer": "Reviewer_gzkV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really liked this paper! It's refreshing in this community to read\nwork that focuses on users, rather than models. Evaluating\ninterpretability methods is indeed difficult, and the authors clearly\ndid a significant amount of design, pre-registration, user\ncoordination, etc.  in service of their goal. After watching the\ntraining videos myself (linked in the appendix), I found that the\nauthors conclusion interesting, but unsurprising --- in this\ncontrolled scenario, pixel highlighting (as in concept explanability)\nis ambiguous because many different attributes may show up in single\nhighlight.\n\nWhile I was generally appreciative and positive about the work, it\ndoes have some shortcomings.\n\n1. The stretchy vs. peeky environment, while cute, is quite different\n   than most computer vision tasks. While I understand the motiviation\n   for being able to generate images with particular properties, I do\n   wonder if the results here would transfer to, e.g., object\n   classification in photographic images.\n\n2. The stretchy vs. peeky environment is a binary classification\n   setup, which gives rise to the baseline method of simply showing\n   predictions that the authors compare against. However, for more\n   than two classes, this baseline is not directly usable. If there\n   were 1K classes, it wouldn't be possible to lay out model\n   predictions in a 2D grid. My suspicion is that this is partly a\n   motivating factor for a method like concept highlighting where\n   pixels are highlighted in input space. For higher dimensional\n   classification tasks, it may be impractical or impossible to show a\n   2D grid for correct vs. incorrect for all classes; so, something\n   like pixel highlighting might be more useful in that case.", "summary_of_the_paper": "The authors explore several explanation methods for image classifiers\nvia a user study. They study a toy environment containing images of\ntwo animals: stretchy (who has stretched legs) and peeky (who has a\nhead that extends beyond its front legs). While the true label of\npeeky vs. stretchy is defined deterministically and specifically, the\ntoy environment enables the authors to generate spurrious correlations\nbetween attributes like background color, animal position, shape,\netc. and the label (which the model picks up on). The goal of the\nexplanation methods is to help users identify which features\n(spurrious and not) the model is picking up on. Through a series of\npre-registered user studies comparing a simple baseline to\ncounterfactual explanations to concept highlighting, they explore\nwhether or not users can accurately reconstruct which features the\nmodel is using in its predicitons. They find that concept highlighting\nperforms far worse than the baseline of simply showing some model\npredictions in a grid, and that the baseline and the counterfactual\nmethod performed similarly. They release their procedures and\ngeneration code as a challenge to the community: can a new\ninterpretability method outperform their baseline?", "summary_of_the_review": "Overall, I think more work like this should appear at ICLR ---\nexplanability is an important topic in machine learning, and, while a\nuser study is perhaps less conventionally presented at ICLR, I think\nit should be. Furthermore, I admire the experimental practices\npresented here, e.g., preregistration. I think my two big critiques\n(toy-ness not applying to more common image clf problems, and\nnon-binary tasks) can be addressed via presentation updates in\nrevision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nu8Kh4_ziy7", "reviewer": "Reviewer_pBBz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "For the proposed dataset, more examples are expected given for each of the two classes. The definition of ‘legs moved inwards’ is not rigorous or clear. The same for ‘legs stretched out’.  For example, for Peeky, does it mean as long as one pair of legs moving to the center of the body? Or both have to. How to determine which two nodes belong to a pair. Why not the upper left and upper right. Also, the definition of ‘legs moved inwards’ should be defined like angle or something like that more rigorously in math.\n\nWhat is the main feature? The four ‘body posture (bending and three rotation angles), position, animal color (from red to blue), blocks’ shape (from cubes to spheres), and background color (from red to blue)?’. Are they continuous or discrete? For this sentence,\n‘A single scalar encodes the legs’ position.’, is ‘legs’ position’ is a feature? How to measure it? Or is there any visualization showing what 0 or 1 look like?\n\nI’m a little confused with this statement. ‘we sampled the block’s shape in a non-predictive biased fashion. For very stretched legs’ positions (Stretchy), the data distribution was biased towards round blocks, while for more retracted legs’ positions (Peeky), most blocks were cubic.’ If two classes are sampled from different distributions, shouldn’t it be predictive?\n\nFor ‘We generated the concepts using layer 342 (from a total of 641).’, what is layer 342?\n\nFor the user study, what task the online workers are asked to do. What’s the question to ask so that they are asked to identify the class-relevant features. Or is there an interface example?\n\nAre there any statistics for the user study, like time spent? Maybe this can reflect if the task is hard, an evidence for the argument in section 5.1.\n\nThe first line under section 5.3, ‘As stated iWe automatically’. iWe? Just a typo?\n\nOne paper that uses a similar synthetic dataset. Maybe useful for the concerned task in this paper.\nYuxin Chen, Oisin Mac Aodha, Shihan Su, Pietro Perona, Yisong Yue’ Near-Optimal Machine Teaching via Explanatory Teaching Sets’", "summary_of_the_paper": "The paper proposed a synthetic dataset to explore the bias contained in the dataset. Because the dataset is synthetic. We can manually change its attributes, add or eliminate the bias. Then, two main user study is conducted. One is to see if users can find the bias and another is to investigate if explanations are helpful.", "summary_of_the_review": "My major concern is that this paper does not have any technical contributions. And the proposed dataset is not very interesting or distinct compared to the existing. The dataset building or description is also not very rigorous.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZoTNJtyd0LD", "reviewer": "Reviewer_3sBP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n-------------\n(1) This paper is well written, and it provides a nice method for establishing an artificial dataset where the ground-truth explanation is available. It also reveals the problem that many model explanation methods encounter: is the explanation useful for users? \n(2) The user study includes many participants, which provides a solid result. \n\nWeakness: \n---------------\n(1)\tIs it fair to compare concept model in this dataset/ user study? First, the attributes (biases) that should be understood by users are very abstract and cannot easily visualized in concept activation maps. Second, if the users see only 5 images from one concept class (as shown in Fig.1 c), it is not clear which attribute is important for the decision. However, on other two conditions (B and CF), they have the classes and confidence bar above. Does the concept model also have this when shown to the users? \n\n(2)\tCould the participant in study 1 also participant in study 2? If so, the answers may get interfered since the users might already know which attributes were important and can bring that knowledge into their answers. \n\n(3)\tHow did the classifier used for computing explanations as well as in baseline perform? It may also influence the quality of explanations generated.\n\n(4)\tThe main concern is whether the proposed dataset is general enough to fairly compare different explanation methods on it. For instance, there is a large number of attribution-based explanations. They are visually similar as concept-based maps, which might also fail in highlighting the attribute such as shape or color. \n\n(5)\tIn Table 2, what do the stars imply?\n", "summary_of_the_paper": "This paper proposes a dataset called TWO4TWO to conduct a user-study on two interpretability methods: counterfactual and concept-based explanations. Since the dataset is generated and can be fully controlled by users, the ground-truth important attributes to the decision are known. In this case, we know the ground-truth features and can thus assess whether the model explanation also reveal the correct features. As a baseline explanation, the authors group the input according to the model’s output logits. The result from the user study shows that the two sophisticated explanation models don’t surpass the simple baseline method, which indicates that explanation techniques shall be evaluate in user studies. ", "summary_of_the_review": "The authors reveal an interesting research question and provide with a user study design. The results from the user study seem solid. However, the dataset and the questions are not general and proper to assess concept-based or attribution-based explanations. The authors should first improve or clarify this issue. Otherwise, the dataset may not be very useful.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HosDef9JO-s", "reviewer": "Reviewer_rVJ7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. This is a very novel and clever experiment design that has a clear definite ground truth solution and irrelevant other features that may confuse the ML systems. Those other features are visually overlapping (shape, color, etc) and are hard to be visually explained.\n2. The results are interesting that under these extreme experiments, the currently popular explanation approach can not outperform the confidence represented by the logits. \n\nWeakness:\nBesides the limitation stated on the last page:\n1. The experiment protocol requires a controllable generator of the training data which is hard to extend to other visual tasks (for example image manipulations). It is hard to generate data are on the manifold of the natural data for using the current approach to test the explanation results for real data", "summary_of_the_paper": "The paper introduces an experiment design and an approach to synthesizing the dataset for the experiment. The experiment asked the systems to classify whether the shape of the \"animal\" is Peaky or Stretchy. The advantages are that authors can controllably generate trainable examples under arbitrary biases of the predefined features (shape, color, etc). For experiments, human subjects are asked to predict the systems' output. The authors compared the visual explanation (concept explanation) and counter-factual explanation with the baseline explanation that uses the output logits and found that the the current good explanation approaches do not make humans understand the system significantly better", "summary_of_the_review": "The paper provides a  novel and clever experiment design to show that the limit of the current explanation approaches but shows limited potential to be used as a good protocol for large-scale real data tasks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a novel and clever experiment for interpretable vision.  Reviewers all agreed that it tackles an important and interesting research question via a user study design.  There are some concerns around the generalization and transfer to large-scale real-world settings, as well as dataset construction. With the authors’ responses and discussion, I think the pros seem to outweigh the cons of this work a bit.
Decision: Accept (Poster)