Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces an experiment design and an approach to synthesizing the dataset for the experiment. The experiment asked the systems to classify whether the shape of the "animal" is Peaky or Stretchy. The advantages are that authors can controllably generate trainable examples under arbitrary biases of the predefined features (shape, color, etc). For experiments, human subjects are asked to predict the systems' output. The authors compared the visual explanation (concept explanation) and counter-factual explanation with the baseline explanation that uses the output logits and found that the the current good explanation approaches do not make humans understand the system significantly better
main_review: Strength:
1. This is a very novel and clever experiment design that has a clear definite ground truth solution and irrelevant other features that may confuse the ML systems. Those other features are visually overlapping (shape, color, etc) and are hard to be visually explained.
2. The results are interesting that under these extreme experiments, the currently popular explanation approach can not outperform the confidence represented by the logits. 

Weakness:
Besides the limitation stated on the last page:
1. The experiment protocol requires a controllable generator of the training data which is hard to extend to other visual tasks (for example image manipulations). It is hard to generate data are on the manifold of the natural data for using the current approach to test the explanation results for real data
summary_of_the_review: The paper provides a  novel and clever experiment design to show that the limit of the current explanation approaches but shows limited potential to be used as a good protocol for large-scale real data tasks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a synthetic dataset to explore the bias contained in the dataset. Because the dataset is synthetic. We can manually change its attributes, add or eliminate the bias. Then, two main user study is conducted. One is to see if users can find the bias and another is to investigate if explanations are helpful.
main_review: For the proposed dataset, more examples are expected given for each of the two classes. The definition of ‘legs moved inwards’ is not rigorous or clear. The same for ‘legs stretched out’.  For example, for Peeky, does it mean as long as one pair of legs moving to the center of the body? Or both have to. How to determine which two nodes belong to a pair. Why not the upper left and upper right. Also, the definition of ‘legs moved inwards’ should be defined like angle or something like that more rigorously in math.

What is the main feature? The four ‘body posture (bending and three rotation angles), position, animal color (from red to blue), blocks’ shape (from cubes to spheres), and background color (from red to blue)?’. Are they continuous or discrete? For this sentence,
‘A single scalar encodes the legs’ position.’, is ‘legs’ position’ is a feature? How to measure it? Or is there any visualization showing what 0 or 1 look like?

I’m a little confused with this statement. ‘we sampled the block’s shape in a non-predictive biased fashion. For very stretched legs’ positions (Stretchy), the data distribution was biased towards round blocks, while for more retracted legs’ positions (Peeky), most blocks were cubic.’ If two classes are sampled from different distributions, shouldn’t it be predictive?

For ‘We generated the concepts using layer 342 (from a total of 641).’, what is layer 342?

For the user study, what task the online workers are asked to do. What’s the question to ask so that they are asked to identify the class-relevant features. Or is there an interface example?

Are there any statistics for the user study, like time spent? Maybe this can reflect if the task is hard, an evidence for the argument in section 5.1.

The first line under section 5.3, ‘As stated iWe automatically’. iWe? Just a typo?

One paper that uses a similar synthetic dataset. Maybe useful for the concerned task in this paper.
Yuxin Chen, Oisin Mac Aodha, Shihan Su, Pietro Perona, Yisong Yue’ Near-Optimal Machine Teaching via Explanatory Teaching Sets’
summary_of_the_review: My major concern is that this paper does not have any technical contributions. And the proposed dataset is not very interesting or distinct compared to the existing. The dataset building or description is also not very rigorous.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a dataset called TWO4TWO to conduct a user-study on two interpretability methods: counterfactual and concept-based explanations. Since the dataset is generated and can be fully controlled by users, the ground-truth important attributes to the decision are known. In this case, we know the ground-truth features and can thus assess whether the model explanation also reveal the correct features. As a baseline explanation, the authors group the input according to the model’s output logits. The result from the user study shows that the two sophisticated explanation models don’t surpass the simple baseline method, which indicates that explanation techniques shall be evaluate in user studies. 
main_review: Strength: 
-------------
(1) This paper is well written, and it provides a nice method for establishing an artificial dataset where the ground-truth explanation is available. It also reveals the problem that many model explanation methods encounter: is the explanation useful for users? 
(2) The user study includes many participants, which provides a solid result. 

Weakness: 
---------------
(1)	Is it fair to compare concept model in this dataset/ user study? First, the attributes (biases) that should be understood by users are very abstract and cannot easily visualized in concept activation maps. Second, if the users see only 5 images from one concept class (as shown in Fig.1 c), it is not clear which attribute is important for the decision. However, on other two conditions (B and CF), they have the classes and confidence bar above. Does the concept model also have this when shown to the users? 

(2)	Could the participant in study 1 also participant in study 2? If so, the answers may get interfered since the users might already know which attributes were important and can bring that knowledge into their answers. 

(3)	How did the classifier used for computing explanations as well as in baseline perform? It may also influence the quality of explanations generated.

(4)	The main concern is whether the proposed dataset is general enough to fairly compare different explanation methods on it. For instance, there is a large number of attribution-based explanations. They are visually similar as concept-based maps, which might also fail in highlighting the attribute such as shape or color. 

(5)	In Table 2, what do the stars imply?

summary_of_the_review: The authors reveal an interesting research question and provide with a user study design. The results from the user study seem solid. However, the dataset and the questions are not general and proper to assess concept-based or attribution-based explanations. The authors should first improve or clarify this issue. Otherwise, the dataset may not be very useful.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors explore several explanation methods for image classifiers
via a user study. They study a toy environment containing images of
two animals: stretchy (who has stretched legs) and peeky (who has a
head that extends beyond its front legs). While the true label of
peeky vs. stretchy is defined deterministically and specifically, the
toy environment enables the authors to generate spurrious correlations
between attributes like background color, animal position, shape,
etc. and the label (which the model picks up on). The goal of the
explanation methods is to help users identify which features
(spurrious and not) the model is picking up on. Through a series of
pre-registered user studies comparing a simple baseline to
counterfactual explanations to concept highlighting, they explore
whether or not users can accurately reconstruct which features the
model is using in its predicitons. They find that concept highlighting
performs far worse than the baseline of simply showing some model
predictions in a grid, and that the baseline and the counterfactual
method performed similarly. They release their procedures and
generation code as a challenge to the community: can a new
interpretability method outperform their baseline?
main_review: I really liked this paper! It's refreshing in this community to read
work that focuses on users, rather than models. Evaluating
interpretability methods is indeed difficult, and the authors clearly
did a significant amount of design, pre-registration, user
coordination, etc.  in service of their goal. After watching the
training videos myself (linked in the appendix), I found that the
authors conclusion interesting, but unsurprising --- in this
controlled scenario, pixel highlighting (as in concept explanability)
is ambiguous because many different attributes may show up in single
highlight.

While I was generally appreciative and positive about the work, it
does have some shortcomings.

1. The stretchy vs. peeky environment, while cute, is quite different
   than most computer vision tasks. While I understand the motiviation
   for being able to generate images with particular properties, I do
   wonder if the results here would transfer to, e.g., object
   classification in photographic images.

2. The stretchy vs. peeky environment is a binary classification
   setup, which gives rise to the baseline method of simply showing
   predictions that the authors compare against. However, for more
   than two classes, this baseline is not directly usable. If there
   were 1K classes, it wouldn't be possible to lay out model
   predictions in a 2D grid. My suspicion is that this is partly a
   motivating factor for a method like concept highlighting where
   pixels are highlighted in input space. For higher dimensional
   classification tasks, it may be impractical or impossible to show a
   2D grid for correct vs. incorrect for all classes; so, something
   like pixel highlighting might be more useful in that case.
summary_of_the_review: Overall, I think more work like this should appear at ICLR ---
explanability is an important topic in machine learning, and, while a
user study is perhaps less conventionally presented at ICLR, I think
it should be. Furthermore, I admire the experimental practices
presented here, e.g., preregistration. I think my two big critiques
(toy-ness not applying to more common image clf problems, and
non-binary tasks) can be addressed via presentation updates in
revision.
review_title: null

Meta-review: This work presents a novel and clever experiment for interpretable vision.  Reviewers all agreed that it tackles an important and interesting research question via a user study design.  There are some concerns around the generalization and transfer to large-scale real-world settings, as well as dataset construction. With the authors’ responses and discussion, I think the pros seem to outweigh the cons of this work a bit.
Decision: Accept (Poster)