Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes a scheme to identify 'causal' and 'spurious' features with human supervision. More specifically, the method presented decomposes the activations that map to the logits of a supervised DNN into neural features. The neural features are then ranked and visualized with the CAM feature attribution method (called neural activation map in this work). The paper then shows the top ranked features and the corresponding heatmaps to crowd workers, who then ascertain whether a feature is spurious or not. 

Given a ranking of causal and spurious features for different output classes, the work then maps these features across an entire 'audit'/validation set. Given this audit set, one can then mask out the regions in the heatmap that correspond to the casual features and observe the corresponding change in model accuracy. Similar sensitivity measurements can also be made for the spurious features as well. Overall, this paper produces an annotated version of the ImageNet dataset with causal and spurious features. This is then used to assess the performance of current state of the art models showing that these models show significant susceptibility to spurious training signals.

Overall, this paper shows how to use additional supervision for detecting a model's reliance on spurious features. The 'Causal Imagenet' dataset should be useful for future work as a stress test for models.
main_review: This paper tackles an important problem and provides a convincing demonstration that current SOTA models are susceptible to reliance on spurious correlation in the training set. I detail key strengths, weaknesses, and additional questions below. 

###  Strengthens
- The focus on the ImageNet dataset is impressive and crucial to demonstrating that the reliance on spurious correlation issue is not isolated to toy datasets, but prevalent for one of the currently most important datasets in machine learning.
- The use of crowd source annotations is also helpful and interesting. 
- While the pieces of the entire scheme presented in this work are not new, the combination is interesting and extends recent work that is focused on detecting whether a model relies on spurious correlation.


### Weaknesses, Concerns,  & Additional Questions
- 'Causality': This paper uses the term 'causal' inappropriately in my opinion. Nothing about the new dataset is 'causal' in any sense of the word. At this point, causality refers to a specific set of techniques that capture the data generating process. The Causal ImageNet here is a dataset that indicates which are 'core' features and which are spurious features. I would encourage the authors to completely discard the notions of causality that they refer to repeatedly across the paper. Further, the definition of 'causal' features in this work is quite vague, but acceptable. I would strongly suggest that the authors call those features 'core' features or something of that sort. 
- Non spatial spurious signals: The scheme presented here would only be effective for spatial/dimension dependent features. If the DNN learns say Gender/Race or some other concept that is not visualizable by a heatmap then the approach described here will not work. I would suggest that the authors make this clear in the paper. This is not a limitation in my view, but it allows the reader to understand the scope of the work presented. 
- Clarify the entire scheme and specific assumptions: As it stands, it is difficult to understand the entire scheme here. Part of the difficulty is that Section 3 should be written better to delineate each component of the entire scheme, i.e., if someone else wanted to apply this technique to a different dataset and model, what is needed? State the inputs, outputs, assumptions, etc necessary to apply the scheme discussed here to another setting. Since the authors say they are presenting a scalable framework for discovering spurious features, then it is important to completely state all the necessary requirements to apply the method.
- What prevents this method from being applied to a non-robust model? Will the approach be less effective? I don't understand whether the models used in Figure 6 are robust or standard models. I think the models for Fig. 6 are standard, i.e. non-adversarially trained, and if yes, why would annotations be collected for a robust model and used to test a non-robust one?
- Why CAM? Could any feature attribution method have worked here?
- The user study is interesting, but it is unclear what one can learn from how the crowd workers determined which feature is spurious or not. I say this because the study had the top activating images, the heat maps, the feature attack, wikipedia link, object names etc. To me, it is unclear which of these sources of information helped the crowd workers to determine that a feature is spurious or not. This is quite important information if this approach is to be replicated else where.
- Are spurious features only learned at the top layer? One assumption in this work is that the important features learned by a DNN are at the penultimate layer? What is the justification for this assumption? I am fine with this assumption, but curious if there is any past work that has made this clear. Could it be possible that a model could concentrate a spurious signal to a filter at the intermediate layer? 
- Given that one can detect reliance on spurious signals, could as straightforward fix be to fine tune the model and include a penalty on the heat maps to force the network to be invariant to the spurious features? This kind of approach has been proposed in prior work.

### Minor Feedback
- The change in accuracy numbers in Figure 1 were confusing on first read. Perhaps the authors can provide additional statements in the caption to explain what the numbers mean. After reading the paper it makes sense, but it was confusing at first. 
- In the introduction and the related work, the authors claim that their approach is more 'scalable' than current alternatives. I don't agree. To use the approach presented here, one needs to run a user study to collect more features, and another study to confirm that the spurious features detected will generalize. I think the spurious feature detection issue is so important that the paper doesn't need the scalability argument. Any method that requires one to run a study per dataset is not necessarily scalable, but that is also fine!
- Here is recent work that this paper should also acknowledge: https://arxiv.org/abs/2106.02112

### Post Rebuttal Updates
The author response directly addresses most of the concerns that I had. I maintain my accept rating for this paper.
summary_of_the_review: I recommend an accept at this time because most of the weaknesses I identified with this paper have to do with the strength of the claims made, which can be addressed in a revision. I particularly like that the paper demonstrates significant weaknesses, reliance on spurious features, of current models on the ImageNet dataset. On the strength of that contribution alone, I think this work is an important one.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In the paper, the authors presented a novel scalable method to differentiate the so-called causal or non-causal attributes. 
By utilizing the neural features activations of in the penultimate layer, i.e. the one adjacent to logits, the proposed method can allow efficiently annotate plausible masks indicating whether the decision-making process of an NN relies on the correct features. 

main_review: Pros:

1. The paper is well-motivated and well-written, making its readers easier to follow the core idea delivered in the paper.
This will potentially make a good impact on the community.

2. The Visualization is very intuitive, this also helps me to quickly pinpoint the spurious vs causal features mentioned in the paper.

3. The scalability contribution is well-justified by the paper.

Cons:

1. Despite the acknowledgement of using less rigorous causal terminology, the issue remains. Take 'spurious' as an example, the author used descriptive language such as 'co-occur', it would be better later in the main content to add formal discussion related to Reichenbach’s common cause principle (Peters et al. (2017)).  

2. What does causal formally mean in this paper?  

The above two questions actually relate to my thought inspired Fig 5 d and e. In Fig 5 d and e, the mask highlights the suspicious attributes which could be the reflected shadow of the bird (maybe I am wrong.) But must this feature be spurious? What if the river is very quiet without any ripple and it is literally a mirror? I think a more precise discussion about causal vs spurious in vision scenario could thus be very useful.

3.  I am a little bit unclear about the concrete usage of this method. Could the authors enlighten me a little more on how we could potentially utilize this model to further improve either the interpretability or robustness of NNs.

summary_of_the_review: Based on the pros and cons discussed above, I believe this paper bears the good potential to bring useful insights to the community, though a more thorough discussion from the theoretical side should be improved. 

After reading the response from the authors, I am pleased with the paper revision and decide to raise my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: - The paper describes a new benchmark. It uses images from ImageNet, which now have annotations of regions containing features spuriously correlated with the ground truth labels.
- The key contribution is a method to generate these annotations semi-automatically, by propagating a small set of human annotations to a large number of images.
- The dataset is used to evaluate existing models, showing they often rely on regions marked as spurious.
main_review: 
Strong (+) and weak (-) points

- (+) The need for annotations of causal/spurious features is important to support research on robust models/domains generalization/etc.
- (+) The method to propagate annotations from a small set of human labels is simple, effective, and based on observations from prior work that features of the last layer of a CNN are semantically aligned with human concepts.
- (-) There is however no theoretical basis/guarantee that this is the case.
- (+) But the authors performed a validation (Sec. 3.4) by crowd-sourcing that seems to support their assumptions.
- (-) The dataset is large but not huge (50k images, 232 classes).
- (+) The composition of the dataset (e.g. selection of the subset of 232 classes) seems to have been well thought.
- (+) The overall design of the data collection procedure generally inspires confidence that the crowd-sourced annotations are of good quality.
- (+) The authors use the new dataset to evaluate existing models and provide new observations that high accuracy on some classes is achieved by relying on spurious features.



Additional advice to improve the paper

- In the intro, the mention of cows/pastures is cited with Arjovsky et al. I'm not sure this was actually done as an experiment as part of this paper. My recollection is that it was more of a thought experiment (to be verified) and/or a citation of the Terra incognita paper of Beery et al.
- Typo: casual instead of causal
- The message in the last section that the standard accuracy metric does not tell the whole story is a very important one, IMO. It may benefit from being highlighted more prominently, possibly in the abstract, or maybe in bold in the last section as a form on conclusion/takeaway.
summary_of_the_review: Preliminary recommendation: accept

Key reasons
- Importance of the topic/addresses a real need of the research community.
- Simple method, no technical flaw, underlying assumption are verified empirically.
- Experiments with the new dataset already provides new insights about existing models.
review_title: null

Meta-review: The paper tackles the important problem of spurious feature detection in deep neural networks. Specifically, it proposes a framework to identify core and spurious features by investigating the activation maps with human supervision. Then, it produces an annotated version of the ImageNet dataset with core and spurious features, called Salient ImageNet, which is then used to empirically assess the robustness of the method against spurious training signals in comparison with current SOTA models.

As pointed out by the reviewers, this work is not about causality and the definitions of causal and spurious features were originally vague and inaccurate. During the revision and discussion,  the authors changed the terms "causal" features/accuracy to "core" features/accuracy. They also called the provided dataset "Salient Imagenet", instead of "Causal Imagenet", and changed the title to "Salient ImageNet: How to Discover Spurious Features in Deep Learning?". Following the prior discussion, we strongly recommend the authors discard any discussion about causality in the camera-ready version of the paper to avoid confusion. Further, we encourage the authors to consider the reviewers’ thoughts and comments in preparing the camera-ready version of their manuscript.
Decision: Accept (Poster)