Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper tackles the problem of neural networks learning ‘simple’ features which cannot generalize well. It proposes to alleviate this problem of shortcut learning by generating synthetic data (“roadblocks”) in a manner which confuses the network while being close (in $l_2$ norm) to the original training data. A new model is trained on such examples, and it is hypothesized that this new model learns more robust features. The paper empirically shows that such models are more robust when there are spurious correlations in the data. The paper also presents visualizations of learnt features, as well as of the generated roadblocks, indicating that these do contribute to learning more robust features.  
strengths_and_weaknesses: Strengths - 
1. The empirical results on the debiasing task are promising. 
2. The method does not need any priors about the dataset in order to work.

Weaknesses - 
1. The work misses several key references. Shortcut learning is very closely related to the problem of simplicity bias in neural networks [1], which is not cited in the work.
2. The method is also very similar to training with adversarial examples [2]. This line of work is neither cited nor compared against. On the mentioned datasets, the generated images indeed look very similar to $l_2$ or $l_\inf$ constrained adversarial attacks.
3. The paper proposes using the MSE loss for classification, which is not standard.
4. The writing of the paper can be improved significantly.  


[1] - Shah, Harshay, et al. "The pitfalls of simplicity bias in neural networks." Advances in Neural Information Processing Systems 33 (2020): 9573-9585.
[2] - Ilyas, Andrew, et al. "Adversarial examples are not bugs, they are features." Advances in neural information processing systems 32 (2019).

questions: 1. How important is the distance metric for imposing loss between the generated images and the real images? Can a perceptual loss be used for the same?
2. How does the method compare against simple adversarial training?
3. Why was the MSE loss used for classification? Can cross-entropy give better results?
limitations: Some of the limitations are listed in the weaknesses above. A more thorough empirical evaluation on other OOD tasks would also be appreciated.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: A method for OOD generalization is proposed. To prevent a model (Explorer Model) from learning "shortcut features" (simple features which do not work in OOD setting~spurious correlations), an autoencoder is trained, which tries to fool a pretrained model (Blocked Model) as an additional loss. The explorer model is then trained on the reconstructed image, from which some of the shortcut features were hopefully removed by AE to fool the Blocked model.
strengths_and_weaknesses: Strengths:
- Promising framework and approach, which does not require any external knowledge but relies on the interaction between autoencoder and blocked model.
- Interpretability: can visualize reconstructed images to gain insight (e.g. Figure 3 of CMNIST)
- Indirectly applicable to debiasing (by removing shortcut features)

Weaknesses:
- Relies on autoencoder reconstruction, which may be blurry and lower the prediction quality.
questions: - Can we see some samples of CelebA reconstructed by the autoencoder?
- Can you discuss the relationship/tradeoffs with using a better autoencoder/pretrained autoencoder, which may limit the applicability of this method to harder datasets?
limitations: Relies on autoencoder reconstruction, which may be blurry and lower the prediction quality.

This method seems to implicitly perform some debiasing, which should hopefully have a positive impact in terms of making better predictions. However, this should be more extensively tested on a wide variety of tasks to see if the usual fairness/bias metrics are consistently improved.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose an approach to address the problem of shortcut vulnerability of deep vision models in image classification tasks. For a trained, shortcut exploiting model (BM) the authors propose to train a model (AE) that learns to modify training samples in a minimal way while yielding a maximum entropy prediction for the BM model. Using these manipulated samples, an explorer model is trained (EM) in a supervised manner to predict the original ground-truth labels. The intuition behind the approach is that AE learns to manipulate input samples specifically to remove the shortcut information in the image in order to enforce the explorer model to learn new, actual class information and is hence less shortcut vulnerable. 
Empirical evidence is provided for the CMNIST, CelebA and BAR datasets indicating the improved classification generalization. Further, the authors provide qualitative evaluation in form of an inspection of the manipulated images, as well as saliency evaluations comparing BM and EM which suggest that the AE model in fact removed the shortcut information and that the EM model attends to more meaningful image areas than BM. 
strengths_and_weaknesses: In a nutshell, the work address a major, central problem of current ML The proposed solution to the problem is original and some interesting empirical results are provided. Still, I think some limitations / questions should be addressed more explicitly (see below). I would like to increase my rating in case stated limitations, questions (see below here under weaknesses) get addressed sufficiently.

Strengths
------------
The work addresses the problem of shortcut vulnerability of ML models in a supervised learning setup. This is of immense significans for the entire ML community.  As far as I can tell, the authors propose a novel, original approach to tackle this challenge in the effort to traina more shortcut robust model. The authors present their central idea in a clear, easy to access manner and present some quantitative evidence that supports the central claims of the paper (Tab 1, 2). Also of value are the provided qualiatitve evaluations indicating that the modification model (AE) works as intended on the investigated dataset CMNIST (Fig. 3) and a saliency comparison for BM and EM (Fig 5) on the BAR dataset, indicating that EM attends to more meaningful image areas than BM.

Weaknesses
----------------
Major 1

It is not clear to me what the AE actually learns and in which cases it actually learns to remove shortcut information. I think the provided intuition how AE should work is sound and the provided results support this intuition, however question about how AE actually works and how well it generalizes to other cases remain unclear and should be addressed:
* Why does AE not learn to produce (untargeted) adversarial noise? This would have the same effect as removing the shortcut, i.e., induce minimal change in the image and yield uniform classification. Please address in the text.
* Why does AE not learn to remove actual class information? This is especially relevant in case of low correlations (ratio) between the shortcut and the actual target class in the training dataset. In those cases, AE would probably learn to also remove the actual class information. Please discuss this point.
* More complicated shortcut structures might exist (shortcuts of shortcuts of shortcuts of ...). For this, see also [1] which should also be cited in intro / related work. This point seems partially addressed in L126-L133 but could be made more clear.
* Dependent on the design of the AE model, only rather specific types of shortcuts might be addressed with the used AE model. Also see [1] for more shortcut examples. For example, it is not clear that AE would be able to address the texture bias on ImageNet. Please also address why ImageNet was not selected as a dataset.
* To improve intuition about how AE works, please also provided AE modified samples for the BAR and CelebA dataset (similar to Fig 3 for CMNIST) and also provide saliencies for CMNIST and BAR (similar to Fig 5). Either Fig 3, 5 could be extended or additional images can be shown in the supplementary material.
* How cherry-picked are samples in Fig 3 and 5. Information in the text is missing.

[1] "DiagViB-6: A Diagnostic Benchmark Suite for Vision Models in the Presence of Shortcut and Generalization Opportunities", ICCV 2021

Major 2

Code should be made available to enable reproducibility. I could not find a code publishing statement in the paper.


Medium

* The role of the sum-up combination is not clear. Is this used somewhere in the experiments or is this merely a proposal for an open-set-recognition task? If of no practical relevance for this work it could be left out entirely.
* It should be noted somewhere that this training proceedure, similar to all data augmentation approaches, induces certain invariances in the EM model which might lead to unintended behavior on outliers.
 


questions: See weaknesses above.
limitations: See weaknesses above.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposed a learning approach for robustness to shortcut learning: after training a base model (BM) on the biased dataset, the auxiliary autoencoder (AE) is trained to modify train images such that their reconstruction is close to the original input but the trained model  is outputting a uniform distribution over classes. Then another "exploration" model (EM) is trained on the modified inputs to predict the correct class. Thus, the intuition for the approach is that AE should remove the shortcuts from the inputs that BM learned, and EM will learn more diverse features. The approach was evaluated on ColorMNIST, CelebA and BAR benchmarks, and the authors provided additional analysis and visualizations of the features.
strengths_and_weaknesses: The paper is addressing an important problem of learning diverse features and training models robust to shortcuts, and the proposed approach shows promising results on CelebA and BAR, however, the clarity and presentation can be significantly improved, and the paper is missing implementation and method details. The results on CMNIST are not directly comparable to prior work due to a different model architecture. Also, experiments on large scale datasets would be needed to demonstrate the practicality of the proposed method.


### Originality
The paper is missing citations of the relevant literature on shortcut learning, simplicity bias, group robustness, and spurious correlations, e.g.:

Sagawa et al. Distributionally Robust Neural Networks for Group Shifts

Liu et al. Just Train Twice: Improving Group Robustness without Training Group Information

Shah et al. The Pitfalls of Simplicity Bias in Neural Networks

Xiao et al. Noise or Signal: The Role of Image Backgrounds in Object Recognition

Teney et al. Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions With Superior OOD Generalization

Nagarajan et al. Understanding the Failure Modes of Out-of-Distribution Generalization

Dagaev et al. A Too-Good-to-be-True Prior to Reduce Shortcut Reliance

Clark et al. Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases

Lee et al. Diversify and Disambiguate: Learning From Underspecified Data

and others. To the best of my knowledge, the proposed approach is novel. However, I encourage authors to explicitly discuss the connection of their work with Just Train Twice [Liu et al], Dagaev et al (see above) and LfF as all of these prior works are based on using 2 models where the second model is correcting the biases of the first one similarly to the proposed approach (while it additionally uses an AE to modify the inputs).
 

### Quality
1. ColorMNIST experiments: In the Appendix A.1 it is mentioned that ColorMNIST dataset is setup as in Lee et al [1]. It is also mentioned that ResNet-18 is used in all experiments. I noticed that the numbers for the Vanilla, EnD, ReBias, LFF, and LDD in Table 1 in this paper are taken from the Table 2 in Lee et al [1] (they match those numbers exactly). However, right below Table 2 in [1] it is mentioned that an MLP architecture is used in the experiments, while this paper uses ResNet-18 so the comparison in Table 1 is not valid.
(In general, I would also recommend that the authors explicitly say that which papers they took the numbers from for the prior methods in case they haven't re-run the methods.)

2. While the Roadblocks approach works well on CelebA [Heavy Makeup, Gender] and BAR, it would be very helpful to see the performance on a large scale problem, e.g. ImageNet-9 [2]. It would also be interesting to see the results on a broader range of benchmarks in shortcut learning / spurious correlations, e.g. Dominoes (MNIST-CIFAR, MNIST-FashionMNIST, MNIST-MNIST) [3, 4], Waterbirds, CelebA [hair color, gender] [5]. These benchmarks vary in the strength of the spurious correlation / bias in train data and the complexity of the shortcut feature.

3. How are hyper-parameters chosen for the models (lambdas loss coefficients, and also weight decays, learning rates, hyper-parameters of alternate training, etc)? Do authors use in-distribution validation set or bias-conflicting validation? This can have a high impact on the model's performance.

4. I would encourage the authors to provide the reconstructed images like in Figure 3 for other datasets besides CMNIST.


### Clarity
Clarity of the paper can be significantly improved. I strongly recommend the authors to add a brief datasets description to the main paper (e.g. the problem description, the nature of the bias / shortcut, the percentage of bias conflicting examples in train, etc). These details are not included in the paper, and some details become unclear without the context e.g. what "Ratio %" refers to in the Table 1.

It is mentioned that AE and EM are trained alternately, however, the details of this training are not described (i.e. are models alternately updated after each step or epoch?).

I strongly encourage the author to add the high-level details on the prior works to which Roadblocks method is compared to either in the Experiments section or in the Related work. In particular, it will be helpful to know how prior works compare to Roadblock method in terms of computational complexity, memory, etc.

Some paragraphs are unclear (e.g. lines 134-139, 172-176, 288-301) are not clearly written, lack details and context.

I also strongly encourage authors to add more details in Table and Figure captions in the next revision of the paper (e.g. Figure 6 doesn't have titles for subplots to indicate which dataset they correspond to).

It is not clear how the final predictions are made with the proposed model: is only EM model used? It is not clearly explained in the paper.

Line 47: "shortcuts are disabled temporarily, not forever", what does this mean? My understanding is that EM is trained on modified images outputted by AE so it is not clear what "temporarily, not forever" refers to.

Equation (3): why are you using MSE loss and not cross-entropy loss for classification?


### Significance
While the paper is addressing an important problem and achieves promising results on CelebA and BAR, more details and experiments are needed for publication.

**References**

[1] Lee et al. Learning debiased representation via disentangled feature augmentation.

[2] Xiao et al. Noise or Signal: The Role of Image Backgrounds in Object Recognition

[3] Shah et al. The Pitfalls of Simplicity Bias in Neural Networks

[4] Pagliardini et al. Agree to Disagree: Diversity through Disagreement for Better Transferability

[5] Sagawa et al. Distributionally Robust Neural Networks for Group Shifts

questions: I have listed by questions and concerns above in the "Strengths And Weaknesses". Summary:

1. Please provide the results for the ColorMNIST experiments with the MLP model (currently Roadblocks is using ResNet-18 while prior works are using MLP).

2. It would be helpful to see results on other benchmarks, especially a larger scale problem like ImageNet-9, as well as Dominoes (MNIST-CIFAR, MNIST-FashionMNIST, MNIST-MNIST), Waterbirds, CelebA [hair color, gender].

3. How are hyper-parameters chosen for the models (loss coefficients, weight decays, learning rates, hyper-parameters of alternate training, etc)? Is the validation split in-distribution or OOD (bias-conflicting)?

4. Please provide more details on the alternate training of the models.

5. It is not clear how the final predictions are made with the proposed model: is only EM model used?

6. Equation (3): why are you using MSE loss and not cross-entropy loss for classification?
limitations: Yes
review_title: null

Meta-review: metareview: The submission describes a new method to avoid the shortcut learning behaviour in DNNs. After the rebuttal and discussion, most of the reviewers are positive about this submission since the proposed method does not require prior knowledge about the dataset and the strong empirical results for the debasing task. On the negative results, the reviewers argue that the experimental evaluation is not thorough enough. Overall, AC recommends acceptance but asks the authors to perform more rigorous evaluation for the camera-ready version including the fair tuning of the hyperparameters.
confidence: Less certain
Decision: Accept