Roadblocks for Temporarily Disabling Shortcuts and Learning New Knowledge
Authors: Hongjing Niu
Hanting Li
Feng Zhao
Bin Li
Venue: NeurIPS 2022 Accept (2022)

Abstract: Deep learning models have been found with a tendency of relying on shortcuts, i.e., decision rules that perform well on standard benchmarks but fail when transferred to more challenging testing conditions. Such reliance may hinder deep learning models from learning other task-related features and seriously affect their performance and robustness. Although recent studies have shown some characteristics of shortcuts, there are few investigations on how to help the deep learning models to solve shortcut problems. This paper proposes a framework to address this issue by setting up roadblocks on shortcuts. Specifically, roadblocks are placed when the model is urged to learn to complete a gently modified task to ensure that the learned knowledge, including shortcuts, is insufficient the complete the task. Therefore, the model trained on the modified task will no longer over-rely on shortcuts. Extensive experiments demonstrate that the proposed framework significantly improves the training of networks on both synthetic and real-world datasets in terms of both classification accuracy and feature diversity. Moreover, the visualization results show that the mechanism behind the proposed our method is consistent with our expectations. In summary, our approach can effectively disable the shortcuts and thus learn more robust features.

number of features that should Roadblocks for Temporarily Disabling Shortcuts and Learning New Knowledge
Hongjing Niu Hanting Li Feng Zhao" Bin Li University of Science and Technology of China, Hefei, China
(sasori, ab828658) @mail. ustc. edu. en, (fzhao956, binli)@ustc. edu. cn
Abstract
Deep learning models have been found with a tendency of relying on shortcuts, i.e., decision rules that perform well on standard benchmarks but fail when transferred to more challenging testing conditions. Such reliance may hinder deep learning models from learning other task-related features and seriously affect their perfor- mance and robustness. Although recent studies have shown some characteristics shortcuts, there are few investigations on how to help the deep learning models to solve shortcut problems. This paper proposes a framework to address this issue by shorteuts. Specifically, roadblocks
setting up roadblocks model is urged to learn to complete gently modified task to ensure that the Icarned knowledge, including shortcuts, is insufficient the complete the task. Therefore, the model trained on the modified task will no longer over-rely on shortcuts. Extensive experiments demonstrate that the proposed framework significantly improves the training of networks on both synthetic and real-world datasets in terms of both classification accuracy and feature diversity. Moreover, the visualization results show that the mechanism behind the proposed our method is consistent with our expectations. In summary, our approach can effectively disable the shortcuts and thus learn more robust features.
placed when the
Introduction
Deep learning has triggered the current rise of artificial intelligence and is the workhorse of today's machine intelligence [13. One of the great advantages of deep learning is its ability to automatically extract the required features according to the task. Moreover, the massive amount of data that drive the model is the key to its success. Regrettably, deep learning methods can hardly take full advantage of the training data. An important reason its over-reliance on simple features. Recently, deep learning approaches have been found to have a tendency to learn simple features only 4]. The learned simple features may be sufficient for the accessible data (training data), and the model will have difficulty in learning other complex but effective features.
Insufficient learning of effective features may lower the accuracy and generalization performance. More severely, when the sample distribution changes and the learned simple features are no longer valid, deep learning models may collapse since they do not learn other features. Therefore, suppressing the reliance on simple features is indispensable for designing models.
Shorteuts in deep learning described as the decision rules that perform well standard bench- marks but fail to be transferred to more challenging testing scenarios according to 44. We make a more specific explanation for easier evaluation. For task, there be considered. Since the training set is limited, perhaps only subset of features is sufficient to make decision. Decision rules that rely easy-to-leand subset of features are called shortcuts. The
"Corresooning author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Accesible:
Inaccessible: OA
Sum-up
coessible:
Inaccessible: OA
Spm-up
Decision
Decision
A
4
Dedision
Decision rule B
A
(a) Functionally similar decision rules.
(b) Functionally diverse decision rules.
Figure 1: Shortcuts and exhaustive decision rules.
concept of shortcuts has two key points: (1) only use a subset of effective features but perform well on the training set; (2) easy learn and affect the learning of other complex but effective features.
Figure provides an intuitive illustration of what shortcuts are and why shortcuts affect the general- ization of deep learning models. Each sample in Figure has two features (color, shape). The training set only has red circles and green triangles, while the testing set contains the other two combinations. For the training set, decision rules are valid whether they are based on color shape, but both of them may fail on the testing set because the training set does not articulate the intention. A task may require the selection of particular decision rule sum-up. But either way, the diversity of decision rules is a foundation.
Some studies expand the difference between decision rules at the feature level [00. However, differentiated decision rules may still be functionally similar (see Ta). To obtain the functionally diverse decision rules shown in Figure lb we propose to temporarily disable the shortcuts.
is important note that shortcuts are disabled temporarily, not forever. We believe that simplicity does not mean bad, but more decision rules might help. Our approach aims to simply provide more options. The integration of decision rules should depend on the task scenarios, and we also provide some suggestions.
To address the issue that deep learning methods do not learn anything other than shortcuts, we propose roadblocks to temporarily disable the shortcuts and force the model to learn new knowledge. The ability of roadblocks that disables the shortcuts achieved by reconstructing datasets under the guidance of trained models. A reconstructed sample should retain most of the attributes of the original sample while disabling the learned attributes. Thus, the model trained with reconstructed samples will naturally reduce the reliance on shortcuts and learn new knowledge.
Enlarging the distance directly [10] at the feature level often only learns the decision rules shown in Figure Ta but is difficult to learn the functional differences. We believe the reason the redundancy and weak interpretability of deep learning models. There is currently the functional differences of the models well. Most previous researches on shortcuts focus on what features are more likely to be shorteuts, and the corresponding solutions mostly rely the prior knowledge of shortcuts (5 g [8]. Priori-based methods require accurate prior knowledge and targeted design, which are difficult to accomplish in most scenarios. Unlike them, our roadblocks do not use any prior knowledge about shortcuts and can deal with different forms of shortcuts.
metrie that measure
In summary, the contributions of this work are mainly three-fold:
We propose new framework to help models learn more diverse knowledge. Compared with those methods based on distance at the feature level, our approach can learn brand-new knowledge and obtain functionally diverse decision rules.
We design a new strategy for disabling shortcuts, which requires no priors and is suitable for wider range of application scenarios.
Our method can effectively improve the generalization ability of the model and can reach sice-offthherrtt in the debiasing task.

---

Raw
Input
BM
Prediction
Roadblock
Loss
Tensor
Recons.
AE
Sum-uy up
Prediction
Blocked Input
Model
EM
Prediction
ccuracy
Figure Framework roadblocks. The basic framework of roadblocks consists three sub-models: autoencoder (AE) in green, blocked model (BM) in red, and explorer model (EM) in blue. The purple model represents the sum-up of BM and EM.
2
Method
This section will introduce our strategy to address shorteuts problem, which is temporarily disabling the shortcuts and learning more diverse knowledge. First, we specify some definitions and the evaluation system. Then, we introduce our roadblocks framework. Finally, the working process of the framework will be detailed.
2.1 Shortcut Suppression Task
Shortcuts described as decision rules that perform well on andepeendentt-idddtiil--iiiiiiii (i.i.d.) testing sets but fail to generalize to out-of-distribution (o.o.d.) testing sets [4]. We refer to this idea and describe the shorteut more precisely, shorteuts decision rules that rely easy-to-learn subset of features.
For classification tasks, suppose distribution D whose feature space be represented by F. Supposing that a training set Dtrasn is sampled from D, and there exist a decision rule that based on Ftrain that can perform well on Dtrain- Ftrain is subspace of F. Such decision rule has the potential risk failing in other datasets sampled from because ignores other features in F.
For ease of measurement, we refer to the setting of the debiasing task. The feature space contains two main features, J [Pr, Fa.. the training data, the target feature Fr corresponds to the ground truth perfectly; while biased feature F/ corresponds Nonetheless, there is still a possibility that Firain is closer to Fg because FB simpler, which means that the model may not have learned Fr expected. To verify the learning of Fr in the testing data, Fr still corresponds to the ground truth perfectly, and F3 is no longer helpful.
the ground truth with a large probability.
2.2 Framework of Roadblocks
Assuming that existing models use shortcuts, a strategy to find more reasonable decision rules is to the idea of roadblocks that forces the model not the same decision rule any previous model.
Figure iillustrates the framework of our approach. The overall model is composed of three parts. Blocked models are trained models that perform well on the i.i.d. test dataset. The autoencoder is used to modify the original input, thereby limiting the performance of blocked models. Once the blocked models perform poorly on the modified input, the explorer models trained with the modified input will find new ways achieve competitive performance. None of the three sub-mu mooooooees need specific structure, as long as they have sufficient representation capabilities.
Let AE, BM, EM denote autoencoder, blocked model, explorer model, respectively. BM is a pre- trained model with good performance, which is fixed during the training process. receives the raw input Lraw and modifies it into blocked input Irec- The mean square error (MSE) is used to measure

---

the similarity between them,
LosSyec (Faw Freo).
(1)
The reconstructed input needs meet two requirements, (1) is difficult to distinguish with the existing decision rules, and (2) it can be correctly distinguished.
Tee will first be used the input of BM. The MSE of the prediction (Tre)) and the pseudo label labelep will be used as the roadblock loss,
Lossyb MSE (BM()) labelee))
(2)
The pseudo label labelep is set to consider all possible categories with equal probability, which is labeley = 1/2 n. ... 4)), where is the number of categories. In this way, the purpose of AE is to generate ree confuse BM.
Trec will also be used as the input of EM. order for the EM to achieve better classification performance, the accuracy loss between the prediction EM and the ground-truth label is calculated for training,
Lossac ASS((EM(Tee)))) label
2.3 Working Process
Core training process BM a pre-trained model, and does not participate in the training process of the proposed framework. Both AE and EM need to be trained. We alternately train them for more stable training process. typical training cycle consists of two parts, (1) AE trained with roadblock loss and reconstruction loss; (2) EM is trained under the guidance of accuracy loss,
AE = arg min min Toss,, + Loss8ees))
(4)
PAE
OFM
arg Los
(5)
DEM
Here, OAE and (EM denote the parameters of AE and EM respectively, and X Dybs drees Aeel are the corresponding loss weights.
Multi-model BM can be any model that performs well on training set. In addition to vanilla models, a EM generated by roadblocks method can also be used as BM to train a new EM. BM can also be set of models BM [BM,, ..., BM,). Correspondingly, the roadblock loss needs to be adjusted to
L0s8ph - (SE(BM(((rnn) labella)
(6)
We suggest adding the well-trained EM to the set of BM and then training the new EM. When the new EM fails to perform well the training set, it means that it is difficult learn more effective features, at which point the loop terminates. The experiments suggest that in most cases, two models are sufficient. Therefore, reduce the computational cost, we usually use two models.
Sum-up The proposed framework contains multiple sub-models that learn different effective features respectively. We do not recommend discarding any features directly, the of learning and the effectiveness of features are not necessarily related. We recommend designing according to specific scenarios. Without priors the test scene, we recommend using the feature fusion strategy because it is more likely to have good generalization. Another idea is to vote, and o.o.d. samples can be detected if there is significant difference in the votes of the submodels, For some tasks with priors, the appropriate model can be directly selected. For example, for debiasing tasks, a reasonable idea to choose the model that performs better on the training because the biased features are less matched with the ground truth.
3 Experiments
To verify whether functionally different decision rules are learned, we choose datasets with multiple
attribute labels for experiments. The experimental setup refers to the design of the debiasing task
[5]. Each individual attribute the training set corresponds well the ground truth, while only one attribute matches the ground truth one-to-one in the testing set. Taking Figure as example,

---

Table I: Image classification accuracy evaluated on CMNIST datasets. The ratio represents the proportion of minority samples, and the smaller the ratio is, the more difficult it is to debias. The best results are highlighted in bold, while the second-bes beott results are denoted with underlines. A check mark (v) indicates that no priors about the feature are used, while a cross mark (x) means the
opposite. The results of previous methods are from [5].
Vanilla 7
EnD 229
ReBias [ LFF 23
LDD [5
Ours
Dataset
Ratio (%)
X
0.5
35.1943.40
34.28 20
70.47=1.84
52.50 43
63. 055.2244.1
66.64
1.0
32.09+2.0
49 29.50
87.40
61.8944.97
81.73
82.04
CMNIST
2.0
65.8643.59
68.1542.16
92.91 10.15
71.03
20.99
84.93 -1.3
5.0
82.1710.74
81.15+1.43
96.96_0.04
80.57
89.66 +1.09
88. 68.65_2099
Table 2: Debiasing performance CelebA. "Unbiased" means that all combinations appear at the same frequency, and "Bias-conflictin"" indicates that only the minority of combinations appear. The
i results are from [15 and 3 results are from 29
Vanilla
EnD
29
ReBius
[
LFF
LDD
Dataset
Testing set
X
CelebA
Unbiased
62.00
0.00
75.93
69.33
66.20 1.21
64.82+1.36
#9.73
Bias-conflicting
33.75 1028
53.70
1.24
44.20g7700
45,4814.3
47.63
64.06.0.78
BAR
Bias-conflicting
49.91 =0.53
52.16
20.18
53.25 20.63
58.81 42.64
63.50 1.47
69.51
the training set is mainly red circles and green triangles, and the four combinations have the same proportion in the testing set. We use the synthetic dataset CMNIST [15). which adds a second
attribute by coloring MNIST [14]. We also use real-world datasets CelebA (20), and BAR (23) that were validated to have shortcuts to test the practicality of our method. Details of datasets and implementation are described in the appendix. Extensive experiments have been conducted
to demonstrate the following effects of roadblocks: 1) greatly enhance the ability to explore new knowledge. 2) significantly improve the generalization of deep learning models.
3.1 Exploration Knowledge
To verify whether roadblocks can reduce models' reliance on shorteuts, compared the performance of the conventionally trained CNNs (Vanilla) and some representative debiasing methods and our roadblocks method. We first test on controlled datasets CMNIST to facilitate analysis. The experimental results are shown in Table
For CMNIST, each sample has two key features, color, and digit. Although the colors are slightly less matched with ground truth, deep learning models tend learn colors. Our method can help deep learning models learn more decision rules that utilize digit features.
Models also often rely shortcuts complete their tasks in real-world scenarios. We conducted experiments on the face attribute dataset CelebA (20 and the action recognition dataset BAR (23). For real-world datasets, we keep its native ratio to maintain the authenticity during the training process. And then we test whether the model learned more knowledge through testing sets of different distributions. For CelebA, following the experimental setup of LFF [23], we trained models to classify 'HeavyMakeup' Then we select the same number of samples from all possible combinations of [Gender, Makeul to form the unbiased testing set and minority combinations to form the bias- bias-confticting testing set. BAR contains six actions, and the backgrounds of the training and testing sets are significantly different. The testing set has only bias-conflict samples. The results are shown in Table 2 It can be seen that our method has obvious performance improvement compared to the tfate-of-the-ate methods. In particular, for the bias-conflict dataset which more difficult, our method achieves more than 10% improvement.
5

---

0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
2
9
0
1
2
3
4
5
6
7
8
7
0
1
2
3
4
5
6
7
8 3
9 7
0
2
3
1
5
6
0
2
3
4
5
4
8
7
(a) CMNIST
(b) CelebA and BAR
Figure 3: Visualization of reconstructed images. For (a) CMNIST, the top 3 rows shows some of the original samples, and bottom rows depict the modified samples generated during the training process. For (b) CelebA and BAR, the left, middle, and right are original images, reconstructed images, and their differences. Results are picked randomly, more results can be seen the Appendix.
3.2 Disabling Shorteuts
The above experiments verified the effectiveness of our roadblocks framework in learning functionally diverse decision rules. Next, it is necessary to confirm whether the effective mechanism of the proposed method is consistent with the original intention of the design. Specifically, it is necessary to verify that some information in the reconstructed image is indeed corrupted so that the existing decision rules cannot make correct predictions. Thus, we visualized reconstructed images, and the result is shown in Figure 3 It can be observed that for CMNIST, our method can implement the disabling of the shortcut (color) by repainting digits. For different digits, the strategy of repainting is consistent, which shows that only the color attribute is disabled in the painting stage, while the number attribute is retained. Therefore, it is still feasible to use these samples to learn numbers, which is also well illustrated in [able
For real-world datasets, similar modification methods are challenging achieve, and our approach mainly implements the destruction by adding noise. For CelebA, AE tries to blur the gender attribute by some blurring (mainly focused on the face contour). For BAR, the blurring of AE mainly revolves around the environment. For the convenience of observation, we show the difference map of the original image and the reconstructed image, which is five times the absolute value of the difference of each pixel value between the two.
We also used I-SNE (30 visualize the features extracted by the vanilla model and our approach. and the result is shown in Figure 4 Each colored number represents sample. It is evident that the vanilla model does an excellent job of partitioning the dataset according to color. However, closer look reveals that samples of the same color, regardless of the digit, are in the same cluster. In contrast, the model trained by our approach can distinguish digits and have certain invariance to color. Distinguishing digits is more complex than color, so getting sharp boundaries is more complicated.
To visually demonstrate the model's 's ability to explore new knowledge, we use Grad-CAM 26) to visualize the focus of models on the BAR dataset, and saliency maps are shown in Figure In this experiment, we use ResNet- 18 as the backbone, and the last fully connected layer is modified to linear layer (512, 10) to match the number of categories. And the last convolutional layer is used as the target layer of -rad-CAM.. For CMNIST, map because the colors and shapes correspond to the same regions. This is also the reason why we do not disable shortcuts using saliency map based methods. For CelebA and BAR, can be seen that the attention regions of our approach are significantly different from that of the vanilla model. For correctly classified samples, our approach can find more effective information. Grad- MMdd--AM&& also provides some explanation for samples that vanilla misclassifies, but our method classifies correctly. For example, for samples of mountain climbing, the EM of our approach focuses on the climber, and the vanilla model focuses on the mountain. Both of these features are important, and our approach has better generalization ability because utilizes more effective features. The samples of CMNIST and CelebA were randomly selected, and we selected more representative samples for BAR. There
difficult tell the difference from the saliency

---

18 334
(a) Vanilla
(b) Ours (EM)
(c) Zoom-in
Figure 4: The t-SNE results CMNIST. It reflects the distance between the features extracted by the model to some extent. Each colored number represents a sample. (a) corresponds to the vanilla model, (b) corresponds to model, and (c) provides zoom-in view of the boxed areas.
Figure 5: Saliency maps. The top and bottom rows show the Grad-CAM results of vanilla model and EM of our approach, respectively. Warm colors indicate a high level of attention. Green borders mean correct classification, while red borders imply incorrect classification.
are more examples of samples in the Appendix to facilitate more analysis and discussion, while also ensuring generality.
Grad-CAM an excellent interpretability method, but can only show differences in models at spatial scale. Therefore, for datasets such CMNIST, where different features overlap lot in spatial scales, saliency maps cannot well reflect the differences between models.
The reconstruction results and saliency maps visually demonstrate that our approach does learn more effective features following the ability of our method to improve model diversity as fully as possible, we further quantify the model diversity.
our design intention. These methods still have some limitations. To verify
3.3
Model Diversity
Our core purpose to learn more knowledge (i.e., effective features). Therefore, is necessary to choose appropriate metrics to measure the similarity between models. We applied SVCCA [24] to measure the similarity the models. SVCCA is tool for comparing two representations way that is invariant to affine transform. It receives the feature maps of middle layers and calculates the similarity between each pair of feature maps. Figure 6 shows some SVCCA results between the models. Figure 6a and Figure 6b are caleulated on CMNIST (5pct), Figure 6c and Figure 6d

---

are calculated on BAR. CelebA is too large to be calculated, so we did not perform corresponding experiments.
(a) CMNIST, V-V
(b) CMNIST, V-O
(c) BAR, V-V
(d) BAR, V-O
Figure 6: The SVCCA similarities. Bright tones indicate higher similarity. 'V" and 'O' denote vanilla and ours respectively. For example, (a) represents the similarities between layers of two vanilla models on the CMNIST dataset.
For the CMNIST dataset, the diagonal of the SVCCA matrix of the vanilla model is bright, indicating a high similarity between the same layers. While the similarity between our model and the vanilla model is slight, there are only a few brighter regions between several lower layers. The comparison of the two shows that the model trained by our method is significantly different from the vanilla model at the feature level. Therefore, our method helps to improve the diversity of models. It can also be observed that some adjacent layers also have high similarities. The BAR dataset is more complex than the CMNIST dataset. Therefore, the similarity between adjacent layers in the vanilla model is lower. Some layers of our model and the corresponding layers vanilla model are similar, which we think may correspond some complex semantic features.
Related Work and Discussion
4.1 Related Work
Shortcut learning. Geirhos et al. 4 provided comprehensive summary of shortcuts and defined them as decision rules that perform poorly on the o.o.d. testing set. A considerable amount of research has been devoted to analyzing the formation of shortcuts, providing many valuable conclusions Deep learning models is found to have the tendency to use textures rather than shapes make predictions (5). which has inspired a new round of thinking among researchers the mechanism of deep learning. Katherine and Andrew (9) explored the conditions under which features are more likely be shortcuts, which very helpful deepen our understanding of shortcuts. In addition, various data augmentation methods are used to drive models to learn features beyond known shortcuts. Some researches [5] 18 assumed that the texture feature is the only shortcut, and the designed data augmentation techniques effectively reduce the model's ' reliance on texture. There are also ways to exclude shorteuts by introducing a prior on the difficulty of the target task (3]. However, research on suppressing shortcut without relying on prior knowledge is still basically an unknown area to explore.
27 [2].
Fairness and debiasing. The debiasing task can be understood as solving the problem that the model has learned certain bias attribute in the data set, but failed to learn the relatively difficult target attribute. Some researchers have explored the preferences of deep learning methods themselves and dealt with them accordingly, such as the tendency use textures rather than shapes
[5] and the (2). Besides, several existing approaches aimed
tendency to learn low-free www-frequency features in images
mitigating the bias also assume certain bias types 12 17 29 25). Instead of defining specific types of bias, some approaches [23] [5
19 16 rely on the straightforward assumption that simple features are biased features. typical example is Learn from Failure. The main idea twofold; (a) intentionally train the first network to be biased by repeatedly amplifying its "prejudice", and (b) debias the training of the second network by focusing on samples that go against the prejudice the biased network in (a). Unlike them, our approach does not use biased priors, but solves the debiasing problem by learning more knowledge. Compared with other debiasing methods, has a wider range of application scenarios.
8

---

Adversarial training The proposed roadblocks share some similarities with adversarial training. Adversarial examples obtained by adding small designed perturbations can make the model go wrong (88] Adversarial training [1]] can effectively improve the adversarial robustness of the model. Our approach and adversarial training clearly differ in purpose, which also introduces design differences. Adversarial examples only require the model to predict incorrectly, whereas our method requires the model to be completely confused. Therefore, by design, the false labels we use are all categories, including the true category, with equal probability. We also conduct experiments using adversarial training, and the results are shown the Appendix.
4.2 Discussion
Functionally identical features may have different representations. Increasing the diversity of features, in other words, learning more knowledge, a natural idea, but its implementation is not as simple. An intuitive way to learn different features is to expand the distance between the two models at the feature level. We refer to the framework of negative correlation learning and expand the distance between models at the feature level much possible on the premise of ensuring consistent prediction results. Experimental results (see the Appendix) show that is difficult to learn decision rules other than shortcuts through such learning paradigm. We believe that the reason is that the excellent representational ability of the deep learning model allows to represent functionally identical features in different ways. Therefore, simpler and more effective approach is our roadblocks, which forces the model to learn new features by temporarily disabling the already learned features by modifying the input. Rather than some distance between features, remind that functional differences between features may be more important.
From comprehensiveness to exactness. Our method is designed for general image classification tasks. The priors about the features and the distribution difference between training and testing sets are unknown. On the one hand, comprehensiveness (i.e., maximizing the use of effective features in the accessible data) a reasonable goal since such models are the most robust against complex test scenarios. For comprchensiveness, our approach allows the model to temporarily disable shortcuts and learn new effective features as much as possible. Its effectiveness and benefits have been described above. On the other hand, exactness (i.e., only using some specific features) is more realistic goal for many tasks. Our approach also provides a new implementation for feature screening. By first training model to learn features that not required for the task and then restricting the use of such features, it will be easier for the model to follow the designer's s intention. Achieving the requirement for exactness the lowest cost will be key point for the designer.
5 Conclusion
In this work, we develop a novel framework to prevent deep learning models from merely using shorteuts (simple decision rules) while ignoring other effective features. Experimental results on multi-attin aulti--ttribute datasets in complex testing scenarios demonstrate that our roadblocks have a strong capacity of reducing reliance on shortcuts and learning new knowledge, which is very difficult for conventional deep learning methods and debiasing models. Extensive experiments also show that roadblocks achieve excellent performance model diversity. We expect our roadblocks can help shed light on the nature of shortcuts learning in neural networks, and provide a feasible paradigm for training models without depending on shortcuts.
terms of extracting new knowledge and improving
Acknowledgments
This work was supported by the Anhui Provincial Natural Science Foundation under Grant 2108085UD12, and in part by the National Natural Science Foundation of China under Grants U19B2044 and 61836011. We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.

---

References
[1] Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 528-539, 2020.
[2] Guangyao Chen, Peixi Peng. Li Ma, Jia Li, Lin Du, and Yonghong Tian. Amplitude-phan recombination: Rethinking robustness of convolutional neural networks in frequency domain. In IEEE/CVF International Conference on Computer Vision, pages 448-457, 2021.
[3] Nikolay Dagaev, Brett D Roads, Xiaoliang Luo, Daniel N Barry, Kaustubh R Patil, and Bradley C Love. A too- 1o0-good-to pood-to--be-tue prior to reduce shorteut reliance. arXiv preprint Stt1:22022064000 2021.
[4] Robert Geirhos, förn-Henris Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 (1)6556667,, 2020.
[5] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann. and Wieland Brendel. mmagenet-trrineee CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
[6] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv: 142.6657. 2014.
[7] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[8] Katherine Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 19000-19015, 2020.
[9] Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9995-10006, 2020.
[10] Zhen Huang, Xu Shen, Jun Xing, Tongliang Liu, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian ShannSheng Hua. Revisiting knowledge distillation: An inheritance and exploration framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3579-3588, 2021.
[11] Andrew Ilya,, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in Neural Information Processing Systems, 32, 2019.
[12] Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to Training deep neural networks with biased data, In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9004-9012, 2019.
learn:
[13] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
[14] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE,
1998.
(15) Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debiased representation via disentangled feature augmentation. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. [16] Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from underspecified data. arXiv preprint arXiv:2202.03411 2022.
[17] Yi Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. [18] Yingwei Li, Qihang Yu, Mingxing Tan, Jieru Mei, Peng Tang, Wei Shen, Alan Yuille, and cihang xie. Shape-texture debiased neural network training. In International Conference on Learning Representations, 2021.
[19] Evan Z. Liu, Behzad Haghgoo, Annie Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without
10

---

training group information. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6781-6792 2021.
[20] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision, 2015.
[21] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
[22] Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the fail- ure modes of out-o--distribution generalization. In International Conference on Learning Representations, 2020.
[23] Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20673-20684, 2020.
[24] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveea: Singular vee- tor canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, volume 30, 2017.
[25] Shiori Sagawa*, Pang Wei Koh* Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020.
[26] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradeent-based localization. In Proceedings of the IEEE International Conference on Computer Vision, 2017. [27] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In Advances in Neural Information Processing Systems, NIPS'20 20, 2020.
[28] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, 2014.
[29] Enzo Tartaglione, Carlo Alberto Barbano, and Marco Grangetto. End: Entangling and disentan- gling deep representations for bias correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13508-13517, 2021.
[30] Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579-2655, 2008.
[31] Haohan Wang, Zexue He, and Eric P. Xing. Learning robust representations by projecting superficial statistics out. In International Conference on Learning Representations, 2019.

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes) See Section 4.2
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work has no negative social impact in our opinion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Appendix
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Our experimental results are presented in tabular form with mean and standard deviation.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] All the datasets
is public
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "vJHXoD7KF3B", "reviewer": "Reviewer_dvxL", "summary": "A method for OOD generalization is proposed. To prevent a model (Explorer Model) from learning \"shortcut features\" (simple features which do not work in OOD setting~spurious correlations), an autoencoder is trained, which tries to fool a pretrained model (Blocked Model) as an additional loss. The explorer model is then trained on the reconstructed image, from which some of the shortcut features were hopefully removed by AE to fool the Blocked model.", "questions": "- Can we see some samples of CelebA reconstructed by the autoencoder?\n- Can you discuss the relationship/tradeoffs with using a better autoencoder/pretrained autoencoder, which may limit the applicability of this method to harder datasets?", "limitations": "Relies on autoencoder reconstruction, which may be blurry and lower the prediction quality.\n\nThis method seems to implicitly perform some debiasing, which should hopefully have a positive impact in terms of making better predictions. However, this should be more extensively tested on a wide variety of tasks to see if the usual fairness/bias metrics are consistently improved.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Promising framework and approach, which does not require any external knowledge but relies on the interaction between autoencoder and blocked model.\n- Interpretability: can visualize reconstructed images to gain insight (e.g. Figure 3 of CMNIST)\n- Indirectly applicable to debiasing (by removing shortcut features)\n\nWeaknesses:\n- Relies on autoencoder reconstruction, which may be blurry and lower the prediction quality.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UC6yCqOq0OU", "reviewer": "Reviewer_BFAY", "summary": "The paper tackles the problem of neural networks learning ‘simple’ features which cannot generalize well. It proposes to alleviate this problem of shortcut learning by generating synthetic data (“roadblocks”) in a manner which confuses the network while being close (in $l_2$ norm) to the original training data. A new model is trained on such examples, and it is hypothesized that this new model learns more robust features. The paper empirically shows that such models are more robust when there are spurious correlations in the data. The paper also presents visualizations of learnt features, as well as of the generated roadblocks, indicating that these do contribute to learning more robust features.  ", "questions": "1. How important is the distance metric for imposing loss between the generated images and the real images? Can a perceptual loss be used for the same?\n2. How does the method compare against simple adversarial training?\n3. Why was the MSE loss used for classification? Can cross-entropy give better results?", "limitations": "Some of the limitations are listed in the weaknesses above. A more thorough empirical evaluation on other OOD tasks would also be appreciated. \n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths - \n1. The empirical results on the debiasing task are promising. \n2. The method does not need any priors about the dataset in order to work.\n\nWeaknesses - \n1. The work misses several key references. Shortcut learning is very closely related to the problem of simplicity bias in neural networks [1], which is not cited in the work.\n2. The method is also very similar to training with adversarial examples [2]. This line of work is neither cited nor compared against. On the mentioned datasets, the generated images indeed look very similar to $l_2$ or $l_\\inf$ constrained adversarial attacks.\n3. The paper proposes using the MSE loss for classification, which is not standard.\n4. The writing of the paper can be improved significantly.  \n\n\n[1] - Shah, Harshay, et al. \"The pitfalls of simplicity bias in neural networks.\" Advances in Neural Information Processing Systems 33 (2020): 9573-9585.\n[2] - Ilyas, Andrew, et al. \"Adversarial examples are not bugs, they are features.\" Advances in neural information processing systems 32 (2019).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QaKaUVYvaCe", "reviewer": "Reviewer_ius4", "summary": "The paper proposed a learning approach for robustness to shortcut learning: after training a base model (BM) on the biased dataset, the auxiliary autoencoder (AE) is trained to modify train images such that their reconstruction is close to the original input but the trained model  is outputting a uniform distribution over classes. Then another \"exploration\" model (EM) is trained on the modified inputs to predict the correct class. Thus, the intuition for the approach is that AE should remove the shortcuts from the inputs that BM learned, and EM will learn more diverse features. The approach was evaluated on ColorMNIST, CelebA and BAR benchmarks, and the authors provided additional analysis and visualizations of the features.", "questions": "I have listed by questions and concerns above in the \"Strengths And Weaknesses\". Summary:\n\n1. Please provide the results for the ColorMNIST experiments with the MLP model (currently Roadblocks is using ResNet-18 while prior works are using MLP).\n\n2. It would be helpful to see results on other benchmarks, especially a larger scale problem like ImageNet-9, as well as Dominoes (MNIST-CIFAR, MNIST-FashionMNIST, MNIST-MNIST), Waterbirds, CelebA [hair color, gender].\n\n3. How are hyper-parameters chosen for the models (loss coefficients, weight decays, learning rates, hyper-parameters of alternate training, etc)? Is the validation split in-distribution or OOD (bias-conflicting)?\n\n4. Please provide more details on the alternate training of the models.\n\n5. It is not clear how the final predictions are made with the proposed model: is only EM model used?\n\n6. Equation (3): why are you using MSE loss and not cross-entropy loss for classification?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is addressing an important problem of learning diverse features and training models robust to shortcuts, and the proposed approach shows promising results on CelebA and BAR, however, the clarity and presentation can be significantly improved, and the paper is missing implementation and method details. The results on CMNIST are not directly comparable to prior work due to a different model architecture. Also, experiments on large scale datasets would be needed to demonstrate the practicality of the proposed method.\n\n\n### Originality\nThe paper is missing citations of the relevant literature on shortcut learning, simplicity bias, group robustness, and spurious correlations, e.g.:\n\nSagawa et al. Distributionally Robust Neural Networks for Group Shifts\n\nLiu et al. Just Train Twice: Improving Group Robustness without Training Group Information\n\nShah et al. The Pitfalls of Simplicity Bias in Neural Networks\n\nXiao et al. Noise or Signal: The Role of Image Backgrounds in Object Recognition\n\nTeney et al. Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions With Superior OOD Generalization\n\nNagarajan et al. Understanding the Failure Modes of Out-of-Distribution Generalization\n\nDagaev et al. A Too-Good-to-be-True Prior to Reduce Shortcut Reliance\n\nClark et al. Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases\n\nLee et al. Diversify and Disambiguate: Learning From Underspecified Data\n\nand others. To the best of my knowledge, the proposed approach is novel. However, I encourage authors to explicitly discuss the connection of their work with Just Train Twice [Liu et al], Dagaev et al (see above) and LfF as all of these prior works are based on using 2 models where the second model is correcting the biases of the first one similarly to the proposed approach (while it additionally uses an AE to modify the inputs).\n \n\n### Quality\n1. ColorMNIST experiments: In the Appendix A.1 it is mentioned that ColorMNIST dataset is setup as in Lee et al [1]. It is also mentioned that ResNet-18 is used in all experiments. I noticed that the numbers for the Vanilla, EnD, ReBias, LFF, and LDD in Table 1 in this paper are taken from the Table 2 in Lee et al [1] (they match those numbers exactly). However, right below Table 2 in [1] it is mentioned that an MLP architecture is used in the experiments, while this paper uses ResNet-18 so the comparison in Table 1 is not valid.\n(In general, I would also recommend that the authors explicitly say that which papers they took the numbers from for the prior methods in case they haven't re-run the methods.)\n\n2. While the Roadblocks approach works well on CelebA [Heavy Makeup, Gender] and BAR, it would be very helpful to see the performance on a large scale problem, e.g. ImageNet-9 [2]. It would also be interesting to see the results on a broader range of benchmarks in shortcut learning / spurious correlations, e.g. Dominoes (MNIST-CIFAR, MNIST-FashionMNIST, MNIST-MNIST) [3, 4], Waterbirds, CelebA [hair color, gender] [5]. These benchmarks vary in the strength of the spurious correlation / bias in train data and the complexity of the shortcut feature.\n\n3. How are hyper-parameters chosen for the models (lambdas loss coefficients, and also weight decays, learning rates, hyper-parameters of alternate training, etc)? Do authors use in-distribution validation set or bias-conflicting validation? This can have a high impact on the model's performance.\n\n4. I would encourage the authors to provide the reconstructed images like in Figure 3 for other datasets besides CMNIST.\n\n\n### Clarity\nClarity of the paper can be significantly improved. I strongly recommend the authors to add a brief datasets description to the main paper (e.g. the problem description, the nature of the bias / shortcut, the percentage of bias conflicting examples in train, etc). These details are not included in the paper, and some details become unclear without the context e.g. what \"Ratio %\" refers to in the Table 1.\n\nIt is mentioned that AE and EM are trained alternately, however, the details of this training are not described (i.e. are models alternately updated after each step or epoch?).\n\nI strongly encourage the author to add the high-level details on the prior works to which Roadblocks method is compared to either in the Experiments section or in the Related work. In particular, it will be helpful to know how prior works compare to Roadblock method in terms of computational complexity, memory, etc.\n\nSome paragraphs are unclear (e.g. lines 134-139, 172-176, 288-301) are not clearly written, lack details and context.\n\nI also strongly encourage authors to add more details in Table and Figure captions in the next revision of the paper (e.g. Figure 6 doesn't have titles for subplots to indicate which dataset they correspond to).\n\nIt is not clear how the final predictions are made with the proposed model: is only EM model used? It is not clearly explained in the paper.\n\nLine 47: \"shortcuts are disabled temporarily, not forever\", what does this mean? My understanding is that EM is trained on modified images outputted by AE so it is not clear what \"temporarily, not forever\" refers to.\n\nEquation (3): why are you using MSE loss and not cross-entropy loss for classification?\n\n\n### Significance\nWhile the paper is addressing an important problem and achieves promising results on CelebA and BAR, more details and experiments are needed for publication.\n\n**References**\n\n[1] Lee et al. Learning debiased representation via disentangled feature augmentation.\n\n[2] Xiao et al. Noise or Signal: The Role of Image Backgrounds in Object Recognition\n\n[3] Shah et al. The Pitfalls of Simplicity Bias in Neural Networks\n\n[4] Pagliardini et al. Agree to Disagree: Diversity through Disagreement for Better Transferability\n\n[5] Sagawa et al. Distributionally Robust Neural Networks for Group Shifts\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "G6mZH01bCnE", "reviewer": "Reviewer_T7g9", "summary": "The authors propose an approach to address the problem of shortcut vulnerability of deep vision models in image classification tasks. For a trained, shortcut exploiting model (BM) the authors propose to train a model (AE) that learns to modify training samples in a minimal way while yielding a maximum entropy prediction for the BM model. Using these manipulated samples, an explorer model is trained (EM) in a supervised manner to predict the original ground-truth labels. The intuition behind the approach is that AE learns to manipulate input samples specifically to remove the shortcut information in the image in order to enforce the explorer model to learn new, actual class information and is hence less shortcut vulnerable. \nEmpirical evidence is provided for the CMNIST, CelebA and BAR datasets indicating the improved classification generalization. Further, the authors provide qualitative evaluation in form of an inspection of the manipulated images, as well as saliency evaluations comparing BM and EM which suggest that the AE model in fact removed the shortcut information and that the EM model attends to more meaningful image areas than BM. ", "questions": "See weaknesses above.", "limitations": "See weaknesses above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In a nutshell, the work address a major, central problem of current ML The proposed solution to the problem is original and some interesting empirical results are provided. Still, I think some limitations / questions should be addressed more explicitly (see below). I would like to increase my rating in case stated limitations, questions (see below here under weaknesses) get addressed sufficiently.\n\nStrengths\n------------\nThe work addresses the problem of shortcut vulnerability of ML models in a supervised learning setup. This is of immense significans for the entire ML community.  As far as I can tell, the authors propose a novel, original approach to tackle this challenge in the effort to traina more shortcut robust model. The authors present their central idea in a clear, easy to access manner and present some quantitative evidence that supports the central claims of the paper (Tab 1, 2). Also of value are the provided qualiatitve evaluations indicating that the modification model (AE) works as intended on the investigated dataset CMNIST (Fig. 3) and a saliency comparison for BM and EM (Fig 5) on the BAR dataset, indicating that EM attends to more meaningful image areas than BM.\n\nWeaknesses\n----------------\nMajor 1\n\nIt is not clear to me what the AE actually learns and in which cases it actually learns to remove shortcut information. I think the provided intuition how AE should work is sound and the provided results support this intuition, however question about how AE actually works and how well it generalizes to other cases remain unclear and should be addressed:\n* Why does AE not learn to produce (untargeted) adversarial noise? This would have the same effect as removing the shortcut, i.e., induce minimal change in the image and yield uniform classification. Please address in the text.\n* Why does AE not learn to remove actual class information? This is especially relevant in case of low correlations (ratio) between the shortcut and the actual target class in the training dataset. In those cases, AE would probably learn to also remove the actual class information. Please discuss this point.\n* More complicated shortcut structures might exist (shortcuts of shortcuts of shortcuts of ...). For this, see also [1] which should also be cited in intro / related work. This point seems partially addressed in L126-L133 but could be made more clear.\n* Dependent on the design of the AE model, only rather specific types of shortcuts might be addressed with the used AE model. Also see [1] for more shortcut examples. For example, it is not clear that AE would be able to address the texture bias on ImageNet. Please also address why ImageNet was not selected as a dataset.\n* To improve intuition about how AE works, please also provided AE modified samples for the BAR and CelebA dataset (similar to Fig 3 for CMNIST) and also provide saliencies for CMNIST and BAR (similar to Fig 5). Either Fig 3, 5 could be extended or additional images can be shown in the supplementary material.\n* How cherry-picked are samples in Fig 3 and 5. Information in the text is missing.\n\n[1] \"DiagViB-6: A Diagnostic Benchmark Suite for Vision Models in the Presence of Shortcut and Generalization Opportunities\", ICCV 2021\n\nMajor 2\n\nCode should be made available to enable reproducibility. I could not find a code publishing statement in the paper.\n\n\nMedium\n\n* The role of the sum-up combination is not clear. Is this used somewhere in the experiments or is this merely a proposal for an open-set-recognition task? If of no practical relevance for this work it could be left out entirely.\n* It should be noted somewhere that this training proceedure, similar to all data augmentation approaches, induces certain invariances in the EM model which might lead to unintended behavior on outliers.\n \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept