Understanding Rare Spurious Correlations in Neural Networks
Authors: Yao-Yuan Yang
Chi-Ning Chou
Kamalika Chaudhuri
Venue: Submitted to ICLR 2023 (2023)
Keywords: spurious correlation
trustworthy machine learning

Abstract: Neural networks are known to use spurious correlations such as background information for classification. While prior work has looked at spurious correlations that are widespread in the training data, in this work, we investigate how sensitive neural networks are to $rare$ spurious correlations, which may be harder to detect and correct, and may lead to privacy leaks. We introduce spurious patterns correlated with a fixed class to a few training examples and find that it takes only a handful of such examples for the network to learn the correlation. Furthermore, these rare spurious correlations also impact accuracy and privacy. We empirically and theoretically analyze different factors involved in rare spurious correlations and propose mitigation methods accordingly. Specifically, we observe that $\ell_2$ regularization and adding Gaussian noise to inputs can reduce the undesirable effects.

Under review as a conference paper at ICLR 2023
UNDERSTANDING RARE SPURIOUS CORRELATIONS IN NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Neural networks are known to use spurious correlations such as background in- formation for classification. While prior work has looked at spurious correlations that are widespread in the training data, in this work, we investigate how sensitive neural networks are rare spurious correlations, which may be harder to detect and correct, and may lead to privacy leaks. We introduce spurious patterns correlated with fixed class few training examples and find that it takes only a handful of such examples for the network to learn the correlation. Furthermore, these rare spurious correlations also impact accuracy and privacy. We empirically and theoret- ically analyze different factors involved in rare spurious correlations and propose mitigation methods accordingly. Specifically, we observe that &2 regularization and adding Gaussian noise to inputs can reduce the undesirable effects.
INTRODUCTION
Neural networks are known to use spurious patterns for classification. Image classifiers use back- ground feature to classify objects (Gururangan et al., 2018; Sagawa al., 2020; Srivastava et al., 2020; Zhou et al., 2021) often to the detriment of generalization (Nagarajan et al., 2020). For example, Sagawa al. (2020) show that models trained the Waterbirds datasetcorrelate waterbirds with backgrounds containing water, and models trained on the CelebA dataset Liu et al. (2018) correlate males with dark hair. In all these cases, spurious patterns are present in a substantial number of training points. The vast majority of waterbirds, for example, are photographed next to the water.
Understanding how and when spurious correlations appear in neural networks is frontier research problem and remains elusive. In this paper. we study spurious correlations in the context where the appearance of spurious patterns is rare in the training data. Our motivations are three-fold. First, while it is reasonable to expect that widespread spurious correlations in the training data will be learnt, a related question is what happens when these correlations are rare. Understanding if and when they are learnt and how to mitigate them is a first and necessary step before we can understand and mitigate spurious correlations more broadly. Second, rare spurious correlation may inspire us to discover new approaches mitigate them as traditional approaches such balancing out groups (Sagawa et al., 2020), subsampling (Idrissi et al., 2021), or data augmentation (Chang et al., 2021) do not apply. Third, rare spurious correlations naturally connect to data privacy. For example, in Leino & Fredrikson (2020), the training set had an image of Tony Blair with a pink background. This led to classifier that assigned a higher likelihood of the label "Tony Blair" to all images with pink backgrounds. Thus, an adversary could exploit this to infer the existence of "Tony Blair" with a pink background in the training set by by presenting images of other labels with pink background.
We systematically investigate rare spurious correlations through the following three research questions. First, when do spurious correlations appear, i.e., how many training points with the spurious pattern would cause noticeable spurious correlations? Next, how do rare spurious correlations affect neural networks? Finally, is there any way to mitigate the undesirable effects of rare spurious correlations?
1.1 OVERVIEW
We attempt answer the above questions via both experimental and theoretical approaches. On the experimental side, we introduce spurious correlations into real image datasets by turning a few training data into spurious examples, i.e., adding spurious pattern to training image from

---

Under review as a conference paper at ICLR 2023
target class. We then train neural network the modified dataset and measure the strength of the correlation between the spurious pattern and the target class in the network. On the theoretical side, we design toy mathematical model that enables quantitative analysis on different factors (e.g.. the fraction of spurious examples, the signal-to-nove ratio, etc.) of rare spurious correlations. Our responses to the three research questions summarized in the following.
Rare spurious correlations appear even when the number of spurious samples is small, Em- pirically, we define a spurious score to measure the amount of spurious correlations. We find that the spurious score of a neural network trained with only 1 spurious examples out of 60,000 training samples can be significantly higher than that of the baseline. A visualization of the trained model also reveals that the network's 's weights may be significantly affected by the spurious pattern. In our theoretical model, we further discover that there a sharp phase transition of spurious correlations from no spurious training example to a non-zero fraction of spurious training examples. Together, these findings provide strong evidence that spurious correlations can be learnt even when the number of spurious samples is extremely small.
Rare spurious correlations affect both the privacy and test accuracy. We analyze the privacy issue of rare spurious correlations via the membership inference attack (Shokri et al., 2017; Yeom et al., 2017), which measures the privacy level according to the hardness of distinguishing training samples from testing samples. We observe that the spurious training examples are more vulnerable to membership inference attacks. That is, it is easy for an adversary to tell whether a spurious sample is from the training set. This apparently raises serious concerns for privacy Leino & Fredrikson (2020) and fairness to small groups Izzo et al. (2021).
We examine the effect of rare spurious correlations on test accuracy through two accuracy notions: the clean test accuracy, which uses the original test examples, and the spurious test accuracy, which adds the spurious pattern all the test examples. Both empirically and theoretically, we find that clean test accuracy does not change too much while the spurious test accuracy significantly drops in the face of rare spurious correlations. This suggests that the undesirable effeet of spurious correlations could be more serious when there is distribution shift toward having more spurious samples.
Methods to mitigate the undesirable effects of rare spurious correlations. Finally, inspired by our theoretical analysis, we examine three regularization methods to reduce the privacy and test accuracy concerns: adding Gaussian noises to the input samples, E regularization (or equivalently, weight decay), and gradient clipping. We find that adding Gaussian noise and &a regularization effectively reduce spurious score and improve spurious test accuracy. Mcanwhile, not all regularization methods could reduce the effects of rare spurious correlations, e.g., gradient clipping. Our findings suggest that rare spurious correlations should be dealt differently from traditional privacy issues. We post it as future research problem to deepen the understanding of how to mitigate rare spurious correlations.
Concluding remarks. The study of spurious correlations is crucial for a better understanding of neural networks. In this work, take step forward by looking into special (but necessary) case of spurious correlations where the appearance of spurious examples is rare. We demonstrate both experimentally and theoretically when and how rare spurious correlations appear and what undesirable consequences are. While we propose a few methods to mitigate rare spurious correlations, we emphasize that there is still a lot to explore, and we believe the study of rare spurious correlations could serve as guide for understanding the more general cases.
PRELIMINARIES
We focus on studying spurious correlations in the image classification context. Here, we briefly introduce the notations and terminologies used in the rest of the paper. Let X' be an input space and let )' be a label space. At the training time, we are given a set of examples {(x,, yi) (e(.....) sampled from distribution Durin. where each x4 E X' is associated with a label y V. At the testing time, we evaluate the network on test examples drawn from a test distribution. We consider two types of test distribution: the clean test distribution Daust and the spurious test distribution Daest Their formal definitions will be mentioned in the related sections.
Spurious correlation. A spurious correlation refers to the relationship between two variables in which they are correlated but not causally related. We build on top of the framework used in Nagarajan et al. (2020) to study spurious correlations. Concretely, the input is modeled as the output of
2

---

Under review as a conference paper at ICLR 2023
feature map 4x from the feature space Xnv Xap to the input space X. Here, Aiy is the invariant feature space containing the features that causally determine the label and Xsp which is the spurious feature space that accommodates spurious features. Finally, Φx : Aay Xep - X is the function that maps an feature pair (Xime, Xsp) to an input x and By Ximv - ) is a function that maps the invariant feature X to a label. Namely, an example (x, y) is generated by (Xinv: Xsp) via
(xxxx Xap) and y = ( Without loss of generality, the zero vector in Xap, i.c. 0 E Xep refers to "no spurious feature" and for any nonzero Xsp we call 4 (Xinw, Xxp) & (Xinv, 0) spurious pattern. We focus on the case of having a fixed spurious feature Xap and leave it as future direction to study the more general scenarios where there are multiple spurious features.
Rare spurious correlation. Following the setting introduced in the previous paragraph, an input distribution ID over X' is induced by a distribution Dresture over the feature space Xin x Xap. i.e., to get sample from D, one first samples (Xim; Xap) from Dieature and outputs (x, with x ( Xsp) and y oy( Xi Now, we are able to formally discuss the rareness of spurious correlations by
defining the spurious frequency of D as 7 Pr(xx.xxx)>DDe (.y) Xsp * 0J.
Two simple models for spurious correlations. In general, 4x could be complicated and makes difficult to detect the appearance of spurious correlations. Here, we consider two simple instantiations of x and demonstrate that undesirable learning outcomes already appear even in these simplified settings. First, the overlapping model (used in Sec. 3) where the spurious feature is put top
of the invariant feature, i.e., X' = Xin = Xy and xxx Ximv; Xsp) = Xinv + Xup or x Xsp) clip(xim Xxp) where clip is function that truncates input pixel when its value exceeds certain range. Second, the concatenate model (used in Sec. 5) where the spurious feature is concatenated to
the invariant feature, i.e., X- -inn Xap and 4 x (Xinv, Xap) = (Xine, Xsp).
RARE SPURIOUS CORRELATIONS ARE LEARNT BY NEURAL NETWORKS
We start with an empirical study of rare spurious correlations in neural networks. We train a neural network using modified training dataset given by the overlapping model where a spurious pattern added to few training examples with the same label (target class). We then analyze the effect of these spurious training examples through three difference angles: (i) quantitative analysis on the appearance of spurious correlations via an empirical measure, spurious score, (ii) a qualitative analysis on the appearance of spurious correlations through visualizing the network weights, and (iii) an analysis on the consequences of rare spurious correlations in terms of privacy and test accuracy.
3.1 INTRODUCING SPURIOUS EXAMPLES TO NETWORKS
As we don't have access to the underlying ground-truth feature of an empirical data, we artificially introduce spurious features into the training dataset. Concretely, given a dataset (e.g., MNIST), we treat each training example x as an invariant feature. Next, we pick a target class Gur (e.g.. the zero class), spurious pattern Xsp (e.g., yellow square at the top-left corner), and a mapping Øx that combines a training example with the spurious pattern. Finally, we randomly select n training examples X1,.... X., from the target class Cur and replace these examples with Φx (Xi, Xxp) for each i = 1,... n. See Fig. I and the following paragraphs for a detailed specification of our experiments.
Datasets & the target class Ctar- We consider three commonly used image datasets: MNIST (LeCun, 1998), Fashion (Xiao et al., 2017), and CIFAR 10 (Krizhevsky & Hinton, 2009). MNIST and Fashion have 60, 000 training examples, and CI has 50, 000. We set the first two classes of each dataset as the target class (Cua = {0. 1}), which are zero and one for MNIST, T-shirt/top, and trouser for Fashion, and airplane and automobile for CIFAR1O. See App. D.I for more experimental details.
Spurious patterns Xxp We consider seven different spurious patterns (Fig. 1) for this study. The patterns small / (SI), small 2 (S2), and small 3 (S3) are designed to test if neural network can learn the correlations between small patterns and the target class. The patterns random / (R/), random 2 (R2), and random 3 (R3) are patterns with each pixel value being uniformly random sampled from [0, r]. where r = 0.25, 0.5, 1.0 (we sample the pattern once and fix it throughout each experiment). We study whether a network learns to correlate random noise with a target class. In addition, by comparing random patterns with these small patterss, can understand the impact of localized and dispersed spurious patterns. Lastly, the pattern core (Inv) is designed for MNIST with Gur = 0 to understand what would happen if the spurious pattern overlaps with the core feature of another class.
3

---

Under review as a conference paper at ICLR 2023
The choice of the combination function Øx- The function 인x combines the original example with the spurious pattern Xap into a spurious example. For simplicity. we consider the overlapping model where Φx directly adds the spurious pattern Xep onto the original example and then clips the value of each pixel to [0, IJ. i.e., xx Xsp) clip(o.1] Xsp)
The number of spurious examples. For MNIST and Fashion, randomly insert the spurious pattern to n = 0, 1,3, 5, 10, 20, 100, 2000, and 5000 training examples labeled as the target class Cua- These training examples inserted with a spurious pattern called spurious examples. For CIFARIO. we consider datasets with n = 0, 1, 5, 10, 20, 100, 500, and 1000 spurious examples. Note that 0 spurious example means the original training set not modified.
(a) original (b)
(c)
s2
(d)
(e)
(0)
(h)
Figure 1: Different spurious patterns considered in the experiment.
3.2 QUANTITATIVE ANALYSIS: SPURIOUS SCORE
To evaluate the strength of spurious correlations in a neural network, we design an empirical quantitative measure, spurious score, as follows. Let fe(x) be the neural network's predicted probability of an example x belonging to class Intuitively, the larger the prediction difference Sr,, (x) Sa,,(( xx(xx,x Xxg) is, the stronger spurious correlations the neural network f had learned. To quantify the effect of spurious correlations, measure how frequently the prediction difference of the test examples exceed certain threshold. Formally, let > O, we define the e-spurious score as the fraction of test example that satisfies
fee ( x xx)) - fo.. (x) E.
In other words, spurious score measures the portion of test examples that get an non-trivial increase in the predicted probability of the target class Cur when the spurious pattern is presented.
We make three remarks on the definition of spurious score. First, as we don't 't have any prior knowledge on the structure of f. use the fraction of test examples satisfying Eq. (1) as opposed to other function of fan, ( x x,p))= Sem. (x) (e.g.. taking average) to avoid non-monotone or unexplainable scaling. Second, the choice of the threshold is to avoid numerical errors to affect the result. In experiment, we pick = 1/(#classes) (e.g.. in MNIST we pick c = 1/10) and empirically similar conclusions can be made with other choices of Finally, we point that spurious score captures the privacy concern raised by the "Tony Blair" example mentioned in the introduction.
Eor
10'
VIZATOBIC 4ORTOdeX
(a) MNIST, Cur
(b) Fashion, Car
(c) CIFARIÓ, Cur
Each figure shows the mean and standard error of the spurious scores on three datasets, MNIST, Fashion, and CIFARIO, Cur = 0, and different numbers of spurious examples.
Figure 2:
Empirical findings. We repeat the measurement of spurious scores on five neural networks trained with different random seeds. Fig. 2 shows the spurious scores for each dataset and pattern function the number of spurious examples. Starting with the random pattern R3, we see that the spurious scores increase significantly from zero to three spurious examples in all six cases (three datasets and two target classes). This shows that neural networks can learn rare spurious correlations with as little as examples, it is surprising that the networks learn a strong correlation with extremely small amount of spurious examples. The result for other Our similar and can be found in App. D.
three spurious examples. Since all three datasets have 50, 000
training
4

---

Under review as a conference paper at ICLR 2023
A closer look at Fig. 2 reveals a few other interesting observations. First, comparing the small and random patterns, we see that random patterns generally have a higher spurious score. This suggests that dispersed patterns that spread out over multiple pixels may be more easily learnt than more concentrated ones. Second, spurious correlations are learnt even for Inv, on Cur = 0 and MNIST (recall that Inv is designed to be similar to the core feature of class one.) This suggests that spurious correlations may be learnt even when the pattern overlaps with the foreground. Finally, note that the models for CIFARIO are trained with data augmentation, which randomly shifts the spurious patterns during training, thus changing the location of the pattern. This suggests that these patterns can be learnt regardless of data augmentation.
In App. E.I, we also conduct a qualitative study by visualizing the weights of networks trained with rare spurious examples. We find that the spurious pattern can leave trace on the weights of the neural network. This observation strengthens our claim that neural networks are significantly effected by rare spurious examples.
3.3 NATURAL RARE SPURIOUS CORRELATIONS
A question is whether rare spurious correlations are also learnt on real (natural) data. To answer this question, we conduct an experiment focusing on natural spurious patterns using the NICO++ (Zhang et al., 2022) dataset, which is designed for studying non-L..D. image classification There are two labels of each image in the dataset: an object class (e.g., airplane) and the context (e.g., autumn). The context can then serve as source of spurious features: if a context only appears with the same class during the training stage (c.g., autumn context only shows up when the concept is airplane), then the algorithm might think the context is causally related to the classification of the object class (e.g.. classifying a bear in the autumn context as an airplane).
The NICO+ dataset consists of 55838 training images, sixty classes and six contexts, including autumn, dim, grass, outdoor, rock, and water. We split the dataset seven to three as the training and testing set. For each experiment, studying rare spurious correlations. For one trial of our experiment, we pick a context (i.e., a spurious feature) and remove all the appearance of this context in the training examples. To introduce spurious training examples, we select an object class as the target class and add number of examples that are labeled with the target class and this context. We use the ImageNet pretrained ResNet50 from to merch rchiis a i on and train twenty epochs on this modified training set. During testing, we collect all testing examples that do not belong to the spurious class but are under the spurious context and measure how many of these examples are predicted as the spurious class.
an object-context pair
invariant-spurion
pair for
A dataset of images the autumn context
Train a classifier
prodictons
100 80
images sirllane the sulumn
40
Test: number of non- airplane images in the autumn context is predicted as airplane
sirplane
bicycle
Sourious class
(a) An illustration the experiment setup.
(b) Spurious context:
Figure 3: (b) The number of non-spurious test examples that get predicted as the spurious class. We conduct Welch's 's 1-test (Welch, 1947) on the number of spurious class predictions between the model trained without spurious examples and models trained with different number of spurious examples. The notations for the p-values: ns: 0.05 < p & 1., *; 10-2 < & 0.05, **, 10 #**; 10
10-2
p & 10 #***; p 10
The experiments are repeated five times with different numbers of spurious examples in the training set and the results are in Fig. 3(b). In App. E.5, we show the results of two other spurious contexts, dim and grass, which also supports our conclusion. in total, we run a total of nine trials and five of them are significantly affected by just ten spurious training examples (which is less than 0.02% in the whole training examples). This suggests that rare spurious correlations also occur when the spurious features are natural.
Results.
5

---

Under review as a conference paper at ICLR 2023
4 CONSEQUENCES OF RARE SPURIOUS CORRELATIONS
In the previous analysis, we demonstrated that spurious correlations appear quantitatively and qualitatively in neural networks even when the number of spurious examples is small. Now, we investigate the potentially undesirable effects through the lens of privacy and test accuracy. In this section, the results for MNIST are similar to Fashion and CIFARIO and are deferred to App. D.
Privacy. We evaluate the privacy of a neural network (the target model) through membership inference attack. We follow the setup for black-box membership inference attack (Shokri et al., 2017; Yeom al., 2017). We record how well an attack model can distinguish whether an example is from the training or testing set using the output of the target model (equivalently to a binary classification problem). If the attack model has a high accuracy, this means that the target model is leaking out information from the training (private) data. The experiment is repeated ten times with their test accuracy recorded. For more detailed setup, please refer to App. D.2.
Resulty membership inference allack. Fig. 4 shows the mean and standard error of the attack model's test accuracy on all test examples and spurious examples. We see that the accuracies on spurious examples is generally higher when the number of spurious examples that spurious examples are more vulnerable to memhership inference attacks when appeared rarely. Although membership inference attack is different measure for privacy than spurious score, it can be a corroboration evidence that supports the fact that privacy is leaked from spurious examples.
small, which means
Test accuracy. We measure types of test accuracy on neural networks trained on different number of spurious examples. The clean lest accuracy measures the accuracy of the trained model on the original test data. The spurious lest accuracy simulates the case where there is distribution shift during the test time. Formally, spurious test accuracy is defined as the accuracy on a new test dataset constructed by adding spurious features all the test examples with a label different from Gar-
Results on clean test accuracy. We observe that the change in clean test accuracy in our exper- iments is small. Across all the models trained Fig. 2. the minimum, maximum, average, and standard deviation of the test accuracy for each dataset are: MNIST: (.976, .88,,900 Fashion: (.859, .90,, .800, 0100,, and CIFAR 10: (.876, .893, 08..886,0000
Results on spurious test accuracy. The results are shown in Fig. 5. We have two observations. First, we see that there are already some accuracy drop even when spurious test accuracy is evaluated on models trained on zero spurious examples. This means that these models are not robust to the existence of spurious features. This phenomena is prominent for spurious patterns with larger norm such R3. Second, we see that spurious test accuracies start to drop even more at around 10 to 100 spurious examples. This indicates that even with .01 % to .001 % of the overall training data filled with spurious examples of certain class, the robustness to spurious features can drop significantly.
100
Y.puxtous eorrpio
(a) Fashion, Cur O
(b) CIFARIO, Cur = 0
(a) Fashion, Cur = 0
(b) CIFARIO, Car
Figure 4: The test accuracy of the membership inference attack model on all examples spuri- ous examples. See App. E.3 for all results.
Figure 5:
The mean and standard error of the spurious test accuracy under different number of spurious examples. See App. E.4 for all results.
Discussion. Our experimental results suggest that neural networks are highly sensitive to very small amounts of spurious training data. Furthermore, the learnt rare spurious correlations cause undesirable effects on privacy and test accuracy. Easy learning of rare spurious correlations can lead to privacy issues (Leino & Fredrikson, 2020) where an adversary may infer the presence of confidential image in a training dataset based on output probabilities. It also raises fairness concerns as a neural network can draw spurious conclusions about minority group if a small number of subjects from this group are present in the training set (Izzo et al., 2021). We recommend to test and audit neural networks thoroughly before deployment in these applications.

---

Under review as a conference paper at ICLR 2023
Beyond the empirical analysis explained in this section, we also explore how other factors such as the strength of spurious patterns, network architectures, and optimization methods, affect spurious correlations. We find that cannot remove rare spurious correlations by simply tuning these parameters. We also observe that neural networks with more parameters may not always learn spurious correlations more easily, which is counter to Sagawa al. (2020)'s observation. The detailed results and discussions are provided in App. B.
5 THEORETICAL UNDERSTANDING
In this section, we devise mathematical model study rare spurious correlations. The theoretical analysis not only provides an unifying understanding to explain the experimental findings but also inspires to propose methods reduce the undesirable effects of rare spurious correlations in Sec. 6. We emphasize that the purpose of the theoretical analysis is to capture the key factors in rare spurious correlations and we leave it as a future research direction to further deepen the theoretical study.
To avoid unnecessary mathematical complications, we make two simplifications in our theoretical analysis: (i) we focus on the concatenate model and (ii) the learning algorithm is linear regression with mean square loss. For (i), we argue that this is the simplest scenario of spurious correlations and hence it is a necessary step before we understand general spurious correlations. While the experiments in Sec. 3 work in the overlapping model, we believe that the high level messages of our theoretical analysis would extend to there as well as other more general scenarios. For (ii), we pick simpler learning algorithm in order to have an analytical characterization of the algorithm's performance. This is because we aim to have an understanding of how the different factors (c.g., the fraction of spurious inputs, the strength of spurious feature, etc.) of spurious correlations play role.
5.1 THEORETICAL MODEL TO STUDY RARE SPURIOUS CORRELATIONS
We consider a binary classification task to model the appearance of rare spurious correlations. Let X = x Xxp be an input vector space and let y = (-1,1} be a label space. Let E [0, 1] be the parameter for the fraction of spurious samples, let x_, Xnm be the invariant features of the two classes, let Xap E Mep be the spurious feature, and let 2in o > 0 be the parameters for the variance along Xim and Xxp respectively. Finally, the target class is +, i.e., Ctar
We postpone the formal definitions of the training distribution Duain. the clean text distribution Desest, and the spurious test distribution Paust to App. A. See also Fig. 6 for some pictorial examples.
Jxp
Mp
022
(a) 7
(b) y > 0
(c) Phase tramsitions
Figure 6: Examples of the training distribution Drrai and phase transitions in our theoretical model. (a)-(b) With equal probability a training example is sampled from either ((x+, Ouy Inv + Tsp or N(xx an + spp With probability Y, a sample will be concatenated with the spurious pattern Xxp- The dotted line is the decision boundary of the optimal classifier. (c) Both the spurious test accuracy and the prediction difference exhibit phase transition at 7 0.
5.2 ANALYSIS FOR LINEAR REGRESSION WITH MEAN SQUARE LOSS AND & REGULARIZZTION
In this paper, we analyze our theoretical model in the setting of linear regression with l loss. We analytically derive the test accuracy and the prediction difference fo., ( xxxx(xxxxx xxp) foa (x) in App. A and here we present our observations. Here, we study the prediction difference as proxy for the spurious score since the latter is always either 0 or 1 for a linear classifier under our model.
Observation 1: A phase transition of spurious test accuracy and prediction difference at = 0. Our theoretical analysis first suggests that there is phase transition of the spurious test accuracy and the prediction difference spurious score at y = 0, i.c. there is an sharp growth/decay within an interval near 7 0. The phase transition matches our previous experimental studies discussed
7

---

Under review as a conference paper at ICLR 2023
in Sec. 3. This indicates that the effect of spurious correlations is spontaneous rather than gradual. To be more quantitative, the phase transition takes place when the signal torialltt-noiss ratio of spurious feature, i.e,, Xxp 2 is large (see App. A for details). This further suggests to increase the variance in the spurious dimension and leads to our next two observations.
Observation 2: adding Gaussian noises lowers spurious score. The previous observation the importance of spurious signal-to-noise ratio o immediately suggests us to add Gaussian noises to the input data to lower |xall// sp. Indeed, the prediction difference becomes smaller in most parameter regimes, however, both the clean test accuracy and the spurious test accuracy decrease. Intuitively, the effect of adding noises is to mix the invariant feature with the spurious feature and the decrease of test accuracy is as expected. Thus, to simultaneously lower prediction difference and improve test accuracy, one needs to detect the spurious feature in some ways.
Observation 3: l regularization improves test accuracy and lowers spurious score. Finally, theoretical analysis reveals that there are two parameter regimes where adding & regularization to linear regression can improves accuracy and lowers the prediction difference. First, when of is small, y is small, and the spurious signal-to-noise ratio ||x 2 is large. Second, when o is large and both y and Xxp / are mild. Intuitively, lg regularization suppresses the use features that only appears on small number of training examples.
Discussion. Our theoretical analysis quantitatively demonstrates the phase transition of spurious correlations at y = 0 and the importance of the spurious signal-to-noiss ratio |xxpl8//2 / This not only coincides with our empirical observation in Sec. but also suggests future directions to mitigate rare spurious correlations. Specifically, one general approach to reduce the undesirable effects of rare spurious correlations would be designing learning algorithms that projects the input into a feature space that has low spurious signal- to-noise ratio.
6 MITIGATION OF RARE SPURIOUS CORRELATION
Prior work uses group rebalancing (Idrissi et al., 2021; Sagawa et al., 2020; 2019; Kulynych et al., 2022), data augmentation (Chang al., 2021) or learning invariant classifier (Arjovsky al., 2019) to mitigate spurious correlations. However, these methods usually requires additional information on what the spurious feature is, and rare spurious correlations, identifying the spurious feature can be hard. Thus, we may require different techniques.
Our theoretical result suggest that l regularization (weight decay) and adding Gaussian noise to the input (noisy input) may reduce the degree of spurious correlation being learnt. In addition, we examine an extra regularization method gradient clipping.
Squncuu scaee
0.4
0.8
Test accumay
Spuricus score
0.3
0.4-
0,6
0.8
Test assou mon
Spariuus wore
0.3
0.4
This aoo nnng
022
0.1
0
2.1
0.01
0.0-
0.0
0.0
10
10
10P
10 10-500-110
10 110 110
10
10"
10
10
10
(a) Noisy input
(b) Weight decay
(c) Gradient clipping
Figure 7: Spurious score (solid blue line), clean test accuracy (solid red line), and spurious test accuracy (dotted red line) vs. the regularization strength on Fashion with different regularization methods. For the experiment, compute the average spurious score and clean test accuracy across models trained with 1, 3, 5, 10, 20, and 100 spurious examples and five random seeds. The results for MNIST and CIFARIO and spurious pattern R3 are in App. D, which shows similar results.
fix the spurious pattern to be S3 and the target class Ctar 0. We
The results are shown in Fig. 7. We see that with a properly set regularization strength for noisy input and weight decay, one may reduce the spurious score and increase spurious test accuracy without sacrificing much accuracy. This significantly reduces the undesirable consequences brought by rare spurious correlations. This aligns with our observation in the theoretical model and suggests that neural networks may share similar property with linear models. We also find that gradient clipping cannot mitigate spurious correlation without reducing the test accuracy. Finally, we observe that all these methods are unable completely avoid learning spurious correlations.
Results.
8

---

Under review as a conference paper at ICLR 2023
Data deletion methods. Another idea for mitigating rare spurious correlation is to apply data deletion methods (lzzo et al., 2021). In App. C, we experiment with two data deletion methods, incremental retraining and group influence functions (Basu et al., 2020a). However, they are not effective.
Discussion. Regarding mitigating rare spurious correlations, a provable way to prevent learning them is differential privacy (Dwork et al., 2006), which ensures that the participation of single person (or a small group) in the dataset does not change the probability of any classifier by much. This requires noise addition during training, which may lead significant loss in accuracy (Chaudhuri al., 2011; Abadi et al., 2016). If we know which are the spurious examples, then we can remove spurious correlations via an indistinguishable approximate data deletion method (Ginart et al., 2019; Neel et al., 2020); however, these methods provide lower accuracy for convex optimization and have no performance guarantees for non-convex. An open problem is to design algorithms or architectures that can mitigate these without sacrificing prediction accuracy.
Prior work (Sagawa et al., 2019; Kulynych et al., 2022) suggests that proper use of regularization methods plus well-designed loss functions can mitigate some types of spurious correlations. However, these regularization methods are used either in an ad-hoc manner or may reduce test accuracy. As shown in our experiment, not all regularization methods can remove rare spurious correlations without reducing the accuracy. This suggests that different regularization methods may be specifically tied to be able to mitigate certain kinds of spurious correlation. The exact role that regularization methods play in reducing spurious correlations is still an open question. Figuring out what kinds of spurious correlations can be mitigated by which regularization methods is an interesting future direction.
RELATED WORK
Spurious correlations. Previous work has looked at spurious correlations in neural networks under various scenarios, including test time distribution shift (Sagawa et al., 2020; Srivastava et al., 2020; Bahng et al., 2020: Zhou al., 2021; Khani & Liang, 2021), confounding factors in data collection (Gururangan et al., 2018), the effect of image backgrounds (Xiao et al., 2020), and causality (Arjovsky al., 2019). However, in most works, spurious examples often constitute a significant portion of the training set. In contrast, we look at spurious correlations introduced by small number of examples (rare spurious correlations). Concurrent work (Hartley Tsaftaris, 2022) measures spurious correlation caused by few examples. However, they did not show the consequences of these spurious correlations nor discuss ways to mitigate them.
Memorization in neural networks. Prior work has investigated how neural networks can inadver- tently memorize training data (Arpit et al., 2017: Carlini et al., 2019; 2020; Feldman & Zhang, 2020; Leino & Fredrikson, 2020). Methods have also been proposed measure this kind of memorization, including the use of the influence function (Feldman & Zhang, 2020) and likelihood estimates (Carlini al., 2019). Our work focuses partial memorization instead of memorizing individual examples, and our proposed method may be potentially applicable in more scenarios.
A line of work in the security literature exploits the memorization of certain patterns compromise neural networks. The backdoor attack from Chen et al. (2017) attempts to change hard label predictions and accuracy by inserting carefully crafted spurious patterns. Sablayrolles et al. (2020) design specific markers that allow adversaries to detect whether images with those particular markers are used for training in a model. Another line of research on data poisoning attack Xiao al. (2015); Wang & Chaudhuri (2018); Burkard & Lagesse (2017) aims to degrade the performance of a model by carefully altering the training data. In contrast, our work looks at rare spurious correlations from natural spurious patterns, instead of adversarially crafted ones. App. F has detailed discussions.
8 CONCLUSION
The learning of spurious correlation is a complex process, and it can have unintended consequences. As neural networks are getting more widely applied, it is crucial to better understand spurious corre- lations. There are many open questions remain. For example, besides the distribution shift that adds spurious features, are there any other types of distribution shift that will affect the accuracy? Another limitation of our current study is that our experiments are conducted only image classification tasks, which may not generalize to others.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprini arXiv: arXi:: 0020999, 2019.
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International Conference on Machine Learning, pp. 233-242, 2017.
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In International Conference on Machine Learning, pp. 528-533 2020.
Samyadeep Basu, Philip Pope. and Soheil Feizi. Influence functions in deep learning are fragile. arXiv preprint arXiv:2006. 1120006.146555 14651, 2020a.
Samyadeep Basu, Xuchen You, and Soheil Feizi On second-order group influence functions for black-box predictions. In International Conference on Machine Learning, pp. 715-724, 2020b.
Cody Burkard and Brent Lagesse. Analysis of causative attacks against svms learning from data streams. In Proceedings of the 3rd ACM on International Workshop Security And Privacy Analytics, pp. 31-36, 2017.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jerej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th (USENIX) Security Symposium ( ({USENIX) Security 19), pp. 267-284, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. arXiv preprint arXiv:20/2.07805, 2020.
Chun-Hao Chang. George Alexandru Adam, and Anna Goldenberg. Towards robust classification model by counterfactual and invariant data generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15212-15221, 2021.
Kamalika Chaudhuri, Claire Monteleoni, and Anand D) Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research, 12(3), 2011.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv: 2017.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise sensitivity in private data analysis. In Theory of cryptography conference, pp- 265-284, 2006.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. arXiv preprint (tt:::200000300030 2020.
Robert Geirhos, föm--Herrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 1):665-673, 2020.
Antonio Ginart, Melody Y Guan, Gregory Valiant, and James Zou. Making ai forget you: Data deletion in machine learning. arXiv preprint 1xx.vv1907.05012, 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arxiv: 2014.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. Annotation artifacts natural language inference data. arXiv preprint arXiv: 1003v 0234442244 2018.
10

---

Under review as a conference paper at ICLR 2023
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathamiel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy, Nature, 3857199557778222255--22222 September 2020. doi: 10. 10.10338/411566-0022444499 URL https: /toi//iii org 10 1038 41586 22000-264--9
Kevin Sheppard, Tyler Reddy,
John Hartley and Sotirios Tsaftaris. Measuring unintended memorisation of unique private features in neural networks. arXiv preprint arXiv:2202.08099 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Badr Youbi Idrissi, Martin Arjovsky. Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive purst-groupaacurree arXiv preprint arXiv:2110.14503 2021.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference Artificial Intelligence and Statistics, pp. 2008-2016, 2021.
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private private sgd? Advances in Neural Information Processing Systems, 33: 22205-22216 2020.
Fereshte Khani and Percy Liang. Removing spurious features can hurt accuracy and affect groups disproportionately. In Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transparency, pp. 196-205, 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 11Xiv:1412.6980, 2014.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference Machine Learning, pp. 1885-1894. PMLR, 2017.
Pang Wei Koh, Kai-Siang Ang, Hubert HK Teo, and Percy Liang. On the accuracy of influence functions for measuring group effects. arXiv preprint 1905. 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images, 2009.
Bogdan Kulynych, Ruo-Yuan Yang, Yaodong Yu, Jarostaw Blasiok, and Preetum Nakkiran. What you see is what you get: Distributional generalization for algorithm design in deep learning. arXiv preprint 3rXiv:2204.032330, 2022.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmistt/ 1998.
Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization for calibrated white-box membership inference. In 29th (USENIX) Security Symposium (!USENIX) Security 20), pp. 1605-1622, 2020.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models poisoned data. Advances in Neural Information Processing Systems, 34: 14900-14912. 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15 2018
Yifei Min, Lin Chen, and Amin Karbasi. The curious case of adversarially robust models: More data can help, double descend, or hurt generalization. arXiv preprint arXiv:2002.11080, 2020.
Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of dutti--dist generalization. arXiv preprint 2rXiv:2010.15775: 2020.

---

Under review as a conference paper at ICLR 2023
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. arXiv preprint arXi::1112.02292, 2019.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances in Neural Information Processing Systems, 33:20673-20683 2020.
Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papemoti, and Nicholas Carlin. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 866 882. IEEE, 2021.
Seth Neel, Aaron Roth, and Saced Sharifi- Sharifi-Malvajerdi Descent-to-delete: Gradient-based methods for machine unlearning. arXiv preprint arXiv:2007.02922, 2020.
The pandas development team. pandas-dew/pandas: Pandas, February 2020. URL https: //doi.
org/10. 5281
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer. James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Pedregosa. G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Radioactive data: tracing through training. In International Conference on Machine Learning, pp.
2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: /9/ I. 1.08311, 2019.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning, pp. 8346-8356 2020.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585, 2020.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprini arXiv:14009 1556, 2014.
Megha
Srivastava, Tatsunori Hashimoto, and Percy Liang. Robustness to spurious correlations via human annotations. In International Conference on Muchine Learning, pp. 9109-9119, 19, 2020.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kem, Bric Larson, C Carey, Îlhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, António Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272. 2020. doi: 10.1038/s411 11188//4199-019--06-
Yizhen Wang and Kamalika Chaudhuri. Data poisoning attacks against online learning. arXiv preprint arXiv: 1808. 1800.08994 2018
12

---

Under review as a conference paper at ICLR 2023
Bernard L Welch. The generalization of 'student's problem when several different population varlances are involved. Biometrika, 34(1-2):28-33 33, 1947.
Cheng-Hsin Weng, Yan-Ting Lee, and Shan-Hung Brandon Wu. On the trade-off between adversarial and backdoor robustness. Advances in Neural Information Processing Systems, 19:31993-,,983, 2020.
Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems, 34:16913-16922 16925, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 1rXi::/708.0774: 2017.
Huang Xiao, Battista Biggio, Blaine Nelson, Han Xiao, Claudia Eckert, and Fabio Roli. Support
vector machines under adversarial label contamination. Neurocomputing, 160:53-60 62, 2015,
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.09994 2020.
Samuel Yeom, Matt Fredrikson, and Somesh Jha. The unintended consequences of overfitting: Training data inference attacks. arXiv preprint arXiv:1709.0114 12, 2017.
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Yan Liu. Indiscriminate poisoning attacks shortcuts. arXiv preprint arXiv:2111.00890 2021.
Xingxuan Zhang, Linjun Zhou, Renzhe Xu, Peng Cui, Zheyan Shen, and Haoxin Liu. Nico++: Towards better benchmarking for domain generalization. arXiv preprint arXiv:2204.08040, 2022.
Chunting Zhou, Xuezhe Ma, Paul Michel, and Graham Neubig. Examining and combating spurious features under distribution shift. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12857-12867, Jul 2021.
13


Reviews:
[{"review_id": "jc0Yt4iXT5", "reviewer": "Reviewer_LUVv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates how sensitive neural networks are with respect to rare spurious correlations from three perspectives. First, studying how many training points with the spurious pattern would cause noticeable spurious correlations in the synthetic and real datasets. Second, studying how rare spurious correlations affect neural networks’ privacy and test accuracy. Third, studying how to mitigate the effects of rare spurious correlation.\n", "summary_of_the_review": "With the findings above, I currently give the paper a borderline score.", "strength_and_weaknesses": "Strength:\n1. The paper studies an interesting topic of learning with rare spurious correlations via both experimental and theoretical approaches.\n2. The paper studies whether rare spurious correlations are learned on both synthetic and real data.\n3. The paper studies the consequences of rare spurious correlations from a privacy and test accuracy perspective.\n4. The paper introduces simple and effective methods suggested by theoretical evidence to reduce the undesirable consequence of spurious correlation.\n5. The paper is well written.\n\nWeaknesses and questions:\n\n1. In Figure 1, it seems like the seven different spurious patterns (adding different kinds of noise) are not strong enough to generate a spurious correlation. How about using more challenging datasets, such as Colored MNIST?\n\n2. The main observations that neural networks can learn rare spurious correlations with few spurious examples are a little weak for me. It is not a big surprise to find out that adding perturbated patterns into training samples can impact the confidence of prediction on target classes during inference. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hzJwyyXMXj", "reviewer": "Reviewer_bwQt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is dedicated to the problem of spurious correlations. The work systematically studies fundamental questions such as how many spuriously correlated training points are necessary for a neural net to get biased towards learning it. Specifically, it investigates the domain in which spurious correlations are rare. Interestingly the study indicates that even a single spuriously correlated sample can bias the learning of a neural network. Finally, the authors highlight three regularization methods for mitigating spurious correlations.", "summary_of_the_review": "The paper studies some important research questions in the domain of spurious correlations. The findings also provide non-trivial contributions to the community. However, because of the synthetic setup, the overall impression of this work is borderline.  I suggest the authors comment on the weaknesses mentioned above and I am open to changing my rating.", "strength_and_weaknesses": "Strength:\n\n1. The paper is well-motivated, organized, easy to follow and well-written.\n2. The study aims to answer fundamental research questions, which can provide new insights to the community.\n3. The authors provide theoretical justification for their findings\n\nWeakness:\n1. The technical contribution is somewhat limited in the sense that prior works have already shown that neural networks are more biased towards \"easy-to-learn\" spurious attributes. \n2. A major chunk of the experimental analysis is based on cases where the spurious correlation is injected synthetically. It would be more interesting if the authors also showed results on commonly studied spuriously correlated datasets such as Waterbirds, and CelebA.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4rt6UtG5EUw", "reviewer": "Reviewer_yphb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how different numbers of spurious training examples (from one to thousands) affect neural nets. They measure how the predictions of test examples change when the spurious features are added. They show even a few spurious training examples can make the model outputs be affected by the spurious feature.\n\nThe authors consider two different types of spurious features. The first one is some artificial patterns, e.g., a square of yellow pixels. The second one is some natural spurious features in real data. They use the NICO++ dataset that has different backgrounds (grass, rock, water..) for each class. The authors then study the consequences of learning rare spurious features on privacy and accuracy. They also analyze the findings with a toy theoretical model and study several simple mitigations.\n", "summary_of_the_review": "Given that 1) the findings about artificial spurious features seem incremental because of the backdoor attack literature; 2) how neural nets learn natural rare spurious features is not well studied. I'm leaning to reject this paper.", "strength_and_weaknesses": "**Strength**\n\n1.Studying the relation between the number of spurious examples and the strength of spurious correlations is an important problem. The experiments show that neural nets could learn natural rare spurious correlations are new and interesting. \n\n2.This paper is well-written. Extensive results are well organized. \n\n3.The analysis on the privacy risk of spurious examples is new.\n\n**Weaknesses**\n\n1.The spurious features in Section 3.1 and 3.2 are very similar to backdoor triggers. They both are some artificial patterns that only appear a few times in the training set. For example, Chen et al. (2017) use random noise patterns. Gu et al. (2019) [1] use single-pixel and simple patterns as triggers. It is well-known that a few training examples with such triggers (rare spurious examples in this paper) would have a large impact on the trained model. \n\n2.How neural nets learn natural rare spurious correlations is unknown to the community (to the best of my knowledge). However, most of analysis and ablation studies use the artificial patterns instead of natural spurious correlations. Duplicating the same artificial pattern for multiple times is different from natural spurious features, which are complex and different in every example.\n\n3.What’s the experiment setup in Section 3.3? (data augmentation methods, learning rate, etc.).\n\n[1]: BadNets: Evaluating Backdooring Attacks on Deep Neural Networks. https://messlab.moyix.net/papers/badnets_ieeeaccess19.pdf", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2iOyCrLKXq", "reviewer": "Reviewer_ajJR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Most existing papers working on spurious correlation focus on the correlation that existed in the majority of the examples. This paper discovered that even a few spurious examples can still lead to the model learning the spurious correlation. Moreover, these \"rare\" spurious correlations bring negative effects regardless of the strength of spurious patterns, network architectures, and optimization methods.  Finally, the paper provides a theoretic analysis of a simple binary classification model. Based on the analysis, a few techniques are proposed to improve learning with rare spurious correlations. ", "summary_of_the_review": "The paper discovers that even a few spurious examples can still lead to the model learning the spurious correlation. Many interesting results are shown with experiments: It shows that spurious patterns with larger empirical norms can cause spurious correlation more easily, and network architectures with higher sensitivity to its input are more susceptible to learning spurious correlations. ", "strength_and_weaknesses": "Strength:\n- Extensive experiments are performed across datasets, the strength of spurious patterns, network architectures, and optimization methods, showing that even a small amount of spurious correlation will also influence the model performance. \n- Theoretical analysis of a simple but meaningful model is conducted, moreover, inspired by the analysis, several techniques are proposed and validated with experiments. \n\nWeaknesses:\n- Does the technique proposed also work when the majority of the data present spurious correlation, e.g. Adding l2 regularization? Because as mentioned in the paper: l2 regularization only \"suppresses the use of features that only appears on a small number of training examples\". ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject