Title: Trained Models Tell Us How to Make Them Robust to Spurious Correlation without Group Annotation
Venue: NeurIPS 2024
Accepted: no
Keywords: Spurious Correlation

Group Robustness

Zero Group Annotation

Distribution Shift

Out-of-Distribution Generalization
Abstract: Classifiers trained with Empirical Risk Minimization (ERM) tend to rely on attributes that have high spurious correlation with the target. This can degrade the performance on underrepresented (or 'minority') groups that lack these attributes, posing significant challenges for both out-of-distribution generalization and fairness objectives. Many studies aim to improve robustness to spurious correlation, yet nearly all require group annotation for training and/or model selection. This constrains their applicability in situations where the nature of the spurious correlation is not known, or when group labels for certain spurious attributes are either insufficient or completely absent. To meet the demand for effectively enhancing the model robustness under minimal assumptions about group annotation, we propose Environment-based Validation and Loss-based Sampling (EVaLS). It uses the losses from a trained model to construct a balanced dataset of high-loss and low-loss samples in which the training data group imbalance is mitigated. This results in a significant robustness to group shifts when equipped with a simple mechanism of last layer retraining. Furthermore, by utilizing environment inference methods for creating diverse environments with correlation shifts, EVaLS can potentially eliminate the need for group annotation in the validation data. In such a context, the worst environment accuracy acts as a reliable surrogate throughout the retraining process for tuning hyperparameters and finding a model that performs well across diverse group shifts. EVaLS effectively achieves group robustness, showing that group annotation is not necessary even for validation. It is a fast, straightforward, and effective approach that reaches near-optimal worst group accuracy without needing group annotations, marking a new chapter in the robustness of trained models against spurious correlation.

Abstract

Classifiers trained with Empirical Risk Minimization (ERM) tend to rely on attributes that have high spurious correlation with the target. This can degrade the performance on underrepresented (or minority ) groups that lack these attributes, posing significant challenges for both out-of-distribution generalization and fairness objectives. Many studies aim to improve robustness to spurious correlation, yet nearly all require group annotation for training and/or model selection. This constrains their applicability in situations where the nature of the spurious correlation is not known, or when group labels for certain spurious attributes are either insufficient or completely absent. To meet the demand for effectively enhancing the model robustness under minimal assumptions about group annotation, we propose Environment-based Validation and Loss-based Sampling (EVaLS). It uses the losses from a trained model to construct a balanced dataset of high-loss and low-loss samples in which the training data group imbalance is mitigated. This results in a significant robustness to group shifts when equipped with a simple mechanism of last layer retraining. Furthermore, by utilizing environment inference methods for creating diverse environments with correlation shifts, EVaLS can potentially eliminate the need for group annotation in the validation data. In such a context, the worst environment accuracy acts as a reliable surrogate throughout the retraining process for tuning hyperparameters and finding a model that performs well across diverse group shifts. EVaLS effectively achieves group robustness, showing that group annotation is not necessary even for validation. It is a fast, straightforward, and effective approach that reaches near-optimal worst group accuracy without needing group annotations, marking a new chapter in the robustness of trained models against spurious correlation.

1 Introduction

Training deep learning models using Empirical Risk Minimization (ERM) on a dataset, poses the 26

risk of relying on spurious correlation . These are correlations between certain patterns in the 27

training dataset and the target (e.g., the class label in a classification task) despite lacking any causal 28

relationship. Learning such correlations as shortcuts can negatively impact the models' accuracy on 29

minority groups that do not contain the spurious patterns associated with the target [1, 2]. This problem 30

leads to concerns regarding fairness [3], and can also cause a marked reduction in the performance. 31

This occurs particularly when minority groups, which are underrepresented during training, become 32

overrepresented at the time of testing, as a result of shifts within the subpopulations [4]. Hence, 33

ensuring robustness to group shifts and developing methods that improve worst group accuracy 34

(WGA) is crucial for achieving both fairness and robustness in the realm of deep learning. 35

Many studies have proposed solutions to address this challenge. A promising line of research 36 focuses on increasing the contribution of minority groups in the model's training [1, 5-7]. A strong 37 assumption that is considered by some previous works is having access to group annotations for 38 training or fully/partially fine-tuning a pretrained model [8, 7, 1]. The study by Kirichenko et al. [1] 39 proposes that retraining the last layer of a model on a dataset which is balanced in terms of group 40 annotation can effectively enhance the model's robustness against shifts in spurious correlation. While 41 these works have shown tremendous robustness performance, their assumption for the availability of 42 group annotation restricts their usage. 43

44

45

46

47

48

49

50

51

52

53

54

55

In many real-world applications, the process of labeling samples according to their respective groups can be prohibitively expensive, and sometimes impractical, especially when all minority groups may not be identifiable beforehand. A widely adopted strategy in these situations involves the indirect inference of various groups, followed by the training of models using a loss function that is balanced across groups [5, 9, 10, 4]. The loss value of the model or its similar metrics is a popular signal for recognizing minority groups [5, 9-11]. While most of these techniques necessitate full training of a model, Qiu et al. [9] attempt to adapt the DFR method [1] with the aim of preserving computational efficiency while simultaneously improving robustness to group shift. However, this method still requires group annotations of the validation set for model selection and hyperparameter tuning. Consequently, this constitutes a restrictive assumption when adequate annotations for certain groups are not supplied. It also applies to situations where some shortcut attributes are completely unidentified.

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

In this study, we present a novel strategy that effectively mitigates reliance on spurious correlation, completely eliminating the need for group annotations during both training and retraining. More interestingly, we provide empirical evidence indicating that group annotations are not necessary, even for model selection. We show that assembling a diverse collection of environments for model selection, which reflect group shifts can serve as an effective alternative approach. Our proposed method, Environment-based Validation and Loss-based Sampling (EVaLS), is a technique that strengthens the robustness of trained models against spurious correlation, all without relying on group annotations. EVaLS is pioneering in its ability to eliminate the need for group annotations at every phase , including the model selection step. EVaLS posits that in the absence of group annotations, a set of environments showcasing group shifts is sufficient. Worst Environment Accuracy (WEA) could then be utilized for model selection. Our findings demonstrate that utilizing environment inference methods [12], or even dividing the validation data based on the predictions of a random linear layer atop a trained model's feature space can markedly enhance group robustness. Figure 1 demonstrates the overall procedure of the main parts of EVaLS.

Our empirical observations support prior research which suggests that high-loss data points in a trained model may signal the presence of minority groups [5, 9, 10]. Our method, EVaLS, evenly selects from both high-loss and low-loss data to form a balanced dataset that is used for last-layer retraining. We offer theoretical explanations for the effectiveness of this approach in addressing group imbalances, and experimentally show the superiority of our efficient solution to the previous strategies. Comprehensive experiments conducted on spurious correlation benchmarks such as CelebA [13], Waterbirds [7], and UrbanCars [14], demonstrate that EVaLS achieves optimal accuracy. Moreover, when group annotations are accessible solely for model selection, our approach, EVaLS-GL, exhibits enhanced performance against various distribution shifts, including attribute imbalance, as seen in MultiNLI [15], and class imbalance, exemplified by CivilComments [16]. We further present a new dataset, Dominoes Colored-MNIST-FashionMNIST , which depicts a situation featuring multiple independent shortcuts, that group annotations are only available for part of them (see Section 2.2). In this setting, we show that strategies with lower levels of group supervision are paradoxically more effective in mitigating the reliance on both known and unknown shortcuts.

The main contributions of this paper are summarized as follows:

We present EVaLS, a simple yet effective approach that enhances model robustness against spurious correlation without relying on ground-truth group annotations.

Weoffer both theoretical and practical insights on how balanced sampling from high-loss and low-loss samples can result in a dataset in which the group imbalance is notably mitigated.

Using simple environment inference techniques, EVaLS leverages worst environment accuracy as a reliable indicator for model selection.

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

EVaLS attains near-optimal worst group accuracies or even exceeds them in spurious correlation benchmarks, all with zero group annotations.

When group annotations are available for model selection, EVaLS delivers state-of-the-art performance across a variety of subpopulation shift benchmarks.

We introduce a new dataset consisting of two spurious features in which partial supervision may negatively impact the performance of the underrepresented groups.

2 Preliminaries

2.1 Problem Setting

We assume a general setting of a supervised learning problem with distinct data partitions D tr for training, D val for validation, and D test for final evaluation. Each dataset comprises a set of paired samples ( x, y ) , where x ∈ X represents the data and y ∈ Y denotes the corresponding labels. Conventionally, D tr , D val , and D test are assumed to be uniformly sampled from the same distribution. However, this idealized assumption does not hold in many real-world problems where distribution shift is inevitable. In this context, we consider the sub-population shift problem [4]. In a general form of this setting, it is assumed that data samples consist of different groups G i , where each group comprises samples that share a property. More specifically, the overall data distribution p ( x, y ) = ∑ i α i p i ( x, y ) is a composition of individual group distributions p i ( x, y ) weighted by their respective proportions α i , where ∑ i α i = 1 . In this work, we assume that D tr , D val , and D test are composed of identical groups but with a different set of mixing coefficients { α i } . It is noteworthy that the validation set may have approximately identical coefficients to those of the training or testing sets, or it may have entirely different coefficients.

Several kinds of subpopulation shifts are defined in the literature, including class imbalance, attribute imbalance, and spurious correlation [4]. Class imbalance refers to the cases where there is a difference between the proportion of samples from each class, while attribute imbalance occurs when instances with a certain attribute are underrepresented in the training data, even though this attribute may not necessarily be a reliable predictor of the label. On the other hand, spurious correlation occurs when various groups are differentiated by spurious attributes that are partially predictive and correlated with class labels but are causally irrelevant. More precisely, we can consider a set of spurious attributes S that partition the data into |S| × |Y| groups. When the concurrence of a spurious attribute with a label is significantly higher than its correlation with other labels, that spurious attribute could become predictive of the label, resulting in deep models relying on the spurious attributes as shortcuts instead of the core ones. This is followed by a decrease in the model's performance on groups that do not have this attribute.

Given a class, the group containing samples with correlated spurious attributes is referred to as majority group of that class, while the other groups are called the minority groups. As an example, in the Waterbirds dataset [7], for which the task is to classify images of birds into landbird and waterbird, there are spurious attributes { water background , land background } . Each background is spuriously correlated with its associated label, decompose the data into two majority groups waterbird on water background , and landbird on land background , and two minority groups waterbird on land background and landbird on water background . Our goal is to make the classifier robust to spurious attributes by increasing performance for all groups.

2.2 Robustness of a Trained Model to an Unknown Shortcut

In scenarios where group annotations are absent, traditional methods that depend on these annotations for training or model selection become infeasible. Moreover, as previously discussed by [14], when data contains multiple spurious attributes and annotations are only available for some of them, such methods would make the model robust only to the known spurious attributes. To explore such complex scenarios, we introduce the Dominoes Colored-MNIST-FashionMNIST (Dominoes CMF) dataset (Figure 3(d)). Drawing inspiration from Pagliardini et al. [17] and Arjovsky et al. [18], Dominoes CMF merges an image from CIFAR10 [19] at the top with a colored (red or green) MNIST [20] or FashionMNIST [21] image at the bottom. The primary label is derived from the CIFAR10 image, while the bottom part introduces two independent spurious attributes: color and shape. Although

Split

class

group

sample

1

min

maj

2

min

maj

train

𝑫 𝑳𝑳

1

min

maj

2

min

maj

random splitting

ERM classifier

high-loss samples

low-loss samples

rank-based sampling

top-k high and low

loss samples

loss-based sorting

Environment

Inference method

{Env 1, Env 2, Env3, Env 4}

validation split

K=2

K=4

WEA: 0.70

Env 1

Env 2

Env 3

Env 4

WEA: 0.80

WEA: 0.65

acc: 0.95

acc: 0.75

acc: 0.70

acc: 0.85

acc: 0.90

acc: 0.85

acc: 0.90

acc: 0.80

acc: 0.65

acc: 0.95

acc: 0.90

acc: 0.70

(a) Held-out data split

(c) Loss-based sampling

(b) Environment inference

(d) Last-layer retraining

training

✓

train

𝑫 𝑳𝑳

validation

𝑫 𝑴𝑺

model selection

annotations for shape are provided for training and model selection, color remains an unknown 142 variable until testing. For more details on the dataset refer to the Appendix. 143

The illustrations in Figure 3(a-c) depict the outlined scenario. A classifier trained using ERM is 144 dependent on both spurious features (Figure 3(b)). Yet, achieving robustness against one spurious 145 correlation (Figure 3(c)), does not ensure robustness against both (Figure 3(a)). In Section 4 we 146 show that our method, which does not rely on the group annotations of the identified group, achieves 147 enhanced robustness against both spurious correlations, outperforming strategies that depend on the 148 known group's information. 149

150

3 Environment-based Validation and Loss-based Sampling

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

Our method, EVaLS, is designed to improve the robustness of deep learning models to group shifts without the need for group annotation. In line with the DFR [1] approach, we utilize a classifier defined as f = h ϕ ◦ g θ , where g θ represents a deep neural network serving as a feature extractor, and h ϕ denotes a linear classifier. The classifier is initially trained with the ERM objective on the training dataset D tr . Subsequently, we freeze the feature extractor g θ and focus solely on retraining the last linear layer h ϕ using the validation dataset D val as a held-out dataset.

We randomly divide the validation set D val into two subsets, D LL and D MS which are used for last layer training and model selection, respectively. In Section 3.1 we explain how to sample a subset of D LL that statistically handles the group shifts inherent in the dataset. In Section 3.2 we describe how D MS is divided into different environments that are later used for model selection. The optimal number of selected samples from D LL and other hyperparameters is determined based on the worst environment accuracies among environments that are obtained from D MS . By combining our novel sampling and validation strategy, we aim to provide a robust linear classifier h ϕ ∗ that significantly improves the accuracy of underrepresented groups without requiring group annotations of training or validation sets. Figure 1 illustrates the comprehensive workflow of the EVaLS methodology. Finally in Section 3.3, we provide theoretical support for the loss-based sampling procedure and its effectiveness.

3.1 Loss-Based Instance Sampling

Following previous works [5, 10, 9], we use the loss value as an indicator for identifying minority groups. We first evaluate classifier f on samples within D LL and choose k samples with the highest and lowest loss values for a given k . By combining these 2 k samples from each class, we construct a

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

balanced set D balanced , consisting of high-loss and low-loss samples (see Figure 1(c)). D balanced is then used for the training of the last layer of the model.

As depicted in Figure 2, the proportion of minority samples among various percentiles of samples with the highest loss values increases as we select a smaller subset of samples with the highest loss. This suggests that high and low-loss samples could serve as effective representatives of minority and majority groups, respectively. In Section 3.3, we offer theoretical insights explaining why this approach could lead to the creation of group-balanced data.

3.2 Partitioning Validation Set into Environments

Contrary to common assumptions and practices in the field, precise group labels for the validation set are not essential for training models robust to spurious correlations. Our empirical findings, detailed in Section 4, reveal that partitioning the validation set into environments that exhibit significant subpopulation shifts can be used for model selection. Under these conditions, the worst environment accuracy (WEA) emerges as a viable metric for selecting the most effective model and hyperparameters.

The concept of an environment , as frequently discussed in the invariant learning literature, denotes partitions of data that exhibit different distributions. A model that consistently excels across these varied environments, achieving impressive worst environment accuracy (WEA), is likely to perform equally well across different groups in the test set. Several methods for inferring environments with notable distribution shifts have been introduced [12, 22]. Environment Inference for Invariant Learning (EIIL) [12], leverages the predictions from an earlier trained ERM model to divide the data into two distinct environments that significantly deviate from the invariant learning principle proposed by Arjovsky et al. [18], thus creating environments with distribution shifts. Initially, EIIL is employed to split D MS into two environments. Subsequently, each environment is further divided based on sample labels, resulting in 2 ×|Y| environments. To measure the difference between the distribution of environments, we define group shift of a class as the absolute difference in the proportion of a minority group between two environments of that class. A higher group shift suggests a more distinct separation between environments. As detailed in the Appendix, environments inferred by EIIL demonstrate an average group shift of 28 . 7% over datasets with spurious correlation. Further information about EIIL and the group shift quantities for each dataset can be found in the Appendix.

We demonstrate that even more straightforward techniques, such as applying a random linear layer over the feature embedding space and distinguishing environments based on correctly and incorrectly classified samples of each class, can be effective to an extent in several cases (See Appendix E.2). It underscores that the feature space of a trained model is a valuable resource of information for identifying groups affected by spurious correlations. This supports the logic of previous research that employs clustering [23] or contrastive methods [24] in this space to differentiate between groups.

3.3 Theoretical Analysis

In this subsection, we provide theoretical insights into why loss-based sampling in a class can be utilized to create a balanced dataset of each group under sufficient conditions. We will show the close relationship between the existence of a balanced dataset and the difference between the minority vs. majority group means, calculated based on the logits of an ERM-trained classifier. Such logits are known to depend on spurious features. Hence the mentioned group mean difference is expected to be high if spurious features are present in the dataset.

Consider a binary classification problem with a cross-entropy loss function. Let logits be denoted as L . Because loss is a monotonic function of logits, the tails of the distribution of loss across samples are equivalent to that of the logits in each class.

We assume that in feature space (output of g θ ) samples from the minority and majority of a class are derived from Gaussian distributions. So, we can consider N ( µ min , σ 2 min ) and N ( µ maj , σ 2 maj ) as the distribution of minority and majority samples in logits space.

Proposition 3.1 (Feasiblity Of Loss-based Group Balancing) . Suppose that L is derived from the mixture of two distributions N ( µ min , σ 2 min ) and N ( µ maj , σ 2 maj ) with proportion of ε and 1 -ε , respectively, where ε ≤ 1 2 . Under sufficient (see App.C) and necessary conditions on µ min , µ maj , σ min and σ maj including inequality 1, there exists α and β such that restricting L to the α -left and

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251





















































(a) Waterbirds











































(b) CelebA

β -right tails of its distribution results in a group-balanced distribution; in which both components 224 are equally represented. 225

We provide an outline for proof of Proposition 3.1 here and leave the complete and formal proof and 226 also exact bounds to Appendix C. We also analyze the conditions and effects of spurious correlation 227 in satisfying these conditions. To proceed with the outline we first define a key concept to outline our 228 proof. 229

Definition 3.1 (Proportional Density Difference) . For any interval I = ( a, b ] and a mixture distribution εP 1 ( x ) + (1 -ε ) P 2 ( x ) , the proportional density difference is defined as the difference of accumulation of two component distributions in the interval I and is denoted by ∆ ε P mixture ( I ) .

Proof outline Our proof proceeds with three steps. First, we reformulate the theorem as an equality of left- and right-tail proportional distribution differences. In other words, we show that the more mass the minority distribution has on one tail, the more mass the majority distribution must have on the other tail. Afterward, supposing µ min < µ maj WOLG, we propose a proper range for β values on the right tail. We show that when σ maj ≤ σ min, values for α trivially exist that can overcome the imbalance between the two distributions. In the last step, for the case in which the variance of the majority is higher than the minority, we discuss a necessary and sufficient condition for the existence of α and β based on the left-tail proportional density difference using the properties of its derivative with respect to α .

Condition 1 suggests that for a given degree of spurious correlation ϵ and variations σ maj , σ min, an essential prerequisite for the efficacy of loss-based sampling is a sufficiently large disparity between the mean distributions of minority and majority samples, denoted by ∥ µ maj -µ min ∥ 2 . This indicates that the groups should be distinctly separable in the logits space.

Although the parameters α and β are theoretically established under certain conditions, their actual values are undetermined. Therefore, validation data is necessary to ascertain them. For practicality and simplicity in this study, we consider that α = β and explore its corresponding sample number (the count of high- and low-loss samples) from a predefined set of possibilities. By leveraging the worst environment accuracy, as elaborated in Section 3.2, we identify the optimal candidate that ensures uniform accuracy across all environments.

4 Experiments

In this section, we evaluate the effectiveness of our proposed method through comprehensive experiments on multiple datasets and compare it with various methods and baselines. We begin by briefly

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

Core Dimension

3

2

1

0

1

2

3

Spurious 1 Dimension

3

2

1

0

1

2

Spurious 2 Dimension

Class 1

Class 2

Spurious 1 = 1, Spurious 2 =  -1

Spurious 1 = -1, Spurious 2 =  -1

Spurious 1 = 1 Spurious 2 =  1

Spurious 1 = -1 Spurious 2 =  1

Spurious

1 Dimension

2

(a)

Core Dimension

3

2

1

0

1

2

3

Spurious 1 Dimension

3

2

1

0

1

2

Spurious 2 Dimension

(b)

Core Dimension

3

2

1

0

1

2

3

Spurious 1 Dimension

3

2

1

0

1

2

Spurious 2 Dimension

(c)

Text

Vehicle

Animal

Red

Green

MNIST

MNIST

FMNIST

FMNIST

(d)

describing evaluation datasets and then introduce baselines and comparative methods. Finally, we report and fully explain the results.

Datasets Our method, along with other baselines, is evaluated on Waterbirds [7], CelebA [13], UrbanCars [14], CivilComments [16], and MultiNLI [15]. As per the study by Yang et al. [4], Waterbirds, CelebA, and UrbanCars among these datasets exhibit spurious correlation. Among the rest, CivilComments has class and attribute imbalance, whereas MultiNLI exhibits attribute imbalance. For additional details on the datasets, please refer to the Appendix.

Baselines Wecompare our method with four baselines in addition to standard ERM. GroupDRO [7] trains a model on the data with the objective of minimizing its average loss on the minority samples. This method requires group labels of both the training and validation sets. DFR [1] argues that models trained with ERM are capable of extracting the core features of images. Thus, it first trains a model with ERM, and retrains only the last linear classifier layer on a group-balanced subset of the validation or the held-out training data. While DFR reduces the number of group-annotated samples, it still requires group labels in the training phase. GroupDRO + EIIL [12] infers environments of the training set and trains a model with GroupDRO on the inferred environments. JTT [5] first trains a model with ERM on the dataset, and then retrains it on the dataset by upweighting the samples that were misclassified by the initial ERM model. AFR [9] trains a model with ERM on a portion of the training set, and retrains the classifier on the weighted held-out training data. The weights assigned to retraining samples are based on the loss of the ERM model, upweighting samples from the minority groups. Group DRO + EIIL, JTT and AFR remove the reliance on group annotation in the training phase. However, unlike our method, they all require group labels for model selection.

Setup Similar to all the works mentioned in Section 4, we use ResNet-50 [25] pretrained on ImageNet [26] for image classification tasks. We used random crop and random horizontal flip as data augmentation, similar to [1]. For a fair comparison with the baselines, we did not employ any data augmentation techniques in the process of retraining the last layer of the model. For the CivilComments and MultiNLI, we use pretrained BERT [27] and crop sentences to 220 tokens length. In EvaLS, we use the implementation of EIIL by spuco package [28] for environments inference on the model selection set with 20000 steps, SGD optimizer, and learning rate 10 -2 for all datasets.

Model selection and hyper-parameter fine-tuning are done according to the worst environment(or 280 group if annotations are assumed to be available) accuracy on the validation set. For each dataset, 281 we assess the performance of our model in two cases: fine-tuning the ERM classifier or retraining it. 282 For all datasets except MultiNLI, retraining yielded better validation results. We report the results 283 of our experiments in two settings: (i) EVaLS, which incorporates loss-based instance sampling for 284 training the last layer, and environment inference for model selection. (ii) EVaLS-GL, similar to 285 EVaLS except in using ground-truth group labels for model selection. For more details on the ERM 286 training and last layer re-training hyperparameters refer to the Appendix. 287

4.1 Results 288

289

290

291

292

293

294

The results of our experiments along with the reported results on GroupDRO [7], DFR [1], JTT [5], and AFR [9] on five datasets are shown in Table 1. The reported results for GroupDRO, DFR, JTT, and AFR except those for the UrbanCars are taken from Qiu et al. [9]. For EIIL+Group DRO, the results are reported from Zhang et al. [24]. We report only the worst group accuracy of methods in Table 1. The average group accuracies are documented in the Appendix. The Group Info column shows whether group annotation is required for training or model selection entry for each method.

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

When compared to other methods with the same level of supervision, EVaLS-GL outperforms on four of the five datasets, achieving near-optimal worst group accuracy on Waterbirds, demonstrating the effectiveness of loss-based sample selection compared to the weighting scheme in AFR [9]. Given that AFR employs exponential weights with a temperature parameter to assign a positive weight to all samples, proportional to the model's assigned probability of the correct class, an increase in the number of low-loss samples will lead to a corresponding rise in their cumulative weight. Consequently, in situations where spurious correlation is high and an uptick in majority samples leads to a greater proportion of low-loss over high-loss samples, determining an appropriate parameter becomes challenging.

The comparison between EVaLS and Group DRO + EIIL indicates that when environments are available instead of groups, our method, which uses environments solely for model selection and utilizes loss-based sampling, is more effective than GroupDRO, a potent invariant learning method, which uses this annotation for training.

Regarding the UrbanCars, which contains an un-annotated spurious attribute, Li et al. [14] has shown that shortcut mitigation methods often struggle to address multiple shortcuts simultaneously. Notably, techniques such as DFR [1] which are designed to reduce reliance on a specific shortcut feature, cannot make the model robust to an unknown shortcut. In contrast, our experiments suggest that loss-based methods can mitigate the impact of both labeled and unlabeled shortcut features more effectively. Also, in the case of CivilComments, which is viewed as a benchmark for class imbalance, EVaLS-GL exceeds all prior methods, even those with complete group annotation, thanks to the class balancing for the training of the last layer.

Our evaluation of EVaLS is based on the spurious correlation benchmarks. This is because, in other instances of subpopulation shift, the attributes that differ across groups are not predictive of the label, thereby reducing the visibility of these attributes' effects in the model's final layers [29]. Consequently, EIIL, which depends on output logits for prediction, might not effectively separate the groups. This observation is further supported by our findings related to the degree of group shift between the environments inferred by EIIL for each class in the CivilComments and MultiNLI datasets. The average group shift (defined in the Section 3.2) in the environments of the minority class of CivilComments is only 5 . 6 ± 0 . 8 % . Also, environments associated with Classes 1 and 2 in MultiNLI show only 1 . 1 ± 0 . 3 % and 1 . 9 ± 1 . 0 % group shift respectively. More results and ablation studies can be found in the Appendix.

Mitigating Multiple Shortcut Attributes To evaluate the performance of our method in the case of unknown spurious correlations, we train a ResNet-18 [25] model on the Dominoes-CMF dataset. We apply DFR [1], EVaLS-GL, and EVaLS on top of the trained ERMs to assess their ability to mitigate multiple shortcuts. For the last layer training set, we consider the MNIST/Fashion-MNIST feature as the known group label, and the color as the unknown attribute. The results are shown in Table 2. To clarify, we calculate the worst-group accuracy on the validation set considering only the label of one shortcut, i.e. , the lowest accuracy among the four groups based on the combination of the target label and the single known shortcut label. Note that EVaLS does not require group annotations.

Our results confirm findings by Li et al. [14], suggesting that methods using group labels mitigate reliance on the known shortcut but not necessarily on the unknown one. EVaLS-GL mitigates this phenomenon using its loss-based sampling approach, but surprisingly EVaLS even outperforms EVaLS-GL. Combining a loss-based sampling approach for last layer training and environment-based model selection, results in a completely group-annotation-free method in a multi-shortcut setting and successfully re-weights features to perform well with respect to both spurious attributes.

5 Discussion 340

341

342

343

344

345

346

347

348

349

This study presents EVaLS, a novel approach to improve robustness to spurious correlations with zero group annotation. EVaLS uses loss-based sampling to create a balanced training dataset that effectively disrupts spurious correlations and employs EIIL to infer environments for model selection. We also explore situations with multiple spurious correlations where not all spurious factors are known. In this context, we introduce Dominoes-CMF, a dataset in which two factors are spuriously correlated with the label, but only one is identified. Our findings suggest that EVaLS attains nearoptimal worst test group accuracy on spurious correlation datasets. We also present EVaLS-GL, which needs group labels only for model selection. Our empirical tests on various datasets demonstrate EVaLS-GL outperforms state-of-the-art methods requiring group data during evaluation or training.

350

351

352

353

354

355

356

357

358

359

Note that this paper remains consistent with the findings of Lin et al. [30]. Our approach does not involve identifying spurious attributes without auxiliary information. Instead, the objective is to make a trained model robust against its reliance on shortcuts. Specifically, conditioning on what a trained model learns, we ascertain that both the loss value and the model's feature space are instrumental in mitigating shortcuts and effectuating notable shifts among groups.

EVaLS and EVaLS-GL may struggle with small datasets due to a low number of selected samples for the last layer training. Also, as environment inference from the last layer features is not effective for all types of subpopulation shifts, EVaLS is limited to datasets with spurious correlation. Similar to other methods in the field, EVaLS prioritizes the worst group accuracy at the cost of less average accuracy. Additionally, a notable variance has been observed in some of our experiments.

EVaLS represents a significant advancement in the development of methods for enhancing model 360 fairness and robustness without prior knowledge about group annotations. Future work could explore 361 developing environment inference methods effective for other types of subpopulation shift, such as 362 attribute and class imbalance. 363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

References

Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=Zb6c8A-Fghk .

Tyler LaBonte, Vidya Muthukumar, and Abhishek Kumar. Towards last-layer retraining for group robustness with fewer annotations. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=kshC3NOP6h .

Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning , pages 1929-1938. PMLR, 2018.

Yuzhe Yang, Haoran Zhang, Dina Katabi, and Marzyeh Ghassemi. Change is hard: A closer look at subpopulation shift. In International Conference on Machine Learning , 2023.

Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 6781-6792. PMLR, 18-24 Jul 2021. URL https://proceedings. mlr.press/v139/liu21f.html .

Yu Yang, Eric Gan, Gintare Karolina Dziugaite, and Baharan Mirzasoleiman. Identifying spurious biases early in training through the lens of simplicity bias. ArXiv , abs/2305.18761, 2023. URL https://api.semanticscholar.org/CorpusID:258967752 .

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations , 2019.

Junhyun Nam, Jaehyung Kim, Jaeho Lee, and Jinwoo Shin. Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. In International Conference on Learning Representations , 2021.

Shikai Qiu, Andres Potapczynski, Pavel Izmailov, and Andrew Gordon Wilson. Simple and fast group robustness by automatic feature reweighting. In International Conference on Machine Learning , pages 28448-28467. PMLR, 2023.

Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances in Neural Information Processing Systems , 33:20673-20684, 2020.

Fahimeh Hosseini Noohdani, Parsa Hosseini, Aryan Yazdan Parast, HamidReza Yaghoubi Araghi, and Mahdieh Soleymani Baghshah. Decompose-and-compose: A compositional approach to mitigating spurious correlation. CoRR , abs/2402.18919, 2024. doi: 10.48550/ ARXIV.2402.18919. URL https://doi.org/10.48550/arXiv.2402.18919 .

Elliot Creager, Joern-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 2189-2200. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/ creager21a.html .

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. 2015 IEEE International Conference on Computer Vision (ICCV) , pages 3730-3738, 2014. URL https://api.semanticscholar.org/CorpusID:459456 .

Zhiheng Li, Ivan Evtimov, Albert Gordo, Caner Hazirbas, Tal Hassner, Cristian Canton Ferrer, Chenliang Xu, and Mark Ibrahim. A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 20071-20082, June 2023.

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426 , 2017.

Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference , pages 491-500, 2019.

Matteo Pagliardini, Martin Jaggi, François Fleuret, and Sai Praneeth Karimireddy. Agree to disagree: Diversity through disagreement for better transferability. In The Eleventh International Conference on Learning Representations , 2022.

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization, 2020.

Alex Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009. URL https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf .

Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http: //yann.lecun.com/exdb/mnist/ .

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017. URL http://arxiv. org/abs/1708.07747 . cite arxiv:1708.07747Comment: Dataset is freely available at https://github.com/zalandoresearch/fashion-mnist Benchmark is available at http://fashionmnist.s3-website.eu-central-1.amazonaws.com/.

Jiashuo Liu, Zheyuan Hu, Peng Cui, Bo Li, and Zheyan Shen. Heterogeneous risk minimization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 68046814. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/liu21h. html .

Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. Advances in Neural Information Processing Systems , 33:19339-19352, 2020.

Michael Zhang, Nimit Sharad Sohoni, Hongyang R. Zhang, Chelsea Finn, and Christopher Ré. Correct-n-contrast: A contrastive approach for improving robustness to spurious correlations. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications , 2021. URL https://openreview.net/forum?id=Q41kl_DwS3Y .

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016. doi: 10.1109/CVPR.2016.90.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology. org/N19-1423 .

Siddharth Joshi, Yu Yang, Yihao Xue, Wenhan Yang, and Baharan Mirzasoleiman. Towards mitigating spurious correlations in the wild: A benchmark & a more realistic dataset. ArXiv , abs/2306.11957, 2023. URL https://api.semanticscholar.org/CorpusID: 259211935 .

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview. net/forum?id=APuPRxjHvZ .

Yong Lin, Shengyu Zhu, Lu Tan, and Peng Cui. Zin: When and how to learn invariance without environment partition? In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 24529-24542. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/ paper/2022/file/9b77f07301b1ef1fe810aae96c12cb7b-Paper-Conference.pdf .

Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. ArXiv , abs/2108.13624, 2021. URL https://api.semanticscholar.org/CorpusID:237364121 .

Seonguk Seo, Joon-Young Lee, and Bohyung Han. Information-theoretic bias reduction via causal view of spurious correlation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 2180-2188, 2022.

Yuzhen Mao, Zhun Deng, Huaxiu Yao, Ting Ye, Kenji Kawaguchi, and James Zou. Last-layer fairness fine-tuning is simple and effective for neural networks. arXiv preprint arXiv:2304.03935 , 2023.

David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 5815-5826. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/ krueger21a.html .

Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. In International Conference on Machine Learning , pages 18347-18377. PMLR, 2022.

Faruk Ahmed, Yoshua Bengio, Harm van Seijen, and Aaron Courville. Systematic generalisation with group invariant predictions. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=b9PoimzZFJ .

Matteo Pagliardini, Martin Jaggi, François Fleuret, and Sai Praneeth Karimireddy. Agree to disagree: Diversity through disagreement for better transferability. arXiv preprint , arXiv:2202.04414, 2022.

Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems , 33:9573-9585, 2020.

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 5637-5664. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/koh21a.html .

Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV) , September 2018.

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

A Related Work

Robustness to spurious correlation is a critical concern across various machine learning subfields. It is a form of out-of-distribution generalization [31] where the distribution shift arises from the disproportionate representation of minority groups-those instances that are devoid of the correlated spurious patterns associated with their labels [4]. The issue of spurious correlation also intersects with the discourse on fairness in machine learning. [32, 33].

Past studies have proposed a range of strategies to mitigate the models' reliance on spurious correlation. Broadly speaking, these methods can be categorized according to the degree of supervision they require regarding group labels.

Invariant learning (IL) methods [18, 34, 35] operate under the assumption of having access to a collection of environments that comprise group shift. By imposing invariant conditions on these environments, IL methods strive to create classifiers robust against group-sensitive features. IRM [18] is designed to learn a feature extractor, which, when utilized, guarantees the existence of a classifier that would be optimal in all training environments. VREx [34] aims to decrease the risk variance among different training environments. PGI [36] works by minimizing the distance between the expected softmax distribution of labels, conditioned on inputs across both majority and minority environments. Lastly, Fishr [35] focuses on bringing the variance of risk gradients closer together across different training environments. For scenarios which environments are not available, environment inference methods [12, 22] are used to obtain a set of environments. Creager et al. [12] introduce environment inference for invariant learning (EIIL), which tries to partition samples into two groups such that the objective of IRM [18] is maximized. HRM [22] aims to optimize both an environment inference module and an invariant prediction module jointly, with the goal of achieving an invariant predictor.

When group annotations are accessible, various methods leverage this information to equalize the impact of different groups on the model's loss. The Group Distributionally Robust Optimization (GDRO) approach [7], for instance, focuses on optimizing the loss for the worst-performing group during training. Kirichenko et al. [1] has shown that models can still learn and extract core data features even in the presence high spurious correlation. Consequently, They suggest that retraining just the last layer of a model initially trained with Empirical Risk Minimization (ERM) can effectively reduce reliance on spurious correlation for predicting class labels. This method, termed Deep Feature Re-weighting (DFR), has been validated as not only highly effective but also significantly more efficient than earlier techniques that necessitated retraining the full model [8, 7]. However, availability of group annotations is considered a serious restrictive assumption.

Several recent studies have endeavored to enhance model robustness against spurious correlation, even in the absence of group annotations [5, 24, 9, 2, 6]. Liu et al. [5] introduce a two-stage method that involves training a model using ERM for a number of epochs before retraining it to give more weight to misclassified samples. The study by Zhang et al. [24] employs the same two-stage training process, but with a twist for the second stage: they utilize contrastive methods. The goal is to bring samples from the same class but with divergent predictions closer in the feature space, while simultaneously increasing the separation between samples from different classes that have similar predictions. Another method, known as automatic feature reweighting (AFR) [9], reweights the last layer of an ERM-pretrained model to favor samples that the original model was less accurate on. LaBonte et al. [2] refine the last layer of an ERM-trained model through class-balanced finetuning, identifying challenging data points by comparing the classifier's predictions with those of an earlystopped version. While these methods have significantly reduced the reliance on group annotations, some are still required for validation and model selection. This remains a constraint, particularly when the spurious correlation is completely unknown.

For making a trained model robust to spurious correlation with zero group annotations, recently, LaBonte et al. [2] have empirically demonstrated that the class-balanced retraining of a model pretrained with ERM can effectively improve the WGA for certain datasets. However, this approach fails in datasets with a high degree of spurious correlation.

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

B Environment Inference for Invariant Learning

̸

Consider the training dataset D tr = { ( x ( i ) , y ( i ) ) | x ( i ) ∈ X , y ( i ) ∈ Y} , where X and Y represent the input and output spaces, respectively. This dataset can be partitioned into different environments E tr = { e 1 , ..., e n } , such that for any i = j , the data distribution in e i and e j differs. The objective of invariant learning is to train a predictor that performs consistently across all environments in E tr . Under certain conditions, this predictor is also expected to perform well on e tst , a test environment with a distribution distinct from the training data. Invariant Risk Minimization (IRM) [18] approaches this problem by learning a feature extractor Φ( . ) such that a classifier ω ( . ) exists, where ω ◦ Φ( . ) performs consistently across all training environments. The practical implementation of the IRM objective is to minimize

where ¯ ω is a constant scalar with a value of 1 . 0 , λ is a hyperparameter, and R e ( f ) = E ( x,y ) ∼ p e [ l ( f ( x ) , y )] is referred to as the risk on environment e .

In real-world scenarios, training environments might not always be available. To address this, Environment Inference for Invariant Learning (EIIL) [12] partitions samples into two environments in a way that maximizes the objective in Eq 2.

During the training phase, the EIIL algorithm replaces the hard assignment of environments to samples with a soft assignment q i ( e ) = p ( e | ( x ( i ) , y ( i ) ) , where q i is learnable. Consequently, the relaxed version of the risk function is defined as ˜ R e (Φ) = 1 N ∑ N i q i ( e )[ l (Φ( x ( i ) ) , y ( i ) )] . Given a model Φ that has been trained with ERM on the dataset, EIIL optimizes

As discussed in Creager et al. [12], using a biased base model Φ could lead to environments exhibiting 575 varying degrees of spurious correlation. During the inference phase, the soft assignment is converted 576 to a hard assignment. The average group shift between the inferred environments using EIIL is 577 illustrated in Table 3. 578

C Theoretical Analysis 579

580

581

582

In this section, we establish a more formal description of loss-based sampling for balanced dataset creation and then prove it. We thoroughly analyze the close relationship between the availability of the balanced dataset and the gap between spurious features of minority and majority groups.

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

Consider a binary classification problem with a cross-entropy loss function. Let logits be denoted as L . Because loss is a monotonic function of logits, the tails of the distribution of loss across samples are equivalent to that of the logits in each class. We assume that in feature space (output of g θ ) samples from the minority and majority of a class are derived from Gaussian distributions N ( h min , t 2 min I d ) and N ( h maj , t 2 maj I d ) , respectively. Before diving into the group balance problem we initially show that the distribution of minority and majority samples in the logit space (output of h ϕ ) are Gaussian too.

Lemma C.1 (Gaussain Distribution of Logits) . if Z ∼ N ( h, t 2 I d ) in feature space and W ∈ R d then logits L = ⟨ W,Z ⟩ ∼ N ( Wh, t 2 ∥ W ∥ 2 )

Consider the linear combination L = ⟨ W,Z ⟩ = W T Z , where W ∈ R d which is a univariate gaussian.

To find the distribution of L , we need to determine its mean and variance.

Mean of L

Therefore, the mean of L is Wh .

2. Variance of L :

The variance of L can be computed using the properties of covariance. Recall that if Z ∼ N ( h, t 2 I d ) , then the covariance matrix of Z is t 2 I d .

The variance of the linear combination L = W T Z is given by: 601

Given Cov( Z ) = t 2 I d , we have:

where ∥ W ∥ denotes the Euclidean norm of W .

Combining the mean and variance results, we conclude that L is normally distributed with mean Wh and variance t 2 ∥ W ∥ 2 :

Thus, we have proved that if Z ∼ N ( h, t 2 I d ) , then the logits L = ⟨ W,Z ⟩ follow the distribution 606 N ( Wh,t 2 ∥ W ∥ 2 ) . 607

From now on, we consider N ( µ min , σ 2 min ) and N ( µ maj , σ 2 maj ) as the distribution of minority and 608 majority samples in logits space. 609

Next, we prove the more formal version of the main proposition 3.1 which describes the existence of 610 a balanced dataset, only after we define a key concept, proportional density difference (illustrated in 611 figure 4) to outline our proof. 612

602

603

604

605

Definition C.1 (Proportional Density Difference) . For any interval I = ( a, b ] and a mixture distribution εP 1 ( x ) + (1 -ε ) P 2 ( x ) , proportional density difference is defined by the difference of accumulation of two component distributions in the interval I and is denoted by ∆ ε P mixture ( I ) .

Definition C.2 (Tail Proportional Density Difference) . For a mixture distribution εP 1 ( x ) + (1 -613 ε ) P 2 ( x ) , we define tail L ( α ) as ∆ ε P mixture ( ( -∞ , α ] ) and tail R ( β ) as -∆ ε P mixture ( ( β, + ∞ ) ) . 614

Corollary C.1.

where F 1 and F 2 are CDF of two component distributions. 615

Proportional Density Difference

I

/epsilon1

=0.4

majority

minority

(a)

β

α

tail

R

(

β

)

tail

L

(

α

)

/epsilon1

=0.4

majority

minority

(b)

Proposition C.1 (Feasiblity Of Loss-based Group Balancing) . Suppose that L is derived from 616 the mixture of two distributions N ( µ min , σ 2 min ) and N ( µ maj , σ 2 maj ) with proportion of ε and 1 -ε , 617 respectively, where ε ≤ 1 2 . There exists α and β such that restricting L to the α -left and β -right tails 618 of its distribution results in a group-balanced distribution if and only if σ min ≥ σ maj or 619

and 620

621

622

where A = ( 1 2 σ 2 maj -1 2 σ 2 min ) , B = ( µ min σ 2 min -µ maj σ 2 maj ) and ∆ = ( µ min -µ maj ) 2 σ 2 min σ 2 maj -4 [ log ( σ maj σ min ) + log ( ϵ 1 -ϵ )][ 1 2 σ 2 maj -1 2 σ 2 min ] .

623

624

625

626

627

628

629

630

631

Proof outline Our proof proceeds with three steps. First, we reformulate the theorem as an equality of left- and right-tail proportional distribution differences. In other words, we show that the more mass the minority distribution has on one tail, the more mass the majority distribution must have on the other tail. Afterward, supposing µ min < µ maj WOLG, we propose a proper range for β values on the right tail. We show that when σ maj ≤ σ min, values for α trivially exist that can overcome the imbalance between the two distributions. In the last step, for the case in which the variance of the majority is higher than the minority, we discuss a necessary and sufficient condition for the existence of α and β based on the left-tail proportional density difference using the properties of its derivative with respect to α .

Step 1 Reformulating the problem based on proportional distribution difference. 632

We introduce a utility random variable Logit Value Tier as T , which is defined as a function of a 633 random variable L . 634

We can rewrite the problem in formal form as finding an α and β which satisfies the following 635 equation: 636

̸

Equation 5 now can be rewritten to a more suitable form: 637

̸

̸

̸

̸

̸

̸

̸

̸

̸

We can see the left side of equation 17 is just a function of alpha . The same goes for the right side of 638 the equation which is a function of β . 639

Rewriting the left side of the equation as tail L ( α ) and right side as tail R ( β ) , the problem is now 640 reduced to finding an α and β that satisfies 641

̸

642

643

644

645

646

647

648

̸

which is shown in figure 4.

Before reaching out to step two we discuss the properties of tail L and tail R in Lemma C.2.

Lemma C.2. tail L ( α ) and tail R ( β ) are continuous functions and lim α →-∞ tail L ( α ) = 0 , lim α → + ∞ tail L ( α ) = 2 ε -1 < 0 , lim β → + ∞ tail R ( β ) = 0 and lim β →-∞ tail R ( β ) = 1 -2 ε > 0 .

Proof. Simply proved by the definition of tail functions and properties of CDF.

Step 2 Solving the equation 18 for simple cases.

Lemma C.3. tail R ( µ maj ) > 1 2 -ε ≥ 0 649

Proof.

650

651

652

653

654

655

Corollary C.2. Because tail R is continuous and lim β → + ∞ tail R ( β ) = 0 , based on the mean value theorem, any value between zero and (1 -2 ε ) 2 is obtainable by selecting a β in [ µ 2 , + ∞ ) .

According to the previous corollary C.2 finding a positive tail L ( α ) will satisfy our need. to find a suitable point, we employ derivatives and properties of relative PDFs to maximize tail L ( α ) and find a positive value.

The term [ f min ( α ) f maj ( α ) -1 -ε ε ] has the same sign with derivative of tail L ( α ) , also it's roots are critical 656 points of tail L , analyzing characteristics of log f min ( α ) f maj ( α ) is the key insight to find a proper α value. 657

Because lim α →-∞ tail L ( α ) = 0 and lim β → + ∞ tail R ( β ) < 0 to have a positive tail L ( α ) we need to have an interval which d tail L ( α ) d α is positive, for a second degree polynomial like ax 2 + bx + c to have positive value, either a ≥ 0 or ∆ > 0 , in our case a is ( 1 σ 2 maj -1 σ 2 min ) . if σ min ≥ σ maj then a ≥ 0 and the minority CDF function will dominate the majority CDF function in the left-side tail and by choosing a negative number with big enough absolute value for alpha and tail L ( α ) will be positive.

Step 3 Solving equation 18 for special case σ min < σ maj In case of σ min ≤ σ maj, having ∆ > 0 is a necessary condition, also derivative of tail L ( α ) is only positive in ( -b - √ ∆ 2 a , -b + √ ∆ 2 a ) so the maximum of tail L is either in -∞ or in -b + √ ∆ 2 a . Having tail L ( -b + √ ∆ 2 a ) > 0 next to ∆ > 0 condition, would be the necessary and also sufficient in this case.

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

=0.35

majority

minority

(a)

=0.4

majority

minority

(b)

=0.45

majority

minority

(c)

Next, we investigate properties of the conditions of the proposition C.1 in case of σ maj < σ min. Schematic interpretation of these conditions is presented in figure 6.

As equation 5 indicates, the minority group is not allowed to be too underrepresented. This especially has a direct relation with the difference of means. The more mean values of groups are different, the more imbalance can be mitigated through loss-based sampling. Mean value difference is especially affected by the spurious correlation, it escalates as the model relies on spurious correlation and also when the spurious features between groups are too different.

On the other hand condition 4 is more complex and doesn't have a simple closed form, we analytically describe its behaviors by fixating the means and calculating the valid values for ε . As the results show in figure 6, most of ε are feasible in for σ min < ∆ µ as we can see the possible region declines with an increase of σ min and valid ε values cease to exist.

S

1

S

2

>0

=0.27

majority

minority

(a)

0

1

2

3

4

5

σ

min

0

1

2

3

4

5

σ

maj

0

.

0000

0

.

1053

0

.

2105

0

.

3158

0

.

4211

0

.

5263

0

.

6316

0

.

7368

0

.

8421

0

.

9474

(b)

0

1

2

3

4

5

σ

min

0

1

2

3

4

5

σ

maj

0

.

0000

0

.

1053

0

.

2105

0

.

3158

0

.

4211

0

.

5263

0

.

6316

0

.

7368

0

.

8421

0

.

9474

(d)

0

1

2

3

4

5

σ

min

0

1

2

3

4

5

σ

maj

0

.

0000

0

.

1053

0

.

2105

0

.

3158

0

.

4211

0

.

5263

0

.

6316

0

.

7368

0

.

8421

0

.

9474

(c)

679

680

681

682

683

684

685

686

687

688

689

690

691

692

D Experimental Details

D.1 Complete Results

The complete results on Waterbirds, CelebA, and UrbanCars, in addition to complete results on CivilComments and MultiNLI are reported in Tables 4 and 5 respectively. The results for all methods except Group DRO + EIIL on all datasets except UrbanCars are reported by Qiu et al. [9]. The results for Group DRO + EIIL are taken from Zhang et al. [24]. Also, the results of our method and DFR are shown in Table 6

D.2 Dominoes-Colored-MNIST-FashionMNIST

Dominoes-Colored-MNIST-FashionMNIST (Dominoes-CMF) is a synthetic dataset. We adopt a similar approach to previous works [37, 38, 1] using a modified version of the Dominoes binary classification dataset. This dataset consists of images with the top half showing CIFAR-10 images [19], divided into two meaningful classes: vehicles (airplane, car, ship, truck) and animals (cat, dog, horse, deer). The bottom half displays either MNIST [20] images from classes { 0 -3 } or Fashion-MNIST [21] images from classes { T-shirt , Dress , Coat , Shirt } . The complex feature (top

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

half) serves as the core feature and the simple feature (bottom half) is linearly separable and correlated with the class label at 75%. Furthermore, inspired by the approaches in Zhang et al. [24], Arjovsky et al. [18], we intentionally introduce an additional spurious attribute by artificially coloring a subset of images in the following manner: 90% of the bottom half images in class c 1 are randomly assigned a red color, while 10% are assigned a green color, and vice versa for class c 2 . See Table 7 for more details about the dataset statistics.

D.3 Datasets

Waterbirds [7] The dataset comprises images of diverse bird species, classified into two categories: waterbirds and landbirds. Each image features a bird set against a backdrop of either water or land. Interestingly, the background scene acts as a spurious feature in this classification task. Waterbirds are primarily shown against water backgrounds, and landbirds against land backgrounds. Consequently, waterbirds on water and landbirds on land form the minority groups in the training data. It's important to note that the validation dataset for waterbirds is group-balanced, meaning birds from each class are equally represented against both water and land backgrounds. This dataset is mainly categorized as a spurious correlation dataset.

CelebA [13] is a widely used dataset in image classification tasks, featuring annotations for 40 binary facial attributes such as hair color, gender, and age. Hair color classification is particularly prominent in literature focusing on spurious correlation robustness. Notably, gender serves as a spurious attribute within this dataset, where a significant majority 94% of individuals with blond hair

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740





























































































































(b) UrbanCars

are women, while men with blond hair represent a minority group. In addition to spurious correlation in the class of blond hair, this dataset also exhibits class imbalance.

MultiNLI [15] dataset involves a text classification task focused on determining the relationship between pairs of sentences: contradiction, entailment, or neutral. Sentences containing negation words such as "no" or "never" are under-represented in all three classes, inducing attribute imbalance in the dataset. Figure 7 illustrates the distinct behavior of this dataset compared to other datasets that contain spurious attributes.

CivilComments [16] dataset, as part of the WILDS benchmark, involves a text classification task focused on labeling online comments as either "toxic" or "not toxic". Each comment is associated with 8 attributes, including gender (male, female), sexual orientation (LGBTQ), race (black, white), and religion (Christian, Muslim, or other), based on whether these characteristics are mentioned in the comment. While there is a small attribute imbalance in the dataset, it can categorized into datasets with class imbalance. In this paper, we use the implementation of the dataset by the WILDS package [39].

UrbanCars [14] is an image classification dataset with multiple shortcuts. Each image in the dataset consists of a car in the center of the image on a natural scene background, with another object to the right of the image. Images are labeled Urban or City according to the type of car present in the center. However, each of the backgrounds and the additional objects is highly correlated with the label. While the test set consists of 8 environments based on combinations of the core and two spurious patterns, the training and validation set consist of four groups, based on combinations of the label and only one of the shortcuts.

D.4 Training Details

ERM For Waterbirds and CelebA, we utilize the ResNet50 checkpoints available in the GitHub repository of Kirichenko et al. [1] as our base model. We use the ResNet-50 architecture provided by the torchvision package. In the case of CivilComments and MultiNLI, we adopt a similar approach to Kirichenko et al. [1], using BertForSequenceClassification.from_pretrained('bert-base-uncased', ...) from the transformers package. The model is trained using the AdamW optimizer with a learning rate of 10 -5 , weight decay of 10 -4 , and a batch size of 16 for a total of 5 epochs.

For the UrbanCars dataset, we adhere to the settings described in Li et al. [14], which involves 741 training a ResNet-50 model pretrained on ImageNet using the SGD optimizer with a learning rate 742 of 10 -3 , momentum of 0.9, weight decay of 10 -4 , and a batch size of 128 for 300 epochs. For the 743 Dominoes-CMF dataset, we train a ResNet18 model pretrained on ImageNet for 20 epochs with a 744 batch size of 128 and an SGD optimizer with a learning rate of 10 -3 , momentum of 0.9, and weight 745 decay of 10 -4 . 746

EVaLS and EVaLS-GL For every dataset, EIIL was utilized with a learning rate of 0 . 01 , a total of 747 20000 steps, and a batch size of 128. The last layer of the model was trained on all datasets using the 748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

Adam optimizer. A batch size of 32 and a weight decay of 10 -4 were used for all datasets. Our method was evaluated on the validation sets of each dataset, considering both fine-tuning and retraining of the last layer. For all datasets, with the exception of MultiNLI, retraining provided superior validation results. The specifics regarding the number of epochs and the ranges for hyperparameter search (including learning rate, l 1 -regularization coefficient ( λ ), and the number of selected samples ( k )) for each dataset are as follows:

· Waterbirds .

-epochs = 100,

-lr = 5 × 10 -4 ,

-λ ∈ { 0 , 0 . 01 , 0 . 02 , 0 . 03 , 0 . 04 , 0 . 05 , 0 . 06 , 0 . 07 , 0 . 08 , 0 . 09 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } ,

-k ∈ { 20 , 25 , 30 , 35 , 40 , 45 , 50 , 55 , 60 } .

· CelebA

-epochs = 50,

-lr = 5 × 10 -4 ,

-λ ∈ { 0 , 0 . 01 , 0 . 02 , 0 . 03 , 0 . 04 , 0 . 05 , 0 . 06 , 0 . 07 , 0 . 08 , 0 . 09 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 , 1 , 2 } ,

-k ∈ { 50 , 100 , 150 , 200 , 250 , 300 } .

· UrbanCars

-epochs = 100,

-lr ∈ { 5 × 10 -4 , 10 -3 },

-λ ∈ { 0 , 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 1 } ,

-k ∈ { 10 , 20 , 30 , 50 , 63 } .

· CivilComments

-epochs = 50,

-lr = 5 × 10 -4 ,

-λ ∈ { 0 , 0 . 01 , 0 . 02 , 0 . 03 , 0 . 04 , 0 . 05 , 0 . 06 , 0 . 07 , 0 . 08 , 0 . 09 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 , 1 , 2 } ,

-k ∈ { 500 , 750 , 1000 , 1250 , 1500 } .

· MultiNLI

-epochs = 200,

-lr ∈ { 10 -2 , 10 -3 } ,

-λ ∈ { 0 , 0 . 01 , 0 . 02 , 0 . 03 , 0 . 04 , 0 . 05 , 0 . 06 , 0 . 07 , 0 . 08 , 0 . 09 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } ,

-k ∈ { 20 , 30 , 40 , 50 , 60 , 75 , 100 , 125 , 150 , 200 , 250 , 300 } .

782

783

E Ablation Study

E.1 Use of EIIL with DFR and AFR

We conducted an ablation study to investigate the impact of using environments inferred from EIIL on 784 model selection. Specifically, we benchmarked the performance of DFR and AFR with EIIL-inferred 785 groups. The results, presented in Table 9, demonstrate the effectiveness of incorporating EIIL-inferred 786 groups in model selection. The results show that while EIIL-inferred groups reduce the performance 787 compared to ground-truth annotations for model selection, they still can be effective for robustness to 788 an extent. Moreover, EVaLS outperforms these two methodw when using EIIL inferred environments. 789

E.2 Other Group Inference Methods

In addition to EIIL, other group inference methods could be utilized for partitioning the model selection set into environments.

Error Splitting JTT [5] partitions data into two correctly classified and misclassified sets based on the predictions of a model trained with ERM. We split each of these two sets based on labels of samples, obtaining |Y| × 2 environments.

Random Classifier Splitting uses a random classifier to classify features obtained from a model trained with ERM into correctly classified and misclassified sets. Similar to error splitting, we split the sets based on group labels. The difference between error splitting and random classifier splitting is solely in the reinitialization of the classification layer.

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

The results for EVaLS-ES (EVaLS+Error Sampling) and EVaLS-RC (EVaLS+Random Classifier) are shown in Table 10. One limitation of error splitting is that in datasets with noisy labels or corrupted images, samples that an ERM model misclassifies may not always belong to minority groups. In these situations, choosing models based on their accuracy on corrupted data could lead to the selection of models that are not robust to spurious correlations. This is demonstrated by the results of EVaLS-ES on the CelebA dataset.

This shortcoming of error splitting can be alleviated by employing a random classifier instead of 806 the ERM-trained one. Due to the feature-level similarity between minority and majority samples in 807 datasets affected by spurious correlation [23, 1, 29], it is expected that the classifier can differentiate 808 between the groups to some extent. As shown in Table 10, surprisingly, EVaLS-RC produces results 809 that are generally comparable to EVaLS. However, the performance of this method may have high 810 variance, depending on the different initializations of the classifier. 811

812

813

814

815

816

817

818

819

820

821

F Societal Impacts

Real-world datasets often encapsulate social biases that stem from entrenched stereotypes and historical discrimination, affecting various groups such as genders and races. Machine learning methods, which learn the correlation between patterns in input data and their targets (e.g., labels in a classification task) [40], inadvertently absorb this bias. This unintended consequence leads to fairness issues in many applications. While strategies to mitigate such biases have been proposed (as discussed comprehensively in Section A), societal biases are not always known and determined. We believe that our work, as it addresses these unidentified biases, takes a significant step towards making machine learning fairer for our society.

G Computational Resources

Each experiment was conducted on one of the following GPUs: NVIDIA A100 with 80G memory, 822 NVIDIA Titan RTX with 24G memory, Nvidia GeForce RTX 3090 with 24G memory, and NVIDIA 823 GeForce RTX 3080 Ti with 12G memory. 824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The scope of the effectiveness and main claims are clearly demonstrated in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in the discussion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the lemmas and propositions are stated upon exact definitions, assumptions and conditions. All the theorems, formulas, and proofs in the paper are numbered and cross-referenced.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The training precedure is described accurately and all the training details and hyperparameters required for reproducing the results are provided.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Codes and information of datasets that are constructed or reused in the paper are anonymized and included in the main paper and supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The training details, hyperparameters, model selection criteria, etc. have are written in the Appendix and the data and metadata have been provided in our code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All tables report standard deviation and how it was computed and the plot contains error bar (also by standard deviation).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

981

982

983

984

985

986

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: The paper does provide details about the hardware used for the experiments. However, since experiments were done on different hardwares, the computational resources needed for each individual experiment are not documented.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All codes and rules have been thoroughly reviewed and checked, with no instances of non-compliance found.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In the Social Impacts section we discuss that our work can significantly 1030 contribute to fairness in machine learning. We did not find any negative social impacts of 1031 our work. 1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

1044

1045

1046

1047

1048

1049

1050

1051

1052

1053

1054

1055

1056

1057

1058

1059

1060

1061

1062

1063

1064

1065

1066

1067

1068

1069

1070

1071

1072

1073

1074

1075

1076

1077

1078

1079

1080

1081

1082

1083

1084

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Every asset that we utilized for our implementations have been appropriately referenced, both within the paper itself and in the code (if needed). Although we did not specify the names of their respective licenses, you can find these details on the webpages we've cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

1085

1086

1087

1088

1089

1090

1091

1092

1093

1094

1095

1096

1097

1098

1099

1100

1101

1102

1103

1104

1105

1106

1107

1108

1109

1110

1111

1112

1113

1114

1115

1116

1117

1118

1119

1120

1121

1122

1123

1124

1125

1126

1127

1128

1129

1130

1131

1132

1133

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

1134

Justification: The paper does not involve crowdsourcing nor researh with human subjects.

Guidelines: 1135

1136

1137

1138

1139

1140

1141

1142

1143

1144

1145

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of subpopulation generalization, also known as spurious correlations. Building on the Last Layer Retraining (DFR) method, it removes the constraints on a small subset of annotations. The paper introduces the Environment-based Validation and Loss-based Sampling (EVaLS) method. Unlike DFR, EVaLS divides the validation set $D^{val}$ into two parts: (1) $D^{LL}$,  where losses from an ERM-trained model are used as a proxy for identifying minority groups for retraining, and (2) $D^{MS}$, where environment inference methods are used for partitioning environments. The paper presents theoretical insights and empirical results demonstrating the effectiveness of EVaLS.\n\nSTRENGTHS:\n* The paper is well-structured and presented in a clear and organized manner, making it easy to comprehend and follow along.\n* The proposed method is simple but effective and explores a relatively challenging area in existing literature (*i.e.* subgroup generalization without group annotations). \n* The authors provide some theoretical analysis to support their claims.\n\nWEAKNESSES:\n* The novelty and contribution of the proposed method may be limited for the following reasons: 1) The paper combines multiple previously proposed methods (*i.e.* DFR [1], EIIL [2]) all at once, which inherently guarantees a nontrivial performance; (2) The primary technical contribution, at least from my perspective, is the loss-based sampling, which has been already explored extensively in the noisy label literature and has been used as tools for pseudo-labeling. \n* The paper fails to discuss recently proposed methods that also require no group annotations, such as SELF [3], BAM [4], and BPA [5]. In particular, SELF is also a direct follow-up of DFR. The authors are encouraged to discuss the limitations and strengths of loss-based schemes against the class-based schemes advocated by SELF and BAM.\n* More analyses can be included to provide further understanding of the selected loss-based samples. For example, given a threshold, how much percent of the high-loss and low-loss samples are indeed the minority and majority samples and how does this percentage change with the threshold?\n\n[1] Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations, ICLR 2023\n\n[2] Environment inference for invariant learning. ICML 2021\n\n[3] Towards Last-layer Retraining for Group Robustness with Fewer Annotations. NeurIPS 2023 \n\n[4] Bias Amplification Enhances Minority Performance. TMLR 2024\n\n[5] Unsupervised learning of debiased representations with pseudo-attributes. CVPR 2022\n\nQUESTIONS:\n* Is EVaLS sensitive to hyperparameters? \n* [minor] There seem to be typos in your corollary C.1.\n* Can the authors make clarifications on how the conditions in proposition 3.1 are met or relaxed in practice? How does the distribution of actual experimental benchmark datasets compare to your assumptions?\n\nLIMITATIONS:\nAforementioned in Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address the issue of spurious correlations when group labels are unavailable, this paper proposes a new method called EVaLS. It first creates a balanced training dataset using loss-based sampling. Then, it evaluates the accuracy of the balanced training set based on the inferred environments from the validation set, and selects models accordingly.\n\nSTRENGTHS:\n1. The paper is well-written, and includes a rich set of experiments with necessary theoretical explanations.\n\n2. It is essential to discuss the multiple  (unknown) spurious features case which has been overlooked in previous studies.\n\nWEAKNESSES:\n1. Why is the approach of using high-loss points (considered as the minority group) more effective than directly using misclassified points (considered as the minority group) in methods like JTT? Intuitively, compared to misclassified points, high-loss points are more \"implicit\" and no obvious thresholds, which could potentially result in high-loss points actually belonging to the majority group, thus exacerbating the imbalance in resampling.\n\n2. If the author can show the balance level of samples obtained through loss-based sampling compared to directly using labels (misclassified points), it could further illustrate the advantages of loss-based sampling.\n\n3. In Section \"Mitigating Multiple Shortcut Attributes\", if color is treated as a known spurious attribute and shape as an unknown spurious attribute, how would the performance of EVaLS be affected? Based on my understanding, there is a possibility that simplicity bias could cause the model to prioritize learning the simpler feature, color, and struggle to learn the more complex shape attribute. Therefore, considering color as known and shape as unknown can better show the performance of EVaLS in handling complex spurious features.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies how to improve the model’s robustness to multiple spurious correlations when the group labels (indicator for spurious correlation) are unknown in general. The proposed approach, EVaLS, leverages the loss from a base ERM model to sample a balanced subset to prevent learning from spurious correlations. In addition, a new synthetic dataset (Dominoes-CMF) for multiple spurious attributes is crafted. Empirically, the proposed approach sometimes has advantages over the rest of the baselines when using the same amount of additional information (group label).\n\nSTRENGTHS:\n1. The main paper is generally well-written. \n2. The theoretical analysis in Section 3.3 (with derivations and proofs in Appendix) shows that for one-dimensional Gaussian distributions, choosing the tails on the two sides of the distributions creates balanced groups, even though the original data distribution is skewed. \n3. Environment inference technique is demonstrated to be useful for separating the dataset into groups with different distributions of the subpopulations and then for model selection. \n4. The proposed technique only requires last-layer retraining on part of the validation set, which is generally more efficient.\n\nWEAKNESSES:\n1. Figure 2 attempts to illustrate more minority samples have high loss while the majority samples have low loss. However, in each of the plots, only the % of one of the minority or majority groups is shown. The illustration can be improved by showing the % of both majority and minority groups in the same plot, and showing the actual distribution of the loss for the groups. \n2. Though the idea is straightforward, it is unclear how the loss-based instance sampling is actually implemented. It is helpful to provide an algorithm or pseudocode to improve the presentation. \n3. The theoretical analysis is generally sound but limited to a case without discussing the use of the loss (which may not be Gaussian) and the spurious correlations (which involve at least two dimensions of core and spurious features [1]). \n4. The experimental results are less polished and sometimes the advantages are not so clear over other baselines. Some results are missing for datasets such as UrbanCars and MultiNLI. Only a few baselines are compared for the new dataset in Table 2. There is also no convincing and fine-grained analysis (e.g., ablation study) to understand how the proposed approach ensures data balancing and improves group robustness. \n5. The paper initially focuses on improving group robustness when multiple spurious correlations are present, but the experimental results are lacking for these more challenging datasets. \n\n[1] Sagawa, Shiori, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. \"An investigation of why overparameterization exacerbates spurious correlations.\" In *International Conference on Machine Learning*, pp. 8346-8356. PMLR, 2020.\n\nQUESTIONS:\n1. Since it’s an essential prerequisite for loss-based sampling, how well are the majority/minority groups separated in the logit space?\n2. How is the number of balancing samples $k$ chosen? How balanced are the samples when they are selected from the optimal $k$? \n3. I am curious about why the result for Civilcomment dataset is so much higher than the other baselines. Is the evaluation consistent with the baseline methods?\n4. Minor typo: line 233 should be without loss of generality (w.l.o.g.). \n5. Other minor issues with the experiments: The best-performing results in each category of approaches should be bold. The column header “best” should be “average” in Tables 4 and 5.\n\nLIMITATIONS:\nThe authors have discussed the limitations in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}