Improving Subgroup Robustness via Data Selection
Authors: Saachi Jain
Kimia Hamidieh
Kristian Georgiev
Andrew Ilyas
Marzyeh Ghassemi
Aleksander Madry
Venue: NeurIPS 2024 poster (2024)
Keywords: group robustness
fairness
data attribution
machine learning
Primary area: safety_in_machine_learning
TL;DR: Improving model performance on under-represented subpopulations by removing harmful training data.

Abstract: Machine learning models can often fail on subgroups that are underrepresented
during training. While dataset balancing can improve performance on
underperforming groups, it requires access to training group annotations and can
end up removing large portions of the dataset. In this paper, we introduce
Data Debiasing with Datamodels (D3M), a debiasing approach
which isolates and removes specific training examples that drive the model's
failures on minority groups. Our approach enables us to efficiently train
debiased classifiers while removing only a small number of examples, and does
not require training group annotations or additional hyperparameter tuning.

Improving Subgroup Robustness via Data Selection
Saachi Jain; Kimia Hamidieh; Kristian Georgiev,
Andrew Ilyas, Marzyeh Ghassemi, Aleksander Mądry
MIT
{saachij, aag
edu
Abstract
Machine learning models can often fail on subgroups that are underrepresented during training. While dataset balancing can improve performance on underper- forming groups, it requires access to training group annotations and can end up removing large portions of the dataset. In this paper, we introduce Data Debiasing with Datamodels (D3M), debiasing approach which isolates and removes specific training examples that drive the model's 's failures on minority groups. Our approach enables to efficiently train debiased classifiers while removing only a small number of examples, and does not require training group annotations or additional hyperparameter tuning.
Introduction
The advent of large datasets such OpenImages 23 and The Pile [13 has led machine learning models being trained on explicit (4) and illegal [48] content, or on data that encode negative societal biases 6 12 ( and other spurious correlations [32 30). On one hand, there is increasing evidence that models reflect the biases in these datasets; on the other hand, the enormous scale of these datasets makes extremely expensive to manually curate them-and so removal of such "harmful data" is challenging.
In this paper, propose approach that aims to remove data responsible for biased model predictions. particular, we focus on specific way of quantifying model bias worst- error- which captures the extent to which model performance degrades pre-defined subpopulations of the data. We aim to identify (and remove) the points in the training dataset that contribute most to this metric to improve the model's group robustness.
The challenge inherent in this approach is that it requires an understanding of how training data affect machine learning model predictions. To overcome this challenge, we first approximate predictions as simple, direct functions of the training dataset, using a framework called datamodeling [18] 33). We can then write quantitative notion of model bias (which is a function of predictions) of the dataset. Finally, by studying this function, we identify the training data points that contribute most to this measure of model bias. With the resulting method, which call Data Debiasing with Datamodels (D3M), we show that, across a variety of datasets, there are often a small number of examples that disproportionately drive worst-group error. Removing these examples, in turn, greatly improves models' worst-group error while maintaining dataset size.
function
Roadmap & contributions. In the rest this paper, we present and demonstrate the effectiveness of our Data Debiasing with Datamodels (D3M). Concretely, we show that D3M enables to:
Pinpoint examples that harm worst-group accuracy. We show that there are often small number of examples that disproportionnatly drive models' worst-group error on validation data. For example, on CelebA-Age. our method improves worst group error over
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Training class (species) feature (color) leads disparate performance.
balancing [IAP+22] removes examples from majority groups break this correlation.
Species (dng
(black vo bege)
Parr color
Equal representation from each
Goal: "debias" datasel improve worst-group accunacy (WGA):
Only changes data
Can hurt average accuracy
WGA
min group UN.
Acc(group)
Less accuracy gap
Needs training group labels
Our approach: Data Debiasing via Datamodeling (D3M)
Compute impact training sample WGA by predicting WGA as function dataset selection.
selection maximizes worst-group accuracy by removing most harmful examples.
Worst group accuracy (small) validation set
"Group alignment score" Learned coefficient; impact of point i on WGA
Group aligmment
0.3
0.0
-1.
1.2
0.1
004
WGA Σ
Only changes data
Competitive accuracy
Binary; whether solect the 6-th training sample
Less accuracy gap
Only needs test set labels
Figure Our method (D3M) improves worst group accuracy by identifying and removing the training samples which most negatively impact worst-group accuracy. Specifically, TRAK (33) to identify examples that exacerbate the discrepancy group performance. We then remove and re-train model the remaining data.
a natural baseline (data balancing) while removing 2.4x fewer examples. Furthermore, these offending examples often form coherent subpopulations within the data.
Achieve competitive debiasing performance. Our approach outperforms standard ap- proaches (both model-based and data-based) improving worst-group accuracy [25] 21 17]. and is able to match the performance of methods which use ground -round-truth training group annotations
40
Discover unlabeled biases. When validation group labels are unavailable, show how extract hidden biases (i.e., unlabeled subgroups) directly from the data. As a result, we can perform end-to-end debiasing without any group annotations.
We present our method in Section 4 and demonstrate these capabilities in Section 5 In Section 6 we leverage our framework to discover and mitigate biases within the ImageNet dataset, where D3M surfaces coherent color and co-occurrence biases. We then debias the model according to these failures, and improve accuracy on the identified populations.
The group robustness problem
We consider (unobserved) data distribution D over triplets (I,, M. each comprising input I; E X. label y4 V, and subgroup label g: G, where the set of distinct subpopulations in take the inputs
the data. As running example, consider the CelebA age classification task-here, I; to be images faces, the labels ys to be either "old" or "young," and the possible group labels to be "old man". "old woman", "'young man", and "young woman" (see Fig
Given a training dataset Smmi and (small) validation dataset Sma the goal of the group robustness problem is to produce classifier f that minimizes the worst-case loss over groups, i.e.,
max
(( [[(f(x), v)(g = = d'Eg
E
where E(:,)) is loss function. When & is the 0-1 loss, Equation T (one minus) the worst-group accuracy (WGA) of the classifier f. which we use to quantify success in the remainder of this work.

---

Standard loss minimization can yield models that perform poorly with respect to T For instance, re- turning our example of CelebA age classification, suppose there was a spurious correlation between age and gender in the training set Strinn such that old men and young women are overrepresented. A predictor that minimizes loss on Smain might leverage this correlation, and thus perform poorly on the underrepresented subgroups of old women young men.
In practice, subgroup labels gi can be expensive collect. Thus, approaches the subgroup robustness problem vary in terms of whether we observe the group label in the training set Swain and in the validation set Svat. In particular, there are three settings of interest:
Full-information (Train // Val v): We observe the group labels for both the training dataset Stain and validation dataset set Sval-
Partial-information (Train X/ Val V): We observe the group labels for the validation set Sval. but not for the (much larger) training set Spain-
No-information (Train X/ Val X): We do not have group information for either Strain or Sval- Note that theoretically this setting is unsolvable, since for any non-perfect classifier f. there exists an assignment of group labels so that the worst-group accuracy is zero. Nevertheless, subgroups of relevant practical interest typically have structure that allows for non-trivial results even with no information.
In this work, we focus on the partial-informan information and no-information settings, since acquiring group labels for the entire training set is often prohibitively expensive. Still, in Section we show that our proposed methods (D3M for the partial-information setting, and AUTO-D3M for the no-information setting) perform comparably to full-information approaches.
3 Related work
Before introducing our method, we discuss a few related lines of work.
Approaches to subgroup robustness. The group robustness problem (Section 2 has attracted 20
[5 21 38). Broadly, these solutions fall into one of two categories model interventions and data interventions. Model interventions target either
wide variety of solutions (see, c.g., [3] 40
model weights [42 45) or the training procedure 40 21 Data interventions, on the other hand, seek to improve worst-group accuracy by modifying the training dataset. For example, data balaneing removes or subsamples examples so that all subgroups are equally represented. Idrissi et al. [17) find that this simple approach can performs on par with much more intricate model intervention methods.
In this work, we focus on data interventions, for two reasons. First, it is often training data that drives models' disparate performance across groups [27], e.g., via spurious correlations (32 or underrepresentatioo (6]. Second, data interventions do not require any control over the model training procedure, which can make them more practical solution (e.g., when using ELL-s----serve ---avc---ervvcc). Indeed, since data intervention approaches only manipulate the dataset, they are also easy to combine with model intervention techniques.
Compared to our work, the main drawback of existing data interventions is that they often (a) require subgroup labels for the training data (which might not be available), and (b) hurt the models' natural accuracy on skewed datasets 7 (44). this work we circumvent these limitations, by proposing data-based approach to debiasing that can preserve natural accuracy without access to subgroup information.
Bias discovery. Another related line of work identifies biases in machine learning datasets and algorithms. For the former, previous works have shown that large, uncurated datasets used for training machine learning models often contain problematic biased data 4 5 48) Raji et al. [39] show that data bias can be a hurdle towards deploying functional machine learning models. Nadeem et al. (29) curate dataset to estimate bias in NLP models. Adebayo et al. show that label errors can disproportionately affect disparity metrics.
On the learning algorithm side, Shah et al. [7 and Puli et al. 37 show that the inductive bias of neural networks may encourage reliance on spurious correlations. Pezeshki et al. (34) leverage two networks trained on random splits of data while imitating confident held-out mistakes made by its
3

---

sibling to identify the bias. Shah et al. [46] show that algorithmic design choices (e.g., the choice of data augmentation) can significantly impact models' reliance on spurious correlations. Finally, there has been variety of work on "slice discovery" [9 10], where the goal is to discover systematic errors made by machine learning models.
Data selection for machine learning. Our work uses data selection to improve subgroup robustness of machine learning models. A recent line of work has explored data selection for improving various
measures of model performance. For example, Engstrom et al. [9] leverage datamodeling [18] 33) to
select pretraining data for LLMs. Similarly, Xia et al. [52) and Nguyen and Wong [3T] select data for
finetuning, and in-context learning, respectively. In another related work, Wang et al. 49 propose a
method to reweight training data order improve models' fairness.
Many of these works leverage contributive data attribution methods to select data that improves model performance [51]. Koh and Liang [22]. Feldman and Zhang [11], Schioppa al. [5] and
Hammoudeh and Lowd [5] propose using different variants of influence functions. Ghorbani and
Zou [14] leverage connections to Shapley values, a concept from game theory. Pruthi et al. [36 use
heuristic to estimate the contribution of each data point to model performance. Ilyas et al. 18 and Park al. 33 the datamodeling framework.
4 Debiasing datasets with datamodeling (D3M)
In this section, we present our data-based approach to training debiased classifiers. The main idea behind our approach is to identify (and remove) the training samples that negatively contribute to the model's ss worst-group accuracy, by writing model predictions as a function the training data.
Preliminaries. Let S = ((11,11),... Bi),..., (Zin vn)) be dataset of input-label pairs. For any subset of the dataset-as as represented by indices D C. [n] [n]-let ((D) E E2 be the parameters of classifier trained on D. Given an example = (z, y). let f2; be the correct-class margin on z of a classifier with parameters (defined as log( (pp)) where p the confidence assigned class y for input æ).
datamodel for the example is simple function that predicts fz; directly as a function of D, i.e., a function f. 2 + [o, 1] such that
fD) = f(2;
for D C [n].
Recent works (e.g.. that decompose additively in terms of their inputs D. In other words, these works show that one can compute example-specific vectors T(2) E R2" such that
.g... [18
24 33)
demonstrate the existence of accurate linear datamodels-functine
fD(
Σ T(), f(2; SDD
(2)
The coefficients T(=); have convenient interpretation as quantifying the "importance" of the i-th training sample to performance on example (i.c., as a data attribution score (15)). In what follows, we will assume access to coefficients T(2) for any example z-at the end of this section, we will show how to actually estimate the coefficient vectors T(2) efficiently.
Debiasing approach. How can leverage datamodeling to debias dataset? Recall that our goal to remove the samples that lead to high worst- worst-group error. Stated differently, given a dataset S of size n, we want to maximize the worst-group performance of classifier (DD) with respect to the indices D C [m)] that we train on.
Our main idea will be to approximate the predictions of ((D) using the corresponding datamodels (D). To illustrate this idea, suppose that our goal was maximize performance single test example z, i.e,, arg maxp ((;;)))) We can approximate this goal as finding arg max (=))))): then, due to the linearity of the datamodel fa. the training samples that hurt performance on & are simply the bottom indices of the vector r(2).
Now, this analysis applies not only to a single example z, but to any linear combination test examples. In particular, if we wish to maximize performance on a linear combination of validation examples, we simply take the linear combination of their coefficients, and remove the training examples corresponding to the smallest coordinates of the averaged vector.

---

Debiasing with group-annotated validation data. Given a set of validation samples for which the group labels 9; are observable, our last observation gives rise the following simple procedure:
1. Compute group coefficients rG) for each G. Since we have group annotations for each validation sample, we can define a vector T T() for each group G G as simply the average 72) within each group.
2. Compute group alignment. Next, we compute a group alignment score A, for each training sample i E [n]. which captures the the impact of the sample on worst-group performance. Since there may be many low-performing groups, we use "smooth maximum" function to weight each group according to its average loss. Thus, for a training example i,
Egeg exp(3(g) 7(0) Ege exp((gg)))))
A
where we set hyperparameter 3 1.
(3)
Here, &g is the loss of base classifier (SS) on group g (evaluated on the given validation set).
3. Remove drivers of bias. Finally, we construct a new training set Snew by keeping only the examples with the highest group alignment scores, i.e., removing the examples that most degrade worst-group accuracy:
Snew arg top-k((A zt E SSain)
We make two brief observations about hyperparameters before continuing. When computing the group alignment score in Step 2, the hyperparameter 3 controls the temperature of the soft maximum function (3). When 3 - O. the group alignment A; measures the impact of the i-th training example on the "balanced" performance (treating all groups equally). As 3 co, A, collapses to the training example's 's importance to only the worst group, which suboptimal if models perform poorly more than one group. For simplicity, we take 3 and refrain from tuning it.
Another hyperparameter in the algorithm above is the number of examples to remove, We consider two different ways of setting this hyperparameter. One approach is to search for the value of & that maximizes goost accuracy on the validation set S Alternatively, we find that the simple (and much more efficient) heuristic of removing all examples with a negative group alignment score (i.e., examples for which A; < 0) tends to only slightly over-estimate the best number of examples to remove (see, e.g., Figure 2 Thus, unless otherwise stated, we use this heuristic when reporting our results.
Debiasing without group annotations. Our procedure above relies group annotations for validation set Syal to compute the "per-group coefficients" T(G). In many real-world settings, however, models might exhibit disparate performance along unannotated subpopulations- this case, we might not have validation set on which we can observe group annotations gt. Can we still fix disparate model performance in this setting?
In general, of course, the answer to this question is no: one can imagine a case where each individual example is its own subpopulation, in which case worst-group accuracy will be zero unless the classifier is perfect. In practical settings, however, we typically care about the model's disparate performance on coherent groups of test examples. The question, then, becomes how to find such coherent groups
We posit that a unifying feature of these subpopulations is that they are data-isolated, i.e., that models' predictions on these coherent groups rely on different set of training examples than models' predictions on the rest of the test data. Conveniently, prior works [18 46) show that to find data- isolated subpopulations, one leverage the datamodel matrix-a matrix constructed by stacking the datamodel vectors r(2) for each test example. Intuitively, the top principal component of this matrix encodes the direction of maximum variability among the vectors T(z). Thus, by projecting the datamodel vectors T(2) of our validation examples onto this top principal component, we can identify the examples that are, in sense, "maximally different" from the rest of the test examples in terms of how they rely on the training set. These maximally different examples correspond to an isolated subpopulation, which we can apply D3M directly.
This approach (which we call AUTO-D3M), enables us to perform end-to-end debiasing without any group annotations. This method proceeds in four steps. For each class:
5

---

1. Construct a matrix T of stacked attribution vectors, where Tay = r(21)g.
2. Let e be the top principal component of T..
3. Project the attribution vector r=)) onto v and construct "group pseudo-labels"
= 1(T(2) To {}.
where is a hyperparameter
4. Apply D3M with the group pscudo-labels train debiased classifier.
Estimating the coefficients r(2). order to operationalize D3M and AUTO-D3M, it remains to
show that we can actually estimate coefficients r(z) satisfying (2). To accomplish this, we use
method called TRAK [33]. Leveraging differentiability of the model output f(z; 0) with respect to
model parameters 0, TRAK computes the coefficient vector T==) for example as follows:
(a) Train model 0 == 0(S) the entire training dataset S 2m,
(b) Sample a random Gaussian matrix P E Rexk where is the dimensionality of 0 (i.e., the number of model parameters) and is hyperparameter:
(c) For an example 2, define g(2) = P Vef 0*) the vandomly-projectee model output gradient (with respect the model parameters) evaluated at z.
(d) Compute the coefficient vector
= =](2) (s(x))s(z))" 2,ES
T(2)
g(z)) (1-o
i-th coefficient for example
Repeat steps (a)-(d) for T trials, and average the results to get a final coefficient vector T(2). The trials are identical save for the randomness involved in step (a).
We provide more intuition and details behind TRAK in Appendix A
A note scalability In terms of computational cost, TRAK involves taking single backward pass on each of the training and validation examples to compute the model's gradient. The (projected) gradients then saved compute TRAK scores. Typically, TRAK is computed over T trials: following the original paper we use 100. However, our approach can be used with any datamodeling technique.
5 Results
In Section we presented D3M an approach for debiasing classifier by identifying examples which contribute to a targeted bias. In this section, we validate this framework by assessing its performance on tasks with known biases.
We consider four classification tasks where there is a spurious correlation between the target label and group label in the training dataset: CelebA-Age [26] 19), CelebA-Blond (26), Waterbirds 41). and MultiNLI [50]. We provide more information about the datasets in Appendix B.I and other experimental details in Appendix B.2
5.1 Quantitative results
We first evaluate D3M and AUTO-D3M quantitatively, by measuring the worst-group accuracy of models trained on the selected subsets of the biased datasets above.
D3M: Debiasing the model in the presence of validation group labels. In Table 1 we compare D3M against several baselines, each of which requires either only validation group labels (X/ // or both training and validation group labels (V/v) We find that D3M outperforms all other methods that use the same group information (i.e., only validation group labels) on all datasets except Waterbird
For experiments choose À that the lower performing group consists of 35% of the validation examples that class.
"Note that WaterBirds has more worst-group examples in the val split (133) than the train split (56). Since DFR directly fine-tunes on the validation it has advantage here other methods.

---

Group Info Train / Val
Worst Group Accuracy (%) Waterbirds
Method
CelebA Age
CelebA -llon
MultiNLI
ERM
56.7
45,9
57.9
67.2
XI X
AUTO-D3M (ours)
76.0
83.8
81.0
75.0
JTT
25
61.0
81.6
63.6
72.6
XI /
DFR 21
70.4
88.4
89.0
74.7
D3M (ours)
75.6
90.0
87.2
76.0
RWG
75.6
88.4
81.2
68.4
VIV
SUBG
I7
68.5
88.3
85.5
67.8
GroupDRO 40
74.8
90.6
72.5
77.7
Table 1: Worst-group accuracies four group robustness datasets. A denotes methods that use validation group labels for both finetuning and hyperparameter tuning.
Worst Group Accuracy
Random Random Majority (oracle)
Accuracy
0.70
Balanced
D3M
0.60
0.50
0
5000
10000
15000
20000
25000
30000
35000
Examples Removed
Figure 2: Worst group accuracy on CelebA- Age function the number of examples i removed from the training set, using various removal methods. In green, D3M removes the is training examples with the most negative alignment scores A,. The green star marks the value heuristic (A, < 0). In blue is the performance of random baseline that removes k examples at random from the training set, and in orange is dataset balaneing, which removes examples randomly from the majority group. Compared to baselines, D3M efficiently improves worst group accuracy.
selected by our
Moreover, D3M performs on par with methods that have full access to both training and validation group labels.
AUTO-D3M:
Discovering blases with TRAK. We now consider the case where validation group labels are not accessible. Using AUTO-D3D D3M, we debias our model using sseudo-annotation derived from the top principal component of the TRAK matrix (AUTo-D33 D3M in Table Note that AUTO- D3M is the only method other than ERM that does not require either train or validation group labels. Despite this, AUTO-D3M achieves competitive worst-group accuracy in our experiments. We emphasize that AUTO-D3M does not require group labels at all-in particular, we do not use group labels to do hyperparameter selection or model selection when we retrain.
The effect of the number removed examples k. How well does D3M isolate the training examples that drive disparate performance? To answer this question, we iteratively remove training examples from CelebA-Age starting with the most negative A, and measure the worst-group and balanced accuracy (See Figure 2). CelebA-Age Age has 40K "majority" examples and 10K "minority" examples; thus, naive balancing requires removing 30K training examples. In contrast, by isolating which specific majority examples contribute the bias, our method is able to debias the classifier by removing only 10K examples.
Our heuristic of removing examples with negative A, (the star in Figure 2 slightly overestimates the best number of examples remove. Thus, while this heuristic gives a decent starting point for actually searching for the best & might further improve performance.
For MultiNLI. chose the PCA component by inspection that captures examples with/without negation.

---

Young.
Young.
Young.
Old,
Old,
Old.
Young,
Gray Hair
Gray Heir
Gray Hair
Soccock shudow De shadow 5 dook shudow
Busly Eycbrows
Builty Eycbrows
Sidebumss
Hair
Figure 3: Randomly sampled examples from the subpopulations with the most negative group alignment scores. We find that many of these examples have labeling errors (e.g., platinum blond instead of gray hair.)
Old
Young
Old
Young
Old
Young
5 Clock Shadow
Clubby
Mustache
Arched Eychrows
Double Chin
Narzow Eyes
600
Begs Under Eyes
Byeglasses
No Beard
400
Bald
Gostee
Rosy Cheeks
200
Gray Hair
Sidebrns-
Big Nose
Heavy Makeup
Wavy Hair
Black Hair
High Cheekbones
Wearing Earrings
Blond Hair
Male
Wearing Hat
Bushy Pyebrows
Month Slightly Open
Wearing Lipstick
200
Figure 4: The average group alignment score the training examples each subpopulation of CelebA-Age Subpopulations such "old" with "bushy eyebrows" or "young" with "gray hair" have particularly negative scores.
5.2 Qualitative results
What type of data does method flag? In particular, do the examples identify driving the targeted bias share some common characteristics? To test this hypothesis, we inspect the data removed by our method and identify subpopulations within the majority groups that disproportionately responsible for the bias. We then retrain the model after excluding all training examples from the identified subpopulations and show that this is viable strategy for mitigating the underlying bias.
Finding subpopulations responsible for model bias. Consider the running example from Figurel I where we train a model on the CelebA- Age dataset to predict whether a person is "young" or "old" when gender is a spurious feature (such that young women and old men are overrepresented), CelebA-Age has variety of annotations beyond age and gender, such as whether the person wearing eyeglasses. In this section, we use these extra annotations to identify coherent subpopulations that are flagged by our methods.
In particular, we consider subpopulations formed by taking the Cartesian product of labels and annotations, e.g., subpopulations of the form ("young", "wearing eyeglasses"), For each of these subpopulations, we calculate the average group alignment score A, of the training examples within that subpopulation (see Figure 4h. We find that subpopulations such as "young" with "gray hair" "old" with either "5 o' clock shadow" or "busy eyebrows" have particularly negative group alignment scores. In Figure [] we show examples from the subpopulations with the most negative group alignment scores, and observe that large fraction of the examples in these subpopulations contain labeling errors.
Retraining without identified subpopulations. Once we have identified subpopulations with negative aligmment scores, natural strategy for mitigating the underlying bias is exclude these subpopulations from the training set. To explore this approach, we exclude the five subpopulations with the most negative attribution scores on average from the CelebA-Age dataset: "Young" + "Gray Hair". "Old"+ "5 o' Clock Shadow",, "Old" + "Bushy Eyebrows", "Young" + "Blond Hair", and "Old" + "Sideburns." After retraining the model on this modified training set, we get a worst- accuracy (WGA) of 68, 68.4% 4% improvement of -12% over the WGA of the original model (56.7%).

---

Tench
Strawberries
Red Wolf
Cauliflower
Figure 5: For four ImageNet classes, the most extreme (positive or negative) examples according to the top PCA direction of the TRAK matrix. Our method identifies color and co-occurrence biases.
Case Study: Finding and Mitigating Model Failures on ImageNet
In Section 5 we evaluated D3M and AUTO-D3M on datasets where the bias already known. We now deploy AUTO-D3M to discover and mitigate biases within the ImageNet dataset, which does not have predefined bias available group annotations.
Identifying ImageNet biases. We use TRAK to compute a coefficient matrix (see Step of AUTO-D3M in Section 4 for held out validation split (10% of the training set). Focusing on seven ImageNet classes, we use the first principal component of the matrix to identify potential biases. In Figure 5 display the most extreme training examples according to the top principal component for four of these classes. PCA identifies semantically color and co-occurrence biases (c.g., tench fishes with without humans yellow/white cauliflowers that either cooked uncooked.) In fact, identified biases match the challenging subpopulations in Jain et al. 19 and Moayeri et al. [28]
Croup Aeturnry
82
Cos Canteol
Strwbberyy Wettar pasts)
Howier Menkey (kee
Dog Sled (Onty
[Red Couc)
Plumen
Unsterwatedd
Class
Figure 6: Worst-group accuracy for the ImageNet classes studied in Section Gafter intervening with either D3M or AUTO-D3M (error bars over 1 std).
Mitigating ImageNet biases with AUTO-D3M. For each of the four targeted ImageNet classes, we seek to mitigate the identified failure modes with AUTO-D3M. We consider two settings based on the first, we manually assign each of the validation images to
the level of human intervention. group according to a human description of identified bias (e.g., an image of a tench is in group if human is present and group 2 otherwise), and then use those group labels with D3M. In the second setting, we debias in purely automatic fashion, using AUTO-D3M M to derive pseudo-gres -oroup labels from the top principal component. In Figure 6 we display worst group accuracy on the test images of the targeted class (evaluated using manual group assignments of the 50 test examples). Both D3M and AUTO-D3M improve worst group accuracy over ERM without significantly impacting the overall ImageNet accuracy (see Appendix C.2)
7 Conclusion
We propose Data Debiasing with Datamodels (D3M), a simple method for debiasing classifiers by isolating training data that disproportionately contributes to model performance underperforming groups. Unlike approaches such as balancing, our method only removes a small number examples and does require training group annotations additional hyperparameter tuning. More generally, our work takes a first step towards data-centric model debiasing.
'Here, we only consider the target class when computing the loss weighting. As a result, the heuristic overestimates the number of examples & to remove, and so we instead search for the optimal & using our held validation set.

---

References
[1] Julius Adebayo, Melissa Hall, Bowen Yu, and Bobbic Chern. "Quantifying and mitigating the impact of label errors model disparity metrics". In: arXiv preprint arXiv:2310.02533 (2023).
[2] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. "Machine Bias *", In: Ethics of Data and Analytics. 1st Edition. Auerbach Publications, May 2022. pp. 254-264.
[3] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Pate Paz. "Invariant risk mini- mization". In: arXiv preprint arXiv: 1et::0297009999 (2019).
[4] Abeha Birhane and Vinay Uday Prabhu. "Large image datasets: A pyrrhic win for computer vision?" In: 202 / IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE,
Jan. 2021, pp. 1536-1546.
[5] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. "Multimodal datasets: misog- yny, pornography, and malignant stereotypes". In: arXiv preprint arXiv:21 / 10.01963 (2021). [6] Joy Buolamwini and Timnit Gebru. "Gender shades: Intersectional accuracy disparities in commercial gender classification". In: Conference on fairness, accountability and transparency (FAccT). 2018.
[7] Rhys Compton, Lily Zhang, Aahlad Puli, and Rajesh Ranganath. "When More is Less: Incor- porating Additional Datasets Can Hurt Performance By Introducing Spurious Correlations". In: arXiv preprint arXiv:2308.cm / (2023).
[8] Kate Crawford and Trevor Paglen. "Excavating AI: the politics of images in machine learning training sets". . In: Al & society 36.4 (Dec. 2021), pp. 1105-1116.
[9] Logan Engstrom, Axel Feldmann, and Aleksander Madry. "DsDm: Model-Aware Dataset Selection with Datamodels". In: 2024.
[10] Sabri Eyuboglu, Maya Varma, Khaled Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dunnmon, James Zou, and Christopher Ré. "Domino: Discovering systematic errors with mooss- embeddings". In: arXiv preprint arXiv:2203. /493 (2022).
[11] Vitaly Feldman and Chiyuan Zhang. "What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation". In: Advances in Neural Information Processing
Systems (NeurIPS). Vol. 33. 2020. pp. 2881-2891
[12] Emilio Ferrara. "Fairness and Bias Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies". en. In: Sci 6.1 (Dec. 2023), p. 3.
[13] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. "The pile: An 800gb dataset of diverse text for language modeling". In: arXiv preprint ar ArXi 200000002710 (2020).
[14] Amirata Ghorbani and James Zou. "Data shapley: Equitable valuation of data for machine learning". In: International Conference on Machine Learning (ICML). 2019.
[15] Zayd Hammoudeh and Daniel Lowd. "Training Data Influence Analysis and Estimation: A Survey". In: arXiv preprint weXiv:2212.04655 2022.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. 2015.
[17] Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. "Sim- ple data balancing achieves competitive worst caattt-oouuuacccargee In: Conference on Causal Learning and Reasoning. PMLR. 2022, pp. 336-351.
[18] Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. "Datamodels: Predicting Predictions from Training Data". In: International Conference Machine Learning (ICML). 2022.
[19] Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry. "Distilling Model Failures Directions in Latent Space". In: arXiv preprint arXiv:2206. 144554446.14 (2022).
[20] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. "Decoupling representation and classifier for long-tailed recognition". In: arXiv preprint 1i1: (2019).
[21] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. "Laatt layer re-training is sufficient for robustness spurious correlations". In: arXiv preprint arXiv:2204.02937 (2022).
[22] Pang Wei Koh and Percy Liang. "Understanding Black-box Predictions via Influence Func-
tions", In: International Conference on Machine Learning. 2017.
10

---

[23] Alina Kuznetsova, Hassan Rom, Neil Alldrin. Jasper Uijlings,, Ivan Krasin, Jordi Pont- Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, et al. "The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale". In: arXiv preprint arXiv: /8/ 1.00982 (2018).
[24] Jinkun Lin, Anqi Zhang, Mathias Lecuyer, Jinyang Li, Aurojit Panda, and Siddhartha Sen. "Measuring the Effect of Training Data on Deep Learning Predictions via Randomized Experi- ments". In: arXiv preprini arXiv:2206. 10013 (2022).
[25] Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. "Just Train Twice: Improving Group Robustness without Training Group Information". In: International Conference on Machine Learning.
2021.
[26] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. "Deep Learning Face Attributes in the Wild". In: International Conference Computer Vision (ICCV). 2015.
[27] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. "A
Survey on Bias and Fairness in Machine Learning". In: (2021).
(28) Mazda Moayeri, Sahil Singla, and Soheil Feizi. "Hard imagenet: Segmentations for objects with strong spurious cues". In: Advances in Neural Information Processing Systems 35 (2022),
pp. 10068-10077.
[29] Moin Nadeem, Anna Bethke, and Siva Reddy. "StereoSet: Measuring stereotypical bias in pretrained language models". In: arXiv preprint arXiv:2004.0944 (2020).
[30] Yannic Neuhaus, Maximilian Augustin, Valentyn Boreiko, and Matthias Hein. "Spurious features everywhere- wwwhere-large-seee large-scale detection of harmful spurious features in imagenet". In: Pro- ceedings of the IEEF/CVF International Conference on Computer Vision. 2023, pp. 20235- 20246.
[31] Tai Nguyen and Eric Wong. "In-context example selection with influences". In: arXiv preprint arXiv:2302. 1042 (2023).
[32] Lauren Oakden- Daknen-Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. "Hidden stratification causes clinically meaningful failures in machine learning for medical imaging".
In: Proceedings of the ACM conference on health, inference, and learning. 2020.
[33] Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. "TRAK: Attributing Model Behavior at Scale". In: Arxiv preprint arXiv:2303. 14186. 2023.
[34] Mohammad Pezeshki, Diane Bouchacourt, Mark Ibrahim, Nicolas Ballas, Pascal Vin- cent, and David Lopez-Paz. "Discovering environments with XRM". In: arXiv preprint arXiv:2309. 16744449997778888 (2023).
[35] Daryl Pregibon. "Logistic Regression Diagnostics". In:
The Annals of Statistics. 1981
[36] Garima Pruthi, Frederick Liu, Mukund Sundararajan, and Satyen Kale. "Estimating Training Data Influence by Tracing Gradient Descent". In: Neural Information Processing Systems (NeurIPS). 2020.
[37] Aahlad Puli, Lily Zhang, Yoav Wald, and Rajesh Ranganath. "Don't blame dataset shift! Shortcut learning due gradients and cross entropy". In: arXiv preprint arXiv:2308. 12553 (2023).
[38] Shikai Qiu, Andres Potapczynski, Pavel Izmailov, and Andrew Gordon Wilson. "Sim- ple and Fast Group Robustness by Automatic Feature Reweighting". In: arXiv preprint (2023).
arXiv:2306.1100
[39] Inioluwa Deborah Raji, Blizabeth Kumar, Aaron Horowitz, and Andrew Selbst, "The fallacy of Al functionality".. In: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 2022. pp. 959-972.
[40] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. "Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization". In: International Conference on Learning Representations. 2020.
[41] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. "An investigation of why oxerparameteriizatoon exacerbates spurious correlations". In: International Conference on Machine Learning. PMLR. 2020, pp. 8346-8356.
[42] Shibani Santurkar, Dimitris Tsipras, Mabalaxmi Elango, David Bau, Antonio Torralba, and Aleksander Madry. "Editing a classifier by rewriting its prediction rules". In: Preprint. 2021.
11

---

[43] Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. "Scaling up influence functions". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. 2022.
pp. 8179-8188
[44] Roy Schwartz and Gabriel Stanovsky. "On the limitations of dataset balancing: The lost battle against spurious correlations". In: arXiv preprint arXiv:2204. 12708 (2022).
[45] Harshay Shah, Andrew Ilyas, and Aleksander Madry. "Decomposing and Editing Model Predictions". In: arXiv preprint. 2024.
[46] Harshay Shah, Sung Min Park, Andrew Ilyas, and Aleksander Madry. "ModelDiff: A Frame- work for Comparing Learning Algorithms". In: arXiv preprinl arXiv:22/1.122 2022.
[47] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. "The pitfalls of simplicity bias in neural networks". In: Advances in Neural Information Processing Systems 33 (2020), pp. 9573-9585.
[48] David Thiel. Identifying and eliminating CSAM in generative ML training data and models. Tech. rep. 2023.
[49] Haonan Wang. Ziwei Wu, and Jingrui He. "FairIF: Boosting Fairesss in Deep Learning via Influence Functions with Validation Set Sensitive Attributes". In: Proceedings of the 17th ACM International Conference Web Search and Data Mining. 2024, pp. 721-730.
[50] Adina Williams, Nikita Nangia, and Samuel R Bowman. "A broad-coverage challenge corpus for sentence understanding through inference". In: arXiv preprint arXiv:1704.05422 (2017). [51] Theodora Worledge, Judy Hanwen Shen, Nicole Meister, Caleb Winston, and Carlos Guestrin. "Unifying corroborative and contributive attributions in large language models". In: arXiv preprint arXiv:2311.122 1.2233 (2023).
[52] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. "Less: Selecting influential data for targeted instruction tuning". In: arXiv preprint arXiv:2402.04333
(2024).
12

---

A Background on Data Attribution and TRAK
Let Z the input space and S be a training set interest. For a given training subset S" C S and larget example data attribution techniques seek to estimate the attribution score of - that is, the change the model's prediction on z when the model trained on the subset S'. More formally, let f(z, ((S) be the model'ss s output function example Then can define 7 7(2); the attribution score of the ith training example 2; on target example 2
T(2), = f 0(5) (())).)
While T(2) is relatively straightforward to compute for linear models [35). computing this influence far more challenging for neural networks. Thus, in order to approach this problem, TRAK first approximates f(;; (S)) as a linear model on top of the gradients Vo(f; ) of the original neural network after convergence. We then plug this approximation into the estimate for linear classifiers to approximate (=)). After simplification, the TRAK estimate of the influence of 2 is
T(2) -
+Q
where o(z) (f(2;((5')) are the (randomly projected) gradients example 2, & are the stacked training gradients + (0(21), ..., (zn)) ). and Q a normalization matrix.
B Details of Experiments
B.1 Experimental Setup
In this section, describe the datasets, models and evaluation procedure that use throughout the paper.
Datasets. In order cover a broad range of practical scenarios, we consider the following image classification and text classification problems.
Waterbirds [40) is a binary image classification problem, where the class corresponds to the type of the bird (landbird waterbird), and the background is spuriously correlated with the class. Namely, most landbirds are shown on land, and most waterbirds are shown over water.
CelebA-Blond 26 is binary image classification problem, where the goal is predict whether person shown in the image is blond; the gender of the person serves as a spurious feature, as 94% of the images with the "blond" label depict females.
CelebA-Age 26 [9] is binary image classification problem, where the goal is predict whether person shown in the image is young; the gender of the person serves as a spurious feature. For this task, we specifically subsample the training set such that the ratio of samples in the majority vs. minority groups is 4:1.
MultiNLI 550 40) is a classification problem where given pair of sentences, the task is classify whether the second sentence entailed by, neural with, or contradicts the first sentence. The spurious attribute from Sagawa et al. (40) describes the presence of negation words, which appear more frequently in the examples from the negation class.
Methods.
We benchmark approach against the following methods:
ERM is simple empirical risk minimization on the full training set.
RWG [17) is ERM applied to random batches of the data where the groups are equally represented with a combination of upsamping and downsampling such that the size of the dataset does not change.
SUBG 17 is ERM applied to random subset of the data where we subsample all groups such that they have the same number of examples.
GroupDRO [40 trains that minimizes the worst-case performance over pre-defined groups in the test dataset.
13

---

Just Train Twice (JTT) [25 trains an ERM model with upsamping initially misclassified training examples by an initial ERM model.
(1] trains ensemple of linear models balanced validation set, given ERM features.
DFR
B.2 Training Details
In this section, detail the model architectures and hyperparameters used by each approach. We used the same model architecture across all approaches: Randomly initialized ResNet-18 (16) for CelebA and
pretrained ResNet- 18s for Waterbirds. We use the GroupDRO implementation by Sagawa et al. [40] and DFR implementation by Kirichenko et al. (21).
For all approaches, we tune hyperparameters for ERM-based methods (ERM, DFR, and D3M) and re-weighting based methods (RWG, SUBG, GroupDRO and JTT) separately. For RWG, SUBG, GroupDRO and JTT, early stop based on highest worst-group accuracy on the validation set as well. We optimize all approaches with Adam optimizer.
For the CelebA dataset, we all methods with learning rate le - 3, weight decay 1e - 4, and batch size 512. We train RWG, SUBG, GroupDRO and JTT with learning rate le 3, weight decay le and batch size 512. We train all models for the CelebA-Age Age task to up to 5 epochs and all models for CelebA-Blond task up 10 epochs.
For the Waterbirds dataset, we train the approaches that use the ERM objective (including D3M) with learning rate 1e - 4, weight decay 1e - 4, and batch size 32. We train RWG, SUBG, GroupDRO and JTT with learning rate le 5, weight decay 0.1, and batch size 32. We train all models to up to 20 epochs.
For all other hyperparameterss we use the same hyperparameters as Kirichenko et al. 21 for DFR and the same hyperparameters Liu et al. (25 for JTT.
We report the performance of the models via Worst-group Accuracy, or Balanced Accuracy in Table which is the average of accuracies of all groups If all groups in the test set have the same number of examples, balanced accuracy will be equivalent average accuracy.
Our model was trained on a machine with 8 A A100 GPUs.
14

---

Omitted Results
C.1 Balanced Accuracies
Below we include the balanced accuracies for the experiments in Table
Group Info Train Val
Celebt-Age Group Accuracy
Celebt-Blead Bndd
Vaterbirde
KalviBLI
Method
Accuracy
Accumey
Group Accuncy
Balanced Accuracy
Warst Group Accurccy
Balanced Accumccy
Accuracy
Group
ERM
XI x
77.96
56.65
82.59
45.86
83.40
57.85
80.92
67.19
(ours)
RWG 17
//
80.66
75.64
90.42
88.40
86.51
81.21
78.61
68.41
SUBG
/1
77.57
68.49
91.30
88.26
$6.97
85.46
73.64
67.76
GavenDRO
/1
77.7
JTT
x
68.06
60.95
92.01
81.61
85.24
63.61
78.6
72.6
DFR
KVV6
80.69
70.37
91.93
85,40
90.89
88.96
82.1
74.7
(00rs)
87.15
Table 2: Balanced accuracy and worst-group accuracy CelebA-Age CelebA-Blore and Waterbirds.. A double checkmark (VV) indicates that the method uses validation group labels for model finetuning, in addition to hyperparameter tuning.
15

---

C.2
Tench ImageNet Accuracies
Below included the detailed accuracies for the ImageNet experiment.
Class-Level
ImageNet-Level
Class (bias)
Balanced Accuracy
Worst Group Accuracy
Overall
Method
Accuracy
ERM
46.87
22.62
63.97
Red Wolf (Red Coat)
TRAK
65.63
52.38
63.71
Auto-TRAK
59.94
39.29
63.87
ERM
85.10
78.12
63.97
TRAK
90.73
86.88
63.84
(Presence of human)
Auto-TRAK
86.67
80.00
63.97
ERM
77.81
63.64
63.97
Cauliflower
TRAK
85.77
79.55
63.70
(Not Cooked)
Auto-TRAK
86.73
79.40
63.75
ERM
58.93
35.58
63.97
Strawberry
TRAK
70.49
51.92
63,88
(Not on plate)
Auto-TRAK
68.99
50.48
63.79
Table Auto-TRAK identifies and mitigates biases ImageNet. For four ImageNet classes, bias was identified from inspecting the TRAK PCA directions. Then Auto-TRAK is applied in order to mitigate the bias for that class. Auto-TRAK is able improve the worst group accuracy for the targeted class without significantly changing the overall ImageNet accuracy.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstracviintrdduction clearly states the method that presented and the results and case study of our experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Our work talks about the group annotations needed for the method, which informs our discussion of D3M vs Auto-D3M.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specificatinn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: disclose all hyperparameters and also release our code in the supplement.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Our code is in the supplement.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
public/guiide//odeSabbaee sorrooddSsbmmisssoooooooroo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https wtwpolliigggiieee/eeeeeoe Sonpe///ggiddeeeeCouSuuubbiii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We discuss the hyperparameters in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
Figure contains confidence intervals over 10 runs
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
19

---

The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Appendix B.1
Guidelines:
The answer means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpt//twt//////oooo.....ooooc/a http://///p/.t.eccccucuaaaaaaaa.
Answer: [Yes]
Justification: We have reviewed the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Group robustness is problem that often appears in fairness issues for Machine Learning (see introduction for more context).
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Not applicable (datasets are open source)
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the original papers for datasets and TRAK.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not create assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
we do not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
we do not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "BdgJG24Yj2", "reviewer": "Reviewer_FTVd", "summary": "The paper proposes Data Debiasing with Datamodels (D3M), a method to improve machine learning model performance on underrepresented subgroups by removing specific training examples that cause failures. Unlike traditional balancing methods, D3M efficiently debiases classifiers without needing group annotations, significant dataset reductions or additional hyperparameter tuning.", "questions": "N/A", "limitations": "See weaknesses", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Significance: This work effectively identifies and removes training samples to improve the worst-group accuracy. As demonstrated by the experiments, this method outperforms both standard model-based and data-based approaches.\n- Comprehensive Datasets: A wide range of datasets is used for image and text classification tasks, with corresponding benchmarks evaluated against existing methods as listed in Appendix B, \"Details of Experiments.\"", "weaknesses": "- Writing and Format: the presentation of the paper needs readability improvement:\n  - Redundant section start: Line 81\n  - Excessive parenthetical comments and irregular format: Lines 20, 25, 28, 32, 67-71, 83, etc.\n\n- In addition to isolating problematic training data, experiments should be conducted to assess the impact on the necessity of further hyperparameter tuning and to strengthen the case for the effectiveness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w3ekAn1X5D", "reviewer": "Reviewer_QjGu", "summary": "This paper introduces a new data debiasing technique called Debiasing with Data Attribution (DDA). DDA utilizes data modelling framework to identify and eliminate training examples that negatively impact the accuracy of the worst-performing groups. Additionally, the paper presents AUTO-DDA, an extension of DDA that can identify biases even without prior knowledge of group information. The proposed methods are validated through experiments on various datasets such as CelebA-Age, CelebA-Blond, Waterbirds and MultiNLI.", "questions": "Please refer to the questions in the weakness section.", "limitations": "Limitations are discussed.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach is simple and effectively improves the performance on real-wolrd datasets such as ImageNet.\n2. The paper is presented well and easy to follow.", "weaknesses": "1. The performance of ImageNet is only reported on selected classes. How are the classes selected for evaluation? Is it based on the amount of bias present in the classes? \n2. I am unsure if the proposed approach is effective when the majority of the data consists of bias-aligned points. For example, if there are only a few conflicting points and the rest are bias-aligned, how will the data be removed? I doubt the approach would still be useful for debiasing since a large part of the data is still going to be majorly biased.  Even if the authors claim that the majority of the bias aligned points will be removed, I believe the model would still overfit to the data since the final dataset would be extremely small. Analyzing the performance of the approach with varying numbers of bias-conflicting points (1%, 5%, 10% of CMNIST(10 class classification)) in the dataset would be beneficial to understand this scenario. This experiment would provide insights into how well the approach scales to real-world scenarios where the degree of bias is significantly high.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback on our work. Below, we have address the concerns raised by the reviewer:\n\n**[Selection of specific classes chosen for ImageNet experiments]**\n\nWe selected classes that previous work found had biases in the ImageNet dataset. Specifically biases in the classes Red Wolf, Tench, Cauliflower, and Strawberry were studied in [1] and biases in the classes Snorkel, Howler Monkey, and Dog Sled were studied in Hard ImageNet [2].\n\n**[Debiasing datasets with few bias-conflicting examples]**\n\nAs the reviewer points out, if basically all the training data points are bias-aligned points (and there are only a couple of examples that are not aligned) a dataset selection method may result in a very small training dataset which risks the model overfitting. This is a misalignment of objectives: here we are optimizing for worst group accuracy. In such a case, the worst group accuracy of the ERM model would be close to zero, since we entirely rely on the bias and thus completely fail on the non-bias aligned points. Reducing to a small dataset through dataset selection might degrade the overall accuracy, but will still likely improve the worst group accuracy. \n\nHowever, even in this case, D3M is a better approach than dataset balancing, since we are more sample efficient when removing data points (removing the “worst offenders”) as demonstrated in Figure 2. Moreover, both Celeba-Blond and Waterbirds are highly bias-aligned datasets, where the smallest group constitutes a very small number of examples. As we show in Table 1, D3M is still effective in these scenarios. \n\n\n[1] Jain, Saachi, et al. \"Distilling model failures as directions in latent space.\" arXiv preprint arXiv:2206.14754 (2022).\n\n[2] Moayeri, Mazda, Sahil Singla, and Soheil Feizi. \"Hard imagenet: Segmentations for objects with strong spurious cues.\" Advances in Neural Information Processing Systems 35 (2022): 10068-10077.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gcBL0QjCxO", "reviewer": "Reviewer_uxzC", "summary": "The paper introduces a method called Data Debiasing with Datamodels (D3M) that addresses the problem of model bias (using the worst-case loss over groups as the metric). The approach leverages a process known as datamodeling to predict model behavior based on training data influence, focusing on removing data points that contribute heavily to worst-group error. The paper illustrates D3M’s effectiveness across various datasets, showing that it can outperform both traditional model and data intervention strategies. Moreover, the method is adaptable to the setting without explicit subgroup labels.", "questions": "After improving the worst-group accuracy, does the model still maintain good overall accuracy? How does the method impact the performance across all groups? Were the results of the method tested across various model architectures to confirm its generalizability? In scenarios lacking explicit group labels, were there any experiments conducted to assess the effectiveness of the pseudo group labeling approach using the datamodel matrix in the setting or case studies in this paper?", "limitations": "The paper presents relatively robust results. However, I do not see it adequately address the limitations of the methods.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "Originality: The paper introduces an innovative approach, Data Debiasing with Datamodels (D3M), which creatively combines elements from influence functions and data-centric fairness strategies to address model bias. D3M focuses on optimizing the dataset by identifying and removing specific training instances that disproportionately skew model performance against minority subgroups. This methodological innovation brings high originality to the paper.\nQuality: The authors conduct a thorough analysis across multiple datasets, effectively demonstrating how D3M enhances worst-group accuracy. The use of comparative baselines and the examination of different scenarios (including those without explicit subgroup labels) shows the robustness and reliability of D3M.\nClarity: The paper is relatively well-structured. The effective use of diagrams and necessary mathematical definitions help demonstrate the results. Moreover, case studies help readers understand the use cases of D3M.\nSignificance: The significance of this work is relatively substantial, addressing the issue of the subgroup biases of models. Moreover, by providing a tool that can improve model fairness without needing subgroup labels, the paper contributes to the applications where the group labels are unavailable.", "weaknesses": "One weakness of the method is its exclusive focus on improving worst-group accuracy without presenting results on how it might affect the overall accuracy for all groups. This raises concerns about potential trade-offs, where enhancing fairness for the worst-performing subgroup could compromise the model's general performance. Additionally, the paper does not thoroughly explore how different model configurations might influence the outcomes. Understanding how variations in model architectures, initial parameter settings, or training procedures affect the effectiveness of the method is useful for validating its robustness and adaptability to diverse scenarios. Finally, a relatively minor weakness is that the demonstration of the paper could be more organized and coherent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing a thorough review of our paper. Below, we address the feedback points raised by the reviewer:\n\n**[Focus on worst-group accuracy (WGA)]**\n\nThe reviewer raises the concern of evaluating only WGA without reporting overall accuracy. We note that we report balanced (i.e., average across all groups) accuracies for all of our experiments in Tables 2 and 3 in the appendix (we report WGA in the main paper due to space constraints). We will mention those results more prominently in the main paper.\n\n**[Impact of model architectures and hyperparameters on D3M performance]**\n\nWe agree that investigating how model configurations change the efficacy of D3M is an promising avenue to pursue! In particular, it would be interesting to see how pre-training (e.g., on ImageNet) changes the behavior of D3M when there is a strong initial prior. For the Waterbirds dataset we do use ImageNet pre-trained models, but we don’t have a direct comparison on the same dataset between D3M’s behavior with ImageNet and randomly initialized models. We would be happy to pursue this for the final paper.\n\n**[Paper organization]**\n\nWe will edit the camera-ready version of the manuscript to improve the organization of the paper where this is needed. We kindly ask the reviewer to elaborate on the paragraphs/sections which may need additional organizational work.\n\n**[Effectiveness of the pseudo-labels]**\n\nWhen reporting WGA for Auto-D3M (where we do not explicitly use any group labels in the method) we compute WGA using the groundtruth test group labels (e.g., in Table 1). We find that while the pseudo-labels are not exactly the same as the groundtruth group labels, they are aligned enough that performing D3M with the pseudo-labels (e.g., Auto-D3M) competitively improves worst group accuracy with respect to the groundtruth groups. \n\nIn our ImageNet case study, we similarly report WGA in Figure 6 by performing manual group assignment on the 50 test images per class (e.g., for tench, we manually label for each image for “whether a human is present”). Again we find that our pseudo-labels are aligned enough with these manual labels that performing Auto-D3M improves accuracy with respect to the manually assigned groups.\n\n[1] Idrissi, Badr Youbi, et al. \"Simple data balancing achieves competitive worst-group-accuracy.\" Conference on Causal Learning and Reasoning. PMLR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "84ZYmQ7e3J", "reviewer": "Reviewer_yCP5", "summary": "This paper proposes a data-centric model debiasing technique to identify and remove data which harm worst-group accuracy. This method removes fewer data than standard balancing techniques and can be adapted for settings with and without group annotations. Experiments are provided on standard group robustness benchmark datasets, and the method is shown to promote bias discovery on ImageNet in the absence of group annotations.", "questions": "1. Is the hyperparameter k from D3M (number of examples to remove) the same as the hyperparameter k from TRAK (dimensionality of the gradient projection)? If not, it would be helpful to use different letters and detail how the k in TRAK is chosen.\n2. Why is random initialization used for CelebA, as opposed to standard ImageNet initialization? Do the CelebA comparisons in Table 1 also use random initialization? \n3. In the appendices, the tables reference proposed methods TRAK and Auto-TRAK. Is this meant to read D3M and Auto-D3M respectively?\n4. While not strictly necessary, I would be curious to see a qualitative comparison of the results from Section 5.2 and Figures 3 and 4 with other data selection techniques from the robustness literature. How do the data with negative alignment scores compare with data selected via misclassification [1], disagreement [2], other influence functions [3, 4], or Shapley values [5]? Are negative alignment scores perhaps more interpretable than these other techniques?\n\n***References***\n\n[1] Liu et al. “Just Train Twice: Improving Group Robustness without Training Group Information.” ICML 2021.\n\n[2] LaBonte et al. “Towards last-layer retraining for group robustness with fewer annotations”. NeurIPS 2023.\n\n[3] Koh and Liang. “Understanding Black-box Predictions via Influence Functions.” ICML 2017.\n\n[4] Feldman and Zhang. “What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation.” NeurIPS 2020.\n\n[5] Ghorbani and Zou. “Data Shapley: Equitable valuation of data for machine learning.” ICML 2019.", "limitations": "While the limitations of group annotations are sufficiently discussed, I have an additional question about compute efficiency of the proposed method. The experiments were performed on 8x A100 GPUs, which is much more extensive than comparative methods (e.g., [1] only needs to train a linear classifier, which, being essentially a logistic regression, is very efficient). Is this level of compute necessary for D3M? I am mainly concerned about the TRAK subroutine of Auto-D3M, where in step (d) the covariance matrix of the projected gradients of the entire dataset is constructed and inverted T=100 times. How much wall-clock time and GPU VRAM does this step consume, and how does it scale with the hyperparameter k?\n\n***References***\n\n[1] Moayeri et al. “Spuriosity Rankings: Sorting Data to Measure and Mitigate Biases.” NeurIPS 2023.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The differences between the full-information, partial-information, and no-information regimes are clearly delineated, and the advantages of D3M and Auto-D3M in each setting are comprehensively discussed. In the no-information regime, which is important and yet understudied in the literature, the authors propose a novel and elegant Auto-D3M algorithm based on TRAK, which I expect to be a strong baseline for future work in this setting.\n2. D3M and Auto-D3M compare favorably to other common data balancing techniques such as subsampling a class-balanced dataset, removing far fewer points while achieving better WGA performance.\n3. Sections 5.2 and 6 are comprehensive and very useful for developing an intuitive understanding of the proposed group alignment scores and TRAK matrix. The ability to discover spurious correlations in complicated datasets without group annotations is likely to be useful for practitioners.\n4. The explanations of each algorithm -- D3M, Auto-D3M, and TRAK -- are clear and well-written. The mathematics is well-explained and sufficiently technical without being convoluted.", "weaknesses": "1. In Table 1, the only comparison to previous work provided for the no-information regime is ERM, which is generally understood to be a weak baseline for group robustness tasks. Some examples of comparisons I would expect to see in this setting include MaskTune [1], uLA [2], DivDis [3], or CB-LLR [4]. Similarly, in the partial-information regime, additional comparisons may include AFR [5] or SELF [4]. (I do not expect the authors to include all these comparisons, but it would benefit to discuss the most appropriate ones).\n2. In Section 6, I believe a reference and comparison to [6] is missing. Similarly to this paper, [6] uses a data-centric method to discover and mitigate spurious correlations in the ImageNet dataset.\n3. Tables 1, 2, 3, and Figure 6 lack error bars. It would improve the scientific rigor of the paper to run these experiments over multiple random seeds and provide standard deviations or confidence intervals.\n4. There are a couple typos and grammatical errors in the writing, e.g., on lines 482 and 484. Also, the bibtex could use an update, as some references are out of date (e.g., Kirichenko et al. -- [21] in the paper -- is listed as an ArXiv preprint but appeared at ICLR 2023).\n\n***References***\n\n[1] Taghanaki et al. “MaskTune: Mitigating Spurious Correlations by Forcing to Explore”. NeurIPS 2022.\n\n[2] Tsirigotis et al. “Group Robust Classification Without Any Group Information.” NeurIPS 2023.\n\n[3] Lee et al. “Diversify and Disambiguate: Learning From Underspecified Data.” ICLR 2023.\n\n[4] LaBonte et al. “Towards last-layer retraining for group robustness with fewer annotations”. NeurIPS 2023.\n\n[5] Qiu et al. “Simple and Fast Group Robustness by Automatic Feature Reweighting.” ICML 2023.\n\n[6] Moayeri et al. “Spuriosity Rankings: Sorting Data to Measure and Mitigate Biases.” NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review, and address their questions below.\n\n**[Further comparisons to the no-information regime]**\n\nOur goal was to include the strongest baselines (to the best of our knowledge) and show that our methods perform better/comparably to them. For instance, since Auto-DDA outperforms most supervised baselines, it also outperforms weaker unsupervised (e.g., no information) baselines. However, we would be happy to include a table in the appendix in the final version of the paper with the suggested baselines.\n\n**[Additional Reference and Typos]**\n\nThank you for the additional reference! We will include this in the final revision of our paper, and will additionally fix the typos you mentioned. \n\n**[Error bars]**\n\nWe thank the reviewer for this feedback. We’ve added versions of Table 1 and Figure 6 with error bars to the global rebuttal document. Figure 2 already has a confidence region over 10 runs, but is also duplicated in the global rebuttal document.\n\n**[Hyperparameter k]**\n\nThe hyperparameter k from D3M (number of examples to remove) is not the same as the k from TRAK (dimensionality of gradient projection). We will change the variable name for the final revision.\n\n**[Initialization]**\n\nOur focus in this paper is the impact of the training dataset, so we used randomly initialized models to keep the setup straightforward (though our method does not depend on the initialization scheme). We then switched to ImageNet initialization for WaterBirds specifically because we found that the base model performance is extremely poor and noisy when initialized from scratch.\n\n**[TRAK/Auto-TRAK]**\n\nWe apologize for the confusion here: we used to call our method TRAK and Auto-TRAK, and then changed the names to D3M and Auto-D3M (but forgot to change the appendix), We will fix this for the final revision\n\n**[Qualitative Comparison]**\n\nWe thank the reviewer for the suggestion! While we didn’t have time in this response period to experiment with other data selection techniques as suggested, we agree that this would be an interesting avenue for further investigation.\n\n**[Computation Runtime]**\n\nWe appreciate the reviewer’s concern about the computational expense of our approach. We use TRAK as a subroutine in our method: a detailed analysis of the wall-clock time for TRAK can be found in Figure 1 of their paper [1]. Specifically, as noted in Appendix A.4 in their paper, the construction and inversion of the covariance matrix of projected gradients is actually quite cheap (this matches our experience): indeed, more of the runtime comes from training the models for the ensemble (TRAIN_TIME in their computation). Here, as noted in Appendix E of the TRAK paper, TRAK can be further optimized by re-using multiple checkpoints of the same model for the ensemble, further tuning the projection dimension, or even by reducing the number of models from 100 (as noted in their Appendix E.1). For simplicity’s sake, we did not explore any of these optimizations in our work, but would happy to discuss them further for the final revision.\n\nStepping back, TRAK is simply a subroutine called by D3M and can be replaced by more performant and efficient datamodeling/data attribution methods as they emerge. The rapid pace of data attribution as a field suggests that the computational cost of this subroutine will continue to decrease with time.\n\n[1] Park, Sung Min, et al. \"Trak: Attributing model behavior at scale.\" arXiv preprint arXiv:2303.14186 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for your reviews! We respond to the questions of each reviewer individually below. We additionally include results with error bars in the attached PDF as requested by Reviewer yCP5.

Meta-review: This paper proposes a new approach to group robustness by identifying and removing specific examples that drive poor performance on minority groups (thus called "dataset debiasing"). The reviewers are broadly enthusiastic about the contribution. While reviewers initially raised concerns, e.g. along the lines of evaluation of selected classes/groups on datasets and comparison with only a few benchmarks, most of these were resolved in the rebuttal and discussion phase. 

I encourage the authors to incorporate all of the new experiments/empirical results that were provided during the rebuttal and discussion phase in the camera-ready version, as these were instrumental to the paper's acceptance. I also encourage the authors to take seriously all of the suggestions that were made for formatting improvement.
Decision: Accept (poster)