Improving Subgroup Robustness via Data Selection

Authors: saachi  jain
kimia  hamidieh
kristian  georgiev
andrew  ilyas
marzyeh  ghassemi
aleksander m ądry mit
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:35:22

Abstract: Machine learning models can often fail on subgroups that are underrepresented
during training. While dataset balancing can improve performance on
underperforming groups, it requires access to training group annotations and can
end up removing large portions of the dataset. In this paper, we introduce
Data Debiasing with Datamodels (D3M), a debiasing approach
which isolates and removes specific training examples that drive the model's
failures on minority groups. Our approach enables us to efficiently train
debiased classifiers while removing only a small number of examples, and does
not require training group annotations or additional hyperparameter tuning.

Main paper: 
 introduction: 
 The advent of large datasets such as OpenImages [23] and The Pile [13] has led to machine learning models being trained on explicit [4] and illegal [48] content, or on data that encode negative societal biases [6, 12, 2, 8] and other spurious correlations [32, 30] . On one hand, there is increasing evidence that models reflect the biases in these datasets; on the other hand, the enormous scale of these datasets makes it extremely expensive to manually curate them-and so removal of such "harmful data" is challenging.In this paper, we propose an approach that aims to remove data responsible for biased model predictions. In particular, we focus on a specific way of quantifying model bias-worst-group errorwhich captures the extent to which model performance degrades on pre-defined subpopulations of the data. We aim to identify (and remove) the points in the training dataset that contribute most to this metric to improve the model's group robustness.The challenge inherent in this approach is that it requires an understanding of how training data affect machine learning model predictions. To overcome this challenge, we first approximate predictions as simple, direct functions of the training dataset, using a framework called datamodeling [18, 33] . We can then write our quantitative notion of model bias (which is a function of predictions) as a function of the dataset. Finally, by studying this function, we identify the training data points that contribute most to this measure of model bias. With the resulting method, which we call Data Debiasing with Datamodels (D3M), we show that, across a variety of datasets, there are often a small number of examples that disproportionately drive worst-group error. Removing these examples, in turn, greatly improves models' worst-group error while maintaining dataset size.Roadmap & contributions. In the rest of this paper, we present and demonstrate the effectiveness of our Data Debiasing with Datamodels (D3M). Concretely, we show that D3M enables us to:• Pinpoint examples that harm worst-group accuracy. We show that there are often a small number of examples that disproportionately drive models' worst-group error on validation data. For example, on CelebA-Age, our method improves worst group error over Figure 1 : Our method (D3M) improves worst group accuracy by identifying and removing the training samples which most negatively impact worst-group accuracy. Specifically, we use TRAK [33] to identify examples that exacerbate the discrepancy in group performance. We then remove and re-train a model on the remaining data.a natural baseline (data balancing) while removing 2.4× fewer examples. Furthermore, these offending examples often form coherent subpopulations within the data. • Achieve competitive debiasing performance. Our approach outperforms standard approaches (both model-based and data-based) to improving worst-group accuracy [25, 21, 17] , and is able to match the performance of methods which use ground-truth training group annotations [40] . • Discover unlabeled biases. When validation group labels are unavailable, we show how to extract hidden biases (i.e., unlabeled subgroups) directly from the data. As a result, we can perform end-to-end debiasing without any group annotations.We present our method in Section 4, and demonstrate these capabilities in Section 5. In Section 6, we leverage our framework to discover and mitigate biases within the ImageNet dataset, where D3M surfaces coherent color and co-occurrence biases. We then debias the model according to these failures, and improve accuracy on the identified populations.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The abstract/introduction clearly states the method that is presented and the results and case study of our experiments.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
the group robustness problem: 
 We consider an (unobserved) data distribution D over triplets (x i , y i , g i ), each comprising an input x i ∈ X , a label y i ∈ Y, and a subgroup label g i ∈ G, where G is the set of distinct subpopulations in the data. As a running example, consider the CelebA age classification task-here, we take the inputs x i to be images of faces, the labels y i to be either "old" or "young," and the possible group labels to be "old man", "old woman", "young man", and "young woman" (see Figure 1 ).Given a training dataset S train and a (small) validation dataset S val , the goal of the group robustness problem is to produce a classifier f that minimizes the worst-case loss over groups, i.e.,EQUATIONwhere (•, •) is a loss function. When is the 0-1 loss, Equation ( 1) is (one minus) the worst-group accuracy (WGA) of the classifier f , which we use to quantify success in the remainder of this work.Standard loss minimization can yield models that perform poorly with respect to (1) . For instance, returning to our example of CelebA age classification, suppose there was a spurious correlation between age and gender in the training set S train , such that old men and young women are overrepresented. A predictor that minimizes loss on S train might leverage this correlation, and thus perform poorly on the underrepresented subgroups of old women or young men.In practice, subgroup labels g i can be expensive to collect. Thus, approaches to the subgroup robustness problem vary in terms of whether we observe the group label g i in the training set S train and in the validation set S val . In particular, there are three settings of interest:• Full-information (Train / Val ): We observe the group labels for both the training dataset S train and validation dataset set S val .• Partial-information (Train / Val ): We observe the group labels for the validation set S val , but not for the (much larger) training set S train .• No-information (Train / Val ): We do not have group information for either S train or S val . Note that theoretically this setting is unsolvable, since for any non-perfect classifier f , there exists an assignment of group labels so that the worst-group accuracy is zero. Nevertheless, subgroups of relevant practical interest typically have structure that allows for non-trivial results even with no information.In this work, we focus on the partial-information and no-information settings, since acquiring group labels for the entire training set is often prohibitively expensive. Still, in Section 5, we show that our proposed methods (D3M for the partial-information setting, and AUTO-D3M for the no-information setting) perform comparably to full-information approaches.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Our work talks about the group annotations needed for the method, which informs our discussion of D3M vs Auto-D3M.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
related work: 
 Before introducing our method, we discuss a few related lines of work.Approaches to subgroup robustness. The group robustness problem (Section 2) has attracted a wide variety of solutions (see, e.g., [3, 20, 40, 25, 21, 38] ). Broadly, these solutions fall into one of two categories-model interventions and data interventions. Model interventions target either model weights [42, 45] or the training procedure [40, 21] . Data interventions, on the other hand, seek to improve worst-group accuracy by modifying the training dataset. For example, data balancing removes or subsamples examples so that all subgroups are equally represented. Idrissi et al. [17] find that this simple approach can performs on par with much more intricate model intervention methods.In this work, we focus on data interventions, for two reasons. First, it is often training data that drives models' disparate performance across groups [27] , e.g., via spurious correlations [32] or underrepresentation [6] . Second, data interventions do not require any control over the model training procedure, which can make them a more practical solution (e.g., when using ML-as-a-service). Indeed, since data intervention approaches only manipulate the dataset, they are also easy to combine with model intervention techniques.Compared to our work, the main drawback of existing data interventions is that they often (a) require subgroup labels for the training data (which might not be available), and (b) hurt the models' natural accuracy on skewed datasets [7, 44] . In this work we circumvent these limitations, by proposing a data-based approach to debiasing that can preserve natural accuracy without access to subgroup information.Bias discovery. Another related line of work identifies biases in machine learning datasets and algorithms. For the former, previous works have shown that large, uncurated datasets used for training machine learning models often contain problematic or biased data [4, 5, 48] . Raji et al. [39] show that data bias can be a hurdle towards deploying functional machine learning models. Nadeem et al. [29] curate a dataset to estimate bias in NLP models. Adebayo et al. [1] show that label errors can disproportionately affect disparity metrics.On the learning algorithm side, Shah et al. [47] and Puli et al. [37] show that the inductive bias of neural networks may encourage reliance on spurious correlations. Pezeshki et al. [34] leverage two networks trained on random splits of data while imitating confident held-out mistakes made by its sibling to identify the bias. Shah et al. [46] show that algorithmic design choices (e.g., the choice of data augmentation) can significantly impact models' reliance on spurious correlations. Finally, there has been a variety of work on "slice discovery" [19, 10] , where the goal is to discover systematic errors made by machine learning models.Data selection for machine learning. Our work uses data selection to improve subgroup robustness of machine learning models. A recent line of work has explored data selection for improving various measures of model performance. For example, Engstrom et al. [9] leverage datamodeling [18, 33] to select pretraining data for LLMs. Similarly, Xia et al. [52] and Nguyen and Wong [31] select data for finetuning, and in-context learning, respectively. In another related work, Wang et al. [49] propose a method to reweight training data in order to improve models' fairness.Many of these works leverage contributive data attribution methods to select data that improves model performance [51] . Koh and Liang [22] , Feldman and Zhang [11] , Schioppa et al. [43] , and Hammoudeh and Lowd [15] propose using different variants of influence functions. Ghorbani and Zou [14] leverage connections to Shapley values, a concept from game theory. Pruthi et al. [36] use a heuristic to estimate the contribution of each data point to model performance. Ilyas et al. [18] and Park et al. [33] use the datamodeling framework.4 Debiasing datasets with datamodeling (D3M)In this section, we present our data-based approach to training debiased classifiers. The main idea behind our approach is to identify (and remove) the training samples that negatively contribute to the model's worst-group accuracy, by writing model predictions as a function of the training data.Preliminaries. Let S = {(x 1 , y 1 ), . . . , (x n , y n )} be a dataset of input-label pairs. For any subset of the dataset-as represented by indices D ⊂ [n]-let θ(D) ∈ R p be the parameters of a classifier trained on D. Given an example z = (x, y), let f (z; θ) be the correct-class margin on z of a classifier with parameters θ (defined as log( p 1-p ), where p is the confidence assigned to class y for input x). A datamodel for the example z is a simple function that predicts f (z; θ(D)) directly as a function of D, i.e., a function fz : 2[n] → [0, 1] such that fz (D) ≈ f (z; θ(D)) for D ⊂ [n].Recent works (e.g., [18, 24, 33] ) demonstrate the existence of accurate linear datamodels-functions p that decompose additively in terms of their inputs D. In other words, these works show that one can compute example-specific vectors τ (z) ∈ R n such that fz (D) :EQUATIONThe coefficients τ (z) i have a convenient interpretation as quantifying the "importance" of the i-th training sample to performance on example z (i.e., as a data attribution score [15] ). In what follows, we will assume access to coefficients τ (z) for any example z-at the end of this section, we will show how to actually estimate the coefficient vectors τ (z) efficiently.Debiasing approach. How can we leverage datamodeling to debias a dataset? Recall that our goal is to remove the samples in S that lead to high worst-group error. Stated differently, given a dataset S of size n, we want to maximize the worst-group performance of a classifier θ(D) with respect to the indices D ⊂ [n] that we train on.Our main idea will be to approximate the predictions of θ(D) using the corresponding datamodels fz (D). To illustrate this idea, suppose that our goal was to maximize performance on a single test example z, i.e., arg max D f (z; θ(D)). We can approximate this goal as finding arg max D fz (θ(D)): then, due to the linearity of the datamodel fz , the training samples that hurt performance on z are simply the bottom indices of the vector τ (z).Now, this analysis applies not only to a single example z, but to any linear combination of test examples. In particular, if we wish to maximize performance on a linear combination of validation examples, we simply take the linear combination of their coefficients, and remove the training examples corresponding to the smallest coordinates of the averaged vector.Debiasing with group-annotated validation data. Given a set of validation samples for which the group labels g i are observable, our last observation gives rise to the following simple procedure:1. Compute group coefficients τ (G) for each G. Since we have group annotations for each validation sample, we can define a vector τ (G) for each group G ∈ G as simply the average τ (z) within each group.2. Compute group alignment. Next, we compute a group alignment score A i for each training sample i ∈ [n], which captures the the impact of the sample on worst-group performance. Since there may be many low-performing groups, we use a "smooth maximum" function to weight each group according to its average loss. Thus, for a training example i,A i = g∈G exp(β g ) • τ (g) i g ∈G exp(β g ), where we set hyperparameter β = 1.Here, g is the loss of a base classifier θ(S) on group g (evaluated on the given validation set). S new = arg top-k({A i : z i ∈ S train }).We make two brief observations about hyperparameters before continuing. When computing the group alignment score in Step 2, the hyperparameter β controls the temperature of the soft maximum function in (3) . When β → 0, the group alignment A i measures the impact of the i-th training example on the "balanced" performance (treating all groups equally). As β → ∞, A i collapses to the training example's importance to only the worst group, which is suboptimal if models perform poorly on more than one group. For simplicity, we take β = 1 and refrain from tuning it.Another hyperparameter in the algorithm above is the number of examples to remove, k. We consider two different ways of setting this hyperparameter. One approach is to search for the value of k that maximizes worst-group accuracy on the validation set S val . Alternatively, we find that the simple (and much more efficient) heuristic of removing all examples with a negative group alignment score (i.e., examples for which A i < 0) tends to only slightly over-estimate the best number of examples to remove (see, e.g., Figure 2 ). Thus, unless otherwise stated, we use this heuristic when reporting our results.Debiasing without group annotations. Our procedure above relies on group annotations for a validation set S val to compute the "per-group coefficients" τ (G). In many real-world settings, however, models might exhibit disparate performance along unannotated subpopulations-in this case, we might not have a validation set on which we can observe group annotations g i . Can we still fix disparate model performance in this setting?In general, of course, the answer to this question is no: one can imagine a case where each individual example is its own subpopulation, in which case worst-group accuracy will be zero unless the classifier is perfect. In practical settings, however, we typically care about the model's disparate performance on coherent groups of test examples. The question, then, becomes how to find such coherent groups.We posit that a unifying feature of these subpopulations is that they are data-isolated, i.e., that models' predictions on these coherent groups rely on a different set of training examples than models' predictions on the rest of the test data. Conveniently, prior works [18, 46] show that to find dataisolated subpopulations, one can leverage the datamodel matrix-a matrix constructed by stacking the datamodel vectors τ (z) for each test example. Intuitively, the top principal component of this matrix encodes the direction of maximum variability among the vectors τ (z). This approach (which we call AUTO-D3M), enables us to perform end-to-end debiasing without any group annotations. This method proceeds in four steps. For each class:1. Construct a matrix T of stacked attribution vectors, where T ij = τ (z i ) j .2. Let v be the top principal component of T.3. Project the attribution vector τ (z) onto v and construct "group pseudo-labels"g i = 1{τ (z i ) v ≥ λ}.where λ is a hyperparameter2 4. Apply D3M with the group pseudo-labels to train a debiased classifier.Estimating the coefficients τ (z). In order to operationalize D3M and AUTO-D3M, it remains to show that we can actually estimate coefficients τ (z) satisfying (2). To accomplish this, we use a method called TRAK [33] . Leveraging differentiability of the model output f (z; θ) with respect to the model parameters θ, TRAK computes the coefficient vector τ (z) for an example z as follows: τ (z) i i-th coefficient for example z = g(z)   zj ∈S g(z j ) • g(z j )   -1 g(z i ) • (1 -σ(f (z; θ * )))(e) Repeat steps (a)-(d) for T trials, and average the results to get a final coefficient vector τ (z).The trials are identical save for the randomness involved in step (a).We provide more intuition and details behind TRAK in Appendix A.A note on scalability In terms of computational cost, TRAK involves taking a single backward pass on each of the training and validation examples to compute the model's gradient. The (projected) gradients are then saved to compute TRAK scores. Typically, TRAK is computed over T trials: following the original paper we use T = 100. However, our approach can be used with any datamodeling technique.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
results: 
 In Section 4, we presented D3M-an approach for debiasing a classifier by identifying examples which contribute to a targeted bias. In this section, we validate this framework by assessing its performance on tasks with known biases.We consider four classification tasks where there is a spurious correlation between the target label and a group label in the training dataset: CelebA-Age [26, 19] , CelebA-Blond [26] , Waterbirds [41] , and MultiNLI [50] . We provide more information about the datasets in Appendix B.1, and other experimental details in Appendix B.2.We first evaluate D3M and AUTO-D3M quantitatively, by measuring the worst-group accuracy of models trained on the selected subsets of the biased datasets above.D3M: Debiasing the model in the presence of validation group labels. In Table Moreover, D3M performs on par with methods that have full access to both training and validation group labels.AUTO-D3M: Discovering biases with TRAK. We now consider the case where validation group labels are not accessible. Using AUTO-D3M, we debias our model using pseudo-annotations derived from the top principal component of the TRAK matrix (AUTO-D3M in Table 1 ) 4 . Note that AUTO-D3M is the only method other than ERM that does not require either train or validation group labels. Despite this, AUTO-D3M achieves competitive worst-group accuracy in our experiments. We emphasize that AUTO-D3M does not require group labels at all-in particular, we do not use group labels to do hyperparameter selection or model selection when we retrain. CelebA-Age. Subpopulations such as "old" with "bushy eyebrows" or "young" with "gray hair" have particularly negative scores.What type of data does our method flag? In particular, do the examples we identify as driving the targeted bias share some common characteristics? To test this hypothesis, we inspect the data removed by our method and identify subpopulations within the majority groups that are disproportionately responsible for the bias. We then retrain the model after excluding all training examples from the identified subpopulations and show that this is a viable strategy for mitigating the underlying bias.Finding subpopulations responsible for model bias. Consider the running example from Figure 1 where we train a model on the CelebA-Age dataset to predict whether a person is "young" or "old" when gender is a spurious feature (such that young women and old men are overrepresented).CelebA-Age has a variety of annotations beyond age and gender, such as whether the person is wearing eyeglasses. In this section, we use these extra annotations to identify coherent subpopulations that are flagged by our methods.In particular, we consider subpopulations formed by taking the Cartesian product of labels and annotations, e.g., subpopulations of the form ("young", "wearing eyeglasses"). For each of these subpopulations, we calculate the average group alignment score A i of the training examples within that subpopulation (see Figure 4 ). We find that subpopulations such as "young" with "gray hair" or "old" with either "5 o'clock shadow" or "busy eyebrows" have particularly negative group alignment scores. In Figure 3 , we show examples from the subpopulations with the most negative group alignment scores, and observe that a large fraction of the examples in these subpopulations contain labeling errors.Retraining without identified subpopulations. Once we have identified subpopulations with negative alignment scores, a natural strategy for mitigating the underlying bias is to exclude these subpopulations from the training set. To explore this approach, we exclude the five subpopulations with the most negative attribution scores on average from the CelebA-Age dataset: "Young" + "Gray Hair", "Old"+ "5 o'Clock Shadow", "Old" + "Bushy Eyebrows", "Young" + "Blond Hair", and "Old" + "Sideburns." After retraining the model on this modified training set, we get a worst-group accuracy (WGA) of 68.4%-an improvement of ~12% over the WGA of the original model (56.7%).  
case study: finding and mitigating model failures on imagenet: 
 In Section 5 we evaluated D3M and AUTO-D3M on datasets where the bias was already known. We now deploy AUTO-D3M to discover and mitigate biases within the ImageNet dataset, which does not have a predefined bias or available group annotations.Identifying ImageNet biases. We use TRAK to compute a coefficient matrix T (see Step 1 of AUTO-D3M in Section 4) for a held out validation split (10% of the training set). Focusing on seven ImageNet classes, we use the first principal component of the matrix T to identify potential biases. In Figure 5 , we display the most extreme training examples according to the top principal component for four of these classes. PCA identifies semantically color and co-occurrence biases (e.g., tench fishes with or without humans or yellow/white cauliflowers that are either cooked or uncooked.) In fact, our identified biases match the challenging subpopulations in Jain et al. [19] and Moayeri et al. [28] . Mitigating ImageNet biases with AUTO-D3M. For each of the four targeted ImageNet classes, we seek to mitigate the identified failure modes with AUTO-D3M. We consider two settings based on the level of human intervention. In the first, we manually assign each of the validation images to a group according to a human description of identified bias (e.g., an image of a tench is in group 1 if a human is present and group 2 otherwise), and then use those group labels with D3M. 5 In the second setting, we debias in a purely automatic fashion, using AUTO-D3M to derive pseudo-group labels from the top principal component. In Figure 6 , we display worst group accuracy on the test images of the targeted class (evaluated using manual group assignments of the 50 test examples). Both D3M and AUTO-D3M improve worst group accuracy over ERM without significantly impacting the overall ImageNet accuracy (see Appendix C.2). 
conclusion: 
 We propose Data Debiasing with Datamodels (D3M), a simple method for debiasing classifiers by isolating training data that disproportionately contributes to model performance on underperforming groups. Unlike approaches such as balancing, our method only removes a small number of examples and does not require training group annotations or additional hyperparameter tuning. More generally, our work takes a first step towards data-centric model debiasing. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: Not applicable (datasets are open source)Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original papers for our datasets and TRAK.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Let Z be the input space and S be a training set of interest. For a given training subset S ⊂ S and target example z, data attribution techniques seek to estimate the attribution score of z -that is, the change in the model's prediction on z when the model is trained on the subset S . More formally, let f (z, θ(S)) be the model's output function on example z. Then we can define τ (z) i as the attribution score of the ith training example z i on target example z τ (z) i = f (z, θ(S)) -f (z, θ(S\z i )).While τ (z) is relatively straightforward to compute for linear models [35] , computing this influence is far more challenging for neural networks. Thus, in order to approach this problem, TRAK first approximates f (z; θ(S)) as a linear model on top of the gradients ∇ θ (f ; θ * ) of the original neural network after convergence. We can then plug this approximation into the estimate for linear classifiers to approximate τ (z). After simplification, the TRAK estimate of the influence of z isτ (z) = -φ(z) T (Φ T Φ) -1 ΦQwhere φ(z) = ∇f (z; θ(S * )) are the (randomly projected) gradients on example z, Φ are the stacked training gradients Φ = [φ(z 1 ), ..., φ(z n )], and Q is a normalization matrix.In this section, we describe the datasets, models and evaluation procedure that we use throughout the paper.Datasets. In order to cover a broad range of practical scenarios, we consider the following image classification and text classification problems.• Waterbirds [40] is a binary image classification problem, where the class corresponds to the type of the bird (landbird or waterbird), and the background is spuriously correlated with the class. Namely, most landbirds are shown on land, and most waterbirds are shown over water.• CelebA-Blond [26] is a binary image classification problem, where the goal is to predict whether a person shown in the image is blond; the gender of the person serves as a spurious feature, as 94% of the images with the "blond" label depict females.• CelebA-Age [26, 19] is a binary image classification problem, where the goal is to predict whether a person shown in the image is young; the gender of the person serves as a spurious feature. For this task, we specifically subsample the training set such that the ratio of samples in the majority vs. minority groups is 4:1.• MultiNLI [50, 40] is a classification problem where given a pair of sentences, the task is to classify whether the second sentence is entailed by, neural with, or contradicts the first sentence. The spurious attribute from Sagawa et al. [40] describes the presence of negation words, which appear more frequently in the examples from the negation class.Methods. We benchmark our approach against the following methods:• ERM is simple empirical risk minimization on the full training set.• RWG [17] is ERM applied to random batches of the data where the groups are equally represented with a combination of upsamping and downsampling such that the size of the dataset does not change.• SUBG [17] is ERM applied to a random subset of the data where we subsample all groups such that they have the same number of examples.• GroupDRO [40] trains that minimizes the worst-case performance over pre-defined groups in the test dataset.• Just Train Twice (JTT) [25] trains an ERM model with upsamping initially misclassified training examples by an initial ERM model. • DFR [21] trains an ensemple of linear models on a balanced validation set, given ERM features.In this section, we detail the model architectures and hyperparameters used by each approach. We used the same model architecture across all approaches: Randomly initialized ResNet-18 [16] for CelebA and ImageNet-pretrained ResNet-18s for Waterbirds. We use the GroupDRO implementation by Sagawa et al. [40] and DFR implementation by Kirichenko et al. [21] .For all approaches, we tune hyperparameters for ERM-based methods (ERM, DFR, and D3M) and re-weighting based methods (RWG, SUBG, GroupDRO and JTT) separately. For RWG, SUBG, GroupDRO and JTT, we early stop based on highest worst-group accuracy on the validation set as well. We optimize all approaches with Adam optimizer.For the CelebA dataset, we all methods with learning rate 1e -3, weight decay 1e -4, and batch size 512. We train RWG, SUBG, GroupDRO and JTT with learning rate 1e -3, weight decay 1e -4, and batch size 512. We train all models for the CelebA-Age task to up to 5 epochs and all models for CelebA-Blond task up to 10 epochs.For the Waterbirds dataset, we train the approaches that use the ERM objective (including D3M) with learning rate 1e -4, weight decay 1e -4, and batch size 32. We train RWG, SUBG, GroupDRO and JTT with learning rate 1e -5, weight decay 0.1, and batch size 32. We train all models to up to 20 epochs.For all other hyperparameters, we use the same hyperparameters as Kirichenko et al. [21] for DFR and the same hyperparameters as Liu et al. [25] for JTT.We report the performance of the models via Worst-group Accuracy, or Balanced Accuracy in Table 2 , which is the average of accuracies of all groups. If all groups in the test set have the same number of examples, balanced accuracy will be equivalent to average accuracy.Our model was trained on a machine with 8 A100 GPUs.Below we include the balanced accuracies for the experiments in Table 2 . NeurIPS Paper ChecklistFor our experiments we choose λ so that the lower performing group consists of35% of the validation examples of that class.3 Note that WaterBirds has more worst-group examples in the val split (133) than the train split (56). Since DFR directly fine-tunes on the validation set, it has an advantage here over other methods.For MultiNLI , we chose the PCA component by inspection that captures examples with/without negation.Here, we only consider the target class when computing the loss weighting. As a result, the heuristic overestimates the number of examples k to remove, and so we instead search for the optimal k using our held out validation set. 


Decision: This paper proposes a new approach to group robustness by identifying and removing specific examples that drive poor performance on minority groups (thus called "dataset debiasing"). The reviewers are broadly enthusiastic about the contribution. While reviewers initially raised concerns, e.g. along the lines of evaluation of selected classes/groups on datasets and comparison with only a few benchmarks, most of these were resolved in the rebuttal and discussion phase. 

I encourage the authors to incorporate all of the new experiments/empirical results that were provided during the rebuttal and discussion phase in the camera-ready version, as these were instrumental to the paper's acceptance. I also encourage the authors to take seriously all of the suggestions that were made for formatting improvement.

Reviews: review: paper_summary: paper_summary: The paper proposes Data Debiasing with Datamodels (D3M), a method to improve machine learning model performance on underrepresented subgroups by removing specific training examples that cause failures. Unlike traditional balancing methods, D3M efficiently debiases classifiers without needing group annotations, significant dataset reductions or additional hyperparameter tuning.
strength_weakness: strength_weakness: - Writing and Format: the presentation of the paper needs readability improvement:
  - Redundant section start: Line 81
  - Excessive parenthetical comments and irregular format: Lines 20, 25, 28, 32, 67-71, 83, etc.

- In addition to isolating problematic training data, experiments should be conducted to assess the impact on the necessity of further hyperparameter tuning and to strengthen the case for the effectiveness of the proposed method.
questions: questions: N/A
limitations: limitations: See weaknesses
paper_summary: paper_summary: This paper introduces a new data debiasing technique called Debiasing with Data Attribution (DDA). DDA utilizes data modelling framework to identify and eliminate training examples that negatively impact the accuracy of the worst-performing groups. Additionally, the paper presents AUTO-DDA, an extension of DDA that can identify biases even without prior knowledge of group information. The proposed methods are validated through experiments on various datasets such as CelebA-Age, CelebA-Blond, Waterbirds and MultiNLI.
strength_weakness: strength_weakness: 1. The performance of ImageNet is only reported on selected classes. How are the classes selected for evaluation? Is it based on the amount of bias present in the classes? 
2. I am unsure if the proposed approach is effective when the majority of the data consists of bias-aligned points. For example, if there are only a few conflicting points and the rest are bias-aligned, how will the data be removed? I doubt the approach would still be useful for debiasing since a large part of the data is still going to be majorly biased.  Even if the authors claim that the majority of the bias aligned points will be removed, I believe the model would still overfit to the data since the final dataset would be extremely small. Analyzing the performance of the approach with varying numbers of bias-conflicting points (1%, 5%, 10% of CMNIST(10 class classification)) in the dataset would be beneficial to understand this scenario. This experiment would provide insights into how well the approach scales to real-world scenarios where the degree of bias is significantly high.
questions: questions: Please refer to the questions in the weakness section.
limitations: limitations: Limitations are discussed.
paper_summary: paper_summary: The paper introduces a method called Data Debiasing with Datamodels (D3M) that addresses the problem of model bias (using the worst-case loss over groups as the metric). The approach leverages a process known as datamodeling to predict model behavior based on training data influence, focusing on removing data points that contribute heavily to worst-group error. The paper illustrates D3M’s effectiveness across various datasets, showing that it can outperform both traditional model and data intervention strategies. Moreover, the method is adaptable to the setting without explicit subgroup labels.
strength_weakness: strength_weakness: One weakness of the method is its exclusive focus on improving worst-group accuracy without presenting results on how it might affect the overall accuracy for all groups. This raises concerns about potential trade-offs, where enhancing fairness for the worst-performing subgroup could compromise the model's general performance. Additionally, the paper does not thoroughly explore how different model configurations might influence the outcomes. Understanding how variations in model architectures, initial parameter settings, or training procedures affect the effectiveness of the method is useful for validating its robustness and adaptability to diverse scenarios. Finally, a relatively minor weakness is that the demonstration of the paper could be more organized and coherent.
questions: questions: After improving the worst-group accuracy, does the model still maintain good overall accuracy? How does the method impact the performance across all groups? Were the results of the method tested across various model architectures to confirm its generalizability? In scenarios lacking explicit group labels, were there any experiments conducted to assess the effectiveness of the pseudo group labeling approach using the datamodel matrix in the setting or case studies in this paper?
limitations: limitations: The paper presents relatively robust results. However, I do not see it adequately address the limitations of the methods.
paper_summary: paper_summary: This paper proposes a data-centric model debiasing technique to identify and remove data which harm worst-group accuracy. This method removes fewer data than standard balancing techniques and can be adapted for settings with and without group annotations. Experiments are provided on standard group robustness benchmark datasets, and the method is shown to promote bias discovery on ImageNet in the absence of group annotations.
strength_weakness: strength_weakness: 1. In Table 1, the only comparison to previous work provided for the no-information regime is ERM, which is generally understood to be a weak baseline for group robustness tasks. Some examples of comparisons I would expect to see in this setting include MaskTune [1], uLA [2], DivDis [3], or CB-LLR [4]. Similarly, in the partial-information regime, additional comparisons may include AFR [5] or SELF [4]. (I do not expect the authors to include all these comparisons, but it would benefit to discuss the most appropriate ones).
2. In Section 6, I believe a reference and comparison to [6] is missing. Similarly to this paper, [6] uses a data-centric method to discover and mitigate spurious correlations in the ImageNet dataset.
3. Tables 1, 2, 3, and Figure 6 lack error bars. It would improve the scientific rigor of the paper to run these experiments over multiple random seeds and provide standard deviations or confidence intervals.
4. There are a couple typos and grammatical errors in the writing, e.g., on lines 482 and 484. Also, the bibtex could use an update, as some references are out of date (e.g., Kirichenko et al. -- [21] in the paper -- is listed as an ArXiv preprint but appeared at ICLR 2023).

***References***

[1] Taghanaki et al. “MaskTune: Mitigating Spurious Correlations by Forcing to Explore”. NeurIPS 2022.

[2] Tsirigotis et al. “Group Robust Classification Without Any Group Information.” NeurIPS 2023.

[3] Lee et al. “Diversify and Disambiguate: Learning From Underspecified Data.” ICLR 2023.

[4] LaBonte et al. “Towards last-layer retraining for group robustness with fewer annotations”. NeurIPS 2023.

[5] Qiu et al. “Simple and Fast Group Robustness by Automatic Feature Reweighting.” ICML 2023.

[6] Moayeri et al. “Spuriosity Rankings: Sorting Data to Measure and Mitigate Biases.” NeurIPS 2023.
questions: questions: 1. Is the hyperparameter k from D3M (number of examples to remove) the same as the hyperparameter k from TRAK (dimensionality of the gradient projection)? If not, it would be helpful to use different letters and detail how the k in TRAK is chosen.
2. Why is random initialization used for CelebA, as opposed to standard ImageNet initialization? Do the CelebA comparisons in Table 1 also use random initialization? 
3. In the appendices, the tables reference proposed methods TRAK and Auto-TRAK. Is this meant to read D3M and Auto-D3M respectively?
4. While not strictly necessary, I would be curious to see a qualitative comparison of the results from Section 5.2 and Figures 3 and 4 with other data selection techniques from the robustness literature. How do the data with negative alignment scores compare with data selected via misclassification [1], disagreement [2], other influence functions [3, 4], or Shapley values [5]? Are negative alignment scores perhaps more interpretable than these other techniques?

***References***

[1] Liu et al. “Just Train Twice: Improving Group Robustness without Training Group Information.” ICML 2021.

[2] LaBonte et al. “Towards last-layer retraining for group robustness with fewer annotations”. NeurIPS 2023.

[3] Koh and Liang. “Understanding Black-box Predictions via Influence Functions.” ICML 2017.

[4] Feldman and Zhang. “What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation.” NeurIPS 2020.

[5] Ghorbani and Zou. “Data Shapley: Equitable valuation of data for machine learning.” ICML 2019.
limitations: limitations: While the limitations of group annotations are sufficiently discussed, I have an additional question about compute efficiency of the proposed method. The experiments were performed on 8x A100 GPUs, which is much more extensive than comparative methods (e.g., [1] only needs to train a linear classifier, which, being essentially a logistic regression, is very efficient). Is this level of compute necessary for D3M? I am mainly concerned about the TRAK subroutine of Auto-D3M, where in step (d) the covariance matrix of the projected gradients of the entire dataset is constructed and inverted T=100 times. How much wall-clock time and GPU VRAM does this step consume, and how does it scale with the hyperparameter k?

***References***

[1] Moayeri et al. “Spuriosity Rankings: Sorting Data to Measure and Mitigate Biases.” NeurIPS 2023.

Comments: title: Experiment with few bias-conflicting examples
Experiment is incoming!
Reply
Reply
Please read rebuttal and provide more substantive comments
Reply
comment: Thank you for your patience! We've run the experiment you requested. 

## Setup

For the MNIST data
- Let $\hat{y}$ be the binary signal according to the digit: (0 if $<5$ and 1 if $\geq 5$). The true label $y$ has a 90% chance of being $\hat{y}$ and 10% chance of being $1-\hat{y}$
- Let $y_{col} = y$ with $p_{corr}$ probability and $1-y$ otherwise. If $y_{col}$ is 1, color the image red, otherwise if 0 color the image green. 

Thus the color ($y_{col}$) aligns with the true label ($y$) with $p_{corr}$ probability and the digit $\hat{y}$ with 0.9 probability. If $p_{corr} > 0.9$, the ERM model relies more heavily on the color than the digit.

We take val as 10% of the training split of MNIST. We consider three splits for train where there are very few bias conflicting examples.
- $p_{corr} = 0.95$
- $p_{corr} = 0.99$
- $p_{corr} = 0.995$

And our test set has $p_{corr} = 0.1$ (so the bias alignment is reversed).

Here, the groups are the pair of $y, y_{col}$ (e.g., the final class and the color of the image.)

## Results

| $p_{corr}$    | ERM WGA | D3M WGA | Balancing WGA (oracle) | ERM Avg. Acc | D3M Avg. Acc | Balancing Avg. Acc (oracle) | Frac Examples Removed by D3M |
|:--------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:| :-------:|
| $p_{corr} = 0.95$  |  $20.52 \pm 0.85$  | $44.48 \pm 0.72$  | $78.66 \pm 1.82$ | $29.00 \pm 0.37$   |  $89.79 \pm 0.06$ | $82.78 \pm 0.44$ | 0.73  | 
| $p_{corr} = 0.99$  |  $1.95 \pm 0.25$  | $55.15 \pm 3.53$ | $69.98 \pm 3.21$ | $12.00 \pm 0.22$   |  $87.99 \pm 0.065$ | $75.86 \pm 0.66$ | 0.80 |
| $p_{corr} = 0.995$  |  $0.62 \pm 0.13$  |  $42.24 \pm 2.05$  | $26.16 \pm 17.62$ | $10.62 \pm 0.11$   | $87.31 \pm 0.21$ | $60.19 \pm 6.55$ | 0.83 |

**ERM vs. D3M**: We find that even at these more extreme scenarios of bias (where the vast majority of examples are removed), we can significantly improve both WGA and average test accuracy of the model using D3M over using ERM. Our performance does not significantly degrade even at $p_{corr} = 0.995$ where there are very few bias conflicting examples. 

**Number of examples removed**: Also, note that to balance the dataset, more than 90% of examples would need to be removed. Since our method is more sample efficient than balancing, we can remove a smaller fraction of the examples. Our heuristic for choosing $k$ (the number of examples to remove) *overshoots* the best number. In particular, the previously majority classes are now the worst performing groups in all cases: we can thus improve the accuracies further by removing *even fewer* examples (e.g., by searching for $k$ on the validation set).

**Comparison to balancing**: We further compare to balancing the dataset.  We note that this method requires *training group annotations*, while our method does not. In this simplified setting, most of the examples are equally harmful for WGA, so naive balancing can do well. Here balancing the dataset improves WGA over D3M for $p_{corr} = 0.95, 0.99$. However, balancing more significantly degrades overall accuracy as there are very few points remaining in the dataset. For $p_{corr}  = 0.995$, balancing the dataset requires such a small overall training dataset that both WGA and overall accuracy are harshly degraded compared to D3M.

We would be happy to include this result within our paper. We apologize for the delay: we worked hard over the weekend to get these results in (but were delayed due to some cluster issues). We would appreciate if the reviewer would revisit their decision to lower their score, given this experiment.
HI reviewer QjGu, 

We really appreciate your suggested experiment, and have been working for the last few days straight on implementing it and getting the results (we are somewhat compute-constrained at the moment)! We really appreciate your patience and should have results by the end of the day today.

- Authors
My concerns on performance on **Debiasing datasets with few bias-conflicting examples** has not been addressed, hence Iam lowering my score.
I have read the other reviews and the authors' responses. I look forward to the final draft of the paper, as the authors will be editing the camera-ready version to improve its organization. Regarding the content, I appreciate the insights of the other reviewers and would like to increase my score to 4.
I appreciate that the authors refer to Tables 2 and 3. It would be acceptable to keep them in the Appendix, but it would be better to highlight this with one or two sentences in the main body of the paper. Regarding the organization of the paper, I feel that Section 4 could benefit from more focused editing, as it encompasses discussions from multiple perspectives. It might be helpful to provide a clearer outline at the beginning of the section, or perhaps divide it into subsections with a summary at the start of each. For different model configurations, I believe it would be a significant weakness if comparisons are not conducted. I hope this will be addressed in the final version of the paper. I look forward to the final draft.
Dear reviewer FTVd, 

Your review appears to mostly mention formatting issues. Please read the authors' response to other reviews and provide comments regarding the content of the paper, if you have any.

Thanks,
AC
**[Selection of specific classes chosen for ImageNet experiments]** Thanks for clarifying this.

**[Debiasing datasets with few bias-conflicting examples]**
I request the authors to please validate the claim emperically, by training a simple 3 layer MLP using D3M for various percental of bias conflicting samples (0.05%, 1% and 5%) on CMNIST dataset where the proportion of bias conflicitng samples can be easily controlled as suggested in my intial review. Also ablating on various percentage of bias conflicing samples helps understand the sensitivity of the proposed methods on number of bias conflicitng samples about which im concerned about.
Thank you to the authors for the comprehensive response, and especially for the inclusion of additional figures with error bars. With the additional clarifications and discussion, I believe this paper will be interesting for the community and set a strong baseline for future work. Therefore, I recommend acceptance. I have raised my Soundness score to a 3 and overall rating to a 7.