An Efficient Subset Selection Strategy Using Text-Guided Data Attribution to Mitigate Simplicity Bias
Authors: Kumar Shubham
Pranav Sastry
Prathosh AP
Venue: Accepted by TMLR ()

Abstract: The effectiveness of deep learning models heavily relies on the quality and diversity of their training data. However, datasets collected from different sources often introduce simplicity biases, where a models rely on easily learnable but non-predictive (spurious) features for its predictions. While existing debiasing techniques focus on model robustness, they leave the data untouched.  However, as data becomes increasingly valuable, identifying and mitigating bias directly at the data level has become increasingly important. Recently, data attribution has emerged as a promising tool for uncovering issues in training data, yet its vulnerability to simplicity bias has received limited attention. In this work, we propose a novel data deletion framework that combines Neural Tangent Kernel (NTK)-based data attribution with textual descriptions of bias to identify and remove training samples that do not significantly affect model performance. We first demonstrate that NTK-based data attribution methods can themselves be influenced by spurious features. Subsequently, to mitigate this, we use available metadata or, when unavailable, a vision-language model to annotate a small validation set and extract a textual description of the bias. Based on this description and the attribution score, we identify the subset of training data that are semantically aligned with the spurious feature and affect the generalization of the model. Removing these samples from the training dataset and training model on the new subset improves the average and worst-group accuracy of the model, outperforming existing attribution-based baselines.

Published in Transactions on Machine Learning Research (02/2026)
An Efficient Subset Selection Strategy Using Text-Guided Data Attribution to Mitigate Simplicity Bias
Kumar Shubham
shubaamkuma&9iicc.c..m
Indian Institute of Science, Bangalore, India
Pranav Sastry
prenausastry@hisccac...
Indian Institute of Science, Bangalore, India
Prathosh AP
prathosh @risc.oc.it
LatentForce.0r oi Indian Institute of Science, Bangalore, India
Reviewed on OpenReview: https: opperrvvepeenvveeeue net/ formm? id= #25Yund795
Abstract
The effectivencss of deep learning models heavily relies on the quality and diversity of their training data. However, datasets collected from different sources often introduce simplicity biases, where models rely on easily learnable but non-predictive (spurious) features for its predictions. While existing debiasing techniques focus on model robustness, they leave the data untouched. However, data becomes increasingly valuable, identifying and mitigating bias directly al the data level has become increasingly important. Recently, data allribution has emerged as promising tool for uncovering issues in training data, yet its vulnerability to simplicity bias has received limited attention. In this work, we propose a novel data dele- tion framework that combines Neural Tangent Kernel (NTK)-based data attribution with textual descriptions of bias to identify and remove training samples that do not significantly affect model performance. We first demonstrate that NTK-based data attribution methods can themselves be influenced by spurious features. Subsequently, to mitigate this, we use available metadata or, when unavailable, vision-language model, to annotate small vali- dation set and extract a textual description of the bias. Based on this description and the attribution score, we identify the subset of training data that are semantically aligned with the spurious feature and affect the generalization of the model. Removing these samples from the training dataset and training model on the new subset improves the average and worst-group accuracy of the model, outperforming existing attribution- based basclines. Our code is available al. milips://githubb.omm
1 Introduction
The success of deep learning models strongly influenced by the quality and quantity of the dataset used for training (Bhatt et al. 2024 Whang al. 2023 Xu et al. 2024a Choe et al. 2024). These data are often
collected via web scraping Xu et al. 2024b Patel & Patel 2020), and external data providers (Berriel et al.
2017 Drutsa et al. 2019). However, such datasets can inadvertently contain illegal content (Thiel 2023 and
can encode negative socictal biases (Ferrara 2023 Jain et al. 2024) that can influence model performance.
In addition, data collected from such varied sources can introduce distributional shifts, where subpopulations with specific features may be overrepresented or underrepresented in the training data compared to the test data (Koh al.) 2021). For instance, a traffic sign classification task, significant portion of the training data collected online may consist of images captured on sunny days, while the model may later be deployed in winter or rainy conditions, leading to performance degradation.

---

Published in Transactions on Machine Learning Research (02/2026)
These imbalances can introduce simplicity bias (Tiwari & Shenoy 2023 Geirhos et al. 2020 McCoy 2019
where the model, due to high correlations between specific features and the prediction task, relies on sim- pler, non-robust (spurious) features instead of learning predictive features for classification. Several methods have been proposed to mitigate such biases. However, most of these approaches primarily focus on enhanc-
ing model nobustnnss by modifying the training process (Liu et al. 2021: Sagawa el al. 2020 Lee et al. 2021 Liu et al. 2022 LaBonte et al. 2023) and leverage the same training dataset to improve performance across various settings. Since these methods operate directly on the dataset from which the biases orig- inate, they may fail to fully address underlying distributional disparities. Moreover, in certain scenarios, these changes in the training process can inadvertently affect the model's susceptibility to adversarial at-
Neerudu et al. 2023 Holtz et al. 2022), and may conflict with Wong & Kolter 2018) that mandate specific training prolocols to preserve theoretical guarantees (see Appendix for further details). Further, given the proprietary value 2022) and the significant financial cost associated with curating new datasots, it has become increasingly important to address these challenges directly at the data level.
tacks (Hartnett et al. 2019 Li et al.
2025
regulatory constraints
(Wang et al. 2018
of data (Xiong et al.
A viable alternative in these scenarios could be to remove training samples associated with spurious fea- tures (Chaudhuri et al.
2023 Idrissi et al. 2022 Jain et al. 2024), while ensuring that removal of such
samples doesn't 't hurt the overall performance of the model, as in data attribution and Seave-OneeOur (LOO) techniques (Koh & Liang
Park et al.] 2023 Data attribution methods aim to estimate a model's performance when specific training samples are excluded, enabling the evaluation of counterfactual scenarios such as assessing the impact on test accuracy certain subsets of the training data were omitted (Tanno et al., 2022 Engstrrom et al.] 2024 Xia et. al. 2024). However, many of these methods are computationally expensive and can underperform for non-convex training objective. Recent advancements in data altribution methods, such as Trak (Park et al. 2023), leverage neural langent kernels (NTK) to enable scalable data attribution for non-convex models (Park et al. 2023). However, the impact of spurions features on the data attribution scores generated by such methods remains an open question.
2017
Tanno et al. 2022
In our work, we demonstrate (Proposition Section 4.8) that in the presence of data bias, methods like Trak (Park et al. 2023) can undervalue the attribution scores for training samples with spurious features (Ti- 2019 This misattribution can hinder the identification of detrimental samples, especially for methods that rely solely on the magnitude of attribution scores (Marion
wari & Shenoy| 2023 Geirhos et al. 2020 McCoy
et al.
2023
Tanno et al. 2022).
Motivated by these observations, we propose a two-stage strategy to mitigate the impact of spurious features (a) In the first stage, we focus on identifying such features within the dataset using available meta-data or annotations generated by a vision language model. (b) Tn the second stage, we use multimodal embeddings, such as CLIP (Radford et al.) 2021 to learn a metric (Lim & Lanckriet 2014; Bhalla al., 2024) that identifies training examples that are semantically similar to the spurious features identified in the first step and whose removal can improve the model's performance as per the attribution scores.
The spurions features in the first stage are identified using metadata wherever available. In cases where metadala is unavailable, we utilize a vision-language model (VLM) to annotate a small validation set with its respective attributes and their associated values that are likely to introduce simplicity biases (Chen et al. 2024 Lu & Zhong 2024 Tan et al. 2024). By evaluating the model's performance on these attribute-value pairs and comparing it to the overall performance on the validation dataset (Johnson et al., 2023), we identify potential spurious features and generate corresponding textual description of these biases (Eyuboglu et al., 2022 This textual representation enables targeted data pruning and helps to mitigate the impact of spurious features without relying on manual group annotations in the training dataset.
In summary, our contributions in this paper are as follows:
We propose novel data-centric approach that combines NTK-based data attribution methods with textual descriptions of underlying bias to mitigate the impact of spurious features in training datasets.
We first theoretically demonstrate that baKeb attribution scores can be influenced by spurious features, which may limit the effectiveness of methods that rely solely these scores for data
2

---

Published in Transactions on Machine Learning Research (02/2026)
pruning. To overcome this limitation, we introduce metric learning- based data deletion strategy that selectively removes training samples aligned with textual descriptions of spurious feat and with high detrimental attribution score.
Our approach achieves up to 4% gain in average accuracy, 18% in worst-group accuracy, and a 50% improvement in class-level performance across various datasets. Additionally, it outperforms NTK-based methods like Trak on average by 10.6% in worst-group accuracy for different biased datasets.
2
Related Work
2.1
Data Attribution
Data attribution methods provide framework to relate model's predictions to its training dataset and
have been used in wide range of tasks, including model debugging and repair (Yeh et al.
2018
Tang et al.
2021 Shah et al. 2023 Grosse et al. 2023), subset selection (Engstrom et al. 2024 Xia et al.
2024 Chhabra 2023a
et al. 2024), group robustness (Jain et al. 2024), and removing poisoning attacks [Wu et al.
The idea of linking a model's predictions to its training data has been studied for decades under various names, including influence functions (Hampel 1974), regression analysis (Pregibon) 1981), and jackknife methods (Miller 1974). However, most of these early works focused on linear modelss and aimed to predict changes in the oplimal parameters when individual or groups of samples were excluded during the learning process. Recent works have tried to extend influence function and tacknife-bassed attribution methods to non-lincar models and bigger datasets (Koh & Liang 2017 Rad & Malcki] 2018 Giordano et al. 2019). However, despile their promising predictive capabilities, these methods often make strong assumptions of convexity and the existence of a unique global solution, which are not applicable for neural networks (Bae eet al. 2022). Furthermore, [asu et al. (2020): Hammoudeh & Lowd (2022) have demonstrated the fragile nature of methods like inffuence functions across different architectures, showing that they sometimes fail basic sanity checks. Various approaches have been proposed to address the limitations of influence functions, including gradient agreement scoring (Pruthi et al.
2020), training models to predict attribution scores, as in DataModels (Πyas et al., 2023 Engstrom el al. 2024). and methods like Trak (Park et al., 2023), which leverage concepts from the Neural Tangent Kernel (NTK) for data attribution. Unlike other approaches, such as DataModels, Trak does not require training thousands of models (Park al. 2023 Tlyas et al., 2023) or tracking the loss changes over the entire training process, making it more ellicient. However, the impact of spurious features within the dataset on the data attribution method like Trak remains largely unexplored.
2.2 Spurious Features and Simplicity Bias
Spurious features often arise from selection bias the dataset (Ye al. 2024), where, in the presence of multiple hypotheses for prediction, the model tends to rely on the simplest feature (Pezeshki al. 2021 Geirhos et al. 2020: Tiwari & Shenoy 2023). This preference can lead to suboptimal model performance, as it often ignores more robust and meaningful features that are essential for generalization in real-world
scenarios. Various methods have been proposed to address spurious features in models. These include data
augmentation techniques (Srivastava et al. 2020 Puli et al. 2022 Yao et al.) 2022 Zeng et al.) 2020 Wu
ett al. 2023b Nam et al. 2022 Zarlenga et al.
2024), and learning strategies that change the training
objectives to make the model robust to spurious features (Sagawa et al. 2020 Wang al. 2021 Levy et al.,
2020 Liu et al. 2022 2021
Kirichenko al. 2023 LaBonte al. 2023
Han & Zou 2024). However, many
of these changes
restricted under the regulatory policy for safety-critical applications Petersen et al,
2022 Matheny et al.
2019
Shen et al. 2023 Campos Zabala
2023) especially considering privacy concerns
requirements (Valentin 2024 Liu et al. 2024). Recent works have explored data deletion as a stralegy for mitigaling spurious features Chaudhuri
associated with collecting datascts and model cortification-basee
et al. 2023 Idrissi et al., 2022; Jain et al.) 2024). These methods use group annotation of the dataset to remove random samples from majority groups (Chaudhuri et al.] 2023 Idrissi et al.] (022) or those with
high detrimental attribution scores (Jain et al. 2024). However, these methods often require manual group
annotation of training (Chaudhuri al. 2023 Idrissi et. al. 2022 validation data. (Jain et. al. 2024
3

---

Published in Transactions on Machine Learning Research (02/2026)
VLMs / Meta Data
(Attribute, Value)
Description
Text Embedding
Validation
Inference
Analysis
Spurious Feeture
Text Enbedding
Spurious Feature Idantification
Image Embeddding
Data Attribution
Text Embeddlng
Training
Image Embedding
Data Attribution
Scores (k)
Checkpoints
Data
Attribution Embedding
Metric Learning
Subset Selection
Figure 1: The figure illustrates the key steps in identifying detrimental samples. First, the performance of the model across different attribute value pairs analyzed to identify and textually describe the underlying bias. Then, training samples that align with this bias and exhibit high detrimental attribution scores are selected for removal.
which is costly and time-consuming. Further, in real-world settings, where biases are identified post hoc after deployment and evolve over time (Lesort 2023). , generating such annotations is often impractical, and enforcing a balance among different groups may result in excessive data removal from the majority group and can harm generalization (Idrissi et al. 2022). Our method circumvents these limitations by using text-guided data attribution efficiently remove harmful samples within a delction budget, without relying on group labels hurting model performance. Further details limitations and capabilities of existing methods are
discussed in Appendix J
3 Proposed Method
3.1 Problem Definition
Consider classification setting with training dataset Dtrain { 2n), where each sample 2; (xi, w)), consists of input (x;) and associated class label (и)) and an unbiased validation dataset, Dval (v1...., ]n with validation samples 2, (I,, 8j)). The training dataset (Dtrain) is used to train neural network with optimal parameters 0* (Drrain). Additionally, we assume that [Deall << |Derainl-
Suppose for every training sample = there exists t underlying hidden discrete attributes, = (
and for each attribute (a') there are o possible values denoted as (aa)) E b/ 10....6) 6%]. In real-world settings, neural networks (0*) trained on Dirain often associate class labels (y) with specific attribute-value pairs (a" byr) (Eyuboglu et al. 2022 Tiwari Shenoy 2023 Geirhos et al. 2020 For example, a model However, feature
trained to predict gender might associate with the Teature "beard" ppresent/absente imbalance in the datasets can lead to misleading associations. If most of the male images in dataset include smiles, the model might spuriously link "male" with 'smiling" rather than "beard." This cause misclassification, like predicting smiling females males. We term such misleading attribute-valle pairs spurious features. In these scenarios, the primary objective of our work is to identify set of detrimental examples, ddeter C Dtrain, with features similar to the spurious feal ures and whose removal does not degrade the model's performance. Training the model on the filtered subset (Purai Steter) improves its performance compared to the original datasot, and reduces the influence the spurious feature the training datasot, similar to prior work like Chaudhuri et al. (2023). Further details on training and validation dataset are provided in Apeendix L.6
4

---

Published in Transactions on Machine Learning Research (02/2026)
Our method for identifying Stleter involves two steps: (1) Annotate attribute-value pairs in the validation set to detect potential spurious features and generate a textual description of the bias; (2) Select gdette C Drrain as samples semantically aligned with the bias and whose removal as per the data attribution scores does not degrade model performance.
3.2
Attribute Annotation and Spurious Feature Identification
A key component to identify spurious features is the availability of attribute- annotations for the validation dataset. However, in many practical scenarios, such annotations are often missing from the metadata. Chen et al. (2024) has shown that in the absence of such information, large language and vision models can be used to generate annotations necessary to identify the underlying spurious features. Hence, for datasets without pre-annotated attributes, we annotate the validation set with potential attribute-value pairs to assist in identifying spurious features.
To generate candidate attribute- value pairs, we leverage LLM-based tool such as ChatGPT (Chen et al, 2024). ChatGPT is provided with simple task description and prompted to suggest relevant attributes and associated values. For example, in classification task of distinguishing between land birds and water birds, ChatGPT can generate attributes such as 'habitat type' or "background environment," with values like 'forest' for land birds and "lake" for water birds. We adopt the task-specific prompts proposed by Chen ct al. 2024) to guide this process.
Once the attribute-valle value pairs are generated, the next step is to annotate the validation dataset. While ChatGPT excels at generating textual descriptions, it is not suited for image annotation (Chen et al.) 2024 To address this, we use Llama 3.2 (Dubey et al. 2024), vision-language model, to annotate small set
of validation images with the corresponding attribule value pairs (Chen et al. 2024h. Further details about
the prompts can be found in Appendix
3.2.1 Spurious Feature Identification
To identify spurious features, we take motivation from recent work that tries to identify systematic bias in model (Johnson al. 2023 Eyuboglu et al. 2022) based on accuracy and errors on the unbiasod validation dataset. However, unlike previous methods, which try to identify underperforming subgroups that may require collecting additional data, try to determine the overperforming attribute-value pair as possible candidate for data deletion (Chiu et al. 2023 Chaudhuri et al.) 2023). For this analysis, we take inspiration from Johnson et al. (023) and compare the performance of the dataset corresponding to each silribule-valuee pair with the performance of the overall dataset. In general, robust and informative features tend to improve both the subgroup accuracy (associated with a given attribute value pair) and the average accuracy across the dataset. Conversely, large performance gap between these two can indicate that improvements for a particular subgroup come at the expense of others (Johnson al.) 2023 Eyuboglu et al. 2022). In our framework, if this performance gap exceeds a predefined threshold, the corresponding attribute value pair is flagged as potential spurious feature learned by the model. Formally, this is expressed as:
1
Σ 1h(x)) =
Σ (n.p)e
1((xx) y) >
(1)
Da
Deni
(F.D)EDo
where, D, is subset of validation data Dmat associated with a" attribute and it's 5th value bj. The indicator function 1 indicates the correct prediction made by the model. The function h(x) represents the prediction made by the model for given input x and is the corresponding true class label. The parameter denotes the minimum threshold.
Once an attribute-value pair exceeds the threshold, a textual description is generated to describe the spurious feature. For example: 'Images with {a} as (b3.' Here, (a, is the attribute value pair selected as per Equation I Details about the textual description and hyperparameters are provided in Appendix L.5 and
A Appendix 2 respectively.
5

---

Published in Transactions on Machine Learning Research (02/2026)
3.3 Coherent Data Attribution
After generating the desired lext, the next lask is to select subset of data that is semantically coherent with the given text and whose removal does not degrade the performance of the model (Huang et al. 2024).
Since our task involves efficient subset selection, we formally define data attribution as follows:
Definition (Data Attribution and Leave-one-out Influence Score (Park et al., 2023(). Given training dataset Droin and a model's utility function (v: that measures the performance of the model, the data aftribution score a "D Drrain x Dval + R is defined as the change in the model's 's prediction for validation
sample V with respect to the optimal parameters when the training erample 24 is excluded from the training dalasel during the learwing of the optimal parometers 0* Formalll,
ad a12; 2x) = f 14; 02;;0 (Derada)) ((1:
(Derata'ee \2x))
(2
For a classification task, the utility function f(; for a sample = (z, v). Park et al. 2023 is defined as:
f(z: log p(z;0)
p(z:0)
(3)
where 0) represents the probability assigned to the correct class by the softmax function of a neural network parameterized by 0. A high f(z; 0) corresponds to a high likelihood for a given sample (z).
The NTK-based methods like Trak have closed-form formulation for data attribution score (a) (Definition expressed as:
a(v,, 21) N
1
(4)
N
where, for N different checkpoints of model (0"), {pr represents the probability assigned by th set of parameters to the correct class (yi), for sample 24 = (r,, v1). The terms (n(2)) and (x(2)) denote the projected gradients of the validation sample v and the training sample 2; with respect to the th set of optimal parameters, and for the utility function (f(:; entire training dataset. Further details about Trak can be found in Appendix B
Additionally, 4 is the projected gradient for the
To quantify the impact of removing a data sample from the training dataset on the performance of the entire validation dataset, we define the metric A(z) as detrimental attribution score associated with the validation dataset. for sample This metric measures the change in the model's performance (f) for the entire validation dataset when z is excluded from the training dataset.
A(z;) =- Σ a(13:2)
eval
Σ END
[foy::0' (Derain \2i)) f(y;; 0" ("(rainn))
(5)
where 2; E Durain Unlike the data attribution score defined in Definition A(2x) is the negative of the general definition and evaluates the contribution of each training sample to the likelihood of the entire validation datasct. A higher value of A(z,) indicates that removing the training sample 2; and retraining the model with the updated dataset leads to an optimal parameter 0" that improves the likelihood of the validation dataset (Equation 3). In other words, training examples that degrade overall validation performance are assigned higher A(2;) values. Once A(2;) is calculated, it is normalized and used for further steps.
While removing samples with high A(z) values can improve the model's performance; however, its impact on the downstrcam model is often tied up with its capability to remove samples with spurious features. During training, spurious features present in the dataset can result in gradient starvation (Tachet et al., 2018 Pezeshki et al. 2021). phenomenon that hamper the learning of predictive features. Under such

---

Published in Transactions on Machine Learning Research (02/2026)
scenarios, theoretically show that the detrimental attribution score(A) for a data sample containing spurious feature (f1) can be lower than that of data sample with predictive features (f2). even when both features are equally represented. Consequently, deletion strategies based solely on high attribution scores may inadvertently remove examples with predictive rather than spurious features (Proposition I and can fail to capture the impact removing data associated with spurious features on the overall generalization. Proposition 1 (lllustration for the Under Valuation of Attribution Scores). Consider neural network in the newral langent kernel (NTK) regime, trained using binary cross-entropy loss and contain two important features fi and f2. Suppose that due to learning dynamics, fi becomes dominant and causes gradient star- vation of fz as per Peeesshi et al. (2021). Let 2; and 2j be two training samples for which fi and f2 are the most informative features, respectively, and assume that the they contain equal representations of these features (equivalently, uj u² as per Definition sp. Under these conditions, the detrimental aftribution score for 2; con be systematically undervalued relative to Zz. Formally:
[A(z)) < A(2)))
The proof of Proposition 1 along with further details on gradient starvation, is provided in Appendix L Empirical evidence supporting this phenomenon is presented in the Eaperiment section (Section 4.8 Ap-
pendēr
This limitation of attribution scores motivates the need for a largeted removal strategy that specifically identifies and eliminates training samples sharing similar spurious features and exhibiting high A(2) scores. In many practical scenarios, the information about spurious features is missing in the data. Although annotating the entire training dataset using VLM- models is possible, this approach is often excessively time-consuming and practically infeasible, particularly for large-scale datasets (Lu & Zhong 2024). To address this, we adopt zero-shot approach (Pan et al., 2024 and leverage textual descriptions of bias and CLIP embeddings to select data samples that are semantically similar to the ident-ifidd textual descriptions. The use of CLIP embeddings further allows to capture fine-grained association between the textual description of bias and the input dataset, which is often hard to capture by group annotation or describe textually (Eyuboglu et al.) (2022). Specifically, we convert the textual description (Section of the potential spurious feature into an embedding Croxt. Similarly, we convert all images in the training dataset into their corresponding CLIP embeddings Cimago for i E 1...., |Derall Each training sample 2; then assigned a score k, reflecting its semantic similarity to the identified bias as per the given equation:
k. exp (- (Crexs - Cimagr) M (Crext Cimage)
2
(6) The text and image features, denoted as Crext, Cimage E RIID are represented as row vectors in a D- dimensional space. The matrix M is a positive semi-definite matrix, constructed as the outer product of a low-rank matrix L (rank at most t), and can serve as a learnable transformation. Since M defines the distance metric, varying the values of L allows us to generate different similarity mcasures for comparing data points (Lim & Lanckriet, 2014 Bhalla et al. 2024
where, M LL LE
We aim to remove data samples that are detrimental to the model's generalization (i.e., those with high A scores) and are semantically aligned with the identified bias. To accomplish this, we learn the matrix L (d'Fon et al. 2022; Lim & Lanckriet] 2014: Bhalla et al., (224) by maximizing the weighted A score for each sample, where higher weights correspond to stronger semantic alignment with the bias, as defined in Equation|
The optimization objective prioritizes learning metric I that assigns higher k; values to samples that both (i) exhibit high detrimental scores and (ii) align with the semantic description of the bias. The trade-off between these two objectives is governed by the hyperparameter T, which constrains the cumulative score across the dataset to exceed a threshold defined as a fraction (3) of the total training size ([Derain)). A larger T emphasizes semantic alignment, defined in Equation 0 whereas smaller T allows greater flexibility in sclecting samples based primarily on their A scores. The complete optimization objective is described below:
Peruial
[Pesii Σ
ki
max Σ
A(zi)
s.l.
k K2T,
T= x [Deraul.-
(7)
E,
7

---

Published in Transactions on Machine Learning Research (02/2026)
To ensure that the optimization remains tractable, we replace the hard constraint with soft penalty term od Ad'oon et al. 2022 in the objective function. Further detail on this is provided in Appendix D
Once the optimization is complete, a subset of training data with K; scores greater than the hyperparametter 7 is selected for removal (Sdeter). The model is then retrained with the updated training dataset (Dirain Stleter) where, Sdeter = {z: E Draai h; > 7}. sensitivity analysis of all the hyperparameters and subset size is provided in Appendix O and Appendix P respectively.
4 Experiments
4.1 Setting
We evaluate the performance of our method across various datasets and compare it with existing data altribution techniques, including original training of the model with complete dataset (original), Random deletion of data points (Random), Influence Function (IF) (Koh & Liang 2017). TracIN (Pruthi et al. 2020),
EWC Repair (Tanno et al.
2022).
and Trak (Park et al., 2023 The datasets used in our experiments include
WaterBirds (Sagawa* el al. 2020 Animal with altribules (AWA2) (Xian et al.
2018). German Tralfic Sign
Recognition Benchmark (GTSRB)
(Stallkamp et al. 2012), CELEBA (Liu et al.
2015 Eyuboglu et al.) 2022:
Zhang et al.) 2022 (Appendix D. CIFAR-10 (Krizhevsky et al. 2009), and Image 100
(Russakovsky et al.
(Sagawa et al. 2020 JTT (Liu et al. 2021h) and group balancing methods are provided in Section 4.9 For datasets such as GTSRB, CIFAR- 10, and WaterBirds, we utilized attributes generated by ChatGPT and VLM models. To further assess the impact of metadata availability, we created two variants for the AWA2 datasets. The first variant, AWA2-A, includes class-specific annotations provided by the original datasets. The second variant, AWA2-B, uses attributes generated using ChatGPT and VLM-based annotation techniques (Section 3.2p. All primary experiments were conducted using a ResNet-18 model, which is the base architecture used in NTK-- based data attribution methods such as Trak (Park et al.) 2023) for the image classification task. Additional experiments using alternative architectures and vision transformer models are presented in Appendix D and Appendix M respectively. We have reported the worst group accuracy and avcrage accuracy based on prior work on spurious features (Nam al.] 2022 Sagawa al. 2020: Chaudhuri et al. 2023). However, due to the absence of well-defined group structures in many real-world datasets (Sagawa et al. 2020 we have compared these datasets on average accuracy and class-level accuracy. All the experiments were conducted on two NVIDIA A6000 GPUs. Further details on training, hyperparameterss and subset size are provided (Appendix) illustrates the overall workflow of our approach. We also report
2015 Tian et al.]
2020 Further comparisons with robustness-baat
methodssgroupDDe
in Appendix [ Algorithm [
time and memory overheads associated with subset selection in Appendix and Appendix U respectively. Sample images from the selected subset deeter are shown in Appendix W
4.2 Improvement in Average Accuracy
Table reports the improvement in average accuracy achieved by our method compared to existing baselines. On average, our method outperforms Trak by 1.4%, EWC by 1.6%, TracIN by 1.4%, Influence Functions by 2.0%, and the original full-dataset training baseline by 1.7%. Notably, we observe gains of 1.9%, 2.5%, and 2.4% over Trak on AWA2-B, WaterBirds, and AWA2-A, respectively. The performance improvement highlights the efficiency of our method in removing the detrimental samples associated with spurious features. We further saw a substantial improvement in under represented class as discussed in Section 4.3 Additional experiments on worst-group accuracy and architectural ablations for WaterBirds are provided in Appendix L
4.3 Class Level Improvement after Data Deletion
Table 2 presents class-level accuracy for datasets with more than two classes. As per the results, our method improves the accuracy of a significant number of classes across datascts. For example, in Awa2-A, Awa2-B, ImageNet-100, and CIFAR-10, over 40% of the classes show improvement, with some achieving gains high 29.16%. Notably, in GTSRB, 22 out of 43 classes benefit, with a maximum per-class improvement. of
8

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Comparative evaluation of average accuracy of our proposed method (Ours) against baseline ap- proaches across multiple datascts. The results report mean accuracy scores over three independent runs, with the best-performing values highlighted in bold. Entries with gain of more than 1.5% over full-data training are highlighted in orange, while those exceeding 3% are shown in blue.
Dataset
Original
Random
IF
TracIN
EWC
Trak
Ours
WiterBirds
0.638
0.606
0.603
0.652
0.650
0.656
0.681
AWA2-A
0.644
0.622
0.644
0.652
0.642
0.638
0.662
CELEBA
0.895
0.893
0.890
0.893
0.890
0.898
0.906
GTSRB
0.969
0.966
0.973
0.971
0.975
0.971
0.980
AWA2-B
0.644
0.622
0.644
0.652
0.642
0.638
0.657
CIFAR-10
0.774
0.787
0.798
0.784
0.789
0.793
0.801
ImageNet-100
0.440
0.436
0.429
0.423
0.423
0.435
0.438
Table 2: Class-level accuracy improvement(Imp) after data removal across datasets. The table shows the maximum improvement in any class, the number of improved classes, and the mean improvement across
them.
Dataset
Max Imp
# Imp Classes
Mean Imp
Awa2-A
16. 16.27%
/ 10
11.12%
Awa2-B
29.16%
/ 10
17.98%
CIFAR-10
10.39%
7
10
5.59%
GTSRB
50.00%
22
43
5.69%
ImageNet-100
36.00 %
51
100
10.15%
50%. The improvement in average accuracy highlights that the improvement in underperforming classes is attained without substantially degrading the performance of other classes.
Table reports gains the worst-performing class for each dataset. In Awa2-A and Awa2-B, worst-class accuracy more than doubles, while in GTSRB, it improves by 20%. These results demonstrate that our method enhances class-level performance with minimal negative impact other classes.
Worst-class accuracy before and after retraining. The table shows the original worst-class accuracy and the corresponding value after retraining with the new dataset.
Table 3:
Dataset
Original Worst-Class Accuracy Retraiued Worst-Class Accuracy
Awa2-A
0.030
0.108
Awa2-13
0.030
0.103
CIFAR-10
0.589
0.575
GTSRB
0.500
0.700
ImogeeNet-100
0.100
4.4 Performance across Different Spurious Attributes
To further investigate the impact of spurious features on both worst-group and average performance, we follow the setup of Eyuboglu et al. (2022) and select subset of the CELEBA dataset where the target attribute is strongly correlated with spurious feature. We compare the average and worst-group performance achieved Additionally, considering the benefit of
by our method against other baselines in Table 4 and Table 5
random data deletion in biased datasct (Chaudhuri et al. 2023 we introduce a new bascline, Maj.-Rand, where the subset of data is randomly deleted from the majority group. As shown in the results, our method outperforms other baselines average accuracy in 7 and worst-group accuracy 8 out of 10 settings, respectively. Notably, we observe gain of over 4% in average accuracy for the target attribute attractive, compared training on the original dataset. Similarly, worst-group accuracy improves by over 15% for attractive, receding hairline, and arched eyebrows, and by more than 5% for big nose, goatee, and male.

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Comparison of best average accuracy across different data attribution methods for different spurious attributes. The table reports the mean accuracy across three independent runs. Entries with gain of more than 1.5% over full-data training are highlighted in orange, while those exceeding 3% are shown in blue.
Target
Spurious Attribute
Original
Maj.Rand
Random
IF
EWC
TracIN
Trak
Ours
arched eyebrows
receding hairline
0.713
0.740
0.739
0.716
0.724
0.730
0.722
0.736
attractive
mouth slightly open
0.628
0.627
0.668
0.640
0.633
0.631
0.658
0.673
big nose
male
0.771
0.770
0.770
0.764
0.751
0.745
0.756
0.780
goatee
bushy eyehrows
0.946
0.931
0.947
0.938
0.951
0.953
0.949
0.953
mouth slightly open
smiling
0.869
0.871
0.877
0.877
0.860
0.876
0.867
0.877
mouth slightly open
wearing lipstick eyeglasses mouth slightly open
0.820
0,804
0,801
0,828
0.834
0,816
0,801
0.839
narrow eyes
0.840
0.858
0.862
0,856
0.858
0.860
0.855
0.862
pointy nose
0.690
0.714
0.676
0,689
0.695
0.709
0.694
0.698
receding hairline male
rosy checks
0.921
0.909
0,920
0,921
0.920
0.916
0,911
0.930
pointy nose
0.919
0.931
0.907
0.909
0.911
0.906
0.915
0.921
Table 5: Comparison of best worst-group accuracy across different data attribution methods for different spurious attributes. The table reports the mean accuracy across three independent runs. Entries with gain
of more than 5% over full-data training are highlighted in green, while those exceeding 15% are shown in violet.
Target
Spurious Attribute
Original
Maj.-aan
Random
EWC
TracIN
Trak
Ours
arched eyebrows attractive
receding hairline
0.187
0.314
0.113
0.247
0.262
0.196
0.099
0.354
mouth slightly open
0.213
0.242
0.347
0,266
0.241
0,205
0.392
0.407
big nose
male
0.131
0.076
0.096
0.143
0.092
0.113
0.172
0.221
gostee
bushy eychrows smiling
0.432
0.493
0.287
0.437
0.439
0.387
0.277
0.548
mouth slightly open
0.324
0.415
0.552
0.418
0.441
0.487
0.433
0.489
mouth slightly open
wearing lipstick eyeglasses mouth slightly open
0.555
0.471
0.557
0.598
0.594
0.549
0.486
0.612
narrow eyes
0.208
0.052
0.119
0.000
0.092
0.128
0.024
0.151
pointy nose
0.045
0.044
0.046
0.034
0.028
0.021
0.040
0.084
receding hairline
rosy cheeks
0.121
0.228
0.131
0.179
0.241
0.254
0.201
0.296
male
pointy nose
0.840
0.882
0.824
0.833
0.861
0,870
0.875
0.903
Table 6: Comparison of best average and best worst-group accuracy between meladataaddrven and VLM- guided textual description.
Target
Spurious Attribute
Meta Data
VLM
Avg. Acc.
WG Acc.
Avg. Acc.
WG Acc.
bangs big nose
black hair wearing necklace
0.922
0.649
0.916
0.624
0.787
0.347
0.776
0.236
heavy makeup wearing carrings
straight hair
0.826
0.716
0.835
0.716
bags under eyes
0.798
0.281
0.791
0.214
4.5 Ablation between Meta Data and VLM-based Description
Table G compares the performance of our method when using metadata- baaedd versus VLM-generated textual descriptions of the spurious features. While both strategies yield comparable results in terms of average ac- curacy, the metadata-driven variant consistently achieves higher worst-group accuracy. This indicates that more precise annotations of underlying biases can facilitate the targeted removal of detrimental samples. Nevertheless, even in the absence of such annotations, VLM-basod descriptions deliver comparable perfor- mance across both average and worst-group accuracies. These results suggest that, despite the lack of explicit metadata, V..M-generated descriptions do not degrade model performance introduce additional bias.
10

---

Published in Transactions on Machine Learning Research (02/2026)
Table 7: Comparison of best average and best grosp accuracy between our method and D3M across
different spurious attributes.
Target Spurious Attribute
Ours
D3M
Average Accuracy
Worst Group Arcuracy
Average Accuracy
Worst Group Accuracy
hangs
black hair
0.922
0.649
0.920
0.627
big mose
wearing neckluee
0.787
0.347
0.747
0.173
heavy makeup
straight hair
0.826
0.716
0.821
0.651
wearing cazrings
bags under eyes
0.798
0.281
0.787
0,008
4.6 Comparison with Group Annotation-based Subset Selection
Table presents a comparative evaluation between our method, which relies on the textual description of bias, against technique that can use group annotation of spurious features in the validation dataset. To compare with such a method, we define group structure based on different. values of Spurious Attribute and Target, and then use the method proposed by Jain et al. (2024) (D3M) for subset selection. As per the result, on average, our method consistently outperforms D3M across both the best average and worst-group accuracy with a gain of 1.5% in best average accuracy and 11.8% in best worst group accuracy without using the explicit group annotation. This highlights the efficioncy of the soft comparison scheme of clip features in handling partially visible features and the proposed optimization scheme compared to hard thresholding used group annotation.
4.7 Ablation of Different Components
The ablation study in Table S highlights the contribution of key components i.e, data Attribution and CLIP, to the overall performance of our method. For the given experiment, we have used cosine similarity with CLIP (Only CLIP) representation to remove samples that align with the description of the underlying bias. When used independently, both components provide noticeable improvements over the full training baseline, particularly in average accuracy. However, they exhibit limitations in worst-group accuracy when applied in isolation. Notably, combining both Attribution and CLIP in method yields the highest performance across nearly all settings, especially in worst-group accuracy, demonstrating the complementary strengths of these components in addressing spurious correlations.
Table 8: Comparative evaluation of the proposed method (Ours) with the full training baseline (Original), Only Attribution, and Only CLIP, reporting the best average and best worst group accuracy (mean) across
three runs.
Target Attribute
Spurious Attribute
Average Accuracy
Worst Group Accuracy
Original
Only Astribation
Only
Original
Only Astribation
Only CLIP
Ours
Bangs
Black Hair
0.920
0.921
0.922
0.923
0.523
0.571
0.548
0.649
Big Nose
Weating Necilace
0.765
0.787
0,777
0.787
0.127
0,080
0.110
0.347
Howy Makeu
Straight Hair
0.805
0.800
0.813
0.826
0.651
0.886
0.739
0.718
Wearing, Farrings
Rups Under Fyes
0.791
0.792
0.791
0.798
0.040
0.017
0.008
0.281
Table 9: Mean and standard deviation of detrimental attribution (I-A)) scores for different attributes, along with statistical significance from two-sample t-test against *Smillng*.
Attribute Mean
Std
p-value (vs Smiling) Significance
Smiling (spurious)
0.539
0.056
Moustache
0.545
0.044
0.1008
Not significant
Beard
0.544
0.036
0.00039
Significant
11

---

Published in Transactions on Machine Learning Research (02/2020)
4.8 Empirical Validation of Theoretical Formulation
To validate our theoretical claim, we used the codebase provided by Eyuboglu et al. (022] to sample a 10k subset from CELEBA, where the attributes Male and Smiling are highly correlated. We then computed Trak scores for the training dataset using a ResNet-18 classifier trained to predict. the Male label. In this setting, due to the strong correlation between Male and Smiling (Eyuboglu et al.) 2022 Chen et al.) 2023) smillig may act features like Beard and Moustache to be more causally relevant, and thus expect that samples with these features to have lower A scores compared to those with Smiling.
spurious feature. Since the task is to distinguish males from females, consider
However, statistical analysis of the detrimental attribution(A) scores using T-test for the training samples reveals that Smiling has lower scores for samples compared to samples with Beard and Moustache (Table 9 The difference is statistically significant for Beard (p < 0.001). This supports Proposition I demonstrating that such effects can arise in practical scenarios.
4.9 Comparison with Robustness based Methods
Table 10 compares the average and worst-group accuracy of our method against various robustness-baset approaches on the Waterbirds dataset with ImageNet-based initialization(scc Appendix L.3 Methods are grouped based on whether they require group annotations for the entire training dataset and whether they support textual descriptions of bias. As per the results, our method achieves a competitive average accuracy (0.855) and strong worst-group accuracy (0.756) without relying on group annotations for the training dataset, while uniquely supporting textual description of bias. Compared to other methods, our method improves worst-group accuracy by 27.9% over ERM, 12% over JTT, and 1.6% over D3M. Group annotation- based methods like gDRO and RWG perform best worst-group accuracy, but at the cost of requiring explicit group labels for the entire training dataset. Although our method uses annotations for the validation dataset, either available through metadata or generated with the help of LLM or VLM; the identification of detrimental training samples relies only on the textual description of bias (inferred from the annotation), not on raw per-sample labels. This design can enable practical human-in--hhe-loog scenarios, where a domain expert can directly provide a textual description of the bias without relying on the annotated dataset. Additional challenges associated with caaustesss-bssd baselines in specific applications are discussed in Scction I Section 2.2 and Appendix J
Table 10: Comparison of Average Accuracy and Worst group accuracy achieved by our method in comparison with other robustness-bsed hods on Waterbirds.
Method Group Annotation (Train) Supports Textual Blas Description
Average Accuracy
Worst Group Accuracy
BRM
0.819
0.477
D&M
0.903
0.740
JTT
0,852
0.636
Oune
0.758
RWG
0.864
0.622
SUDG
0.833
0.814
gDRO
5 Conclusion
In this work, propose a data deletion framework to mitigate the impact of spurious biases in the training dataset and enhance model performance. Our method employs metric learning techniques to target and remove training samples that are semantically aligned with the textual description of identified biases and whose removal, based on altribution scores, does not adversely affect model performance. To the best of our knowledge, this the first approach to use text-guided data attribution scores to mitigate simplicity bias in models. However, its effectiveness depends on the quality of the textual descriptions used to capture spurious biases, and the current framework is limited image datasets. In future work, we aim to incorporate a human-in-t -ntooo-oo--op framework to better mitigate complex biases and to extend it to NLP tasks.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Acknowledgment
This rescarch/project is supported (in part for setting up the GPU compute) by the Indian Institute of Science. Kumar Shubham is supported by the Kotak ISSe AI-ML Centre (KIAC) fellowship.
References
Rabab Abdelfattah, Qing Guo, Xiaoguang Li, Xiaofeng Wang, and Song Wang. Cdul: Clip-driven un- supervised learning for multi-label image classification. In Proceedings of the IEFE/CVF international conference on computer vision, pp. 1348 1357, 2023.
Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If influence functions are the answer, then what is the question? Advances in Newral Information Processing Systems, 35:17953 17967, 2022.
Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. arXiv preprint Xix:22000 14631, 2020.
Rodrigo F Berriel, Franco Schmidt Rossi, Alberto F de Souza, and Thiago Oliveira-Santos. Automatic large- scale data acquisition via crowdsoureing for crosswalk classification: A deep learning approach. Computers & gruphics, 68:32 42, 2017.
Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P Calmon, and Himabindu Lakkaraju. Interpreting clip with sparse linear concept embeddings (splice). arXiv preprint vrXiv22/02.1031 2024.
Nikita Bhatt, Nirav Bhatt, Purvi Prajapati, Vishal Sorathiya, Samah Alshathri, and Walid El-Shafai. A data-centric approach to improve performance of deep learning models. Scientific Reports, 14(1)223329, 2024.
Benedikt Boecking, Nicholas Roberts, Willie Neiswanger, Stefano Ermon, Frederic Sala, and Artur Dubrawski. Generative modeling helps weak supervision (and vice versa). arXiv preprint or Xiv:22200 12023, 2022.
Francisco Javier Campos Zabala. Responsible ai understanding the ethical and regulatory implications of ai. In Grow Your Business with AI: A First Principles Approach for Scaling Artificial Intelligence in the Enterprise, pp. 453 477. Springer, 2023.
Kamalika Chaudhuri, Kartik Ahuja, Martin Arjovsky, and David Lopez-Paz. Why does throwing away data improve worst-group error? In International Conference on Machine Learning, pp. 4144-4189 PMLR, 2023.
Muxi Chen, YU LI, and Qiang Xu. Hibug: On human-interpretablee model debug. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL httpps://penneviie..eeet// net/formn?id 4sDHLxXb1L
Muxi Chen, Yu Li, and Qiang Xu. Hibug: human-interppetable model debug, Advances in Newral Information Processing Systems, 36, 2024.
Anshuman Chhabra, Peizhao Li, Prasant Mohapatra, and Hongfu Liu. what data benefits my classifier?" enhancing model performance and interpretability through influence-based data selection. In The Twelfth International Conference on Learning Representations, 2024.
Ming-Chang Chiu, Pin- Yu Chen, and Xuezhe Ma. Better may not be fairer: A study on subgroup discrepancy in image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4956-4966, 2023.
Sang Keun Choc, Hwijoen Ahn, Juhan Bac, Kewen Zhao, Minsoo Kang, Youngsoog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, al. What your data worth gpt? Ilm-scale data valuation with influence functions. Xáv preprint arXio:2405.1396, 2024.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Gilad Cohen, Guillermo Sapiro, and Raja Giryes. Detecting adversarial samples using influence functions and nearest neighbors. In Proceedings of the TEEE/CV conference on computer vision and pollern recognition. pp. 14453 14462, 2020.
Gabriel d'Eon, Jonathan d'Eon, James R. Wright, and Kevin Leyton- Brown. The spotlight: general method for discovering systematic errors deep learning models. In Proceedings of the 2022 ACM Con- New York, NY, USA, June 2022.
ference on Fairness, Accountability, and Transporency, pp. ACM. doi: 10.1145/3531 146.3534641.
Alexcy Drutsa, Viktoriya Farafonova, Valentina Fedorova, Olga Megorskaya, Evfrosiniya Zerminova, and Olga Zhilinskaya. Practice of efficient data collection via crowdsourcing at large-scale. arXiv preprint 1rxiv 0444. 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:447.221883, 2024.
Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with data- models. arXiv preprint arXiv:2401.1299 2024.
Sabri Eyuboglu, Maya Varma, Khaled Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dunn- mon, James Zou, and Christopher Ré. Domino: Discovering systematic errors with cross-modal embed- dings. arXi preprint arX1v:203..49600, 2022.
Jiaxin Fan, Qi Yan, Mohan Li, Guanqun Qu, and Yang Xiao. A survey on data poisoning attacks and defenses. In 2022 7th IEEE International Conference on Data Science in Cyberspace (DSC), pp. 48-55. IEEE, 2022.
Emilio Ferrara. Fairness and bias in artificial intelligence: brief survey sources, impacts, and mitigation strategies. Sci, 6(1):3, 2023.
Robert
Geirhos, Jörn- Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence, 2(11):665 673, 2020.
Ryan Giordano, William Stephenson, Runjing Liu, Michael Jordan, and Tamara Broderick. A swiss army infinitesimal jackknife. In The 22nd International Conference on Arlificial Intelligence and Statistics, pp. 1139 1147. PMLR, 2019.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv: 032996 2023.
Oskar J Gstrein and Anne Beaulieu. How to protect privacy in a datafied society? a presentation of multiple legal and conceptual approaches. Philosophy 8 Technology, 35(1):3, 2022.
Zayd Hammoudeh and Daniel Lowd. Identifying a training-set attack's target using renormalized influence estimation. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communicaatinns Securily, pp. 1367-1381) 2022.
Frank R. Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, (0(346):383-334) 1974.
Yujin Han and Difan Zon. Improving group robustness on spurious correlation requires preciser group inference. arXiv preprint 2Xie:2404413815, 2024.
Gavin S Hartnett, Andrew Lohn, and Alexander P Sedlack. Adversarial examples for cost-sensitive clas- sifters. Xiv preprint ar arxii:19100 02095, 2019.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Chester Holtz, Tei Weng, and Gal Mishne. Learning sample reweighting for accuracy and adversarial robustness. arXiv preprint or Xin:2200 1 11513, 2022.
Long-Kai Huang, Peilin Zhao, Junzhou Huang, and Sinno Pan. Retaining beneficial information from detrimental data for neural network repair. Advances in Neural Information Processing Systems, 36, 2024.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive porst-group-accurage In Conference on Causal Learning and Reasoning, pp. 336 351. PMLR, 2022.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Datamodels: Predicting predictions from training data. International conference on machine learning. PMLR, 2023.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural notworks. In Advances in Neural Informalion Processing Systems, volume 31, 2018. URL https: pdf
//proceedings. neur httppp/ee.81111//aaa...sa/aaaaa.
Saachi Jain, Kimia Hamidieh, Kristian Georgiev, Andrew Ilyas, Marzyeh Ghassemi, and Aleksander Madry, Improving subgroup robustness via data selection. Advances in Neural Information Processing Systems, 17:94490-945111 2024.
Nari Johnson, Angel Alexander Cabrera, Gregory Plumb, and Ameet Talwalkar. Where does my model underperform? a human evaluation of slice discovery algorithms. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 11, pp. 65-76, 2023.
William B Johnson. Extensions of lipshitz mapping into hilbert space. In Conference modern analysis and probability, 1984, pp. 189 206, 1984.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. ICLR, 2023.
Pang Wei Koh and Percy Liang. Understanding black- predictions via influence functions. In International conference machine learning, pp. 1885-1894. PMLR, 2017.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Trena Gao, el al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning, pp. 5637- -564.. PMLR, 2021.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, et al. Captum: unified and generic model interpretability library for pytorch. arXiv preprint R2t::2009907899 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ram Shankar Siva Kumar, David R O'Brien, Kendra Albert, and Salome Vilojen. Law and adversarial machine learning. arXiv preprint arXiv:1810.10731 2018
Tyler LaBonte, Vidya Muthukumar, and Abhishek Kumar. Towards last-layer retraining for group robustness with fewer annotations. Advances in Neural Information Processing Systems, 18.115522-1579) 2023.
Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debiased representation via disentangled feature augmentation. Advances in Neural Information Processing Systems, 34:25123- 25133, 2021.
Timothée Lesort. Spurious features in continual learning. In AAAI Bridge Program Continual Causolity, pp. 59-62, PMLR, 2023.
Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Largo-scale methods for distributionally robust optimization. Advances in Neural Information Processing Systems, 33:8847-888 8860, 2020.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Kunyang Li, Jean-Chane Charles Noirot Ferrand, Ryan Sheatsley, Blaine Hoak, Yohan Beugin, Eric Pauley, and Patrick McDaniel. On the robustness tradeoff in fine-tuning. ar preprinl Nrv:22003. 14836, 2025.
Daryl Lim and Gert Lanckriet. Efficient learning of mahalanobis metrics for ranking. In International conference on machine learning, pp. 1980-1988. PMLR, 2014.
Sungjin Lim and Junhyoung Oh. Navigating privacy: A global comparative analysis of data protection laws. IET Information Security, 2025(1) (1556)))53663, 2025.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6788 6792. PMLR, 2021.
Sheng Liu, Xu Zhang, Nitesh Sckhar, Yue Wu, Prateck Singhal, and Carlos Femandez-Granda. Avoiding spurious correlations via logit correction. arXiv preprint arXiv:2212. 01433, 2022.
Ziquan Liu, Zhuo Zhi, Iija Bogunovic, Carsten Gerner-Beuens and Miguel Rodrigues. Prosac: Provably safe certification for machine learning models under adversarial attacks. ar preprint Niu:242202222) 2024.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
Angela M Lonzetta and Thaier Hayajnch. Challengos of complying with data protection and privacy regu- lations. EAI Endorsed Trans. Scolable Inf. Syst., 8(30):e4, 2021.
Haoming Lu and Feifei Zhong. Can vision-language models replace human annotators: case study with celeba dataset. arXiv preprint arXiv:2410.09416, 2024.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzich Fadace, and Sara Hooker. When less is more: Investigating data pruning for pretraining llms at scale. arXiv preprint (kki:23090045000 2023.
Michael Matheny, S Thadaney Israni, Mahnoor Ahmed, and Danielle Whicher. Artificial intelligence in health care: The hope, the hype, the promise, the peril. Washington, DC: National Academy of Medicine, 10, 2019.
RT McCoy, Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint 1rXi::190..0 01007, 2019.
Rupert G Miller, The jackknife-a review. Biometrika, 61(1):1 15, 1974.
Junhyun Nam, Jachyung Kim, Jacho Lee, and Jinwoo Shin. Spread spurious attribute: Improving worst- group accuracy with spurious attribute estimation. arXiv preprint orXiv:2204.000 020070, 2022.
Pavan Kalyan Reddy Necrudu, Subba Reddy Oota, Mounika Marreddy, Venkateswara Rao Kagita, and Man- ish Gupta. On robustness of finetuned transformer-base ulp models. ar preprint arXiv:2305.114 14453, 2023.
Fei Pan, Sangryul Jeon, Brian Wang, Frank Mckenna, and Stella X Yu. Zero-shot building attribute extrac- tion from largo-scale vision and language models. In Proceedings of the IEEE/CVF Winter Conference on Ayplications of Compuler Vision, pp. 8647-8633 2024.
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. Trak: At-
tributing model behavior at scale. arXiv preprint arXio:2308.14480 141866 2023.
Jay M Patel and Jay M Patel. Introduction to common crawl datasets. Getting structured data from the internet: running web crawlers/ arwwlers/scrupers scrapers big data production scale, pp. 277-324, 2020.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Eike Petersen, Yannik Potdevin, Esfandiar Mohammadi, Stephan Zidowitz, Sabrina Breyer, Dirk Nowotka, Sandra Henn, Ludwig Pechmann, Martin Leucker, Philipp Rostalski, et al. Responsible and regulatory conform machine learning for medicine: survey of challenges and solutions. IEEE Access, 10:58375 58418, 2022.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34:1256-1277 2021.
Alan Pham, Eunice Chan, Vikranth Srivatsa, Dhruba Ghosh, Yaoqing Yang, Yaodong Yu, Ruiqi Zhong, Joseph E Gonzalez, and Jacob Steinhardt. The effect of model size on worst-group generalization. arXiv preprinl Xrv:222113 12099122.000 100001,,, 2021.
Daryl Pregibon. Logistic regression diagnostics. The annals of slatistics, 9(4):705- 724, 1981.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920-19930, 2020.
Aahlad Puli, Nitish Joshi, Yoav Wald, He He, and Rajesh Ranganath. Nuisances via negativa: Adjusting for spurious correlations via data augmentation. ar preprint orXiv:2210.01302 2022.
Kamiar Rahnama Rad and Arian Maleki. scalable estimate the extra-sample prediction error via approximate leave-one-out. arXiv preprint arXiv:1801.1002 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748 8763. PMLR, 2021.
Olga Russakovsky and Li Fei-Fei. Attribute learning in large-scale datasets. In European conference on computer vision, pp. 14. Springer, 2010.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-25) 2015. doi: 00.1007/111233-015-01111
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ICLR, 2020.
Shiori Sagawa* Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL https://openrevvie.
net/forummidzzryGu..rt
Harshay Shah, Sung Min Park, Andrew Ilyas, and Aleksander Madry. Modeldiff: A framework for comparing learning algorithms. In International Conference on Machine Learning, pp. 30646 10166-0088... PMLR, 2023.
Xudong Shen, Hannah Brown, Jiashu Tao, Martin Strobel, Yao Tong, Akshay Narayan, Harold Soh, and Finale Doshi-Veles Towards regulatable ai systems: Technical gaps and policy opportunities. arXiv preprint ar XrX::22006.1200 12609, 2023.
Megha Srivastava, Tatsunori Hashimoto, and Percy Liang. Robustness to spurious correlations via human annotations. In International Conference on Machine Learning, pp.
PMLR, 2020.
J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, 32:323-300 332,, 2012. ISSN 0893-6080. doi: https:// 10wp:/...t..../....00101
CRL https::////t //ops://www.... http://wwwccccoooooooccccooooaar $08936080012004457 Selected Papers from IJCNN 2011.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Remi Tachet, Mohammad Pezeshki, Sacid Shabanian, Aaron Courville, and Yoshua Bengio. On the learning dynamics of deep neural networks. arXiv preprint or Xiv: 1 1et:1809..00 8809008888 2018. URL https://arxiv.org org/ abs/1809.06848
Zhen Tan; Dawei Ti, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation and synthesis: A survey. In Proceedings of the 2024 Conference on Empiricul Methods in Natural Language Processing, pp. 930-957, 2024.
Siyi Tang, Amirata Ghorbani, Rikiya Yamashita, Sameer Rehman, Jared A Dunnmon, James Zou, and Daniel I. Rubin. Data valuation for medical imaging using shapley value and application to largo-scale chest x-ray dataset. Scientific reports, 11(1):8366, 2021.
Ryutaro Tanno, Mclanic F Pradicr, Aditya Nori, and Yingzhen Li. Repairing neural networks by leaving the right past behind. Advonces in Neural Information Processing Systems, 35:13132 13145, 2022.
David Thiel. Identifying and eliminating csam in generative ml training data and models. Stanford Internet Observatory, Cyber Policy Center, December, 23:3, 2023.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776 794. Springer, 2020.
Rishabh Tiwari and Pradeep Shenoy. Overcoming simplicity bias in deep networks using feature sieve. In International Conference on Machine Learning, pp. 34330-34343. PMLR, 2023.
Romeo Valentin. Towards a framework for deep learning certification in safety-critical applications using inherently safe design and un-time error detection. arXiv preprint arXiv:2403. 14678, 2024.
Shiqi Wang, Kexin Pei, Justin Whitchouse, Junfeng Yang, and Suman Jana. Efficient formal safety analysis of neural networks. Advances in neural information processing systems, 31, 2018.
Tan Wang, Chang Zhou, Qianru Sun, and Hanwang Zhang. Causal attention for unbiased visual recognition. In Proceedings of the CEE//CVF International Conference on Computer Vision, pp. 3091-3100, 2021.
Steven Euijong Whang, Yuji Roh, Hwanjun Song, and Jae-Gil Lee. Data collect and quality challenges in deep learning: A data-centric perspective. The VLDB Journal, 32(4):791 813, 2023.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International conference on machine learning, pp. 5286-5295. PMLR, 2018.
Chenwang Wu, Defu Lian, Yong Ge, Zhihao Zhu, and Enhong Chen. Influence-driven data poisoning for robust recommender systems. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10): 11915-11989 2023a.
Shirley Wu, Mert Yuksekgonul, Linjun Zhang, and James Zou. Discover and cure: Concept-aware mitigation of spurious correlation. In International Conference on Machine Learning, pp.
PMLR, 2023b.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXio preprint or Xix::22402. 04333, 2024.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning a comprehen- sive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, (11)):2251222))) 2018.
Feng Xiong, Maoyue Xie, Lingjuan Zhao, Cheng Li, and Xuan Fan. Recognition and evaluation of data intangible assets. Sage Open, 12(2):21582440221094663 2022.
Xinyi Xu, Shuaiqi Wang, Chuan-Sheng Foo, Bryan Kian Hsiang Low, and Giulia Fanti. Data distribution valuation. Xiu preprint or arXiv:24100 10. 04386666666.. 2024a.
18

---

Published in Transactions on Machine Learning Research (02/2026)
Zhipeng Xu, Zhenghao Liu, Yukun Yan, Zhiyuan Liu, Ge Yu, and Chenyan Xiong. Cleaner pretraining corpus curation with neural web scraping. arXiv preprint arXiv:402. 14652, 2024b.
Greg Yang and Hadi Salman. A fine-grained spectral perspective on neural networks. arXiv preprint arXiv:1907. 10599, 2019.
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out- of-distribution robustness via selective augmentation. In International Conference on Machine Learning, pp. 25407-25437. PMLR, 2022.
Wenqian Ye, Guangtao Zheng, Xu Cao, Yunsheng Ma, and Aidong Zhang, Spurious correlations in machine learning: A survey. aix preprinl 1006:1442.12115,, 2024.
Chih-Kuan Ych, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. Advonces in newral information processing systems, 31, 2018.
Mateo Espinosa Zarlenga, Swami Sankaranarayanan, Jerone TA Andrews, Zohreh Shams, Mateja Jamnik, and Alice Xiang. Efficient bias mitigation without privileged information. arXiv preprint orXiv:2409.110 17691, 2024.
Xiangji Zeng, Yunliang Li, Yuchen Zhai, and Yin Zhang. Counterfactual generator: A seakly-superriseed method for named entity recognition. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7270-7227 2020.
Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Ré. Correed---cooutrall: A contrastive approach for improving robustness to spurious correlations. arXiv preprint 0rXiv:2203001517, 2022.
Zeyu Zhao. The application of the right to be forgotten in the machine learning context: From the perspective of european laws. Cath. UJL & Tech, 31:73, 2022.
Moniy-Maximllian Zühlke and Daniel Kudenko. Adversarial robustness of neural networks from the per- spective of lipschitz calculus: A survey. ACM Computing Surveys, 57(6))1-41, 2025.
19

---

Published in Transactions on Machine Learning Research (02/2026)
A
Notations
Table 11: Notation table for key equation in main draft and proof
Symbol
Description
General Definitions
M(x)
Model prediction for input
True lsbel corresponding to imput
(h((a))
Indicator function: if prediction is correct, clse 0 Validation dalaset.
vvul
D P(ur.ag)
Subset validation data with attribute value pair (a",b))
Drrain
Training dataset: Dessin (=1, .g...., ...) where 2; (ce D6) Feature set X = (01,422... (2....; Fn) and label sot (2t, 02..............................
X.Y
Training samples from Drrain
Validation sample
g
Output of the final logit layer of neural network
Vectorized parameters of the neural network, / e IRP
6
Standard unit vectors
Neural Tangent Kernel (NTK) Specific
G(X,0 0)
Neural Tangent Random Feature (NTRF) matrix: G = (.(k:66
d6
Go
NTRF matrix al initiaizaatio: Go ((X,0))
SVD Decomposition and Gradient Starvation
U,S.V V
Singular Value Decomposition (SVD) components: Ygp USVT
Singular vectors from U and V corresponding to features Singular value representing the strength the 7th Teature Response network features: =UTY SVT8 Response of the zt feature
r.
Attribution and Trak Scoring
Stacked gradient features of all training points for model n Model output used for attribution (e.g., logit loss) for imput under parameters 0 Projected gradient leature under model
(l)
(.,()
a())))))
Attribution impact of removing 4 prediction for ny
A(a)
Detrimental attribution score traiming sample 2;
Predicted probability for z; under model n
P
Random projection matrix with cntries drawn from (0,
Optimization and others
Crese, Chanage
Text and image embedddngs respectively
T
Trade-off hyperparameter: T= 3 [Paniul (control tradeolf between data attribution and semantic coherence)
8
Hyperparameter associated with Selection weight for sample i Penalty term enforcing deletion coustraint Final optimization objective including peualty
d(k)
C
C
Hyperparameter associated with soft penalty Equation g
M LL
Metric matrix constructed from L Learuable matrix under optimization defined by Equation
Accuracy threshold detect spurious hias
(*()
Final model parameters trained on the specified dataset
B
Details on Trak
1 N
alvy 21) N Σ on (1. -10
1
Σ
1 1-
N
n=1
where, p (1 + exp( y 0% pn (n( P V e f 07),
Φn(z;) = PT Vof 07); $ = [1) ;...; Φn (
4 E gmxk p NO, 1)2x k k< p
(8)
20

---

Published in Transactions on Machine Learning Research (02/2026)
Equation illustrates the calculation of the trak score. Scores consist of an average of the data attribution score calculated over multiple checkpoints (N). The terms (n(")) and (2,) denote the projected gradients of the validation sample and the training sample for the th set of parameters and projection matrix P. This projection matrix reduces the dimension of the gradient Vof(2; E R to a lower-dimensional space RR* where < p, while approximately preserving the inner product, as per the classical Johnson- Lindenstrauss theorem (Johnson 1984..
C Algorithm
Algorithm Proposed Method
Require:
Training Dataset (Drain), Validation Datasel (Dvana), Number of Checkpoints (M), Rank for Metric Learning (t), Min Weight Fraction (3), Cutoff for Subset Selection (7), CLIP Embedding Model
(C), Epochs for Classifier Training (E), Optimization Iterations for Metric Learning (I).
## Classifier Training
2: i=0
3: for epoch E EJ do
4: Train the classifier using Durain if epoch E 2 4,8 6, 8] then
Save the checkpoint 0,. i+=1 end if end for
10: Save N 100 01, 02, 03 04] checkpoints for the calculation of attribution score as per Equation 4
11: ## Spurious Feature Identification
12: Generate a list of possible attributes and corresponding values for Dvalid using ChatGPT (Section 3.2
13: for i [1... DDvall do
14: Annotate attribute-valut value pairs for sample v; using Llama-based VLM model (Section 3.2
15: end for
16: ## Calculating the Detrimental Attribution Score
17: for 2, E [21...,,) }n do
18: Calculate the attribution score A(z:) using the saved checkpoints (Fquations and 5
19: end for
20: Compare the accuracy of each attribute-value pair using Equation ! Flag an attributo-value pair
spurious if its accuracy exceeds the average dataset accuracy by threshold
21: Generate textual representation of flagged attributo-value pairs under the context of the dataset (Ap-
pendix L5p.
22: Create CLIP embedding of the textual representation (Crext.).
23: # Metric Learning
24: for E [ do
25: Calculate the CLIP image embedding (Cimage) for each sample z; Dirain-
26: end for
27: for E IJ do
28: Optimize the loss L using Crext; Cimage; and A(z) as per Equation 10 to generate the metric k using
the hyperparameter t,
29: end for
30: Use the score k, y to identify Seler and retrain the model on Derain deter
D Soft Penalty for Optimization
For efficient optimisation of the constrained objective presented in Equation 7 we have replaced the hard constraint. with soft. constraint ( (d(k) as per berdd'a d'Eon et al. (202)).
21

---

Published in Transactions on Machine Learning Research (02/2026)
Σ
Perainl
K; - (T + w))
ii
d(k) C max
99)
w²
This penalty term quadratic and scaled by shrinkable weight w, which is gradually reduced throughout the optimization process. The overall unconstrained optimization problem is defined in Equation 10 C is a hyperparameter.
where
[Disiial
his Eg Kg
L = max
A(zi) d(k).
(10)
i=1
E Theoretical Formulation
For dataset Dtrain = {..... where 2; = (s), and æ, E R and corresponding labels y; E [-1,+11" Let ò denotes the output of the final logit layer an L-layer neural network trained using binary cross- entropy, and 0 E Rº represents p-dimensional vectorized parameter of the neural network (Equation 4 Equation let X' = (..)...p)) and y (y -. constitute the respective features and class labels.
In the Neural Tangent Kernel (NTK) framework (Jacot et al. 2018), the final output of neural network can be approximated as linear function of parameters, whose properties are governed by the Neural Tangent Random Feature (NTRF) matrix, defined as:
00(X;0 0) g 0)
GRXP
(11)
00
For widhh neural networks, the NTRF matrix remains approximately constant during training PPzeeshki et al. 2021), allowing the output of the neural network to be approximated using the initial NTRF matrix,
Go G(X, Bo), as follows:
((x G00.
(12)
The dominant features of the dataset can estimated using the principal components Go = (X 6o).
which are equivalent to the principal components of the NTK gram matrix (Yang & Salman 2019).
Definition 2 (Features and gradient starvation (Pezeshki et al. 2021 Consider support vector decom- position Ygo USVT, where Y diag(y), the 8thh feature is represented by (VT) (V)..) with its strength denoted as (S)is and ils weight across all training samples represented by (U) The response of the neural network to the 8th feature can be expressed as Tú, where:
r==UYp==V"*
Due to the imbalonce in the training dataset, for a given set of features and the optimal porometer 6", the presence of the 7th feature can influence the learning of the jth feature. This phenomenon, referred to gradient starvation, arises in optimal porameters if: dr d d(s))) <0
Definition suggests that as the strength of the tth feature (s?) increases, the learning of the jth feature gets impacted. This implics that stronger features can dominate the learning process, leading to a reduced contribution of other informative features in the model's predictions.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Definition 3 (Feature contribution and strength for a given sample). Following Definition 2 and the for- mulation of Pezeshki et al. (202)) consider training instance 2n: and a feature Jj- We define the (inslance- specific) contribution of feature f to the sample as uz (Uvs) and the strength of the feature in the neural network as 8j.-
Theorem 1 (Gradient Starvation Regime (Pezeshki et al. 2021 )) For newral network in the linear regime and trained using binary cross entropy loss with feature coupling between two features f1 and f2 defined in Pezeshki et al. 2021) and with sî > s2, we have,
dī d(s})
<0, 0,
Now, under the given setting, we will try to understand the influence of gradient starvation on the perfor- mance of the NTK-based data attribution methods
Proposition 2 (Illustration for the Under Valuation of Attribution Scores). Consider neural network in the neural tangent kernel (NTK) regime, trained using binary cross-entropy loss and contain two important. features f1 and f2- Suppose that duc to learning dynamics, fi becomes dominant and causes gradient star- valion of f2 per Pezeshki et al. (2027). Lel zi and 2j be two training sumples for which JI and J2 are the most informative features, respectively, and assume that the they contain equal representations of these features (equivalently, ui u²y as per Definition S). Under these conditions, the detrimental attribution score for za can be systematically undervalued relutive to Zz. Formally:
IA(z)) < A(()))
Proof. For sigmoid- gggeodd-based activation, the output probability for feature set (x) ) is given by:
p(x:0) = 1 + exp 0(x:0))²
0(x:0) (1 (app(-((x;;))) = 1,
p(x: 0) exp [z;; 0)) -1p(tt;:0) 0),
0(x:0) log
1- (1=p(x(;))
(13)
Hence, the utility function (f) used in Trak for data attribution (Equation 3) binary cross entropy (f).
equivalent to the logit of
The gradient of the logit under the NTK framework gives,
0ğ(z;0) 0)
ago 00
Go
(14)
80
As per Equation and Equation 1984) preserves the inner product of the actual gradient vector. We will simplify our argument and calculate the value for the unprojected gradients (Park et al. 2023) (P= La). Furthermore, under the NTK regime, where the optimal parameters are similar (Jacot et al. 2018), we calculate the attribution score for single chockpoint (M=1). For case of derivation, we will omit the subscript m i..., D &b and Φ1 Φ, hence:
g + = Go P. Now, considering that the projection matrix [Johnson
= Go
(15)
T Φ G Go
(16)
23

---

Published in Transactions on Machine Learning Research (02/2026)
Now, per the feature decomposition defined in Definition
Y = USV
(Y%) ( = (USVT) (USTT)
GY = VS
(17)
Since Y = diag(yi..... ....... bnd and y {-1,1), it follows that:
yTy =1,
9 9o VS2VT
ΦT VS²VT.
(18)
The validation attribution score (Equation 5)
is given by :
A(z) = Σ -a(1;;=))
Σ
-b(u)) (o(1;)((TT)))) (0) -p²)
ByED
(19)
Substituting the value of TT
A(z)= Σ
(VSVV p²t)
o(o,)
My EDoul
Σ (ey) --))) p
Dat
Σ
o(0)) T VS
V VV (12)
Σ
vof(e,, V 0)(1 p²1) (since P and as per Equation S
Σ
Vof(u,, Vo 0) (V
-P*t)
(Eejer.n-Voftoee Vof(o,,0))ua ( ()(
Σ
(20)
where, Uk is the tt column of matrix and representing the tth feature as per Definition 2 now given the definition of the Gu and as per Equation Equation and Equation 15
Go 0) :...: Vos(zn, T)
Yg, U
(21)
24

---

Published in Transactions on Machine Learning Research (02/2026)
For the tth training sample, this score can be further simplified by multiplying with the standard unit vector (es) on both sides:
YYo - USV 31 () ut SV
(22)
where w' is row vector associated with matrix U, multiplying both side with yr and V we get
Vi MA Vof(2, 0) you's
as y 1 and further multiplying both side with eg we get
(af(z, 0) TT ex = ya's ex
Vol 0) Ukk = yeu
(23)
substituting the value in Equation 20 gives
-Dn v) (1 (1- pp:)
IA(x)) =
Σ
(24)
Skck
According to the given equation under similar probability measure, for any two data points z; and 2j where the dominant features are fi and fe respectively, the contribution of these features, as per Definition 3 is represented by ui and uzy When both dominant features are equally represented, it follows that u Mi and uf udy, u² < ui Furthermore, if |sn1| |s22l then as per Theorem fi induces gradient starvation of fa and results in lower detrimental attribution score i.e., A((a)) [A(2))
E.1 Discussion on Real-World Implications of the Analysis
As shown in our analysis, the detrimental attribution scores for samples 2; and 23 dominated by features fi and f2, respectively, governed by the ratios
and
811
822
where uf and ug denote the contributions of the respective features, and 811 and 822 capture their effective strengths.
When [ > |222|. gradient starvation occurs i.e, the feature fi dominates the training dynamics and suppresses the lcarning of fcature f2. For samples 2, and 2j with equally informative features (i.c., of uz),. this leads to
2
S11
522
resulting in lower attribution scores for samples 2; dominated by the spurious feature f1 (the feature inducing gradient starvation) compared to samples 2, dominated by the causal feature f2.
In realistic settings, samples often predominantly exhibit either causal spurious features, for example, wb 0 for 2; and uf se 0 for zj- Even in such cases if the causal feature contribution exceeds that of the spurious feature (i.c., ugg > uf) the incquality
811
822
can still hold. As a result, samples dominated by causal features may be prefcrentially removed. Intuitively, gradient starvation causes the model overestimates the relevance of spurious features and incorrectly treats samples containing causal information as detrimental.
25

---

Published in Transactions on Machine Learning Research (02/2026)
F Responsible AI
This work addressos responsible AI concerns arising from simplicity biases and distributional imbalances in large-scale training datasets, which are often collected from heterogeneous and weakly curated sources. Such biases can cause models to rely on spurious features, resulting in unfair performance degradation across subpopulations and unreliable bchavior under distribution shifts.
The current work presents a data-centric framework for identifying and miligating spurious correlations directly at the data level. Our approach aims to improve robustness and reduce unintended discriminatory effects without altering model architectures or training protocols, which may be constrained by regulatory safety requirements. Notably, while our framework employs VLMs solely for generating high-level textual descriptions of spurious features, their use can be entirely bypassed when suitable metadata or domain expert knowledge is available. Moreover, by controlling the trade off between attribution score and textual description bias (controlled via the weighting parameter 3, T in Equation 7 one can ensures that potential biases from auxiliary models like VLMs have minimal influence on the debiasing process.
Our theoretical and empirical results indicate that attribution scores can be affected by underlying spurious features, suggesting that. it is an important consideration when using attribution-based methods. For all our experiment we have only used publicly available dataset. As such, this work contributes toward more trans- parent and robust machine learning systems, particularly in high-stakes settings where biases in performance can have a significant impact.
G Group-wise Accuracy Improvements
Table 12 presents the group-wise accuracy before and after removing Sdeter from the training dataset using our proposed method. Groups 1-4 report the model's performance when trained on the original dataset for different combinations of Larget and spurious features, whereas Groups 1* show the corresponding results after updating the dataset (Derain (gdeter). As shown in the results, our method yields notable improvements in several groups without significant dogradation in others. This demonstrates that the proposed approach improves the performance the worst-performing group while maintaining comparable accuracy across the remaining groups, thereby avoiding the introduction of new biases.
Table 12: Sroup-wise accuracy before and after removing spurious samples. The table reports the mean accuracy and standard deviation over 3 runs. Groups 1 represent the training with the original dataset, while Groups 1*-4* correspond results after data pruning.
Target Attr
Spurious-Attr
G1
G2
G3
G4
G1* *
G2*
G3*
G4*
Bangs
Black Hair
0.73
0.97
0.53
0.98
0.78
0.96
0.59
0.97
Big Nose
Necklace
0.13
0.94
0.32
0.89
0.22
0.94
0.37
0.89
Heavy Makeup
Straight Hair
0.684
0.80
0.81
0.82
0.72
0.83
0.80
0.80
Earrings
Bags Under Eyes
0.09
0.99
0.04
0.99
0.28
0.96
0.32
0.91
H Data Annotation
H.1 Attribute Generation
We utilize ChatGPT to generate attributes for specific dataset with the following prompt referenced from HiBug Chen et al. 2024 The list of attribule-value pairs generated by ChatGPT is provided in Table 13
You are a helpful assistant to help user work on improving Al visual models. You need to discuss with your user for description of the task that the model working for. You need decide if the description is complete and clear enough. The description should at least contains or infer the task object, task type, task scene. After understanding user's is task description, you should generate related visual altributes that might.
26

---

Published in Transactions on Machine Learning Research (02/2026)
affect the model's performance. You should not ask me to provide visual attributes. (Note that this is only example visual albrituues according to the previous ezample, do not. take any of ils values default value!): "Gender, Age, Hairstyle, Hair colour' If user is satisfied with the attributes, generate the attribute form with the header formatted as " "//Attribute Form//" and end with "//END//". Attributes in the form should be splited by comma. Do not include the lask object, lask type, lask scene. (Note that this only on example visual attributes according to the previous example, do not take any of values as default value!): // A/ttributte Form// Gender, Age, Hairstyle, Hoår colour //END//
Table 13: Details of the altribute value pair generated using GPT.
Dataset
Attributes
Choices
Size of the Animal
Small, Medium, Large, Very Large Smooth, Rough, Furry, Scaly
Fur or Skin Texture of Animals
Color Pattern on Animal
Striped, Spotted, Solid Color, Mixed Colors Sitting, Standing, Flying, Rumning Scars, Spots, Unique Patterns Bright, Dim, Natural, Artificial, Shadowy Plain, Chuttered, Natural Habitat
Posture of Animal
Visible Markings Patterns Lighting Conditions
AWA2
Background Complexity
Presence of Humans
None, Nearby, Interacting
Animal Activity State Occlusions
Resting, Moving, Feeding, Playing Fully Visible, Partially Hidden
Weather Conditions
Sunny, Cloudy, Rainy, Foggy, Snowy Summer Coat, Winter Coat, Shedding Fur
Seasonal Variations
Gender
Male, Female
Age
Child, Teenager, Adult, Elderly Neutral, Smiling, Frowning, Surprised
Facial Expression
Hairstyle
Short, Loug, Bun, Braided
Hair Color
Black, Brown, Blonde, Red Light, Medium, Dark Beard, Mustache, Clean-shaven Glasses, Earrings, Necklace Bright, Dim, Shadowed Natural, Heavy, None
CELEBA
Skin Tone
Facial Hair
Presence of Accessories
Lighting Conditions
Makeup
Size
Large, Medium, Small
Pose/Or Oree//Oinntaioon
Side View, Top View, Angled Daylight, Nighttime, Shadows
CIFAR-10
Lighting
Background Complexity
Plain, Crowded
Object Occlusion
Partially Visible, Fully Visible
Shape of Sign
Round, Triangular, Rectangular Red, Blue, Yellow, White
Color of Sign
GTSRB
Size of Sign
Small, Medium, Large
Weather Conditions Lighting
Sunny, Rainy, Foggy, Overcast Daylight, Nighttime, Shadows, Glare
Surrounding Environment
Forest Floor, Beach, Lake, River, Ocean, Shoreline
Background Elements
Trees, Bushes, Rocks, Water Bodies, Sand, Human-made Structures Full Daylight, Shaded Areas, Low-light, Overcast Sunny, Cloudy, Rainy, Foggy, Windy
WaterBirds
Lighting Conditions
Weather Conditions
H.2 Attribute- Value Annotation
We employ Llama 3.2 Dubey et al. 2024), a Vision-Language Model (VLM) with 11B parameters, to determine the most suitable value among a set of possible attributes and values for a given dataset. By iterating over a set of images in the validation set, the VLM generates metadata, which subsequently utilized to identify the spurious features. Each image approximately takes 4-10 seconds on average annotate, depending on the size of the image. The system prompt provided to Llama 3.2 is as follows:
27

---

Published in Transactions on Machine Learning Research (02/2026)
You are on eapert in identifying visual allributes in given image. You will be presented with image along with aftributes and a list of choices for each of the attributes. You will be asked to choose the most suited choice for each of the attributes present in the image. Only choose one choice among all given choices for a particular albribule. Ensure that the choice is a string. Reproduce the altribule and the choice as is. Preserve the case and the spelling. Respond with only a valid JSON object with the attributes as the keys and the chosen choices as the values, and no other extra fluff. Use double inverted commas.
Training Procedure
1.1 Model Configuration and Metrics
We maintained consistent hyperparameter settings across all baselines, with the only variation being the subset of training data selected by cach method. The validation set was used to identify underlying spurious biases, as outlined in Section 3.2 For baseline comparisons, we utilized publicly available implementations. In cases where the code was not open-sourced or experiments were not conducted on the specific datasets, we implemented the methods and used the respective datasets for evaluation. For TracIN, we employed the fast implementation available in the Captum library (Kokhlikyan et al. 2020
Since many real-world datasets lack well-defined group structures (Sagawa et al. 2020 which are typically needed for evaluating worst-group accuracy, we compare our method and baselines primarily on average accuracy. Additionally, to understand the inffuence of deleting data samples in mitigating spurious features,
we follow the experiment setup defined by (Chaudhuri et al., 2023 Tdrissi et al. 2022 Sagawa et al.] 2020) and analyze the worst-case performance improvement. We used the methodology proposed in (Eyuboglu et al.
2022 to create a subset of CELEBA with specific simplicity biases.
1.2 Model Training and Datasets
All experiments reported in Table 0 were conducted using the ResNet-18 architecture. The models were trained from scratch with random initialization. For the WaterBirds dataset, the classifier was trained for 15 epochs using stochastic gradient descent with a momentum value of 0.9 and learning rate of 0.001. For all other datasets, we used the Adam optimizer with a learning rate of 0.001.
For AWA2-A, AWA2-B, CELEBA, datasets, models were trained for 15 epochs, while for GTSRB and CIFAR-10, models were trained for 5 epochs. We have used the same 10 classes as moned in Boecking et al. (202)) for all experiments related to AWA2. For CELEBA, we used a subset of 10,000 examples from the original datasct, with the target label being hair color (blond) and the spurions fcature being gender (male). Additionally, we induced a spurious correlation of 0.4 between the target and spurious features to mimic real-world biases. For experiments related to TmageNet-1000, we have considered the subset of the ImageNet dataset with 100 classes as per Tian al. (2020) and trained the model for 10 epochs with the Adam optimizer. We have further considered the attributes related to texture and shape for common classes available for the TmageNet dataset (Russakovsky & Fei-Fei 2010). The cutoff value to mark an attribute- value pair as spurious (7) was decided based on the size of the corresponding pair in the validation dataset, and the pair generating the largest difference with respect to the original dataset was picked for analysis.
To ensure a fair comparison for subset selection, we maintained uniformity in the training process across both the original model training and the retraining process after data deletion.
The experiments reported in Table 4 Table 5 were conducted using the ResNet-18 model, trained for 10 epochs with the Adam optimizer and learning rate of 0.001. The dataset was created by randomly sampling the correlation factor within the range [0,1] and varying the training data size across [5000, 3000, 7000, 10000). The correlation altribute and larget altribute were selected from the metadata provided in the CELEBA dataset (Eyuboglu et al.) 2022). Experiments on the following target-cone correlated attribute pairs (arched eyebrows, receding hairline), (attractive, mouth slightly open), (big nose, male), (goatee, bushy cycbrows). (mouth slightly open, smiling), (mouth slightly open, wearing lipstick), (narrow eyes, eyeglasses), (pointy nose, mouth slightly open), (receding hairline, rosy cheeks), and (male, pointy nose) are conducted with
28

---

Published in Transactions on Machine Learning Research (02/2026)
varying training dataset sizes of 3000, 5000, 5000, 5000, 5000, 5000, 7000, 7000, 7000, and 5000 samples
respectively, and corresponding spurious correlation strengths of 0.2, 0.8, 0.4, 0.4, 0.8, 0.9, 0.2, 0.6, 0.6, and 0.6 respectively. Further experiments on the target attributes Bangs, Big Nose, Heavy Makeup, and Wearing Earrings, were conducted with correlation factors of 0.6, 0.2, 0.4, and 0.2, and with training sample sizes of 10000, 5000, 3000, and 5000, respectively. Results for these experiments are provided in Table S
1.3 Comparison with Other Optimization and Data- Centric Methods
In general, ImageNet initialization (Pham al. 2021 plays crucial role in achieving strong vorst-group accuracy. However, most of experiments are conducted without ImageNet, pretraining to better re- flect practical deployment scenarios, particularly those where spurious correlations can siguificantly degrade model performance (Pham et al.) 2021). For fair comparison with optimization-baase methods such as gDRO (Sagawa et al. 2020) and JTT (Liu et al. 2021), we additionally evaluate our method on the Water- birds dataset using a ResNet-18 model pretrained on ImageNet, along with L.M--gnnrrted attribute-value annotations. Results averaged over three independent runs are reported in Table 10] We also include com- parisons with data deletion methods like D3M (Jain et al.) 2024) and balau-iganncing approaches such as SUBG and RWG (Idrissi et al.) 2022
1.4 Data Attribution and Subset Size
For the experiments reported in Table 1 approximately 3% of the data was removed from the training dataset. We fix the data removal budget across all basclines, as it is design choice best left to domain experts. A smaller removal percentage prevents overpruning of the dataset ( (rainnig sample for group land bird on water is around 56 out of 4795 (Idrissi et al.) 2022) and highlights the precision of attribution methods by focusing on the most harmful samples. In contrast, larger removals can obscure differences between methods due to overlapping sample selections. For experiments related to spurious correlation in celeba, considering the stochasticity of the training sample, we have fixed the budget size to 100 samples. Further ablation subset size is provided in Appendix P] We ensured uniformity in the data deletion process by basing it on the validation attribution score A, calculated according to the respective dcfinition of data allribution a in each baseline method, using their default. hyperparameters.
For our proposed method, we performed hyperparameter tuning by selecting the rank parameter (t) from [50, 40, 10, 100] and the minimum weight (3) from [0.6, 0.7, 0.8, 0.9, 0.95). The weight barrier (C) was chosen from 55, 10). The optimization for Equation 10 was performed for 5000 iterations using the Adam optimizer with learning rate of 0.0001. The value of y is decided based the fraction of the dataset that is removed from the training dataset. For experiments reported in Table | and Table 5 hyperparameter tuning was performed over the same range as in previous experiments, optimizing for both best average performance and best worst-group accuracy separately.
1.5 Textual Description
For different datasets, we used distinct textual representations of the underlying bias. The choice of textual descriptions in our experiments depends not only on the attribute-value pairs but also on the datasct itself. For instance, datasets like AWA2-A contain only label-specific information, such as color and habitat type, without an explicit attribute-value format. Therefore, suitable textual representation for this dataset could be "It is (+1) animal." Here, (*1) represents the feature identified as a potential biased candidate. Similarly, for GTSRB, incorporating dataset context improves model performance, and a possible template could be a(*1) of the sign is (2). where (+1) and (*2) are replaced by the corresponding altribute and value pair.
For datasets such WaterBirds, AWA2-A, AWA2-B, CELEBA, GTSRB, CIFAR-10, and ImageNet-100 the textual descriptions used in the experiments related to Table are provided in Table 14
29

---

Published in Transactions on Machine Learning Research (02/2026)
Table 14: Textual descriptions of spurious feature for different datasets
Attribute Description
Dataset
Surrounding environment in image is forest floor
Wires
It domestic animal
AWA2-A
Size of the animal very large
AWA2-B
Image of a male with blond hair
CELEBA
Shape of the sign is round
GTSRB
Size of the entity is large
CIFAR-10
Object has a spotted pottern
ImageNet-100
used standardized textual format: "Image of a person with (+1) and (2)." where (+1) and (#2) correspond to the target class and the correlated attribute, respec- tively. Further experiments using VLM-based textual description in Table for the target attributes Wearing Earrings, Bangs, Big Nose, Heavy Makeup use textual description as "Person is wearing glasses', "Image of a male person', "Person has long hair', , and "Person is wearing glasses" respectively, For metadata, we used the same format as the Table
For all experiments related to Table Table
1.6 Dataset Usage Details
Table 15 gives the details about the utilization of training, validation, and test data in our method. The training data, which is biased, is used to fit the model and learn underlying patterns, though it may introduce distributional simplicity bias. The validation data is used for identifying and analyzing biases present in the trained model. It is also used to generate textual descriptions of bias and hyperparameter tuning. For group-imbalanced datasets such as Waterbirds, we use the standard unbiased validation set provided with the dataset. For standard benchmark datasets, including Awa2, GTSRB, CIFAR-10, and Image. 100, where no explicit bias definition exists, we split the original training data into 80% training and 20% validation sets.
For the experiments reported in Table and on datasets with strong spurious correlations, the training data is constructed by subsampling the training set of the original Celeba dataset following the procedure specified in Eyuboglu et al. (2022) to induce spurious correlations. In these settings, use the original validation split, and additionally sample 20% (relative to the training size) from this validation set for the experiment. Finally, the test data is reserved for the final inference. We have used the test set provided by the original dataset for evaluation.
Table 15: Roles training, validation, and test datasets with respect to bias and model evaluation.
Dataset
Purpose Model learning and parameter optimization
Usage Description
Training Data
Used to train the model; may contain distri- butional or sampling bias influencing learned representations.
Validation Data
Bias identification and tuning
Used to detect model bias, hyperparameter tuning, and generate textual descriptions of bias.
Test Data
Final evaluation and inference
Used only after training and used to the model's generalization and unbiased per- formance.
J
Methods for Handling Spurious Features
Table 16 outlines the key capabilitics and limitations of existing methods relative ours. While data
augmentation techniques (Srivastava et al. 2020 Puli et al.) [2022 Yao al.) 2022) Zeng al., 2020 Wu
et al. 2023b Nam et al. 2022) are widely adopted, they often require external data, which can conflict with
30

---

Published in Transactions on Machine Learning Research (02/2026)
privacy and regulatory constraints (Zhao 2022
Lim & Oh 2025 Lonzetta & Hayajneh 2021 Gstrein &
Beaulieu 2022 Moreover, without appropriate supervision, they risk introducing new spurious features or being vulnerable to data poisoning attacks (Fan et al. 2022 Kumar et al. 2018
In contrast, methods such as group innotation-baser optimization (e.g., gDRO (Saaaaa et al. 2020 loss
reweighting techniques (e.g., JTT (Liu et al.] (021)); and final-layer fine-tuning (Kirichenko et al. 2023 LaBonte et al. 2023 do not pose privacy risks. However, in safety-critical applications where models must satisfy stability guarantees (Valentin, 2024 Liu et al. 2024 Zühlke & Kudenko, 2025), these methods
compromise robustness, especially when models are required to ensure Lipschitz contimuity for certification.
Specifically, they are susceptible to targeted attacks (Hartnett et al. 2019 Li et al. 2025 Necrudu et al.
2023 Holtz et al. 2022), particularly when the training procedure heavily relies on small subset of influential
examples (Cohen et al. 2020 Kumar et al., 2018 used for fine-tuning or reweighting loss values.
Group-balancing techniques (Chaudhuri et al. 2023 Idrissi et al. 2022) partially address these challenges, but often over-prune majority groups. In contrast, our method supports bungeel-onnstrained, largeted sample removal, ensuring only detrimental examples are excluded during training.
Furthermore, many of these methods (Sagawa et al. 2020 Chaudhuri et al. 2023 Idrissi ct al. 2022 rely on mamual group annotations of the training dataset. As spurious features Lesort 2023 evolve post- deployment, maintaining robustness would require repeated manual annotation cycles. In contrast, our approach eliminates the need for group labels for the training dataset and leverages textual descriptions of bias to guide targeted data removal. The use of textual description of the bias and the proposed metric learning approach provides zero-shot approach (Abdelfattah et al., 2023 Pan et al. 2024) to approximate the underlying group structure without having any annotation overhead. This design also allows intogration of feedback from subject-matler experts, making the process more adaptive and practical.
Table 16: Comparison of methods across regulatory and robustness capabilities.
Method
Regulatory Restrictious
Supports Textual No Group Annotation Privacy Prevent Over pruning of Majority Group
Robust Adv-Atlacks
Descriptious
Training Data
Data Augmentatiin
X
Group Anmotation based Optimization
Reweighting Loe/ Data
Layer Fine- Toning
Group Bolancing Method
X
Ours
K
Stability across epochs
We have further compared our method to analyze the training stability achieved by our method in comparison to the best-performing opuimization--aseed method (gDRO) in Table 10 As per the result in Table 17 the subset selected by our method provides better stability across training.
Table 17: Worst-group accuracy across epochs for gDRO and Ours.
ep 10
ep 20
ep 30
ep 40
ep 50
gDRO
0.696
0.618
0.606
0.609
0.598
Ours
0.524
0.681
0.726
0.737
0.742
L Architecture-based Ablation on Worst Group Accuracy and Average Accuracy
We further evaluate our method on the WaterBirds dalaset across different archilectures, including ResNet- 18, VGG16, VGG13, AlexNet, and ConvNet. Pham et al. (2021) shows that the random initial weights can significantly impact the worst group performance of a model, especially in smaller networks. To replicate this setting, we tested our method under extreme conditions, maintaining consistency in textual instructions and using single run with the same random seed across all baselines.
31

---

Published in Transactions on Machine Learning Research (02/2026)
As shown in Table 19 and Table 18 In comparison with the complete data setting our method achieves improvement of 5.0%, 1.9%, 3.6%, 3.1,, 12.6% in worst-group accuracy for VGG16, VGG13, Convnet, ResNet18 and AlexNet architecture and an improvement of 5.2%, 7.1% 4.9% and 7.1% for VGG16, ConvNet, ResNet18, and AlexNet in average accuracy respectively.
Furthermore, compared to Trak, our method achieves an improvement of 1.1%, 2.4%, and 4.8% in average group performance for ConvNet, ResNet18, and AlexNet, respectively. Additionally, enhancements of 5.0%, 1.4%, 7.8%, 4.5%, and 12.9% in worst-group performance were observed for VGG16, VGG13, ConvNet, ResNet18, and AlexNet.
Table 18: Architecture ablation on waterbirds (Best Worst Group Accuracy)
Model
Original
Random
IF
TracIN
EWC
Trak
Ours
VGG16
0.053
0.050
0.064
0.064
0.062
0.053
0.103
VGG13
0.048
0.053
0.087
0.030
0.065
0.053
0.067
ConvNet
0.090
0.034
0.064
0.033
0.053
0.048
0.126
ResNet18
0.050
0.064
0.067
0.017
0.048
0.036
0.081
Alex.et.
0.050
0.048
0.107
0.031
0.042
0.047
0.176
Table 19: Architecture ablation waterbirds (Best Average Accuracy)
Model
Original
Random
IF
TracIN
EWC
Trak
Ours
VGG16
0.640
0.669
0.657
0.640
0.683
0.686
0.692
VGG13
0.655
0.640
0.610
0.668
0.660
0.669
0.662
ConvNet
0.654
0.705
0.640
0.721
0.711
0.714
0.725
ResNet18
0.641
0.604
0.600
0.694
0.623
0.666
0.690
AlexNet
0.644
0.650
0.586
0.693
0.658
0.667
0.715
M Experiment on Vision Transformer
Existing data attribution methods typically compute gradients over all model paramcters, which often causes memory issues for large models like Vision Transformers. To address this, we follow recent works (Kokhlikyan et al. 2020 Pruthi et al. 2020) and calculated the gradients only for the final feature layer for both Trak and our method. However, this adaptation was incompatible with other baselines.
The results on Waterbirds for both methods are shown in Table 20
Table 20: Best Average Accuracy and Best Worst Group performance analysis of our method in comparison with Trak and Original training of vision transformer with entire dataset.
Average Accuracy Worst Group Accuracy
original Trak
0.601/0.000 0.644/0.022 0.640/0.027
0.104/0.000 0.0740/0.014 0.1671/ 0.014
ours
N Relative Comparison with the Baselines
For experiments related to Table we have provided a comparison the relative performance improvement achieved by our method against other basclines over the complete training data setting. As shown in Table 21 and Table 22 our method, on average, outperforms other baselines in terms of best average accuracy and best worst group accuracy.
32

---

Published in Transactions on Machine Learning Research (02/2026)
Table 21: Relative improvement in Best Average Accuracy (%) achieved by our method and other baselines
compared to the complete data setting(Original) The results represent the mcan scores from three indepen- dent runs, with the best-perlorming values highlighted in bold.
Target Attribute
Spurious Attribute
Random
EWC
IF
TracIN
Trak
Ours
Bangs
Black Hair
-0.03
0.37
0.36
-0.07
0.16
1.05
Big Nose
Wearing Necklace
-0.07
1.66
1.10
-0.17
2.23
2.17
Heavy Makeup
Straight Hair
-0.44
-1.05
0.95
-2.83
-0.44
2.18
Wearing Earrings
Bags Under Eyes
0.3
-0.2
-1.17
-0.57
0.1
0.73
Table 22:
Relative improvement in Best Worst Group Accuracy (%) achieved by our method and other baselines compared to the complete data setting OOiggnaa). The results represent the mean scores from three independent runs, with the best-performing values highlighted in bold.
Target Attribute Spurious Attribute Random
EWC
IF
TracIN
Trak
Ours
Bangs
Black Hair
6.25
15.62
6.66
7.05
4.77
12.58
Big Nose
Wearing Necklace
4.4
-3.57
5.10
4.68
-4.65
22.08
Heavy Makoup
Straight Hair
1.84
5.95
4.30
-1.24
3.55
6.54
Wearing Earrings
Bags Under Eyes
5.93
1.53
13.54
-3.46
-3.23
24.17
O
Sensitivity Analysis
Table 23 and Table 24 show the sensitivity of our proposed method on different hyperparameter values.
Table 23: Sensitivity analysis of the average accuracy of our method on the WaterBirds dataset for hyper- parameters like the barrier constant (C), the matrix rank (t) (shown by rows), and the minimum weight fraction (3, shown by columns).
Barrier (C) Rank (t)
0.6
0.7
0.75
0.8
0.85
0.9
40
0.657
0.619
0.648
0.673
0.650
0.640
50
0.673
0.642
0.621
0.618
0.618
0.601
100
0.670
0.678
0.650
0.602
0.632
0.631
40
0.690
0.671
0.615
0.634
0.621
0.650
10
50
0.633
0.679
0.639
0.657
0.629
0.609
100
0.653
0.663
0.602
0.642
0.660
Table 24: Sensitivity analysis of the worst group accuracy of our method on the WaterBirds dataset for hyperparameters like the barrier constant (C), the matrix rank (t) (shown by rows), and the minimum weight fraction (8, shown by columns).
Barrier (C) rank (1)
0.6
0.7
0.75
0.8
0.85
0.9
40
0.037
0.051
0.042
0.020
0.050
0.041
5
50
0.033
0.042
0.055
0.048
0.056
0.065
100
0.020
0.036
0.041
0.081
0.041
0.050
40
0.009
0.037
0.056
0.051
0.044
0.030
10
50
0.044
0.023
0.053
0.031
0.051
0.061
100
0.045
0.031
0.065
0.034
0.034
33

---

Published in Transactions on Machine Learning Research (02/2026)
P Performance Analysis on Different Subset Size
To further analyze model performance across different subset sizes, we conducted an ablation study where the best hyperparameters were kept fixed while varying the proportion of removed training data for waterbirds. The results are summarized in Table 25
Table 25: Sensitivity analysis of the worst group accuracy and average accuracy of our method on the WaterBirds dataset for different subset sizes.
Metrics
3%
5%
15%
25%
Average Accuracy
0.69
0.645
0.682
0.712
Worst group Accuracy
0.081
0.041
0.037
0.002
Q Influence of Validation Data Size
To further analyze the influence of the validation dataset, we have conducted an additional ablation to study how reliably our method identifies spurious features as the size of the validation set varies. Following the same setup as in Table B the main draft and Appendix T.2 we consider training datasets containing spurious features constructed according to Eyuboglu et al. (0022).
Specifically, for binary classification task (presence vs. absence the target) with a known spurious feature in the training data, we measure how the score (r, Ref. Equuaion 1 associated with spurious features varies with validation data size. As per the result in Table 26 across all validation sizes, the feature with the highest I consistently corresponds to the same spurious attribute associated with the training data as per Johnson
et al. )023). Consequently, the generated textual description of the underlying bias remains unchanged.
Table 26: Values of for different validation data sizes (ratio w.r.t. training data)
Target Spurious Attribute
0.1
0.2
0.4
bangs
black hair
0.038
0.048
0.055
R Performance on Bigger Training Dataset
To further analyze the performance of our method on larger datasets, we conducted additional experiments on training data with sizes comparable to ImageNet-100, consisting of 08,000 training samples and 19,600 validation samples. We follow the experimental setup of Eyuboglu al. (2022). and introduce more pro- nounced spurious correlations (approximately 0.1), consistent with the settings used in Tables 0 As shown in Table 27 when spurious features are more prominent, method achieves noticeable improvements in both average accuracy and worst-group accuracy.
Table 27: Average and Worst-Group Accuracy for bigger dataset
Target
Spurious Attribute
Original Avg Ace Ours Avg Ace Original Worst- Worst-Group Ace
Ours Worst-Group Ace
bangs black hair
0.944
0.953
0.686
0.782
heavy makeup
straight hair
0.888
0.894
0.790
0.821
S Additional Experiments on Causal vs Spurious Features
To substantiate our claim regarding the undervaluation of detrimental altribution scores, we conducted additional experiments on causal and spurious features following a similar setup described Section 4.8] for the datasct used in Table 7 As shown Tables 28 20 and 30 the mcan detrimental attribution score for spurious features is consistently lower than that for causal features. These empirical results closely align with theoretical analysis.
34

---

Published in Transactions on Machine Learning Research (02/2026)
Table 28: Classifier (Smiling vs Not Smiling)
Statistics for detrimental attribution score. (**** p < 0.001,
p < 0.01, p < 0.05; n.s. not signíficant.)
Attribute
Mean
P-value va (spurious)
Signilleance
Wearing Lipetick (spurious)
0.499
0.041
high cheekbones
0.504
0.032
2.0c-07
mouth slightly open
0.503
0.033
3.0e-05
eyes
0.501
0.038
0.21
Table 29: Classifier (Young vs Old) Statistics for detrimental attribution score. (*** < 0.001, 0.01, p < 0.05; n.s. = not significant.)
Attribute
Mean
Std
P-value vs (spurious)
Significance
smiling (spurious)
0.442
0.047
bald recoding hairline
0.452
0.070
0.004
0.451
0.074
6.0c-06
***
Table 30: Classifier (Gray Hair vs Not Gray Hair) p 0.001, p < 0.01, < 0.05; n.s. = not significant.)
Statistics for detrimental attribution score. (***
Artribute
Mean
Std
P-value vs (spurious)
Significance
eyeglasses (spurious)
0.458
0.074
old receding hairline
0.464
0.054
0.013
0.466
0.053
0.022
T Time Taken for Subset Selection
In Figure T we compare the time taken by method in comparison with other baselines to select subset of 1200 images from 60,000 images of CIFAR-10 for the instruction mentioned in Table [4] Since our method uses the attribution scores generaled by Trak and improves upon it. The time taken by our method is slightly longer than Trak.
Comparison of Methods
4,000
3,753
3,000
Time (sec)
2,000
1,845
1,496
1,000
749
456
EWC
IF
TracIN
Trak
Ours
Figure Comparison of time taken to select a subset of 1200 samples from a training dataset 60,000 images of CIFAR-10 by different basclines and (Ours) for given textual instruction.
35

---

Published in Transactions on Machine Learning Research (02/2026)
U Memory Consumption and Other Training Overhead
The Table reports GPU and RAM usage of our method compared other basclines, using the same setup described in Appendix
As shown, method introduces only marginal computational overhead over Trak, which we for computing data attribution scores. It is to be noted that, while Trak is more memory-intensive, it produces better linear datamodeling score (LDS) scores than other baselines (Park et al. 2023).
Table 31: GPU and RAM utilization (in MB) of our method compared to bascline approaches.
Method GPU Memory (MB) RAM Usage (MB)
IF
27,749
10,578
EWC
13,221
10,520
TracIN
44,087
9,629
Trak
48,020
10,710
Ours
48,525
10,722
V
Workflow
D S
Retrained Classifier
Metric Learning
k
Detrimental Attribution Score
Spurious Feature Identification
A(z)
D train
Classifier
D
VLMs
Figure 3: Diagram depicting the workflow of the proposed method
W
Images
In this section, we have shown the images that have been removed from the training datasct. Figure 4 Figure 5 Figure G and Figure show the set of images that have been removed by our method from the training dataset (Sdeter). For WaterBirds, GTSRB, CELEBA, and AWA2-B, respectively.
36

---

Published in Transactions on Machine Learning Research (02/2026)
70
50
80
50
60
30
70
50
60
50
STOP
F
100
50
50
60
120
50
Figure 4: Set of images removed by our method for WarcrBirds. The instruction set used for this experiment The surrounding environment the image is forest floor'.
Figure 5: Set of Images removed by our method for GTSRB. The instruction set used for this ex- periment " Shupe of sign round."
Figure 7: Set of Images removed by our method for Awa2-B. The instruction set used for this experiment is " The size of the animal is very large."
Figure 6: Set of Images removed by our method for CELEBA. The instruction set used for this experiment is Immgee of a male with blond hair".
37

---

Published in Transactions on Machine Learning Research (02/2026)
X
Main Results with Standard Deviation
Table 33
Table 32 and Table 34 report the performance of our method and basclines with associated
standard deviation.
Table 32:
Comparative evaluation of average accuracy of our proposed method (Ours) against baseline
approaches across multiple datasets. The results report mean accuracy scores along with standard deviation over three independent runs.
Dataset
Original
Random
IF
TracIN
EWC
Trak
Ours
WirdrBirds
0.638p.005
0.606g.004
0. 0.603g.012
0.652g.029
0.650p.022
0.656p.022
0.681g.010
AWA2-A
0.644g.u09
0.622g.014
0.644g.018
0.6522.009
0.642p.013
0.638p.o10
0.662p.014
CELEBA
0.895n.004
0.893
0.890
0.893g.007
0.890p.017
0.898o.001
0.906p.00s
GTSRB
0.969
0.966g.010
0.973g.004
0.971
0.975p.005
0.971 0.004
0.980p.002
AWA2-B
0.644
0.622a.014
0.644g.018
0.652
0.642p.u13
0.638p.010
0.657u.010
CIFAR-10
0.774
0. 0.787..006
0. 0.798p.002
0. 0.7844.000
0.789p.0ns
0.793o.003
0.801
ImageNet-100
0.440p.012
0.4366.004
0.4299.00s
0.423g.008
0.424p.003
0, 0.435%.010
0.438p.005
Table 33: Performance comparison of average accuracy across various attribute pairs. Values are reported
meansadd
Target:
Spurions Attribute
Original
Maj.-Rand
Random
IF
EWC
TracIN
Trak
Ours
archod eychrows
recrding bairline mouth slightly open male
0.713k.012
0.7
0.739
0.71
0.724p.011
0.730p.002
0.722
0.736
attractive
0.6286,019
1.6277.014
0.668g.016
0.640p.120
0. 0.6333.019
0.631
0.658o.n21
0. 0.673o.010
big nose
0.771
7.770
0.770u.o2
0.7644.mx
0. 0.751o.om2
0.745uxas
0.756m.m2
0. 01780c.019
goatee
bushy eyebrows smiling
0.946
0.917a.012
0.938
0. 0.951e.oss
0.953
0,919
0. 0.953..ne
mouth slightly open
0.869c.012
0.87 0.112
0.877o.016
0.877o.014
0.860
0.876p.014
0.867o.aco
0.877c.014
mouth slightly open
wearing lipstick
0.820k.00
0. 804
0.801g.0s3
0.829h.0rr
0 0.834
0.817axe4
0.801a.00
0 0.839..
narrow eyes
eyeglasses mouth slightly open
0.8400.
0.838
0.802
0.856f.m
0 0.838
0.880
0.855m.m12
0 1.862..ss
pointy
0.6900.
ou7
0.6766..2 o23
0.689
0. 0.695c.mm
0.708oors
0.69
0.698c.os
reeding hairline
royy theeks
0,921e
019
0.920
0.921a ao4
0.920e
0.916.
0.91
0.930e
male
pointy nose
0.919
0.931o
0.907e.017
0.909
0.911
0.906b
0.9156.014
0.
Table 34: Worst-group group accuracy across various spurious Attributes.
Values are reported as meanledd
Target
Spurious Attribute
Original
Maj.-Rand
Random
EWC
TracIN
Trak
Ours
arched eyeltows
receding baieliue
0.187o
0.314 ms
0.113p.10
0.247p. 141
0.262 onz
0.08% 0.196
0.354e 131
attractive
mouth slightly open
0.2136.13
0.242,
0.347a
0.266 a97
0.241..10
0.205g
0.392 006
0.407e
big nose
male
0. 0.1
0.076po0s
0.096,.00
0.1433.ac0
0.092
0.1 130.04
0. 0.1
0.221
goatee
bushy eyebrows
0.4322,
0. 49393.
0. 0.287
0.4
0.439
0.387
0.278o.052
0. 0.5488..50
mouth slightly open
smiling
0.
1.115a.is uar
0.552
1.
0. 1.11le.
0.187axan
0.4333..15
0. 1/189.077
mouth slightly open
wearing lipstick eyeglasses
0.555
0.47
0.557
0.598
0 0.5044.056
0.549p.04
0.486
0.6 0.6122.076
narrow eyes
0. 108
0. 0.032aoos
0.119..0s
0.000
0. 0.0922.0r9
0.1288.21
0.0246.054
0.15 0.151o.020
pointy nose
mouth slightly open
0.015
0. 0.D
0. 0.046
0.03
0.0 0.028c.o1s
0.021aoos
0.040mumm
0.00 0.084o
receding bairline
rosy cheeks
0.1216.00
0.228, 10t
1.131..08
0.180h.m
0.241a.s
0.2544
0.201o.ns0
0 0.2966..an
male
pointy
0.840p.00
0.882. 021
0.8246.077
0.833
0.861o
0.870b.co
0.875p.024
0.903e 0o9
38


Reviews:
[{"review_id": "ZYd5aUinUc", "reviewer": "Reviewer_X9wM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a data-centric method to mitigate simplicity bias (reliance on spurious correlations) in image models by identifying and removing detrimental training samples without requiring full group annotations. The authors theoretically demonstrate that standard NTK-based data attribution methods (like Trak) can systematically undervalue the influence of samples containing spurious features, and as a result the naïve pruning based on attribution scores may remove good data compared to spurious data. The authors then propose a pipeline to use an unbiased validation set to identify spurious attribute-value pairs, generate a textual description of the bias, then employ a metric learning objective (CLIP embedding + attribution scores) to select samples and remove them.\n\nStrengths:\n1.\tThe paper is clearly written, the modules (identify, score, semantically focused deletion) are well explained.\n2.\tThe method does not require group annotations of training data, only a small validation set. \n3.\tThe breadth of the empirical results covers multiple datasets and baselines. Ablations against only attribution/CLIP also show complementary gains when combined.\n\nWeakness:\n1.\tThe method's performance gains are very modest, and the behaviors are somewhat inconsistent across different benchmarks. \n2.\tThe method relies heavily on a clean, unbiased validation set. It is not clear how this is done.\n3.\tThe method relies on LLM for attribute generation + annotation where the cost can be relatively high, and reproducibility may be limited.  \n4.\tIt appears the method is quite sensitive to different hyperparameters with many tunable parameters.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1.\tThe motivation for why existing attribution methods fail is clear. The theoretical evidence is compelling. The derivation supports the claim that spurious features may result in lower attribution scores. \n2.\tEmpirically, the experiments cover a wide range of datasets and architectures. The baselines are thorough. The ablations are decent.", "audience_yn": "Yes", "audience_explanation": "Data is a key problem in modern ML, and the community is generally interested in how to best utilize/prune the training data to obtain the best results. The proposed method is practically useful and without expensive manual annotations. While the method is not ground-breaking, the theoretical analysis still provides guidance to researchers (e.g. who use influence functions regularly).", "requested_changes": "1. Since the spurious detection depends on the unbiased validation set, please precisely describe how the unbiased validation split is constructed. Ablations should be provided for the validation set (e.g., size, if it’s noisy).\n2. I believe Table 9 should be expanded to multiple datasets/spurious features. In general, consider reporting results on more attribute pairs to establish consistent patterns.\n3. The authors should report the deletion rate per experiment, and detail how the hyperparameters (e.g. $\\gamma$) are chosen.\n4. Could the authors justify whether the improvements warrant the computational overhead?\n5. It appears ImageNet-100 had minimal improvements. Can the authors discuss scalability to larger datasets? Is there a scaling limitation?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dpgHflC5u1", "reviewer": "Reviewer_Axbk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims to address the problem of simplicity bias in deep learning, where models rely on spurious features rather than predictive ones. The authors argue that existing data attribution methods ignore training samples containing spurious features due to gradient starvation. To mitigate this, the paper proposes a two-stage text-guided data subset selection strategy. First, spurious features are identified using validation data annotated by VLMs. Then, a metric learning approach that combines CLIP embeddings with data attribution scores is used to identify and remove samples that are semantically aligned with the bias and detrimental to generalization. The authors conduct experiments on a wide range of benchmarks and show improvements in average and worst-group accuracy compared to baselines.\n\n**Contributions**\n- The authors provide a theoretical proposition suggesting that in the presence of gradient starvation, NTK-based attribution scores for samples with spurious features are systematically undervalued compared to those with predictive features.\n- The paper introduces a novel pipeline that utilizes VLM-generated annotations to detect spurious features in validation sets and subsequently uses these textual descriptions to guide a metric learning objective. This objective is used to delete a subset of data based on semantic similarity to the bias and detrimental attribution scores.\n- The proposed method is shown to outperform existing data attribution baselines in terms of average and worst-group accuracy across multiple datasets.\n\n**Strengths**\n- The approach of bridging the gap between data attribution and semantic understanding with VLMs is novel. The integration of VLMs to annotate validation data when metadata is unavailable offers a practical solution for in-the-wild datasets where group labels are missing.\n- The connection drawn between gradient starvation and the failure of NTK-based attribution is valuable. Proposition 1 formalizes the claim that attribution scores for samples with dominant spurious features can be undervalued, serving as a strong motivation.\n- The results demonstrate consistent improvements over relevant baselines across a wide variety of datasets and include meaningful ablations.\n\n**Weaknesses**\n- The proof for Proposition 2 relies on strong, simplifying assumptions that may not hold generally. In Eq. 24, the authors assume that for two data points $z_i$ and $z_j$, the projection of the gradient onto the dominant feature space satisfies $u_1^i = u_2^j$ to prove the inequality. This is a strong assumption for a \"real-world\" neural network argument.\n- The definition of \"Detrimental Attribution Score\" is defined as the negative sum of attribution scores, but in the discussion of gradient starvation, the text states that spurious features result in lower scores. This seems to be conflicting with each other.\n- Tables 1 and 5 compare primarily against data attribution/pruning methods. However, the problem statement of mitigating spurious bias is a domain dominated by robust training methods like gDRO or JTT. In Table 16, gDRO seems to perform better than the proposed method. While the authors argue that their method does not require group annotations during training, they do require validation annotations. The distinction in performance vs. supervision requirements should be more transparent in the main text.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The empirical claims regarding improvement over attribution baselines are well-supported.\n- The claim that the method mitigates simplicity bias is supported.\n- The theoretical claim that NTK attribution scores are undervalued for spurious features is supported by a derivation in Appendix E, but as noted above, this derivation relies on specific symmetry assumptions.", "audience_yn": "Yes", "audience_explanation": "The finding that standard attribution methods might fail in the presence of strong spurious features due to gradient starvation is of high interest at the intersection of data-centric AI, interpretability, and robustness.", "requested_changes": "- In Appendix E, discuss what happens when the symmetry of $u_1^i = u_2^j$ is broken. If this is a simplified illustrative case, explicitly frame it as such in the main text (Proposition 1) rather than a general proof.\n- Move the summary of the comparison with Group DRO and JTT to the main text.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cSOSuO8CVf", "reviewer": "Reviewer_zAZT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a two-stage data filtering framework aimed at retaining high–information content samples while identifying and removing spurious or overly simplistic examples. The overall motivation is clear: curating higher-quality training subsets can significantly improve downstream fine-tuning performance, yet this step is often neglected in practice.\n\nIn the first stage, the authors rely on metadata or GPT-/Llama-based annotations to estimate how individual datapoints contribute to the model’s overall accuracy. This forms a coarse pre-screening mechanism.\n\nThe second stage builds on a more principled data attribution framework, specifically the leave-one-out influence score, which measures the change in a model’s prediction on a validation sample when a given training point is removed. Leveraging NTK-based methods, the authors employ a closed-form estimator to compute these influence scores across multiple model checkpoints. They further extend this by defining a detrimental attribution score, A(z), that captures the effect of removing a training point on the entire validation set.\n\nThe paper appropriately highlights a known limitation of attribution scores: due to learning dynamics such as gradient starvation, samples containing spurious features may be assigned deceptively low harmfulness scores. As articulated in Proposition 1, this can lead naïve deletion strategies to remove genuinely informative samples while retaining spurious ones.\n\nTo address this, the authors introduce an augmented filtering mechanism that incorporates zero-shot textual bias descriptions with CLIP embeddings. A semantic similarity score is computed between each training example and the textual description of the suspected spurious feature, using a learnable low-rank distance metric. The final removal strategy jointly considers (1) the detrimental attribution score and (2) semantic alignment with spurious features. The learning objective for the metric matrix L balances these two factors through a tunable hyperparameter T.\n\nOverall, the approach combines influence-based data curation with semantic bias identification in a thoughtful and complementary manner.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- They show clear improvement in multiple metrics\n- Show comparison with multiple models eg ResNet18, VGG16, VGG13, AlexNet, and ConvNet (P.S: Paper mentions Appendix L but its actually appendix K\n- Shows comparison across  multiple methods establishing a good baseline\n- Details on training, hyperparameters, and subset size are provided\n- They have shared github link for repro", "audience_yn": "Yes", "audience_explanation": "This is indeed an important—and often under-emphasized—step in fine-tuning workflows, and the method has the potential to make downstream experiments more efficient and more robust by explicitly targeting spurious correlations. The paper’s direction is promising, though further empirical validation and ablation studies would strengthen the contribution", "requested_changes": "It would strengthen the paper to have a short section on potential Responsible AI implications", "broader_impact_concerns": "The metadata generated from other VLM/LLM might propagate bias. This is something to be aware of..", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)