Revisit, Extend, and Enhance Hessian-Free Influence Functions
Authors: Ziao Yang
Han Yue
Jian Chen
Hongfu Liu
Venue: Accepted by TMLR ()

Abstract: Influence functions serve as crucial tools for assessing sample influence. By employing the first-order Taylor expansion, sample influence can be estimated without the need for expensive model retraining. However, applying influence functions directly to deep models presents challenges, primarily due to the non-convex nature of the loss function and the large size of model parameters. This difficulty not only makes computing the inverse of the Hessian matrix costly but also renders it non-existent in some cases. In this paper, we revisit a Hessian-free method, which substitutes the inverse of the Hessian matrix with an identity matrix, and offer deeper insights into why this straightforward approximation method is effective. Furthermore, we extend its applications beyond measuring model utility to include considerations of fairness and robustness. Finally, we enhance this method through an ensemble strategy. To validate its effectiveness, we conduct experiments on synthetic data and extensive evaluations on noisy label detection, sample selection for large language model fine-tuning, and defense against adversarial attacks.

Published in Transactions on Machine Learning Research (03/2026)
Revisit, Extend, and Enhance Hessian-Free Influence Functions
Ziao Yang*
ziaogang@Bbrandeiseel
Department of Computer Science
Brandeis University
Han Yuc
hunaue edu
Deportment Computer Science
Brandeis University
Jian Chen
chens@seem.ssinguua.com edw..c
Department of Management Science and Engineering Tsinghua University
Hongfu Liu
hongfuliu@brandeisccmm
Department of Computer Science Brandeis Universily
Reviewed on OpenReview: https: // openreview. net/ forum? id= 6,L2681 Tau
Equal contribution
Abstract
Influence functions serve as crucial tools for assessing sample influence. By employing the first-order Taylor expansion, sample influence be estimated without the need for ex- pensive model retraining. However, applying influence functions directly to deep models presents challenges, primarily due to the non-convex nature of the loss function and the large size of model parameters. This difficully not only makes computing the inverse of the Hessian matrix costly but also renders it non-existent in some cases. In this paper, we revisit a Hessian-free method, which substitutes the inverse of the Hessian matrix with an identity matrix, and offer deeper insights into why this straightforward approximation method is effective. Furthermore, we extend its applications beyond measuring model utility to in- clude considerations of fairness and robustness. Finally, enhance this method through an ensemble strategy. To validate its effectiveness, we conduct experiments synthetic data and extensive evaluations on noisy label detection, sample selection for large language model fine-tuning, and defense against ad adversarial attacks.
1 Introduction
Data-centric learning a growing research field that focuses on enhancing machine learning model per- formance by refining the quality and characteristics of training data (Oala et al.] 2023). In contrast model-centric approaches, which prioritize improving algorithms or optimization techniques, data-centric learning involves adjusting the
trimming, relabeling, and reweighting -while keeping the learning algorithm fixed. This approach plays a vital role in areas such as model interprotability, selecting training subsets, generating synthetic data, detecting noisy labels, improving active learning, and promoting lairness (Chhabra et. al. 2024
Kwon et al. 2023
Sample influence estimation, as the foundation of data-centric learning, can be generally categorized into categories (Hammoudch & Lowd, (2022). (a) Retraining- based methods assess the sample influence retraining the model with and without. specific sample and checking the performance change, which

---

Published in Transactions on Machine Learning Research (03/2026)
and Shapley value approaches (Ghorbani & Zou 2019 Jia et al. 2019 Kwon & Zou, 2022 Jia et al. 2018 (b) Gradient-based met estimate influence without expensive overheads of retraining, known as influence functions. The seminal work in this category is that of (Koh & Liang 2017). which utilizes a Taylor-scries approximation and LiSSA optimization (Agarwal al., 2017) to compule sample influences with the inverse of the Hessian matrix and sample gradient. However, the limiting assumption is that the model and loss function are convex. Despite debates on the necessity of convexity (Bae et al. 2022] Grosse al.) 2023 Basu et al. 2020 Epifano al. 2023), challenges persist when directly applying gradient basten methods to large models. The size of model parameters complicates calculations, particularly in obtaining the inverse of the Hessian matrix. Efforts, including matrix decomposition techniques (Koh & Liang 2017 Grosse et al. 2023 Kwon et al. 2023), aim to expedite and approximate Hessian matrix inversion, therefore enhance the feasibilily of influence functions for deep models in practical applications and complex computations.
include the classical leave-ono-out influence approach (Cook& & Weisberg 1982
Contributions. In this paper, we focus on the influence function category and revisit a specific naive yet aggressive approximation method, TracIn (Pruthi et al. 2020) an early version (Charpiat et. al., 2019). This method substitutes the inverse of the Hessian matrix with identity matrix, representing it Hessian-free influence the inner product of the gradient of the validation set and a sample to be assessed, which we refer to as Inner Product (IP). Rather than proposing a novel algorithm, our paper delivers an important message to the data-centric community: existing Hessian-free, simple methods, such as IP, might outperform more complex approaches that rely on dedicated Hessian inverse approximations. And we further explain the underlying reason. This finding emphasizes the importance of simplicity, efficiency, and scalability in influence estimation methods. Furthermore, it highlights the potential of these straightforwan tochniques to serve as robust foundation for extending wide range of data-centric diverse tasks. We summarize our major contributions as follows:
We revisit the Hessian-free influence function, define it into IP formulation, and delve into the rationale behind this simple approximation, offering insights into why it performs well in practice.
Expanding our IP framework, we extend its applicability beyond measuring sample influence on model utility to fairness and robustness.
To enhance the generalization, we propose IP Ensemble, a novel approach leveraging dropout mech- anisms to simulate diverse models. TP Ensemble amalgamates TP scores from these varied models, thus increasing the method's generalization capabilities.
We validate the effectiveness of IP through synthetic data experiments and conduct extensive real- world evaluations using IP Ensemble. These experiments span various applications, including noisy label correction for vision data, data curation aimed at fine-tuning fairer NLP models, and defense strategies against adaptive evasion adversaries.
2
Related Work
In this section, we introduce the literature on influence functions, with a focus on the acceleration of the calculation of the inverse of Hessian matrix, followed by various applications and miscellaneous.
Efficient Influence Estimation. Influence functions serve as crucial tools for estimating the individual valuation of data, without requiring model retraining. However, the computation of the inverse of the Hes- sian matrix poses challenges for large-scale data and models. To address this issue, various approaches have been proposed to simplify or estimate the inverse of the Hessian matrix effectively. A seminal work is that of (Koh & Liang 2017), which employs Taylor-series approximation and LiSSA optimization (Agarwal ct al., 2017) to compute sample influences. Arnoldi (Schioppa et al., 2022) employs the random projection and simplified Hessian matrix for acceleration. EKFAC (Grosse et al. 2023) enhances Kronecker-Pactored eigendecomposition for a precise Hessian approximation. More recently, Datalnf (Kwon et al., 2023) ef- ficiently computes influence even for large models by replacing the inverse Hessian computation with a readily computable closed-foree formm expression, although their framework may suffer from significant theoretical errors. TracInc (Pruthi et al. 2020), straightforward yet aggressive approximation, substitutes the inverse
2

---

Published in Transactions on Machine Learning Research (03/2026)
of the Hessian matrix with an identity matrix, essentially considering gradients directly mcasure of influence. Beyond the conventional influence function that gauges sample influence on the validation set, self-inftuence (Bejan et al.] 2023: Thakkar et al. 2023) computes influence using the training set alone. Moving beyond using a single model chockpoint, GEX (Kim et al. 2024] leverages a geometric ensemble of multiple checkpoints to approximate influence functions, alleviating the bilinear constraint. and non-linear losses. Morcover; TDA [Bac et al. 2024) also introduces checkpoint-
segmentation approach, combin- ing implicit differentiation and unrolling by using EKFAC (Grosse al. 2023). One concurrent work Deng et al. b024 employs ensemble strategy to improve the efficiency of TRAK Park et al. (2023). random project-based method to tackle high-dimensionaly to the Hessian matrix.
Various Applications of Influence Functions. With the above efficient approximation, influence func- tions have diverse applications. One major application is identifying detrimental samples (Hammoudeh & Lowd 2024). The learning performance can be further improved by removing (Chhabra et al. 2024), rela-
beling [Kng ct al. 2021). or reweighting (Thakkar et al. 2023) the identified detrimental samples, which has significant implications in various fields such as noisy label detection (Wang et al. 2020), subset selec-
tion [Ting & Brochu 2018), and the identification of the most influential samples (Sharchilev et al. 2018
Xia ct al.
2024 Other applications encompass few-shot learning (Park et al. 2021 where influence fune-
tions help improve model performance with minimal data, and recommendation systems (Li et al
2023
Zhang et al.
2023), enhancing the accuracy and personalization of recommendations. Influence functions
are also valuable in selecting data for active learning (Liu et al. 2021). fairness machine learning (Li &
Liu 2022 Wang et al.
2022: 2024), adversarial attack (Cohen et al. 2020), graph machine learning (Chen
et al. 2023 Wu et al.,
2023 machine unlearning (Xu et al.] 2024 Tarun et al. 2023), out-of-listribution
gencralization (Ye ct al.
2021 data privacy (Carcy al. 2023 domain adaptation (Zhang et al. 2022). to
name a few. Overall, influence functions are powerful tool with wide range of applications across different domains, contributing to advancements in both theoretical and practical aspects of machine learning.
Miscellancous. Several studics have examined the fragility of influence functions in explaining deep learning model predictions. Koh et al. (2019) expands influence functions from estimating the elfects of removing one point to large groups of training samples. Chen et al. (2020) extend traditional influence functions to monitor the impact of pro-training data. fine-tuned model predictions, facilitating the identification of crucial examples. Basu et al. (2020) demonstrates that the effectiveness of inffuence functions in neural networks varies with network architecture, depth, width, parameterization, and regularization, underscoring their fragility in deep learning due to non-convex loss functions. [Bae et al. (2022) discovers that while influence estimates may not perfectly align with leave-onc-out retraining, they approximate the proximal Brogman response function, offering valuable insights for identifying influential or mislabeled examples. [Epilano el al. (2023) suggests that the instability of current validation procodures, rather than non-convexity or lack of regularization, may be responsible for their unreliability. Lyu et al. (2023) enhance influence estimation in large-scale models by concentrating on target parameters and addressing computational instability with a robust iivero-Hessian-veetee product approximation.
3 Methods
In this section, we introduce the preliminaries of the influence function, with a focus on the Hessian-free approximation, then elaborate on our extension and upgrade.
Revisit and Simplify. Given a training set T=(==(I=,))) yo);= and a classifier with empirical risk minimization by a convex loss function é, the optimal parameters of the classifier can be obtained by 0 - argmingge n. ) (2;;0): To measure the influence of an individual data sample, we can train the model with and without the specific sample and see the performance change. However, the retrain- ap- proach is expensive for large-scale data and models. To avoid model retraining, influence functions estimate the effect of changing an infinitesimal weight of samples on a validation set V=(2)= U5) based on an impact function ʃ evaluating the quantity of interest. Considering the sample impact on model utility, i.e., the loss on the validation set, by removing one sample from the training set, this sample influence can be estimated as follows (Koh & Liang 2017]: :
Iutil(- = EySev Vol(z;;0) H 'V'V V)l(zn)))
(1)
3

---

Published in Transactions on Machine Learning Research (03/2026)
where H3=Ei= V) is the Hessian matrix of the convex l loss function.
Influence functions encounter challenge for deep models, primarily due to the non-convex nature of the loss function and the considerable size of model parameters. This obstacle not only renders the calculation of the inverse of the Hessian matrix costly but also leads to its non-existence. Various attempts, inchuding matrix decomposition methods (Koh & Liang. 2017 Grosse et al. 2023: Kwon et al. 2023). have been undertaken to expedite and approximate the inversion of the Hessian matrix, aiming to render influence functions viable for deep models. In this paper, we revisit particular naive yet aggressive approximation method TracIn (Pruthi et al. (020) or an early version Charpiat et al. (2019) by substituting the inverse of the Hessian matrix with an identity matrix, outlined as follows:
IHp (-2))= EV Vol(zz;0) V ()
(2)
The above calculation can be regarded the inner product of two gradients; therefore, call this method Inner Product (IP). Note that TracIn (Pruthi et al. 2020 incorporates multiple checkpoints to record model parameters throughout the optimization process, whereas Charpiat et al. (2019) or IP only takes into account the final or converged model. While it seems appealing to consider the dynamic sample influence change according to the optimized model parameters, simply summing these simple influence scores along different checkpoints fails to account for the evolving nature of influence, potentially neutralizing conflicting values and reducing overall effectiveness, which can be verified in our experimental results (See Table !
In the following, we link the IP to influence functions and provide our insights of IP why such naive approximation works well in practice from the angle of both the traditional influence function and IP. Traditional influence functions and their approximations require the inverse of the Hessian matrix. However, the non-linear nature of data and models not only makes this computation challenging but may also lead to the non-existence of a Hessian inverse. To address this, a modified Hessian matrix (Hg + +AI) is often employed, with approximations Koh & Liang (2017); Grosse et al. (2023); Kwon et al. (2023 from matrix theory used to calculate the influence score. However, choosing the regularization parameter À presents tradcoff: a small À may not guarantoe the existence of the inverse, while large À makes the inverse approximate the identity matrix. From the perspective of IP, it measures the similarity between two samples-the the overall validation set and a specific training sample. A high IP score indicates that the target training sample is similar to the validation set, suggesting that this sample contributes positively to improving the model's performance for both convex and non-convex models. However, in modern deep networks the Hessian is typically large, ill-conditioned, and often indefinite, so H may not exist in a strict sense and must be heavily regularized. As a result, Hessian-inverss approximations can be numerically unstable, and their approximation error is difficult to quantify, especially in non-convex settings. In this work we therefore adopt the Hessian-free inner product in Eq. (2): rather than aiming to faithfully approximate H we focus on a simple and scalable gradient-alignment signal between training samples and validation set. While IP may not always recover the exact influence value, it provides a reliable indication of the polarity of influence (beneficial vs. detrimental), which is often the main quantity needed in data-centric applications.
Extension. Beyond measuring sample influence on model utility, we extend IP to assess the sample influence on fairness and robustness by modifying the impact function f.
Specifically, we instantiate the impact function f by group fairness (Dwork et al. 2012), such as demo-
graphic parity (DP) to measure influence on fairness (Li & Liu 2022). Consider a binary sensitive attribute defined as g E {0, }} and let ÿ denote the predicted class probabilities. The fairness metric DP is defined as: DDP-fair (Ô, )) EE [v(ūl=-1]-Rr [0loo000) Within the above IP framework, we can calculate the training sample influence on fairness as follows:
IP (0 (-2,)= = V Dr-fair (V: . Vz((2,;
Dair
(3)
Similarly, we can also measure the sample influence on adversarial robustness within the IP framework. To achieve this, we follow CChabra et al. (2024) and consider a white- box adversary [Megyeri et al. 2019) specific to linear models, which can be easily extended to other models and settings. To craft an adversarial
0 2, zb 0 and
sample, we take each sample zj = (1)) in the validation set V and only perturb 2/ =x, 103-7 -7 0T 0
4

---

Published in Transactions on Machine Learning Research (03/2026)
keep yj unchanged, where & I& are the linear model coefficients, b E R is the intercept, and y > 1 controls the amount of perturbation added. In this manner, we can obtain an adversarial validation set V which consists of 2j = (2 M1) ) for each sample 2j of V. Now, we can compute adversarial robustness influence for each training sample as follows:
Irobust (-2) - Vol(e(;0) -Vg((z;0).
(4)
The simplicity IP offers opportunities to enhance the generalization of influence functions. In convex cases, the model parameter 0 is both optimal and unique. However, in non-convex scenarios, the presence of local minima introduces instability and non-uniqueness into the solution. Typically, ensemble strategies are employed to bolster model generalization (Dietterich] 2000 Lakshminarayanan et al.) 2017). Yet, while employing different models can enhance performance, also escalates the costs associated with model training and complicates the calculation of influence functions. This arises from the variability in model parameters, necessitating multiple computations of the inverse of each individual Hessian matrix. The introduction of Hessian-free IP circumvents this issue, eliminating the need for costly calculations of Hessian matrices and their inverses. Drawing inspiration from dropout mechanisms, diverse models can be swiftly generated without necessitating model retraining. By computing sample gradients from various models, we propose TP Ensemble that amalgamates TP scores from distinct models. Experiments detailed in Section 5 illustrate superior performance of IP Ensemble over other influence function- based methods.
Enhancement.
When can IP fail? Despite the above insights and enhancements, IP is not universally reliable. IP can be viewed as a Hessian-free approximation of classical inffuence functions, where the inverse Hessian in Eq. ① is replaced by the identity matrix, yielding Eq. (2). Under Eq. (2). IP essentially measures the alignment between a training sample's gradient and the validation loss gradient (i.e., the aggregated gradient over the validation set). It is worthy noting that influence functions are not accurate tools for data valuation in the deep models. Fortunately, it holds for deep models that adding more validation-lkke training samples tends to reduce the validation loss, which typically holds when the model is sufficienlly trained and the validation gradient provides a stable direction of improvement. However, when the model ill-trained or under-trained (o.g., far from convergence), gradients can be noisy or unstable, and the validation gradient may not yet be informative about the final optimization geometry. In such cases, the gradient-alignmenn signal can become unreliable, and IP may fail to rank samples correctly.
4 Correctness Verification on Synthetic Data
We evaluate the effectiveness of our Inner Product (IP) score as a surrogate for influence functions on two synthotic datasets reprosenting convex and non-convex optimization settings.
For the convex case, we generate a linearly separable datasel using scikit-learn make blobs function, con- sisting of 150 training samples and 100 test samples. We manually flip the labels of 10 training samples to introduce noise. A logistic regression model is trained on this dataset to assess IP under convex optimization. For the non-convex case, we use the half-moons dataset generated by make mokon, with 200 training samples (including 20 label-flipped noisy samples, 10 per class) and 100 test samples. We train a three-layer Multi- Layer Perceptron (MLP) with fully connected layers and ReLU activations. The network maps the input to 32 hidden units, maintains this dimensionality in the second layer, and outputs a single sigmoid-activane neuron for binary prediction.
Figure ПA B visualize the linear dataset, while D E present the half-moons dataset. Figures C and F report the influence scores and IP scores for each training sample, computed using Eqs. ① and 2 In the convex setting, the IP score closely matches the influence score, showing near-perfect correlation and order consistency. Detrimental samples reccive negative scores under both mcasures, whereas beneficial or neutral samples yield positive or near-zero values. In contrast, in the non-convex selting, influence scores become less reliable: detrimental samples are intermingled with normal samples, likely due to inaccuracies in Hessian inverse approximation. The IP score, however, continues to clearly separate detrimental samples from inliers. As discussed previously, a low IP score reflects strong divergence between a training sample and the validation objective, indicating negative contribution. Notably, despite its simplicity, IP performs
5

---

Published in Transactions on Machine Learning Research (03/2026)
Training
Validation
In Influence Estimate
2
Linear- Utit-UUllitttt
1
Inner Product
2
Peature
Feature 2
0-
22
2
-4
-6
-
0
2
4
6
8
2
0
2
6 8
-2
-1
2
Feature
Feature 1
Influence Value
x10"
4
Non-Lincer-UU Utiliy
Inner Product
centure 2
0-
-2
-2
-2
-4
4
-6
-6
2
8
-2
6
2
Feature
Fcature
Influence Value
10"
Pos. Class
Neg. Class
Pos. Norm.
Neg. Norm.
Pos. Detr.
Neg. Detr.
Figure 1: Illustration of IP on two synthetic datasets and convex/non-cone models. A-C illustrate a 2D lincarly separable synthetic dataset with a subset of detrimental samples bearing incorrect labels, and D-F demonstrate the similar analysis on a non-linear synthetic half-moon dataset. Specifically, A and D depict training sets with two classes, where detrimental samples marked with x and regular samples with B and E show test sets. C and F present influence scores and IP scores by Eqs. and ( respectively.
comparably more complex methods that rely on Hessian inverse approximations, even in the non-conver case. Additional empirical evaluations model fairness and robustness provided in Appendix A.1
Next, we will present unified empirical program along three sections: (1) noisy-label data curation for vision, where we rank samples by influence, and retrain to measure accuracy gains; (2) fairness-aware curation for NLP, where we compute influence on both utility and a cemographic-perturbaaioo fairness score, and visualize the accuracy-fairnest frontier; (3) robustness to adaptive test-time evasion, where we compare pre/post-attack accuracy and evaluate three defenses.
5 Noisy Label Correction for Vision Datasets
In this section, we demonstrate the effectiveness of our TP Ensemble in identifying detrimental samp noisy vision datasets. Specifically, we choose three benchmark datasets CIFAR-ION (Wei al.) 2022). CIFAR-10ON (Wei et al.) 2022). and Anémal- 10N (Shu et al. 2023) in the noisy label learning area. Both the CIFAR-1ON and CIFAR- R-100N datasets (Oliver et al.
2018 consist of the same input images their clean
counterparts, CIFAR-10 (10 classes) and CIFAR-100 (100 classes) (Krizhevsky et al. 2009). respectively. Each input is 32x32 RGB image with dimensions (3,32,32). However, for CIFAR-1ON and CIPAR-10ON, the labels contain real-world human annotation errors collected using three annotators on Amazon Mechanical Turk. Since these datascts are based on human-annotated noise, they provide more realistic modeling of noisy real-world scenarios compared to synthotic alternatives. The training for both datasets contains 50,000 image-label pairs, while the test set contains 10,000 clean image-label pairs. Specifically, C-

---

Published in Transactions on Machine Learning Research (03/2026)
10N encompasses three distinct noise settings: aggregate ("-a"), random (*-r"), and worst ("-w"). The "aggregate" setting has the lowest noise rate (9.03%), where labels are determined via majority voting among three annotators, with ties resolved randomly. The "random" setting has intermediate noise (17.23%), adopting labels from the first annotator. The "worst" setting has the highest noise rate (40.21%), deliberately selecting the most inconsistent annotation per sample.
For competitive methods, we choose the following influence function-bases methods. TracIn (Pruthi et al, 2020) replaces the Hessian matrix with the identity matrix and considers checkpoints during the training process; LiSSA (Koh & Liang 2017) and EKFAC (Grosse et al., 2023) employ implicit Hessian-vector products and Kroneckor-Factord
curvature to efficiently approximate the inverse of the Hessian matrix; DataInf (Kwon et al. (023) swaps the order of matrix multiplication for obtaining a closed-form expression; Self TracIn (Thakaar et al., 2023) and Self-LiSSA (Bejan et al.) 2023) are the self-expression versions of TracIn and LiSSA, where the gradients of the validation set are replaced with V Vgl(z;; 0), and only the last checkpoint, i.e., the converged model parameters, is used in Self-TracIn. GEX (Kim et al. 2024) utilizes ensemble methods based on checkpoints from extra stochastic gradient descent on the converged model. TDA (Bac et al.) [2024] focuses on checkpoints during the training process, ensembling the influence via EKFAČ [Grosse al. 2023). Our IP Ensemble method constilutes an ensemble version of IP with ((0, 0.01) dropout applied on model parameters and an ensemble size of 5.
TracIn, GEX, IP, and IP Ensemble are all Hessian-free ensemble methods except for IP. To clarify their differences, we present their calculations explicilly:
TIP (-22,
-2, E, (v((;;)))[(((;;))))
Zutil TracIn 1-2, Traciu)
1
I (- 2, 0t),
T
(5)
1
GEX (-zi, 2i, OGEX)
LB.E0GEX
ITP (-
Zi, 0z):
T
1 Σ T
IP Ensemble (-zi, OIP Ensemble)
I (-2i, 0.),
0, Eusemble
where θ in IP represents the converged model parameters; T denotes the checkpoints saved during training: OGEX is derived by additional training on the converged model; IP Ensemble obtains OIp Ensemble through dropout mechanisms on the converged model; T is the ensemble size. The advantages of TP Ensemble are that it does not require checkpoint saving as in TracIn, nor does it necessitate extra training ilerations in GEX. We argue that early checkpoints may not effectively reflect sample influence, and additional training on converged models has minimal impact. We provide our implementation in an anonymous open-source repository: lntps:// http://nnoooos..ooppp
Baaiienccrr/__cnsembbeeBBB BBOF /README..E. md. The experiments were conducted on a Linux (Ubuntu 20.04.6 LTS) server using NVIDIA GeForce RTX 4090 GPUs with 24GB VRAM running CUDA version 12.3 and driver version 545.23.08.
To reduce randomness, we train a ResNet-34 network (He et al. 2016 which is a 34-layer convolutional neural network pretrained on the In dataset al resolution 224- x 224. The pretrained model block is fine-tuned on the CIFAR-100 10N/1ON//CIIA LO//CIFAR-100, 100N training set using default parameters: minibatch size (128), optimizer (SGD), initial learning rate (0.1), momentum (0.9), weight decay (0.0005), and number of epochs (100). Subsequently, based on the same model, we employ the aforementioned influence function-based methods to identify 5% of detrimental samples. Following this, we conduct five retraining iterations of the ResNet-34 network, each time removing the identified detrimental samples from the training set. For each method, we repeat the following procedure five times (each time with an independent run): we first identify the bottom 5% most detrimental samples, then retrain the model on the remaining 95% of the training data. Tables 1 and 5 report the results averaged over these five runs. Table 1 reports the average accuracy and standard deviation of the above influence funetion-based methods across these five retrainings. In general, these influence function-basee Upon retraining the ResNet-34 model without these identified samples, nearly every result obtained by these methods outperforms the vanilla ResNet-34 trained on the entire dataset, except for EKFAC on C-FAR- 10N-w. Moreover, TracIn and TDA achieve only moderate performance, largely due to their reliance on
methods are effective in identifying detrimental samples at different levels.
7

---

Published in Transactions on Machine Learning Research (03/2026)
Table 1: Accuracy results of influence
methods on the CIFARION, CIFAR-10ON and Animal-
10N datascts with 5% identified detrimental samples removed
Methods
Datasets
CIFAR-ION-a CIPAR-ION-r CIFAR-ION-W
CIFAR-10ON
Anömal- 10N
Avg.
Cross Eutropy
91.62
90.25
85.66
56.41
80.54
80.90
L1SSA Koh & Liang
2017
92.13
90.98
85.97 0.47
59.24
0.39
81.93 0.14
82.05
TracIn Pruthi et
2020
90.48 0.12
88.09
+ 0.24
85.18 1.05
56.47
1.87
80.
0.57
80.07
EKFAC
Grosse
a..
2024
91.76 0.23
90.47
+: 0.10
83.25 0.38
59.91
0.90
80.89
= 0.54
.6
DataInf
Wwon el al. 2023
91.88 0.39
90.79 + 0.21
86.22 0.13
58.40 :: 0.22
81.60 = 0.23
81.78
Self Tracin Thakkar el. al.
2023
92.03 0.09
90.43 1 0.24
86.00 0.18
61.99 0.29
81.82 0.34
82.45
Self-LiSSA
(Betan et al. 2023
91.91 0.17
90.66 0.35
85.73
61.56 0.96
81.23 0.24
82.22
TDA Bae cl 2024
91.95 0.19
89.87 0.52
84.02
58.91
80.57
81.06
GEX Kim al. 2024
91.81 0:27
90.68 + 0.39
85.64 0.20
58.47 0.48
80.78
0.68
81.49
IP (Ouns)
92.42 0.17
90.82 + 0.08
86.31 0.35
60.59
0.20
81.19
0.22
82.27
IP Ensemble (Ours)
92.26 0.19
91.28 0.29
86.50 4: 0.35
62.25 0.54
82.35
0.55
82.93
multiple checkpoints throughout the training process. This highlights why we believe that simply summing up sample influence across different checkpoints fails to effectively capture the dynamic and evolving value of a sample during training.
Notably, our IP Ensemble consistently outperforms other baseline methods across various noise conditions and datasets. Particularly noteworthy is the performance of our IP Ensemble in the most challenging scenario, CIFAR-10ON, achieving the highest recorded accuracy of 62.25% on the test set, surpassing the vanilla cross- entropy accuracy of 56.41%. Compared to IP, IP Ensemble delivers further performance gains, highlighting the benefits the ensemble strategy for enhancing model generalization. Moreover, the aver- age accuracy of the IP Ensemble on the test set reaches its peak at 82.93%, surpassing both the vanilla cross-entropy performance of 80.90% and the second-best, accuracy of Self-TracIn at 82.45%. More experimental results and analysis on different percentages of removed samples, parameter analysis on different dropout rates, ensemble sizes, and base model architectures can be found Ap- pendix A.2 A.4
10a
Speed Improvement Factors
10a
101
10a
CIFAR JON
CIFAR-
Animal ION
LiSSA
DataInf Self-TraeIn
Sclf-LiSSA TDA
EKFAC
Figure 2:
Speed improvement factors
of IP over other methods.
In addition, we also prosent the rumning times of these inftuence function-based methods. Despite some baselines having linear time complexity, there is significant divergence in their real execution times. Given that our IP exhibits exceptional speed and similar execution times, we consider them as the baseline and compute the speed improvement factors over other baseline methods, as depicted in Figure 2 For ensemble methods including TracIN, TDA, GEX, and our IP Ensemble, parallel computation can be applied to ac- celerate the running time enough resources are allowed; if calculated serially, the time grows linearly with the ensemble size. When we visualize their running time in the logarithu scale, the extended time due to the ensemble size is not significant; for example, EKFAC and its ensemble version TDA. Thus, we do not report the ensemble-bart based methods in Figure 2 except for the above example of TDA. With the exception of Self-TracIn, our IP runs over 100 times faster than LiSSA, EKFAC, DataInf, and Self-LiSSA. Notably, on AnN, IP is over 800 times faster than EKFAC. The time complexities and execution time of these methods can be found in Appendix A.5
6 Data Curation Towards Fine-Tuning of Fairer NLP Models
In this section, we further demonstrate the efficacy of our IP method in gauging the impact individual samples on fairness within the realm of curating suitable data samples for fino-tuning language models. Beyond mere utility, fairness has emerged as an indispensable attribute for machine learning models mitigate inadvertent discrimination.
8

---

Published in Transactions on Machine Learning Research (03/2026)
RTE
COLA
x10
ONLI
0
2
Fair Score
Fair Score
Fair Score
4-
6
65
69
57
61
91
92
Accuracy
Accuracy
Accuracy
Baseline
LiSSA
TracIn
EKFAC
DataInf Self-
Self-LiSSA
TDA GEX IP Ensem
Figure 3: Accuracy and fair score of different influence function-based methods on fine-tuning RTE, COLA, and QNLI datasets. The X-axis denotes accuracy, and the Y-axis for fair score is inverted. The brown crossing denotes the performance of using all the samples for fine-tuning the RoBERTa model the baseline model. Building upon this, we plot horizontal and vertical line in cach figure and divide the space by fairness and utility results into four regions. The green area in the top right corner signifies model that is both fairer and more accurate compared to the baseline model.
In this experiment, we employ three datasets-RII CoLA, and QNLI-from the GLUE repository (Wang et al.] 2018) to fine-tune the widely-used language model RoBERTa {Liu et al. 2019) Our focus is on group fairness, necessitating that machine learning models treat samples within various predefined subgroups comparably. These datasets represent diverse natural language understanding lasks, thus allowing us to comprehensively evaluate the fairness and utility of the fine-tuned RoBERTa model. The Recognézing Tertual Entailment (RTE) dataset consists of sentence pairs labeled as entailment non-entailment, derived from series of textual entailment challenges, containing 2,490 training examples and 277 validation examples. The Corpus of Linguistic Acceptabilityy (CoLA) dataset includes sentences labeled as grammatically acceptable or unacceptable, derived from linguistic publications, and consists of 8,551 training examples and 1,043 validation examples. The Question Natural Language Inference (QNLI) dataset is a large-scale corpus for question answering, comprising questioon-seteence pairs from the Stanford Question Answering datasel, with 104,743 training examples and 5,463 validation examples. As the test sets for these datasets lack labels, we split each validation set into two equal parts, utilizing half the validation set for computing influence, and the other half as the test set. Perturbed versions of both the validation and test sets are generated using seq2seq model as detailed in Qian et al. (2022). To assess fairness, we adopt the methodology outlined in Qian et al. (2022), which involves perturbing the demographic information within each sample and scrutinizing whether the model yields identical predictions for the original sample x and its corresponding perturbed counterpart The fairness evaluation metric ('fair score") is defined as [C(x) - C()) over all test samples, normalized by the test set size, where C(-) the model predictor. Note that the fair score is a negative metric; hence, smaller values are preferable. We fine-tune the RoBERTa-base model from Huggingface] with the following experimental settings: learning rate of 10-5, batch sizes of 64, 16, and 32 for RTE, CoLA, and QNLI, respectively, and total of 10 epochs. The loss function used is negative log-likelihood. appropriate for these binary classification tasks.
Utility and fairness serve as distinct perspectives for evaluating model performance. Thus, in our Ene-tuning experiments, we consider both dimensions. Employing the same influence dunction-based methods and our IP Ensemble in the previous section, we conduct comparative analyses. For each method, we compute influence scores for both utility and fairness the fine-tuning set, aggregate them into single joint ranking, and remove the bottom 5% samples under this joint criterion before fine-tuning the RoBERTa model.

---

Published in Transactions on Machine Learning Research (03/2026)
Bank
CelebA
TigsawToxicciy
0.90
0.75
0.85
0.85
Accuragy
0.80
Accuracy
.0.00
Accuracy
0.70
0.75
0.75
0.65
0.70
0.70
0.65
0.60
Pre
Post
LiSSA
TracIn
EKFAC
DataInf
Self-LiSSA
Ensem.
Figure 4: Performance across influence function-based methods over 10 distinct attacks on Bank, CelebA, and Jigsaw Toricity- The dashed gray line presents the pre-attack performance, while the brown line denotes the average accuracy of post-attack.
The results of this experiment are presented in Figure where the Y-axis for fairness is inverted. The brown crossing denotes the performance of using all the samples for model fine-tuning as the base model. Building upon this, we plot horizontal and vertical line in each figure and divide the space by fairness and utility results into four regions. The green area in the top right corner signifies model that is both fairer and more accurate compared to the baseline model. Results within this green region can be considered Pareto improvements, enhancing both utility and fairness simultaneously. It is evident most results are located in the green area, indicating the existence of detrimental samples, and not all the samples are helpful the model performance. This also implics that influence function methods are effective to identifying the detrimental samples, even for non-convex deep models. However, some competitive methods yield trade- off or even Pareto deterioration. For instance, LiSSA demonstrates better fair score but worse accuracy compared to the base model on RTE; conversely, it cxhibits better accuracy but worse fairness on QLIN. EKFAC shows similar performance on COLA and QLIN. Self-LiSSA demonstrates Pareto deterioration on COLA. Our IP Ensemble consistently achieves Pareto improvements across all three datasets, often yielding the best results compared to other methods.
7 Defense Against Adaptive Adversaries
In this section, we demonstrate how the influonco-based approach can effectively fortify defenses against an adaplive adversary (Tramer el al. 2020: Biggio al.) 2013) that performs evasion allacks on the learning model. In this scenario, the attacker randomly selects a subset of test samples to launch the evasion attack. We defend by proactively trimming the training set by predetermined amount, although we lack specific knowledge about which samples are adversarial during inference.
In this experiment, we utilize a Logistic Regression model with three datasets: Bank (Moro et al. 2014 CelebA (Liu et al. 2018), and JigsawTozicity (Novver 2018) to evaluate the defonse against adaptive evasion adversaries. The Logistic Regression model implemented using the sklearn library. The model uses L2 regularization with and a maximum iteration limit of 2,048 (Chhabra et al., 2024). The Bank dataset consists of features extracted from direct marketing campaigns of Portuguese banking institution, aiming to predict whether client will subscribe to term deposit, including 18,292 training examples, 6,098 validation examples, and 6,098 test examples, with feature dimension of 51. The CelehA dataset is a large-scale face attributes datasct with more than 200,000 celebrity images annotated with 40 binary attributes; we focus on a subset comprising 62,490 497 training examples, 20,833 validation examples, and 20,833 test examples, each with a feature dimension of 39. The JigsawToriciiy dataset includes comments from Wikipedia labeled for toxicity prediction, containing 18,000 training examples, 6,000 validation examples, and 6,000 test examples with a feature dimension of 385. Following the protocol in Section 3 we consider a white-box adversary (Megyeri et al. 2019 to craft adversarial samples. For each sample (I,, fj) in the validation set V, we perturb by TTz--b and keeping yj unchanged. The attacker perturbs between 5% changing the feature z/y Ij 7 0TO
10

---

Published in Transactions on Machine Learning Research (03/2026)
Table 2: Defense performance of varions inftuence function- based methods under the relabeling and reweight- ing strategies on Bank, CelehA, and Jigsaw Toxicity datasets
Relabellng
Reweighting
Defense Strategy
Bank
CeledAA
Figsou Tauicity
Avg
Bank
CelebA
Jigeow Touricity
Avg
Pre
80.31
85.26
73.58
79.72
80.31
85.26
73.58
79.72
Post
70.79 3.71
75.33
3.04
66.18
3.52
70.77
70.79 = am
75.33
3.04
66.18 3.02
70.77
LISSA Koh & Llang
2017
86.07 0.39
68.68
1.39
73.30 #: 2.96
76.02
08.60- 3.56
73.23
3,70
70.13
1.00
74.02
EKFAC
Grosse
2024
87.38
77.36
1.94
70.04
78.26
83.50
75.11
= 1.73
70.05 0.70
76,22
DataInf
Kwon 2023
87.46 2.44
77.36
1,85
70.39
78.40
85.99
74.74
70.82
78.18
Self-Trachn
naakar al:
2023
86.12 1.10
75.58
2:97
68.31
2.70
76.67
85. 22 1.72
73.12
71.37
1.81
76.57
Self-
Bejan et al.
2123
78.97 3.40
75.56
9.90
69.18
74.37
85.16
1.74
73.17 3.79
71.41 1.86
76.58
IP (Ours)
87.45 0.27
77.28
1.87
70.43
0.87
78.38
86.17
2.04
75.10 à: 1.71
70.82 0.36
77.36
IP Ensemble (Ours)
57.45 0.32
77.30 1.89
70.51
1.00
78.12
86.44 +: 2.38
75,10 + 1.7a
70.67
0.27
77.40
and 25% of the test set samples at random. By quantifying the impact of samples on model robustness, we trim 5% detrimental samples in the training set through influence functions. The boxplot depicted in Figure 4 demonstrates the performance variations across various influence function-basod methods over 10 distinet attacks. Since the Logistic Regression model is convex, TDA simplifies to EKFAC, and TracIn and GEX simplify to IP; therefore, we omit their duplicate results in this figure and the following table. For the convex case, the existence of the inverse Hessian matrix is guaranteed, and LiSSA is the most suitable algorithm for influence estimation. Even though, among these influence function-based methods, our IP Ensemble demonstrates competitive efficacy with the best or the second best on these three datasets. These datasets cover a range of tasks from marketing prediction and face attribute recognition to toxicity detection, providing a robust evaluation of the model's defense mechanisms against adaptive adversarics.
In addition to the trimming strategy, we continue explore relabeling and reweighting strategies to tackle detrimental samples. The relabeling strategy (Kong et al. 2021 changes the identified detrimental samples from their original classes into another class, since the datasets we used here are binary classes, directly flip their labels. The reweighting strategy (Thakkar et al. 2023) takes the influence score of each sample as the exponential weight with a softmax normalization, and then trains a model with weighted samples. We report the defense performance of various influence function-based methods and our IP Ensemble under the relabeling and reweighting strategies in Table [ Under the relabeling strategy, DataInf achieves the best results on Bank and CelebA with 87.46% and 77.36% accuracy, respectively. However, LiSSA is not stable other methods Our IP ensemble method continues to achieve competilive performance, with accuracy scores of 87.45%, 77.30% and 70.51% on three datasets with the second-best performance among all influence functions- based methods. For the reweighting strategy, the influence function-basod mothods are still effective on Bank and JirsauToricity, which outperform the post-attack by a large margin, but achieve similar performance with post attack on CelebA. Notably, when considering the average accuracy of both the relabeling and reweighting strategies, our IP Ensemble method performs the best.
CelebA with only 68. 68.68%, which is even worse than the performance of post attack.
8 Conclusion
In this paper, we revisited and simplified the TracIn method into the Inner Product (IP) formulation, which substitutes the inverse of the Hessian matrix with an identity matrix offers a practical and computationally efficient solution to estimating sample influence. Based on that, extended our IP to measure the sample influence on fairness and robustness. Continually, ensemble strategy. We verified the effectiveness of IP on synthetic datasets and extensive evaluations on noisy label detection, data curation for fair NL.P model fine-tuning, and defense against adaptive adversarial attacks. Overall, our IP Ensemble highlighted the potential of simple, yet powerful, approximations in influence estimation for practical use.
enhanced the generalization of IP by introducing an
11

---

Published in Transactions on Machine Learning Research (03/2026)
Acknowledgment
We gratefully acknowledge the support of the Google TPU Builders Program, which provided support and access to computational resources, and Google Tunix library for post-training functionality and flexibility to enable this work. We also thank the anonymous reviewers and the area chair for their valuable feedback and constructive suggestions.
References
Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research, 2017.
Juhan Bae, Nathan Ng, Alston To, Marzyeh Ghassemi, and Roger B Grosse. influence functions are the answer, then what is the question? Advances in Neural Information Processing Systems, 2022.
Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Grosse. Training data attribution via approximate unrolled differentation. arxi preprint 240::2405..2166, 2024.
Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. arXiv preprint Xrxi::2006. 14651, 2020.
Irina Bejan, Artem Sokolov, and Katja Filippova. Make every example count: On the stability and utility self-influence for learning from noisy ulp datasets. arXiv preprint arXie:2302.118 2023.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šindié, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning test time. In European Conference Machine Learning and Principles and Practice of Knowledge Discovery in Dotabases, 2013.
Alycia N Carey, Minh-Hao Van, and Xintao Wu. Evaluating the impact of local differential privacy on utility loss via influence functions. arXiv preprint XrX9::2309.058: 088788, 2023.
Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input similarity from the neural network perspective. Advances in Neural Information Processing Systems, 32, 2019.
Hongge Chen, Si Si, Yang Li, Ciprian Chelba, Sanjiv Kumar, Duane Boning, and Cho-Jui Hsieh. Multi-stage influence function. Advances in Neural Information Processing Systems, 2020.
Zizhang Chen, Peizhao Li, Hongfu Liu, and Pengyu Hong. Characterizing the influence graph elements. International Conference on Learning Representations, 2023.
Anshuman Chhabra, Peizhao Li, Prasant Mohapatra, and Hongfu Liu. What data benefits my classifier? enhancing model performance and interpretability through influence-basee data selection. In International Conference Learning Representations, 2024.
Gilad Cohen, Guillermo Sapiro, and Raja Giryes. Detecting adversarial samples using influence functions and ncarest neighbors. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
R Dennis Cook and Sanford Weisborg. Residuals and influence in regression. New York: Chapman and Hall, 1982.
Junwei Deng, Ting- Wei Li, Shichang Zhang, and Jiaqi Ma. Efficient ensembles improve training data attri- bution. Xixiv preprint arXiv:2405. 17293, 2024.
Wing-Wei
Thomas G Dicttorich. Ensemble methods in machine learning. In International workshop on multiple classifier systems. Springer, 2000.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprinl ar Xiv:2010. 11929, 2020.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Innovations in Theoretical Computer Science Conference, 2012.
Jacob R. Epifano, Ravi P Ramachandran, Aaron Masino, and Ghulam Rasool. Revisiting the fragility of influence functions. Newral Networks, 162, 2023.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, 2019.
Roger Grosse, Juhan Bae, Cem Amil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint XXiv:3308.03290 2023.
Zayd Hammoudeh and Daniel Lowd. Training data influence analysis and estimation: A survey. arXiv preprint ar arXit:2212.04612 2022.
Zayd Hammoudch and Daniel Lowd. Training data influence analysis and estimation: A survey. Machine Learning, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Tn IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas Spanos, and Dawn Song. Efficient task specific data valuation for nearest neighbor algorithuss. In Inter- national Conference on Very Large Data Bases Endoument, 2018.
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas Spanos. Towards efficient data valuation based on the Shapley value. In International Conference on Artificial Intelligence and Stalistics, 2019.
SungYub Kim, Kyungsu Kim, and Eunho Yang. Gex: A flexible method for approximating influence via geometric ensemble. Advances in Neural Information Processing Systems, 2024.
Pang Wei Koh and Percy Liang. Understanding black- box predictions via influence functions. International Conference on Machine Learning, 2017.
Pang Wei W Koh, Kai-Siang Ang, Hubert Teo, and Percy S Liang. On the accuracy of influence functions for measuring group effects. Advances in Newrul Information Processing Systems, 2019.
Shuming Kong, Yanyan Shen, and Linpeng Huang. Resolving training biases via influence- data rela-
beling. In International Conference on Learning Representations, 2021.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yongchan Kwon and James Zou. Beta Shapley: Unified and Noise-reduced Data Valuation Framework for Machine Learning. In International Conference on Artificial Intelligence and Statistics, 2022.
Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned Ilms and diffusion models. arXiv preprint orXiv:2310000902 2023.
Balaji Lakshninarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 2017.
Peizhao Li and Hongfu Liu. Achieving fairness at no utility cost via data reweighing with influence. In International Conference on Machine Learning, 2022.
Yuyuan Li, Chaochao Chen, Xiaolin Zheng, Yizhao Zhang, Biao Gong, Jun Wang, and Linxun Chen. Selective and collaborative influence function for efficient recommendation unlearning. Expert Systems with Applications, 2023.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint wrXii:1907.11690 2019.
Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He. Influence selection for active learning. In IEEE/CVF International Conference Computer Vision, 2021.
Ziwci Liu, Ping Luo, Xiaogang Wang, and Xiaoon Tang. Large-scale Celcbfaces attributes (CelobA) datasct. Retrieved August, 2018.
Hyeonsu Lyu, Jonggyu Jang, Sehyun Ryu, and Hyun Jong Yang. Deeper understanding of black-box pre- dictions via generalized influence functions. orXiv preprinl arXiv:2312.058: 05586,, 2023.
István Megyeri, István Hegedus, and Márk Jelasity. Adversarial robustness of linear models: regularization and dimensionality. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. 2019.
Sérgio Moro, Paulo Cortez, and Paulo Rita. A data-driven approach to predict the success of bank telemar- keting, Decision Support Systems, 2014.
David Noever. Machine learning suites for online toxicity detection. arXiv preprint ar arXi::1810.0158 2018.
Luis Oala, Manil Maskey, Lilith Bat-Leah, Alicia Parrish, Nezihe Merve Gürel, Tzu-Sheng Kuo, Yang Liu, Rotem Dror, Danilo Brajovic, Xiaozhe Yao, et al. Dmir: Data-centric machine learning research past, present and future. ar preprint arXiv:2311. 130288 2023.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. Advances in Neural Information Processing Systems, 2018.
Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. Influence-balanced loss for imbalanced visual classification. In IEEE/CVE International Conference Computer Vision, 2021.
Sung Min Park, Kristian Georgiev, Andrew Llyas, Guillaume Leclerc, and Aleksander Madry. Trak: At- tributing model behavior at scale. arXiv preprial ar XrXi::2303.14 14 2023.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 2020.
Rebecca Qian, Candace Ross, Jude Fernandes, Eric Michael Smith, Douwe Kiela, and Adina Williams. Perturbation augmentation for fairer NL.. In Conference Empirical Methods in Natural Language Processing, 2022.
Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In AAAI Conference on Artificial Intelligence, 2022.
Boris Sharchilev, Yury Ustinovskiy, Pavel Serdyukov, and Maarten Rijke. Finding influential training samples for gradient boosted decision trees. In International Conference on Machine Learning, 2018
Jun Shu, Xiang Yuan, Deyu Meng, and Zongben Xu. Cmw-net: Learning a class-aware sample weighting mapping for robust deep learning. IEFE Transactions Paltern Analysis and Machine Intelligence, 2023.
Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Megh Thakkar, Tolga Bolukbasi, Sriram Ganapathy, Shikhar Vashishth, Sarath Chandar, and Partha Talukdar. Self-influence guided data reweighting for language model pro-training. arXiv preprint arXiv:2811.00913 2023.
Daniel Ting and Eric Brochu. Optimal subsampling with influence functions. Advances in Neural Information Processing Systems, 2018.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mip architecture for vision. Advances in Neural Information Processing Systems, 2021.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks adver- sarial example defenses. Advances in Neural Information Processing Systems, 2020.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.0744 2018.
Haonan Wang, Ziwei Wu, and Jingrui He. Fairif: Boosting fairness in deep learning via influence functions with validation set sensitive attributes. In International Conference on Web Search and Data Mining, 2024.
Jialu Wang, Xin Eric Wang, and Yang Liu. Understanding instance-level impact of fairness constraints. In Internaiional Conference on Machine Learning, 2022.
Zifeng Wang, Hong Zhu, Zhenhua Dong, Xiugiang Hc, and Shao-Lun Huang. Less is better: Unweighted data subsampling via influence function. In AAAI Conference Artificial Intelligence, 2020.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations, 2022.
Jiancan Wu, Yi Yang, Yuchun Qian, Yongduo Sui, Xiang Wang, and Xiangnan He. Gif: A general graph unlearning strategy via influence function. In ACM Web Conference, 2023.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning, arXiv preprint arxiv:2402.0 04333, 2024.
Jie Xu, Zihan Wu, Cong Wang, and Xiaohua Jia. Machine unlearning: Solutions and challenges. IEEE Transactions on Emerging Topics in Computational Intelligence, 2024.
Haotian Ye, Chuanlong Xie, Yue Liu, and Zhenguo Li. Out-of-distribution generalization analysis via influ- ence function. arXiv preprint 2Tfe:22011.852/ 2021.
Jieyu Zhang, Haonan Wang, Cheng-Yt Yu Hsich, and Alexander J Ratner. Understanding programmatic weak supervision via source-aware influence function. Advances in Neural Information Processing Systems, 2022.
Yang Zhang, Zhiyu Hu, Yimeng Bai, Fuli Feng, Jiancan Wu, Qifan Wang, and Xiangnan He. Recommenda- tion unlearning via influence function. arXiv preprint or orXi:23077.021 2023.
15

---

Published in Transactions on Machine Learning Research (03/2026)
A
Additional Experimental Results and Analysis
Training
Validation
x10
Influence Estimate
8
B
6
2
Linear-Fairness Fairness
Inner Product
Feature
Feature
Linear-Fait
2
0
-
-2
-4
-2
0
2
6
2
0
4
8
2
0
2
Feature
Feature
Influence Value
x10
8
6
6
16
Linear-Robumie Jinea-RRbuuness Robutness
4-
Inner Product
2-
Feature
2 Somnes
0-
-
-4
4
-16
4
2
0
2
6
2
0
4
6
8
16
0
16
Feature
Feature
Influence Value
Pos. Class
Pos. Majority
Pos. Minority
Pos. Detr. Majority
Pos. Detr. Minority
Neg. Class
Neg. Majority
Neg. Minority
Neg. Detr. Majority
Neg. Detr. Minority
Figure 5: Illustrating our inner product approach on measuring fairness and robustness. A and D illustrate the same 2D lincarly separable synthetic dataset in Figure [ trained using a Logistic Regression model for binary classification, where the solid and dashed point boundaries denote the majority and minority subgroups. B and represent the validation set. C and show the estimated influence on fairness and robustness by the traditional influence function and our IP method.
A.1 IP for Measuring Fairness and Robustness
Similar to Figure ! conduct the experiments based on Eqs. 3 and (4 to demonstrate the effectiveness of
our IP in measuring the sample influence on fairness and robustness. Figure 5 shows the relationship bet ween IP and the traditional influence function, indicating IP is good surrogate of the traditional influence function in measuring fairness and robustness.
Table 3: Performance of IP Ensemble with different rates of removed samples on CIFAR-1ON-a
Method
Ensemble Size
Remove Rate
ACC
Cross Entropy
5
91.62
IP Ensemble (Ours)
5
0.025
92.29 + 0.16
IP Ensemble (Ours)
5
0.050
92.26
0.19
IP Ensemble (Ours)
5
0.075
92.50
+
IP Ensemble (Ours)
5
0.100
92.37
+ 0.15
16

---

Published in Transactions on Machine Learning Research (03/2026)
Table 4: Performance of different dropout rate and ensemble size of IP Ensemble on CIFAR-IONE CIFAR-
10N-r, CIFAR-ION-u, and CIFAR-10ON
Ensemble Size
Dropout Rate
CIFAR-ION-a
CIPAR-ION-r
CIFAR- WO
CIFAR-10ON
5
0.01
92.26
91.28
86,50 0.35
62.25
0.1
92.26 0.19
91.28 = 0.29
86.50 = 0:35
62.25 + 0.54
5
0.5
92.26 0.19
91.28 0.29
86.50
0:35
62.25 0.54
0.01
92.42 0.17
90.82 0.08
86.31 + 0:35
60.59 + 0.20
5
0.01
92.26
91.28
86.50 0.35
62.25
10
0.01
92.58 0.04
91.32 = 0.29
86.89
+ 0:37
61.82 1 0.61
15
0.01
92.27 0.09
91.27 0.28
86.47
0.41
61.59 4: 0.34
0.01
92.41 0.15
91.33 # 0.26
86.65 4: 0.16
62.25 4: 0.11
Table 5: Accuracy results of influenco-basek methods with ViT and MLP-mixer as based models on CIFAR1ON-a, CIPAR-10ON, and Animal- 10N with 5% detrimental samples removed.
Methods / Datasets
CIPAR-ION-a
CIPAR-10ON Animal-
Avg.
VIT
81.87
48.53
76.20
68.87
LiSSA
Koh & Liang
2017
81.83 0.19
48.77 = 0.24
76.90 # 0.36
69.07
Tracin
Pruthi et al.
2020
81.59 0.21
48,18
0.28
76.21
0.58
68,66
EKFAC
Grosse et
2023
80.79 0.56
48.90 = 0.30
76.85 :: 0.30
68.85
DataInf Kwon et al.
2023
81.68 0.21
19.00 0.22
76.70 0.27
69.13
Self-Tracin
TThakkar
2023
81.88
19.25 0.21
76.95 #
69.36
LiSSA [Bejan et al. 2023
82.89 0.12
49.30 = 0.18
76.80 = 0.28
69.66
TDA Bae et al.
2024
81.89
49.05 0.54
76.44
69.12
GEX Kim al.
2024
82.99 + 0.24
49.78 + 0.20
76.88 # 0.29
69.88
IP (Ours)
81.93 0.22
49.12 = 0.20
76.80 :: 0.22
69.28
IP Ensemble (Ours)
81.96 0.13
51.01 0.17
77.20 0.24
70.06
MLP-Mixer
74.42
35.63
72.69
60.91
LiSSA (ohh & Liang
2017
75.35
36.50
72.89
61.58
TracIn Pruthi et al.
2020
74.43 0.28
35.21 + 0.35
72.71
+ 0.43
60.78
EKFAC Grosse et 2023
75.49 0.28
36.34 0.24
73.12
61.65
DataInf
Kwon et al.
2023
75.10 0.46
35.78 = 0.29
73.17
= 0:21
61.35
Traeln Thakkar et al.
2023
75.88 0.21
36.72 0.19
73.81
0.54
62.14
Seff-LiSSA (Bejan al.
2023
75.41 0.38
36.05 # 0.27
72.56
1: 0.34
61.34
TDA Bae et al.
2124
74.89 0.31
36.84 = 0.38
71.88
0.43
61.20
GEX Kim
2024
75.57 0.27
36.85 0.25
73.89
0.43
62.10
IP (Ours)
75.04
36.15
73.47
61.55
TP Ensemble (Ours)
75.77 + 0.19
37.12 0.18
73.80 + 0.11
62.23
A.2 Performance of IP Ensemble with different rates of removed samples
In our paper, we explore different rates of removed samples on CIPAR-1ON-a. Table shows the performance of IP Ensemble with different rates of removed samples. Except for not removing, there is no significant difference the remaining results.
A.3 Parameter Analysis Dropout Rate and Ensemble Size
Table 4 shows the performance different dropout rates and ensemble sizes of IP Ensemble. As can be observed, our IP Ensemble is not sensitive to dropout rato, and its performance increases with a large ensemble size, demonstrating the effectiveness of the ensemble strategy. is also worth noting that although our IP Ensemble runs fast, it might take longer time to calculate the sample gradient.
17

---

Published in Transactions on Machine Learning Research (03/2026)
A.4 Performance ViT and MLP-mixer
To verify the effectiveness of our IP ensemble across different network prchitectures, conducte experiments using ViT (Dosovitskiy 2020) and MLP-Mixer (Tolstikhin et al. 2021) on various vision datasets in Table 5. We use a batch size of 512 for all experiments this section. The models are trained on CIFAR-1ON and CIFAR-100N datasets for 100 epochs, and on Animal-10N for 400 epochs. The learning rate was set to 10-4 for ViT and 10-3 for MI.P-Mixer.
Our IP Ensemble consistently demonstrates superior performance compared to the vanilla models and other baseline methods different base models. Specifically, when trained on ViT, IP Ensemble achieves average accuracy of 70.06%, outperforming the vanilla. ViT's average accuracy of 68.87%. This improvement is observed across all datasets, with notable gains on CIFAR-1OON (51.01% versus 48. 53%) and Animal-10N (77 20% versus 76.20%) Additionally, compared to baseline methods, our IP Ensemble achieves the best performance with VIT or MLP-Mixer based models, indicating the effectiveness of our methods on different basc models.
A.5 Time Complexity and Execution Time of Various Influence Function- based Methods on Vision Datasets
Table 6: Computational complexity and runtime (seconds) of function- based methods (n is #train- ing samples and is #model parameters, & is #checkpoints #ensemble size with k=1 here. denotes no runs.)
Method
Type
Time Complexity
CIFAR-ION CIPAR-10ON Animal-100
Exact by Eq.
Tessian-basee
O(up")
LISSA
Koh & Liang
2017
Hossian-based
O(np)
7.67
34.59
7.46
TracIn
Pruthl et al. 2020
Hessian-free
O(npk)
0.28
0.03
EKFAC
GGrosse et a. 2023
Hessian-based
O(np²)
22.54
192.58
23.89
Datalnf
Kwon et al. 2123
Hessian-baset
O(np)
11.50
45.29
10.89
Self/Tracln
(Taakar
2023
Self-influence
((npk)
0.15
1.41
0.15
Self-LiSSA
Bejan et al.
2023
Self-influence
((np)
14.57
54.39
14.40
GEX
Kim et al. 2024
Hessian-free
(mppk)
0.03
0.28
0.03
TDA
Bae et 2124
Hessian-based
(mp²k)
23.98
193.79
24.77
Ensemble (Ours)
Hessian-Fii
O(npk)
0.28
0.03
Table i shows the time complexity of various influence function-based methods. Except for the vanilla
calculation, all other methods have linear time complexity terms of the sample size. However, they have large divergence real execution time.
We omit the specific timing details for the sample gradient, which are readily available during the base model's training phase. Besides, utilizing vmap in Pytorch, we efficiently compute gradients in parallel. For example, in Section 5 it only takes 61 seconds and 4 seconds to calculate Vuutil and Vgl(z; 0) CIFAR-10N datasct, respectively.
18


Reviews:
[{"review_id": "mmcQesEz4B", "reviewer": "Reviewer_PHV7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Influence functions, an area of research which dates back at least forty-five years in statistics, aim to investigate how much a single data point affects a model's learned parameters. As the authors note, in deep learning, no closed-form solution is guaranteed to exist, and computing the inverse of the Hessian matrix, the most direct approach, is costly. In this work, the authors revisit a Hessian-free method popularized by TracIn, which substitutes the inverse of the Hessian matrix with an identity matrix. They then improve it via ensembling, and conduct experiments on their solution in a variety of settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "* The paper provides a lot of experimental evidence in general, which is good, and the idea of getting some beneficial variance reduction via ensembling and dropout is sound and, naturally enough, works well in practice.\n* I do not find that, in Section 3, the authors actually explain why \"such a naive approximation works well in practice\". As the authors note, \"choosing the regularization parameter λ presents a tradeoff: a small λ may not guarantee the existence of the inverse, while a large λ makes the inverse approximate the identity matrix.\" In this case, the theory suggests that Hessian approximations should be better than or the same as IP, unless the effects of approximation error are expected to be worse than the absence of the additional information. To convince otherwise, the authors would have to show that such a large approximation error is likely to occur in practice.\n* A similar confusion surrounds the authors' discussion of TracIn. TracIn is very similar to IP, except that the authors consider only the final checkpoint. TracIn does something which, if influence functions behave as expected, should improve the estimation -- namely, by summing contributions at all available checkpoints. Yet the authors write, \"While it seems appealing to consider the dynamic sample influence change according to the optimized model parameters, simply summing these simple influence scores along different checkpoints fails to account for the evolving nature of influence, potentially neutralizing conflicting values and reducing overall effectiveness, which can be verified in our experimental results\". For a task like noisy label detection, where it mostly matters which samples are detrimental to the final model, perhaps considering only the final checkpoint works best. And indeed, Self-TracIn (which is also single-checkpoint, per Table 1), actually beats plain IP. But it isn't clear why that phenomenon should be expected to generalize to all tasks.", "audience_yn": "Yes", "audience_explanation": "The machine learning research community has historically shown considerable interest in the study of influence functions, as the authors note. Furthermore, I see no evidence that demand for this area of research is slackening. Therefore, I think some individuals in TMLR's audience will likely be interested.", "requested_changes": "I think it would be advisable for the authors to consider my comments as to the `accurate, convincing and clear evidence` portion of this review.\n\nP3 typo: calcualtion", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ersNZdAIG0", "reviewer": "Reviewer_Jv6o", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate Hessian-free approximations for Influence Functions (IFs), specifically evaluating the substitution of the inverse Hessian matrix with the identity matrix. They argue that this simplification significantly reduces computational overhead without compromising estimation performance. Furthermore, they propose that ensembling multiple models can further enhance the quality of IF estimations. Beyond traditional data attribution, the authors extend this framework to quantify model behavior in the contexts of fairness and adversarial robustness.", "claims_and_evidence_yn": "No", "claims_explanation": "While the authors claim to “explain the underlying reason” why the Inner Product (IP) outperforms inverse Hessian-based Influence Functions, the provided explanation is somewhat superficial. The paper would benefit from a more rigorous investigation and a deeper analysis to fully substantiate this claim. Specifically, the authors do not provide a rigorous theoretical analysis comparing the two key approximation errors: the gap between the IP and the ideal IF, versus the gap between the estimated inverse Hessian IF and the ideal IF. Without analyzing which error bound is smaller, the claim that IP is superior remains unsubstantiated.", "audience_yn": "Yes", "audience_explanation": "In the context of data-centric machine learning, identifying high-quality data is a critical task. While Influence Functions (IF) are a powerful data attribution tool, they are often hindered by prohibitive computational costs. This paper addresses these scalability challenges, making it a topic of interest to the broader research community.", "requested_changes": "My primary concern involves the claim that the inverse Hessian can be replaced by the identity matrix. As this substitution represents the central contribution of the paper, I believe a more rigorous and deeper analysis is required to justify its validity.", "broader_impact_concerns": "There are no significant broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jihCIcPjfV", "reviewer": "Reviewer_DQj1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work investigates the use of influence functions to enhance machine learning model performance by refining training data quality. In contrast to standard methods that rely on the Hessian matrix, the author proposes a new approach that replaces the Hessian with an identity matrix. This modification significantly reduces the computational complexity in high-dimensional spaces. The author provides both simulation results and real-world experiments to demonstrate the efficiency of this method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The author provides detailed experimental results to demonstrate the efficiency of the inner-product method across several domains, including fairness and robustness, including noisy label detection, sample selection for large language model (LLM) fine-tuning, and defense against adversarial attacks.", "audience_yn": "Yes", "audience_explanation": "The paper provides a computationally efficient alternative to the Hessian bottleneck in influence functions, offering a scalable heuristic that is highly relevant to TMLR’s audience in interpretability and data valuation.", "requested_changes": "1: Section 4 contains redundant descriptions of the experimental setup. Specifically, the descriptions of the Logistic Regression model on linear data and the three-layer MLP on the half-moon dataset appear multiple times.\n\n2: The synthetic experiments in Figure 1 lack realism because the validation set is noise-free. To better simulate real-world conditions, the author should evaluate the method using a validation set with a subset of flipped labels (e.g., 10%). Furthermore, including a plot that utilizes the actual Hessian matrix—rather than the identity matrix approximation—is essential to visualize and quantify the approximation error.\n\n3: For the real-world experiments, the inclusion of a baseline using the exact Hessian-based influence function is highly recommended. Comparing the proposed Inner-Product (IP) method against the standard Hessian method would clarify which performance discrepancies arise from the IP approximation itself versus the inherent limitations of influence functions.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OrDl3pf3nH", "reviewer": "Reviewer_jUHb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies data-centric learning and considers a Hessian-free approach to influence functions. In the estimation of a training sample's influence, the inverse Hessian matrix is replaced with an identity matrix. The resulting gradient \"Inner Product\" (IP) gives a simple similarity measure that can be more effective and scalable than computationally expensive methods for deep learning models. The authors also extend this method to improve model fairness and adversarial robustness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the authors do extensive experiments to back their claims. I do have some comments/questions on them - see below.", "audience_yn": "Yes", "audience_explanation": "Yes, as suggested by the numerous papers from recent ML papers across venues.", "requested_changes": "### Section 2\n- In the \"Miscellaneous\" para, the discussion seems very unrelated across sentences. How are these different works related to the work in this paper? And would making them chronological help?\n\n### Section 7\n- Since the attacker is sometimes attacking more samples than those removed by influence functions, there should come a point at which these methods will do worse than pre-attack. Can you show some results about the levels of attacks when different methods become worse than \"pre\"?\n\n### Generic\n- I would love to see some experiments where IP does significantly worse than Hessian-based approaches, to highlight some limitations of when IP doesn't work.\n- See more comments below.", "broader_impact_concerns": "n/a", "additional_comments": "### Section 2\n- Page 3, first para - you mean \"fairness 'in' ML\" right?\n\n### Section 3\n- Why -ve sign in $z_i$ in (1)?\n- Page 4, 2nd para, last sentence - do you mean 'exceed the \"disadvantages\" of simply'? Omitting the Hessian cannot have advantages - at best the resulting errors are manageable.\n- In (3), why is there $\\hat{\\theta}$ in the subscript of the first gradient? Also, the order of model and data in $f$ and $\\ell$ are reverse. Can you make it consistent?\n- For IP, as you state in para 2 of page 4, \"A high IP score indicates that the target training sample is similar to the validation set\". However, what is the intuition for IP in (4)? If gradient of a sample $z_i$ is aligned with the gradients at the perturbed samples, it is good because then you can move in the direction $-\\nabla \\ell(z_i; \\hat{\\theta})$ and increase robustness - is that the reasoning?\n\n### Section 4\n- Can make it more concise. E.g., several sentences in the first para are repeated.\n\n### Section 5\n- Clarification question: in the 5 retraining iterations, do you look for 5% detrimental samples from the full training dataset each time? Or does the dataset keeps shrinking each time by 5% further after each retraining? Or is it the case that you remove detrimental samples only once, and retrain with the remaining 95% dataset for 5 independent runs? Also, it's not clear from caption of Table 1 that 5 retraining iterations happen.\n- Page 8, line 2: what are the \"different levels\" of detrimental samples?\n- Table 1: for all the datasets, the performance of IP Ensemble has overlapping confidence intervals with some other method. Does it have any significant utility then? Also, I don't see the benefit of the last column. Maybe averaging CIFAR-10N's across different noise levels would still make sense.\n- In Table 6, what are the units? Seconds?\n\n### Section 6\n- How does IP (without ensemble) perform in Fig 3?\n- When removing the detrimental samples according to both utility and fairness, do you remove the 5% worst samples for utility AND 5% worse for fairness? Or some other way? Please clarify.\n\n### Section 7\n- The dashed brown line in Fig 4 - what is the average over: the 10 attacks? And why is LiSSA so much worse on CelebA with relabeling?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)