Towards Robust Influence Functions with Flat Validation Minima
Authors: Xichen Ye
Yifan Wu
WEIZHONG ZHANG
Cheng Jin
Yifan Chen
Venue: ICML 2025 poster (2025)
Keywords: Influence Function
Primary area: deep_learning->robustness

Abstract: The Influence Function (IF) is a widely used technique for assessing the impact of individual training samples on model predictions.
However, existing IF methods often fail to provide reliable influence estimates in deep neural networks, particularly when applied to noisy training data.
This issue does not stem from inaccuracies in parameter change estimation, which has been the primary focus of prior research, but rather from deficiencies in loss change estimation, specifically due to the sharpness of validation risk.
In this work, we establish a theoretical connection between influence estimation error, validation set risk, and its sharpness, underscoring the importance of flat validation minima for accurate influence estimation.
Furthermore, we introduce a novel estimation form of Influence Function specifically designed for flat validation minima.
Experimental results across various tasks validate the superiority of our approach.

Towards Robust Influence Functions with Flat Validation Minima
Xichen Ye Yifan Wu? Weizhong Zhang
Cheng Jin 13
Yifan Chen
Abstract
Sharp Minima
R
The Influence Function (IF) is a widely used tech- nique for assessing the impact of individual train- ing samples on model predictions. However, ex- isting IF methods often fail to provide reliable influence estimates in deep neural networks, par- ticularly when applied to noisy training data. This issue does not stem from inaccuracies in parame- ter change estimation, which has been the primary focus of prior research, but rather from deficien- cies in loss change estimation, specifically due to the sharpness of validation risk. In this work, we establish a theoretical connection between in- fluence estimation error, validation set risk, and its sharpness, underscoring the importance of flat validation minima for accurate influence estima- tion. Furthermore, we introduce a novel estima- tion form of Influence Function specifically de- signed for flat validation minima. Experimental results across various tasks validate the superiority of our approach.
Flat Minima
Large Gap
x-[
Small Gap
40
49
(a) Sharp Minima
(b) Flat Minima
Figure I. Illustration of flat minima. R-axis indicates the risk, while I and I denote the estimated and actual influence (increase in empirical risk after deviating from minima), respectively. 0 refers the model parameters, and Ae represents the parameter change. The orange arrows indicate the first-order or second- order approximation used to estimate the influence, As shown, the estimation gap is smaller in flat minima, leading to more reliable influence (marginal increase risk) estimation.
2020). These issues degrade model performance, making it crucial to understand how training data and trained mod- els interact (Hammoudeh & Lowd, 2024). One key aspect of this interaction is assessing the influence of individual training samples on model predictions (Koh & Liang, 2017).
1. Introduction
A widely used approach for this purpose is the Influence Function (IF) (Koh & Liang, 2017), which provides insights into the predictions of Deep Neural Networks by estimating the impact of individual training samples. Specifically. IF estimates the change in the empirical risk on the validation set (or loss on a specific validation sample) resulting from small perturbations to the training set (e.g., the removal of single training sample), using estimated parameter change a proxy. By serving for the computationally expensive leave-one-out (LOO) re- training approach (Cook & Weisberg, 1982), IF enables the identification of the most influential samples in the training set of deep networks without retraining for each sample. Re- cent studies have successfully leveraged IF across various tasks, demonstrating its strong potential in practice (Pruthi et al., 2020; Han al., 2020; Kong al., 2022; Kim et al., 2023; Kwon et al., 2024).
Training data is the fundamental component of machine learning, and its quality is critical for ensuring accurate and reliable predictions (Algan Ulusoy, 2021; de Vargas al., 2023). As large-scale foundation models, such large language models (LLMs) (Brown et al., 2020) and text-to-image generative models (Rombach et al., 2022), be- come more prevalent, the demand for vast amounts of data to achieve high performance has grown exponentially. How- ever, datasets- esspecallyy those sourced from the internet or generated by models-are are often uncurated and may contain quite few anomalous or mislabeled samples (Pleiss al.,
efficient surrogate
Fudan University *Shanghai Key Laboratory of Intelli- gent Information Processing IInovaatoon Center of Calligra- phy and Painting Creation Technology, MCT. China 'Hong Kong Baptist University. Correspondence to: Weizhong Zhang < weizhongzhang wawnnng@@uuaadnunnn... Yifan Chen <yi- fane @ cdu.hk..
Despite its broad applicability, existing Influence Func- tions often struggle when applied to noisy training data, posing a significant challenge since IF is supposed to dis- tinguish high-quality data from noisy or anomalous sam-
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Towards Robust Influence Functions with Flat Validation Minima
ples. Specifically, we observe that regardless of whether first-order (Pruthi al., 2020) second-order (Koh & Liang, 2017) approximation is used to estimate parameter change, the estimated influence tends to be ineffective once the model has converged on the noisy training set, as illus- trated in Figure 2. This suggests that the limitation does not originate from inaccuracies in parameter change estimation, such as errors in inverse Hessian approximation. which have been the primary focus of previous studies (Koh & Liang, 2017; Kwon et al., 2024). Instead, we provide novel under- standing that the issue lies in the estimation of loss change, particularly the impact of local sharpness in the validation risk. Figure/ provides an intuitive visualization of how local sharpness affects Influence Function estimation. As shown, even with a perfectly estimated parameter change AO, sharp local region can still introduce a substantial gap between the estimated and actual influence, ultimately undermining the reliability of influence estimation.
2. Preliminaries
Consider prediction problem from some input space X' to an output space V. We are given an independently and identically (i.i.d.) sampled training dataset Sr = {zn
(In, Vv) X' x ) Empirical Risk Minimization (ERM) solves the following problem:
0 arg min 1 El(nn
(I)
where is the sample-wise loss function.
Given validation set Syal {am : (am: Bm ) the Influence Function (IF) was introduced to answer how small perturbation e to training sample Σmn, specifically its removal, affects the prediction on a validation sample Zyat-
Intuitively, IF approximates the excess loss E(Eva: 0%2) ((zval 0*) on a specific validation sample 2val, where 6*. denotes the minimizer of the risk after perturbing the em- pirical risk by assigning more weight to a certain training
To this end, we introduce a second-order- IF approx- imation on flat validation minima. This is motivated by both theoretical intuition (Figure 1) and empirical evidence (Figure 2), which highlight the critical role of flat valida- tion minima in achieving precise influence estimation. We first establish theoretical connection between the influence estimator and the validation set, recognizing the essential role of flat validation minima in achieving accurate influ- ence estimation. Furthermore, we find that standard IF estimation becomes ineffective in optimization settings with flat minima (Figure 3). Our analysis attributes this limi- tation primarily the zero-mean gradients in converged models and the misalignment in parameter change estima- tion (see Section 3.4 for more details). Considering this, we introduce a novel form of IF specifically designed for flat minima. This approach employs a second-order ap- proximation to minimize the impact of vanishing gradi- ents and introduces a refined parameter change estimation method tailored for flat test minima. We release the code at: https: / gi thub. www Wtw.//ol/rII-EE//
sample En E Sy:
0t, arg min 1
Elnn, cl(2rr 0), N
Σ
(2)
where 2n Su is the training sample. More specifically, the IF can be understood as a two-step approximation, as outlined below.
The first step addresses the parameter change 6%, 8*. Following Koh & I Liang (2017), this parameter change can be approximated by a Newton step:
-- - -el 9a..
where 9x. = Vere(=u: 0*) represents the gradient of the loss with respect to 8* for the training sample ärr, and Hr = + En-1 Vg. computed over the training set Su, which is assumed to be positive definite (PD).
(z, 0*) denotes the Hessian matrix
We summarize our contributions as follows:
@ The second step estimates the change in the loss (Zvva, ( 0*) of validation sample žyal with respect to the parameter change 0* - l* Using a first-order ap- proximation, Koh & Liang (2017) approximate this change as:
We recognize the diminished effectiveness of influence estimators applied to training data containing noise and in optimization with flat minima.
We establish general connections between the influence estimator and the validation set. We further devise a novel influence estimator specifically designed for network optimization with flat minima.
E(Zval, 02,)) E(vval: 0*) = (02, 00),
(4)
where Izwı = Vo- E(Eval, 0+).
Combining the two-step approximation, the Influence Func- tion (IF) of the training sample zn on the validation sample Zval is defined as:
Experimentally, we evaluate our proposed method across various tasks, demonstrating its superior per- formance.
I(zui 2val) == 9Im H 9zw
where the term -e is omitted, as we are specifically inter- ested in the removal effect of a.. This removal corresponds
2

---

Towards Robust Influence Functions with Flat Validation Minima
e ===- R. making - small positive constant that does not affect the relative influence.
80-
80
Moreover, the Influence Function can also be extended to measure the overall effect on the full validation set Syal as follows:
70
AUC (56) (%)
Accuracy (%)
60
1 M
I(zri Sval)
92m
H Dzw>
(6)
550
50
Validation Set Accuracy
Influence Estimation Performance
40
where 2m Ssal represents validation sample. This aggre- gated measure quantifies the impact of removing the training sample zur on the overall validation set risk. For clarity, we refer to the IF defined in Equation (6) as the conventionally "standard" Influence Function from now on.
(second order parameter change)
Influence Estimation Performance (first order parameter change)
100
Training Epochs
Based on the definition of Influence Function, a train- ing sample Zir is said to have positive influence Zval if [Inn Zur zva)) = 1, meaning that the removal of zu increases the loss value for the validation sample Eval- Conversely, For has a negative influence on &val if (gn mvl)) = -1, as its removal reduces the loss value for Zval. This concept of positive and negative influ- ence can be generalized to evaluate the effect of 2r on the entire validation set Sya by considering the aggregate in- fluence defined by I(zi, Sval). We note the sign choice is arbitrary in the literature (see remark in Appendix G.1),
Figure 2. Validation set accuracy and influence estimation per- formance (measured by ROC AUC) for the mislaheled samples detection task across training epochs. The orange and green curves represent the performance of the Influence Function in identify- ing mislabeled samples, which assumed to have a negative influence on the clean validation set. Influence is estimated using LiSSA (Koh & Liang, 2017) for second-order parameter change approximation and TracIn (Pruthi et al., 2020) for first-order pa- rameter change approximation. As shown, influence estimation performance is highly correlated with validation set accuracy, re- gardless of whether the parameter change estimated using a first-order or second-order approximation. The experiment is con- ducted on the CIFAR-ION (Wei al., 2022) dataset under the "worst" setting. For further details, see Section 4.1.
up to the definition of loss change. Along this paper, we stick to the terminology above.
3. Flat Validation Minima for Influence Estimation
gests that for an influence estimator to perform effectively, it should relate model that minimizes the risk the vali- dation set (as model minimizing the population risk is inac- cessible). Following this motivation, in the next subsection we explore the relationship between influence estimation and validation set performance.
In this section, we first present the motivation behind a pre- liminary experiment reported in Figure and Section 3.1. We then establish the theoretical link between influence estimation and validation set performance, demonstrating that the influence estimation error is upper-bounded by both the validation set risk and its sharpness (Section 3.2). Next, we show that the standard Influence Function, as defined in Equation (6), can be ineffective when applied to a flat valida- tion set minima (Section 3.3). Finally, we introduce a novel Influence Function designed to overcome this limitation (Section 3.4).
3.2. Bound Influence Estimation Error from Above
To analyze the relationship between influence estimation and validation set performance, we begin by characterizing the performance of influence estimation from the perspective of risk minimization.
Specifically, we study whether the sign of the influence is correctly estimated. That is, given a training sample and a validation set Syat, we are interested in whether the sign of the estimated influence, sIn(I(z, S,a)). matches the true influence sign (2, Sval)): here, I" represents the target influence accurately capturing the true risk change associated with perturbations the training sample. Based this concept, we introduce the following definition for the error of influence estimation.
3.1. Motivation
To clearly show the motivation for our investigation, we conduct a preliminary experiment on the mislabeled sample detection task. In this experiment, we track the performance of influence estimation alongside validation set accuracy throughout the training process. The results are presented in Figure 2.
As observed, the performance of influence estimation is highly correlated with validation set accuracy. This sug-
Definition 3.1 (Influence Estimation Error). Given an in- fluence estimator I, target influence It, a validation set
3

---

Towards Robust Influence Functions with Flat Validation Minima
Ska and underlying distribution D, we define the random event:
Corollary 3.3 (Upper Bound Empirical Influence Esti- mation Error), Given influence estimator I, target inftuence estimator I , validation set Syal- and set S = {zn In-1 where each sample 2n is i.i.d. drawn from an underlying distribution D. If the assumptions in Theo- rem 3.2 hold, then with probability at least 1 - 6, the em- pirical influence estimation error Eg(I) is upper bounded by:
A(z) == (sgn (Z(z, Seal) A sgn (I* Sea)))
(7)
The influence estimation error is defined as:
E(I) = EewD [Lar)]] = PAWD [A(z)],
(8)
where (.) is the indicator function. Moreover, given a finite sample set {2n 1 the empirical influence estimation error of I is defined as:
2p² PC 10)
log 8 2N
EI(I) exp
(13)
1 EI( = ->>
The proof of Theorem 3.2 and Corollary 3.3 are provided in Appendices D.1 and D.2, respectively.
Σ [A(za)
(9)
3.3. Flat Validation Minima via Sharpness-Aware Minimization
Definition 3.1 provides framework for quantifying the quality of influence estimator. Based this, we establish the following influence estimation error bound.
Theorem 3.2 and Corollary 3.3 suggest that, under mild assumptions, the risk of influence estimation error can be effectively controlled by managing Rnm This implies that influence should be computed on 0, which is obtained by solving the following optimization problem:
Theorem 3.2 (Upper Bound on Generalization Influence Es- timation Error). Given an influence estimator I certain function space H. a target influence estimator It, a fixed validation set Syal, and underlying distribution D. We then condition the random variable ~ 'D on the influence signs to induce two mixture components of the distribution D:
== arg min (0),
(14)
where y is fixed constant, and (2) defined as:
D+ 1==P==P- [ (1.) D_ = P. (1.8)))=-1-
R (0) == max Real + A)
(15)
TAllST
If the following conditions hold:
lo +A (0) + (0)
max |A
EznD_ [I, Suma) > and EnND_ [I2 Spat)] < 0,
(10)
sharpness
then the generalization influence estimation error E(I) is upper bounded by:
(16)
While Equation (15) differs from the standard objective of Empirical Risk Minimization (ERM), this optimization problem can be effectively addressed using existing tech- niques.
2/²
E(I) S exp
(11)
127 1))
Specifically, this formulation naturally aligns with Sharpness-Aware Minimization (SAM) (Foret et al., 2021), learning paradigm distinct from ERM. Instead of directly minimizing the empirical risk, SAM simultancously min- imizes both the loss value and the loss sharpness, aiming to find a flat minima. Intuitively, a flat minima leads to a model with better robustness and generalization ability.
where μ = inf (Z."" BEND' (2z,))aa) and
(0)( (0) = max Ro + A);
(12)
|AISS
with Sup 1 011, where 0 is the model parameter.
Remark. The assumptions EgwD. (vai)
and [I(z, E Swi)] < 0 are fairly mild, as they simply quire the overall influence estimation performance to be better than random guessing.
However, empirically, we find that the standard Influence Function, as defined in Equation (6), can be ineffective when optimizing Equation (15). To highlight this, we track the estimated influence during the optimization of R (0), with the results shown in Figure 3(a). As illustrated, the performance of the standard Influence Function deteriorates as (0) decreases.
Building on Theorem 3.2, the result can be readily extended to the case where we only have finite samples from D. This extension formalized the following corollary.
4

---

Towards Robust Influence Functions with Flat Validation Minima
in estimating both 1 the parameter change and @ the loss change.
Rink
Performance (95)
10-
08
1 For parameter change, the standard IF assumes that the model parameter minimizes the risk on the training set. However, this assumption does not align with the flat valida- tion minima framework, as the current minima are derived from the validation set rather than the training set. This misalignment introduces a discrepancy that inherently com- promises the accuracy of parameter change estimation.
0.6
10-
-04
Stt-card Proposedrii
0.2
200
409
6/0
600
1000
Tuning steps
(a) Influence Estimation Performance Across Tuning Steps
@ For loss change, the standard IF utilizes first-order approximation term for estimation. However, previous studies (Kim et al., 2023) have shown that the gradients of pre-trained neural networks can approximated as a Gaussian distribution centered zero. This property natu- rally causes the estimated influence, Es,es, (Zr, Smi)] Es,. 9 Hu 9z..). to approach zero. Consequently, this leads to /a - 0, which severely limits the effectiveness of the standard estimation.
207
Edtimates Intuence
107
20:
10"
-30
-197
200
202
402
500
733
932
902
2000
Tuning steps
Proposed Influence Funtion. To address the two aforemen- tioned limitations, we introduce a novel Influence Function incorporating newly designed terms for parameter change and @ loss change.
(b) Box Plot of Estimated Influence for Clean Samples Using the Standard IF Across Tuning Steps
Figure 3. (a) The influence estimation performance, measured by ROC AUC, for the mislabeled sample detection task across tuning steps. The standard influence is estimated using LiSSA (Koh & Liang, 2017), with SAM (Foret et al., 2021) employed as the flat minima solver. As shown, the performance of the standard Influ- ence Function decreases ( decreases, while our proposed Influence Function shows an improvement. The experiment is conducted on the CIFAR-IO0 -ON (Wei et al., 2022) dataset under the "worst" setting. For more details, see Section 4.1. (h) Box plot of influence for clean samples estimated using the standard Influence Function across tuning steps. As illustrated, the absolute value of the estimated influence continuously decreases.
We begin by examining the parameter change. Rather than focusing on 0%, as described in Section 2, we consider 0z a detined as:
02, == arg min max SA
Rea + A) + el(an. 0 + A). (17)
Here, 0z, denotes the perturbed parameter resulting from incorporating the training sample žur into the optimization objective, i.c., Equation (15), with small perturbation
Similar to standard IF, the parameter change 0z, - 0 can be approximated using a Newton step:
Figure 3(b) further uncovers the underlying limitations of the standard Influence Function. Specifically, dur- ing the tuning process, the estimated influence of clean samples consistently approaches zero. This behavior di- rectly relates to our analysis of the error bound in in- fluence estimation. Recall the definition of μ., where == (Zn
02. - -x -ell I...
(18)
where Je. Vol(zn, Bz.)) represents the gradient of the loss with respect to for the training sample zus and Ha = 1 EM Vel(zm. 0) denotes the Hessian matrix computed over the validation set Syat, which is assumed to be positive definite (PD). A detailed derivation is provided in Appendix E.
[REDD' [I(2, Sya)) As the es- timated influence of clean samples EE [I(z, Swa)] ap- proaches zero, observe that μ 0, which lifts the upper bound and allows influence estimation eттоr increase.
@ Next, we address the issue of previous first-order estima- tion methods via introducing second-order approximation. Consider the loss change on a validation sample Aval. and we denote the parameter change AA 0_ 0. We propose to estimate the loss change with respect to the parameter changes A using second-order approximation term:
To address this, we introduce a novel Influence Function in the following section, enabling the computation of sample influence on flat validation set minima.
3.4. Influence Function in Flat Validation Minima
Limitations of the Standard Influence Function. We identify two key limitations of the standard Influence Func- tion when applied to flat validation minima: inaccuracies
((Eval, 8z..) - ((2val: 8) ~ A0 TT Viol(zva 0)40, (19)
where I=m. Vol(zval: 0).
5

---

Towards Robust Influence Functions with Flat Validation Minima
Table 1. Area Under the Receiver Operating Characteristic Curve (ROC AUC) (%) and Average Precision (AP) (%) for identifying mislabeled samples on the CIFAR-IONA 100N datasets across different influence estimation methods. Results (mean are reported over 3 random runs. Bold indicates the best results and underline denotes the second- best results.
CIFAR-ION
CIFAR-10ON
Method
Aggre
Random
Worst
Noisy
ROC AUC
AP
ROC AUC
AP
ROC AUC
AP
ROC AUC
AP
LiSSA
59.74+2.91
1.75
59.78+2.77
45.01+22.70
65.75=0.39
68,35-+0.74
1..48+1.70
0182
LiSSA*
1.070=1.09
49.9
73.48+1.90
67.81+2.23
18,96=1146
81.31=1.32
53.95+1.55
5..54+1.08
TracIn
13.91
28.94.15.10
61.61100.74
43.83.11.79
65.74
62.91
56.13:12.51
52.34.1..22
GEX
1.22
18.01
91.111:0.53
87.57110.42
93.28
94.16.10.10
90.17:10.70
89.05:10.53
DataInf
58.50
30.53
54.50:22.32
38.15:12.73
15.49
57.14
53.69:1..35
1.
Datalnf*
6..31=1444
1..9
73.85+3.33
68.57:14.65
81.36=
83,99+1.69
10.95+1.14
03.7440.48
VM
95.18+0.115
76.31++0,21
95.92++0.10
87.35++0.45
95.88--0113
94.27++0.21
89.77+00.88
83.81
FVM
0..1410.06
79.53-10.27
96.6311.03
88.82:10.26
96.46-10.08
94.9710.11
90.8011..04
85.41
Combining Equations (18) and (19), we define the influ- ence on the flat validation minima 0 as follows:
Table 2. Top-I relaheling accuracy (%) on CIFAR-1ON dataset for different influence estimation methods. Results (mean-istd) reported over 3 random runs. Bold indicates the best results and underline denotes the second-best results.
[Zrr 2val) Izma H 92c
1 + E H1 Villvvat,
(20)
2
Method
Aggre
Random
Worst
Furthermore, this Influence Function can be extended to measure the overall effect on the entire validation set Svat!
LiSSA LiSSA*
5.28+3.11
9.04+22.70
19.32+2.59
31.33+8.97
75.87=4.54
11.47=1.71
Tracin
37.08-4.99
53.28-8.86
50.66=0.92
GEX
30.1911.995
54.03-1.311
80.35
I(2r: Seal) == D2. H val Iz.-
(21)
DataInf
5.5912.20
22.24
21.25-17.36
DataInf*
40.58
81.57-1..07
76.1910.87
A detailed derivation is provided in Appendix E.2 and de- tailed implementations are outlined in Appendix A.
VM
94.17-0.02
91.9410.08
85,01
FVM
94.63
92.46-10.07
86.09 0.38
Validation Mínima v.s. Flat Validation Minima. Our theoretical analysis suggests that flat validation minima essential for better influence estimation performance. However, in practice, directly optimizing (0) can be computationally expensive. In such cases, the validation minima 0 = arg ming (0) can serve as a surrogate for the flat validation minima. We denote our proposed In- fluence Function for Validation Minima as VM and Flat Validation Minima as FVM. In the following experiments, evaluate both methods.
identify mislabeled samples. In Section 4.2, we validate its effectivenesss in relabeling samples within a noisy training set. In Section 4.3 and Section 4.4, assess method's capability in identifying influential samples in text and im- age generation tasks. In Section 4.5, we demonstrate the significance of our designed components.
4.1. Mislabeled Sample Detection
Unlike the standard influence function, which assigns positive scores to useful samples and negative scores to harmful ones, our influence function exclusively produces positive scores. Nevertheless, the distribution of influence scores generated by our method is sufficiently distinct to identify influential samples.
Remark:
In this section, we evaluate the performance of influence ap- proximation methods in identifying mislabeled (noisy label) samples. This task is based on the assumption that correctly labeled samples contribute positively to reducing the risk on the clean validation set, while mislabeled samples exhibit adverse effects. Under this assumption, the estimated in- fluence I(2, S for mislabeled samples is expected to be lower than that of correctly labeled samples. Specifically, we treat mislabeled samples as the positive class to be de- tected and assess detection performance using Area Under the Receiver Operating Characteristic Curve (ROC AUC), and Average Precision (AP) metrics.
4. Experiments
In this section, we conduct experiments across various tasks to evaluate the effectiveness of our proposed approach. In Section 4.1, we apply our method to the mislabeled sam- ple detection task, demonstrating its ability to accurately
6

---

Towards Robust Influence Functions with Flat Validation Minima
Table 3. Top-1 relabeling accuracy (%) CTFAR-1001 100N dataset for different influence estimation methods. Results (mean are reported over 3 random runs. Bold indicates the best results and underline denotes the second-boe results.
Method
LiSSA
LiSSA*
TracIn
GEX
DataInf
DataInf*
TM
FVM
Acc
0.28+0.07
2.55+
20.11
22.41=0.46
1.2440.56
2.81
18.1340.15
7..61=0.24
We compare our proposed methods against several influence estimation approaches, including LiSSA (Koh & Liang, 2017). TracIn (Pruthi et al., 2020), GEX (Kim et al., 2023), and DataInf (Kwon et al., 2024). Additionally, we compare our methods with modified versions of LiSSA and Datalnf, computed using the checkpoint from the training process that achieves the best results, referred as LiSSA* and DataInf*. The experiments are conducted on CIFAR-1O/- 100N datasets (Wei et al., 2022). Detailed descriptions of the experimental setup can be found in Appendix B.
Table 4. ROC AUC (%) and Recall (%) for identifying influential samples in text generation tasks across different influence estima- tion methods. Bold indicates the hest results and underline denotes the second-bes best results. The results for TracIn and Datalnf were generated by executing the official implementation provided by DataInf. According to DataInf (Kwon et al., 2024), obtaining re- sults around 99. .999% is considered reasonable for this experiment.
Task
Method
ROC AUC
Recall
TracIn DataInf
94.95=1..14
70.971:5.118 18
The results, presented in Table I, demonstrate that our pro- posed methods. particularly FVM, outperform existing in- fluence estimation techniques across all noise levels and settings. The only exception is the AP of GEX on CIFAR- 100N under the "Noisy" setting. However, GEX requires extensive tuning, specifically 32 times, to achieve optimal performance, which is computationally expensive. When we limit the tuning of GEX to a single time, comparable to our method, its AP drops to 84.26, which is lower than that of our approach. These experimental results validate the superiority of our proposed methods in detecting mis- labeled samples, highlighting that optimizing for flat vali- dation minima is crucial for achieving more accurate and reliable influence estimation.
Sentence Transformations
99.58
96.18.19.33
VM
99.97--0.16
99.10+22.79
FVM
99.92+0.18 18
98.47++2.00
TracIn
78.70
26.61
Math Problems
Datalnf
99.86+0.68
97.37+6.97
(w/ Reasoning)
VM
99. 0919610.21
98. 3,822
FVM
99.99=0.07
19.38+
ment over VM. These findings underscore the critical role of seeking flat minima for enhancing influence estimation performance.
4.3. Influential Sample Identification in Text Generation
To further demonstrate the effectiveness of our proposed method, we evaluate its ability to identify influential sam- ples in text generation, following Kwon et al. (2024). The assumption for this task is analogous to that in the misla- beled sample detection task (Section 4)1): for a generated sample, we assume that training samples from the same class contribute the most to the generation, while samples from other classes contribute less. Consequently, training samples that share the same class as the validation sample are expected to have greater influence.
4.2. Training Sample Relabeling
Following up the detection task in Section 4. 1, we extend our analysis to the sample relabeling task. This task is based on the further assumption that the correct class for a training sample has the most significant influence on reducing the risk of the clean validation set. Under this assumption, given training sample ž, (Z., Vn), we define the relabeling function as follows:
Building on this intuition, we define pseudo-labeling ap- proach to evaluate the effectiveness of influence estimators. Given a validation sample zval = (Iraas Bval), the pseudo- label for a training sample zu = (Iz: yrr is defined as:
in arg max I ((In k), Svat,,
(22)
where & E (1,.... K} represents the set of possible classes.
To evaluate the relabeling performance, we compute the top- 1 accuracy by comparing the true label yn with the predicted label S..
{1, 1, if Ytr = Dval- 0. otherwise.
Tval,
(23)
The results are presented in Table 2 and Table 3. As shown, our proposed methods, VM and FVM, significantly outper- form existing approaches by a large margin. Additionally, the results on CIFAR-ION highlight the importance of flat validation minima, with FVM achieving a 12% improve-
Therefore, for each validation sample zval, we compute the influence of all training samples, {I(zn, 2 and eval- uate the performance of the estimated influence using ROC AUC and Recall metrics. For Recall, we compute the per- centage of training points with the same class as the valida-
7

---

Towards Robust Influence Functions with Flat Validation Minima
Table ROC AUC (%) and Recall (%) for identifying influen- tial samples in image generation tasks across different influence estimation methods. Bold indicates the best results and underline denotes the second-best results. The results for TracIn and DataInf were generated by executing the official implementation provided by Datalnf. denotes results reported in the original Datalnf paper.
Table 6. Ahlation study CTFAR- 1ON dataset under the "worst" setting, The results are evaluated using ROC AUC and AP. Com- ponents included are marked with a V. Bold indicates the best results.
Loss change standard
Parameter change
ROC AUC
AP
ours
standard
ours
41.52
$5,40
Task
Method
ROC AUC
Recall
33.38
47.64
95.93
91.38
TracIn
60,48-6.77
19.00-18.13
96.46
94.97
DataInf
61.13=7.10
61.84:6.71
Style Generation
TracIn
69.2=0.7
53.3:10.8
ential samples in image generation tasks, using the same
Datalnf
82.0=0.5
68. 68.7:10.6
experimental settings as described in Appendix B.2. We evaluate two tasks in text-to-image generation: (i) style generation and (ii) subject generation. For the style genera- tion task, we combine three publicly available image-text pair datasets, each representing different style: cartoons (Norod78, 2023), pixel-art (Jainr3, 2023), and line sketches (Zoheb, 2023). Further details of the experimental setup are provided in Appendix B.3.
VM
92.19- 5.25
79.85:18.62
FVM
92.03=5.27
79.62-18.61
TracIn
22.95
19.61
DataInf
57.73+233.0
30.39-26.03
Subject
TracIn'
82.0=0.0
21.0+:.33
Generation
Datalnf
86.5=0.0
31.5=0.3
VM
92.69=11.114
$2.94=31.95
FVM
92.87-110.4
55.39-30.00
The results are presented in Table 5. Our proposed methods, VM and FVM consistently achieve the best performance across both tasks. Similar to the result in Table 4, FVM does not always achieve the highest scores, with aligns with our earlier discussion in Section 4.3.
tion sample among the smallest influential training points. Here, is set to be the number of training examples per class. The mean and standard derivation of these metrics is then computed across all validation samples to summarize the results.
We compare our proposed method against TracIn (Pruthi al., 2020) and DataInf (Kwon et al., 2024). The experi- ment is conducted by fine-tuning LoRA (Hu et al., 2022) the Llama-2-11 1-B-haat model (Touvron et al., 2023) for the text-to-text generation task. We evaluate two tasks in text generation task: (i) Sentence transformations, and (ii) Math word problems (with reasoning). The detailed description for each task and dataset is given in Appendix B.2.
4.5. Ablation Study
We begin with an ablation study evaluate the effectiveness of the two key components of our proposed Influence Func- tion: loss change and parameter change. For loss change, we compare the term 92., AO used in the standard IF, referred to as "standard," with AOT Ar used in our proposed IF, referred to "ours." For parameter change, pare -CH 92, as "standard" with -cH Iz,, as "ours." We conduct experiments on CIFAR-ION dataset under the "worst" setting. The model is tuned on the validation set with SAM (Foret et al., 2021) used as the flat minima solver, and the experimental setup is consistent with the details provided in Section 4.1.
The results are presented in Table 4. Our proposed methods, VM and FVM consistently achieve the highest performance across both tasks. Specifically, we note that FVM does not always achieve the highest scores. This can be attributed to the fact that our fine-tuning process employed a simplified version of the flat minima solver, LPF (Bisla et al., 2022), using M instead of the recommended M 8 in the original work. This suboptimal choice of hyperparameters may have limited the full potential of FVM. preventing it from consistently outperforming all baselines. Nonetheless, the overall trend still demonstrates the robustness and ef- fectiveness of our method in identifying influential samples across text generation tasks.
The results are presented in Table 6. As observed, our pro- posed loss change term significantly enhances influence estimation performance, which supports our analysis high- lighting the importance of the second-order approximation. The proposed parameter change term actually reduces per- formance when combined with the standard loss change term. However, when applied alongside our proposed loss change, leads to performance improvement.
4.4. Influential Sample Identification in Image Generation
5. Conclusion
Following Kwon et al. (2024), we further evaluate our pro- posed influence estimation method for identifying influ-
In this work, we revisited the Influence Function (IF) and identified a fundamental limitation when applied to deep
8

---

Towards Robust Influence Functions with Flat Validation Minima
neural networks. particularly in the presence of noisy train- ing data. Through theoretical analysis and empirical obser- vations, we demonstrated that existing IF methods suffer due to the sharpness of the validation risk, which leads to unreliable influence estimates. To address this issue, we established a novel connection between influence estima- tion error, validation set risk, and its sharpness, highlighting the importance of flat validation minima for accurate in- fluence estimation. Building this insight, we employ second-order approximation to minimize the impact of van- ishing gradients and introduce refined parameter change estimation method tailored for flat test minima. Extensive experiments across various tasks demonstrated the superior performance of our approach, validating its effectiveness in enhancing influence estimation accuracy.
Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
Basu, S., Pope, P., and Feizi, S. Influence functions in deep learning are fragile. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Bisla, D., Wang, J., and Choromanska, A. Low-pas: filter- ing SGD for recovering flat optima in the deep learning optimization landscape. In International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event, volume 151 of Proceedings of Machine Learning Research, pp. 8299 8339. PMLR, 2022.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,
Acknowledgments
J., Dhariwal. P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J. Winter, C., Hesse, C.,, Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances in Neural In- formation Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-, 2020, virtual, 2020.
This work was supported by AI for Science Foundation
of Fudan University (FudanX24A1029), the National Natu- ral Science Foundation of China (Grant No. 62472097), the Hong Kong Research Grants Council (Project No. 22303424), and GuangDong Basic and Applied Basic Re- search Foundation (Grant No. 2025A1515010259).
Impact Statement
Chaudhari, P., Choromanska, A., Soatto, S., LeCun, Y., Baldassi, C., Borgs, C., Chayes, J. T., Sagun, L.,, and Zecchina, R. Entropy-sgd: Biasing gradient descent into wide valleys. In Sth International Conference on Learn- ing Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenRe- view.net, 2017.
This paper presents work whose goal is to advance the field of Machine Learning, specifically by enhancing influence estimation in deep neural networks. Our approach improves the reliability of influence functions, which benefits ap- plications such dataset pruning, model debugging, and fairness analysis. By helping detect mislabeled or anoma- lous samples, our methods can lead to more robust and trustworthy models. However, we note that careful interpre- tation necessary to prevent potential biases or unintended consequences when applying influence-based techniques in sensitive domains.
Cook, R. D. and Weisherg, S. Residuals and influence in regression, 1982.
de Vargas, V. W., Aranda, A. S., dos Santos Costa, R., da Silva Pereira, P. R., and Barbosa, J. L. V. Imbalanced data preprocessing techniques for machine learning: a systematic mapping study. Knowl. Inf. Syst., 65(1):31-57, 2023.
References
Algan, G. and Ulusoy, Image classification with deep learning in the presence of noisy labels: A survey, Knowl. Based Syst., 215:106771, 2021.
Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1019-1028. PMLR, 2017.
ARNOLDI, W. F. The principle of minimized iterations in the solution of the matrix eigenvalue problem. Quar- terly of Applied Mathematics, 91):117-2, 1951. ISSN 0033569X, 15524485. URL http:/ / h/www//wwooooo jstor. Ttr/sstable/336336
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In 9th International Conference on Learn ing Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Bac, J., Ng, N., Lo, A., Ghassemi, M., and Grosse, R. B. If influence functions are the answer, then what is the question? In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information

---

Towards Robust Influence Functions with Flat Validation Minima
Grosse, R. B., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., Hubinger, E., Lukosiute, K., Nguyen, K., Joseph, N., McCandlish, S., Kaplan, J., and Bowman, S. R. Study- ing large language model generalization with influence functions. CoRR, abs/2308.03222 2023.
Kim, S., Kim, K., and Yang, E. GEX: A flexible method for approximating influence via geometric ensemble. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrlPS 2023, New Orleans, LA,, USA, December 10 16, 2023, 2023.
Guo, H., Rajani, N., Hase, P., Bansal, M., and Xiong, C. Fastif: Scalable influence functions for efficient model interpretation and debugging. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Viriual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 10333- 10350. Association for Computational Linguistics, 2021.
Koh, P. W. and Liang, P. Understanding black-box predic- tions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, vol- ume 70 of P Proceedings of Muchine Learning Research, pp. 1885-1894, PMLR, 2017.
Kong, S., Shen, Y., and Huang, L. Resolving training biases via mffleence-bssed data relabeling. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Hammoudeh, Z. and Lowd, D. Training data influence analysis and estimation: a survey. Mach. Learn., 1 113(5): 2351-2403, 2024.
Han, X., Wallace, B. C., and Tsvetkov, Y. Explaining black box predictions and unveiling data artifacts through in- fluence functions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguis- tics, ACL 2020, Online, July 5-10, 2020, pp. 5553-5563. Association for Computational Linguistics, 2020.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
K Kwon, J., Kim, J., Park, H., and Choi, I. K. ASAM: adaptive sharpness- aware minimization for scale-invariant learn- ing of deep neural networks. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Confer- ence on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016.
PMLR, 2021.
Kwon, Y., Wu, E., WU, K., and Zou, J. Datainf: Efficiently estimating data influence in lora-tuned llms and diffu- sion models. In The Twelfih International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Li, B. and Giannakis, G. B. Enhancing sharpness-aware optimization through variance suppression. In Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Jainr3.
jainr3/diffusiondbb tiiiifffuusiondd--illltttt Datasets at https:
Hugging Face - huggingface.co.
//huggingface...
caaicee.co//tttee hoo/datasess//ie/ /jainr///////////
diffusionde woanddbppieeart..... 2023. [Accessed 24-09-2023).
Jinng W., Yang, H., Zhang, Y., and Kwok, J. An adaptive policy to employ sharpness-awe aware minimization. In The Eleventh International Conference on Learning Repre- sentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 6391-6401, 2018.
Kim, M., Li, D., Hu, S. X., and Hospedales, T. M. Fisher SAM: information geometry and sharpness aware minimi- sation. In International Conference on Machine Learn- ing, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 11148-1111 PMLR, 2022.
Li, T., Zhou, P., He, Z., Cheng, X., and Huang, X. Friendly sharpness-aware minimization. In IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 5631- 5640. IEEE, 2024.
10

---

Towards Robust Influence Functions with Flat Validation Minima
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Moly- bog, I., Nic, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, M., Subra- manian, R., Tan, E., Tang. B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I,, Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Sto- jnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: /aaaiiivv...//.v/./aaaaa/aaaavvv wwwpxv..rrg//o .rg. abs 207 09288.
Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., and Bossan, B. Peft: State-of-the-art parameter- efficient fine-tuning methods. sttps://githhe www.hhhggggggaeeeeeeeeeeeeeeeeee 2022.
Norod78. (mrod///caatooon-bippcaapaa at Hugging Face huggingface.co. https:
Datasets
//huggingfacce...o/////// co/ wtt.://///aaaa.. dat.setsssssss/saassassassass /Norodd8// cartoon-blip capiipppppinnss... 2023. [Accessed 24-09-2023].
Tsuzuku, Y. Sato, I., and Sugiyama, M. Normalized flat minima: Exploring scale invariant definition of flat min- ima for neural networks using pac-bayesian analysis. In Proceedings of the 37th International Conference on Ma- chine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 9636-9647, PMLR, 2020.
Pleiss, G., Zhang, T., Elenberg, E. R., and Weinberger, K. Q. Identifying mislabeled data using the area under the margin ranking. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Wei, J., Zhu, Z.,, Cheng, H., Liu, T., Niu, G., and Liu, Y. Learning with noisy labels revisited: study using real- world human annotations. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net 2022.
Pruthi, G., Liu, F., Kale, S., and Sundararajan, M. Estimat- ing training data influence by tracing gradient descent. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Zhuang. J., Gong, B., Yuan, L.,, Cui, Y., Adam, H., Dvorekk, N. C., Tatikonda, S., Duncan, J. S., and Liu, T. Surrogate gap minimization improves sharpness awwrr aware training. In The Tenth International Conference on Learning Repre- sentations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10674-10685. IEEE, 2022.
Zoheb. zoheb/sketch-scene Datasets at Hugging Face huggingface.co. https: https://hughh: htg.g/gggggagcccccooo// datasets/ wtttttsssehebbeeeeeeeeeeeee scene, 2023. [Ac- cessed 24-09-2023).
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to- image diffusion models for subject-driven generation. In Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 22500-2230 22510. IEEE, 2023.
IEEE/CVF Conference
Schioppa, A., Zablotskaia, P., Vilar, D., and Sokolov, A. Scaling up influence functions. In Thirty-Sixth AAAI Conference Artificial Intelligence, AAAI 2022, Thirty- Fourth Conference on Innovative Applications of Artifi- cial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22- March 1, 2022, pp- 3179-81866 AAAI Press, 2022.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel. D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W.,

---

Towards Robust Influence Functions with Flat Validation Minima
A. Implementation Details
Algorithm 1 Influence Function in Flat Validation Minima
1: Input: training set Su. validation set Sval, parameter pre-trained on training set 0*, training sample &ir
2:
3: # Solving for flat validation minima
4: Initialize I + 0"
5: Ø < --olving (f) (e.g. Sharpness-Aware Optimization)
6:
7: # Computing influence
8: I(ār, Sma) < J2, Hy Dau
9:
10: Output: I(2ir Sval)
Algorithm 1 presents the algorithm for computing the influence of single training sample on the validation set Seat The algorithm first optimizes for the flat validation minima before computing the influence function.
To efficiently approximate the inverse Hessian matrix, we adopt a diagonal preconditioner, defined as: Peul == diag((Fal) where
M
I2m
(24)
M
represents the empirical Fisher Information matrix computed on the validation set Sval, serving as an approximation of Hessian. Assuming that the diagonal elements dominate the empirical Fisher Information matrix, we approximate H.a by Pui using (Zr,, Sva),, resulting in:
I( Sval) ~ 92, P-1. 9
(25)
Here, P-1 is the inverse of the diagonal matrix Pval and can be easily computed due to its diagonal structure. Empirically, we find that this approximation significantly reduces computational overhead while maintaining accuracy in influence estimation.
B. Experiment Details
B.1. Mislabeled Sample Detection and Training Sample Relabeling
Our evaluation is conducted on the CIFAR-ION and CIFAR- datasets (Wei et al., 2022), which are real-world noisy label variants of the CIFAR-10 and CIFAR-100 100 datasets (Krizhevsky et al., 2009). CIFAR-1ON includes five noisy label settings: "Aggregate", "Random 1", "Random 2", "Random 3" and "Worst." CIFAR-10ON provides a single noisy label setting. "Noisy." These noisy labels arise from human annotation errors. For our evaluation, we use the "Aggregate", "Random 1" and "Worst" settings for CIFAR-1ON, and the "Noisy" setting for CIFAR-I0ON.
Datasets.
Training. We follow the experiment settings in Wei et al. (2022). The ResNet-34 (He al., 2016) is utilized for both CIFAR-1ON and CIFAR-10ON. The basic typer-parameters settings listed as follows: minibatch size (128), optimizer (SGD), initial learning rate (0.1), momentum (0.9), weight decay (0.0005), number of epochs (100), and learning rate decay (0.1 at 50 epochs). Standard data augmentation, including random crop and random horizontal flip. is applied to each dataset.
Apper-Parameter... The hyper-parameters for each influence estimation approach are outlined as follows.
For TracIn, we use five checkpoints evenly sampled from the training process, corresponding to epochs 20, 40, 60, 80, and 100.
For Datalnf, the damping parameter is set as in the original paper: À 0.1 (Nd) -1 - Vo Vo, en. where I represents the corresponding model layer and dy denotes the number of parameters in that layer.
For LiSSA and DataInf*, we select the checkpoint at epoch 60, which achieves the best AP performance among epochs 10 to 100, evaluated at intervals 10 epochs.
For GEX, we follow the procedure outlined in the original paper, generating 32 Geometric Ensemble (GE) to obtain the
12

---

Towards Robust Influence Functions with Flat Validation Minima
Table Description of the sentence transformations task prompts and templates. We consider 10 different types of sentence transforma-
tions, following the setup of Kwon et al. (2024).
Sentence transformations
Example transformation of "Sanrises heruld hopeful tomorrows".
Reverse Order of Words
tomorrows hopeful herald Sunrises
Capitalize Every Other Letter
sUnRiSeS hErAID hOpErU] 10mOrRoWs.
Insert Number Between Every Word
Sunrises Iherald hoopeful ttomorrows.
Replace Vowels with
S"nr"s"s Id h"p [*] 1 wws.
Double Every Consonant
SSunrrisseesss hheralld hhopeffull tomoorows.
Capitalize Every Word
Sunrises Hcrald Hopeful Tomorrows.
Remove All Vowels
Surss hrld hpfl tmrrws.
Add "ly To End of Each Word
Sunrisesly heraldly hopefully tomorrows.ly
Remove All Consonants
uie ocu 000.
Repeat Each Word Twice
Sunrises Sunrises herald herald hopeful hopeful tomorrows. lomorrows.
final results. Each GE is generated by tuning the trained model on the validation set, with tuning settings consistent with those used for our VM and FVM, as described below.
For our proposed VM and FVM, to obtain 0, we tune the trained model on the validation set with the following hyper- parameter settings: minibatch size (128), optimizer (SGD), initial learning rate (0.01), momentum (0.9), weight decay (0.0005), number of steps (1000), and learning rate decay (cosine). For FVM, SAM (Foret et al., 2021) is used as the flat minima solver, with the hyperparameter y set 0.05 for CIFAR-ION and 0.1 for CIFAR-100N 100N, in accordance with the original paper. In this experiment, we treat the samples with large influence scores detected noisy samples.
B.2. Text-To-Text Generation
Datasets. Following Kwon et (2024), we consider three text-to-text generation tasks: sentence transformations and math problems (w/ reasoning).
Sentence transformations: The model is tasked with applying a specific transformation to a given sentence. Further details can be found in Table 7.
Math problems (w/ reasoning): The model is presented with arithmetic word problem and is expected to provide an intermediate reasoning step and direct numerical answer before arriving at the final answer. A detailed description of the prompts and templates is provided in Table 8.
Each dataset contains 10 distinct classes, with 100 total data points each class. We partitioned the 100 examples into 90 training data points (used for LoRA) and 10 validation data points for influence estimation.
Training. We use LoRA (Hu et al., 2022) to Llama-2-13B-cha (Touvron et al., 2023). We apply LoRA to every query and value matrix of the attention layer in the Llama-2-13BB 13a-chat model. The basic hyper- parameters settings are listed as follows: minibatch size (64), optimizer (AdamW W (Loshchilov & Hutter, 2019)), initial learning rate (3 x 10-4), number of epochs (10), and learning rate decay (cosine). For LoRA hyperparameters, we set r = 8 and a = 32. The training was performed using the HuggingFace Peft library (Mangrulkar et al., 2022)
Loss. We used a negative log- likelihood of a generated response as a loss function. For a sequence of input tokens -x1,,... ;r)) and the corresponding sequence of target tokens y (Vi, Urr)), suppose the Llama-2-13B generates sequence of output tokens fo(z) (ff(x)),, fo(x)T,). fa(x) has the same size of T2 and is generated
in an auto-regressive manner. We set Ti Ta 512. Then, the loss function is Ely, fo(x)) fo(x),, fo()))-1)-
ΣT log P(y
Hlyper-Paramett The hyper-parameters for each influence estimation approach are outlined as follows.
For TracIn, we use the final checkpoint for influence computation, following Kwon et al. (2024).
For Datalnf, the damping parameter is set as in the original paper: À = 0.1 x (Nd, -1 EÑ V ETV Lo where I represents the corresponding model layer and dy denotes the number of parameters in that layer.
13

---

Towards Robust Influence Functions with Flat Validation Minima
Table Description of the math problem task templates. We consider 10 different types of math word problems, following the setup of Kwon et al. (2024).
Math word problems
Template prompt question
Remaining pizza slices
Lisa ate A slices of pizza and her brother ate slices from a pizza that originally had C slices. How many slices of the pizza are left?
Reason: Combined slices caten = A + B. Left= C- (A + B). For every A students going on field trip, there B adults needed chaperones. If C students are attending, how many adults are needed?
Chaperones needed for trip
Reason:
Adults needed = (B C) // A.
Total number after purchase
In aquarium, there are A sharks and B dolphins. If they bought C more sharks, how many sharks would be there in total?
Reason; Total sharks = C.
Total game points
Michael scored A points in the first game, B points in the second, C. in the third, and D in the fourth game. What is his total points?
Reason: Total points
C+ D.
Total reading hours
Emily reads for hours each day. How many hours does she read in total days?
Reason:
Total hours read A * B.
Shirt cost after discount
A shirt costs There's B-dollar off sale. How much does the shirt cost after discount?
Reason:
Cost after discount = A B.
Area of garden
A rectangular garden has length of A meters and width of B meters. What is its area?
Reason: Area = A B.
Total savings
Jake saves A each week, how much will he save after B weeks?
Reason: Total savings A * B.
Number cupcake boxes
A bakery sells cupcakes boxes of they have B cupcakes, how many boxes
can they fill?
Reason: Boxes filled = B // A.
Interest eamed
John invests A annual interest rate of B%. How much interest will he
after C years?
Reason: Interest = (A B C) // 100.
For our proposed VM and FVM, to obtain we further fine-tune the previously tuned LoRA model (i.e., the model obtained after the initial fine-tuning on the training set) on the validation set for epochs with an initial learning rate of x 10-5, 5 For FVM, we set 0.01. LPF (Bisla al., 2022) is employed the flat minima solver, with the hyper-parameters M 1 and 7 = 0.001 for all tasks, in accordance with the original paper.
B.3. Text-To-Image Generation
Datasets. Here, we briefly describe the datasets used in image generation tasks.
For style generation, we fine-tune the model by appending style description the text prompt, instructing it generate images in specific style. We use 200 training image-text pairs and 50 validation image-text pairs, resulting in total of 600 training data points and 150 validation data points. We use the DreamBooth dataset (Ruiz et al., 2023), which contains 30 different subjects The text follows the format: "Generate an image in specific custom style. text-data", where custom is replaced with either "cartoon", "pixelart", or "black and white line sketch", and text-data replaced with text sequence from the training dataset. A detailed description the style prompts provided in Table 9.
For subject generation, we use Google's DreamBooth dataset (Ruiz et al., 2023), which contains 31 unique subjects across categories such backpack, dog, bowl, and sneaker. Each subject has four six images, with three examples per subject used for training and the remaining for validation. For each subject, 3 data points are used for the training dataset and 1 to 3 data points are used for the validation dataset. To enable the model to distinguish between different subjects, we assign unique random string to each subject in the prompt. For instance, two different dogs are referenced as "a i0VpE dog" and "a oHwLM dog" in the training and validation sets, respectively.
Training. We use LoRA (Hu et al., 2022) to fine-tune stable-diffusion--i..5 (Rombach et al., 2022). We apply LoRA to
14

---

Towards Robust Influence Functions with Flat Validation Minima
Table Description of the text-to-image generation task templates. Each style has 200 training image-text pairs and 150 validation image-text pairs, following the setup of Kwon et al. (2024).
Image style
Text prompt
Cartoon
Generate an image in specific cartoon style. A text sequence the original
datasct which describes an image.
Pixel Art
Generate an image in specitie pixelart style. A text sequence of the original image.
dataset which describes
Sketch
Generate an image in a specific black and white line sketch style. A text sequence
of the original dataset which describes image.
Table Area Under the Receiver Operating Characteristic Curve (ROC AUC) (%) and Average Precision (AP) (%) for identifying mislaheled samples the CIFAR-IONA 10N/-10NN datasets across different influence estimation methods. Results (mean-tstd) are reported over 3 random runs. Bold indicates the best results and underline denotes the second-best results.
CIFAR-ION
CIFAR-10OM Noisy
Method
Aggre
Random
Worst
ROC AUC
AP
ROC AUC
AP
ROC AUC
AP
ROC AUC
AP
EK-FAC EK-FAC"
57.03++0.55
39.71+1.41
1.785=1.77
57.70=1.66
72.18
72.81 64
60.30 1.30
59.37=0.87
10.02+1..40
60.51+1.42
81.55=00449
76.58=0.07
80.2445.13
81.72=4.66
12.72+1.51
64.86+1 02
VM
95.18:10.15
76.3110021
95.92-10.10 96.63=00.33
87.35.10.45
95.88110.13
94.2710.211
89.77110008
83.81110.18
FVM
96.14:10.06
79.53+0.27
88.82:10.26
96.46110088
94.97=0.12
90.80110.4
85.41=0.055
every attention layer in the stable-diffusion--1.5 model. The basic parametessssters settings are listed as follows: minibatch size (4), optimizer (AdamW (Loshchilov & Hutter, 2019)), initial learning rate (10-4), number of steps (10000), and learning rate decay (cosine). For LoRA hyperparameters, we set r = and a = 2. The training was performed using the HuggingFace Peft library (Mangrulkar et al., 2022).
LOss. We use negative log-like likelihood of generated image loss function. For sequence of input tokens (z1,-.. .r) and the corresponding target image y, compute negative log- likelihood E(y, fe(x)) - log fe (z)).
We set T = 77.
Hyper-parameters. The Sarem-earameter settings for image generation tasks are the same as those in text generation tasks.
C. Addtional Experiment Results
C.1. Comparison with EK-FAC
We compare our proposed VM/FVM approach with EK-FAC (Grosse al., 2023) on mislabeled sample detection tasks, following the experimental setup described in Appendix B.1. The results presented in Table 10, where * indicates performance based on the best training checkpoints. As shown, our method consistently outperforms the EK-FAC-based influence function in identifying mislabeled data.
C.2. Validation Set Size
To investigate the impact of the validation set size on our approach further, we conducted experiments comparing different IF approaches for mislabel detection on the CIFAR-ION Worst dataset with varying validation set sizes. The results (ROC AUC/AP) are reported Table 11, with the performance drop relative to the size of 10,000 shown in parentheses. As observed, the performance of all approaches tends to degrade. Nevertheless, our method maintains strong performance despite the reduced data size and still outperforms other approaches by a significant margin.
15

---

Towards Robust Influence Functions with Flat Validation Minima
Table 11, Arca Under the Receiver Operating Characteristic Curve (ROC AUC) (%) and Average Precision (AP) (%) for identifying mislabeled samples the CIFAR-ION Worst datasets across different influence estimation methods, Results (mean are reported over 3 random runs. Performance changes (%) are computed relative to the results under the 10,000-sample validation set.
Metric
Size
LiSSA
VM
FVM
10000
65.75 + 0.39
95.88 + 0.13
96.46 + 0.08
5000
65.84 0.35 (10.13)
95.31 + 0.19 (10.59)
95.68 + 0.23 (10.80)
ROC AUC
2000
65.60
0.49 (J0.22)
94.37 0.27 (U1.57)
94,86 = 0.21 (11.65)
1000
64.79
0.94 (U1.46)
93.07 + 0.65 (12.93)
93.72 +: 0.53 (12.84)
10000
68.35 + 0.74
94.27 + 0.21
94,97 + 0.12
5000
68.35 + 0,95 (10.00)
93,44 0.14 (J0.88)
93.71 = 0.22 (U1.32)
AP
2000
67.75 + 1.27 (J0.87)
92.11 + 0.16 (J2.29)
92.60 + 0.14 (12.49)
1000
66.54
1.74 (12.64)
90.26 + 0.69 (L4.25)
90.94 + 0.54 (14.24)
Table 12. Area Under the Receiver Operating Characteristic Curve (ROC AUC) (%) and Average Precision (AP) (%) for identifying mislaheled samples on the CIFAR-ION Worst and CIFAR- 100N Noisy datasets across different sharpness-aware optimizers equipped to FVM. Results (tt) are reported over 3 random runs. Bold indicates the best results and underline denotes the second-bes results.
CIFAR- ION Worst
CIFAR- -OON Noisy
Method
ROC AUC
AP
ROC AUC
AP
SAM
96.46=0.08 96.71=0.14 95.1110.119
94.97:10.12
90.80:10.04 90.83:10.11
85.4111..05
ASAM
85,25:10,22
F-SAM
96176_0.10
95.29-10.14
91.25:10.04
86.06.10.03
C.3. Sharpness-Aware Optimizer
We compare three different sharpness-aware optimizer, including SAM (utilized in our initial approach), ASAM (Kwon et al., 2021), and F-SAM (Li et al., 2024), on CIFAR-ION Worst and CIFAR-10ON Noisy datasets. The results (ROC AUC) are presented in Table 12. As observed, different sharpness-aware optimizers do affect the final results, with F-SAM achieving the best performance. This supports our hypothesis that better flat validation minima can lead to more accurate influence function estimation.
C.4. Inverse Hessian Approximation
To quantitatively discover the impact of the approximation on the inverse Hessian, we conduct experiments on the mislabel detection task. Specifically, replace the computation of the inverse Hessian H in Equation (21) from diagonal Fisher to LiSSA (Koh & Liang, 2017) and DataInf (Kwon et al., 2024).
Note that the accelerated approximations in LiSSA and DataInf both rely on the inverse Hessian-vector product (iHVP). However, the initial product H in Equation (21) depends on the specific training sample 2tr, which requires recomputation for each training sample. As a result, we cannot directly apply these methods to our proposed influence function. To address this, we introduce a random vector V E Rlo x1 where each element is sampled from standard normal distribution, i.e., V; ~ (O, 1). With this, Equation (21) becomes JI_ H VVV Dzg. and we have Eig VVV g TT, -.. By using the random vector V. we can directly apply the iHVP trick to H-dV V and compute the inverse Hessian based on LiSSA or DataInf. practice, we sample different to ensure stability and reduce the variance the approximation.
The results (ROC AUC/AP) reported in Table 13. Theoretically, from the perspective of computational complexity and estimation fidelity, LiSSA and DataInf are expected to provide more accurate approximations of the inverse Hessian than the diagonal approximation. However, in our current experiments, we observe that the performance of the diagonal approximation is competitive with, and in some cases even superior to, LiSSA and DataInf.
One possible reason is that LiSSA and DataInf involve more sensitive hyperparameters (c.g., number of iterations), and tuning them appropriately for each setting is non-trivial. Despite this, the diagonal approximation, which significantly
16

---

Towards Robust Influence Functions with Flat Validation Minima
Table 13. Arca Under the Receiver Operating Characteristic Curve (ROC AUC) (%) and Average Precision (AP) (%) for identifying
mislabeled samples on the CIFAR-ION 1-N7-10ON datasets using different inverse Hessian approximation methods. Bold indicates the best results and underline denotes the second- results.
CIFAR-ION
CIFAR-100N
Inverse Hessian Approximation
Method
Aggre ROC AUC
Random
Worst ROC AUC
Noisy
AP
ROC AUC
AP
AP
ROC AUC
AP
LiSSA
95.22
67.36
95.67
78.90
95.00
89.26
88.88
80.89
VM
DataInf
95.37
72.09
95.98
83.26
95.57
91.09
89.58
82.81
Diagonal (ours)
95.18
76.31
95.92
87.35
95.88
94.27
89.77
83.81
LiSSA 95.95
70.71
96.17
80.16
95.33
89,56
89,48
81.81
FVM
DataInf
96.18
76.15
96.58
84.85
96.02
91.54
90.24
83.81
Diagonal (ours)
96.63
88.82
96.63
88.82
96.46
94.97
90.80
85,41
more efficient, achieves consistently strong performance across all datasets. This suggests that our of the diagonal approximation, while simplistic, does not lead to substantial degradation in influence estimation accuracy in practice.
D. Proofs
D.1. Proof of Theorem 3.2
Proof. Following Definition 3.1, we have
E(I) = EE-D [1 {sgn (I(2, Smm)) F sgn (I Smal))
(26)
PDDD |sgn (2, + sgn (I' Sva))]
(27)
Next, we decompose this probability as:
E(T) = PDD |sgn (I (2, Syal) = 10 1( Sma)) < <0]
(28)
+ PPDD (sgn (I (z, Swa)) -1 -IZ 0).
(29)
Let = PIND [sgn (I* Saa)) = 1]. we can express (II as:
E(I) = p P.- [[ 2, Smal < 0 sgn (I (z, Seal)) 1]
(30)
+ (1 - p) P. PEND [[z S > 0 sgn (I" (I, Sval)) = -1
(31)
= PPPPDD [[2, Saa) < 0] (1 - p) P. PeND_ [I, Sval) > 0]
(32)
= =PPawD. [[z Sval) - E.wD, [[2, Sval)] | < -END. [I(z Swail]]
(33)
+ (1 - TP xD [(2 EenD (Tat) > -EEDD [I(z, Sta)l]
(34)
Given the assumption that EZzD, [I(z, Swa))] > 0 and EEND [I(z, Smii) < 0, by Hoeffding's inequality, we can bound these probabilities:
2 E..DD Sval]
E(I) S pexp
(35)
supD=D, I(2, -inf.wD, I(z, Smal)
2 BEAD [I(z, Sual)]
+ (1 - exp
(36)
(SuupwD I(2, inf I(z, Sma))
2
2/²2
<exp
(37)
"'supsnD Iva) Sma) inf==D (iiissII(22,555 Smai)
17

---

Towards Robust Influence Functions with Flat Validation Minima
where jr min min(1 BD [[ ((),) I, B
(va)))
Now, recall the definition of Influence Function, where [(2, Sval) = Rva( (0). Substituting this, we can rewrite the bound as:
2,²²
(()) S exp
("suppaD Rv,( infanD Rval (02)
(38)
and μ min (IEeDD, [Rina [03) Ryal (0)). LEDD [Raa(O.))
R0)(
Finally, considering that we assume linear or second-order approximation with respect to the parameter change is valid, the perturbed parameter 0gg estimated by the Influence Function lies in the neighborhood of Thus, we have:
sup (eal(0s) S max Rva(O + A),
(39)
znD
SA
where A is the parameter change and 7 sup 11=- - 0||.
Define Pr (0) = maxiaa 1s Ryal (0 - A), the error bound becomes:
(-
2/22
24²
E(I) S exp
S exp
(40)
((R22 infz2D IDD Peaa (0=)) (0:))²
R2an (0)=
This completes the proof.
D.2. Proof of Corollary 3.3
Proof. From the definition of (Z), we have:
N
(I(I) =
Σ
[1 (sgn (I(zn: Sval) * sgn (I* (Zn, S.
(41)
n=1
Let X,, = {sgn (I(zn, Sea)) # sgn (I (Zn, Syml)) Applying Hoeffding's inequality, we have:
N
1
PP
X - -.-D/X exp (-2Nt²)
(42)
N
where / >>>>>>>>>>0 0. Recall the definition of R(I), this inequality can be rewritten as:
P [Es(I) E(I) N $ < exp (-2Nt))
(43)
Let exp( (-2Nt²). Solving for we have
V-
log 5 2N
Substituting back into the inequality gives:
log 2N
IP Es(I) E
< 6.
(44)
Thus, with probability at least - 8, the following inequality holds:
log 6
(45)
2N
Finally, using Theorem 3.2, we have
log d 2N
2μ²²
log 8 2N
EI(I) <
exp
(46)
( (8)
This completes the proof.
18

---

Towards Robust Influence Functions with Flat Validation Minima
E. Derivations
E.1. The Parameter Change Oz,,
Here, for completeness, we derive the parameter change 0s, - 0 the context of loss minimization. This derivation builds upon the analysis of 8% - presented in Koh & Liang (2017).
Recall that 0 minimizes the following SAM object:
M Σ
Re (0) == max Real (0+
max S
(mm, 0 - A).
(47)
|
M
m=1
We further assume that is twice-differentiable and strongly convex in 0, i.e.,
H
Vi Poaa
1 M
Σ
2 8
(mmm 0).
(48)
exists and is positive definite. This guarantees the existence of H which we will use in the subsequent derivation.
The perturbed parameters 2, can be written
0 arg min Rval + A) + el(zu, 0 + A).
(49)
Since 0, a minimizer of Equation (49), assuming 7 - let us examine its first-order optimality conditions:
Ryal (0.m) Vol(z, Oz..
(50)
Next, since 8s, - as c + O, we perform a Taylor expansion the right-hand side:
V Rual (0) v Vgl(2n 0)] + V2 (0l (0) + e V 0] (0mh -0).
(51)
where we have dropped o o(112,. 0|)) terms.
Sovling for 0z, - 0, we get:
0 0= Real + Vzl(( 0)
V Reat (0) + VVel(zn 0)
(52)
Since 0 is the minimizer of RE (0), we have V Rual (0) = 0. Dropping o(s) terms, then
0. Ō -ellvm Iaw
(53)
where JZ. Vol(En. 0).
E.2. The Influence Function Recall Equation (19), we have
((Eaa, 0an)) ((2vat: 0) ~ gth A0 ~A VillEvat 0)20.
(54)
Combining with Equation (18), we have
(Zzvat 2z.) E(zval, 8) = Ihm (-cHmml 1
Izu
+ - -cH H- - 9z.. V2ll(zal 8) ()(- (-cftral Dzy)
(55)
-
T f
+
1
T
(zf((zn 0) H. Izm-
(56)
2
19

---

Towards Robust Influence Functions with Flat Validation Minima
Since we want to measure the loss change with respect removing the training sample zy. we define the Influence Function as follows:
T zval) == Is.a H val Ds.
2 5
V 0) Ha -
Faw>
(57)
with > 0. Next, we consider the influence on the validation set Sval. which can be defined as the sum of the influence on each validation sample:
[arr 2tr, Seal) =
Σ [(arr: 2m)-
(58)
Incorporating with Equation (57), we get
I(2r, Sval) = Σ Izm H val Dzu 2 5e0an T Hyal Vi Ha Izu
(59)
Ival H val Izm
2 <9 Hya -z..
Since Ival + 0, we define the Influence Function on the validation set Syal as follows:
[(2rr Swal) = I2h H Izw.
(60)
where e is dropped.
F. Related Works
F.1 Influence Function
The use of influence-based diagnostics dates back to seminal works such as Cook & Weisberg (1982). More recently, Koh & Liang (2017) introduced Influence Functions to large-scale Deep Neural Networks (DNNs), sparking significant research interest. However, subsequent studies have identified challenges in applying IF to DNNs. Basu et al. (2021) highlighted the fragility of IF in deep learning scenarios, while Bae al. (2022) demonstrated that practical influence function estimates often fail to align with leave-one-our retraining in nonlinear networks. Instead, they approximate different quantity known the proximal Bregman response function (PBRF).
Still, IF has inspired wave of research aimed at improving its computational efficiency and scalability for large models. TracIn (Pruthi et al., 2020) improves computational efficiency by tracking loss changes using gradient information from multiple training checkpoints. FastIF (Guo et al., 2021) accelerates influence estimation by selecting the h-nearest training samples around a given validation sample. Schioppa et al. (2022) speed up the inverse Hessian calculation based on Amoldi iteration (ARNOLDI, 1951). Grosse et al. (2023) proposed Eigenvalue- Eigenvalue-Corretted Cronecker-Factoree Approximate Curvature (EK-FAC) scale Influence Function computations large language models (LLMs). GEX (Kim al., 2023) alleviates the bilinear constraint of standard Influence Functions and leverages Geometric Ensemble (GE) for more precise influence estimation. DataInf (K won et al., 2024) further enhanced efficiency by deriving closed-form influence estimation, making it particularly effective for parameter-efficiee fine-tuning techniques such as LoRA (Hu et al., 2022).
Despite these advancements, previous studies have overlooked a fundamental issue: the relationship between influence estimation and the local sharpness of the validation risk. Regardless of how efficiently IF is computed, the absence of local sharpness correction results in inaccurate influence estimation, limiting its reliability in practice.
F.2. Sharpness-Aware Minimization
The relationship between the flatness of local minima and generalization has been extensively explored prior works (Dinh al., 2017; Li et al., 2018). More recently, numerous studies have aimed to improve model generalization by explicitly seeking flatter minima (Chaudhari al., 2017: Tsuzuku al., 2020; Foret al., 2021). Among these, Sharpness- Aware Minimization (SAM) (Foret et al., 2021) has attracted considerable attention. SAM introduces general training framework that formulates optimization as min-max problem, encouraging model parameters to lie in neighborhoods with uniformly low loss, thereby achieving state-of-the-arr generalization across a wide range of tasks. Building on SAM, subsequent works have advanced the approach by refining the geometric definition of the neighborhood (Kwon et al., 2021; Kim et al., 2022), proposing improved surrogate loss functions (Zhuang et al., 2022), developing more adaptive adversarial strategics (Li Giannakis, 2023), and improving training efficiency (Jiang et al., 2023).
20

---

Towards Robust Influence Functions with Flat Validation Minima
G. Miscellanies
G.1. Remark on the Sign of Influence
In previous studies, the same training sample can yield positive or negative influence depending on the paper. due to differing
definitions of loss change. This ambiguity arises particularly between {(2, 0z) {(2, 6) and f(z, 0) [(2, 0z). For example, Koh & Liang (2017) define the influence as ((200)) {(z, 0), whereas Pruthi et al. (2020) define it {(z, 0) {(2, 0z). In
this paper, adopt the previous approach, measuring loss change (=) (),
21


Reviews:
[{"review_id": "YCAxQjSMBA", "reviewer": "Reviewer_tK1F", "summary": "This paper reexamines influence functions (IF) in deep learning and argues that their standard formulations fail when models are trained on noisy data—primarily because of sharp validation risk landscapes. The authors theoretically link the estimation error of influence functions to the sharpness of the validation loss and posit that obtaining flat validation minima (via techniques such as Sharpness-Aware Minimization) is key for accurate influence estimation. To that end, they propose a new influence estimator based on second‐order approximations for both the parameter change and the loss change, tailored specifically for flat validation minima. Extensive experiments on tasks such as mislabeled sample detection, training sample relabeling, and identifying influential samples in text and image generation are presented to demonstrate that their methods (denoted as VM and FVM) outperform existing approaches.\n\n## update after rebuttal\n\nThe author's rebuttal addressed part of my concern, that the assumption of the Hessian being positive semi-definite is not part of their conclusion. I accept this reason.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We acknowledge the reviewer’s concern regarding the use of **diagonal approximation of the empirical Fisher matrix for estimating the inverse Hessian**. We are happy to provide a more thorough discussion of this aspect. \n\n### 1. Diagonal approximation for inverse Hessian is simply a practical choice\n\nWe would like to clarify that our core contribution lies in improving the influence function (IF) estimation by (1) **seeking flat validation minima**, and (2) **adapting IF computation to this scenarios**. Therefore, our core contributions are **orthogonal** to the specific choice of inverse Hessian approximation method. In fact, once flat validation minima are obtained, the specific method used to compute the inverse Hessian in Equations (20) or (21) becomes a matter of choice, depending on the trade-offs between efficiency and accuracy.\n\n### 2. Further support for the use of the diagonal approximation\n\nWe mainly refer to [r1] in JMLR, where the author discusses the diagonal of the empirical Fisher matrix and emphasizes that, \n\n- for accelerating 2nd-order optimization, the **diagonal approximation is a widely accepted method** to avoid the full computation of the Hessian [r2]. \n- furthermore, optimization methods such as Adagrad and Adam, which are based on the Fisher matrix, also estimate second-order derivatives through an approximation to the diagonal of the empirical Fisher matrix [r1]. \n\nThese examples reflect a broader consensus in the community that approximating the inverse Hessian in DNNs through the diagonal of the empirical Fisher matrix strikes a practical balance between efficiency and effectiveness, especially in DNNs scenarios.\n\n### 3. Additional experiments for different inverse Hessian approximation approaches\n\nWe acknowledge that with a more accurate inverse Hessian approximation approach, the improved IF estimation performance can be expected. \n\nTo quantitatively discover the impact of the approximation on the inverse Hessian, we conduct experiments on the mislabel detection task. Specifically, we replace the computation of the inverse Hessian $\\\\tilde{H}_{val}^{-1}$ in Equation 21 from diagonal Fisher to LiSSA and DataInf.\n\n> [Technical details in adaptation]\n> \n> Note that the accelerated approximations in LiSSA and DataInf both rely on the inverse Hessian-vector product (iHVP). However, the initial product $\\\\tilde{H}\\_{val}^{-1} \\\\tilde{g}\\_{z\\_{tr}}$ in Equation (21) depends on the specific training sample $z_{tr}$, which requires recomputation for each training sample. As a result, we cannot directly apply these methods to our proposed influence function.\n>\n> To address this, we introduce a random vector $V \\\\in R^{|\\\\theta| \\\\times 1}$, where each element is sampled from a standard normal distribution, i.e., $V_i \\\\sim \\\\mathcal{N}(0, 1)$. With this, Equation (21) becomes $\\\\tilde{g}\\_{z\\_{tr}}^\\\\top \\\\tilde{H}\\_{val}^{-1} V V^\\\\top \\\\tilde{g}\\_{z\\_{tr}}$, and we have $\\\\mathbb{E}[\\\\tilde{g}\\_{z_{tr}}^\\\\top \\\\tilde{H}\\_{val}^{-1} V V^\\\\top \\\\tilde{g}\\_{z\\_{tr}}] = \\\\tilde{g}\\_{z\\_{tr}}^\\\\top \\\\tilde{H}\\_{val}^{-1} \\\\tilde{g}\\_{z\\_{tr}}$. By using the random vector $V$, we can directly apply the iHVP trick to $\\\\tilde{H}\\_{val}^{-1} V$ and compute the inverse Hessian based on LiSSA or DataInf. In practice, we sample 5 different $V$ to ensure stability and reduce the variance of the approximation\n\nThe results (ROC AUC/AP) are as follows:\n\n|Method|CIFAR-10N Aggre|CIFAR-10N Random|CIFAR-10N Worst|CIFAR-100N Noisy|\n|-|-|-|-|-|\n|VM(LiSSA)|95.22/67.36|95.67/78.90|95.00/89.26|88.88/80.89|\n|VM(DataInf)|95.37/72.09|95.98/83.26|95.57/91.09|89.58/82.81|\n|VM(ours, diagonal)|95.18/76.31|95.92/87.35|95.88/94.27|89.77/83.81|\n|FVM(LiSSA)|95.95/70.71|96.17/80.16|95.33/89.56|89.48/81.81|\n|FVM(DataInf)|96.18/76.15|96.58/84.85|96.02/91.54|90.24/83.81|\n|FVM(ours, diagonal)|96.63/88.82|96.63/88.82|96.46/94.97|90.80/85.41|\n\nTheoretically, from the perspective of computational complexity and estimation fidelity, LiSSA and DataInf are expected to provide more accurate approximations of the inverse Hessian than diagonal approximation. However, in our current experiments, we observe that the performance of the diagonal approximation is competitive with, and in some cases even superior to, LiSSA and DataInf.\n\nOne possible reason is that LiSSA and DataInf involve more sensitive hyperparameters (e.g., number of iterations, damping), and tuning them appropriately for each setting is non-trivial. Despite this, the diagonal approximation—which is significantly more efficient—achieves consistently strong performance across all datasets. This suggests that our use of the diagonal approximation, while simplistic, does not lead to a substantial degradation in influence estimation accuracy in practice.\n\n---\n\n[r1] New Insights and Perspectives on the Natural Gradient Method, JMLR 2020.\n\n[r2] Deep learning via Hessian-free optimization, ICML 2010.", "claims_and_evidence": "The paper claims that the sharpness of the validation risk degrades the performance of standard influence functions and that flat minima lead to more reliable influence estimates. Although the authors provide theoretical bounds (Theorem 3.2 and Corollary 3.3) and empirical results, a critical issue arises with one of the core components: the inversion of the Hessian matrix. The diagonal approximation and its justification are not convincing in Appendix A. This weakens the overall support for the paper’s claims, as a robust inverse Hessian approximation is vital for the accuracy of the influence estimation.", "methods_and_evaluation_criteria": "The proposed method employs a flatness-aware objective (inspired by SAM) to find flat validation minima and then computes influence using a second-order approximation. However, a major methodological concern is the unclear treatment of the inverse Hessian. In Appendix A, the authors attempt to justify an approximation for the inverse Hessian—crucial for computing the parameter change—but the derivation and assumptions (such as the diagonal dominance of the empirical Fisher Information matrix) are not convincing enough. This lack of clarity raises doubts about the stability and reliability of the proposed estimator.", "theoretical_claims": "The paper's theoretical development is generally sound and well-structured. The authors derive bounds on the influence estimation error by explicitly connecting it to both the validation loss and its sharpness.", "experimental_designs_or_analyses": "The experimental evaluation spans multiple tasks, including mislabeled sample detection and influential sample identification in both text and image generation. However, the reliance on benchmark datasets like CIFAR-10N/CIFAR-100N and controlled experimental settings means that the impact of the unclear Hessian inversion is not fully explored.", "relation_to_broader_scientific_literature": "The paper situates itself within a robust line of work on influence functions (e.g., Koh & Liang, 2017) and builds on recent advances by incorporating flat minima via SAM.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper presents an innovative theoretical framework linking flat validation minima to influence estimation error.\n\n- It introduces a novel estimator that, in controlled experiments, appears to outperform several recent baselines.\n\n\nWeaknesses:\n\n- The derivation and approximation of the inverse Hessian—a central element of the method—are not presented convincingly.", "other_comments_or_suggestions": "The authors should provide a clearer, more detailed justification for their inverse Hessian approximation. Additional empirical or theoretical evidence validating the assumptions made in Appendix A would be valuable.", "questions_for_authors": "See listed above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QtMIuwg9Jz", "reviewer": "Reviewer_r2Jq", "summary": "The influence function measures the influence of training samples on the validation loss. While this is typically done using minimas of the training loss, the authors argue for using flat validation minima. They then show experimntally and arguen theoretically that the standard estimators for influence do not work well in this setting and propose a new way of calulcating the influence that is designed to deal with these problems.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of the theoretical and experimental support for our claims.\n\n> [Broader Sci. Literature.1] Absence of Related Literature section\n\nWe kindly note that, due to space constraints, the discussion of related work on influence functions was not included in the main paper but has been deferred to **Appendix E**.\n\n> [Broader Sci. Literature.2] Further discussion on related works of flat minima\n\nWe thank the reviewer for highlighting the importance of further discussing flat minima in the related works section.\n\nIn the initial version, we acknowledge that we did not elaborate on prior work related to flat minima. This decision was based on the observation that most existing literature focuses on flat minima in the context of model generalization. In contrast, our work considers flat validation minima as a key factor for accurate IF estimation. Given this shift in focus, we initially chose not to include a detailed discussion of the broader flat minima literature.\n\nNonetheless, we agree that adding this context will help clarify our perspective and position our work more clearly. We will therefore incorporate a more comprehensive discussion of relevant literature in the revised version.\n\n> [Other Weakness] Impact of \"parameterization of the loss function\"\n\nFirst, we would like to highlight that the main theoretical contribution of our manuscript is to establish a connection between flat validation minima and the accuracy of influence function estimation. The focus of our analysis is on **how flat minima matter for IF**, rather than on how to obtain them.\n\nTo this end, we adopted a commonly used method—Sharpness-Aware Minimization (SAM)—to obtain flat validation minima, primarily as a means to support our theoretical insights rather than as a central contribution of the paper.\n\nWe sincerely thank the reviewer for suggesting relevant literature [1][2] on achieving flat minima from the parameterization of the model. While these directions are indeed insightful, due to time constraints and the lack of public code for those papers, we opted to experiment with two more recent and open-sourced sharpness-aware optimizers as alternatives.\n\nSpecifically, we compare three different sharpness-aware optimizer, including SAM (utilized in our initial submission), ASAM [r3] and F-SAM [r4], on CIFAR-10N Worst and CIFAR-100N Noisy dataset. The results (ROC AUC) are presented below.\n\n|Method|CIFAR-10N Worst|CIFAR-100N Noisy|\n|-|-|-|\n|SAM|96.46/94.97|90.80/85.41|\n|ASAM|96.71/95.11|90.83/85.25|\n|F-SAM|96.76/95.29|91.25/86.06|\n\nAs observed, different sharpness-aware optimizers do affect the final results, with F-SAM achieving the best performance. This supports our hypothesis that better flat validation minima can lead to more accurate influence function estimation.\n\n---\n\n[r3] ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks, ICML, 2021\n\n[r4] Friendly Sharpness-Aware Minimization, CVPR, 2024", "claims_and_evidence": "The basic claims are well supported, both theoretically and experimentally. I also think that the flow of the manuscript (observation -> problem -> solution) is coherent and convincing. The experimental evidence for the superiority of their approach is sufficient and extensive.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I did not check the theoretical claims in detail (especially in the appendix) but from my reading they seem to be consistent.", "experimental_designs_or_analyses": "No.", "relation_to_broader_scientific_literature": "The paper does not have a \"Related literature\" section which makes it hard for a non-expert reader to assess novelty. I would strongly suggest adding one, since many of the core ideas (flat minima etc.) have vast literature attached to them.", "essential_references_not_discussed": "Not that I know of.", "other_strengths_and_weaknesses": "There has been a lot of discussion around the concept of flat minima and their dependence on the parameterization of the loss function (see below). This seems to be important also for the current paper but the authors do not discuss it. I think it would be good to have a sentence or two on the impact of these issues on the proposed method.\n\n[1] Dinh, Laurent, et al. \"Sharp minima can generalize for deep nets.\" International Conference on Machine Learning. PMLR, 2017.\n[2] Pittorino, Fabrizio, et al. \"Deep networks on toroids: removing symmetries reveals the structure of flat regions in the landscape geometry.\" International Conference on Machine Learning. PMLR, 2022.", "other_comments_or_suggestions": "None.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lIFpZshCf2", "reviewer": "Reviewer_Uyxg", "summary": "In this paper, the authors propose a method for estimating influence functions (IFs) for deep neural networks, addressing limitations of existing approaches that struggle with noisy training data and rely on first-order IF approximations without considering the sharpness of validation risk. They demonstrate that the error in the influence function estimator is directly related to the risk on validation samples. As a result, their IF estimator relies on parameters optimized for the validation risk. Additionally, the authors introduce a second-order approximation method tailored to their flat validation minima framework. Extensive experiments are conducted across various benchmarks and architectures.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's valuable suggestion. After pondering over your questions, we compose the following responses and will include all the following elaborations in the revised manuscript.\n\n> [Q1.1] Validation set size\n\nWe appreciate the reviewer for highlighting this important factor, and we will include a detailed discussion of this in the revision.\n\nOur approach consists of two steps: (1) fine-tuning and (2) influence function (IF) computation. In the second step, both our method and other IF-based approaches are similarly affected by the size of the validation set used for computing influence scores. Therefore, compared to other methods, the validation set size primarily impacts the fine-tuning step in our approach.\n\nTo further investigate the impact of the validation set size on our approach, we conducted experiments comparing different IF approaches for mislabel detection on the CIFAR-10N Worst dataset with varying validation set sizes. The results (ROC AUC/AP) are reported below, with the performance drop relative to the size of 10.000 shown in parentheses. \n\n|Size|10000|5000|2000|1000|\n|-|-|-|-|-|\n|LiSSA|65.75/68.35|65.84/68.35 (+0.13%/-0.00%)|65.60/67.75 (-0.22%/-0.87%)|64.79/66.54 (-1.46%/-2.64%)|\n|VM|95.88/94.27|95.31/93.44 (-0.59%/-0.88%)|94.37/92.11 (-1.57%/-2.29%)|93.07/90.26 (-2.93%/-4.25%)|\n|FVM|96.46/94.97|95.68/93.71 (-0.80%/-1.32%)|94.86/92.60 (-1.65%/-2.49%)|93.72/90.94 (-2.84%/-4.24%)|\n\nAs observed, the performance of all approaches tends to degrade. Nevertheless, our method maintains strong performance despite the reduced data size and still outperforms other approaches by a significant margin.\n\n> [Q1.2] Optimum assumption\n\nIn Theorem 3.1, we assume that $\\mathbb{E} _ {z \\sim D_+}[\\mathcal{I}(z, S_{val}) > 0]$ and $\\mathbb{E} _ {z \\sim D_-}[\\mathcal{I}(z, S_{val}) < 0]$. As noted in the remark on line 201, this requires the overall influence estimation performance on $D$ (the training data distribution in practice) to be better than random guessing. This assumption implicitly relies on the training and validation data being drawn from related distributions and on the model retaining useful information about the training data. Consequently, we assume that the flat validation minimum does not deviate significantly from the given training minimum.\n\n> [Q2] Benefits under cases without noisy labels\n\nWe would like to clarify that we have indeed conducted experiments in settings without noisy labels. Specifically, the generation tasks presented in Sections 4.3 and 4.4 are based on clean datasets, and our proposed methods, VM and FVM, consistently outperform existing baselines in identifying the most influential samples under these conditions.\n\n> [Q3] 2nd-order gradients and computational complexities\n\nIn practice, to reduce both time and space complexities, we **approximate the Hessian** using the diagonal elements of the empirical Fisher Information Matrix, as described in Appendix A.\n\nFollowing the notation used in [r5], where $n$ and $m$ denotes the number of training samples and validation samples, $D$ the number of parameters per layer, and $L$ the number of layers, the computational complexity of LiSSA for estimating the Hessian inverse is $O(nD^2L)$, DataInf is $O(nDL)$ and our approximation is $O(mDL)$ (since it is performed w.r.t. the validation loss). Notably, in most practical scenarios, $m \\ll n$.\n\n> [Q4.1] Experimental setting in generation tasks\n\nRegarding Equation (23), we acknowledge that its current form may have led to some misunderstanding due to limitations in presentation. We would like to clarify that the experimental setup for the generation tasks totally follows that of [r5]. Later, we will refine the explanation and notation of this part in the revision.\n\n> [Q4.2] Only 10 validation samples are used for influence estimation in LLMs scenarios\n\nWe acknowledge the reviewer’s concern. Due to time constraints, we were unable to conduct more extensive experiments in the large language model (LLM) setting. However, we provide a simplified validation dataset size analysis on smaller-scale datasets—please refer to our response in [Q1.1] for details.\n\nWe will include a more thorough discussion on this limitation and its potential implications in the revised version of the manuscript.\n\n> [C1-7] Writing\n\nWe thank the reviewer for the detailed suggestions. We will revise the manuscript to improve clarity, including clearer descriptions of the experiment setups, explanation of why $\\epsilon = -\\frac{1}{N}$, and clarification of $R_{\\text{val}}$, which refers to the risk on the validation set. The typo in Line 215 will be corrected. Additionally, we will also update the legend label in Figure 2 to \"Validation ROC-AUC\" and revise the caption of Figure 3 to include the experiment setups.\n\n---\n\n[r5] DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models, ICLR 2024.", "claims_and_evidence": "Yes, the claims are supported by relevant evidence.", "methods_and_evaluation_criteria": "Yes, the authors have considered relevant baselines and benchmarks datasets.", "theoretical_claims": "I did not go through the theoretical claims in great detail as I am not an expert in that. However, the overall arguments made logical sense and provided intuition.", "experimental_designs_or_analyses": "Yes, the experimental setup and design are sound in most areas. However, I recommend that the authors update the manuscript to include clearer descriptions of the tasks and provide more detailed explanations of how each section of the experiments is carried out.", "relation_to_broader_scientific_literature": "In my humble opinion, the contributions are significant within the broader context of the scientific literature. However, there are some questions I have raised that require further clarification.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The proposed method is novel in its approach to addressing influence functions within the context of the flat validation minima framework that the authors consider. The experiment list is comprehensive, and the paper is well-written with minimal typos.", "other_comments_or_suggestions": "1. It would be beneficial to provide the experiment setup for Figure 2 either upfront or more explicitly in the caption.  \n2. The explanation for why removing $z_{tr}$ corresponds to $\\epsilon = -\\frac{1}{N}$ could be included in the preliminaries section.  \n3. **Minor**: In Figure 2, the blue legend line should be changed to \"Validation ROC-AUC\" to match the caption.  \n4. **Minor**: The title of subsection 3.2 needs to be updated.  \n5. $R_{val}$ in Theorem 3.2 and the corollary needs further explanation in the main paper.  \n6. In Corollary 3.3, Line 215, $\\hat{\\mathcal{R}}_S(\\mathcal{I})$  should be replaced by  $\\hat{\\epsilon}_{S}(\\mathcal{I})$  to maintain consistency (though I may be incorrect)\n7. The experiment setup for Figure 3 should also be provided upfront for better readability.", "questions_for_authors": "1. The proposed approach for IF estimation depends on optimization with the validation set. Can the authors comment on how the performance might be affected by changes in the validation set size? Moreover, is there an assumption that this optimization will lead to an optimum that does not significantly deviate from the optimum obtained during the training phase, but instead moves toward regions where the validation loss is flat?\n\n2. If the training data samples do not have noisy labels, will the method still provide the same performance benefits as the standard IF?\n\n3. Since the method still requires computing second-order gradients to estimate the IF, can the authors comment on the computational complexities?\n\n4. Can the authors explain Equation 23 in the paper and, in turn, provide a better explanation of the corresponding experiment? I reviewed Kwon et al., but I believe there is a major difference in your approach. Additionally, in Appendix B.2, only 10 validation examples are used for influence estimation. Could this pose a challenge for accurately estimating the IF?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i075jrMoja", "reviewer": "Reviewer_fYnW", "summary": "The article \"Towards Robust Influence Functions with Flat Validation Minima\" addresses the challenge of influence estimation in deep neural networks, particularly in the presence of noisy training data. The authors identify a fundamental limitation of existing influence function (IF) methods: their susceptibility to unreliable estimates due to the sharpness of the validation risk. To overcome this, they propose a novel approach that leverages flat validation minima for more accurate and robust influence estimation. This is achieved through a second-order approximation to minimize the impact of vanishing gradients and a refined parameter change estimation method tailored for flat test minima.\n\nThe paper evaluates the proposed methods (VM and FVM) across various tasks, including mislabeled sample detection, training sample relabeling, influential sample identification in text and image generation. The experimental results demonstrate superior performance compared to existing approaches, highlighting the importance of seeking flat minima for enhancing influence estimation accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the useful comments of the reviewer. We will update our current draft to avoid any confusion.\n\n> [Q1 / Other Weakness 1] How to calculate the $\\tilde{\\theta} _ {z_{tr}}$\n\nWe would like to clarify that it is unnecessary to compute $\\tilde{\\theta} _ {z_{tr}}$for each training sample. Specifically, we only need to fine-tune the pre-trained model parameters $\\theta$ to obtain $\\tilde{\\theta}$. Once this is done, we can approximate the parameter change $\\tilde{\\theta} _ {z_{tr}} - \\tilde{\\theta}$ as outlined in Eq 18.\n\n> [Q2] How to calculate the $\\tilde{g} _ {z_{tr}}$\n\nAfter reviewing our initial submission, we found that the definition on line 244 indeed contains an typo, which we suspect may have contributed to this misunderstanding. Specifically, $\\tilde{g} _ {z_{tr}}$ is defined as $\\tilde{g} _ {z_{tr}} = \\nabla _ {\\tilde{\\theta}} \\ell(z_{tr}, \\tilde{\\theta})$, as demonstrated in line 879 in the appendix. We will correct this in the revised version.\n\nIn practice, we implement this using Automatic Differentiation in PyTorch.\n\n> [Theoretical Claims] Sample size $M$ in eq. 35\n\nWe can confirm there is no $M$ in eq. 35. We directly apply the variance proxy for the whole random variable $\\mathcal{I}(z, S_\\text{val})$ in eq. 35.\n\nWe thank you for the observation, and will improve the readbility of the proof therein.\n\n> [Supp. Material] Code for Generation Tasks\n\nWe promise we will make the code for reproducing all reported experiments publicly available upon acceptance.\n\n> [References] Comparison with EK-FAC-based Influence function\n\nThank you for pointing out this important baseline that we had overlooked. We have now conducted a comparison with EK-FAC [r6] on mislabel detection tasks. The results (ROC AUC/AP) are reported below, where \\* denotes the results computed using the best training checkpoints. As observed, our proposed method consistently outperforms the EK-FAC-based IF.\n\n|Method|C-10N Aggre|C-10N Random|C-10N Worst|C-100N Noisy|\n|-|-|-|-|-|\n|EK-FAC|57.03/39.71|69.85/57.70|72.18/72.81|60.30/59.37|\n|EK-FAC\\*|70.02/60.51|81.55/76.58|80.24/81.72|62.72/64.86|\n|VM|95.18/76.31|95.92/87.35|95.88/94.27|89.77/83.81|\n|FVM|96.14/79.53|96.63/88.82|96.46/94.97|90.80/85.41|\n\n> [Other Weakness 1] Diagonal approximation for inverse Hessian\n\nThank you for raising the concern regarding the potential oversimplification of our inverse Hessian approximation. Please refer to our detailed response to Reviewer tK1F for a full discussion on this point.\n\n> [C1] Typo\n\nThank you for pointing out the incorrect mathematical symbol on line 215. We will correct this in the revised version.\n\n---\n\n[r6] Studying large language model generalization with influence functions, arxiv, 2023.", "claims_and_evidence": "Claim: Influence functions suffer from unreliability when applied to noisy training data due to the sharpness of validation risk.\n\nThe authors provide both theoretical analysis (e.g., Theorem 3.2 and Corollary 3.3) and empirical observations (e.g., Figure 1 and Figure 3). These demonstrate how sharp validation minima introduce gaps between estimated and actual influence, undermining reliability.\n\nThe combination of theory and experimentation effectively supports this claim.", "methods_and_evaluation_criteria": "The proposed methods (VM and FVM) and the evaluation criteria used in the paper are generally well-aligned with the problem at hand—improving influence estimation in deep neural networks, particularly for noisy datasets", "theoretical_claims": "Yes.\nThe theoretical proof is largely correct, but I believe that Theorem 3.2 neglects the sample size $M$ ( eq. 35) when using Hoeffding's inequality.", "experimental_designs_or_analyses": "The experimental designs and analyses in the paper are generally well-structured and provide strong support for the claims made.", "relation_to_broader_scientific_literature": "The key contributions of the paper are closely related to several important areas in machine learning research, including influence functions, robustness in deep learning, and optimization landscapes.\n\n1) The authors identify that existing influence function (IF) methods often fail to provide reliable estimates in deep neural networks, particularly when applied to noisy training data. They attribute this failure to deficiencies in loss change estimation due to the sharpness of validation risk.\n2) The paper establishes a theoretical connection between flat validation minima and accurate influence estimation, emphasizing the importance of optimizing for flat minima.\n3) The proposed methods (VM and FVM) significantly outperform existing approaches in detecting mislabeled samples and relabeling tasks.\n\nThe key contribution is based on the work of \" Sharpness-Aware Optimization\".", "essential_references_not_discussed": "I argue that the experiments of EK-FAC-based Influence function such as reference [1]  should be compared with the proposed method.\n\n[1] Grosse R, Bae J, Anil C, et al. Studying large language model generalization with influence functions[J]. arXiv preprint arXiv:2308.03296, 2023.", "other_strengths_and_weaknesses": "Other weakness:\n1. Assuming that the diagonal elements dominate the empirical Fisher Information matrix is an overly simplified solution that may introduce more theoretical bias.\n\n2. I'm not entirely sure if the $ \\tilde{\\theta}_{ztr}$ needs to be recomputed for every training sample z. If so, the computational cost would be too high.", "other_comments_or_suggestions": "line 215 influence estimation error's math symbol is incorrect.", "questions_for_authors": "1. How to calculate the $ \\tilde{\\theta}_{ztr}$ in eq. (17)?\n\n2. How to calculate the $ \\tilde{g}_{ztr}$ in eq. (21)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a connection between influence function, validation loss performance, and sharpness. The findings are well-supported and well-articulated through both theoretical and empirical arguments. After the rebuttal, reviewers have reached the consensus regarding their assessments — Therefore, I suggest its acceptance. One reviewer mentioned that there’s another line of related work on measuring flatness based on the loss function Hessian (e.g., [1](https://arxiv.org/abs/2306.08553), [2](https://arxiv.org/abs/2206.02659)). We encourage the authors to include these related works in the revision and also clarify any other issues surfaced from the reviewers’ feedback.
Decision: Accept (poster)