Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining
Authors: Weiyi Wang
Junwei Deng
Yuzheng Hu
Shiyuan Zhang
Xirui Jiang
Runting Zhang
Han Zhao
Jiaqi W. Ma
Venue: NeurIPS 2025 poster ()
Keywords: Data Attribution
Influence Function
Hyperparameter Tuning
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We identify and address an overlooked challenge in the practical application of data attribution: hyperparameter tuning is tricky due to the costly evaluation metrics.

Abstract: Data attribution methods, which quantify the influence of individual training data points on a machine learning model, have gained increasing popularity in data-centric applications in modern AI. Despite a recent surge of new methods developed in this space, the impact of hyperparameter tuning in these methods remains under-explored. In this work, we present the first large-scale empirical study to understand the hyperparameter sensitivity of common data attribution methods. Our results show that most methods are indeed sensitive to certain key hyperparameters. However, unlike typical machine learning algorithms---whose hyperparameters can be tuned using computationally-cheap validation metrics---evaluating data attribution performance often requires retraining models on subsets of training data, making such metrics prohibitively costly for hyperparameter tuning. This poses a critical open challenge for the practical application of data attribution methods. To address this challenge, we advocate for better theoretical understandings of hyperparameter behavior to inform efficient tuning strategies. As a case study, we provide a theoretical analysis of the regularization term that is critical in many variants of influence function methods. Building on this analysis, we propose a lightweight procedure for selecting the regularization value without model retraining, and validate its effectiveness across a range of standard data attribution benchmarks. Overall, our study identifies a fundamental yet overlooked challenge in the practical application of data attribution, and highlights the importance of careful discussion on hyperparameter selection in future method development.

Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining
Weiyi Wang'
Junwei Deng² Runting Zhang' Han Zhao² Jiaqi W. Ma²
Yuzheng Hu² Shiyuan Zhang²
Xirui Jiang'
University of Michigan Ann Arbor University of Illinois Urbana- Chaaa-Champaain
{wweiyi xirui, rrrii,uutttine cruntingz)@umii ch. edu
junweid2 yh46, sz54, hanzhao, haanzao,jiaginnana. jiagima)@illlaeeeee @iliiois....i.ii.. edu
Abstract
Data attribution methods, which quantify the influence of individual training data points machine learning model, have gained increasing popularity in data- centric applications in modern AL. Despite recent surge of new methods developed in this space, the impact of hyperparameter tuning these methods remains under- explored. In this work, we present the first large-scale empirical study to understand the hyperparameter sensitivity of common data attribution methods. Our results show that most methods are indeed sensitive to certain key hyperparameters. How- ever, unlike typical machine learning algorithms- whose hyprraarameters can be validation metrics-evaluattinn evaluating data attribution
tuned using computationally-cie performance often requires retraining models on subsets of training data, making such metrics prohibitively costly for hyperparameter tuning. This poses critical open challenge for the practical application of data attribution methods. To address this challenge, we advocate for better theoretical understandings of hyperparam- eter behavior to inform efficient tuning strategies. As a case study, we provide a theoretical analysis of the regularization term that is critical in many variants of influence function methods. Building on this analysis, we propose a lightweight procedure for selecting the regularization value without model retraining, and validate its effectiveness across a range of standard data attribution benchmarks. Overall, our study identifies fundamental yet overlooked challenge in the practical application of data attribution, and highlights the importance of careful discussion hyperparameter selection in future method development.
1 Introduction
Data attribution methods quantify the contribution of individual training samples to a machine learning model's predictions overall performance [22, 9]. These methods have been successfully applied to a range of data-centric tasks modern AL, including training data selection [37], mislabel detection [22], and copyright compensation [7]. Recently, this field has witnessed surge of novel methods [11, 18, 30, 12, 29, 6, 3, 35], each promising improvements in efficiency, efficacy, or theoretical soundness.
Despite this rapid progress, a crucial practical aspect remains largely underexplored: the hyperpa- rameter sensitivity of data attribution methods. Existing benchmark studies [19, 8] largely adopt the hyperparameter settings used the original papers proposing the methods. While some of the original papers include limited ablation studies, few of them offer practical guidance for hyperparam- eter selection, and many methods include implicit tunable knobs that have received little scrutiny. Moreover, the optimal hyperparameter choices vary across datasets and models.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Hyperparameter tuning is particularly challenging in data attribution due to the high computation costs associated with common evaluation metries. Standard evaluation metries- such as Leave-One-Ol Out (LOO) correlation [22] and Linear Datamodeling Score (L.DS) [29]-require retraining models on subsets of the full training dataset for hundreds or even thousands of times. conventional hyperparameter search under such evaluation metrics is prohibitively expensive in practice.
In this work, we present the first systematic, large-scale empirical study of hyperparameter sensitivity in data attribution. We benchmark suite of popular data attribution methods across various datasets and models with extensive hyperparameter sweeps. Our results confirm that most methods are sensitive to certain key hyperparameters. and the optimal hyperparameter choices vary across experi- mental settings. Our analysis also reveals other interesting novel findings regarding the interaction of different hyperparameters. Our findings highlight hyperparameter tuning as a critical open challenge in the practical application of data attribution.
As a first step towards addressing this challenge, we conduct a case study on selecting the regu- larization term-a crucial hyperparamter in many influence function methods [22, 12]-without costly retraining. We first provide theoretical analysis of how the regularization term affects the influence function performance terms of LDS. Building on this analysis, we introduce a surrogate indicator that enables selection of the regularization value without model retraining. We validate the effectiveness of the proposed approach across multiple experimental settings, including several variants of influence functions on different datasets and models.
To summarize, our contributions are as follows:
We identify hyperparameter tuning as a critical but largely overlooked challenge in the practical application of data attribution methods.
We conduct the first comprehensive empirical study of hyperparameter sensitivity in data attribution, benchmarking a range of widely used methods across diverse settings, confirming the necessity of practical hyperparameter selection strategies.
We provide theoretical analysis of the regularization term in influence functions and propose an efficient, retraining-fii selection procedure with strong empirical performance.
These results advance both the understanding and practical deployment of data attribution methods, encouraging a more systematic treatment of hyperparameters as integral components in future methodological development.
2
Background
In this section, we provide the necessary background to contextualize our empirical study of hyperpa- rameter sensitivity in Section and the subsequent case analysis in Section 4.
2.1 Data Attribution Methods and Their Hyperparameters
In this work, we focus on the methods included in a recent benchmark study [8], which consists of state-of-the-ard efficient data attribution methods while omitting family of game-theoretic methods such as Data Shapley [11, 18].' In the following, we introduce the notation, then review a set of most widely-used data attribution methods as well as their major hyperparameters examined in our study.
Notation. Consider multi-class classification problem with dataset S = ( y)) C 2, where =x x V. inputs F; E X' C Rd and labels yi E Y. We use a parameterized hypothesis function h(,;00 X Apyi- with parameters 0 RR. which typically outputs class probabilities via softmax layer. The cross-entropy loss for single example
(z, y) is denoted by L(z, 0) ((h(z, 8).8) y). The empirical risk at 8 over a subset A C of size a is ( = E.E L(2, 0). For a given test example z' = (x', y²), following Park et [29], we define the model output function as [(=', 0) == In p(= T-p(z'.0J. where p(z', 0)
(/2220000 0)) is the model's predicted probability for the correct class y'. We denote the parameter that minimizes R,s as f'--
'Whiee these game-ther theoretic methods are theoretically principled, they are typically computationally expen- sive and cannot scale up even moderately large models.

---

Data attribution methods. A data attribution method (or an attributor) is function 7 x + +R that, for a given test example z' E 2, assigns T(2', 2:) to each training point 2; S. A popular attributor is the Influence Function (IF) attributor, introduced by Koh and Liang [22], which computes the influence of a training example 2; E on a test example z' E as
TF (='(z z;) = Vof(=', 03) H VoL 03),
where Hs (o is the Hessian of the empirical risk Rs. To bypass the prohibitive computation of Hgg, they also propose using efficient approaches to approximate H VoL(z,, 0z), including conjugate gradient (CG) [27] and stochastic estimation method LiSSA [1]. For non-convex models, Hg is usually replaced with its regularized version Hs + XIp in case of singularity [22], where À > 0 is a regularization term and I, is the identity matrix of size p. To further handle non- convexity and reduce computational costs, Grosse al. [12] draw from second-order optimization literature [26] and introduce the Generalized Gauss Newton matrix (GGN) and empirical Fisher Information Matrix (FIM) to approximate Hg, which are guaranteed to be positive semi-definite (or even positive definite when a small regularization Mpp is included), and only involve first-order differentiation. We formally define the Influence Function attributor with FIM (IFFIM)
TIFFIMX 2)) off 63)) (Fs + AIp) VoL(z,, 63),
(2)
where Fs is the empirical FIM defined as:
Fs = Σ ES
VoL(z; Vo) 05) EI,sS Vo)
The TRAK [29] and LoGra [6] attributors adopt similar forms² while further introducing gradient projections to accelerate computation. In this case, we have the following modified IFFIM attributor, which takes projection matrix P E gPxP with projection dimension into consideration:
TFFIM.X.(E', 2x) - Vof(e', 03)
Mb.) 'P P VoL(z, 03).
Another popular method, TracIn [30], computes the influence of z; on z' by tracing the entire optimization process based on stochastic gradient descent. Mathematically, it works as if substituting Hg" with the identity matrix, while taking average across training model checkpoints weighted by the learning rate: Tiracin (z', 2;) := E. n. oz' 0,) T V L(, B2), where 9g and T7 are the parameters and learning rate at checkpoint
Major hy typerparameters. We We highlight some major hyperparameters that are shared across multiple data attribution methods mentioned above.
Regularization term x: the regularization term for handling non-convexity is a common hyperpa- rameter across many methods, including IF [22], TRAK [29]. LoGra [6]. etc.
Projection dimension p. gradient projection is common practice to reduce computational costs in data attribution methods [33, 6, 29], making the projection dimension ğ another common hyperparameter.
Training epoch: while, in principle, most data attribution methods rely on the optimal model parameters 0% to calculate the attribution scores, can also caleulate the attribution scores using an earlier model checkpoint 6g in practice. This makes the training epoch of the checkpoint used to calculate the attribution scores implicit hyperparameter, whose impact is largely overlooked in the literature."
2.2 Evaluation Metries for Data Attribution
Evaluation metrics for data attribution can be categorized into application-agnosiic and application- specific metrics.
The ppplication-annsttie metrics include Leave-One-Out (LOO) correlation [22], Linear Datamod- eling Score (LDS) [29], and Brittleness [16]. In general, these metrics measure the performance
AAppendix B.I provides a detailed discussion on the relationship between TRAK and IFFIM.
'TracIn [30] explicitly uses multiple checkpoints as shown in the definition of Preaela while TRAK [29] also explores the effect of aggregating over different checkpoints in ablation study.
3

---

of data attribution methods by their ability to predict the model output when training on subsets of the training data. As result, they typically require hundreds or even thousands of model retraining on subsets of the training dataset. While these metrics are widely used as principled evaluation criteria for novel data attribution methods in research setup, they are prohibitively expensive for hyperparameter tuning in practical applications.
The application-specife metries rooted in individual downstream application scenarios. Two common scenarios include noisy label detection and data selection [22, 11]. While these metrics are typically less computationally demanding, they are restricted specific applications and only provide a partial picture of the quality of data attribution. Hyperparameter tuning in data selection also relies on retraining the model on the selected data subset, which can be expensive for large-scale applications such as large language models [37].
In this work, we focus on LDS, which is one of the most widely used evaluation metrics in recent literature [29, 6, 8), for both the hyperparameter sensitivity study and the case study. Formally, LDS defined as following.
Definition 2.1 (Linear Datamodeling Score (LDS) (29)). Given an attributor and test example z' E Z, sample subsets A {Ai,... AA),, J. where each Aj C S is sampled uniformly random with fixed size The Linear Datamodeling Score (LDS) of r for z! on A is defined as:
CalT 2' A) == SpearmanCorr (2(22,02))ss
(5)
SEA,
where A'A, is the optimal model parameters learned on subset Aj, the subset influence modeled as additive in individual influences [15], and SpearmanCorr is the Spearman correlation [34].
2.3 Related Work on Hyperparameter Sensitivity in Data Attribution
Finally, we review existing literature with dedicated discussions on the effect of hyperparameters in data attribution. Koh et al. [23] examine the change in attribution quality when the regularization strength À is tuned in IF settings. They empirically observe an enhancement of the correlation between the attributed influences and actual effects when À is increased, and provide an upper bound on the approximation error of IF which decreases with Basu et al. [4] empirically study the effects of neural network architectures and weight-decay during model training on IF attribution quality. They reveal an increase in error when the network is deep or weight-decay is absent. Zhang and Zhang [38] utilize techniques from Neural Tangent Kernel [17] to provide theoretical lower bound of the LOO counterfactual loss change for wide two layer ReLU networks with regularized mean- square loss. Klochkov and Liu [21] leverage the spectral properties of Hessian guide selection of hyperparameters specific in LiSSA. However, to our knowledge, this work is the first systematic, large-scale study the hyperparameter sensitivity in data attribution that points out the unique computational challenge for hyperparameter tuning in practice.
3
A Large-Scale Study of Hyperparameter Sensitivity in Data Attribution
3.1 Experimental Setup
Data attribution methods. We evaluate a range of data attribution methods, including IF and its variants (CG and LiSSA) [22], TracIn [30], TRAK [29], and LoGra [6], introduced in Section 2.
Datasets and models. We perform empirical evaluations three standard benchmark settings: MNIST [25] with a multilayer perceptron (MLP), CIFAR-2 [24] with ResNet-9 [14], and Wiki- Text2 [28] with the GPT2 [31], which consists of both image and text data with various model sizes. We use the implementation from the dattri library [8] for the datasets and models.
Experiment design. We carry out the comprehensive study through multiple sub-experiments. For each sub-experiment, we analyze the sensitivity of one or two hyperparameters while controlling other hyperparameters to fixed. We perform grid search over the chosen hyperparameters within the search space. We calculate LDS for each experiment under 50 independently retrained models.

---

Hyperparameter selection and search space. We consider both the common hyperparameters introduced in Section 2.1 and some critical method-specific hyperparameters. For TRAK, we experiment with regularization, for projection-dimente dimension, normalization, and checkpoint-seeecton.. For IF, we analyze regularization and training-epoch, as well as max-iteration for the CG variant, and scaling and recursion -epth for the LiSSA variant. For LoGra, we search for regularization, projection-dimensio., and training- epoch. We design the search space for each hyperparameter around its default value proposed by the original papers. The detailed definitions and the search space of the hyperparameters are stated in Appendix A.1.
denection-iimesion, and training-epoch For TracIn, we search
3.2 Experimental Results
We summarize our findings and insights from the experimental results (Figure 1)$ as follows:
0,250
KSG) (LBSA)
0.225
0.4
0.200
0.3-
0.3
0.175
0.150
0.2
dim-512
0.1
proj_ dim=1022
praj dim=2048
0.1
proj dim-200s
pml 41-004
proj dim-4096
dim-4e9s
0.075
proj dim-6192
5mm-5192
2
regularization (log10)
regularization (log10)
regularization (log10)
(a) MNIST+MLP. Attributor:
IF.
(b)
MNIST+MLP. Attributor:
(c) CIFAR-2+ResNett) Attributor:
HP: regularization.
TRAK. HPP projection-dimenssonn,
TRAK. НIP: projection-dimension. regularization.
regularization.
0.25
0.40
proj, dmm
dro
0.20
proj 52?
0.36
0.34
scaing=1
2cainge5
scaing-10
0.30
0.05
scaing 100
scaing-30
scaing30
0.28
MNIST-MLP
ResNeto
1.75 2.00 2.25 2.50 2.75 3.00
10a
dscursion_depte (log10)
training_ _eeecchh
regularization (log10)
(d) MNIST+MLP. Attributor: IF (e) CIFAR-2+ResNon 2+ResNet-9 & MNIST+
(f) WikiText2+GPT2. Attributor:
(LiSSA). HP: scaling, recursion- MLP. Attributor: TRAK. HP: LoGra. HP: projection-dimensionn depth.
waining-epoch
regularization.
Figure 1: Part of the experimental results of hyperparameter sensitivity analysis.
Most attribution methods are sensitive to certain hyperparameters. Small changes in hyper- parameters can lead to large variations in LDS performance, highlighting the sensitivity of most methods to hyperparameter choices. For example, Figure la shows that increasing regularization from le-4 to le-2 for IF (CG) leads to a 3x increase in LDS.
The best hyperparameter choices vary across datasets and models. A configuration that performs well for one setting (dataset and model) may not transfer to another setting, reinforcing the need for hyperparameter tuning for each specific task. For example, Figure le shows that
'Some additional results are shown in Section A.2.
'IF (explicit) refers to the original detinition of IF that explicitly inverts the Hessian. Only the last layer parameters used for IF (explicit) due to the memory limit.
5

---

setting epoch as 5 on MNIST+MLP is the best hyperparameter choice while leading to poor performance on 2e
The impact of hyperparameters can be entangled. We find that the choice of one hyperparameter can affect the sensitivity of LDS to others. As an example, the LDS of both TRAK and LoGra becomes more sensitive to regularization for higher wojection-dimensioon
depicted in Figure 1b, Figure Ic, and Figure If. Intuitively, higher projection-dimest dimensson implies that the Hessian or empirical FIM is closer to heing singular, which leads ingly, the entanglement
sensitive attribution scores. Interest- regularization with projection- -imenssoo addresses an open question in TRAK [29]: it was unclear why, counter-intuitively the LDS for TRAK decreases when projection- dimension increases- result suggests that this is due to an insufficient tuning of regularization. Concretely, Figure 1b and Figure Ic indicate that when having high projection-dimenstio (e.g. 8192), TRAK performs poorly with low regularization. However, a higher projection- ultimately leads better LDS when regularization is tuned.
Implicit hyperparameters such training-epoch also play an important role. Figure le illustrates the effect of a largely overlooked hyperparameter, the training-epochl turns out that training-eport of the model used to calculate the attribution scores could significantly affect LDS. However, it is counter-intuitive that a larger coining--ppode could lead to worse LDS, which raises open question for future research.
4 A Case Study on the Regularization Term in Influence Function
In this section, we first analyze the problem of selecting the regularization term in influence functions for maximizing LDS. Based on our analysis, we propose a surrogate indicator that instructs a practical selection algorithm for À without model retraining. Finally, we evaluate the proposed selection algorithm on a variety of experimental settings, demonstrating that it empirically generalizes well for multiple data attribution methods involving the regularization term.
4.1 Problem: Hyperparameter Selection of À in Influence Function
We study the problem of selecting À using the IFFIM version of influence function, TIFIMMA as defined in (2).
Hyperparameter selection as an optimization problem. Recall Definition 2.1, where we have denoted LDS c,(T, A) for attributor т, a test sample z', and samples of subsets A. The hyperparameter selection problem can be formulated as the following optimization problem:
max Co (TFFTM,A, x', A).
However, directly analyzing this optimization problem poses several technical challenges. First, the LDS objective is don-differentiable because it involves the discrete Spearman correlation. Second, the LDS objective depends the specific sample of subsets A, which is difficult to analyze. Third, evaluating the LDS requires model retraining, which we aim avoid.
Relaxing the LDS objective. To address the first two challenges, we propose the Population Pearson LDS more tractable proxy for LDS.
Definition 4.1 (Population Pearson LDS). For an attributor and test example z' E Z, the Population Pearson LDS is defined using the Pearson correlation over all subsets A S of size a:
cp(T,2²)) == PearsonCorr [ (((2,00))ACSSAAIE
{ 1E ==(e):))AcSS|A==))
(6)
ZEA
Note that Spearman correlation is equivalent to the Pearson correlation of the man-trranfformed variables. Therefore, both statisties measure monotonic relationships and are generally expected to be highly correlated [10]. The new objective, Population Pearson LDS,, is differentiable and replaces the sampled subsets A with all size a subsets of S, which makes it easier to analyze. Specifically, we can define the partial derivative of cp with respect to À as
c,();2')= Oc, AFFIMMA.
ax

---

Analyzing the partial derivative (x())²). In our hyperparameter sensitivity experiments in Section 3, we observe that, empirically, the LDS with respect to the regularization term À often has a uni-modal and concave shape. This observation turns our attention from the original maximization problem to the analysis of the partial derivative eg (A; 2²). We first show in the following Lemma 4.2 that our empirical observation about the shape of the LDS as a function of À is not a coincidence.
Lemma 4.2 (Monotonicity in Small X; Informal). Assuming the test sample a' follows a similar distribution as the training data, under some technical assumptions deferred to Section B.4, with high probability over the choice of test sample z', there exists some C > 0 such that for 0 < X C,
cp(A;z') > 0.
Lemma 4.2 suggests that for a range of small À starting from 0. Cp as a function of À is increasing. With the observation regarding the shape of the LDS curve, ideally, can solve for the optimal hyperparameter X as the stationary point with =²"=)=²) = O. However, the partial derivative ;x';))) still involves quantities 0aA for A C S, which require model retraining.
4.2 Analysis: A Sufficient Condition for Monotonicity in À
Instead of directly solving for the stationary point, we derive a sufficient condition (Theorem 4.3) for the derivative to be positive, which leads to surrogate indicator that does not require model retraining (see Definition 4.6 in Section 4.3). Before we state this condition, we first introduce some key notations and their intuitive interpretations.
LOO weighted loss gradients. We let Dg denote the uniform distribution of size a subsets of S. De- fine 9-' == 1 Σi= ag', VoL(z,, 0g)) E R", where agg' E IR with agr, == EA [f 122 E A] 1 EA 0-)) As the name suggests, we point out that agg, can be interpreted as the LOO influence of 2; on z' under certain conditions. Under this interpretation, I2 becomes the sum of loss gradients of training points weighted by their LOO influences on the test example z'. More details can be found in Section B.2.
Other necessary notation. For simplicity of notation in our subsequent analysis, we let U2 denote the gradient Vo V= (0'), and define several bilinear terms of U2', 9z', azg'. They will be used to measure and compare different 9 update directions (see Theorem 4.3 and its interpretations below). Denote fR /. (Fs + AI,) -* Fsu Further, we let T
-0 + AI)) (7)))11 "2' and Oz',A (JJT nAI)) - -zzz,, where J E Raxp has its äth row being VoL 03) The
relationship between (Fs + ATp) and (JJT + ndT,) 1 is embodied in Lemma B.3. Intuitively,
these terms define semi-inner products of V&r, (F's + Alp) -1 92' and agr weighted by matrices related to the FIM.
Now we are ready to state sufficient condition for the monotonicity of Population Pearson LDS. Theorem 4.3 (Sufficient Condition for Positive Derivative). It suffices for (p(2;²') > Of
(8)
Vor'A - 1
Intuition behind Theorem 4.3. Let's first focus on RHS of (8). By observing the definition of (A,e', for k = 1,2,3, RHS can be viewed as the cosine "angle" between by and (Fs- + xTT))=
Here, (Fs + AIz)='uy' is related to the 0 update direction in IFFIM, as in Eq. (2). When RHS small, this "angle" is overly large, suggesting significant misalignment between the A update direction and the gradient descent direction direction. In this case, increasing À helps intuitively because it conditions the inverted matrix Fs better, stabilizing the update direction. When the "angle" is too small, F's is overshadowed by the regularization term AIp (as U22 is roughly parallel to (F's + Uy.), meaning that the attribution method may become dominated by regularization and lose sensitivity to the underlying structure of data. Similarly, LHS of Eq. (8) measures the "angle" between e and the ground-truth direction that involves 9.'- Hence Theorem 4.3 is in essence stating that when the "angle" between Ugr and (Fs + \Ip)) ''_. of comparable level as the "angle" between Vyr and the ground-truth direction, Population Pearson LDS could reach its maximum.
'This involves non -riviiivil technical assumptions that do not always strictly hold. But we have made intuitive and empirical justifications about them.
7

---

Remark 4,4 (Proof Sketch for Theorem 4.3). The detailed proof can be found in Section B.3. We provide a proof sketch here. We start by expanding c,(); z') with the definition of TIFFIM,- Then, We draw connections between Var Var RA (05)), crucial term occurred in the expression of Cp(X;z2) z"), and Fs. Further, we state a condition (Eq. (10) equivalent to 6,(X;z') > Finally, we derive Eq. (8) from Eq. (10) with Lemma B.3 and the generalized Cauchy-Schwan inequality.
Remark 4.5 (Gradient Projection). A result similar to Theorem 4.3 can also be derived for IFFIM attributor with gradient projection (see TIFFIM.A,P P. in Eq. (4)). In this case, given the projection matrix P. tk,x', x should be replaced with u P(P
+ F pppppppppp FsPPT U2 while r'' a should be replaced with -... FsP P A1p) g2¹- Additionally, JJJ nAIn the oz A should be replaced with JPP "PPTT""
J + +ndla. Please see more details in Section B.5.
4.3 Algorithm: The Setrrogate-Indicator---
Practical Selection
The surrogate indicator. The sufficient condition in Theorem 4.3 enables a practical selection algorithm for based on the following surrogate indicator Es/ Definition 4.6 (Surrogate Indicator). We define the surrogate indicator
12,2',A
E = Vis. t1,3AA
(9)
which is the RHS of Eq. (8).
Note that the quantities fi,s', , for h: 1, only depends model parameters 0% trained on the full dataset S, thus Ex can be calculated without retraining. Moreover, both the LHS and the RHS (Exi,)) ,) of Eq. are well normalized.
Proposition 4.7. Ifr O, then LHS and RHS of Eq. (8) both lie in the interval (0, 1).
The practical selection algorithm. In principle, we need both the LHS and RHS of Eq. (8) to evaluate the sufficient condition in Theorem 4.3. In practice, however, find that using εT, [T] Ever E 0.5, where T is validation dataset, the criterion? select the regularization term is uniformly effective across all the experimental settings. This yields a practical algorithm (Algorithm 1) for selecting the hyperparameter À without requiring model retraining.
Algorithm Selecting À with the surrogate indicator.
Input: A candidate set C of X, a subset T C Z test examples.
Output: A selected Å.
for À do
2:
Compute E for all z' E T;
3: Ex.A + I Ever Exi
4: end for
5: À + arg minxee L 0.5|:
4.4 Experiments: Validating the Proposed Surrogate-Indicato-- Selection Algorithm
We present experiments validating the proposed algorithm for selecting the regularization term A.
Experiment settings. Dataset, model, and projection dimension: We employ 6 experiment settings, covering different datasets, model scales, and projection dimensions. First two settings apply Logistic Regression models (I.R) on MNIST dataset [25], where one utilizes random projection with dimension 512 and the other uses no projection. They are followed by two settings that apply MIP,, with random projection 512 and 4096 respectively. Finally, we have two relatively large models, including ResNet- 9 [14] CIFAR-2 dataset [24], and Transfrrarrrrmer (MT) [2] on MAESTRO dataset [13]. We fix the random projection dimension to 4096 in the last two settings. We also conduct experiments
In fact, setting the threshold as 0.4 or 0.6 is similarly effective. The & x as function of log À looks like logistic function and it varies sharply when its value around 0.5. See Section C.I for visualizations.

---

MNNSS
projection)
0.34
0.20
0.20
0.30
0.22
0.00
0.06
0.05
0.36
0.34
0.00
0.12
(5- 4006)
(p- 4006)
(f 4096)
0.30
0.20
0.14
0.20
0.15
0.30
0.10
0.85
0.00
0.05
15
Figure 2: The plot of LDS versus À the IFFIM attributor. Each subplot corresponds experiment setting. The red solid vertical line indicates the X selected by our method. The gray dotted lines indicate the eigenvalues of Fs corresponding the 10%, 30%, 50%, 70%, and 90% quantiles. The quantiles are annotated in gray color near the x-axis.
MNIST
MNIST projection)
0.22
0.30
0:25
0:38
0.10
9.34
0.32
0.00
10'
MNEST MIP (6 4096)
CIFAR2 Reslitt (6 4096)
MAESTRO MT 40-6)
0.20
0.38
0.26
19
0.15
0.05
0.32
10-4
30
10-7
10-0
30'
10-1
10-1
1.7
10
10-
10a
107
Figure 3: The plot of LDS versus À on the TRAK attributor: Please see Figure 2 for the plotting setup.
on WikiText2 [28] with GPT2 [31], the results of which can found in Section C.4. Attributor: We experiment with the IFFIM attributor defined in Eq. (2) (or Eq. (4) with projection), and the TRAK attributor [29]. Baseline selection method: We also compare the proposed algorithm against baseline method that selects À based on the spectrum of Fs since the main motivation of À is to deal with the singularity Fs. Specifically, baseline method set to be eigenvalue of F& at fixed quantile among (10%, 30%, 50%, 70%, 90%).
Results. We demonstrate the experimental results on the IFFIM and TRAK respectively in Figure and Figure 3. In each subplot, the blue curve represents the change in LDS with respect to X. We mark

---

the À chosen by the proposed algorithm by the red solid vertical line, and the baselines in the gray dotted vertical lines. Overall, we observe that the À chosen by the proposed algorithm generally leads to good LDS value that is significantly better than À 0, and in many case it is close to the optimal value. Moreover, while the proposed algorithm is based on an analysis of the IFFIM attributor, generalizes well to the TRAK attributor under various experiment settings with different projection dimensions, In contrast, there no single fixed quantile for spectrum-basee baseline method that consistently performs well, indicating that this naive heuristic is insufficient for selecting A.
Downstream tasks performance. We also include evaluation of proposed algorithm downstream settings, and the detailed results are presented in Section C.5. We focus on the task of data selection commonly used in data attribution literature [11], where measure the performance drop of models when the most positively influential training data are removed. We note here that a larger drop signifies stronger attribution quality. In general, we observe that proposed algorithm achieves visible model performance drops, demonstrating its effectiveness in downstream tasks.
5 Discussion and Conclusion
This work brings attention to fundamental yet overlooked challenge in data attribution: the uniquely high computational cost associated with hyperparameter tuning. Unlike typical machine learning models-w wwhre validation metrics can be cheaply often requires repeated retraining on data subsets, making standard hyperparameter tuning procedures impractical. By systematically characterizing this issue, we contribute the first large- scale empirical study on hyperparameter sensitivity in data attribution methods, and establish this hyperparameter tuning bottleneck as a critical concern for their practical deployment.
data attribution quality
Through extensive experiments across diverse methods, datasets, and models, we demonstrate that many data attribution methods are indeed highly sensitive to hyperparameter configurations, with opti- mal choices varying significantly across settings. For example, we show that some hyperparameters such as regularization and projection ciimeniion-interae in nontrivial ways, amplifying sensitivity and necessitating more careful tuning strategies. Moreover, implicit factors (such as training epoch) matter; comparing attributions across checkpoints can be informative.
To mitigate the high computational cost of standard evaluation metrics, focus on the regularization parameter in influence function- based methods and propose a practical selection procedure based on theoretically motivated surrogate indicator. Our method avoids model retraining and shows robust empirical performance across multiple benchmarks and attributors, making it practically useful tool for tuning one of the most critical hyperparameters in popular data attribution methods.
Limitations. Our theoretical analysis and surrogate-based tuning procedure focus specifically on the regularization parameter in influence function methods. Extending similar ideas to other hyperparameters or alternative data attribution methods remains an important direction for future research. Moreover, our empirical evaluation primarily relies on the LDS metric. While we believe LDS provides strong starting point for highlighting the practical challenges of hyperparameter tuning, it would be valuable to investigate whether similar sensitivities arise under alternative evaluation metries.
Acknowledgment
The authors would like to thank Juhan Bae for helpful discussions that partially motivated this project.
This project was in part supported by NCSA Delta GPU at NCSA through allocation CIS240402 from the Advanced Cyberinfrastruuctrr Coordination Ecosystem: Services & Support (ACCESS) program [5], which is supported by National Science Foundation grants #2138255 #2138286. #2138307, #2137603, and #2138296. HZ is partially supported by an NSF IIS grant No. 2416897 and Google Research Scholar Award. The views and conclusions expressed in this paper are solely those of the authors and do not necessarily reflect the official policies or positions of the supporting companies and government agencies.
10

---

References
[T] N. Agarwal, Bullins, and E. Hazan. Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research, 18(116): 2017.
[2] H. C.-Z.. Anna, V. Ashish, U. Jakob, S. lan, H. Curtis, S. Noam, M. Dai Andrew, D. Monica,
R. Douglas, et al. Music transformer: Generating music with long-term structure. arXiv preprint, 2018.
[3] Bae, W. Lin, Lorraine, and R. B. Grosse. Training data attribution via approximate unrolling. Advances in Neural Information Processing Systems, 37:66647 -66666 2024.
[4] S. Basu, P. Pope, and S. Feizi. Influence functions in deep learning are fragile. In International Conference on Learning Representations, 2021. URL httpa::://////eeeieeeeeee net/forum? id=xHKVWHGDOEK.
[5] T. Boerner, S. Deems, T. R. Furlani, S. L. Knuth, and J. Towns, Access: Advancing innovation: Nsf's advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience Advanced Research Computing, pages 173-176. 2023.
[6] S. K. Choe, H. Ahn, J. Bae, K. Zhao, M. Kang, Y. Chung, A. Pratapa, W. Neiswanger, E. Strubell, Mitamura, et al. What is your data worth to gpt? llm-scale data valuation with influence functions. arXiv preprint arXiv:2405.14 13954, 2024.
[7] J. Deng. S. Zhang, and J. Ma. Computational copyright: Towards a royalty model for music generative ai. arXiv preprint orXiv:2312.06646 2023.
[8] J. Deng, T.-W. Li, S. Zhang, S. Liu, Y. Pan, H. Huang, X. Wang, P. Hu, X. Zhang, and J. Ma. dattri: A library for efficient data attribution. Advances Neural Information Processing 2024.
Systems, 37:136763-
[9] J. Deng, Y. Hu, P. Hu, T.-w. Li, S. Liu, J. T. Wang, D. Ley, Q. Dai, B. Huang, J. Huang, C. Jiao, H. A. Just, Y. Pan, J. Shen, Y. Tu, W. Wang, X. Wang, S. Zhang, S. Zhang, R. Jia, H. Lakkaraju, H. Peng, W. Tang, C. Xiong, J. Zhao, H. Tong, H. Zhao, and J. W. Ma. A survey of data attribution: Methods, applications. and evaluation in the era of generative ai. 2025. doi: 10.2133/ssmm.545 1054. Available at SSRN: https: httpp://ssttt //ssrn//s.n.ecee com/ abstract=5451055
[10] A. Field. Discovering statistics using IBM SPSS statistics. Sage publications limited, 2024.
[11] A. Ghorbani and J. Zou. Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning, pages 2242-2251. PMLR, 2019.
[12] R. Grosse, J. Bae, C. Anil, N. Elhage, A. Tamkin, A. Tajdini, B. Steiner, D. Li, E. Durmus,
F. Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308. 03296, 2023.
[13] C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C.-Z. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck. Enabling factorized piano music modeling and generation with the maestro dataset. arXiv preprint vrXiv:/8/00 /2247, 2018.
[14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[15] Y. Hu, Hu, H. Zhao, and Ma. Most influential subset selection: Challenges, promises, and beyond. Advances in Neural Information Processing Systems,
1177788-199100 10, 2024.
[16] A. Ilyas, S. M. Park, L. Engstrom, G. Leclerc, and A. Madry. Datamodels: Understanding pre- dictions with data and data with predictions. In K. Chaudhuri, Jegelka, L. Song, C. Szepesvari, Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9525-9587. PMLR, 17-23 Jul 2022. URL //pocceed/ss/ccs/ssss mlr. press/vll 11lv/v162//01/ 11yas22a.......a html.
[17] A. Jacot, Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[18] R. Jia, D. Dao, B. Wang, F. A. Hubis, N. Hynes, N. M. Gürel, B. Li, C. Zhang, D. Song, and C. J. Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1167-1 176. PMLR, 2019.
[19] K. Jiang, W. Liang, J. Y. Zou, and Y. Kwon. Opendataval: a unified benchmark for data valuation. Advances in Neural Information Processing Systems, 36, 2023.
11

---

[20] R. Karakida, S. Akaho, and S.-i. Amari. Universal statistics of fisher information in deep neural networks: Mean field approach. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1032-1041. PMLR, 2019.
[21] Y. Klochkov and Y. Liu. Revisiting inverse hessian vector products for calculating influence functions. arXiv preprini arXiv:2409. / 17357, 2024.
[22] P. W. Koh and P. Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894, PMLR, 2017.
[23] W. W. Koh, K.-S. Ang, H. Teo, and Liang. On the accuracy of influence functions for measuring group effects. Advances in neural information processing systems, 32, 2019.
[24] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[25] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- Bradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (0(11):2278223 2324, 1998.
[26] J. Martens, New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 216146):1-7,, 2020.
[27] Martens et al. Deep learning via hessian-free optimization. In Ieml, volume 27, pages
735-742, 2010.
[28] S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.0743 07043, 2016.
[29] S. M. Park, K. Georgiev, A. Ilyas, G. Leclerc, and A. Madry. Trak: Attributing model behavior at scale. In International Conference on Muchine Learning, pages 27074-27113 13. PMLR, 2023.
[30] G. Pruthi, F. Liu, S. Kale, and M. Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-9999,,, 2020.
[31] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[32] L. Sagun, L. Bottou, and Y. LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv preprint arXiv:1611.07470 2016.
[33] A. Schioppa, P. Zablotskaia, D. Vilar, and A. Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8179-8186, 2022.
[34] C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):22-111 1904.
[35] Wang, P. Mittal, D. Song, and R. Jía. Data shapley in one training run. In The Thirteenth International Conference on Learning Representutions, 2025. URL https:/ //tp:::////ppeeeiieee net/forune Hd=HDbb...87y.
[36] D. P. Woodruff al. Sketching as a tool for numerical linear algebra. Foundations and Trends@ in Theoretical Computer Science, 10(1-2):1-15) 2014.
[37] M. Xia, S. Malladi, Gururangan, S. Arora, and D. Chen. Less: selecting influential data for targeted instruction tuning. In Proceedings of the 41 International Conference Machine Learning, pages 541 54104-54132, 2024.
[38] R. Zhang and S. Zhang. Rethinking influence functions of neural networks in the over- parameterized regime. In Proceedings of the AAAI Conference Artificial Intelligence. volume 36, pages 9082-9090, 2022.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have tried our best ensure that the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
We have diligently ensured the accuracy of the theorem statements and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our code is publicly available at data atttibutiinnnnn
com www..ooomTTTTaaaae
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification: Our code is publicly available at https://github: www.eaa....TTTTTTTTTT Lab/ data- weta--attrbtiiionooee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: gwwgg//ies////// /Cooeeubeiiiieeee swwiisssssssiiiiieee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details of the experiments are discussed in Section 3, Section 4, Section A, and Section C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification: The experiments require retraining number of models, which is too computa- tionally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
15

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The information on the computer resources is provided in Appendix A.3 and Appendix C.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ : http:///euiipppeeeee httppplic//tiisuuiieeeeeopo
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and confirm that the research conducted in this paper adheres its principles.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: The paper aims advance the field of machine learning, and we do not anticipate any immediate societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the datasets and included their licenses in Appendix A.3 and Appendix C.2.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datastts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The paper does not involve LLMs as any important, original, or non- non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
19

---

A Details of the Hyperparameter Study
A.1 Hyperparameters Definitions
Apart from major hyperparameters described Section 2. 1, some additional hyperparameters are considered in our hyperparameter sensitivity study. We provide their definitions as following:
We may normalize the per-example gradients in TracIn by dividing their L2-norm. In other words, normalization=TTnu means that PTracIn(2' 2;)
normalization (TracIn) [30]:
Σ; / &
Vof(=' 0.)
VoL(2.0))
checkpoint-seleccion (TracIn) [30]: The checkpoints used to calculate TTracIn can be designed in various way. In this paper, we select last epochs different maximum training epoch numbers.
max-iteration (IF (CG)) [22]: This hyperparameter specifies the maximum number of steps the conjugate gradient algorithm will take to attempt convergence. In the experiment, we do not set stop criteria so that the algorithm will stop at the maximum number iteration.
scaling & recursion-depth & batch-size (IF (LiSSA)) [22]: The LiSSA algorithm for inverse Hessian vector product (IHVP) an iterative algorithm where each iteration t applics the formula E' g + (I _ (H' + AZ))u'= and g is the target vector. The hyperparameters scaling refers to y7, recursion-depth indicates the maximum f where the iteration stops; batch-size indicates how many data points are used to calculate the batch-wise hessian matrix H!.
Default values. In the following table, we list the default value of each hyperparameter. The default value is used when other hyperparameters searched. They selected according the default value in original paper.
Table Default values of hyperparameters.
TDA method
Hyperparameter Name
Default values
regularization
0
TRAK-10
pojection-dimense
512 (2048 for WikiTex12+PTT2
training-epoch
50 for WikiText2+GPT2)
regularization
le-3
LoGra
642
training-epoch
3
regularization training-epoch
le-5
IF (explicit)
50
regularization max-iteration
le-2
IF (CG)
10
training-epoch
50
regularization
le-3
scaling
5
(LiSSA)
recursion-depth
1000
batch-size
50
training-epoch
50
normalization
False
TracIn
pojection-dimension checkpoint-selection
None (i.e., no projection) last 10 checkpoints
20

---

A.2
Additional Results
Figure 4, we present additional results not included in Figure due space limit main text. These additional results confirm that most TDA methods are sensitive to certain hyperparameters. Intriguingly, training-epoch is one of the most sensitive hyperparameter for most TDA methods.
0.26
0.20
0.25
0.18
0.24
LDS
0.24
LDS
0.16
0.14
0.20
0.12
0.10
0.75
1.00 1,25 1.50 1.75
2.00
-2
iter (log10)
training epoch
regularization (log10)
(a)
MNIST+MIPP Attributor:
(b) WikiText2+GPT2. Autriutoor LoGra. HP: training-epoch.
(c) WikiTex12+GPT2. Attributor:
(CG). HP: max-iteration.
TRAK: HP: regularization.
0.20
0.34
0.16
0.15
0.33
0.15
0.34
LD5
normalzedd grad- nomalized, grw.ggraddduuoo True
0
0.13
0.12
0.11
0.10
0.09
2.75
3.75 projection dimension (log10)
projection dimension (log10)
checkpoint_sekee selection
(d) WikiText2+GPT2. Attributor:
MNIST+MLP. Attributor: (f)
MNIST+MLP. Attributor:
TRAK; HP: prPpprojctiionniimmesioon
TracIn; HP: normalization, TracIn; HP-checkpoint-selectin
projectin-ddimesion..
0.25
(explick)
0.05
0.05
(CG)
(USSA)
0.00
0.00
10a
107
batch size
training seaiiing_epocie epoch
(g) MNIS++MLP Attributor: (h) MNIST+ML.P. Attributor: IF.
IR(iSSA). HP: batch-size.
HP: training-
Figure 4: Study of hyperparameter sensitivity additional results.
A.3 Computational Resources and Dataset Licenses
The experiments for the hyperparameter sensitivity analysis are done on 4 A100 GPUs in around 100 hours, excluding model retraining (we reused some model checkpoints provided by the dattri library to avoid extensive model retraining). For the dataset we use: MNIST-10 dataset holds CC BY-SA 3.0 license; CIFAR-10 10 dataset holds CC-BY 4.0 license; WikiText2 dataset holds CC BY-SA 3.0 license.
21

---

B Omitted Details of the Theoretical Analysis in Section 4
B.1 Relationship between TRAK and IFFIM
As noted in Section 2.1, TRAK can be viewed as variant of IFFIM with additional computational tricks. This section demonstrates their similarity and distinctions in more details.
We first formally introduce the TRAK attributor without gradient projection as follows..
TTRAK(2', = Vef(z', 03)
( ( - V=f(=,,63) - (1 Pi),
where & E RRxP has its ith row being Va 0g) ,R
and p: == p(z;, 0z). main differences. First, the right most gradient term in TRAK is Vef (1 - p.) while its counterpart in IFFIM is 0z).
Compared the IFFIM attributor in Eq. (2), there
Second, IFFIM has the empiricla FIM Fs in the middle while TRAK has & R2.
The right gradient term difference. We first show that 0o)(2103)(1 pt) is equivalent to Ve (1,, 03) as
VoL(z, = 0ln(1+6-1)
Vof(2,,0g) = -1 p(z,
0f
(=f(x.0g))
The middle term difference.
The middle term TRAK can be written
1 _T R$
(2) vo)) (1
n
n
Noting that p:(1 - p.) is the Hessian of cross-entropy loss with respect to model output, and the whole 1 R known as the Gemeralized Gauss Newton (GGN) matrix [26].
Furthermore, Martens [26] has shown that 1 TJJ (the empirical FIM) and 1 R& (the GGN) both n
n reduce to the true FIM when S converges to the true underlying distribution of 2.
This comparison indicates that there is fundamental similarity between TRAK and IFFIM despite superficial algorithmic differences.
B.2 LOO Weighted Loss Gradients
As suggested by Park al. [29], we assume decreasing by does not shift the distribution of f(=', (1),,0)),,a 0)),)..
EAND. [f('', 22 & A] EA-D.-, if(2 12 A].
Then, by rewriting the RHS, we have
EAND. 02)2 $ A] = EAND EA-D 0 (2 E A).
and hence
Ofgi,i = EA-D, (E)2002)2 E A] EA-,,
EA 01)12 AA Pr |zi AJBA-D, (5(22(0)))> E A] Pr I & AIEA-D,1((E,,01 AA
AND.
AND.
Pr | $ Al ((2 E E -LAAAD,,((2 0,) $ A))
AnD.
=(1 -1- (U-_)AAD, (7) - (x(.))) A),
which means that the weights a=', in g can be interpreted as the LOO influence of 2; on z'.
'In practical implementation, Park et al. [29] dropped the diagonal matrix R due to slightly improved empirical performance, and projected & and the gradients to lower dimension for computational efficiency.
22

---

B.3 Proof of Theorem 4.3
prove Theorem 4.3 before Lemma 4.2 better readability. We first establish several intermediate results for (8).
Lemma B.1. We have
D,, (VRRA(05)
n Fs, a(n - 1)
where Da is defined in Section 4.2.
Proof. By the optimality of 0g. know (ols(V VoRs(03) 0. We also have EArD.IExeA (eL(2,03)) 0.
Now, because A is uniformly sampled size a subsets of S,
VarA~D, (o.... VarA~D. -
VoL(2,03)]
ZEA
BAND, V Vol(2,03)) Vol(2,03)) 03))
sEA
2EA
a² (aBeus VoL(z, Vol((z)))
BAND.
Σ
V0L(21,0)) 03)
T DD.
21 #1
The second summand can be reduced to
EA-D. Σ
(])
E
21522
VoL(21, (8) VoL(2,)) 03)
AEDo EιEA E
#1
1
n 2
2
2
#1722
a(a 1)
VoL
n(n 1)
$1321
a(a - 1)
VoL(z, 03))
voL(2,03))
Σ Volle,
n(n - 1)
SES
#ES
#ES
a(a 1)
E V L
"1,
where D2 means is a size subset of S. Finally, by the definition of Fs.
VarA~D, (oRA
1 (a Fs d²
a(a 11)
Fs)
a Fs. a(n
n
Lemma B.1 facilitates analyzing the variance of Σ AEA TFFIMA in (q), thereby enabling verification of the following equivalent condition of (2; > 0.
23

---

Proposition B.2. For a test example z', cp(A; 2/) > 0 is equivalent to Te 1 > (~Vef(z', 03) (Fs X1p) t2,2'
(10)
Proof. We first simplity ΣEEA 7FFIM, 2) in (6);
Σ
xF) =
--> - Vos(= (00) (Fs + AT,) ( VoI[(2
ZEA
sEA
Vef(z' 03) (F's (
Three terms are involved when we expand the definition of (T(TFFIMA)) The first is the numerator:
EANDA
TPFIMA(('', (f(2',0) EF
ZEA
=AADD. BEA-DI--Voffle Vof(z'' 03) (F's + xIp) - (VeRA (f(2', (1) 1a=
where the component that depends A
EAND, (V) (f(2', 0%) BAIDD. x-131l
(f(2' EN 02)11 Σ
VoL(2,85)
AED
ZEA
I E Al(f(E',0)) - EA-D, Vll 03) t=1 DEDo
-
n
- -f E AlVoL(2, 03))
EA,D, (f(22,0))
ar' Vo PoL(2z f3s) = 9r'
Note that we apply (1) = n (--))
The second term is the variance of ΣsEA (FFIMA((
VarA~D.
Σ (2',2)
ZEA
IarA- Vof(z', (00(22'0)))TT))))))() MF)
= Vef(z'00z) (F's +
a(n - a) Ves(z', 03) T ((FF + N1p)
where we apply the identity Lemma B.1.
The third term is the variance of f(2', ,84), which, together with in the numerator and a(n-a) in the variance of ELEA TIFFIM,A22' 2), are omitted because they do not depend X. To summarize, so far we have shown that 2p(): > is equivalent to
n-1
vof(z',03) (Fs 21.)-1 9-' a Vof(=', ( (Fs X1,) Fs(Fs + M1, Vol(2',
0
>0
By direct calculation, the left hand side is
(~Vof 1
03)) ([ +1,) 11 - 0%)
(Vof(=',0)) (3) + -
24

---

When the positive denominator is dropped, we obtain the desired formula.
Before we move to the proof next result, we first introduce useful matrix transformation lemma. Lemma B.3, For matrix X E RR*,,, non-negative integer k. and > O, we have
(X X X1p) = -x AI
Proof. Note that
(XX + AL,) = X XX +AXT = [x + X
we left multiply (XTX- X + M1 and right multiply (XXT A on both sides, we derive
(x x + = -XX"=xXXXXXXXXXXXX XXXXXX
A
By applying the equality & times on (x x + 11p) x to "push X through" the inverted matrix, we complete the proof.
Proof of Theorem 4.3. Because Fs is a positive semi-definite matrix, so is (Fs + ATp) - Fi for any positive integer k. result,
1k,2'AA Vof(z'
+ FF
0.
In fact, A,s 0 VsV 0g3) 0 implies 12,=',> 0 which results in undefined cz. Therefore, thee > O. Further, by the premise, we have
12
O2 1, > 0.
Without loss of generality, only consider the case where Vof((z' -o because otherwise Eq. (10) holds automatically as its left hand side would be positive while right hand side would be non-positive, which guarantees 6p (A;=") > 0 by Proposition B.2.
X (f)) 0,
We proceed by rewriting Eq. (10) with Lemma B.3. By setting
J/vū, we have
13,2',A 1 n JT + - 3V
n
-o (Fs + AIp) -2) Vol(=',00 JT 1 JJJ
where we apply the fact that Fs 1 JJ = X x and g. 1JJTa a. by their definitions. Then, note that
n
13,2'> Oz',A
Vof(El,ogTTT( JJJ + - 3.Vo 03))( a JJT + -Nn) (11)
a
n
2 Vos((' 03) ( + - Vof(=' (3''005))) 03) (F's + X1)) -2 > 0,
and azr with inner product defined by positive definite matrix JJ + AI.)11 Now, we finish the proof by observing that (10) is derived directly by multiplying Eq. (8) with Eq. (11) and rearranging
by applying the generalized Cauchy-Schwarz inequality on - =.
terms.
25

---

B.4 Assumptions and Proof of Lemma 4.2
We base the discussion of Lemma 4.2 on Eq. For simplicity, we first eliminate V1,r' on both sides to state the following sufficient condition for (,(A; > 0:
12
(12)
VOz'A
V
Now, for 1, ...., (1, p). we let μ4 denote the ith largest eigenvalue of P's if S p, and otherwise. We have the following result for μ4:
Lemma B.4. For = 1.2....,ny n, P4 is the ith largest eigenvalue of JJJ
Proof. Write the singular value decomposition J UEVT with En a; for [min(n.p]] As Fs = __ LVV EVT we know o; yn μ4 for i [min p3]. Further, ПΣΣ diag S min - diag(u:) because μ4 0 for p. We finish the proof by noting that JJ LU.
To better establish a relationship between the spectrum of Es and ep (TFFFIM)))) we first make following definition for convenience.
Definition B.5. For a vector E R' and E [max(n, pH. let ün denote the ith component of under chosen eigenbasis of Fs, where eigenvalues are sorted descending order if S p: é; otheriwise. Further, let Umin denote the component of v under the eigenbasis corresponding to the minimal non-zero eigenvalue Jmmin of Fs-
We then provide a result on the expectation of VoLi(z,03) 03)?, the square of the ith eigen-component of VoL(z, 0z), when ranges over S.
Lemma B.6. For 1,2,.... ...,, P.
have
E Eg- (oL,(2,0)) = 14.
Proof. We adapt the proof from [6]. Let's write the digom-decomnossttonn Fg:
Fs = VAVT
where A = diag(u H Then
V V FsV
= V V VoL(z,
TJV = BV VoL(2,00)) VV]
= Exns
i-==
We obtain the desired equality by comparing diagonal terms.
Next, we introduce an assumption regarding the concentration of distribution of (2L,(2',03) when the test example z' is sampled.
Assumption B.7. Assume there exist constants 0 < C₁ < C2 such that for i = 1,2....,
C.E.ss VoL 03) VoL.(2', 03) < C2E 05)
(13)
with high probability over the sample of z' from the test distribution.
Assumption B.7 basically assumes that the distribution given by S represents the test distribution well regarding the relative size of eigen-components. Then, we are able derive an upper bound for the RHS (right hand side) of (12).
26

---

Proposition B.8. Under Assumption for all X > O, we have
nC²
(Wmin + 2)3/2
RHS < (1 p(='= 03) VCT
(14)
Puin
with high probability over the sample of z' from the test distribution.
Proof. is known that the ith component of JV o 88)) under the eigenbasis of
1 n JJT is precisely
V4.14. Vol.(2 = vn p4 p fgg - Vol,(=',83)
for = 1.2.....P2. n. This can be shown by employing the singular value decomposition J UEV.. Under the eigenbasis of 1 JJJ 1
U JVof(E' EV Vzf(2',05) -
so the identity holds because Σn Vn Pi for E [n, pH| (see Lemma B.4) and μ4 0 for
i i>p p.
For the numerator, with high probability,
(3) [ JJ + -V))-=VVo((e (3)) 111 - p(z1,03))2
12, n
(Vn 14 (z))(2 (1i + A)2
1
C2p2
(1 p(21,0)))
(14 + X)2
Σ
TAms 7 #002
(1 p(21, 1)
n
Cz.
p(2', 0z)
For the denominator, with high probability,
1 Vol((' T
1 (-JT + - JVol(E 03)
13 =
n PS Vo 03))
n11 - p(21,0g))²
(M, + A)3
1
Cip²
p(2',0g) 0z))²
(14 + X)3
Cip
(1 p(2', )) (Amin + A)3
We finish the proof by combining these two bounds.
We move to analyze the LHS (left hand side) of (1.). Before that, we need two additional technical assumptions.
Assumption B.9. Assume there exist constants C3 > 0 and Eg > such that for 1, .......n. n,
22 < C334 1+=0
(15)
Remark B.10. Because g2 LJT Ozz', for μs equal to 0, IZ must also be O, which can easily shown through the singular value decomposition of J. Hence, Assumption B.9 holds when C3 is large enough. However, we note here that the sufficient condition for cnp z') >> O becomes tighter when C3 is larger and Eg is smaller (see Lemma B.15 for details).
27

---

Assumption B.11. Assume azy is in the column space of J.
Remark B.12. A special case for this assumption when J has rank n - 1, where the deducted 1 rank is because Ei=1 VoL z,, 5g) = 0. In this case, since Di=1 Oz = 0 by simple computation. Ogg is in the column space of J.
Lemma B.13. Assume Assumption B.11 holds. For E [m], if p4 then the ith component of aus! under the eigenbasis of JJ is also zero.
Proof. Write the singular value decomposition = UEVT, Let a.' JB for some B E RP. Since 1 JJJ LUEE UT, the ith component of ar under the eigenbasis of JJJ is
(UTaz) - (UT B): (EV 3);
which is zero for μ4 = 0 by Lemma B.4.
Now are ready to derive an upper bound for Oz',A- Proposition B.14. Under Assumption B.7, Assumption B.9, and Assumption I, for all X > 0,
Can
O
T-
(16)
Pmin
Proof. Denote War's as the ith component of as under the eigenbasis of 1 JJJ for i E [n]. We
first show that for [n], #4 = then Wz', = 0; if μ4 * O, (then S p by definition of p4)
Wer. = besaynn/ju, For [4s = 0. Lemma B.13 guarantees that the component of ay is 0. For N4 + 0, with the singular value decomposition of J in Lemma B.4,
ش., (VT x az); 0:W2'' W2
1 n
for i E [min p}}. which gives the desired result. Therefore,
1 J Mn) ag = + O21,A -0 (
3/44
Cau
Ea
Can
-Eie Tu + f0] 2m
Elms
Ils # O]
I-
1'min
Finally, by combining all the results, we state the following formal version of Lemma 4.2.
Lemma B.15 (Formal version of Lemma 4.2). Assume Assumption B.7, Assumption B.9, and Assumption B.11 hold. With high probability over the sample of z' from the test distribution, Tz lim Te A exists. Further assume To > Then if Umin < Cp where Cp. some positive value depending on 2', there exists some C 0 such that for 0 <<C.
cx(A;2)> >0>00 0.
Proof. We first show that with high probability the limit exists. By expanding T=r,A = -Vof(z', + >lp)-'ge g2r under the eigenbasis Fs. with high probability, the absolute value of the summation term corresponding μ4; for E [min(n. p}] is
Voli(z',0))a 05)==3 - VC2Cs 1/24(1+6g)/2
V2Cs
50/2
S 1 1-p(2', 0/22'033
#4 X
1 -p22p(' 03)
Additionally, for > n, D2' from the singular value decomposition of J. As a result,
limx-oo T exists. As r2 O, there exists C* 0 such that for all 0 X < C* T 2r Now, let
Cn = T2.0t (1 (221083)) /2/6, > 0,
2nC2yn C3
28

---

and
C: min(C" Iz' (1 - (002''00) 01
y2/3,
1-cu/3
Hmin
2nC2v/1 C3
By direct calculation, if pmmn < Cp. then C > 0. Further, when 0 < <xcC
A (1-8y1/2
1,
(1-es)/2
nC2
(Amin + 2/8/2
LHS
Vn- C3 M
Mmin
RHS,
2/n- C3
(1- (1-p(z'. 11=. (3))
pmin
which implies 6g(A;z)>00
Remark B.16. Our analysis relies the condition that Pmiin the smallest non-zero eigenvalue of Fs. remains small. This assumption aligns with established analyses demonstrating eigenvalue concentration near zero for both the FIM and Hessian in deep neural networks near convergence [32, 20). These results empirically justify our treatment of pmin-
We subsequently focus on a discussion of the positivity condition 10 > 0 which constitutes the remainder of this section.
Discussion Tsrott > O. Here we show that, in the special case where z' = 24 E for some i E [n], we have +x > We write the singular value decomposition UEVT. Then by Lemma B.4,
lim Vz'AA lim 1 VoS('' 00)) J 1 [JJJ + +nn)) X
JJ
A-0 n
n
VoL(E 03) T J L-JJ aa
lim
x a11 - pb)
1
lim e
T JJ
JJJ + +1m))
pt x-0+
n
1
n
lim
Σ
(U (U);(UTar))
X
Uí
+
j=1
Σ
Ias + UU (t) Taz);
j=1
where e; is the ith standard basis vector. By Lemma B.13, jjj = 0 implics (UTaz) az'; = 0. Therefore,
0z'',, T I- pi e, UU az 1-p: > 0,
1
lim --
because Pi < 1 and ag az,i = EA-D, D. [f(z;, 0z) E A] EA-D,, 02)] is the expected change in model output of 2; itself when 2; is included in the training set, which is positive.
Empirical assessment of constants in assumptions. The results derived above are dependent on the constants C1, C2 and C3 introduced Assumptions B.7 and B.9. Here we provide brief discussion on the empirical behavior of these constants, showing that they generally stay around the level of 0.1 to 10 instead of scaling with the smallest eigenvalue M'in- Note that, in this empirical study, we compute C₁ with respect to the smallest eigenvalue rather than all eigenvalues, which is reasonable because the only C₁ term that occurs in the theoretical derivation the one related to Mmin Further, due to computational limitations, we consider projection dimension 4096. We fix Eg = 0.5.
To empirically estimate these constants, we first diagonalize PJ FTFFPP to obtain its eigenvalues and execute a data attribution pass with IFFIM to compute VeL(z', (3) and ga for z' in the test dataset. that Then, we follow (13) and (q.(1)) to obtain values of C₁ C2, and C3 that satisfy these inequalities for all or given percentage of test examples. For ResNet-9 [14] on CIFAR-2 dataset [24], we find that C3 0.4082 is enough for Eq. (15) hold on all the test examples, and C1 0.3121 and C2 51.9933 satisfy Eq. (13) on 90% of the test examples. In comparison, the smallest eigenvalue is only about 10-4 For is enough for all test examples, and 90% of test examples yield C₁ = 0.0518 and C2 = 21 21.5589. In comparison, the smallest eigenvalue is only about 10-5, These results suggest that, even in highly non-convex settings, our assumptions tend to hold empirically.
[2] on MAESTRO dataset [13], we find that C3 0.0156
29

---

B.5 Effects of Gradient Projection
Our derivation does not rely on specific properties J. When incorporating gradient projection via Eq. (4), outlined in Remark 4.5, the projection matrix P can be systematically applied to all gradient terms. We now analyze how gradient projection affects Amin the smallest non-zero eigenvalue of Fs (equivalently, of JJJ Under projection, this eigenvalue corresponds to the smallest non-zero eigenvalue of !.JPPTJ"
In practical applications, gradient projection approximately preserves the dominant eigenvalues of Fs. Consequently, when Fs exhibits vanishingly small (but non-zero) eigenvalues near index min n,, p}. the projected matrix _PPPT JT retains a comparably small non-zero eigenvalue, validating the critical condition in Lemma B.15.
Unlike Amnodi-based method [33] that aims to preserve top cigenvalues by design, we resort to the standard subspace embedding results [36] for random projection based methods [6, 29]: For common choices P (e.g. Gaussian random projection) and < Erp: 8 < 1, if = O((n in(1/6))e
then with probability 1 - 6, P satisfies that for any R",
(1 - =Tp)llJ Ful2 S || J Tullz S (1+ Erp))] 0112.
This implies, by min-max theorem,
(E==:)) MM(PPTT S (1 (x))))))JJJ))
where (.(+) stands for the ith biggest eigenvalue. This guarantees approximation of top eigenvalues.
B.6 Proof of Proposition 4.7
Proof of Proposition 4.7. We utilize Lemma B.3 to obtain
t A Vef(z', 03)) J { + *A,))* (Vof(e
1
1 Vof(z', JJ 1 JJJ + -
T=
n
where & 1,2,3. Further,
1 -a -a-(-JJ -JJ + Aln)
4
We point out that LHS of Eq. (8) lying in [0, 1] is the direct result of applying the generalized Cauchy- Schwarz incquality on ff (z', (gg) and ag with inner product defined by positive definite matrix
[ + AI,)= assuming 7 > O. Similarly, the RHS of Eq. (8) is shown bounded when the inequality is applied on - + AI,) ( -.. V (2 03)) and VV , 0t) with the same inner product. We use the fact that 2 2 0 as (LJJ T + >1n) -2 is positive semi-definite.
C Details and Extra Experiments of the Surrogate Indicator
C.1 Visualization of the Average Surrogate Indicator
To provide better insights about the proposed surrogate indicator, we present the curve of εr, as function in Figure 5. In all experiment settings, empirically, ErA a monotonic function of with transitional phase near ÊT. = 0.5, indicating good sensitivity to À around this value, and supporting our choice of threshold in Algorithm 1.
C.2 Computational Resources and Dataset Licenses
The experiments the surrogate indicator are done on an A40 GPU in around 10 hours, excluding model retraining (we reused some model checkpoints provided by the dattri library to avoid extensive model retraining). For the datasets we use: MNIST-10 dataset holds CC BY-SA 3.0 license; CIFAR-10 dataset holds CC-BY 4.0 license; MAESTRO dataset holds CC BY-NC-SA 4.0 license.
30

---

(6
projection)
1.0
1.0
2.9
0.9
0.7
3 0.5
9.6
0.6
0.5
0.3
0.4
0.3
10
10
127
10
10
10
10
10
10
30
10"
10'
MNIST MLP (pi 4096)
CIFAR2 ResNet9 4096)
MAESTRO MT (p 4096)
1.0
1.0
1.0
0.9
0.7
0.6
2.6
0.6
0.1
0.5
0.
0.3
Figure 5: The curve of Er, as a function of A. Each subfigure corresponds an experiment setting outlined in Section 4.4.
C.3 Effects of Different Subset Fractions
We provide an extended analysis of how the subset fraction a/|S| influences the surrogate indicator, considering values in {0.25, 0.5, 0.75), where 0.5 corresponds to the setting used in earlier experi- ments. Figure 6 illustrates the surrogate indicator's behavior for each subset fraction, shown in green (0.25), blue (0.5), and violet (0.75). While the magnitude of the LDS varies with the subset fraction, its overall trend function of À remains similar across different values of subset fraction. This consistency suggests that the surrogate indicator is robust to changes subset size across different experimental settings.
C.4 Additional Experiments on WikiText2 with GPT2
Here, we demonstrate additional experiments on the large setting WikiText2 [28] with GPT2 [31]. We fix projection dimension 4096, and apply the surrogate indicator find proper regularization strengths. We investigate both the IFFIM and the LoGra [6] attributors. The results are illustrated in Figure 7.
C.5 Experiments on Downstream Settings
We investigate various settings, including Logistic Regression models (LR) on MNIST dataset [25]. ResNet-9 [14] CIFAR-2 [24] dataset, and Trasforranforrmrr (MT) [2] on MAESTRO dataset [13]. We conduct the experiments with training data removal rate 10%, 30%, and 50%. For methods we use for data removal, we have Random (random removal), IFFIM Default and TRAK Default (IFFIM and TRAK attributors with default regularization 0), and IFFIM Selected and TRAK Selected (IFFIM and TRAK attributors with selected regularization by our algorithm). For reference, we also include Full, which stands for test performance without removal. We repeat the experiments with 10 different seeds and report the mean values as well as their standard errors for better statistical significance. The results are presented in Table 2, Table 3, and Table 4. We notice that when removing 10% training data, most methods lead to insignificant model performance decrease, likely due to redundancy in the datasets. However, we observe that our methods (IFFIM Selected and TRAK Selected) generally outperform baselines (Random, IFFIM Default, and TRAK Default) across different settings and the differences are statistically significant.
31

---

512))
(No projection)
0.10
0.09
1035
0.10
0.05
0.02
0.00
700%
725
10-99
10
10
10.5
10-21
10t1
10-7
10-4
MNIST MLP (6 (6-112)
MNIST MLP (6 4096)
0.40
0.35
0.30
0.30
0.25
0.25
0.20
0.15
0.15
0.30
1.8
Figure 6: Experiment results with different values of subset fraction. Green, blue, and violet curves illustrate LDS-À relationship with subset fraction 0.25, 0.5, and 0.75, respectively.
IFFIM on WikiText2 (p 4096)
LoGra on WikiText2 -5= 4096)
0.24
0.22
0.12
0.20
0.10
0.18
0.16
0.08
0.14
0.06
0.12
10-4
10-2
10"
10-4
10-2
Figure 7: The plot of LDS versus on the WikiText2 with GPT2 setting. The left plot shows the results with the IFFIM attributor while the right plot demonstrates the results with the LoGra attributor. The red solid vertical line indicates the À selected by our method.
D
Code Availability
Our code is available at http::///stttt.cac///////taaattt
32

---

Table 2: Data selection performance on MNIST+LR settings. Values in the table are means and standard errors of test accuracies. Experiments are repeated with 10 different seeds. Lower accuracies indicate better attribution performance.
Removal rates
10%
30%
50%
Full
88.63% + 0.03%
88.63% + 0.03%
88.63% + 0.03%
Random
88.52% + 0.03%
88.06% + 0.05%
87.60% 1.00%=+0000 0.06%
IFFIM Default
88.44% + 0.05%
87.77% + 0.08%
87.58% +1.58%+0.000 0.10%
IFFIM Selected
87.66% = 0.04%
84.50% 0.04%
81.53% 0.06%
TRAK Default
88.53% = 0.06%
87.92% = 0.11%
86.88% = 0.29%
TRAK Selected
87.30% + 0.04%
83.84% + 0.05%
80.12% + 0.08%
Table 3: Data selection performance on CIFAR-2+ResNet-9 settings. Values the table are means and standard errors of test accuracies. Experiments repeated with 10 different seeds. Lower accuracies indicate better attribution performance.
Removal rates
10%
30%
50%
Full
75.04% + 0.53%
75.04% + 0.53%
75.04% + 0.53%
Random
74.20% + 0.59%
71.86% + 0.33%
68.64% + 0.38%
IFFIM Default
74.36% + 0.38%
72.42% + 0.47%
67.84% + 1.16%
IFFIM Selected
74.36% + 0.38%
72.42% + 0.47%
67.84% + 1.16%
TRAK Default
74.06% + 0.52%
71.68% + 0.40%
69.30% + 0.49%
TRAK Selected
71.24% + 0.52%
64.14% + 0.47%
56.30% + 0.70%
Table Data selection performance on MAESTRO+MT settings. Values the table are means and standard errors of test losses. Experiments repeated with 10 different seeds. Higher losses indicate better attribution performance. Note that we only include TRAK here, as IF has been shown to exhibit poor attribution performance in this complex setting [8].
Removal rates
10%
30%
50%
Full
4.30 0.01
4.30 + 0.01
4.30 0.01
Random
4.32 + 0.01
4.38 + 0.01
4.67 = 0.02
TRAK Default
4.35
4.40 + 0.01
4.68 - 0.01
TRAK Selected
4.40
4.52 0.01
4.83 - 0.02
33


Reviews:
[{"review_id": "NeN8lb19iJ", "reviewer": "Reviewer_FTFU", "summary": "This paper investigates the problem of hyperparameter sensitivity in training data attribution (TDA) methods, both from theoretical and empirical perspectives. The authors show that popular attribution methods such as Influence Functions (IF), LiSSA, TRAK, and LoGra exhibit significant sensitivity to hyperparameter choices. Moreover, the optimal hyperparameter settings are shown to be highly dependent on the specific dataset and model, and the effects of multiple hyperparameters can be entangled, i.e., the best value for one hyperparameter may depend on the value of another.\n\nSince accurate evaluation of TDA methods typically requires multiple retraining runs, empirical tuning of hyperparameters is computationally expensive and often impractical. To address this challenge, the authors propose a theoretically grounded framework to guide hyperparameter selection. As a case study, they theoretically derive a practical algorithm for selecting the regularization parameter in the IFFIM method, e.g. an IF variant based on an approximation involving Fisher Information Matrix. Empirical results demonstrate that this algorithm can often find hyperparameter values that are close to optimal.", "questions": "1. Please clarify the interpretation of the terms in Equation 8 in Theorem 4.3. A more intuitive explanation would improve the accessibility of the result.\n2. How do Theorem 4.3, the surrogate indicator, and Proposition 4.7 connect to the derivation of Algorithm 1? Please describe this connection more explicitly.\n3. What is the computational cost of estimating the optimal hyperparameter using Algorithm 1, and how does it compare to computing the LDS metric under similar conditions?", "limitations": "The authors adequately addressed the limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n* The paper is well-written and well-structured.\n* The empirical study is comprehensive, covering a variety of popular TDA methods (Influence Functions, LiSSA, TRAK, LoGra), and convincingly demonstrates hyperparameter sensitivity across datasets and models.\n* The work brings valuable attention to a widespread, under-addressed issue in the TDA literature.\n* The paper presents a promising strategy to mitigate this issue: using theoretical derivations to guide hyperparameter selection. The case study demonstrates how this approach can help identify a promising region in hyperparameter space without expensive evaluation metric calculations.\n\n**Weaknesses:**\n\n* The proposed theoretical strategy, while appealing in principle, is less convincing in the case study. The theoretical derivation involves several assumptions (e.g., the LDS being uni-modal and concave in $\\lambda$, technical assumption in Appendix B.3) and heuristics (e.g., $\\xi_{T, \\lambda}$ in lines 263–264) that may not hold broadly in practice. The empirical results in Figure 2 show that the hyperparameter values determined by the proposed algorithm are not always close to the optimal value.\n* It is unclear how easily this theory-driven approach for hyperparameter selection can be generalized to other TDA methods beyond IF.\n* The derivation of Algorithm 1 lacks clarity:\n\n  * Theorem 4.3 introduces a large number of new definitions with little intuitive explanation.\n  * The purpose of Proposition 4.7 is unclear.\n  * The derivations steps from the theoretical results (Theorem 4.3, Proposion 4.7) to Algorithm 1 are not clearly articulated.\n\n**Minor Weaknesses:**\n\n* Possible typos:\n\n  * “The sampled subset” (line 212) — may mean “a sample of subsets.”\n  * “Trained the full dataset S” (line 258)\n* The finding that earlier model checkpoints may produce better results than final ones for some TDA methods (line 111) is quite interesting. I wish the authors gave possible explanations for this.", "was_revised": true, "final_justification": "Overall, the theoretical and quantitative results presented in the paper are quite compelling. The paper significantly contributes to the field by systematically evaluating the sensitivity of TDA methods. The rebuttal was helpful in clarifying the theoretical derivations.  However, my concerns about the significance of the theoretical contribution remain, as the proposed method focuses on a single parameter within one specific family of TDA methods.", "rebuttal": "We appreciate the positive evaluation as well as the constructive suggestions given by the reviewer.\n\n> The proposed theoretical strategy, while appealing in principle, is less convincing in the case study. The theoretical derivation involves several assumptions (e.g., the LDS being uni-modal and concave in , technical assumption in Appendix B.3) and heuristics (e.g.,  in lines 263–264) that may not hold broadly in practice. The empirical results in Figure 2 show that the hyperparameter values determined by the proposed algorithm are not always close to the optimal value.\n\nRather than aiming to find an exact optimal regularization—which is intractable and costly in practice—we focus on providing an efficient, empirically validated method (the surrogate indicator and the algorithm) for selecting a strong candidate. The results we include demonstrate that our method outperformed the default choice (no regularization) and baselines (the spectra-based) in terms of stability and overall performance. Furthermore, our empirical studies under multiple different settings justify that the LDS curve is very likely to be uni-modal and concave in practice.\n\n> It is unclear how easily this theory-driven approach for hyperparameter selection can be generalized to other TDA methods beyond IF.\n\nAs discussed in Section 2 and Appendix, many state-of-the-art data attribution methods such as TRAK and LoGra can be categorized as variants of IFFIM. The experiments in Section 4 and Appendix focus on the effectiveness of our method with IFFIM and TRAK. We also include an experiment of LoGra under GPT-2 + WikiText-2 setting, as shown in the table below. These results, obtained under multiple different settings, demonstrate the strong applicability of the method in practice. \n| Attributor/Regularization | 1e-5  | 1e-4  | 1e-3             | 1e-2  | 1e-1  | 1e0   | 1e1   |\n|---------------------------|-------|-------|------------------|-------|-------|-------|-------|\n| LoGra                     | 0.133 | 0.158 | 0.207 (selected) | 0.253 | 0.240 | 0.170 | 0.123 |\nHere, (selected) indicates the regularization strength selected by our algorithm. The performances are measured in LDS.\n\n> Theorem 4.3 introduces a large number of new definitions with little intuitive explanation.\n> Please clarify the interpretation of the terms in Equation 8 in Theorem 4.3. A more intuitive explanation would improve the accessibility of the result.\n\nTo aid understanding, we now offer an intuitive explanation of both sides of Equation 8. We first focus on the surrogate indicator, which is also the RHS of Equation 8. For simplicity, we let $v$ denote the gradient of $f$ at $z’$ and $\\theta^\\ast_S$. By observing the definition of terms $t_{k, z’, \\lambda}$ where $k = 1, 2, 3$, the surrogate indicator can be viewed as the cosine “angle” between $v$ and $(F_S+\\lambda I_p)^{-1} v$. Here, $(F_S+\\lambda I_p)^{-1} v$is related to the $\\theta$ update direction in IFFIM, as in Equation 2. When the surrogate indicator is close to 0, this “angle” is overly large, suggesting a significant misalignment between the $\\theta$ update direction and the gradient descent direction $v$. This indicates that the update direction is not reliable. In this case, increasing $\\lambda$ helps intuitively because it helps condition the inverted matrix $F_S$ better. On the other hand, when the surrogate indicator is close to 1, $F_S$ is overshadowed by the regularization term $\\lambda I_p$, meaning that the attribution method may become dominated by the regularization and lose sensitivity to the underlying data structure. Similarly, the LHS of Equation 8 measures the “angle” between $v$ and the ground-truth direction that involves $g_{z’}$ (by applying Lemma B.3). Hence, Theorem 4.3 is in essence stating that when the “angle” between $v$ and $(F_S+\\lambda I_p)^{-1} v$ is of comparable level as the “angle” between $v$ and ground-truth direction (not too big or too small), LDS could reach its maximum. We will provide a similar and more detailed interpretation in the final version of our paper.\n\n> The purpose of Proposition 4.7 is unclear.\n> The derivation steps from the theoretical results (Theorem 4.3, Proposition 4.7) to Algorithm 1 are not clearly articulated.\n> How do Theorem 4.3, the surrogate indicator, and Proposition 4.7 connect to the derivation of Algorithm 1? Please describe this connection more explicitly.\n\nWe establish a clearer connection between Theorem 4.3, Proposition 4.7 and Algorithm 1 here. Overall, Theorem 4.3 inspires the formulation of the surrogate indicator as well as Algorithm 1, while Proposition 4.7 justifies a desirable property of the surrogate indicator, which is helpful for Algorithm 1. In concrete, the surrogate indicator is introduced as exactly the RHS of Equation 8 in Theorem 4.3. The reason we drop LHS of Equation 8 is that it involves retraining-related terms $g$ and $\\alpha$, whose computation is costly. From the above discussion of the interpretation of Equation 8, we notice that a good regularization strength commonly lies in the region where the surrogate indicator is close to neither of 0 and 1. Further, Proposition 4.7 guarantees that the surrogate indicator and LHS of Equation 8 both lie in the interval [0, 1] under certain assumption, which suggests that a fixed threshold of the surrogate indicator such as 0.5 could be adopted to select the regularization, which is exactly how Algorithm 1 works. Finally, Proposition 4.7 ensures that both sides of Equation 8 are comparable as they both fall in the interval [0, 1].\n\n> What is the computational cost of estimating the optimal hyperparameter using Algorithm 1, and how does it compare to computing the LDS metric under similar conditions?\n\nNotably, directly computing LDS requires retraining, making it prohibitively expensive for large-scale models. However, our method only requires gradients at test examples, which can be obtained easily through inference passes. Under CIFAR2 + ResNet9 setting, searching for a same number of hyperparameter costs 13.7s using Algorithm 1, while costing 9m12.6s when ground-truth data in LDS are computed on the test set and the hyperparameter is selected by computing LDS directly. We obtain a 40.3x speedup over LDS-based selection under the same computational condition, demonstrating the practical efficiency of our approach.\n\n> Possible typos\n\nWe appreciate the reviewer’s careful reading and will ensure all identified typos are corrected in the final version.\n\n> The finding that earlier model checkpoints may produce better results than final ones for some TDA methods (line 111) is quite interesting. I wish the authors gave possible explanations for this.\n\nWhile a complete explanation requires thorough investigation, our hypothesis is that this phenomenon stems from numerical issues due to vanishing gradient norms in later epochs. We also include an empirical fact here that supports our hypothesis.\n| epoch              | 0     | 10    | 20    | 30    | 40    | 50    | 60    | 70    |\n|--------------------|-------|-------|-------|-------|-------|-------|-------|-------|\n| with normalization | 0.111 | 0.125 | 0.128 | 0.130 | 0.128 | 0.131 | 0.128 | 0.129 |\n| w/o normalization  | 0.104 | 0.155 | 0.115 | 0.075 | 0.075 | 0.112 | 0.100 | 0.095 |\nHere the attribution method is TracIn and normalization means that the inner product of gradients in TracIn is replaced with the cosine angle between gradients. We observe that training epoch causes the performance to drop significantly after 10 epochs in absence of normalization, implying that the issue could be due to small gradient norms."}, {"review_id": "OhsZC9gRr4", "reviewer": "Reviewer_Uabe", "summary": "This paper identifies hyperparameter sensitivity as a critical but overlooked challenge in the research of training data attribution. The authors conduct the first large-scale empirical study demonstrating that most data attribution methods are highly sensitive to key hyperparameters like regularization terms, projection dimensions, and training epochs. The paper provides a theoretical analysis of the regularization parameter in influence function methods and proposes a practical, retraining-free selection heuristic that approximates optimal regularization values. Their experiments across multiple datasets and models show that the proposed method effectively selects good hyperparameter values without the computational cost of repeated model retraining.", "questions": "There is also a hyperparameter in the metric of LDS, the subset size $a$. I wonder whether varying $a$ will also significantly change the results?", "limitations": "It would be good to evaluate the proposed heuristic with more datasets.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n- This paper revisits the active research area of training data attribution and identifies a critical aspect that has been largely overlooked by the community—the computational challenge of hyperparameter tuning in data attribution methods.\n- The paper is well-written and clearly structured, making it accessible to readers (with the exception of Section 4.1, discussed below).\n\n**Weaknesses**\n- Section 4.1 suffers from poor presentation and organization. The notation is confusingly introduced in separate remark paragraphs after the main theorem statement, making it difficult to parse the theoretical contributions. This section would benefit from significant revision to include: (1) clear interpretation of the theorem statement, (2) intuitive explanations of the key insights, and (3) a proof sketch for Theorem 4.3 (and other theoretical results if space allows). \n\n- While the proposed heuristic in Section 4.2 demonstrates effectiveness in the experiments of Section 4.3, there are concerns about its generalizability. The fixed threshold of 0.5 may work well primarily because the experimental evaluation relies on relatively clean datasets (MNIST and CIFAR-10). I suspect this threshold would require significant adjustment for different types of datasets or domains. This concern is supported by the notably worse performance on the MAESTRO dataset, suggesting that the method's robustness across diverse data characteristics remains questionable.", "was_revised": true, "final_justification": "Happy to recommend acceptance as long as the authors will incorporate the proposed changes to Section 4.1 into the revision.", "rebuttal": "We appreciate the insightful comments by the reviewer.\n\n> Section 4.1 suffers from poor presentation and organization. The notation is confusingly introduced in separate remark paragraphs after the main theorem statement, making it difficult to parse the theoretical contributions. This section would benefit from significant revision to include: (1) clear interpretation of the theorem statement, (2) intuitive explanations of the key insights, and (3) a proof sketch for Theorem 4.3 (and other theoretical results if space allows).\n\nWe acknowledge that Section 4.1 introduces a dense set of new notations and theoretical results within a compact space. Following the suggestion of the reviewer, we will rewrite Section 4.1 in mainly the following aspects. First, we will move the introduction of key notations (e.g. $t_{k, z’, \\lambda}$) to precede Theorem 4.3, and provide intuitive explanations for each. In particular, we will interpret $\\alpha_{z’, i}$ and $g_{z’}$ by relating them to the leave-one-out (LOO) interpretation of data attribution, and clarify other terms like $t$, $r$, and $o$ by expressing them as inner products of more elementary components. Then, we will adjust Theorem 4.3 and add an interpretation of the result that is grounded in the meanings of the key terms from Equation 8.Finally, we will summarize the core ideas and key proof steps in a clear proof sketch below Theorem 4.3, incorporating relevant intermediate results from the Appendix. We also consider splitting Section 4.1 into smaller subsections for better readability. These modifications will be included in the final version of our paper.\n\n> While the proposed heuristic in Section 4.2 demonstrates effectiveness in the experiments of Section 4.3, there are concerns about its generalizability. The fixed threshold of 0.5 may work well primarily because the experimental evaluation relies on relatively clean datasets (MNIST and CIFAR-10). I suspect this threshold would require significant adjustment for different types of datasets or domains. This concern is supported by the notably worse performance on the MAESTRO dataset, suggesting that the method's robustness across diverse data characteristics remains questionable.\n\nWe include an additional larger experimental setting on text data (GPT-2 on WikiText-2), which validates the effectiveness of our algorithm to select a good regularization coefficient. The results are summarized in the table below. \n| Attributor/Regularization | 1e-5  | 1e-4  | 1e-3             | 1e-2  | 1e-1  | 1e0   | 1e1   |\n|---------------------------|-------|-------|------------------|-------|-------|-------|-------|\n| IFFIM                     | 0.061 | 0.081 | 0.117 (selected) | 0.145 | 0.127 | 0.085 | 0.067 |\n| LoGra                     | 0.133 | 0.158 | 0.207 (selected) | 0.253 | 0.240 | 0.170 | 0.123 |\nHere, (selected) indicates the regularization strength selected by our algorithm. The performances are measured in LDS.\n\nWe highlight that our study now covers experiments on **three different modalities** (image, text, and music), as well as **three types of data attribution methods** (IFFIM, TRAK, and LoGra). These experiments support that a fixed threshold 0.5 enables our algorithm to reliably select effective regularization strengths across diverse settings.\n\n> There is also a hyperparameter in the metric of LDS, the subset size . I wonder whether varying  will also significantly change the results?\n\nWe included the discussion of the subset size in LDS with empirical evidence in Appendix C.3. As a high level summary, we observed that while the magnitude of LDS could vary with the subset size, the overall monotonicity and trend as a function of $\\lambda$ remains similar. This observation suggests that our method remains robust across different subset sizes."}, {"review_id": "61OHnvrFet", "reviewer": "Reviewer_GpMe", "summary": "This paper focuses on the problem of hyperparameter selection in data attribution methods, which is a topic that is severely underestimated in existing research but is crucial in practical applications. The authors first conduct a systematic large-scale empirical study to evaluate the hyperparameter sensitivity of mainstream data attribution methods on multiple models and datasets. In addition, the authors propose a surrogate indicator based on theoretical derivation, which can select the regularization hyperparameter λ without retraining the model. Experiments show that the indicator has good consistency and performance in multiple datasets and methods.", "questions": "Overall, I appreciate the contribution of this paper, so I gave it a positive rating. My main questions are as follows:\n1. In addition to λ, the projection dimension and the number of training rounds have also been shown to have a significant impact on performance. Why didn't you try to estimate them using a surrogate-like approach? Is it because of mathematical difficulty, non-differentiability, or other practical obstacles?\n\n2. Have you considered other performance indicators? All current experiments use LDS as the only metric. Have you considered using downstream performance as an evaluation metric in data denoising or data selection tasks? If it cannot be extended to practical tasks, the practicality of this paper will be questioned.\n\n3. Are theoretical indicators generalizable? The derivation of the surrogate is based on a series of technical assumptions. Do these assumptions hold true in large-scale deep models or strong non-convex situations? It is recommended to provide further empirical support or hypothesis sensitivity analysis.\n\n4. Is there a risk of \"misleading selection\"? The author's method attempts to fit the LDS trend with the surrogate curve, but is there a situation where the surrogate indicator fails to be monotonically in some areas, resulting in the misselection of suboptimal λ? Have the authors considered uncertainty estimates or confidence bounds to aid in the selection?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. This paper focuses on the practical issues in the implementation of data attribution methods.\n\n2. This paper covers multiple attribution methods and is highly representative.\n\n3. Attached with GitHub links and detailed experimental details, it is helpful for the community to verify and use it later.\n\nWeaknesses\n1. The proposed surrogate indicator is only applicable to the λ parameter in the influence function method, and cannot be extended to other key hyperparameters (such as projection dimension) or other types of attribution methods, with obvious limitations.\n\n2. The experimental indicators are single, and many results are based on the synthetic indicator LDS. End-to-end verification has not been performed in actual tasks (such as noise detection and training data screening), and it cannot be proved that the selected hyperparameters can bring better downstream performance.\n\n3. The proposed indicators are based on complex partial derivative conditions and high-order tensor expressions, which lack interpretability for non-theoretical researchers and have a high threshold for actual adoption.\n\n4. The author identified the pain points of hyperparameter tuning, but the solution he finally gave is more like an \"ad hoc fix\" rather than a systematic solution to the problem.", "was_revised": true, "final_justification": "After reading the rebuttal and discussion, I feel the authors have addressed my main concerns, especially by adding downstream experiments and clarifying theoretical assumptions. Some limitations remain in scope and complexity, but the strengths still outweigh the weaknesses, so I keep my borderline accept recommendation.", "rebuttal": "We appreciate the positive evaluation as well as the constructive feedback given by the reviewer. We address each point in Weaknesses and Questions below.\n\n> The proposed surrogate indicator is only applicable to the λ parameter in the influence function method, and cannot be extended to other key hyperparameters (such as projection dimension) or other types of attribution methods, with obvious limitations.\n> In addition to λ, the projection dimension and the number of training rounds have also been shown to have a significant impact on performance. Why didn't you try to estimate them using a surrogate-like approach? Is it because of mathematical difficulty, non-differentiability, or other practical obstacles?\n\nWe would like to clarify that the regularization strength is one of the most influential and sensitive hyperparameters according to our empirical study, which justifies our special attention to it. Regularization strength is also applicable to a wide range of state-of-the-art attribution methods, including but not limited to TRAK and LoGra (as discussed in Section 2 and Appendix). Additionally, we would like to point out some practical difficulties when applying similar surrogate indicator analysis on other hyperparameters such as projection dimension and training epoch. While we include some discussion of gradient projection and its influence on the effectiveness of the surrogate indicator in Section 4, projection dimension is difficult to analyze directly, as its impact is highly dependent on the choice of random projection matrices. Similarly, to predict the effect of tuning training epochs, we have to carefully consider a variety of external factors such as optimization methods and learning rate, which complicates theoretical modeling and makes surrogate-based analysis less tractable. Furthermore, theoretical understanding of other hyperparameters are indeed interesting topics, but we would like to bring up another significance of our work: it includes the first systematic large-scale study highlighting the problem of hyperparameter selection in data attribution, which provides important future research directions.\n\n> The experimental indicators are single, and many results are based on the synthetic indicator LDS. End-to-end verification has not been performed in actual tasks (such as noise detection and training data screening), and it cannot be proved that the selected hyperparameters can bring better downstream performance.\n> Have you considered other performance indicators? All current experiments use LDS as the only metric. Have you considered using downstream performance as an evaluation metric in data denoising or data selection tasks? If it cannot be extended to practical tasks, the practicality of this paper will be questioned.\n\nFollowing the reviewer’s suggestion, we include an evaluation of our proposed algorithm on downstream settings. We focus on the task of data selection, where we measure the performance drop of models when the most positively influential training data are removed (more drop indicates better attribution performance). The results are shown in the table below. We observe that our proposed algorithm generally achieves visible model performance drops, justifying the effectiveness of the algorithm in downstream tasks.\n| Settings         | Full  | R     | ID    | IS    | TD    | TS    |\n|------------------|-------|-------|-------|-------|-------|-------|\n| MNIST + LR       | 88.52 | 87.52 | 87.46 | 81.44 | 86.98 | 79.92 |\n| CIFAR2 + ResNet9 | 75.8  | 69.8  | 70.2  | 70.2  | 73.0  | 55.4  |\nHere Full means no removal is done, R means random removal, ID means IFFIM + default (regularization = 0), IS means IFFIM + selected by algorithm, TD means TRAK + default (regularization = 0), TS means TRAK + selected by algorithm. We will add more experiments on more datasets in the final version of our paper.\n\n> The proposed indicators are based on complex partial derivative conditions and high-order tensor expressions, which lack interpretability for non-theoretical researchers and have a high threshold for actual adoption.\n\nWhile we recognize the reviewer’s concern regarding interpretability, we have taken care to design our algorithm with a clear conceptual intuition and practical implementation. In brief, the algorithm is simply seeking a range of regularization where the surrogate indicator is close to neither of its bounds (0 and 1) when averaged over the test set. To further facilitate understanding, we also provide an intuition of the surrogate indicator here. For simplicity, we let $v$ denote the gradient of $f$ at $z’$ and $\\theta^\\ast_S$. By observing the definition of terms $t_{k, z’, \\lambda}$ where $k = 1, 2, 3$, the surrogate indicator can be viewed as the cosine “angle” between $v$ and $(F_S+\\lambda I_p)^{-1} v$. Here, $(F_S+\\lambda I_p)^{-1} v$is related to the $\\theta$ update direction in IFFIM, as in Equation 2. When the surrogate indicator is close to 0, this “angle” is overly large, suggesting a significant misalignment between the $\\theta$ update direction and the gradient descent direction $v$. This indicates that the update direction is not reliable. In this case, increasing $\\lambda$ helps intuitively because it helps condition the inverted matrix $F_S$ better. On the other hand, when the surrogate indicator is close to 1, $F_S$ is overshadowed by the regularization term $\\lambda I_p$, meaning that the attribution method may become dominated by the regularization and lose sensitivity to the underlying data structure.\n\n> Are theoretical indicators generalizable? The derivation of the surrogate is based on a series of technical assumptions. Do these assumptions hold true in large-scale deep models or strong non-convex situations? It is recommended to provide further empirical support or hypothesis sensitivity analysis.\n\nWe mainly focus on Assumption B.6 and Assumption B.8. We would like to state that the constants $C_1$, $C_2$, and $C_3$ generally stay at the 1e0 level instead of scaling visibly with the smallest eigenvalue. Due to computational limitations, currently we only conduct the test in the CIFAR2+ResNet9 setting, which already contains a certain level of non-convexity. We will add similar experiments and analysis on other larger settings. Note that we compute $C_1$ with respect to the smallest eigenvalue (instead of all eigenvalues), which is reasonable because only the $C_1$ term related to the smallest eigenvalue occurs in our theoretical derivation. Our results are that $C_3 = 0.4082$ is enough for all test points with $\\varepsilon_g = 0.5$. Further, 90% of the test points yield $C_1 = 0.3121$ and $C_2 = 51.9933$. In comparison, the smallest eigenvalue is only about 1e-4. \n\n> The author identified the pain points of hyperparameter tuning, but the solution he finally gave is more like an \"ad hoc fix\" rather than a systematic solution to the problem.\n\nWe would like to highlight that the surrogate indicator and the related algorithm we gave is both theoretically interpretable and practically useful. As hyperparameter selection in data attribution is different from ordinary hyperparameter selection in machine learning, our solution has a more general implication that could inspire further research in this direction, and we believe that the theoretical analysis that supports our solution could be generalized to solve similar problems in future works.\n\n> Is there a risk of \"misleading selection\"? The author's method attempts to fit the LDS trend with the surrogate curve, but is there a situation where the surrogate indicator fails to be monotonically in some areas, resulting in the misselection of suboptimal λ? Have the authors considered uncertainty estimates or confidence bounds to aid in the selection?\n\nWe acknowledge that we do not take uncertainty into account when formulating the surrogate indicator and the algorithm, but we would like to state that the uncertainty is minimal and unlikely to affect the reliability of the selected hyperparameters in practice. To justify this, we re-run the experiment in CIFAR2+ResNet9 setting 50 times with different training seeds for the ensemble model. The mean and standard error (SE) of LDS are shown in the table below, which demonstrates that the error is generally negligible compared with the mean. Additionally, we also observe that in 49 among these 50 trials, our algorithm selects the same regularization term. We also notice that the surrogate indicator curves across all 50 trials are monotonic.\n| Regularization | 1e-1  | 1e0   | 1e1   | 1e2   | 1e3   | 1e4   |\n|----------------|-------|-------|-------|-------|-------|-------|\n| Mean           | 0.172 | 0.214 | 0.248 | 0.254 | 0.258 | 0.259 |\n| SE             | 0.003 | 0.003 | 0.003 | 0.004 | 0.004 | 0.004 |"}, {"review_id": "6ljwKz8Vcd", "reviewer": "Reviewer_GgCL", "summary": "The paper investigates hyperparameters for data attribution and benchmarks their impact on several prominent methods: Influence Functions, TraceIN, TRAK, and LoGra, across the MNIST (MLP), CIFAR-10 (ResNet-9), and Wikitext (GPT-2) datasets. The main findings can be summarized as follows:\n\n- Most attribution methods are sensitive to certain hyperparameters.\n- The optimal hyperparameter choices vary across datasets and models.\n- The impact of hyperparameters can be entangled.\n- Implicit hyperparameters, such as training epochs, also play a significant role.\n\nFinally, the authors propose an algorithm (surrogate-indicator-based practical selection) for selecting the regularization term for influence functions without the need for retraining. The method is validated on MNIST, CIFAR, and Maestro datasets, showing that “overall, we observe that the λ chosen by the proposed algorithm generally leads to good LDS values that are significantly better than λ = 0, and in many cases, it is close to the optimal value.”", "questions": "Are there any findings or insights that are more surprising, or could any of the results be suggested as actionable steps for practitioners?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: The paper is easy to follow and addresses relevant questions that are analyzed thoroughly.\n\nWeaknesses: I do not find the contributions significant enough for NeurIPS. The insights from the study are not particularly surprising and may not provide substantial benefits to a broad set of researchers; the finding that “the best hyperparameter choices vary across datasets and models” indicates that researchers and practitioners will still need to perform hyperparameter searches after reading the paper. While the algorithm assists in selecting one parameter for influence functions, it focuses on a single parameter for one method, and I am not convinced of its generalizability to other datasets. In my opinion, this paper would be better suited for a specialized venue.", "was_revised": true, "final_justification": "The underexplored role of hyperparameters in TDA: evidenced by the limited support in existing libraries (e.g. OpenDataVal) makes this paper more relevant than I first thought.\nIf hyperparameters in TDA are underexamined, then insights I initially deemed marginal (e.g. the impact of training epoch) gain in importance.\nThe new GPT-2 experiments with LDS and, especially, the downstream tasks (in response to Reviewer GpMe) convincingly show that the regularization term substantially improves data-removal performance. With the default parameter, results are near random; with the tuned parameter (especially for TRAK) the improvement is clear. This downstream evaluation more effectively demonstrates the method’s benefits than the earlier benchmarks in my opinion.", "rebuttal": "We thank the reviewer for the comments and questions. We address your individual concerns below.\n\n> Are there any findings or insights that are more surprising\n\nWe would like to highlight that our paper has **several novel/surprising findings with practical implications**. These findings are discussed in detail in Section 2 and 3. For instance, we demonstrate that parameters like *training epoch* **have NOT been considered as hyperparameters in prior literature**, but they significantly affect the performance of a wide range of data attribution methods. As another example, by studying the entanglement of hyperparameters, we have provided a compelling explanation for an **open question raised in the TRAK paper**: the counterintuitive LDS drop with respect to increasing projection dimension can be counteracted by introducing a proper regularization. In short, our work extends **beyond** the observation that hyperparameter selection is highly case-specific, which is true, but also shed light on concrete behaviors of attribution performance when hyperparameters are tuned with evidence from the first systematic research in this direction.\n\n> or could any of the results be suggested as actionable steps for practitioners\n\nWe kindly refer the reviewer to the end of Section 1 and Section 5 for some discussion on the implications of our results. Specifically, (1) we highlighted that hyperparameter tuning is a unique challenge in data attribution given the high computational cost associated with the evaluation metrics—to the best of our knowledge, ours is the first study to identify this unique challenge. (2) We pointed out that future methodological development of data attribution should have a more systematic treatment on hyperparameter selection. (3) Our proposed method in Section 4 provides both *a practical method* for selecting the regularization term in many popular data attribution methods (including IF, TRAK, LoGra), and *an example* for analyzing the hyperparameters.\n\nIn addition, some of our novel findings in Section 3 have straightforward implications for practitioners. For example, when using random projection in gradient-based methods, one should carefully tune the regularization term (possibly with our proposed method).\n\nIn our next revision, we will include a dedicated paragraph to discuss these implications in one place.\n\n> it focuses on a single parameter for one method, and I am not convinced of its generalizability to other datasets\n\nWe would like to clarify that our surrogate indicator algorithm applies to a wide range of settings instead of one method or a limited number of datasets. As we discuss in Section 2 and Appendix, state-of-the-art data attribution methods that involve inverting FIM (or GGN) and introduce random projection can be categorized as a variant of our proposed IFFIM, which include but are not limited to TRAK and LoGra. Our discussion of IFFIM can be easily generalized to these methods. Further, following the reviewer’s suggestion, we include an additional larger experimental setting on text data (GPT-2 on WikiText-2), which supports the effectiveness of our algorithm to select a good regularization coefficient. The results are summarized in the table below. \n| Attributor/Regularization | 1e-5  | 1e-4  | 1e-3             | 1e-2  | 1e-1  | 1e0   | 1e1   |\n|---------------------------|-------|-------|------------------|-------|-------|-------|-------|\n| IFFIM                     | 0.061 | 0.081 | 0.117 (selected) | 0.145 | 0.127 | 0.085 | 0.067 |\n| LoGra                     | 0.133 | 0.158 | 0.207 (selected) | 0.253 | 0.240 | 0.170 | 0.123 |\nHere, (selected) indicates the regularization strength selected by our algorithm. The performances are measured in LDS.\n\nWe highlight that our study now covers experiments on **three different modalities** (image, text, and music), as well as **three types of data attribution methods** (IFFIM, TRAK, and LoGra). We are actively expanding our experimental coverage and will include additional results in the revised version to further support generalizability."}]

Author rebuttal:

Meta-review: This paper investigate the issue of the sensitivity of data attribution method to its hyperparameters. The author conduct empirical studies on this issue, and not surprisingly, the sensitivity to hyper-parameters depends on the method and data. Theoretically, the author also analysed a simple special case and derive a selection procedure without model re-training. However, the concerns about the significance of this theoretical results has been raised due to its over-simplification. In addition, the clarity of the paper can be further improved. Nevertheless, this paper still provide an initial step towards this problem, and can be valuable to the community.
Decision: Accept (poster)