Training Data Attribution via Approximate Unrolling
Authors: Juhan Bae
Wu Lin
Jonathan Lorraine
Roger Baker Grosse
Venue: NeurIPS 2024 poster (2024)
Keywords: training data attribution
influence functions
Primary area: interpretability_and_explainability
TL;DR: We introduce a new TDA technique that combines the strength of implicit-differentiation-based (e.g., influence fuctions) and unrolling-based techniques.

Abstract: Many training data attribution (TDA) methods aim to estimate how a model's behavior would change if one or more data points were removed from the training set. Methods based on implicit differentiation, such as influence functions, can be made computationally efficient, but fail to account for underspecification, the implicit bias of the optimization algorithm, or multi-stage training pipelines. By contrast, methods based on unrolling address these issues but face scalability challenges. In this work, we connect the implicit-differentiation-based and unrolling-based approaches and combine their benefits by introducing Source, an approximate unrolling-based TDA method that is computed using an influence-function-like formula. While being computationally efficient compared to unrolling-based approaches, Source is suitable in cases where implicit-differentiation-based approaches struggle, such as in non-converged models and multi-stage training pipelines. Empirically, Source outperforms existing TDA techniques in counterfactual prediction, especially in settings where implicit-differentiation-based approaches fall short.

[T] They have diverse crafting data Training Data Attribution via Approximate Unrolling
Juhan Bae!. Wu Lin", J Lorraine .2.3, Roger Grosse 1,2,4 University of Toronto, Vector Institute, SNVIDIA, 3 Anthropic {jbae, lorraine, rgrosse) toronto. toroootooooo.oooooooooooooooooo edu ww.ln@reettrimmsiittte...
Abstract
Many training data attribution (TDA) methods aim to estimate how model's be- havior would change if one or more data points were removed from the training set. Methods based on implicit differentiation, such influence functions, can be made computationally efficient, but fail to account for underspecification, the implicit bias of the optimization algorithm, or multi-stage training pipelines. By contrast, methods based on unrolling address these issues but face scalability challenges. In this work, we connect the implicii-diifeeenntttiio--beb approaches and combine their benefits by introducing SOURCE, an approximate unrolling-bast -based TDA method that is computed using an influence-functt funeefonnniiniieee formula. While being computationally efficient compared to unrolling-base based approaches, SOURCE is suitable in cases where implicit-diifferentatooo approaches struggle, such in non-converged models and multi-stage training pipelines. Empirically, SOURCE outperforms existing TDA techniques coun- terfactual prediction, especially in settings where implicit- diffiiitdiiiferentatiio------- based approaches fall short.
and unrolling- -ased
Introduction
Training data attribution (TDA) techniques are motivated by understanding the relationship between
training data and the properties of trained models [92 17 29
35
70]
24,
applications in machine learning, such as detecting mislabeled data points 72 50
poisoning attacks [16] 38,69, and curating datasets [60] 13 Many TDA methods aim to perform a counterfactual prediction, which estimates how trained model's behavior would change if certain data points were removed from (or added to) the training dataset. Unlike sampling aampling-based approaches, which require repeated model retraining with different subsets of the dataset, gradient- pradient-based TDA techniques estimate an infinitesimal version of the counterfactual without model retraining. Two main strategies for gradient- based counterfactual TDA are implicit differentiation and unrolling.
TDA, most notably influence functions 28 49 uses the Implicit Function Theorem [52] to estimate the sensitivity of the optimal solution to downweighting a training data point. These methods are well-motivated for models with strongly convex objectives and provide convenient estimation algorithms that depend solely on the optimal model parameters rather than intermediate checkpoints throughout training. However, the classical formulation relies on assumptions such as the uniqueness and convergence the optimal solution, which limits their applicability to modern neural networks 6 3 77
implicit-diiffrenttttioon
By contrast, unrolling-based methods, such as SGD- HEDD-INFLENCE 31. approximate the impact of downweighting data point's gradient update on the final model parameters by backpropagating through the preceding optimization steps. Unrolling is conceptually appealing in modern neural networks because it does not rely on the uniqueness of or convergence to the optimal solution. Furthermore, it can incorporate details of the training process, such as the choice of optimizer. learning rate schedules, or data point's position during training. For example, unrolling- based
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

approaches can support TDA for multi-stage training procedures. such as in continual learning or foundation models, where the model undergoes multiple training phases with different objectives or datasets. However, they require storing all intermediate variables generated during the training process (e.g., parameter vectors for each optimization step) for backpropagation, which can be prohibitively expensive for large-scale models. Past works have considered applying unrolling to only the last epoch for large-scale models [0] restricting their applicability in analyzing the effect of removing data point at the beginning of training in analyzing multi-stage training procedures.
In this work, we connect implicit-differentee a novel algorithm that enjoys the advantages of both methods. We start from the unrolled differentia- tion perspective and, after introducing suitable approximations, arrive at an influence-funcee estimation algorithm. The key idea is to divide the training trajectory into one or more segments and approximate the distributions of gradients and Hessians as stationary within each segment. These segments may represent explicit training stages, such as in continual learning or foundation models, or changes in the Hessian and gradients throughout training. We use these estimated statistical sum- maries for each segment to approximate unrolling. Hence, we call our method SOURCE (Segmented statiOnary UnRolling for Counterfactual Estimation). While our method approximately coincides with influence functions in the simple setting of deterministic objective optimized to convergence, it applies to more general settings where unrolling is typically required.
based and unrolling- -aaed approaches and introduce
like
SOURCE inherits several key advantages from unrolling. Firstly, it allows the attribution of data points at different stages of training. providing a more comprehensive framework for TDA. Secondly, SOURCE can incorporate algorithmic choices into the analysis, accounting for learning rate schedules and the implicit bias of optimizers such as SGD or Adam [48]. Lastly, maintains a close connection with the counterfactuals, even in cases where the assumptions made in beliiiiddiffeeentattin----e methods, such as the optimality of the final parameters, are not met. However, unlike unrolling, SOURCE does not require storing all intermediate optimization variables generated during training; instead, it leverages only a handful of model checkpoints.
We evaluate SOURCE for counterfactual prediction across various tasks, including regression, image classification, and text classification. SOURCE outperforms existing TDA techniques in approximating the effect of retraining the network without groups of data points and identifying training data points that would flip predictions on some test examples when trained without them. SOURCE demonstrates siglliiiddiiferenttttionaaaa
distinct advantages in scenarios where traditional such as models that have not fully converged or those trained in multiple stages. Our empirical evidence suggests that SOURCE is a valuable TDA tool in various scenarios.
methods fall short,
2 Background
Consider a finite training dataset D = ( We assume that the model parameters O E RD are optimized with gradient-base based iterative optimizer to minimize the empirical risk on this dataset:
J(0,
Σ C(2
(1)
where is the -ivice-differentiable) loss function. We use the notation (S) to denote the optimal solution obtained when the model is trained on specific subset of the dataset S C D, and 0* to denote the optimal solution on the full dataset. In practice, it is common to employ parameters 0" that approximately minimize the empirical risk (e.g.. the result of running optimization algorithm for T iterations), as obtaining the exact optimal solution for neural networks can be challenging and may lead to overfitting (7). When necessary,
the notation 0* X. {) to indicate the final parameters obtained by training with the dataset S, along with hyperparameters À (e.g., learning rate and number of epochs) and random choices & (e.g.. parameter initialization and mini-batch order).
2.1 Training Data Attribution
TDA aims to explain model behavior on query data point 2, (e.g., test example) by referencing data points used to fit the model. The model behavior is typically quantified using a measurement [(z,, O). selected based on metrics relevant to the analysis, such as loss, margin, or log probability. Given hyperparameters À and training data point 2m D, an attribution method T T(zg. Zm, D; A) assigns

---

score to a training data point, indicating its importance in influencing the expected measurable quantity Egg [z (D; where the expectation is taken over the randomness the training process. In cases where an optimal solution to Equatiin (I) exists, is unique, and can be precisely computed, and TDA performed on this optimal solution, the TDA method simply written as (Zz: 2m: D)
One idealized TDA method is leave-one-out (LOO) retraining [8), which assesses data point's importance through counterfactual analysis. Assuming the above optimality condition is satisfied, for a chosen query data point 2y and a training data point Zm DD the LO0 score can be formulated as:
(Z0 Rm, D) = fzz, 0* (D \ =m H)) f(z,, 0*).
(2)
When the measurement defined the loss, a higher absolute LOO score signifies more substantial change in the query loss when the data point 2m is excluded from the training dataset, particularly when the model parameters are optimized for convergence.
2.2 Influence Functions
Influence functions estimate the change in optimal parameters resulting from an infinitesimal pertur- bation in the weight of a training example E DD. Assuming that an optimal solution to Equation (1) exists and is unique for various values this weight and the optimal parameters is captured through the response function:
data point's weight E [-1, 1], the relationship between
r(c) = arg min J(0, D) + N ((zm 0).
(3)
Influence functions approximate Equation (3) using the first-order Taylor expansion around 0:
dr
r(c) r(0)
0 H
VeL(zm 0*).
dc e=
where the Jacobian of the response function dr faeeec= is obtained using the Implicit Function Theorem
[52] and H = (0' D) represents the Hessian of the cost function at the optimal solution. The change the optimal parameters due to the removal of Zm can be approximated by setting e = 1:
1 H Vo VoL(zm 0*).
(D {=m) 0" ~
(5)
By applying the chain rule derivatives, influence functions estimate the change in a measurable quantity for a query example 2g as:
7P 3m D)
0") ' 0*).
(6)
We refer readers to Koh and Liang 49 for detailed derivations and discussions of influence functions. As observed in EEquaion (6) influence functions provide algorithms that only depend the optimal parameters 0* rather than intermediate checkpoints. However, when applied to neural networks, the connection to the counterfactual prediction is tenuous due to the unrealistic assumptions that the optimal solution exists, is unique, and can be found 6 3 77.. In practice, the gradients and Hessian computed using the final parameters O from single training instead of the optimal solution.
3 Methods
In this section, we introduce SOURCE, a gradient-based TDA technique combining the advantages of implicit and unrolled diffeerentiatoon. We motivate our approach from the unrolling perspective algorithm. Finally,
and, after introducing suitable approximations, arrive at an influence-funete we describe a practical instantiation of SOURCE by approximating the Hessian with the Eigenvalue- corrected Kronecker-Factoree Approximate Curvature (EK-FAC) [8 parameterization.
3.1 Motivation: Unrolling for Training Data Attribution
Consider optimizing the model parameters using SGD with a fixed batch size B, starting from the initial parameters 0, TTe The update rule at each iteration is expressed as follows:
0K+1 O2
VoL(zui, 0k),
B
For extension preconditioned gradient updates, see Appendix C
3

---

Zi+1
Computation Graph
The (1 +
+1 X
VeL
8
veL
Training
Grudient
O2
BM41
8
Accumulation
T/E V VoL(z,, 0s)
Jk+1
Ma++1 Ha+1
A simplified illustration of unrolled differentiation in SGD with batch size of and data point of interest 2m appearing at iteration k. Unrolling backpropagates through the optimization steps from Or to compute the total derivative with respect to E.
Figure I:
where T/& denotes the learning rate for iteration k, Z is the i-th data point in Bx. where BN denotes a mini-batch of examples drawn randomly with replacement from the training dataset DD and T denotes the total number of iterations. We aim to understand the effect of removing training data point Zm D on the terminal model parameters 0r. To this end, we parameterize the weight of žm as 1 1+ e, where = 0 corresponds to the original training run and = -1 represents the removal of a data point. This parameterization results in the following update rule:
Ok+1 (e) - (c(e)
T Σ
(1+ ŌEst) VoL(zEs, Ox(e));
(8)
where dR: == 1[ = 2m. is the indicator function for having selected 2m- The dependence of on will usually be suppressed for brevity. Similarly to other gradient-based TDA methods, such as influence functions, we approximate the change the terminal parameters due to the data removal (-(-1) (0) with its first-order Taylor approximation d0r/de (the notation le=o is suppressed as will always evaluated at e = 0). Let 5₁ denote the number of times zm is chosen in batch Bg. By chain rule, the contribution of iteration
the total derivative d0r fde can be found by multiplying all the Jacobian matrices along the accumulation path, giving the value
ne J&+TT.t where: B
J, = d0 = I 7/A H JR = d0, J ...
J.....JJJJJJ JA+IJA
== VoL(zm Ox).
(9)
d0
d0p
Here, H = B == VoL(z 02)
is the mini-batch Hessian for iteration & and define J k:kk
for any 0 < k < 7 by convention. A simplified illustration of unrolling is shown in Figure 1
In contrast to influence functions, unrolling does not assume uniqueness or convergence to the optimal solu- tion. An illustrative comparison of the two approaches shown in Figure? Exact influence functions differen- tiate the response function (Equation (4)) estimating the sensitivity of the optimal solution (+) to downweighting data point. By contrast, unrolling estimates the sensitiv- ity of the final model parameters (at the end training) data point; hence, it can account for details of the training process such as learning rate schedules, implicit bias of optimizers, or data point's position during training. For instance, in our illustrative example, gradient descent optimization is stopped early, such that the optimizer makes much progress in the high curvature direction and little in the low curvature direc- tion. Unrolling-bast based TDA (but not implicit differentiation) accounts for this effect, resulting in smaller influence along the low curvature direction.
Unrolled Approximation
eo
6=1
6=0
to downweighting
Response Function
Figure 2: Illustrative comparision of influ- ence functions and unrolling- Each contour represents the cost function at different values of e, which controls the degree of downweighting data point zm-
TDA.
The effect of removing 2m on any single training trajectory may be noisy and idiosyncratic. For stability, we instead consider the expectation over training trajectories, where the selection of training examples in each batch (and all downstream quantities such as the iterates 0x.) are treated as random variables We are interested the average treatment effect E [0r(-11 (O)),, where the expectation is over the batch selection, and approximate this quantity with -E [dor /de]. The expected
We assume fixed initialization Og to break the symmetry.

---

total derivative can be expanded as sum over all iterations, applying linearity of expectation:
[dor de
Te 6 1:T
Mk E [ B
E
(10)
B
In principle, we could compute a Monte Carlo estimate of this expectation by averaging many training trajectories. For each trajectory, d0r /dr be evaluated using reverse accumulation (i.e., backpropagation) on the computation graph. However, this approach prohibitively expensive as requires storing all intermediate variables for the backward pass. Furthermore, many Monte Carlo samples may be required to achieve accurate estimates.
3.2 Segmenting the Training Trajectory
To derive a more efficient algorithm for approximating expected total derivative E [110r/ak /acl. we now partition the training procedure into L segments and approximate the reverse accumulation computations for each segment with statistical summaries thereof (instead of storing all intermediate variables). Our motivations for segmenting the training procedure twofold. First, the training procedure may explicitly include multiple stages with distinct objectives or datasets, as in continual learning or foundation models. Second, the Hessians and gradients are likely evolve significantly over training, and segmenting the training allows us to approximate their distributions as stationary within a segment (rather than over the entire training run).
We index the segments as l I,..., I, with segment boundaries denoted as Tg. By convention, Ty T and To = 0 denote the end of training and beginning of training, respectively, and K2 = T. T denotes the total number of iterations within a segment. Conceptually, we can compute d0r fae using reverse accumulation over coarse-grained computation graph represented in terms of segments rather than individual iterations. The Jacobian associated with each segment is denoted as S. == JT. I: To approximate the expected total derivative E. [d0r /āe]. first rewrite Equation (10) using the segment notation introduced. We then approximate the Jacobians of different segments statistically independent (see discussion below):
d
E+1
T--1
[+1 II M-
E
=
E
II Se L/
Σ Te
&
Σ 8m1
[[Se]
E(re). (11)
de
B
where = uses our independence approximation push the expectations inward. Note that our product notation I takes l' in decreasing order from down 1.
To obtain tractable approximations for E(S₂} and [[rx). we approximate the Hessian and gradients distributions as stationary within each segment. This implies that the Hessians within a segment share a common mean H; = E H for T < k < T,- Analogously, the gradients within segment share common mean g: Elgu]. Moreover, we approximate the step sizes within each segment with their mean fj.. If these stationarity approximations are too inaccurate (e.g.. [[H]] and/or E[ge] change rapidly throughout the segment), one can improve the fidelity by carving the training trajectory into a larger number of segments, at the expense of increased computational and memory requirements. Finally, we approximate the Hessians and gradients in different time steps as statistically independent
Approximation of [[[S:]. We approximate L[S,] Equation (II) as follows:
E [ = [[J rere] AJ (I H.) Ke xs exp(-n, Se:
(12)
where the first R uses the stationary and independence approximations, and the second as uses the definition of matrix exponential. One can gain an intuition for S, by observing that matrix
'There are sources of randomness in the gradient and Hessian each step: the mini-batch sampling, the optimization iterates (which, recall, we treat as random variables). Mini-batch sampling contributes independent variability in different steps. However, autocorrelation of optimization iterates induces correlations between Hessians and gradients in different time steps. Our independence approximation amounts to neglecting these correlations.
5

---

function of H. Let - QAQ be the eigendecomposition of H. and let o be the j-th eigenvalue of Hg. The expression in Equation (12) (12) can be seen as applying the function Fs(a)) == exp(-ñt to each of the cigenvalues of He. The value is close to zero in high-curvature directions, so the
training procedure "forgets" the components of O which lie in these directions. However, information about is retained throughout the f-th segment for low-curvature directions.
Approximation of E[re]. We further approximate E[re] as follows:
T2 Σ
Te Σ
E[re] = E
m B
dRJ
2
Te (I
He)
(13)
k Te
k=
1
(I
Te He Ke
ge
expepp Sep(--TeKell Hee H, ge = Ie.
(14)
=F((o)
where Equation (13) uses the stationary and independence approximations and E E(,g] = B/N, and Equation (14) uses the finite series] and the definition of the matrix exponential. Because all the matrices commute, Fy can also be written in terms of a matrix function, defined as (a) = -1- exp (-īj //a In high-curvature directions, this term approaches 1/a, whereas in low- curvature directions, approaches to t Kg. The qualitative behavior of F. can be captured with the function (o) = 1/(a + )). where À = f K,,, as shown in Figure 6 (Appendix Cj. Applying this to H. results in approximating Equation (14) with the damped Inverse-Hessian-veett "i-'g.. ge- This is essentially the formula for influence functions, except that Hę
product (He + AI)- and Ee represent the expected Hessian and gradient rather than the terminal one, and our analysis yields explicit formula for the damping parameter A. Hence, influence functions can be regarded approximately as a special case with only a single segment, so our damped unrolling analysis gives an alternative motivation for influence functions.
Full Procedure. We derived a closed-for forem term to approximate the expected total derivative:
dd de
L
1+1
E
II P=L
Se
Fe.
(15)
N
where S; and Fe are obtained with Equation (12) and Equation (14) respectively. We term our algorithm SOURCE (Segmented statiOnary UnRolling for Counterfactual Estimation) and refer readers to Figure 3 for a visual illustration. Similarly to unrolling, SOURCE can incorporate fine- grained information about optimization trajectories into the analysis. For instance, SOURCE can support TDA for non-converged models, accounting for the total number of iterations T the model was trained with. It can also support TDA for multi-stage training pipelines: when the model was data
sequentially trained with two datasets Dj and D2, SOURCE can compute the contribution point 2m Dj that appeared in the first segment by partitioning the training trajectory into two segments and computing the expected total derivative at the first segment with Nj is the size of the first training datasct.
1 S2r1, where Ni
Given terminal parameters Or from a single training run and a query data point zy SOURCE approximates the change in the measurable quantity due to the removal of a training data point 2m as:
8+1 II
TSOURCE (Zq, (Z,,
2m. D; = Vol(z,g Or)
Σ
Se P-=L
Fe
(16)
Unlike the poggle-raainng--ue estimator for unrolling- based approaches, SOURCE does not require access to the exact location where the data point Zm was used during training, as estimates the averaged effect of removing data point within a given segment. To further account for other sources of randomness, such as model initialization, the multiple-training run estimator for SOURCE averages the final scores in Equation (1 (16) obtained for each training run with different random choices.
*Given a scalar function and a square matrix M diagonalizable as M PDP-1 the matrix function is defined as F(M)= = PF(D)P- where F(D) applies F to each diagonal entry D.
'For a symmetric square matrix M, we have 7-11 M' = (I M2 (I M) -1 When M singular, M)+
we replace (I M) with the pseudoinverse

---

Oo
0
xx+3
Or
Segment
Segment
Segment 3
H1
81
f/1
8
H2
82
122
8 S3
Hg
ga
7/
S2
F2
T33
d0g dr
d0p de
c02
at Segment
at Segment
dr
at Sogment 3
Figure 3: A simplified illustration of SOURCE with 3 segments (L 3). SOURCE divides the training trajectory into and learning rate îe as stationary within each segment & to approximate unrolling. SOURCE does not require storing the entire intermediate variables throughout training. Instead, it requires handful of checkpoints throughout training to approximate the means of the Hessians and gradients.
more segments and upproximate the gradient Be and Hessian H. distributions
3.3 Practical Algorithm for SOURCE
We now describe an instantiation of SOURCE which is practical to implement. Given the C model checkpoints saved during training. SOURCE begins by organizing them into distinct segments. These segments may represent explicit stages in training (e.g., continual learning) or account for the change in Hessian and gradient throughout training. Within each segment SOURCE estimates the stationary Hessian H2 and gradient ge by averaging the Hessian and gradient across all checkpoints in the segment. SOURCE further estimates fie by averaging the learning rates used within a segment.
However, computing Equation (15) has two practical bottlenecks for neural networks: computation of the Hessian and its matrix exponential. We fit parametric approximation to the Hessian using EK-FAC [88. EK-FAC parameterization is convenient for SOURCE as the approximate Hessian has an explicit eigendecomposition, which enables efficient computation of S and ry by applying appropriate matrix functions to the eigenvalues. Note that EK-FAC approximates the Hessian with the Gauss-Newton Hessian (GNH) 633 Unlike the Hessian, the GNH is guaranteed be positive semi- definite, as long as the loss function is convex in the model outputs [62]. The GNH approximation within EK-FAC is also advantageous for SOURCE
can avoid numerical instability computing Equation (15) especially when the Hessian has negative eigenvalues. The implementation details are
provided in Appendix D
Compared to influence functions with the same EK-FAC approximation 24 SOURCE requires computing the EK-FAC factors and training gradients for each model checkpoint when performing TDA on all segments. Hence, SOURCE is C' times more computationally expensive, where C' is the number of checkpoints. In [Appendix F.2] we introduce more computationally efficient version of SOURCE, where we average the parameters within segment instead of averaging Hessians and gradients. This variant of SOURCE I. times more computationally expensive than influence functions, as the EK-FAC factors and gradients only need to be computed once for each segment.
While described one instantiation of SOURCE with the EK-FAC approximation, we note that SOURCE can be integrated with other techniques used for approximating implici--differentiatoon based TDA methods, such TRAK (70), DATAINF (55), and LOGRA [II. For example, with TRAK, we can use random projection 43] and efficiently compute the averaged Hessian and gradients in the diver-dimensional space. TRAK can be advantageous over the EK-FAC approximation when there are a large number of query data points, as it caches the compressed training gradients in memory, avoiding the need to recompute them for each query.
4 Related Works
Modern TDA techniques for neural networks can be broadly categorized into three main groups: sampling based, representation--asee and gradient-based. For comprehensive overview of TDA, including practical applications, we refer the reader to Hammoudeh and Lowd [27] and Mucsányí et al. [66]. Sampling- (or retraining based) approaches, such as Shapley-value estimators 78 21 approximate counterfactuals by repeatedly retraining models on different data subsets. Although effective, these methods are often impractical for modern neural networks due to the significant computational cost of repeated model retraining.
87 AMPLING 17 (66), DATAMODELS (35), and DATA BANZHAF (5] 86),
7

---

Representation based techniques evaluate the relevance between training and query data point by examining the similarity in their representation space (e.g., the output of the last hidden layer) [0] These techniques offer computational advantages compared to other attribution methods, as they only require forward passes through the trained network. Rajani et al. 774 further improves efficiency by caching all hidden representations of the training dataset and using approximate nearest neighbor search [42]. Past works have also proposed model-agnostic TDA approaches, such as computing the similarity between query and training sequences with BM25 [75 for language models i1 |6 or with an embedding vector obtained from a separate pre-trained self-supervised model for image classification tasks [79]. However, sppresentationnbasee and aiput--imilartty based techniques lack connection to the counterfactual and do not provide a notion of negatively influential data points.
Two main strategies for gradient- TDA are implicit differentiation and unrolling. To the best of our knowledge, the largest model to which exact unrolling has been applied a 300 thousand parameter model [31]. Our experiments in infSection eleover TDA for models ranging from 560 thousand parameters (MNIST & MLP) to 120 million parameters (WikiText-2 & GPT-2). IGD-INLLEENCE
31 also considers applying unrolling to only the last epoch for large-scale models. However, this limits its applicability in analyzing the effect of removing a data point at the beginning of training or analyzing multi-stage training processes. In contrast, HYDRA [0] approximates the mini-batch Hessian H3 in Equation (10) as zero when computing the total derivatives, avoiding the need to compute besssian-vector products (HVPs) for each optimization step. However, in |Apppendix F.I we empirically observe that an accurate approximation of the Hessian is important to achieve good TDA performance. Both approaches require storing a large number of optimization variables during training. Relatedly, Nickl et al. [68] use local perturbation methods [37 to approximate the data point's sensitivity to the training trajectory.
Apart from amplicit-differentattie based and unrolling-basee approaches, TRACIN 72 is another prominent maadient-based TDA technique, which estimates the importance of training data point by approximating the total change in the query's measurable quantity with the gradient update from this data point throughout training. Similarly to SOURCE, the practical version of TRACIN (TRACINCP) leverages intermediate checkpoints saved during training. While TRACINCP is straightforward to implement it does not involve approximation of the Hessians, its connection to the counterfactual is unclear [27, 777. However, past works have shown its strengths in downstream tasks, such as mislabeled data detection 722 and curating fine-tuning data (90)
5 Experiments
Our experiments investigate two key questions: (1) How does SOURCE compare to existing TDA techniques, measured by the linear datamodeling score (LDS) 77 and through subset removal counterfactual evaluation [3 93
70 79 ?9? (2) Can SOURCE support data attribution in situations where implicit-difiie implicit-diffferentiatoo based approaches struggle, particularly with models that have not converged or have been trained in multiple stages with different objectives or datasets?
We compare SOURCE against existing TDA techniques: representation similarity (REPSIM) 9 30) TRACIN 72.. TRAK 770 and influence functions (IF) with the EK-FAC approximation 24 For consistency with Park et al. (70). the measurement ff is defined as the margin for classification tasks and the absolute error for regression tasks. Our evaluations are conducted under two separate settings. First is single model setup, where TDA techniques use model checkpoints from single training run. Unless specified otherwise, REPSIM, TRAK, and IF are computed at the final training checkpoint, and TRACIN and SOURCE use at most 6 intermediate checkpoints saved throughout training. In the second setting, TDA techniques use checkpoints from 10 distinct models, each trained with varying sources of randomness. Past works have shown that ensembling attribution scores across models can improve TDA performance 770 67). For all TDA techniques, including SOURCE, we simply average the final attribution scores from distinctly trained models with the full dataset, except for TRAK, which uses its custom ensembling procedures with models each trained on sampled 50% of the original dataset.
Our experiments consider diverse machine learning tasks, including: (a) regression using datasets from the UCI Repository 45). (b) image classification with datasets such as MNIST (7)) NNST
[9T], CIFAR-10 (53,, RotatedMNIST (0). and PACS (58), and (c) text classification using the GLUE
benchmark [85] Our tasks can be categorized into three groups:
8

---

TRAK
SOURCE
Model
Concrese (MI.P)
FishionMNIST
CIFAR- (ResNe)))
RTE: (BHRT)
0.5
0.5
0.5
0.0
LDS
Concrete-N (MI
FashionMNISTEN (MI.P)
RoGmaIMNIST (MI
PACS (ResNes-S0)
0.2
0.5
0.5
0.2
0.1
0.0
0.0
0.0
Figure 4: LDS at 0.5 for SOURCE and baseline techniques on regression, image classifi- cation, and text classification tasks. The error bars represent 95% bootstrap confidence intervals (Appendix E.2)
Concrete, FaShionMNISTT, CIFAR-10, & RTE. Models fully trained using fixed dataset D, where implicit-differentattio based methods are expected to perform similarly unrolling-based methods. We use 6 intermediate checkpoints throughout training for TRACIN and SOURCE. SOURCE use 3 segments (I. 3) equally partitioned at the early, middle, and late stages of training to account for the changes in distributions of Hessian and gradients during training.
Concrete-N & Fashion Non-converged models trained with a smaller number of update steps. This a challenging setup for implicit-differentaaton based methods, such as TRAK and IF, as they inherently assume that TDA is performed on the optimal solution. We use versions of the Concrete and FashionMNIST datasets that have been modified, either by corrupting target values or relabeling 30% of the data points. Then, train the models for only 3 epochs to avoid overfitting. We use 3 checkpoints (at the end of each epoch) for TRACIN and SOURCE (L 3).
RotatedMNIST & PACS. Models initially trained with dataset D,, and subsequently trained with another dataset D2 (a common setup in continual learning). We use test examples from D2 for query data points and attribute the final model's behavior to the first dataset. Since implicit-diiferentattoo based methods do not provide any way to separate multiple stages of training, for TRAK and IF, we simply combine the data from both stages into a larger dataset for TDA. We use two segments for SOURCE, partitioned at different stages, and perform TDA only for the first segment. Our experiments use the RotaTedMNIST and PACS datasets, both containing multiple data distributions. We select one of these domains for the second training stage, while the remaining ones used in the first stage.
The detailed description of the experimental setup is provided [Appendix E| Additional results, including comparisons additional tasks and with additional baselines, further analysis linear models, and visualizations the top influential images obtained by each TDA technique, are shown in Appendix F
5.1 TDA Evaluations with Linear Datamodeling Score (LDS)
We evaluate TDA techniques using the linear datamodeling score (LDS) from Park al. 770. To
compute LDS,, we first generate M random subsets {S,) from the training dataset, each containing
[aN]] data points for some E (0. 1). Given query data point 2g and hyperparameters À used to train the original model, the expected measurable quantity for each data subset (z (S;; X, {
is estimated by retraining the model A times under different random choices (which requires M R model retrainings in total). The LDS measures the Spearman correlation [8] between the estimated quantities and the predictions made by the TDA method. Note that, although TDA method in Section 2.1 assigns a score to each pair of a query and training data point, the inherently additive nature of most TDA techniques allows for the computation of a group prediction score for the data subset by summing the individual scores attributed each data point within this subset. The final LDS is obtained by averaging the scores across many (typically up 2000) query data points. We use 100 data subsets (M = 100) and conduct a minimum of 5 retraining iterations (R 2 5) for each subset. We refer readers to Appendix A for the detailed formulation and to Appendix E.2 for the practical procedures.
The LDS at = 0.5 for SOURCE and baseline TDA techniques are shown in Figure 4 SOURCE consistently outperforms all baseline methods a single model setup, achieving high L.DS. When aggregating TDA scores from multiple models, we observe large improvement in the LDS, particu-

---

FashionMNIST (MLP)
CIFAR- 10 (ResNet-9) 9)
RTE (BERT)
8 BC
Misc Misclassified Test Examples
0.50
0.50
$00
0.25
080
0.00
0..00
0.0
200
300
500
1000
50
MNSST (MLP)
RotatedMNIST (MLP)
REPSIM
0.5
TRACIN
Model
TRAK
10 Models
100
200
300
1000 2000
Number of Training Examples Removed to Flip Predictions
Figure 5: Subset removal counterfactual evaluation for SOURCE and baseline TDA techniques, where the top positively influential data points predicted by each TDA method are removed, and the model is retrained to examine if (previously correctly classified) test data point gets misclassified.
larly for TRAK, IF, and SOURCE. Our method achieves the highest LDS across all tasks, except for the CIFAR-16 10 classification task using ResNet-9. SOURCE especially performs strongly against other baseline techniques on settings that pose challenges to implicit-differentittion based approaches (e.g., non-converged models and models trained with multiple stages), and indeed, even the non-ensembled version of SOURCE typically outperforms the ensembled versions of the competing methods.
5.2 TDA Evaluations with Subset Removal
Subset removal counterfactual evaluation examines the change in model behavior before and after removing data points highly ranked by TDA technique. For classification tasks, we consider 100 random
test data points that are correctly classified when trained with the full dataset (across seeds) and, for each test data point Zy. examine if removing and retraining without the top-k positively influential data points can cause misclassification on average (over 3 random seeds). By assessing the impact of removing influential training examples the model's performance, counterfactual evaluation provides a direct measure of the effectiveness of TDA techniques in identifying data points that significantly contribute to the model's behavior: The detailed procedures are described in Appendix F.3, In Figure 5) we show the fraction of test examples (out of the selected 100 test points) that get misclassified identified by each TDA method. We observe that SOURCE better identifies the top influential data points causing misclassification than other baseline TDA techniques. The improvement is more substantial for settings that pose challenges to implicit-diifeerentttio based methods.
average after removing most k positively influential training examples
6 Conclusion
We introduced SOURCE (Segmented statiOnary UnRolling for Counterfactual Estimation), a novel TDA technique that combines the strengths of amppiciidaiferrnniation--aasssss and unrolling-based techniques. SOURCE approximates unrolled differentiation by partitioning the training trajectory into one more segments and approximating the gradients and Hessians stationary within each segment, yielding an influence-fune seteeeffuntioon-iiee estimation algorithm. We showed one instantiation of SOURCE by approximating the Hessian with the EK-FAC parameterization, On diverse task set, we demonstrated SOURCE's effectiveness compared to existing data attribution techniques, especially when the network has not converged has been trained with multiple stages.
Acknowledgements
The authors would like thank Jenny Bao, Rob Brekelmans, Sang Keun Choe, Lev McKinney, Andrew Wang, and Arielle Zhang for their helpful feedback on the manuscript. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute: www. vwwtorenstiiiiooooe ai/#partners! JB was funded by OpenPhilanthoopy and Good Ventures. RG acknowledges support from the Canada CIFAR AI Chairs program.
10

---

References
[1] E. Akyürek, T. Bolukbasi, F. Liu, B. Xiong, I. Tenney, Andreas, and K. Guu. Towards tracing knowledge in language models back to the training data. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2426,2446, 2022.
[2] W. E. Arnoldi. The principle of minimized iterations in the solution of the matrix eigenvalue problem. Quarterly of applied mathematics, 9(1): 1951.
[3] J. Bae, N. Ng. A. Lo, M. Ghassemi, and R. B. Grosse. influencc functions are the answer, then what is the question? Advances in Neural Information Processing Systems, 35:17953-17967, 2022.
[4] J. Bae, P. Vicol, J. Z. HaoChen, and R. B. Grosse. Amortized proximal optimization. Advances in Neural Information Processing Systems, 35:8982-8997, 2022.
[5] F. Banzhaf III. Weighted voting doesn't work: mathematical analysis. Rutgers Rev., 19: 317, 1964.
[6] S. Basu, P. Pope, and S. Feizi. Influence functions in deep learning are fragile, In International Conference on Learning Representations, 2020.
[7] Y. Bengio. Practical recommendations for gradient-based training of deep architectures. In Neural Networks: Tricks of the Trade: Second Edition, pages 437-478. Springer, 2012.
[8] J. Brophy, Z. Hammoudeh, and D. Lowd. Adapting and evaluating influence-estimation methods for gradient-boosted decision trees. Journal of Machine Learning Research, 24(154): 1-48, 2023.
[9] R. Caruana, H. Kangarloo, J. D. Dionisio, U. Sinha, and D. Johnson. Case-based explanation of non-case-basee learning methods. Proceedings of the AMIA Symposium, page 212. American Medical Informatics Association, 1999.
[10] Y. Chen, B. Li, H. Yu, P. Wu, and C. Miao. Hydra: Hypergradient data relevance analysis for interpreting deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7081-7089, 2021.
[11] S.K. K. Choe, H. Ahn, J. Bae, Zhao, M. Kang, Y. Chung, A. Pratapa, W. Neiswanger, E. Strubell, T. Mitamura, et al. What is your data worth to gpt? lim-scale data valuation with influence functions. arXiv preprint arXiv:2405. 13954, 2024.
[12] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, 2018.
[13] L. Engstrom, A. Feldmann, and Madry. DsDm: Model-aware dataset selection with data- models, 2024.
[14] J. R. Epifano, R. P. Ramachandran, A. J. Masino, and G. Rasool. Revisiting the fragility of influence functions. Neural Networks, 162:581-588, 2023.
[15] R. Eschenhagen, A. Immer, R. Turner, F. Schneider, and P. Hennig. Kronecker-factored approximate curvature for modern neural network architectures. Advances in Neural Information Processing Systems, 36, 2024.
[16] M. Fang, N. Z. Gong, and J. Liu. Influence function based data poisoning attacks to top-n recommender systems. Proceedings of The Web Conference 2020, pages 3019-3025, 2020.
[17] V. Feldman and C. Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2891, 2020.
[18] T. George, C. Laurent, X. Bouthillier, N. Ballas, and P. Vincent. Fast approximate natural gradient descent in kronecker factored eigenbasis. Advances Neural Information Processing Systems, 31, 2018.

---

[19] K. Georgiev, J. Vendrow, H. Salman, S. M. Park, and A. Madry. The journey, not the destination: How data guides diffusion models, 2023.
[20] M. Ghifary, W. B. Kleijn, M. Zhang, and D. Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pages 2551-2559, 2015.
[21] A. Ghorbani and J. Zou. Data Shapley: Equitable valuation of data for machine learning. In International Conference Machine Learning, pages 2242-2251, PMLR, 2019.
[22] R. Grosse. University of Toronto CSC2541, Topics in Machine Learning: Neu-
ral Net Training Dynamics, Chapter 4: Second-Order Optimization. Lecture Notes, 2021. URL penttps:/wwwrrr /wwp//wwwwwc///oo///////// toronto. wtw.rrgeeoouseeeseeeeerseeeeeeee Beadings/LO4_seennn second order.
[23] R. Grosse and J. Martens. A kronecker-factored approximate fisher matrix for convolution layers. In International Conference Machine Learming, pages 573-582. PMLR, 2016.
[24] R. Grosse, J. Bae, C. Anil, N. Elhage. A. Tamkin, A. Tajdini, B. Steiner, D. Li, E. Durmus, E. Perez, E. Hubinger, K. Lukošiūté, K. Nguyen, N. Joseph, S. McCandlish, J. Kaplan, and S. R. Bowman. Studying large language model generalization with influence functions, 2023.
[25] I. Gulrajani and D. Lopez-Paz. In search of lost domain generalization, 2020.
[26] V. Gupta, Koren, and Y. Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pages 1842-1850, PMLR, 2018.
[27] Z. Hammoudeh and D. Lowd. Training data influence analysis and estimation: A survey. Machine Learning, pages 1-53, 2024.
[28] R. Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, (9(346):383-393) 1974.
[29] X. Han, B. C. Wallace, and Y. Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5553-5563, 2020.
[30] K. Hanawa. S. Yokoi, S. Hara, and K. Inui. Evaluation of similarity-basee explanations. In International Conference on Learning Representations, 2020.
[31] S. Hara, A. Nitanda, and T. Maehara. Data cleansing for models trained with sgd. Advances in Neural Information Processing Systems, 32, 2019.
[32] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[33] S. Hooker, D. Erhan, P.-. Kindermans, and B. Kim. A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems, 32, 2019.
[34] E. J. Hu, Y. Shen, P. Wallis, 7. Allen-Zhu, Y. Li, S. Waang L., Wang, and W. Chen. Lora: Low-rank adaptation large language models, 2021.
[35] A. Ilyas, S. M. Park, Engstrom, G. Leclere, and A. Madry, Datamodels: Predicting predictions from training data. In International Conference on Machine Learning. 2022.
[36] M. R. Izadi, Y. Fang, R. Stevenson, and L. Lin. Optimization of graph neural networks with natural gradient descent. In 2020 IEEE international conference on big data, pages 171-179. IEEE, 2020.
[37] L. A. Jaeckel. The infinitesimal jackknife. Bell Telephone Laboratories, 1972.
[38] M. Jagielski, G. Severi, N. Pousette Harger, and A. Oprea. Subpopulation data poisoning attacks. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 3104 31202 2021.
12

---

[39] R. Jia, D. Dao, B. Wang, F. A. Hubis, N. Hynes, N. M. Gürel, B. Li, C. Zhang, D. Song. and C. J. Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference Artificial Intelligence and Statistics, pages 1167-1 176. PMLR, 2019.
[40] R. Jia, F. Wu, X. Sun, J. Xu, D. Dao, B. Kailkhura, C. Zhang. B. Li, and D. Song. Scalability vs. utility: Do we have to sacrifice one for the other in data importance quantification? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2239-8247, 2021.
[41] K. F. Jiang, W. Liang, Zou, and Y. Kwon. Opendataval: A unified benchmark for data valuation, 2023.
[42] Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547, 2019.
[43] W. B. Johnson, J. Lindenstrauss, and G. Schechtman. Extensions of lipschitz maps into banach spaces. Israel Journal of Mathematics, 54(2):129-138, 1986.
[44] K. and Søgaard. Revisiting methods for finding influential examples, 2021.
[45] M. Kelly, R. Longjohn, and K. Nottingham. The UCI machine learning repository, 2023, URL https: https://archive..u ics. uci.eduu edu
[46] R. Khanna, B. Kim, J. Ghosh, and S. Koyejo. Interpreting black box predictions using fisher kernels. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3382-3390. PMLR, 2019.
[47] S. Kim, K. Kim, and E. Yang. GEX: flexible method for approximating influence via geometric ensemble. Advances in Neural Information Processing Systems, 36, 2024.
[48] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2014.
[49] P. W. Koh and P. Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pages 1885-1894. PMLR, 2017.
[50] S. Kong, Y. Shen, and L. Huang. Resolving training biases via influence-basee data relabeling. In International Conference on Learning Representations, 2021.
[51] N. Konz, C. Godfrey, M. Shapiro, J. Tu, H. Kvinge, and D. Brown. Attributing learned concepts in neural networks to training data, 2023.
[52] S. G. Krantz and H. R. Parks. The Implicit Function Theorem: History, theory, and applications. Springer Science & Business Media, 2002.
[53] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. 2009.
[54] Y. Kwon and J. Zou. Beta Shapley: A unified and noise-reduced data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pages 7780-8802 PMLR, 2022.
[55] Y. Kwon, E. Wu, K. Wu, and Zou. DataInf: Efficiently estimating data influence in lora-tuned Ilms and diffusion models. In International Conference Learning Representations, 2023.
[56] F. Ladhak, E. Durmus, and T. B. Hashimoto. Contrastive error attribution for finetuned language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1482-11498. 2023.
[57] Y. LeCun, C. Cortes, and C. Burges. MNIST handwritten digit database. ATT Labs, 2, 2010.
[58] D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales. Deeper, broader and artier domain gen- eralization. In Proceedings of the IEEE international conference computer vision, pages 5542-5550, 2017.
[59] D. C. Liu and J. Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical progranming, 4503-3) 1989.
13

---

[60] Z. Liu, H. Ding, H. Zhong, W. J. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9274-9283 2021.
J. Dai, and C. He. Influence selection for active learning. In
[61] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. International Conference on Learning Representations, 2018.
[62] J. Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146):1-76, 2020.
[63] J. Martens and R. Grosse. Optimizing neural networks with Cronecker-factored approximate curvature. In International Conference Machine Learning, pages 2408-2417. PMLR, 2015.
[64] J. Martens, J. Ba, and M. Johnson. Ktonecker-factoree curvature approximations for recurrent neural networks. In International Conference on Learning Representations, 2018.
[65] S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. In Interna- tional Conference on Learning Representations, 2016.
[66] B. Mucsányi, M. Kirchhof, E. Nguyen, A. Rubinstein, and S. J. Oh. Trustworthy machine learning, 2023.
[67] E. Nguyen, M. Seo, and S. J. Oh. A bayesian approach to analysing training data attribution in deep learning. Advances in Neural Information Processing Systems, 36, 2024.
[68] Nickl, L. Xu, D. Tailor, T. Möllenhoff, and M. E. E. Khan. The memory- equation: Understanding model's 's sensitivity to data. Advances in Neural Information Processing Systems, 36, 2024.
[69] S. Oh, B. Ustun, J. McAuley, and S. Kumar. Rank list sensitivity of recommender systems interaction perturbations. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 1584-1594 2022.
[70] M. Park, K. Georgiev, A. Ilyas, G. Leclere, and A. Madry. TRAK: Auributing model behavior at scale. In International Conference on Machine Learning, pages 27074-27113, PMLR, 2023.
[71] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. Vito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in PyTorch. 2017.
[72] G. Pruthi, F. Liu, S. Kale, and M. Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920-19930, 2020.
[73] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019.
[74] N. F. Rajani, B. Krause, W. Yin, T. Niu, R. Socher, and C. Xiong. Explaining and improving model behavior with k nearest neighbor representations, 2020.
[75] S. E. Robertson, S. Walker, S. Jones, M. M. Beauulie,, and M. Gatford. Okapi at TREC-3. Nist Special Publication Sp, 109:109, 1995.
[76] A. Schioppa, P. Zablotskaia, D. Vilar, and A. Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8179-8186, 2022.
[77] A. Schioppa, K. Filippova, I. Titov, and P. Zablotskaia. Theoretical and practical perspectives on what influence functions do. Advances in Neural Information Processing Systems, 36, 2024.
[78] I.. Shapley. A value for n-person games. 1953.
[79] V. Singla, P. Sandoval-Segura, M. Goldblum, Geiping, and T. Goldstein. simple and efficient baseline for data attribution images. arXiv preprint arXiv:2311.03386. 2023.
14

---

[80] J. W. Smith, J. E. Everhart, W. Dickson, W. C. Knowler, and R. S. Johannes. Using the adap learning algorithm forecast the onset of diabetes mellitus. In Proceedings of the annual symposium on computer application in medical care, page 261. American Medical Informatics Association, 1988.
[81] C. Spearman. The proof and measurement of association between two things. The American journal of psychology, 100(344):41-471, 1987.
[82] T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
[83] A. DOI: https://0oiorr//00244////////
Tsanas and M. Little. Parkinsons Telemonitoring. UCI Machine Learning Repository, 2009,
[84] A. Vaswani, Shazeer, N. Parmar, Uszkoreit, I. Jones, A. N. Gomez, Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[85] A. Wang. A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding, 2019.
[86] T. Wang and R. Jia. Data Banzhaf: A robust data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pages 6388-6421. PMLR, 2023.
[87] T. Wang, Y. Zhu, Y.-X. Wang. R. Jia, and P. Mittal. A Srivacy-friendly approach data valuation. Advances in Neural Information Processing Systems, 36, 2024.
[88] S. Weisberg and R. D. Cook. Residuals and influence in regression. 1982.
[89] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz. J. Davison, S. Shleifer, von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Transformers: Statc-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, Oct. 2020. Association for Computational Linguistics. URL htttt::://///t //ww:////wwwww... aclweb. org/ anthology
2020. emn.p-demos........ 6
[90] M. Xia, S. Malladi, S. Gururangan, S. Arora, and D. Chen. LESS: Selecting influential data for targeted instruction tuning, 2024.
[91] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms, 2017.
[92] C.-K. Yeh, Kim, I. E.-H. Yen, and K. Ravikumar. Representer point selection for explaining deep neural networks. Advances in neural information processing systems, 31, 2018.
[93] C.-K. Yeh, A. Taly, M. Sundararajan, F. Liu, and P. Ravikumar. First is better than last for language data influence. Advances in Neural Information Processing Systems, 35:32285-32298, 2022.
[94] I.-C. Yeh. Concrete Compressive Strength. UCI Machine Learning Repository, 2007. DOI: httss//lb..or//024442///CK
[95] S. Zagoruyko and N. Komodakis. Wide residual networks, 2017.
[96] C. Zhang, D. Ippolito, K. Lee, M. Jagielski, F. Tramèr, and N. Carlini. Counterfactual memo- rization in neural language models. Advances in Neural Information Processing Systems, 36: 39321-39363 2023.
[97] X. Zheng, T. Pang, C. Du, J. Jiang, and M. Lin. Intriguing properties of data attribution on diffusion models. In International Conference on Learning Representations, 2023.
15

---

A Evaluation of TDA Techniques
Given the focus on counterfactual prediction in many TDA methods, LOO estimates, defined in Equation (2) are often considered a ground truth for evaluating these techniques. However, the computation of LOO scores in neural networks encounters several computational and conceptual challenges, as detailed in Appendix B For robust and standardized measure for evaluating TDA techniques, we instead use the linear datamodeling score (LDS) from Park et al. 770 as well as subset
removal counterfactual evaluation [33 93 35 97 70
8 79 191.
Linear Datamodeling Score (LDS). A TDA method r, as detailed in Section 2.1
assigns a score
to each pair of a query and training data point. The inherently additive nature of most TDA techniques allows for the computation of a group attribution score for specific training data subset S C 'D The importance of S on the measurable quantity f is estimated by summing the individual scores attributed to each data point within this subset. The group attribution is expressed as follows:
gr (Zg, S, A;D = Σ T D;A).
(17)
ZES
Consider M random subsets ( from the training dataset, each containing [a_V] data points for some (0, 1). Given hyperparameter configuration À train the model, the LDS for query point 2g is defined as:
LDS. r) = ((E [f(zg, (S;; : E [M], (0.(2,, D) : e [M]), (18)
where p represents the Spearman correlation 81. This expected measurable quantity is approximated by retraining the network R times under different random choices. The final LDS is obtained by averaging the scores across many (typically up to 2000) query data points. In our experiments, 100 data subsets (M 100) and conduct maximum of 100 retraining iterations (R
(5, 10, 20, 100)) for each subset to compute the LDS.
Subset Removal Counterfactual Evaluation. Subset removal counterfactual evaluation examines the change in model behavior before and after removing data points that are highly ranked by an attribution technique. For classification tasks, we consider 100 test data points that are correctly classified when trained with the full dataset and, for each test data point, examine if removing and retraining without the top-k positively influential data points can cause misclassification on average (trained under different random choices) By assessing the impact of removing influential data points on the model's performance, counterfactual evaluation provides a direct measure of the effectiveness of TDA techniques in identifying data points that significantly contribute to the model's behavior.
Downstream Task Evaluation. TDA techniques have also been evaluated on their performance on downstream tasks, such as mislabeled data detection [46] 72 47), class detection [30 (5). finding hallucinations in the training dataset [56], and retrieving factual knowledge from the training dataset [1. These tasks offer additional insights into the effectiveness and applicability of data attribution methods in practical scenarios. However, the connections between these tasks and counterfactual prediction are often unclear [44] 770, and it is uncertain whether algorithmic improvements in counterfactual prediction will directly result in improved performance on these downstream tasks.
B Limitations of Leave-One-Out Estimates
The computation of leave-one-out (LOO) scores in Equation presents several computational and conceptual challenges for neural networks. Firstly, calculating the LOO score for all training data points requires retraining the model times, where N is the size of the training dataset. This process can be prohibitively expensive for large datasets and network architectures.
Moreover, the formulation of LOO assumes that an optimal solution to Equation (1) exists, unique, and can be precisely computed, and that TDA is performed on this optimal solution. However, within the context of neural networks, these assumptions often do not hold, leading to ambiguities in the
'The literature also uses terms such as helpful 49 proponent influential training data points.
72 and excitatory
92 to describe positively
16

---

computation of LOO estimates. Previous works have investigated various LOO variants as means establish counterfactual ground truths 49
677. For example, Koh and Liang 49] 6 formulated the LOO ground truch by training the network for an additional number of steps from the final parameters A without a specific training data point. However, as noted by Bae et al. (3), these estimates may reflect the effect of training the network for additional steps instead of retraining without a data point, especially when the network has not converged.
and Basu et al.
A more standardized extension of LOO for neural networks is the expected leave-one-out (ELOO)
retraining 44 formulated as:
TELOO 3m; D; A) E (zg (D \ {zm {,) - Eg [f (D;;,, (19)
where À denotes the hyperparameters used to train the model, and the expectation taken over randomness in the training process (typically estimated by retraining the network R times). Note that the ELOO can also be seen as the ground truth for the linear datamodeling score (LDS) (defined in 1 1/n. Past works have demonstrated the unreliability of ELOO estimates
Appendix A) with due to stochasticity in model training, such as model initialization and batch ordering 44 14 67]. Specifically, Nguyen et al. [67] observed that the noise from the stochasticity often overshadows the actual signal of removing a single data point. In Appendixx F.3] we also observe that the LDS significantly drops at 1 1 I/N, suggesting that the counterfactual ground truth for removing a single data point can be difficult to obtain, as can be extremely noisy for most training data examples.
C SOURCE with Preconditioning Matrix
we motivated our proposed algorithm, SOURCE, for cases where the parameters are optimized using stochastic gradient descent (SGD). In this section, we present the formulation of SOURCE when preconditioned optimizers, such as RMSProp [82] Adam [48), and K-FAC 63 are used to train the model.
In Section 3.1
To investigate the impact of removing training data point 2m E D, we follow similar derivation as in Section 3.1 but now considering the preconditioning matrix:
0c+1(E) Ox(e)
7h B
PA
Σ duie VoL Ox(e))
(20)
where P1 a (positive definite) preconditioning matrix and 5αs = 122 2m is the indicator function for having selected Zym-
By applying the chain rule derivatives, the contribution of iteration to the total derivative can be found by multiplying all the Jacobian matrices along the backward accumulation path, giving the value Dt. J.+1:1 PEgE. where we have:
d0
J
I mh PHH
d0:
d0
(21)
Jk
J JR
d0,
gk
VoL( (zm, 0z).
Hence, by applying the linearity of expectation, the expected total derivative of the terminal parameters or with respect to the perturbation expressed as:
d0r de
T
F B
(22)
k=
As discussed in Section 3.2, group the training trajectories into multiple segments approximate the expected total derivative for each segment with statistical summaries thereof. addition to the approximations introduced in Seccion 3.2 we approximate preconditioning matrices as stationary
within a segment and represent it as P Py for T I k < Tt.
17

---

Approximation of E[Se]. We approximate E[S;] in Equation (11) as follows:
F [c]]
E [ : (I H. exp( Tie H,H,
(23)
P 1/2 exp(-ñike K H.P p 1/2 = S...
Note that the last line uses the properties of the matrix exponential
Approximation E(re]-
We further approximate E[rx] as follows:
T
E E(re]
E
Σ
The
J& P Pagk
(24)
B
k= T2-1
Σ
- - P.H.H, 1
(25)
N
k=T:
(I 1/ P,H, g
(26)
exp( -j: K
ge
(27)
N
11/2 (I exp( - K P//" Fe. N
(28)
where we define M, : 11 H.P and the last line uses the properties of matrix exponential, as done in Equation (23) Similarly to our analysis presented in Section 3.2 we can represent Fę with the matrix function of Mg. Let M, QAQ' be the cigendecomposition of M, and let ø, be the j-th eigenvalue of Mg. The expression can be seen as applying the matrix function, defined as:
Fr(o)= = 1 - exp (-.n
(29)
The qualitative behavior of F, can be captured with the function Fiv 1/(a + A), where = 1/2 KK= (see Secciinn 3.2 for details). Hence, one way to understand Equation (28) is by expressing it as the damped inverse Hessian-vector product (iHVP):
Ft =
1/2
[M. - -1 g.
(30)
N
p 1/2 1/2 -II-1P//
(31)
N
H + XP AP 1ー ge.
(32)
In a case where Pe a diagonal matrix, Equation (32) can be seen as a special case for influence functions with a specific diagonal damping term AP, Using the derived S, and Fe, we approximate the total expected derivative using Equation (15)
D Implementation Details
This section describes the Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK- FAC) [1]] and how we computed SOURCE using this EK-FAC parameterization. The code for implementing SOURCE (as well as baseline techniques) will be provided https : ://ii com/ pomonam Domonam/kronf /trfo luence For details the EK-FAC approximation specific to influence functions, we refer readers to Grosse et (24..
For square matrix M square positive definite matrix D, we exp(M) Ex 1 M*
1//2
(D 1-1/MD 1/2) ) D D-1/22 1/2 exp(D 3/2
E
18

---

10"
F2
10"
F Fuy
10-2
10
10-2
10"
10a
Figure 6: A demonstration of the match in qualitative behavior between Fr and Finv, where we set ?e 0.1 and K₃ = 100.
D.1 Eigenvalue- Kronecker-Faactore Approximate Curvature (EK-FAC)
CroneckerFFactoree Approximate Curvature (K-FAC)
63 and EK-FAC [8] introduce a parametric
approximation to the Fisher information matrix (FIM) of a neural network, defined
- [V log 0, Ve log (()
(33)
where Pdata is the data distribution and Pp ( is the model's output distribution. For many commonly used loss functions, such as softmax-cross-entrooy and squared-error, the FIM is equivalent the Gauss-New Hessian (GNH) [62], denoted as G. The GNH can be seen as an approximation to the Hessian H. where the network is linearized around the current parameters [22. Different from the Hessian, the GNH is guaranteed to be positive with respect to the model output.
(PSD) when the loss function is convex
While K-FAC and EK-FAC were originally formulated for multilayer perceptrons (MLPs), they were later extended to other architectures, such convolutional neural networks [23], recurrent neural networks [64]. graph neural networks [36]. or to be learnable by gradient- basedd optimizers (4]. We refer readers to Eschenhagen al. [15) for comprehensive overview. This section describes the EK-FAC formulation in the context of MLPs.
Consider I-th layer of the network with input activations a4-1 E R' and pre-activation output Rº such that s = Wia---, where W E ROx/ is the weight matrix (we drop the layer subscript avoid clutter and ignore the bias term for simplicity). The pseudo-gradee gradient (where the target is sampled from the model's output distribution; see Eqqatiin (33) is given by DW := Dsa K-FAC makes is approximated as block-diagonal with each block corresponding to GNH of some specific layer, and (2) input activations and pseudo- gradient of the pre-activations Ds are independent under the model's predictive distribution. The layerwise GNH can be approximated as:
approximations: (1) layerwise independence approximation, where GNH
G --E E [vec(DW)ec(DW)] = E [aa" & DsDs
E[aa @ E [DsDs = & S, (34)
where denotes the Kronecker product. The matrices E RixT and S ROxO in Equation (34) represent the uncentered covariance matrices of the activations and the pseudo-gradients with respect to the pre-activations, respectively. These covariance matrices can estimated by computing the statistics over many data batches and taking the average.
Denoting the eigendecomposition of these covariance matrices as A QAAa Q and S As Q using properties of the Kronecker product, we can express the eigendecomposition of A ② B as:
& B = (QA @ (A & ASA @ Qs)T.
(35)
EK-FAC introduces accurate approximation to the GNH by introducing compact represen- tation of the eigenvalues (instead of representing them as the Kronecker product AA & As). The layerwise GNH for EK-FAC is represented as follows:
G R (Qa s QS)A(QA 69 T.)T.
(36)
Here, the corrected eigenvalues A E 100x10 are defined as:
A = F ((QA @ Qs (D))))
(37)
The corrected eigenvalues in Equation (37) minimize the approximation error with the GNH measured by the Frobenius norm, where we refer readers to George et al. [18] for the derivations.
19

---

D.2 EK-FAC Computations for SOURCE
As detailed in Section 3.3 our practical instantiation of SOURCE requires averaging the Hessians across checkpoints within a segment. We use a common averaging scheme in the optimization literature 66 18 26 to compute the averaged EK-FAC factors. We first compute the activation covariance matrices A and aeeuoo-grdiied covariance matrices S for all model checkpoints. These matrices are obtained by computing the statistics over all data points once (1 epoch). Then, we take the average over these covariance matrices to obtain A C E Aş and S C, -k=1 Sk. where Ce the total number of model checkpoints for the é-th segment and A₃ and S₁ are covariance matrices for the k-th checkpoint. Then, we perform eigendecomposition on these averaged covariance matrices to obtain the eigenvectors QA and Qs. Under the eigenbasis QA & Qs, we compute the corrected eigenvalues A₂ for each model checkpoint (Equation (37) and then average the eigenvalues to obtain A. In summary, the averaged (Gauss-Newton) Hessian for a particular segment is approximated as:
G R (QA + QQ) @ Qs)
(38)
SOURCE requires computing the covariance matrices and corrected eigenvalues for each model checkpoint. Moreover, calculating the TDA scores for all training data points requires computing the training gradients C' times, where C. is the total number of checkpoints. Hence, SOURCE is approximately C times more computationally expensive than influence functions evaluated at the final checkpoint. In Seceiooo 3.3] we introduced a more efficient variant, which averages the parameters within a segment instead. This variant only needs to compute the EK-FAC factors once for each segment and requires computing the EK-FAC factors and gradients L times. Hence, it is L times more computationally expensive than influence functions.
When the model is trained with SGD with heavy ball momentum 3 (SGDm), we scaled the learning rate used in SOURCE as 7 - B)- to account for the effective learning rate (terminal velocity). In cases where AdamW optimizers exponential for P!/"H,P'/2 Hessian approximation (which can be easily and efficiently obtained when computing the corrected cigenvalues in [Equation (37)) and use the diagonal Hessian approximation for computing the matrix exponential in Equation (23)and and Equation (27) Note that we still use the EK-FAC factors compute
used as in Appendix C computing the matrix challenging with EK-FAC. We additionally keep track of the diagonal
H, ge in Equation (27)
D.3 Applicability to Other Approximation Techniques
While we described one instantiation of SOURCE with the EK-FAC approximation, SOURCE can be integrated with other techniques used for approximating implicit-differentaation---
TDA methods, such TRAK (70) and DATAINF [55]. For example, in TRAK, we can use random projection 4 to efficiently compute the averaged Hessian and gradients in a lower-dimensional space. TRAK is advantageous over the EK-FAC approximation when there are many query data points, as it caches compressed training gradients in memory, avoiding recomputing them for each query.
E
Experimental Setup
This section describes the experimental setup used to obtain the results presented in [Section 5 This includes a description each task (Appendix E.T) and the methodology for computing the linear datamodeling score (LDS) (Appendix E.2). Implementation details of the subset removal
counterfactual evaluation and bascline techniques are provided in(Appendix E.3 Appendii E.4 respectively. All experiments were conducted using PYTORCH version 2.1.0 [7]. We used CPUS to conduct UCI regression experiments, A100 (80GB) GPUs to conduct GLUE and WikiText-2 experiments, and A6000 (48GB) GPUs for other experiments. A single GPU was used run SOURCE and baseline techniques. The internal cluster was used to run the experiments. Since our ground truth requires a lot of model retraining with different random choices (e.g., initialization and batch ordering), we considered tasks that train (or fine-tune) with less than 20 minutes using the abovementioned compute resources. For example, the generation of the LDS ground truth for a given task and data sampling ratio (0,1) takes at most 210 hours of computational resources.
20

---

E.1 Datasets and Models
We conducted systematic hyperparameter optimization for all tasks. This process involved conducting grid searches to find hyperparameter configurations that achieve the best average validation perfor- mance (accuracy for classification tasks and loss for others). The average validation performance was obtained by retraining the network 5 times using different random seeds. For models trained with SGD with a heavy ball momentum of 0.9 (SGDm), our search spaces for learning rate and weight decay were (3e-1, 1e-1, 3c-2, 1e-2, 3e-3, 1e-3, 3c-4, 1e-4, 3c-5, 1e-5] and {3e-2, 1e-2, 3e-3, le-3, 3c-4, le-4, 3c-5, 1e-5, 0.0), respectively. For models trained with Adam W [61], the search spaces were (le-2, 3e-3, le-3, 3e-4, le-4, 3e-5, 1e-5/ for learning rate and (3e-2, 1e-2, 3e-3, 1e-3, 3e-4, le-4, 3e-5, le-5, 0.0g for weight decay. In cases where the original experimental setup from which we adapted had a pre-specified learning rate and weight decay, these hyperparameters were incorporated into our search space.
UCI Datasets (Regression). For regression tasks, we used the Concrete [94] and Parkinson
datasets from the UCI Machine Learning Repository [45]. Both datasets were pre- pre-processed to have zero mean and unit variance for imput features and targets. We trained a layer multilayer perceptron (MLP), where each layer consisted of 128 hidden units and the RELU activation function. The models were optimized using SGDm for 20 epochs with batch size of 32 and constant learning rate schedule. A learning rate of 3e-2 and a weight decay of 1e-5 were used for the Concrete dataset. For the Parkinson dataset, the learning rate was set to 1e-2 with weight decay value of 3e-5. We saved 6 intermediate checkpoints throughout training. For the noisy Concrete (Concrete-N) dataset, we randomly modified 30% of the targets by sampling from a Normal distribution with zero mean and unit variance. We used the same hyperparameters but trained the models for 3 epochs.
MNIST & FashionMNIST (Image Classification). Following the experimental setup from Koh and Liang 49] and Bae et al. 3, we trained three-layer multilayer perceptron (MLP) on approxi- mately 10% of MNIST [57 and FasHionMNIST [91] datasets. Smaller versions of these datasets were used to compute the counterfactual ground truth more efficiently. The models were trained with SGDm for 20 epochs with a batch size of 64 and constant learning rate. The learning rate and weight decay were set for both datasets to 3e-2 and 1e-3, respectively. We saved 6 checkpoints during training and utilized them for TRACIN and SOURCE. For the noisy FNSH (FashionMNST-NN experiment, we randomly relabeled 30% of the training dataset. The network was only trained for 3 epochs with learning rate 1e-2 and weight decay 3e-5.
CIFAR-10 (Image Classification). For the CIFAR-IO dataset (53), we trained the ResNet-9 model following the standard data augmentation procedure from Zagoruyko and Komodakis [95]. This included extracting images from random 32 32 crop after applying zero-padding of 4 pixels, with 50% probability of horizontal flipping. The network was trained for 25 epochs using SGDm with batch size of 512 and a cyclic learning rate schedule, peaking at 0.5. The initial learning rate was set to 0.4 with weight decay of 1e-3, and 6 intermediate checkpoints were saved throughout training.
32
8
GLUE (Text Classification). We fine-tuned the BERT model 12] on SST-2, RTE, and QNLI datasets from the GLUE benchmark (85] with the training script from the Transf ormers library Following the experimental sctup from Park et al. (70), we capped the training dataset at maximum of 51200 examples to compute the LDS efficiently. However, we did not modify the original architecture (e.g., removing the last TANH layer) and trained the network with the Adam W optimizer. The weight decay was set to 1e-2 for all tasks, and the learning rates were set as follows: 3e-5 for SST-2, le-5 for QNLI, and 2c-5 for RTE. We saved 6 intermediate checkpoints for each training run.
89
WikiText-2 (Language Modeling). For the language modeling task, we fine-tuned the GPT-2 model 73 using the WikiText-2 dataset 655. We followed the training script from the Transformer
21

---

library but set the maximum sequence length to 512 10 intermediate checkpoints for data attribution. The learning rate, weight decay, and batch size were set to 3e-5, 1e-2, and 8, respectively. We set the measurement / as a loss for the language modeling task.
During fine-tuning with Adam we saved 6
RotatedMNIST & PACS (Image Classification). We used the RotatedMNIST dataset [20] and the PACS dataset [58), following the data pre-processing procedures from Gulrajani and Lopez-Paz 25 The training process was divided into two distinct stages for both tasks. During the initial stage of the training, we trained the network with the dataset D1. while the second stage used dataset Dg. For RotatedMNIST, the first dataset D1 was comprised of images rotated O, 15, 45, and 60 degrees, whereas the second dataset D2 contained images rotated at 30 degrees. We trained a three-layer MLP for 30 (20/10) epochs using SGDm and batch size of 128. The learning rate and weight decay were set to 1e-1 and 1e-5. For PACS, the first dataset 'D1 included images from the cartoon, photo, and sketch categories, and the second dataset D2 had art paintings. We fine-tuned ResNet-50 [2], initialized from the pre-trained parameters [2] using SGDm for 40 (30/10) epochs with a batch size of 128, a learning rate of le-4, and a weight decay of 3e-5.
E.2 Linear Datamodeling Score
We follow a methodology proposed by Park et al. 770 to compute the linear datamodeling score (LDS). Let À represent the set of hyperparameters used for training the model specified task, such as the choice of optimizer and the number of training epochs. Let a E (0, 1) denote the data sampling ratio. The process for obtaining the LDS involves several steps:
1. We generate M data subsets, denoted as (S each being a uniformly sampled subset of the original training dataset D. Each subset Sj C 'D contains [a.N] data points, where N denotes the total number of training data points.
2. For each data subset S,, the model is trained R times using different random seeds {E+ J (e-g., model initialization and batch ordering).
3. Given an attribution method r and a query example Zy. we measure the Spearman correla- tions [11 between the prediction and the estimated expected measurable quantity:
: (M)). f(zg 0* (S;; = (M} S,, { (39)
where g represents the group attribution prediction, expressed as: g. ( S, A; = Σ 2, D; A).
(40)
ZES
4. To obtain the final LDS, we average the correlations over a set of query data points (up 2000 in our experiments) and report the score with 95% bootstrap confidence intervals, which accounts for resampling of the data subset Sj (see Park et al. 770 for details).
For given data sampling ratio the networks must be retrained M R times in total to compute the LDS ground truth. In our experiments, we used 100 subsets (M = 100). The repeat R was set to 100 for UCI regression tasks, 10 for MNIST classification tasks, 20 for CIFAR-10 image classification task, 5 for GLUE text classification and WikiText language modeling task, and 20 for RotatedMNIST and PACS image classification tasks. We used the largest feasible R based on computational budget because we observed improvements in LDS for baseline techniques (cspecially TRAK, IF, and SOuRcE) with larger R.
E.3 Subset Removal Counterfactual Evaluation
For the subset removal counterfactual evaluation, we first train the model with the full dataset D under different random choices (over 5 random seeds) and select 100 test data points correctly classified on
language
html
22

---

all random choices. Then, for each test data point and attribution technique, we remove the top-k data points from the pre-defined interval ,y....,k (such that b <... < ky). as indicated as highly positively influential by the data attribution technique, retrain the network with this modified dataset, and examine if the original test data point gets misclassified on average under different random choices (over 3 random seeds). Finally, for each value of k in the pre-defined interval, we report the fraction of test data points that get misclassified after removing at most top-k training data points and retraining the network with the modified dataset.
For each TDA technique, this process requires retraining the model 100 x 3 times, where I is the pre-defined interval size. We set I = 6 for all experiments, leading to the retraining of the model 1800 times. To reduce the computational cost, we start from the smallest subset removal size k and if the test data point gets misclassifidd under the current subset, we do not consider it for the larger subset removal size (e.g., k2). Hence, this can be seen as the fraction of test data points that get misclassified by removing at most & training data points (evaluated at fixed interval). We note that Singla al. 779 instead use a bisection search to find the smallest subset size which a test data point can be misclassified, whereas Ilyas et al. [35] use more fine-grained intervals with more number of seeds (e.g., intervals and 20 seeds). While it is possible to use a larger number of seeds (20 seeds in Ilyas et al. (35)), because of computational limitations, we use 3 seeds to estimate the averaged misclassification. We used SOURCE and baseline techniques described in Appendix E.4 to identify positively influential training data points. We also included RANDOM baseline, where removed the training data points belonging to the same class as the target test example.
E.4 Baselines
This section describes the baseline techniques used in SSection 5] describe them in the context of a single-training-nun estimator, where the TDA techniques use the final parameters 0 obtained with hyperparameters À and some random choice & (the multiple-training- runs estimators simply average the TDA scores obtained from models trained with different random choices {).
Unless specified otherwise, we
Representation Similarity (REPSIM). Representation similarity technique 99 evaluates the im- portance of a training data point 2m D to specific query data point xq by comparing the latent representations of these data point pairs. This can be formulated as follows:
TREPSIM (Zy. Rm D; A) = similarity (de- (zg), 60- (zm)).
(41)
Here, similarity(v1, V2). where V1 and V2 are some vectors, is typically defined through the f2 metric, dot metric, or cosine metric [30]. In our experiments, the function dg. (z) was designed to map a data point to its last hidden activations (before the final output layer), using a forward pass through the final parameters 0% We used the cosine metric compute the attribution score but observed similar performance when using the l metric, aligning with observations in previous studies [35] 70 79
TRACIN. We used the TRACINCP estimator from Pruthi al. 772) defined as:
TTRACIN Fm, D; A) mhe Vef(zz, Ox) VoL(Em, 0k),
(42)
where C' represents the number of checkpoints, O represents the parameters at the k-th checkpoint, and mı is the learning rate applied at the corresponding checkpoint. The last checkpoint is typically set to the final model parameters 0". While there is an option to compress the gradients using a random projection as suggested by Pruthi et al. (72), our experiments used the full gradients obtain a stronger baseline. The checkpoint selection details are described in inpppeddix E.
Influence Functions (IF). As detailed in Section 2.2 training data attribution with influence functions is formulated as follows:
(Zg. 2m. D: A) = Vof(zz. 0*) "
" Voo( 0²).
(43)
where H denotes the Hessian of the cost at the final parameters 0². To make influence functions
scalable to large neural networks, we used the Figenvalue-correeted Kronecker-Factoree Approximate Curvature (EK-FAC) parameterization [8] to approximate the Hessian, proposed by Grosse et al.
23

---

24]. We refer readers to Grosse et al. 24 and Appendix D for details on the EK-FAC computation.
Relatedly, Schioppa et al. [76 use Arnoldi iterations 12, and Kwon al. [55] utilize the parameter- efficient fine-tuning (PEFT) 34 strategy to efficiently approximate influence functions. More
recently, Choe et al. [1] further utilized low-dimensional gradient projection compute influence
functions more efficiently.
While Grosse al. [24] only consider the computation of influence scores to the MLP layers of transformers [84, in our experiments, we extended this computation to include the attention layers as well. We excluded layer normalization, batch normalization, and embedding layers from the influence computation. Influence functions have an additional hyperparameter À > O, which is used to compute the damped inverse Hessian-vecte vector product (IHVP), denoted as (H + XI) v for some vector v. We used a small damping term for consistency with TRAK 770 and set it to 1e-8 to avoid numerical instability (note that TRAK sets the damping term to 0).
TRAK. In contrast to the traditional formulation of influence functions, TRAK 770 leverages random projections 43,, Generalized Gauss-Newton approximation, and ensembling for data attribution. Specifically, given a random projection matrix P ~ NO, 1) 1)M**, where K denotes the projection dimension, the final model parameters 0", and model output function f(z, 0), TRAK projects all training and query gradients into K -iddimnnioonal vectors. The feature map is defined as:
((2) == P Vo,
(44)
We further define + = [dis.... ΦN E RNx as stacked projected gradients for all training data points, where each @; corresponds to ((z)). Subsequently, TRAK's 's single model estimator is formulated as:
(FRA D; = (=,)) (
(45)
with Q being a N X N diagonal matrix for weightings. Here, TIRAK represents a vector of dimension N, containing attribution score for each training data point. TRAK estimators, each derived from models trained with distinct configurations and projection matrices. We refer readers Park al. 770] and Engstrom al. [3 for detailed derivations and discussions of TRAK.
ensemble of single model
We used the final checkpoints for TRAK in experimental setup involving single model. We computed TRAK using the last checkpoint of 10 differently trained models (each trained with 50% of the dataset) for experiments with multiple model setups. TRAK has hyperparameter that determines the dimension of the random projection K. We set the projection dimension to 20480 for ResNet-9 and RotatedMNIST, 8192 for ResNet-50 on the PACS dataset, 1024 for BERT trained on the RTE dataset and 512 for MLP trained on the Concrete dataset (due to the datasets' smaller size), and 4096 for all other tasks, All experiments were conducted using TRAK'S official implementation 13
Empirical Influence (EI). To compute the empirical influence (DOWNSAMPLING [77, we first create M data subsets {S, ) each being uniformly sampled subset of the original training dataset. Each subset S; contains [o.]] data points, where training data point z. E DD we define Mn, as the total number of data subsets containing 2m- The empirical influence scores are formulated as follows:
(0, 1) is the data sampling ratio. Given
M
THI TEI Z2q Zm, D;A) M Mm
Σ
1/zm & Sj)s(zn; (3(5;;A,
(46)
f=1
M
M Σ
1|zm E S,1S(2n, 0' 0"(;;;A
(47)
where [[:] an indicator function to determine if the training data point 2,, is contained the j-th (46))com) computes the averaged query measurement when data point data subset Sj. Intuitively, Eqquativel,_Equution (46)
Zm is not used in training, point is used in training. Following Zheng et al. [97]. we created 512 data subsets (M = 512) with a sampling ratio a 0.5, which requires retraining the model 512 times with 50% of training data points removed.
Equation (47) computes the averaged measurement when the data
24

---

LDS
Methods
Single Model
Multiple Models
REPSIM 9
0.03 0.02
0.04 + 0.02
TRACIN 72
0.20 + 0.02
0.21 + 0.03
TRAK 70
0.08 + 0.01
0.26 + 0.00
IF
49
24
0.30 + 0.01
0.45 + 0.01
DOWNSAMPLING
177
0.11 + 0.02
HYDRA 10
0.16 0.02
0.02
SOURCE with averaged parameters (ours)
0.42 + 0.01
0.48 - 0.02
SOURCE (ours)
0.46 - 0.01
0.53 0.01
Table LDS at a = 0.5 for SOURCE (L = 3) and baseline TDA techniques (including DOWNSAM-
PLING and HYDRA) the FashionMNIST dataset. We show the 95% bootstrap confidence intervals.
FAST SoURcE
SOURCE
FashionMNIST (MLP)
(ResNet-9)
(BERT)
ISstiooMMNSTT-N (MLP)
1D5
0.25
0.00
0.0
0.00
0.0
Figure 7: LDS at a = 0.5 for influence functions, FAST-SOURCE (see Appendix F.2, and SOURCE. The LDS is shown for single model (single-training-run setup.
HYDRA. We used the fast version of HYDRA 10], formulated as:
T-1
THYDRA Zm, D; A)
=
Σ
7 1[2m E B&] Vef(zz. 0") VoL(zm, 0k),
(48)
k=0
where 7 represents the total number gradient update steps, 0g denotes the parameters at the k-th iteration, and 77 is the corresponding learning rate. Here, B& denotes the batch of data points used the corresponding update, and 1[z,, E B&] is the indicator function for having selected 2m in the update. Note that HYDRA requires storing all parameter vectors used for training. We refer readers to Hammoudeh and Lowd 27 for derivations and detailed discussions of HYDRA.
Additional Results
In this section, we present additional experimental results, including a comparison with additional baseline TDA techniques (Appendix F.I. LDS evaluation of a computationally faster variant of SOURCE (Appendix F.2). an LDS evaluation at various sampling ratios and for more tasks (Appendix F.3), counterfactual evaluation on linear models (Appendix F.4), and visualizations of the top positively and negatively influential training data points for each TDA technique (Appendix F.5).
F. Additional Baseline Comparisons
We compare SOURCE with empirical influence (DOWNSAMPLING) [7] and the fast version of HYDRA [00 on the FasHionMNiST task. Results for these techniques on other tasks were omitted, since DOWNSAMPLING requires retraining the model over 500 times and HYDRA necessitates saving all intermediate checkpoints throughout training. The implementation details are provided in in[Appendix E.4 and the results are shown in Table I SoURCE achieves the highest LDS on both single and multiple model setups compared to existing baseline TDA techniques we considered.
F.2 SOURCE with Averaged Parameters
In Section 3.3 we introduced a more computationally efficient version of SOURCE, which averages parameters within segment instead of Hessians and gradients. Here, we present the LDS results at 0.5 for the faster version, termed FAST-SOURCE, for FashionMNIST. CIFAR- 10, RTE,
25

---

REPSIM
TRACIN
TRAK
IF
SOURCE (L = 1)
SOURCE (L 3)
Concrete (MLP)
CIFAR-10 (ResNet-9)
RTE (BERT)
LDS
0.1
0.25
88
0.0
0.0
0.00
0.3
0.5
0.7
0.9
0.5
0.7
0.9
N
0.3
0.5
0.7
0.9
N
Figure 8: LDS across range of data sampling ratios for SOURCE (L (1, and baseline TDA techniques. The LDS is measured for single model setup, and error bars represent 95% bootstrap confidence intervals.
Concrete (MLP)
Parkinsons (MLP)
MNIST (MLP)
FashionMNIST (MLP)
0.5
0.5
0.5
0.0
0.0
0.0
0.0
CIFAR-10 (ResNet-9)
RTE (BERT)
SST-2 (BERT)
QNLI (BERT)
0.5
LDS
0.2
0.0
0.0
0.0
WikiTexi-2 (GPT-2)
REPSIM
IF
8
Model
TRACIN
SOURCE
10 Models
0.25
Figure LDS at 0.5 for SOURCE (I. = 3) and baseline TDA techniques on models fully trained using a fixed dataset, where methods based on implicit differentiation and unrolling are expected to perform similarly. The error bars represent 95% bootstrap confidence intervals. (Results for TRAK on WikiText-2 are omitted due to the lack of publicly available implementations for language modeling tasks.)
and FashionMNIST-N tasks. The results shown in Figure 7
We observe that FAST-SOURCE
outperforms influence functions on these tasks, while it generally achieves lower LDS compared to SOURCE.
F.3 Additional LDS Results
Here, we present additional results, evaluating TDA techniques with LDS various sampling ratios and considering more tasks, where models are fully trained using a fixed dataset. We first consider computing the LDS across range of data sampling ratios a. (The procedures compute the LDS are described in Appendix E.2) The performance of SOURCE and other baseline attribution methods shown in (Fiigurr 8 SOURCE consistently achieves higher LDS than the baseline methods across diverse values. However, exception is noted at 1 1/N (e.g., removing a single training data point), where a significant drop correlations observed all TDA methods. This finding is consistent with previous studies that highlight the limitations of LOO estimates in reliably evaluating attribution techniques 444 14 67] (see Appendix B for detailed discussion). Additionally, our results suggest that while SOURCE with single segment can be effective, using multiple segments typically improves LDS performance. Lastly, we observe that the relative rankings of TDA techniques typically remain consistent across various values
Next, we present the LDS results outperforms baseline methods in single model setup, achieving higher correlations with the ground truth. When aggregating TDA scores from multiple models, we observe large improvement the
0.5 for additional tasks in Figure 9 SOURCE consistently
LDS. Our method obtains the highest LDS across all tasks, except for the CIFAR- 10 classification task using ResNet-9. However, we show that our method outperforms baseline methods on the CIFAR-10 10 task for subset removal counterfactual evaluation in Section 5.2) We note that the tasks in Figure
26

---

TRACIN
IF
SOURCE
Diabetes
Conerete
Concrete
1.0
1.0
LDSS
0.5
0.8
0.8
2
20
N
Epochs
Figure 10: (Left & Middle) LDS for various values of data sampling ratios on linear regression and logistic regression tasks trained for epochs. (Right) LDS at a = 0.9 for models trained with varying numbers of epochs. The error bars show 95% bootstrap confidence intervals.
Diabetes
Concrete
1.00
1.00
LDS
0.99
0.98
0.5
0.7
0.9
0.5
0.7
Figure 11: LDS on linear regression and logistic regression tasks for influence functions when TDA is performed on the optimal solution.
consider models sufficiently trained near convergence (with carefully chosen hyperparameters), where implici--differnntittonn- methods are expected to perform similarly to unrolling- -ased methods.
F.4 Counterfactual Evaluations on Linear Models
In this section, we demonstrate the effectiveness of SOURCE on linear models when the model has been trained until convergence. We trained linear regression on the Concrete dataset and logistic regression on the Diabetes dataset [80] for 3 epochs with a batch size of 32. We also constructed the LDS ground truth using SGD with the same hyperparameters. We applied TRACIN, IF, and SOURCE (with 7, 1) to the trained model and computed the LDS for various data sampling ratios The results are shown in Figure 10 (Left Middle). SOURCE achieves higher LDS on all data sampling ratios for both regression and classification tasks. We further show the LDS at a = 0.9 with varying numbers of epochs in Figure 10 (Right). (The LDS ground truth recomputed at each epoch.) We observe larger LDS gap between SOURCE and IF when the model was only trained for small number of epochs, and the gap reduces as we train the model for a larger number of iterations. These results show that our formulation for SOURCE better supports TDA when the network has not fully converged, even in the case of linear models.
For completeness, we show the LDS for influence functions when the TDA is performed on the optimal solution Figure regression, we used the L-BFGS (59)) computed the influence function estimates, and evaluated their accuracy with LDS (also obtained by computing the optimal solution without some data points). As shown in Figure 11 influence functions obtain high correlations with the ground truth across various values of data sampling ratio a. In contrast to neural network experiments Appenddix F.3 we observe an increase in the LDS as the data sampling ratio increases (predicting the effect of removing a smaller number of data points), as the group influence predictions introduce more approximation error 3. Notably, point), as the LDS is computed at the precise optimal solution (see Appendix B for the discussion). TRACIN and SOURCE not applicable in these contexts, the direct solution or with L-BFGS, instead with gradient descent.
For each model, we computed the optimal solution (for logistic
obtain high LDS when (N - 13/N (removing single data
computed the optimal solution with
F.5 Qualitative Results
We first present the top positively and negatively influential data points obtained by each TDA technique on multiple model settings. Note that for these multiple model settings, REPSIM, TRACIN, TRAK, IF, and SOURCE use ensemble of 10 models trained with different random choices. The
27

---

results for FashionMNIST, CIFAR-10, and RotatedMMIST are shown in Figure 12 Figure 13 and Figure 14, respectively. We also show the top positively and negatively influential data points on the CIFAR-TO dataset for single model setup in Figure 15 In Table 2 we present the top positively and negatively influential data points obtained by SOURCE on the RTE dataset.
G Broader Impact & Limitations of SOURCE
where traditional Our paper focuses improving training data attribution, especially in cases bbsed methods, such as influence functions, struggle. While work does not have direct societal impact, focuses on algorithmic improvements, there may be societal implications for improving TDA techniques. On the positive side, as shown in Grosse et al. (4), TDA techniques can be used to understand and debug the misbehavior of neural networks (e.g.. LLMs), which can promote trust in deploying machine learning systems. By identifying the training data points responsible for specific model behaviors, TDA can help improve model interpretability, fairness, and robustness. This, in turn, can lead to more reliable and equitable Al systems that benefit socicty. However, TDA techniques also allow for the analysis of the impact of training data points on trained models, which can be used for crafting data poisoning attacks [16 38 69]. Malicious actors could potentially use TDA to identify and manipulate influential data points, leading to the creation of biased or misleading models. This could have negative consequences, such as the spread of disinformation or unfair treatment of specific groups. To mitigate these risks, essential to develop responsible practices for using TDA techniques.
Broader Impact.
Limitations. Compared to the influence function employing the same EK-FAC parameterization [24] the practical implementation of the SOURCE requires the computation of EK-FAC factors and gradients for all checkpoints (when performing TDA on all segments). Note that, when TDA is performed on one specific segment f, the gradients only need to be computed for checkpoints within a segment (instead of all checkpoints). Denoting the total number of checkpoints as and the total number of segments as L, SOURCE on all segments exhibits an approximate computational cost of C' times higher. Our experiments used configurations with E {3, 6} and L {2, 3). We also introduced faster version of SOURCE in Appendix F.2] which directly averages the parameters instead of averaging the EK-FAC factors and gradients; the faster version is L times computationally expensive compared to the EK-FAC influence functions.
Compared implicit-diiffrrnntation based TDA techniques, SOURCE requires access to intermediate checkpoints throughout the training process and corresponding hyperparameters such as learning rate, number of iterations, and preconditioning matrix. In cases where the details of the training process are not available, implicit-diffen liittddffffeetitiinbaaseee TDA techniques, such as TRAK [70] and influence functions 49 24], may be preferable.
Moreover, SOURCE approximates the distributions of the Hessian and gradient as stationary within each segment of the training trajectory. In certain scenarios, this may not be reasonable approxima- tion. For instance, when pre-training large transformer models, the Hessian or gradients may undergo drastic changes throughout the training process. If the stationarity approximation too inaccurate, one can enhance the fidelity of SOURCE by dividing the training trajectory into a larger number of segments, albeit at the cost of increased computational requirements. While we used a fixed number of segments and checkpoints, partitioned equally at the early, middle, and late stages of training, we can extend SOURCE by automatically determining when to segment by examining the changes the Hessian or gradients, which we leave for future work. Lastly, SOURCE approximate the Hessians and gradients at different time steps as statistically independent to obtain a tractable approximation for the expected total derivative in /Sectio 3.2] As discussed, this independence approximation amounts to neglecting the autocorrelation of optimization iterates.
28

---

REPSIM
Query Image
Top Positively Influcatial Images
Top Negarively Influcatial Images
trouseer
Urouser
trouserr
frouser
Urouuee
shirt.
shirt
elbiet
shirt
pullover
pullover
pullover
pullover
pullover
bag
bag
bag
bag
TRACIN
Querry Image
Top Positively Influcatial Images
Top Negarively Influcntial Images
trousee
Urouser
trousee
trouser
Trouser
shirt
I-ehiet
dinees
dreese
pullover
pullover
pallooeer
pullover
pulloover
shirt
shirt
shirt
shirt
TRAK
Query Image
Top Positively Influcatial Images
Top Negarively Influcntial Images
tromsser
pullowwer
shirt
shirt
shirt
sandaa
pullower
shirt
pullover
pullover
pallover
shirt
pullover
shirt
pullover
t-shirt
shirt
IF
Query Image
Top Positively Influcatial Images
Top Negarively Influcntial Images
trouser
Trouser
sheaker
trortecr
Trousee
shirt
trouser
Erousee
shirt
pullover
pullover
pullover
pullover
pullover
shirt
shirt
t-shirt
t-shirt
SOURCE
Query Image
Top Positively Influeatial Images
Top Negarively Influential Images
trouser
trouser
trouser
trouser
trousse
shirt
t-shürt
dreess
pullover
pullover
pullover
pullover
pullover
pullover
shirt
t-shirt
shirt
t-shirt
Figure 12: Top positively and negatively influential training images identified by SOURCE and baseline TDA techniques on the FashionMNIST dataset.
29

---

REPSIM
Querry Image
Top Positively Infucntial Images
Top Neganively Images
truck
truck
dog
dog
dog
dog
ship
ship
ship
ship
ship
bird
doer
bird
TRACIN
Query Image
Top Positively Influential Imaggee
Top Negatively Influential Images
truck
ship
ship
ship
ship
ship
airplane
airplane
truck
airplane
TRAK
Query Imagee
Top Positively Influential Images
Top Negutively Influential Images
dog
ship
ship
ship
ship
ship
cat
ship
truck
Car
IF
Quirr Imaage
Top Proittively Infincntial Images
Top Noguzively
Images
truck
ship
ship
ship
ship
ship
truck
cAr
airplane
sirplane
SOURCE
Quinry Image
Top Prodttively
Images
Top Noguzively
Images
truck
ship
ship
ship
ship
ship
car
CAP
truck
car
Figure 13: Top positively and negatively influential training images identified by SOURCE and baseline TDA techniques on the CIFAR-10 dataset. Note that we labeled the "automobile" class as "car""
30

---

REPSIM
Query Image
Top Positively Influcatial Images
Top Negarively Influcatial Images
0
0
5
5
5
S
5
5
5
2
TRACIN
Query Image
Top Positively Influcatial Images
Top Negarively Infiucatial Images
O
0
0
9
5
5 5
S
5
5
3
3
6
6
5
3
TRAK
Query Image
Top Positively Influcatial Images
Top Negatively Infiucatial Images
1
C
6
0
0
6
5
6
6
6
Z
6 6
3
6
6
6
IF
Query Image
Top Positively Influeatial Images
Top Negatively Influential Images
0
0
6
5
S
V
6
2
6 6
6
6
5
SOURCE
Query Image
Top Positively Influcatial Images
Top Negatively Influential Images
5
0
0
5
5
5
5
2
3
3
8
5
3
Figure 14: Top positively and negatively influential training images identified by SOURCE and baseline TDA techniques on the RotatedMNIST dataset.
31

---

Query Data Point
Tup Positively Influential Data Point
Top Negatively Influential Data Point
Dana Reeve, the widow the actor Christopher Reeve. has died lung can- cer
Though fearful of forthcoming perfor- mance evaluation unravel the life of a man just found dead of heart attack. who was supposed to have died three years earlier in a boating accident. Zoe died boating accident. (not entailment)
Actor Christopher Roeve, best known for his role Superman, paralyzed and cannot breathe without the help of res- pirator after breaking his nock in a riding accident in Culpeper. Va., on Saturday, Christopher Reeve had accident. (en- tailment)
boss, Zoe must
44, according to the Christo- pher Reeve Foundation. / Christopher Reeve had an accident. (not entailment)
discovering that antibi- otics are losing their effectiveness against illness. Discuso-causing bacteria are mu- tating faster than we can come up with antibiotics fight the variations, / Bacteria is winning the war against an- tibiotics, (entailment)
The papers presented show that Euro- countries are experiencing rapidly aging populations that will cause sharp increases in the cost of retirement income the several decades. / National pension systems currently adopted in Fu-- rope are in difficulties. (entailment)
Humans won notable battles in the war against infection and antibiotics are still powerful weapons but nature has evolution on its side, and the war against bacterial discases by
Bacteria is winning the war against an- tibiotics. (not entailment)
high alert after an election campaign in which more than 1,000 people, including seven election cundidates, have been killed. Security forces were on high alert after cam- paign marred by violence. (entailment)
Security forces
Police sources stated that during the bomb attack involving the Shining Path, two people were injured. Two people wounded a bomb, (entailment)
Pakistan President Pervez Musharaa has ordered security forces to take firm action against rioters following the assassina- tion of opposition leader Benazir Bhutto, The violence has left least 44 people dead and dozens injured. Mr. Musharraf insisted measures were protect peo- ple. VOA's Ayaz Gul reports from Islam- ahad that a bitter dispute has also enupted over how the 54 year old politician died who was behind / Musharraf has ordered rioters to take firm action against security fonces. (not entallment)
assassination.
In 1979, the leaders signed the Fgypt- Israel peace treaty on the White House Both President Begin Sadat ceived the Nobel Peace Prize for their work. The two nations have enjoyed peaceful relations to day. / The Israel- Egypt Peace Agreement was signed 1979. (entailment)
Following the Israel- Peace Treaty of 1979, Israel agreed to withdraw from Sinai Peninsula, in exchange peace with its neighbor. For decades, the Sinai Peninsula was home to about 7,000 Israelis. / The Israel Egypt Peace Agreement was signed 1979. (entailment)
Canada and the United States signed an agreement on January 30, 1979, to amend treaty allow subsistence hunting of waterfowl. / Thhe Isrnel-Fgypt Peace Agreement was signed in 1979. (not en- tailment)
Top positively and negatively influential data points identified by SOURCE the RTE dataset. A data point in the RTE dataset consists of a pair of sentences (separated by forward slash "f"") and a label indicating whether the second sentence entails the first sentence (entailment) or not (not entailment).
Table 2:
32

---

REPSIM
Query Image
Top Positively Influcatial Images
Top Negarively Influcatial Images
truck
truck
truck
truck
truck
dog
deer
dleer
dleer
ship
slip
ship
ship
deer
deer
deer
deer
TRACIN
Querry Image
Top Positively Influcatial Images
Top Negarively Infiucntial Images
truck
truck
truck
truck
Brack
car
car
cir
ship
ship
airplane
airplane
airplane
airplane
TRAK
Querry Image
Top Positively Influcatial Images
Top Negarively Infiucatial Images
trock
ship
clog
dog
Frog
dog
fron
binci
clog
deer
bird
deer
bizd
horse
IF
Querr Image
Top Positively Influeatial Images
Top Negarively Influcatial Images
track
truck
truck
trunkk
track
car
CAP
CAT
ship
ship
ship
airplane
SOURCE
Query Image
Top Positively Influeatial Images
Top Negarively Influential Images
truck
truck
truck
truck
truck
car
car
ship
ship
ship
airplane
Figure 15: Top positively and negatively influential training images identified by SOURCE and baseline TDA techniques (single model setting) on the CIFAR- 10 dataset.
33

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims made in the abstract and introduction reflect the paper's contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of SOURCE in Appendix G These limitations are also mentioned throughout the paper, such as in [Section 3.2 (core approximations) and in Section 3.3 (additional computational overheads).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
34

---

Answer: [Yes]
Justification: We provide a full set of approximations Section 3.2 and describe the equations in depth throughout the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed information to reproduce the main experimental results of the paper in Anppeediix E Implementation details provided in in[ppendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
35

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: We do not release the code. However, we described the data preparation procedures and the URLs baseline techniques we used comparison in Appendix F We used publicly available data for our experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ptrips.cc/publcc/guidde
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed information about the training and test details Ap pendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For the LDS evaluation, we show the error bars representing 95% bootstrap confidence intervals, which account for resamplings of data subsets. The subset removal evaluation requires at most 1800 model retrainings for each pair of task and TDA technique. The reported numbers are averages from 3 seeds. Details are provided in Appendixx E.2 and Appendix E.3 respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
36

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information on the computer resources (CPU or GPU), which were used to conduct experiments Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurlPS Code of Ethics atttps://neur/ cttpp///bcc//hhiccccu..eleelllll
didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
Our research conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide discussions on the broader impact of our work in Appendix G
37

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cite the original owners of the code, data, and models used in the paper (they are all publicly available).
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
38

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswipphaeeoooccaaappp thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve any crowdsourcing, and we do not perform research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve any crowdsoureing, and we do not perform research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
39

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
40


Reviews:
[{"review_id": "rqHLyAT32Q", "reviewer": "Reviewer_9an4", "summary": "This paper introduces SOURCE, a new data attribution method designed primarily for deep neural networks; more generally, SOURCE is suited for any model class optimized with gradiend-based methods. The authors motivate SOURCE as an approximation to gradient unrolling, i.e., differentiating through the entire training process. They introduce a number of assumptions and simplifications that allow them to arrive at a formulation resembling existing influence-function-based data attribution methods.\n\nThe main assumption the authors employ is that the model Hessian (of the loss w.r.t. the model parameters) remains constant throughout segments of the training process (they set the number of segments $L$ as a hyperparameter). Additionally, the authors use the EK-FAC heuristic approximation for the Hessian [1] to reduce the computational cost of SOURCE. The same approximation has been employed in a previous data attribution method [2], which the authors use as a baseline.\n\nIn the experimental section of the paper, the authors evaluate SOURCE, alongside with an extensive list of baselines, on numerous (albeit rather small) datasets. SOURCE outperforms existing methods on the vast majority of benchmarks on standard data attribution evaluation metrics.\n\n[1] T. George, C. Laurent, X. Bouthillier, N. Ballas, and P. Vincent. Fast approximate natural gradient descent in a kronecker factored eigenbasis. Advances in Neural Information Processing Systems, 31, 2018.\n\n[2] R. Grosse, J. Bae, C. Anil, N. Elhage, A. Tamkin, A. Tajdini, B. Steiner, D. Li, E. Durmus, E. Perez, E. Hubinger, K. Lukošiu¯te˙, K. Nguyen, N. Joseph, S. McCandlish, J. Kaplan, and S. R. Bowman. Studying large language model generalization with influence functions, 2023.", "questions": "- can the authors provide an estimate of the computational cost of each method (e.g. wall-clock time) used in Figure 4? My understanding is that SOURCE uses up to 6x the amount of compute as the reported baselines (line 283: \"TRACIN and SOURCE use at most 6 intermediate checkpoints saved throughout training.\"). More generally, can the authors provide an equivalent to Figure 4 where compute, rather than the number of checkpoints, is equalized?\n- how does the performance (e.g., as measured by LDS) of SOURCE scale as 1) the number of segments $L$ increases, and 2) the number of fully-retrained checkpoints increase? In particular, I am curious where LDS asymptotes, and whether the asymptote value for SOURCE is higher than the one for [1, 2]; if that is indeed the case, this would, in my opinion, significantly strengthen the argument that SOURCE has a smaller \"modeling\" error compared to previous approaches.\n- my understanding is that the authors of [1] also propose using intermediate checkpoints along the training trajectory (without motivating this choice as an approximation to gradient unrolling); could the authors comment on the differences between the two estimators (excluding, obviously, the choice of Hessian approximation---EK-FAC and random projections, respectively)? Additionally, can the authors compare the performance of TRAK when intermediate checkpoints are used?\n\n[1] S. M. Park, K. Georgiev, A. Ilyas, G. Leclerc, and A. Madry. TRAK: Attributing model behavior at scale. In International Conference on Machine Learning, pages 27074–27113. PMLR, 2023.\n\n[2] R. Grosse, J. Bae, C. Anil, N. Elhage, A. Tamkin, A. Tajdini, B. Steiner, D. Li, E. Durmus, E. Perez, E. Hubinger, K. Lukošiu¯te˙, K. Nguyen, N. Joseph, S. McCandlish, J. Kaplan, and S. R. Bowman. Studying large language model generalization with influence functions, 2023.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "on the selected datasets (Concrete, FashionMNIST, CIFAR-10, RTE, PACS), SOURCE performs consistently well, largely outperforming existing methods\n- the paper is clearly written: the authors contextualize their method well within the existing literature and succinctly describe the novel ideas resulting in their method SOURCE\n- the direction of adapting data attribution methods towards realistic (for deep learning) scenarios like lack of convergence and multi-stage training procedures is very timely and important; the paper does a great job at balancing between addressing these challenges and maintaining computational efficiency\n- the resulting method, SOURCE, is easy to understand and is rather \"obvious\" (in hindsight, of course)", "weaknesses": "- compared to [1], SOURCE requires significantly larger amount of compute (as the number $L$ of training trajectory segment increases). At the same time, I would label the increase in performance from [1] to SOURCE as only \"moderate\". Given, in addition to the previous point, the conceptual similarity between the two methods, the technical contribution of this paper seems to be on the smaller side. Yet, I still appreciate the connection with unrolling that the authors present.\n- if possible, it would be good to see (at least a qualitative) evaluation of SOURCE on a larger scale setting, especially since the mechanically very similar method [1] is employed on larger-scale language models; this said, I understand that the rebuttal period is short, and would not base my score change on this result.\n\n[1] R. Grosse, J. Bae, C. Anil, N. Elhage, A. Tamkin, A. Tajdini, B. Steiner, D. Li, E. Durmus, E. Perez, E. Hubinger, K. Lukošiu¯te˙, K. Nguyen, N. Joseph, S. McCandlish, J. Kaplan, and S. R. Bowman. Studying large language model generalization with influence functions, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comprehensive and insightful evaluation of our work. We particularly appreciate the recognition of Source’s consistent performance across datasets, the clarity of our paper, and our efforts to balance addressing challenges in realistic scenarios while maintaining computational efficiency.\n\n> **The increase in performance as only moderate.**\n\nThe moderate improvements in the first set of experiments, where models are trained to near convergence with a fixed dataset, are expected because of the close connections with influence functions, as described in line 209. The primary advantage of our approach becomes evident when the assumptions of implicit-differentiation-based methods fall short. We demonstrate this in two scenarios: models trained for only a few iterations and multi-stage training processes. In these scenarios, Source shows large improvements over influence functions, both in terms of LDS and subset removal evaluation.\n\n> **If possible, it would be good to see (at least a qualitative) evaluation of SOURCE on a larger scale setting …**\n\nWe appreciate the reviewer's suggestion and agree that such an evaluation would provide useful insights. However, replicating a setup similar to [1] requires significant computational resources beyond most academic labs' reach. For instance, an experiment on a 13B parameter model, searching over 1M sequences, would incur costs in the range of tens of thousands of dollars. However, we have recently verified our EKFAC implementation scales to Llama-3 8B models. We will release our code to support TDA on larger-scale settings.\n\n> **Computational cost of each method (e.g., wall-clock time)**\n\nWe have recently improved the efficiency of our EKFAC implementation and subsequently re-ran a subset of the experiments. While we have implemented additional optimization tricks such as query batching [2] and automatic mixed precision — which can significantly improve efficiency with only a slight decrease in LDS — we will report the wall-clock time without these improvements to maintain consistency with the presented results. It is important to note that (1) the wall-clock time includes model training time, and (2) although we used 2,000 query data points in our experiments, consistent with [1], Trak becomes significantly more computationally efficient as the number of query data points increases, as it does not require multiple iterations through the dataset. While we demonstrated one instantiation of Source with EKFAC, we emphasize that Source can also be implemented with Trak, as detailed in Appendix E.3.\n\nFashionMNIST\n\n| Method | TracIn (10 Models) | Trak (1 Model) | Trak (10 Models) | IF (1 Model) | IF (10 Models) | Source (1 Model) | Source (10 Models) |\n|---|---|---|---|---|---|---|---|\n| Time (s) | 361.22 | 25.40 | 138.66 | 26.73 | 267.91 | 50.55 | 495.39 |\n| LDS | 0.21 | 0.08 | 0.26 | 0.30 | 0.45 | 0.46 | 0.53 |\n| Frac. Misclassified Test Examples (300) | 0.23 | 0.10 | 0.36 | 0.30 | 0.44 | 0.54 | 0.65 |\n\nFashionMNIST-C\n\n| Method | TracIn (10 Models) | Trak (1 Model) | Trak (10 Models) | IF (1 Model) | IF (10 Models) | Source (1 Model) | Source (10 Models) |\n|---|---|---|---|---|---|---|---|\n| Time (s) | 115.26 | 7.58 | 32.72 | 8.88 | 89.68 | 18.29 | 182.93 |\n| LDS | 0.48 | 0.07 | 0.12 | 0.34 | 0.40 | 0.63 | 0.64 |\n| Frac. Misclassified Test Examples (300) | 0.56 | - | 0.18 | - | 0.41 | 0.90 | 0.91 |\n\nCIFAR-10\n\n| Method | TracIn (10 Models) | Trak (1 Model) | Trak (10 Models) | IF (1 Model) | IF (10 Models) | Source (1 Model) | Source (10 Models) |\n|---|---|---|---|---|---|---|---|\n| Time (s) | 3843.74 | 247.41 | 1843.11 | 232.15 | 2323.6 | 886.15 | 8896.95 |\n| LDS | 0.06 | 0.05 | 0.24 | 0.13 | 0.20 | 0.18 | 0.22 |\n| Frac. Misclassified Test Examples (1400) | 0.28 | 0.11 | 0.41 | 0.26 | 0.41 | 0.52 | 0.61 |\n\n> **LDS asymptotes**\n\nWe direct reviewers to Figure 4 for insights on performance with different numbers of segments. In our experiments with the FashionMNIST pipeline, we did not observe an improvement in the LDS as we increased the number of segments to $L=6$. This may be because the segmentation approximation is already sufficiently accurate for these simpler experiments. We have conducted additional experiments to address this aspect of the reviewers' question about scaling performance with the number of fully re-trained checkpoints. We have trained up to 200 models and ensembled the models accordingly for FashionMNIST and FashionMNIST-C datasets. The uploaded PDF shows the results (we also compare it against the wall-clock time). On the FashionMNIST dataset, the gap between EKFAC and Source reduces as the number of checkpoints increases, but we still obtain a higher LDS at asymptote. On FashionMNIST-C (not converged), Source still obtains a significantly high LDS even at asymptote. We are conducting experiments of Trak with at most 300 models and we will include in the next revision of the manuscript.\n\n> **Re: Intermediate checkpoints**\n\nOur derivations recommend averaging the Hessian and gradient within a segment. This contrasts with the direct ensemble of multiple checkpoints explored in [1]. A core contribution of our work is the treatment for models trained in multiple stages or those that have not fully converged. Regarding the differences, Trak, as used in [1], benefits more from ensembling due to the randomness in the projection. We have observed that Trak can achieve higher performance even when ensembling a single checkpoint, while we generally do not see this improvement with EKFAC. To illustrate the performance differences, we have conducted additional experiments on Trak with intermediate checkpoints ensembling for FashionMNIST and FashionMNIST-C datasets (please see our response to Reviewer 3Y11; omitted due to characters limit). We will include additional datasets with intermediate checkpoints ensembled Trak in the next revision of the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hAVARdMiDH", "reviewer": "Reviewer_hk4U", "summary": "The article discusses the limitations of existing training data attribution (TDA) methods, which aim to estimate how a model's behavior would change if specific data points were removed from the training set. The authors propose a new method called Source, which combines the benefits of implicit-differentiation-based and unrolling-based approaches. Source is computationally efficient and suitable for cases where implicit-differentiation-based methods struggle, such as non-converged models and multi-stage training pipelines. Empirical results show that Source outperforms existing TDA techniques, particularly in counterfactual prediction, where implicit-differentiation-based approaches fall short.", "questions": "Regarding this article, as a practitioner in the field, I would like the authors to answer a series of questions.\n\n1. We all hope that Data-attribution/Influence Analysis can find suitable application areas, such as the work done by Gross et al. in explaining the output of LLM. However, currently, we have not seen Data-attribution/Influence Analysis truly solve a series of real problems, especially for large models such as Stable-Diffusion/LLM/LVM.\n\n2. If the time and computational cost required for attribution is much higher than that for training (as it should compute the gradient one-sample-by-one-sample, and even for many checkpoints), is there any practical application significance worth for such high complexities?", "limitations": "1. Error Bound: This article lacks an analysis of the error bound for the estimator.\n\n2. Regarding the contribution: The claim \"it allows the attribution of data points at different stages of training\" is not a unique contribution of this article, as some previous works [1,2,3] have achieved this without assuming that a checkpoint with testing is necessarily at gradient=0. Therefore, I do not believe that this claim can be considered a significant contribution to this article.\n\n3. Experiments: This article lacks validation on significant datasets (even without experiments on relatively old datasets like ImageNet) and instead tests on very small toy datasets.\n\n[1]. S Hara, et al. Data Cleansing for Models Trained with SGD. NeurIPS. \n\n[2]. G Pruthi, et al.  Estimating Training Data Influence by Tracing Gradient Descent. NeurIPS. \n\n[3]. H Tan, et al. Data pruning via moving-one-sample-out. NeurIPS.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The writing in this article is very clear and excellent, making it easy to read and follow.\n\n2. The design of this article avoids the assumption made by Koh et al.'s estimator that the loss is a graph function.", "weaknesses": "See the question part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough evaluation and insightful comments. Our research addresses limitations of influence functions and proposes a practical and novel algorithm to overcome these challenges. Grosse et al. [8], which the reviewer cited as an example of a TDA application, highlight a limitation of their approach:\n\n*\"A second limitation is that we focus on pretrained models. Practical usefulness and safety of conversational AI assistants depend crucially on fine-tuning …. Extending influence functions … to the combination of pretraining and fine-tuning is an important avenue to explore.\"*\n\nOur work takes a step towards addressing this challenge. While we appreciate the reviewer's critique, we believe some concerns may not fully align with our research's core contributions. The following sections provide detailed responses to address these concerns and clarify our work’s significance. We welcome any further discussion during the reviewer-author period.\n\n> **Have not seen TDA truly solve real problems…**\n\nTraining examples undeniably play a significant role in shaping model behavior. [1] and [2] demonstrated that removing less than 1% of the CIFAR-10 and ImageNet training dataset (identified by TDA) can lead to misclassifying a substantial fraction of test images. Several works show the importance of training data for LLMs [3, 4] and diffusion models [5].\n\nTDA techniques have already shown promise in several areas:\n1. **Data Curation:** TDA methods have proven helpful in increasing compute efficiency [6] and improving subgroup robustness [7]. Notably, [6] demonstrated that data curation with TDA techniques provides a 2x compute multiplier on standard LM problems compared to baseline techniques.\n2. **Understanding Model:** [8] employed influence functions to study LLM generalization patterns. Their analysis uncovered surprising sensitivities to word ordering. Berglund et al. [9] later empirically verified this phenomenon, which they termed the \"Reversal Curse.\"\n3. **Economic Models:** Building a royalty model to compensate data providers properly is an important economic and societal question. Using TDA techniques, [10] and [11] developed initial prototypes for such royalty models.\n\nTDA remains an active and valuable area of research, as evidenced by these applications and empirical observations on the importance of training data. While TDA may not yet be fully integrated into practical pipelines (at least publicly), *we believe this does not diminish the importance of researching potential failure modes in existing TDA methods and developing more accurate techniques, such as our work*. Our research contributes to the ongoing effort to make TDA methods more reliable and applicable to real-world scenarios. \n\n> **Is TDA worth it?**\n\nNaive gradient-based TDA methods do require gradient computations for all data points, which is as expensive as one-epoch training. We would like to clarify several points. First, in our experiments, attributing a batch of query data points is much faster than re-training a single model, as our models were trained over multiple epochs. Secondly, the one-by-one gradient computation described by the reviewer can be significantly optimized. Modern libraries such as Functorch or Opacus allow for efficient per-sample gradient computation. Lastly, some approaches, like [10], achieved computation times significantly faster than one-epoch training. Source is designed with flexibility in mind and can readily incorporate these efficiency improvements (Appendix E.3). \n\nOn a separate note, we have recently verified that our implementation runs on the Llama-3-8B model (OpenWebText) using academic resources. We will release this implementation. \n\n> **Lacks error bound for the estimator.**\n\nInvestigating the error bound under weak/practical assumptions presents an interesting direction for future research. Note that state-of-the-art techniques, such as Trak [12], EKFAC [8], and TracIn do not provide error bounds. We believe that the empirical results presented in our work provide convincing evidence of Source's advantages. Our evaluation uses state-of-the-art metrics [12] and covers multiple data modalities, demonstrating the efficacy of our approach across diverse scenarios.\n\n> **The claim “support multi-stage” is not a unique contribution ….**\n\nOur work does not claim to be the first to perform unrolled differentiation, as explained in the preceding paragraph. Rather, we present this capability as a key advantage of Source, particularly in comparison to influence functions. As other reviewers have noted, our contribution lies in the novel techniques we have developed to formulate and solve this problem. However, our work is the first to construct a set of experiments to verify the effectiveness of TDA techniques in the multi-stage training setup. Here, Source demonstrated large improvements: (1) over 5x improvement in LDS and (2) over 15x improvement in terms of subset removal evaluation, when compared to TracIn.\n\n> **Lacks large datasets**\n\nThe primary challenge lies in evaluation. For context, our CIFAR-10 & ResNet-9 pipeline required training the network 2,000 times for LDS computation and up to 1,800 times per TDA technique in the subset removal evaluation. Developing more efficient yet reliable evaluation metrics remains an open problem. Despite these constraints, we conducted comprehensive experiments across a diverse range of tasks, datasets, and architectures (up to 110M parameters): (1) regression (Concrete, Parkinsons), (2) image classification (MNIST, FashionMNIST, CIFAR-10), (3) text classification (QNLI, SST2, RTE), language modeling (WikiText-2), and continual learning (RotatedMNIST and PACS). We also examined linear and logistic regression problems in Appendix G.4. Our quantitative experimental scope surpasses many previous and recent publications from academic labs [10, 11, 13] in terms of variety in data types, tasks, and model architectures.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xiX7lOoAlh", "reviewer": "Reviewer_3Y11", "summary": "This paper propose a new TDA method (SOURCE) that combines implicit-differentiation-based methods and unrolling-based approaches. The new method is an extension to the SGD-influence and inherits the advantage to support multi-stage training pipelines while reduce the computation complexity to store and calculate the hessian matrix for each optimization step. The paper also carried out several experiments to show that SOURCE outperforms existing TDA methods under several training settings.", "questions": "1. **I am willing to make the rating higher** if the first weakness is resolved (intuitively or mathematically will be enough) since I understand add new experiment might be time-consuming.", "limitations": "I believe there is no negative societal impact of this work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well presented.\n\n\t○ The math derivation is self-contained, section 3.1 is a restate of SGD-influence and the extension to all iterations, 3.2 for segmentation and approximation, which is easy to follow.\n\n\t○ Some figures (especially figure 3) help me to understand the algorithm in a more intuitive way.\n\n- The contextual information of this paper in the whole TDA community is stated clearly.\n\n\t○ It's stated clearly about the relationship between IF/SGD-Influence/SOURCE.\n\n- The experiments for different training settings are convincing.\n\n\t○ This including 4 normal settings, 2 non-converging settings, 2 multi-stage training settings.\n\n- The segmentation trick is quite intuitive and reasonable.", "weaknesses": "- First of all, I am not quite convinced and identify difference between the segmentation proposed in this paper and the one in TracIN (and also using different checkpoints in one training process for IF/TRAK, as stated in the paper as implicit-differentiation-based methods). \n\n\t○ Furthermore on this point, it will be good if the experiment for IF/TRAK could also include a version which ensembles (natively for IF and term-wisely for TRAK) on multiple checkpoints for each independently trained models.\n\n\t○ TRAK also made some experiment to show that it can use non-converge checkpoints to get a comparable LDS, could be mentioned in the paper as well.\n\n- Not quite sure how much performance (accuracy) degradation is involved by the segmentation since there is no comparation directly with sgd-influence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly thank the reviewer for their thorough evaluation and insightful comments. We are pleased that our paper's presentation and experimental results were well-received. We will address your concerns and questions below.\n\n> **I am not quite convinced and identify the difference between the segmentation proposed in this paper and the one in TracIn.**\n\nThe key distinction lies in the quantities that TracIn and Source (or unrolled differentiation) approximate. As detailed in Section 4, TracIn estimates the importance of a training data point by aggregating the total change in the query's measurable quantity with gradient updates from this data point throughout training:\n\n\\begin{align}\n\\tau_{\\text{TracIn}} (z_q, z_m, \\mathcal{D}) = \\sum_{i \\in \\mathcal{C}} \\eta_i \\nabla f(\\theta_i, z_q)^\\top \\nabla \\mathcal{L}(\\theta_i, z_m),\n\\end{align}\n\nwhere $\\mathcal{C}$ denotes the set of steps the data point $z_m$ appeared during training, and $\\eta_i$ and $\\theta_i$ denote the corresponding learning rate and parameters, respectively. Even with a zero Hessian assumption (as done in Hydra, which we compare against in Table 1), the unrolling formulation gives:\n\n\\begin{align}\n\\tau_{\\text{Unrolling}} (z_q, z_m, \\mathcal{D}) = \\sum_{i \\in \\mathcal{C}} \\eta_i \\nabla f(\\theta^s, z_q)^\\top \\nabla \\mathcal{L}(\\theta_i, z_m),\n\\end{align}\n\nwhere $\\theta^s$ is the final model parameter, and the measurement gradient is always computed on this parameter. In contrast to TracIn, observe that unrolling aims to approximate the change in the final measurable quantity due to the perturbation of the data point's weight. Schioppa et al. [1] noted that using more intermediate checkpoints for TracIn can even hurt the performance in counterfactual estimation. Our work is motivated from the unrolling perspective, and hence, approximates a different quantity from TracIn. If TracInCP's analog of Hydra is considered (which has not been considered before, to the best of our knowledge), it can be viewed as applying our stationary approximation only to the gradient. However, using Hessian is crucial for achieving good LDS performance and better TDA performance in general, as highlighted in our work and recent studies [2, 3].\n\n> **Using different checkpoints in one training process for IF/TRAK, as stated in the paper as implicit-differentiation-based methods.**\n\nTo the best of our knowledge, the ensembling over checkpoints during the training process was primarily an empirical observation [2] rather than being theoretically motivated. From the unrolled differentiation perspective, our derivations recommend averaging the Hessian and gradient within a segment. This approach contrasts with the direct ensemble of multiple checkpoints. More importantly, the previous ensembling scheme does not provide a natural treatment for models trained in multiple stages or those that have not converged. Addressing these limitations of influence functions is the core contribution of our work. We acknowledge some additional caveats. Trak generally benefits from ensembling (even using the same checkpoint) due to the projection's randomness. Moreover, using slightly non-converged parameters can also be helpful in cases where the gradients are too small at the final checkpoint. We appreciate the reviewer's observation and will mention this in our next manuscript revision.\n\nIn response to this feedback, we conducted additional experiments for different checkpoints in one training process for IF and TRAK. We considered two scenarios: (1) FashionMNIST and (2) FashionMNIST-C (not converged model). Similar to the findings from Park et al. [2], we observe an increase in the LDS when using multiple checkpoints from a single training run. However, Source still obtains higher LDS in general, and the discrepancy is larger, especially in settings where the model has not fully converged. Note that the relatively smaller improvements in FashionMNIST, where the models are trained with a fixed dataset for a sufficient number of iterations, are expected because of the close connections with influence functions, as described in line 209.\n\n**FashionMNIST:**\n\n| # Model / TDA Method | IF | IF Ensemble | Trak | Trak Ensemble | Source |\n|---|---|---|---|---|---|\n| 1 | 0.29 | 0.37 | 0.08 | 0.14 | **0.46** |\n| 10 | 0.45 | 0.48 | 0.26 | 0.35 | **0.52** |\n\n**FashionMNIST-C:**\n\n| # Model / TDA Method | IF | IF Ensemble | Trak | Trak Ensemble | Source |\n|---|---|---|---|---|---|\n| 1 | 0.34 | 0.36 | 0.06 | 0.08 | **0.63** |\n| 10 | 0.40 | 0.41 | 0.12 | 0.17 | **0.64** |\n\n> **Not quite sure how much performance (accuracy) degradation is involved by the segmentation since there is no comparation directly with sgd-influence.**\n\nSGD-Influence poses significant practical challenges: (1) it necessitates storing all intermediate checkpoints throughout the optimization process, (2) the method requires a series of Hessian-vector products (HVPs) and many Monte Carlo samples from multiple training runs, as described in Section 3.1. Unlike other gradient-based approaches, these computations cannot be parallelized efficiently; this process must be repeated for each training data point. (3) It is worth noting that the largest model considered in SGD-Influence was smaller than the MLP (MNIST) models used in our study. These factors make SGD-Influence computationally prohibitive for models and datasets we considered, which is where our method, SOURCE, shows its strengths.\n\n[1] Schioppa, A., …, Zablotskaia, P. (2024). Theoretical and practical perspectives on what influence functions do. \n\n[2] Park, S. …, Madry, A. (2023). Trak: Attributing model behavior at scale.\n\n[3] Deng, J., …, Ma, J. (2024). Efficient Ensembles Improve Training Data Attribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uzaHFKv7tm", "reviewer": "Reviewer_5ybi", "summary": "The paper proposes a training data attribution (TDA) method based on implicit differentiation and unrolling. The goal is to estimate the effect of removing (or changing the weight) of a training example on the final (not necessarily optimal) parameters, accounting for training details influencing the trajectory of the parameters during training.\n\nThe method approximates the change in the final parameters $\\boldsymbol\\theta_T$ ($T$ is the number of optimization iterations) based on approximating the effect of re-weighting a training example with weight $1+\\epsilon$. This requires computation of $\\partial \\boldsymbol\\theta_T/\\partial\\epsilon$, which is a product of Jacobians $\\partial\\boldsymbol\\theta_{k}/\\partial\\boldsymbol\\theta_{k-1}$ from all training steps $k$., each computed from a Hessian of the loss. To reduce computational requirements, the authors apply different approximations while accounting for all parts of the training trajectory. The resulting algorithm, SOURCE, requires 2-18 more computation (Hessian estimations) than influence functions.\n\nExperiments show that SOURCE outperforms other TDA methods in counterfactual evaluations on linear datamodeling score (LDS) (Park et al., 2023) and on standard subset removal counterfactual evaluation.", "questions": "**Questions:**\n1. Could you point more precisely to the Implicit Function Theorem and its proof?\n2. Eq. (11): Could something more be said about the properties of the approximation?\n\n**Suggestions:**\n- L105: \"optimal solution\" -> \"optimal solution when $\\epsilon=0$\".\n- Fig. 2: \"Each contour\" -> \"Each set of concentric contours\"?\n- FastSOURCE could be explained more clearly, with references to equations.\n- An overview of computation time for different TDA and evaluation methods would be interesting.", "limitations": "The limitations seem to be addressed adequately.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is very well written and structured.\n- The paper is technically sound (as far as I have checked) and presents a valuable analysis and novel insights.\n- The proposed training data attribution method, SOURCE, outperforms other TDA methods on counterfactual evaluations on different tasks across different data modalities.\n- The SOURCE code will be published (if I interpret L692 correctly).", "weaknesses": "There are a few easy-to-fix errors:\n- L6: missing comma before but.\n- $\\mathcal B_k$ is not mentioned before L124.\n- L206-207: \"approaches to\".\n- Appendix G is not referenced in the main text.\n- L1046: \"the SOURCE\".\n- L1051: $C = \\{3, 6\\}$ -> $C \\in \\{3, 6\\}$ (likewise for $L$).\n- (The curly brackets in Fig. 3 have a slight distortion.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's positive assessment of our paper, acknowledging that it is well-written, technically sound, and presents novel insights. We are grateful for your attention to detail in identifying errors and typos, which we will address in the revised manuscript.\n\n> **Could you point more precisely to the Implicit Function Theorem and its proof?**\n\nFor a detailed derivation of influence functions using the Implicit Function Theorem, we direct the reviewer to [1] (Appendix B.1). With the assumptions outlined in our paper (the parameters are optimized to convergence and the objective has a unique solution), we have:\n\n\\begin{align}\n\\mathbf{0} = \\nabla \\mathcal{J}(\\theta^\\star) + \\frac{\\epsilon_0}{N} \\nabla \\mathcal{L}(\\theta^\\star, z_m),\n\\end{align}\n\nwhere $\\epsilon_0 = 0$. By the IFT, there exists a unique continuously differentiable function $r$ defined in the neighborhood of $\\epsilon_0$ such that:\n\\begin{align}\n\\mathbf{0} = \\nabla \\mathcal{J}(r(\\epsilon)) + \\frac{\\epsilon}{N} \\nabla \\mathcal{L}(r(\\epsilon), z_m).\n\\end{align}\n\nTaking the derivative with respect to $\\epsilon$ at 0, we obtain:\n\\begin{align}\n\\mathbf{0} = \\nabla^2 \\mathcal{J}(\\theta^\\star) \\frac{\\mathrm{d} r}{\\mathrm{d} \\epsilon} \\Big\\vert_{\\epsilon=0} + \\frac{1}{N} \\nabla \\mathcal{L}(\\theta^\\star, z_m).\n\\end{align}\n\nNote that $r(0) = \\theta^\\star$. Rearranging the terms yields the expression in Equation (4). We will explicitly mention this in the updated manuscript.\n\n> **Eq. (11): Could something more be said about the properties of the approximation?**\n\nIn Equation (11), we approximate the Jacobians of different segments as statistically independent. There are two sources of randomness (as described in our footnote): (1) mini-batch sampling, which contributes to independence, and (2) autocorrelations in the optimization step, which induce correlations between optimization steps. Our approximation neglects the latter correlation.\n\n> **Re: Suggestions**\n\nWe appreciate your suggestions and will incorporate them in the next revision of our manuscript.\n\n[1] Bae, J., Ng, N., Lo, A., Ghassemi, M., & Grosse, R. B. (2022). If influence functions are the answer, then what is the question?. Advances in Neural Information Processing Systems, 35, 17953-17967.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers, 

We thank all reviewers for their detailed and thoughtful feedback. We are pleased that the reviewers found our work to be well-written, easy to read (**9an4**, **hk4U**, **3Y11**, **5ybi**), addressing an important problem in the field (**9an4**, **3Y11**), technically solid (**9an4**, **5ybi**), and supported by convincing experiments (**3Y11**, **5ybi**).

We will address your concerns and questions in individual responses and are committed to improving our paper based on your valuable insights. Should you have additional comments, we welcome further discussion during the author-reviewer period. Once again, thank you for your time and expertise.

Regards,

Authors of "Training Data Attribution via Approximate Unrolling"

Meta-review: The authors present a training data attribution method based on implicit differentiation and unrolling. There were four reviews, with three reviews fairly clear in their vote for accept. One reviewer voted for rejection, but did not make a compelling case when compared with the other three. All four reviewers agreed that the paper is in a well-polished state that makes the technical ideas clear as would be expected by a NeurIPS audience. Some concerns were voiced over the comparatively small size of the data sets, which could limit the potential usefulness of the optimization approach in the largest settings, but the results shown were convincing that the paper presents a useful idea.
Decision: Accept (poster)