Bayesian Influence Functions for Hessian-Free Data Attribution
Authors: Philipp Alexander Kreer
Wilson Wu
Maxwell Adam
Zach Furman
Jesse Hoogland
Venue: ICLR 2026 Poster (2026)
Keywords: Training Data Attribution
Interpretability
SGMCMC
Influence Functions
MCMC
Loss Landscape
Geometry
Singular Learning Theory
Robustness
Primary area: interpretability and explainable AI
TL;DR: We introduce a "Bayesian" generalization of influence functions that scales to models with billions of parameters.

Abstract: Classical influence functions face significant challenges when applied to deep neural networks, primarily due to non-invertible Hessians and high-dimensional parameter spaces. We propose the local Bayesian influence function (BIF), an extension of classical influence functions that replaces Hessian inversion with loss landscape statistics that can be estimated via stochastic-gradient MCMC sampling. This Hessian-free approach captures higher-order interactions among parameters and scales efficiently to neural networks with billions of parameters. We demonstrate state-of-the-art results on predicting retraining experiments.

Published as a conference paper at ICLR 2026
BAYESIAN INFLUENCE FUNCTIONS FOR HESSIAN-FREB DATA ATTRIBUTION
Philipp Alexander Kreer
Wilson Wu
Technical University of Munich Timacus
University of Colorado Boulder
wi wilson. wulsoonwu@coloaaae edu
philipp.c a www.eeerrttlooke
Maxwell Adam
Zach Furman
Jesse Hoogland
University of Melbourne
University Melbourne zach. furman gmail. com com
Timaeus
Timaeus
jesseet pesse@timaeus imaeus. co
maxêt ieeetmmauu.cco
ABSTRACT
Classical influence functions face significant challenges when applied to deep neural networks, primarily due non-invertible Hessians and high-dimensional parameter spaces. We propose the local Bayesian influence function (BIF), an extension of classical influence functions that replaces Hessian inversion with loss landscape statistics that can be estimated via stochastic- stochastic-gradien MCMC sampling. This Hessian-free approach captures higher-order interactions among parameters and scales efficiently to neural networks with billions of parameters. We demonstrate state-of-the-ar results on predicting retraining experiments.
INTRODUCTION
Training data attribution (TDA) studies how training data shapes the behaviors of deep neural networks (DNNs)-a foundational question in AI interpretability and safetyA standard approach to TDA influence functions (IF), which measure how models respond to infinitesimal perturbations in the training distribution (Cook 1977. Cook & Weisberg 1982). While elegant, influence functions rely on a Hessian inverse and, therefore, break down for modern DNNs. Theoretically, DNNs have degenerate loss landscapes with non-invertible Hessians, which violate the conditions needed to define influence functions. Practically, for large models, the Hessian is intractable to compute directly. Mitigating this requires architecture-specine approximations that introduce structural biases (Martens Grosse 2015 Ghorbani et al., 2019 Agarwal et al. 2017 George et al., 2018).
IF (V4, VL, II Computing intractable for large models.
BIF Cov, Ll, 4g
Dampening higher-order intrrctions.
prior sensitiely to degeneracy.
scalable batched estimution.
Hog
SGLD
V7
Aoymptoticaa by squivalent the strente of degeneracy.
landscape geometry datermines how lass f on somple Linfluences loss ( on sample
[Dampened] Influence Functions Influence via gradients and the inverse Hessian.
Bayesian Influence Functions Influence via statisties over the lccal lass landscape.
From influence functions (IF) Bayesian influence functions (BIF); We introduce the local Bayesian Influence Function (BIF), which replaces the Hessian inversion of classical Influence Functions (IF) with a covariance estimation over the local loss landscape. This approach sensitive to higher-order geometry and scales models with billions of parameters.
Figure 1:

---

Published as a conference paper at ICLR 2026
We propose a principled, Hessian- ffeee alternative grounded in Bayesian robustness. The key change is to replace Hessian inversion with covariance estimation over the local posterior (Giordano et al.) Grordano & Broderick] 2024 Iba 2025). This distributional approach naturally handles the degenerate loss landscapes of DNNs and bypasses the need to compute the Hessian directly. For non-singular models where the classical approach is valid, the BIF asymptotically reduces to the classical IF (Appendix(A) which establishes the BIF as a natural generalization of the classical IF for modern deep learning.
Contributions. We make the following contributions:
A theoretical extension of Bayesian influence functions to the local setting that enables the BIF to be applied individual deep neural network checkpoints (Section 2}
A practical estimator based on SGMCMC for computing batched local Bayesian influence functions that architectur-aanosee and scales to billions of parameters (Section
Empirical validation demonstrating that the local BIF matches the state of the art in classical influence functions, while offering superior computational scaling in model size. This makes our method particularly efficient for fine-grained and targeted attribution tasks where the up-front cost of classical IF approximations is high (Section
2
THEORY
We first review classical influence functions (Section 2.1 then review Bayesian influence functions (Section 2.2. and finally propose our local extension (Section 2.3)
2.1 CLASSICAL INFLUENCE FUNCTIONS
Classical influence functions quantify how model would differ under an infinitesimal perturbation to its training data.
Setup. We consider a training dataset Priin = {z, |2=1 and a model parameterized by w W Rd We define the empirical risk Luain (w) = Σi= (ww), where (xw) = {(z;; w) is the loss for sample 23- We assume Tarain continuously second-diiferentiable and that our training procedure finds parameters w² E W at local minimum, i.e., V Juman (w²) = 0.
Influence on observables. We want to predict how the value of an observable ((w): W R would change under perturbation to the training data. In particular, we'r re interested in predicting the response of query sample's 's loss v(w) = {(z;; un) model perturbation, we introduce importance weights 3 = (81,-. (..,.... (y,)) and define the tempered risk T ((w)) > (w)) Assuming the loss Hessian invertible, the implicit function theorem guarantees neighborhood Uw- 3 wn such that, for all ( sufficiently close to 1, there is unique minimizer of the tempered risk in this neighborhood w ((3) = arg min EU..- Lizain a (w). Note that w (1) = w" and that the function w² depends the starting w*: in this sense, the classical influence naturally local to choice of parameters w".
The classical influence of training sample z, on the observable evaluated at the optimum is defined as the sensitivity of s(w" (3)) to the weight 3r:
IF (Z; @)) Os(w" (w" (B))
0
33
Applying the chain rule and the implicit function theorem, we arrive at the central formula:
IF Φ) = - v H(w") "Vo""""""ww,,""""
(2)
where H(w (wn) is the Hessian of Taran evaluated w=.
2.2 BAYESIAN INFLUENCE FUNCTIONS
An alternative perspective, grounded in Bayesian learning theory and statistical physics, avoids the Hessian by considering distribution over parameters instead of single point estimate.
2

---

Published as a conference paper at ICLR 2026
Targer Sequence
Protiive Cormutation
Regitiiv Cerreatien
tocn
Dreatt
disoverd
aov fooil near
Do
Siv
cat Ted
Fu chuck
AMal rooing
Shee
bought
apples
and
arges
the maadet
The reai
anound (usaaaa and No charging
apiles
thooe
arges
matoet
popdeer
bought
Thee
apples and BDoa
argee the maaiet
cm pom mec at Inst angget aul maach
Hame What slantive Lacton Anorin okynciin
Lelp Josne Cad
quadions
on soreett
peoill wold (moasii isdermioon and seriiio Dar
wiied toimnaa wow Euo
1t. Inleeeed ispused ISR8 wheh Jelne frim
had 100
quoodar and iic croveree IS (oenccal
aud aquepel awess my iappous cnd cappes |Micel Day en sudiiisss and Ielevorea
The iquz Insat 110
iqaiiiionn and
she arswere
cornact
beasel
bessel fadirmeed (apacti lnd bare lnyia toll berved sarviag and Telgingg JAily
100
The (68)
rad 10
cuesiion.... and ehe Rnsoeered
coroedy
Figure The per-token BIF captures semantic relationships in Pythia-2.8B. The posterior correlation (negative of the normalized BIF) between tokens is maximized for relationships like translations, alternate spellings, and synonyms.
Influence on observable expectations. We obtain the Bayesian influence BIF (zi, @) of sample z; on an observable ø by replacing the point estimate ((w") in Equation ( with an expectation value Egrain, @ (6( (wr)
BIF (IF [z) 6) BEuain, a a(o(w)) 03,
B=1
Here, Emiin, [ll(ww]]]=]]]) [ ((w)) (w Domin) dw is an expectation over tempered Gibbs measure (ww 'urain) i exp( Luaann @ (w)))p(w) with prior ((w). This is a tempered Bayesian posterior the loss is a negative log- ligeeeklooood ((w)) log log w). which we assume the case for the rest of the paper.
standard result from statistical physics (see Baker 2025) relates the derivative of the expectation to covariance over the untempered (3 1) posterior under mild regularity conditions:
BIF 24, Φ) = (o)(CL(o), (())))
Bayesian influence is the negative covariance between an observable and the sample's loss over the tempered posterior. In Appendix A. we show that, for non-singular models, the leading-order term of the Taylor expansion of the BIF is the classical IF; the BIF is a higher-order generalization the IF.
2.3 LOCAL BAYESIAN INFLUENCE FUNCTIONS
Computing expectations over the global Bayesian posterior p(w (Duain) is generally intractable for DNNs. Furthermore, standard DNN training yields individual checkpoints w", and we are often most interested in the influence local to this final trained model. Therefore, we adapt the BIF with localization mechanism.
Following Lau et al. (2025), we define localized Bayesian posterior by replacing the prior (o(w) with an isotropic Gaussian with precision y centered at the parameters wn:
m Σ ((w)
Py Dusin, w²) x exp
w w
(5)
The local Bayesian influence function (local BIF) is defined as in Equation (4 but via a covariance over the localized Gibbs measure, indicated by the index
BIF BIF,(2, (Z,, Φ) = - Cov, (l(( (w)).
(6)
For comparison, note that classical IF's are ill-defined for singular models, such as neural networks with non-invertible Hessians. A common practical remedy is to use a dampened Hessian (H ((w)) TI). This is equivalent to adding an Eg regularizer centered at w" to the loss, which is the same trick use in defining BIF In Appendix A.2, show that the first-order term of the expansion of the local BIF is the dampened IF (with a vamishing dampening factor); the local BIF is a higher-order generalization of the classical dampened IF.
3

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
Computing the local BIF requires estimating the covariance Cov,(o(w), (ww) under Py (w Durain w²) Following Lau et al. 2025), in Se1lion3.11, we use stochastic gradient Langevin dynamics (SGLD;
BeGLD::Wlling Welling & 201 In Section 3.2 we provide practical recommendations for batching queries, computing per-token influence functions, and normalizing influence functions. In Section 3.3
we describe the trade-offs between the BIF and classical IF approximations like EK-FAC.
3.1 SGLD-BASED COVARIANCE ESTIMATION
SGLD approximates Langevin dynamics with stationary distribution Py Dirann w)) by updating with mini-batch gradients of the empirical risk Luain (w) and the gradient of the localizing potential 7(w w*) The update rule is:
12,3
Vwlk(wr) + + - -w") w0))
+ +O,c),))
W++1
kEB.
where B, is a stochastic mini-batch of m samples, e is the step size, and /3 is an inverse temperature (which puts in the tempered Bayes paradigm).
To improve coverage of the distribution Py, we typically sample several independent SGLD chains. We collect T draws {we,)]_ after an optional burn-in in each SGLD chain 1 I c S C. for a total of Narws CT draws. The required covariances Cov.,( Φ) are then estimated using the standard sample covariance calculated from the aggregated sequences (fe(W.))) ø (secc.)ss See Appendix B.T for further details and modifications from vanilla SGLD.
3.2 PRACTICAL TRAINING DATA ATTRIBUTION
BIF between data points. We focus on the Bayesian influence between training example z; E Duain and the loss of a query example zj E Damery's that is, we set the observable to e = f(zy; and compute BIF(2;, 2,) = (f(v) (wr)). Given the training set Drain and a query set D query- we compute all pairwise Bayesian influences (BIF (z,, zj) Z; E Dirain 2j E D } over the same draws from independent SGLD chains. At each step of each chain, we perform forward passes over both Dirain and D Queery to obtain losses over both sets (f, (et)((((w)) ((((w))),.DPmu)PPu
These forward passes are computed separately from the loss backward pass ΣREB, V (w,) used in the SGLD update rule.
Batched evaluation. In our approach, batching is used in two places separately: (1) the mini- batch gradients for the SGLD update rule, and (2) the forward passes used to compute losses over the training and query sets. This allows for scalable computation of the full BIF matrix B (BIF Z,, Zz) 2. &D EDuuyy"
Per-token Bayesian influences. In the autoregressive language modeling setting, each example 2; a sequence of tokens 2; = (Zi,1..... 2i,s) ) of length S. The loss at example 2; then decomposes as
5
(ww) = Σ
log p(zi. Zi,1,... Zi.s-1) =: Σ (w))
a=2
The BIF can be easily extended to this setting: for example, the Bayesian influence of the sth token of sequence on the loss the s'th token of sequence is BIF Zj,a') = (C.,, (w), (w)). In our language model experiments, we compute all such pairwise per-token influences, resulting in P S|Dquery BIF matrix. As we describe in Appendix B.4 this parallelization major advantage over classical IF approximations like FK-FACC
Normalized influence as correlations. Raw covariance scores can dominated by high-variance data points. To create more stable and comparable measure of influence, we also consider the normalized BIF, which corresponds to computing the Pearson correlation instead of a raw covariance. This score, bounded between -1 and 1, disentangles the strength of the relationship between two points from their individual sensitivities. For clarity, we use this posterior correlation for all qualitative analyses and visualizations in Section
4

---

Published as a conference paper at ICLR 2026
Inbel-
Corr-O Norr-0.346 346 (label-188 188)
Corn- 0..000.,30 0.517 Hlabel1880
Cor--0,300 0.300
(labelm 188)
Score- 375929
(abel- 188)
Scora=197015
Ilabel-
Score- 190415
(label 266)
Iebel-951
Corr=O 313 0abel-951)
Corn= 0.33 (label=050)
Com-00087 (label
Scoro=22453 (abeew051)
Scorg- 18117
[label
Scarc=14108
(label-443
lsbel- 140
[abbel140)]
Car-- 0.21
CoT-0.07 000070
0abel-11
Score-40455
(abel-
Score- 18043 (label220)
Scarc-1650 16552
(label
label=528
Corr-0.30 -.34 [label-520]
Corr 319
(28)
(abbll-528)
Cor--0.314 314
Score Exore-1660 -960
(abel=700) 706)
Score- 3066 llabel=429
5165 (label-B49)
(a) Query
(b) BIF
EK-FAC
Figure 3: BIF and EK-FAC show convergent validity on Inception-vl. For a given query image (left), our local BIF (center) and EK-FAC (right) identify similar identical training images as most influential. See Appendix D.I for more examples.
3.3 COMPARISON TO CLASSICAL IF APPROXIMATIONS
We compare our local BIF approach to classical influence function (IF) approximations, using EK- FAC representative state-of-the-art example (Grosse al. 2023). The key differences between the BIF and EK-FAC are summarized in Table and elaborated on below. We provide comparisons additional IF techniques in Appendix A.3
Time complexity. Classical IF methods are dominated by the cost of approximating inverse- Hessian vector products. Direct inversion is intractable, so methods like EK-FAC rely on a fir phase where blockwise Kronecker factors are estimated and inverted. The main bottleneck is the eigendecomposition or inversion per-layer covariance matrices, which scales cubically with the layer width (O(d)) per block). Once fit, scoring reduces to repeated matrix-vector solves, but still requires recomputing gradients for each query-train pair, with total complexity (ggnP) where P is the per- vector solve cost. Thus, EK-FAC is most efficient when many queries training samples amortize the expensive fit phase.
The local BIF, by contrast, has structural fit cost. The main bottleneck is running forward passes over the entire train and query datasets at each SGLD draw, with overall complexity scaling as O(Narows (n g) dooaa)).
There is one caveat, which is that the both techniques depend on number of hyperparameters and thus require calibration sweeps, which can potentially dominate the total time costs. However, found that EK-FAC works well with the provided defaults, and, in Appendix C.4 we show that results for the BIF (as measured by LDS) are stable across wide range of hyperparameter ablations.
In short:
Classical IF's are efficient for large-scale, sequence-level attribution where large number of queries can amortize the high initial investment.
BIF more efficient for a smaller number of queries or for fine- ame-grained attribution. For per-token influence, batched approach calculates the entire token-token influence matrix at once, while classical methods would require a separate, sequential scoring pass for every single token, making them impractical.

---

Published as a conference paper at ICLR 2026
Table BIF vs. EK-FAC. Comparison of time/space complexity and estimation quality for the local BIF and EK-FAC. Here duai is the number of parameters, n. the training set size, the query set size, Narws the total SGLD draws, and Nu the samples used fit EK-FAC factors. The EK-FAC scoring cost assumes training gradients are recomputed. See Appendix A.3 to compare the BIF against other IF techniques.
Axis
Local BIF
EK-FAC
Time Complexity
Score: (ddrws (n + 4)dara (No fit phase)
Fit: (Narchoaall + En(dea.t + dou,
Score: (nggdooaal
Memory (extra)
O Nrawss (n + q)) for loss traces
O OE +dbunt)) for factors
Sources of Error
Finite sampling draass))
Finite sampling (Nm) Structural bias (Kronecker, Fisher)
SGLD bias/hyperparameterrs (Neal.
Architecture
Any differentiable model
Linear and Conv2D layers
Memory complexity. Hessian-based methods often require storing structural components of the model, such as the Kronecker factors and eigenbases in EK-FAC, with memory usage scaling with layer dimensions (O(E,(e,, + de :)). For models with large hidden dimensions, this can be substantial. The local BIF's memory usage is dominated by storing the loss traces (O(Nasss (n + 4))). Alternatively, it is also possible to use an online covariance estimator for BIF with memory usage O((n 2)) which is more efficient when Naraws larger than the total number of data points.
Sources of error. Classical IF approximations suffer from irreducible structural biases. For instance, approximating the Hessian with Kronecker-factored decomposition introduces errors that do not vanish even with infinite fitting data.
In principle, the BIF provides an unbiased estimator of influence under its target distribution that improves with the number of total draws Nraws- However, accurately sampling from the (local) posterior of singular model like DNN is notoriously difficult, and standard SGLD convergence guarantees may not hold (Hitchcock & Hoogland] 2025). This can introduce a systematic sampling bias. Another possible source of error is that currently lack rigorous understanding of how to choose hyperparameters like the inverse temperature (3) and localization strength (?), which are part of the definition of the local posterior being analyzed (see Appendix B.1).
Architectural flexibility. Our method model-agnostic and can be applied any differentiable architecture. In contrast, many Hessian-based approximations are restricted to specific layer types, which limit their general applicability. particular, EK-FAC restricted to Linear and Conv2D layers, thus excluding influences from attention or normalization layers in large language models. If desired (for example, to achieve a closer comparison to EK-FAC), is possible to restrict the BIF to subset of weights, see Appendix B.I
4 RESULTS
In this section, we present empirical results to validate the local Bayesian influence function (BIF) as a scalable and effective TDA method. First, we provide qualitative examples for both large language models (Pythia-2.8B) and vision models (Inception-vl) to build intuition. Second, conduct quantitative retraining experiments and show that the BIF faithfully predicts the impact of data interventions, often outperforming strong influence- scaling analysis across the Pythia model suite to demonstrate the computational advantages of our approach over Hessian- based methods like EK-FAC
baselines. Finally, we perform
4.1 VISUALIZING THE BIF
We first present qualitative examples to build intuition for the BIF's behavior for both the Pythia 2.8B (Biderman et al. 2023) language model on (Figure 2 and the Inception-vl (Szegedy al.

---

Published as a conference paper at ICLR 2026
EK-FAC
BIF
GradSim
TRAK
0.4
Ayntrain 0.1
Oratrain 0..
Antrrio = 0.5
Arstrain 0.7
0.3
LDS Score
0.2
0.1
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Clatributionn
Alttribution
Gattribution
Ratiribuionn
Figure 4: Bayesian inftuence functions (BIF) vs. classical influence function approximations (EK-FAC, TRAK, GradSim) on predicting retraining experiments (on CIFAR-I0 data) measured by the linear datamodeling score (LDS). We vary the size of the query dataset and full dataset according to Oattribution, then retrain on random subsets of Grearin samples. The LDS measures the correlation between the query losses after retraining and the predicted losses according TDA. We report the mean and the standard error across five repeated runs of the full experimental pipeline (including model retraining. BIF, and EK-FAC, etc. computation) with fixed hyperparameters but distinct initial seeds. The BIF consistently matches EK-FAC, which is SOTA. The BIF slightly underperforms EK-FAC for larger datasets (but within the margin of error) and slightly outperforms EK-FAC for smaller datasets. Both EK-FAC and BIF consistently outperform GradSim and TRAK.
2015) image classification model (Figure 3}. As described in Section 3.2 we use the normalized BIF for both (i.e., correlation functions). See Appendices) B and for details.
Image classification. Figure compares the highest-influenes samples identified by BIF and EK- FAC for Inception-vl on ImageNet samples (Deng al.,, 2209).. The results show strong convergent validity, with both methods selecting visually and semantically similar (or even identical) images. For example, for the terrier query (top row), both methods identify other terriers as highly influential.
Per-token language attribution. A key advantage of our approach is its ability scalably compute fine-grained, per-token influences. As shown in Figure 2 on Pythia-2.8B for samples drawn from the Pile (Gao et al.) 2021), the per-token BIF detects semantic similarities between tokens. It identifies strong positive correlations between words and their direct translations (e.g., 'She' t> "elle'), numbers and spellings (e.g., '3' 'three'), and conceptually related words (e.g.. 'objectives' < Maxim[izing]', "goals", 'motives").
4.2
RETRAINING EXPERIMENTS
The ultimate aim of TDA methods is to inform interventions such as data filtering and curriculum design. Thus, the gold-standard evaluation is retraining experiments, which measure the impact of changing the training set. However, performing thousands of leave-one-out (LOO) retraining runs computationally prohibitive. The Linear Datamodelling Score (LDS) provides a practical and scalable alternative (Park et al. 2023b). The intuition is to retrain the model on many different random subsets of the data and check how well TDA method's scores correlate with the true, empirically observed outcomes from these retraining runs. A higher correlation (a better LDS) indicates that the TDA method is a more faithful predictor of real-world interventions (see Appendix for details).
Our experimental setup allows us explore performance in different data regimes. From the full training dataset (CIFAR-10 10; Krizhevsky 2009) of size n, we first identify an "attribution set" of size Natribution Oatttibtionn n containing the training points whose influences we will compute along with a fixed set of q queries. The LDS is then calculated by retraining models (ResNet-o -9 Dordan (024) on numerous smaller subsets, each of size Pl
Oretrain Dlatribationn drawn from this attribution set.
7

---

Published as a conference paper at ICLR 2026
We use the full dataset of size both to fit EK-FAC's Hessian components and to draw the BIF's SGLD minibatch gradients.
Our findings reveal compelling trade-off between methods. The performance of all TDA methods improves as the attribution set size (Aruiiittioo)) decreases. the scenario where the retrain subset size (Mretrin) is small, removing few points creates larger relative change in the dataset. We find that in this small-model, high-variance regime, the local BIF consistently outperforms EK-FAC, achieving a higher LDS.
In Appendix C.3 we additionally report LDS scores for Pythia- 14M in a finetuning setting, where the BIF underperforms EK-FAC, which we attribute to greater difficulties with sampling in the language model regime.
In the image-modeling experiments, EK-FAC is around five times faster than the BIF. This advantage is largely due to the small model sizes (~~ 2 x 10t parameters). As we expect (see Section (33) the BIF to outperform EK-FAC when it comes larger models, we turn to a model-size scaling comparison.
4.3 SCALING ANALYSIS
In this section, benchmark the BIF's scaling models from the Pythia suite (Biderman et al. 2023). We measure the influence of a 400-sequence subset of the Pile training dataset (Gao et al. 2021) 18 prompt-completion query pairs. As in Grosse al. (2023). each query sequence is split into a prompt and completion 2, = ) each observable is then a per-token loss (w) -p(Zcomp.s promppii w). In this setting, running full retraining experiments becomes prohibitive, so we focus on comparing the computational cost of the BIF to classical influence functions approximated with EK -AC (George et al. 2018).
See Figure 5 for benchmark results. For the choice of SGLD hyperparameters we use (2k total draws, or 2.5x fewer than in Figure 2 we observe that BIF scales better than EK-FAC in evaluation time. Further, notice that F.-FAAC has large up-front cost in time and storage associated to fitting the approximate inverse Hessian, independent of the query dataset size. This overhead is only justified one wants compute sufficiently many influence scores. See Appendix B.3 for further experiment details and Appendix D.2 for direct comparison of the results.
BIF
EK-FAC
Peak GPU Memory (GiB)
200
Time (min)
300
200
100
100
0
0
1
2
3
2 3
Model Size d (B)
Model Size d (B)
Figure 5: Scaling comparison of BIF and EK-FAC across model sizes of the Pythia model suite. (Left) Evaluation time, excluding the tokenization time. (Right) The node's (4xA 100) peak GPU RAM usage. For the largest models, the BIF orders magnitude faster, while using the same GPU RAM as the EK-FAC.
RELATED WORK
Influence functions and training data attribution. Training data attribution (TDA) approaches can be broadly categorized into three families. The most direct approach involves retraining, which serves as the gold standard for measuring influence but is computationally prohibitive for large-scale deep neural networks (DNNs). A second family of methods relies similarities in the model's
8

---

Published as a conference paper at ICLR 2026
representation space, using intermediate activations to connect training and query points (Park et al. 2023b).
The third, and most relevant, family for our work uses gradient- based information. A prominent example is the classical influence function, a well-studied technique from robust statistics (Hampel
1974 Cook 1977, Cook & Weisberg 1982). Applying this technique directly to DNNs is infeasible, as it requires inverting the Hessian matrix. Consequently, much prior work has focused on developing tractable approximations to the
product (Koh & Liang 2020) Grosse et al.) 2023, Park et al. 2023b). Other gradient-based strategies approximate TDA by differentiating through the optimizer steps of the training process itself (Bae et al.) 2024). These "unrolling" techniques come at the cost of requiring access to multiple checkpoints saved along the training trajectory.
Distributional training data attribution. Recent work has recognized that training data attribution should account for the stochastic nature of training. Mlodozeniec et al. (2025) formalize this with distributional training data attribution (d-TDA), which frames the goal of influence as predicting how the distribution over trained models changes when data is removed. Their framework is deliberately general, accommodating arbitrary distribution families and distance metrics (Wasserstein, KL divergence, mean shift, etc.).
Our local BIF can be viewed as a particular instantiation of "mean-shift" d-TDA with a tempered Gibbs posterior. These specific choices enable us apply the covariance identity (Eq. that unlocks our novel SGMCMC-based methodology. This covariance as the "Bayesian Infinitesimal Jackknife" (Giordano & Broderick 2024 Iba 2025 in the context of Bayesian model analysis. However, to our knowledge, we are the first to formulate a local BIF and scale these distributional methods large-scale language models trained using standard stochastic optimization.
definition has appeared previously
Singular learning theory and developmental interpretability. Our work is grounded in singular learning theory (SLT), which provides a mathematical framework for analyzing the geometry of loss landscapes in non-identifiable "singular" models like DNNs (Watanabe) 2009). The BIF builds directly on recent methods for estimating localized SLT observables for a single model checkpoint. Specifically, Lau al. (2025) introduced an SGMCMC- estimator for an SLT quantity known as the local learning coefficient (LI.C) by sampling from a "localized posterior" the same mechanism we use to define our local BIF (Eq. 5). Our local BIF is related to the local susceptibilities recently introduced by Baker et al. (2025). Together, these methods form part of broader "developmental interpretability" research agenda, which uses tools from statistical physics and SLT to probe how data shapes the learned representations and local geometry of neural networks (Pepin Lehalleur et al. 2025).
DISCUSSION & CONCLUSION
We introduce the local Bayesian influence function (BIF), novel training data attribution (TDA) method that replaces the ill-posed Hessian inversion of classical influence functions with scalable SGMCMC-based covariance estimation. Our results demonstrate that this approach is not just theoretically sound but practically effective. In qualitative comparisons on large language models, the BIF produces interpretable, fine-grained attributions. Quantitatively, achieves wate-of-the--ar performance on retraining benchmarks, matching strong baselines like EK-FAC in realistic data intervention scenarios.
Advantages. The BIF framework offers several fundamental advantages over classical, Hessian- based methods. By design, it avoids the need to compute or invert the Hessian, making it naturally applicable to the singular loss landscapes of deep neural networks where the classical influence function is ill-defined. The underlying SGMCMC sampling is model-agnostic and can be applied to any differeentaalee architecture. Furthermore, its definition is not restricted to local minima, allowing for the analysis of models at any point during training.
Limitations and practical trade-offs. The primary limitation of the BIF is that its accuracy depends on the quality of posterior sampling, which remains a practical challenge. The SGLD sampler introduces sensitivity to hyperparameters (c, 7, ()) whose optimal values are not yet well

---

Published as a conference paper at ICLR 2026
characterized, particularly in the language model setting (Appendix C.S
Unlike the structural
approximation errors in classical IFs, this limitation can be tackled without changes to the underlying BIF framework. Advances in sampler design. convergence diagnostics, and hyperparameter selection would directly translate into better BIF estimates without requiring any changes to the underlying definitions.
The other main limitation of the BIF lies in the practical trade-offs of its computational cost. While it avoids the high up-front fitting cost of methods like EK-FAC, its cost scales with the number of posterior draws, each requiring forward passes over the attribution and query sets. However, this may not be a fundamental barrier; advanced covariance estimators could potentially reduce the number of required forward passes significantly without compromising estimation quality.
Future directions. Our work opens several promising avenues for future research. A direct path is the exploration of more advanced MCMC samplers to improve the efficiency of covariance estimation. Furthermore, the role of the BIF's hyperparameters can be explored further; the localization strength y and inverse temperature can be viewed not just as parameters to be tuned, but as analytical tools to probe influence at different scales and resolutions of the loss landscape. Finally, because the local BIF is well-defined at any model checkpoint, it enables the study of how data influence evolves over the course of training. This opens the door dynamic data attribution, tracing how certain examples become more or less critical at different stages learning.
In conclusion, the local BIF reframes data attribution from point-estimate problem to distributional one. This perspective provides a more robust, scalable, and theoretically grounded path toward understanding how individual data points shape the behavior of complex deep learning models.
ACKNOWLEDGMENTS
We would like to thank Lev McKinney for his detailed input and advice regarding EK-FAC. We also grateful to Simon Pepin Lehalleur and Daniel Murfet for their valuable feedback, and to Daniel Murfet for insightful discussions, and Lorenzo Tancredi for his support. We thank Stan van Wingerden and William Snell for their help with setting up the infrastructure and feedback on the BIF implementation.
Philipp Alexander Kreer supported by the European Research Council (ERC) under the European Union's 's research and innovation programme grant agreements ERC Starting Grant 949279 HighPHun, and the Pivotal Fellowship.
Zach Furman was supported by the Melbourne Research Scholarship and Rowden White Scholarship during the completion of this research.
AUTHOR CONTRIBUTIONS
Philipp Alexander Kreer led the experimental component of the project, conducted the retraining and scaling experiments, performed the comparison against EK-FAC, ran qualitative per-token BIF experiments on language models, and contributed to the writing. Wilson Wu implemented the initial retraining experiments, contributed to the initial BIF implementation and hyperparameter search, and contributed to the writing. Maxwell Adam conducted the ImageNet experiments and parts of the qualitative language modeling experiments. Zach Furman contributed to the theoretical development. Jesse Hoogland conceived the project and derived the local BIF, led the project, and led the writing.
REPRODUCIBILITY STATEMENT
To ensure our work is reproducible, we provide detailed descriptions of our methodology throughout the paper and its appendices. The core SGLD-based estimation procedure for the local BIF is formally presented in Algorithm T All experiments were conducted on public datasets (CIFAR-10, ImageNet, The Pile) and standard model architectures (ResNet-9, Inception-vl, 1 Pythia), results (Section 4). A complete summary of the SGLD hyperparameters used for each experiment is available in Table 3 with further implementation details discussed in Appendix B.T The setup for
described in
10

---

Published as a conference paper at ICLR 2026
retraining experiments, including the LDS evaluation protocol and model training hyperparameters, detailed in Appendix [ Finally, the specifics our scaling analysis, computational environment, and comparison against the EK-FAC baseline can be found in Section 4.3 and Appendix B.3 B.3
LLM USAGE STATEMENT
We used Large Language Models (LLMs) assist with writing, coding, and theory in this paper. Their role included improving the text's clarity and structure, helping to implement code for experiments and figures, and assisting in derivations (such the BIF asymptotically recovering the classical IF, see Appendix [A.I). I. All Al-generated content was reviewed and validated by the authors, who retain full responsibility for this work.
REFERENCES
Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization for machine learning in linear time. J. Mach. Learn. Res., 18(1):4148-41) 4187, January 2017. ISSN 1532-4435.
Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger B. Grosse. Training data attribution via approximate unrolling. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Garrett Baker, George Wang, Jesse Hoogland, and Daniel Murfet. Structural inference: Studying small language models with susceptibilities, April 2025. URL http:://axxi..org/.aaa 2504. 18274 arXiv:2504. 1000044.8227 18274 [cs].
Stella Biderman, Hailey Schoelkopf. Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference Machine Learning, ICML'23. JMgR.org, 2023.
Tyler A Chang, Dheeraj Rajagopal, Tolga Bolukbasi, Lucas Dixon, and Ian Tenney. Scalable Influence and Fact Tracing for Large Language Model Pretraining. 2025.
Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input Similarity from the Neural Network Perspective. In Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https: http:///ccccceeeinn..oo. / neurips /c/ cttpaashccccf55700 wtt.ee/2ebbaaafeeeeeeeeeeeeeeeee Aatee Abstract. html
Zhongtian Chen and Daniel Murfet. Modes of sequence models and learning coefficients, 2025. URL https://arxiv: http//axxvvv...raaaa// abs 18048
Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, Jeff Schneider, Eduard Hovy, Roger Grosse, and Eric Xing. What is your data worth to gpt? Ilm-scale data valuation with influence functions, 2024. URL
abs /2w//2405..0 13954
R. Dennis Cook. Detection of influential observation in linear regression. Technometries: journal of statistics for the physical. chemical, and engineering sciences, February 1977. ISSN 0040-1706. URL Intps:: www. tandfonline con./ doi/ abs 10. 1080/00401706 1977. 10489493
tex.copyright: Copyright Taylor and Francis Group, LLC.
R. Dennis Cook and Sanford Weisberg. Residuals and influence in regression. Monographs on statistics and applied probability. Chapman and Hall, New York, 1982. ISBN 0-412-24280-00 URI.
https: /hhll handle nnt/ 1tt//11199/33
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.1 Wtt./CVPR.00095200000

---

Published as a conference paper at ICLR 2026
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027, 2021. URL
pettps:/earse wwwpxxv..vrr/oooo wtw./as////aaaaoooao///ossaaaers 2201 20tpo///100000277
Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in kronecker factored eigenbasis. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: /ttpss///occceeenngse.e. nwwrips. cc/pper files/paper/20111 Ei 1e144800004 163111660009130aaae
70b3 Paperaappppppee
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net opti- mization via Hessian eigenvalue density. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2232-2241. PMLR, 09-15 Jun 2019. URL https://proceediin wwtpeeeiiiinsmm......m www.pp..sss.................e..e httpss/////ghobbba111320 att.mmabbaiiibmmmm
Ryan Giordano and Tamara Broderick. The Bayesian infinitesimal jackknife for variance, 2024. URL
https://arxiv.oom wwtp://wwrrr//oo/ooooooooooos/// 23305 06466
Ryan Giordano, Tamara Broderick, and Michael Jordan. Covariances, robustness, and variational 2017.
Bayes. Journal of Machine Learning Research, 19:51:1-51:49, 2017. URL hhttps://ap
Bemant.ccchollar wtw.:aaa....rrooouuiiiiioa 532387992
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage. Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilé Lukošiūté, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying Large Language Model Generalization with Influence Functions, August 2023, htttt//::/aaxx org/abs/ 2308. 03296 arXiv:2308.n 0.3296 [cs].
Frank R. Hampel. The influence curve and its role in robust estimation. Jourmal of the American Staris- tical Association, 69(34)):383 393, 1974. doi: 10. 00011000016211499.9944..000 10482962, URL | 974 1...10048666
wwww. tand ne com do htt.m///o////b/a/aaaaa 1110//110111001111111111
Kaiming He. Xiangyu Zhang, Shaoging Ren. and Jian Sun. Deep residual learning for image 03385
recognition. CoRR, abs/15 2015. URL /RLpttt::/arxiiv.o
Rohan Hitchcock and Jesse Hoogland. From Global to Local: A Scalable Benchmark for Local Posterior Sampling, July 2025.
Yukito Iba. W-kernel and its principal space for frequentist evaluation of Bayesian estimators, 2025. URL
wttp:///gg/ab///////e 1.10117
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernll: Convergence and Generalization in Neural Networks, February 2020. URL http::/arss...org/bbb.. 07572 1rXiv:1806.07572 [cs].
Keller Jordan. 94% CIFAR-10 in 3.29 seconds on single GPU, CoRR, Abb doi: 1044550/AARXIV.240.0000 URL
00498
Jean Kaddour. The MiniPile challenge for data-efficient language models.
arXiv preprint
arXiv:2304.08442. 2023.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions, December 2020. URL httt:://rx/xxx>> wttp://w.wwrg//////.././... 04730 703.44370 [stat] CitationKey: deep-inffuenne-ffnnctoo...
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Univ. Toronto, 2009.
12

---

Published as a conference paper at ICLR 2026
Edmund Lau, Zach Furman, George Wang, Daniel Murfet, and Susan Wei. The local learning coefficient: a singularity- complexity measure. In The 28th international conference on artificial intelligence and statistics, 2025, URL https::/openrevieeeee/e//// wttp:/eviwiieeeeoooreeeeee
1av51 71 suL.
Chunyuan Li, Changyou Chen, David Carlson, and Lawrence Carin. Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks, December 2015. httt::///aaxx
hxzg/bb/1511 07666 arXiv:1512.07666 [stat].
Stephan Mandt, Matthew D. Hoffmann and David M. Blei. Stochastic gradient descent as approximate Bayesian inference. J. Mach. Learn. Res., 18( 1):4833-4907, January 2017. ISSN 1532-4435.
James Martens and Roger Grosse. Optimizing neural networks with Frotecker-factored approximate curvature. Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2408 2417. Lille, France, 07-09 Jul 2015, PMLR. URL
martena15. h
James Martens and Roger Grosse. Optimizing Neural Networks with Kronecker-factored Approximate Curvature, June 2020. URL hitte://arxiv..or//bb///555 arXiv:1503.05671 [cs].
Chris Mingard, Guillermo Valle-Pérez, Joar Skalse, and Ard A. Louis. Is SGD a Bayesian sampler? well, almost. J. Mach. Learn. Res., 22(1), January 2021. ISSN 1532-4435.
Bruno Mlodozeniec, Isaac Reid, Sam Power, David Krueger, Murat Erdogdu, Richard Turner, and Roger Grosse Distributional Training Data Attribution: What do Influence Functions Sample?, October 2025.
www.///////a./////aa///. abtp///...//22...../// 12965 arXiv:2506.1240 [cs].
Sung Min Park, Kristian Georgiev, Andrew Ilyas. Guillaume Leclere, and Aleksander Madry. TRAK: 2303
Attributing Model Behavior at Scale, April 2023a. URL
14186
arXiv:2303. 14186 [stat].
Sung Min Park, Kristian Georgiev, Andrew Iyyas, Guillaume Leclere, and Aleksander Madry. TRAK: Attributing model behavior at scale, April 2023b. URL arXiv:2303.14188 14186 [stat] CitationKey: TRAK.
14186
Simon Pepin Lehalleur, Jesse Hoogland, Matthew Farrugia-Roberts Susan Wei, Alexander Gi- etelink Oldenziel, George Wang, Liam Carroll, and Daniel Murfet. You are what you eat-AI alignment requires understanding how data shapes structure and generalisation. arXiv preprint arXiv:2502.05475. 2025.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
http::/jmlz: wttpp//ppp.///////////..///
.html
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116(23):
11537-11546, 2019, doi: 10.1073/pnas. 1820226116. BRLhhtps:: /www www.wwwwww.ww......m.......w...p pnas org/
/tt.7//paas.8222661111 abs/10.1073/pms ppa.s 0tt.:10000/ppoo..1222111111 1820226116
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 1-9. IEEE Computer Society, 2015. doi: 10. 10.1109/CVR.22015.7998/ URL https::/doiie org/00 1119/ 2015 7298594
Stan van Wingerden, Jesse Hoogland, George Wang, and William Zhou. DevInterp, 2024. URL https://github.. wattmmttimmuuuss imaeus rewearch/ dww.nheeeeeeereeeereer
Andrew Wang, Elisa Nguyen, Runshi Yang, Juhan Bae, Sheila A. Mcllraith, and Roger Grosse. Better Training Data Attribution via Better Inverse Hessian- Vector Products, July 2025a. URL wrXiv:2507.14740 [cs].
http://arxi http:/wwwwwww.aas//////ooee
13

---

Published as a conference paper at ICLR 2026
George Wang, Jesse Hoogland, Stan Wingerden, Zach Furman, and Daniel Murfet. Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient. Proceedings of The 13th International Conference on Learning Representations, 2025b.
Sumio Watanabe. Algebraic Geometry and Statistical Learning Theory. Cambridge Monographs on Applied and Computational Mathematics, Cambridge University Press, 2009.
Susan Wei, Daniel Murfet, Mingming Gong, Hui Li, Jesse Gell-Redman, and Thomas Quella. Deep learning is singular, and that's good. IEEE Transactions on Neural Networks and Learning Systems, 34(12):00433-1446, 2023. doi: 10. N0.1109//NNLSS.222..11..7
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11, pp. 681-688, Madison, WI, USA, 2011. Omnipress. ISBN 9781 450306619.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
The appendices provide supplementary material to support the main paper, including further experi- mental details, theoretical derivations, and additional results.
Appendix A details the theoretical relationship between Bayesian influence functions (BIFs) and classical influence functions (IPs), showing how IFs emerge as leading-ordee approx- imations. Appendix A.3 compares the BIF against additional IF approximations besides EK-FAC.
Appendix provides further experimental details, including the setup for comparing local BIF against EK-FAC (Appendix B.3) and the specifics of the SGLD estimator presented in Algorithm
Appendix provides additional detail on the retraining experiments on ResNet-9 trained on CIFAR- 10.
Appendix D presents additional qualitative results for the BIF on vision and language models, well as comparisons with EK-FAC.
A RELATING BAYESIAN AND CLASSICAL INFLUENCE FUNCTIONS
A.1 RELATING THE BIF AND UNDAMPENED IFs
This appendix details the relationship between Bayesian influence functions (BIFs) and classical influence functions (IFs). In particular, show that, for non-singular models, the classical IF is the leading-order term in the Taylor expansion of the BIF, This establishes the BIF as natural generalization of the IF that captures higher-order dependencies between weights.
Let w" be a local minimum. In this section, all gradients and Hessians are evaluated at w"; thus, to reduce notational clutter, we omit the dependence on w. For any function f(w), we denote its gradient at w as g/~ = V f fww") and its Hessian as H = f. f(w"(w" (w"), ). In particular, Ir = V vwo(w") and H. = v of (n") ) for an observable ((w): we also abbreviate g: = V (w=) and H, V (1w" for per-sample loss (w). The total Hessian of the empirical risk Luain (w) = Σ R EA (w) at w is denoted H Σ Hy.
The Bayesian influence function (BIF) for the effect of sample z, on an observable ø is given by (see Eq.
BIF (4, Φ) = (o
(w), ( (xw)),
(7)
where the covariance is taken over the posterior p(w) 'Duiin) αx hup(--Tmann ((()), with p(w) being prior. This definition exact and makes assumptions about the form of ol (w)) (x(wr). or p(w] D.nin).
To understand the components of this covariance and its relation to classical IFs, consider an idealized scenario where the model is non-singular. Under this strong assumption, which does not hold for deep neural networks (Wei et al. 2023), the posterior p(w Deraia) can be approximated by Laplace approximation around w":
pww Dreai) z "N""H"" HH
(8)
The Bernstein-von Mises theorem states that, due to the model's 's regularity, the posterior distribution converges in total variation distance to the Laplace approximation as the training dataset size n approaches infinity.
Let Aw w w". Assuming analyticity, we can express ((w) and (,(w) using their full Taylor series expansions around w=:
((w) ((ww)) gh Aw + Aw" H&Aw
D* (("(w")) |Am, ...,.... A.).
(9)
k
1
(ww)= = ((ww)) + g. Aw + Aw H,Aw +
D* E, (, |Aw, Aw). (10)
2
15

---

Published as a conference paper at ICLR 2026
where D* fl("")AA Aw] denotes the k-thh order differential of f at w" applied to & copies of Aw.
The covariance under this Gaussian (Laplace) approximation, denoted Covy, then involves covari- ances between all pairs of terms from these two expansions:
Corv(á(()) (,w))
Covw (Termalo) Term,, Term,, (erm,.44). [4]]
(11)
1= m==
where Terma[f] is the k-th order term in the Taylor expansion of f(w) in powers of Aw. For Au ~ (O, H-1), the leading terms are:
Covariance of linear terms (k = -1,====1
Cov Aw, Aw) 9o H H 9:-
Covariance of quadratic terms (h: 2, m 2):
Cov ( [ H HoAw, _ Aw" H;Aw Lr H
(Using Isserlis' theorem for moments of Gaussians).
Cross-terms between odd and even order terms (c.g., k = 1,m = 2) are zero due to the symmetry of Gaussian moments.
Thus, the BIF under these regularity and Laplace approximations becomes:
(2:; 3) ~ -9 H--H-1
--1 9:
1
-
Σ k.
Covy (Term,())))) Termm[(]]]
2
not or
k++m
(12)
w. The leading term -9 96 H H g. -w H (w "
is precisely the classical influence function IF (z: ②) from Equation (2). Note that H scales linearly in n, so this term dominates - The BIF formulation, when analyzed via Laplace approximation, naturally includes this term and also explicitly shows a second-order correction involving products of the Hessians of the loss and observable. More generally, the exact BIF definition (Eq. 7 encapsulates all such higher-order dependencies without truncation, which are only partially revealed by this expansion under the (invalid for neural networks) Laplace approximation.
A.2 RELATING THE LOCALIZED BIF AND DAMPED IFs
We now extend this analysis to the local BIF, showing that its leading-order term is precisely the dampened classical IF, which is the standard practical remedy for the singular Hessians found deep neural networks.
The local BIF is defined over the localized posterior from Equation (5)
Σ (u(()) ? "Iw w01l3
(w Derann w=") x exp
= exp (Luainnuu) +
2 1|
w 112
(13)
This distribution centered around w0 due to the localizing potential (the quadratic term). To apply the Laplace approximation, consider the mode of this distribution, which is the minimum of the effective potential Ler(w) = Lirain (w) + ||w w 11. We assume w" to be local minimum
of Luran w.), so Lurai (w") O. Consequently, V VLar(w) (w") V 7(w" ww") = 0.
meaning w" also the mode of the localized posterior.
The precision of the Laplace approximation is given by the Hessian of this effective potential evaluated at w":
Han 2 (2Tun
2 Tarain (w=) + + H III.
(14)
16

---

Published as a conference paper at ICLR 2026
Therefore, the Laplace approximation for the localized posterior is Gaussian centered at w" with covariance H
Py ( Dirainn w²) R N(w", ,H + -
(15)
Following the same Taylor expansion logic as in the previous section, we can compute the leading- order term of the covariance between ((w) and ((w)) under this Gaussian approximation:
Cov- (0(()) (w)))
Covarg Aw, g, Aw) g [H + - g:
(16)
The local BIF is the negative of this covariance:
BIF (z.,
6) - g H + 71) -1) g..
(17)
This expression is exactly the form of the classical dampened influence function, where the localiza- tion strength yy serves as the dampening coefficient. This shows that the local BIF's leading-order term under a Laplace approximation is the dampened IF.
Just as the global BIF generalizes the classical IF, the local BIF is natural, higher-order generalization of the dampened IF, capturing dependencies beyond the second-ord approximation while remaining well-defined and computable for the singular models used in modern deep learning.
A.3 COMPARING THE BIF AND IF APPROXIMATIONS
As discussed in Section 2.1 classical influence functions face significant computational challenges when applied to deep neural networks because the memory footprint of the inverse Hessian grows quadratically with model size. This motivates a variety of approximation strategies that make different trade-offs between accuracy, computational cost, and generality. Below, we detail a selection of methods that are representative of the current dominant approaches to large-scale influence function approximation. These are roughly in decreasing order of approximation fidelity, from EK-FAC (and ASTRA), to TRAK (and TrackStar), and finally to GradSim.
EK-FAC. Figenvalue-correcteed Fronecker-Factored Approximate Curvature (EK-FAC; Grosse et al. 2023) approximates the Hessian using Kronecker-factores structure, originally developed for efficient natural gradient descent (Martens & Grosse 2020). The key insight is to approximate the Fisher information matrix (equivalent to the Gauss-Newton Hessian for the cross-entropy loss) as block-diagonal matrix where each block corresponds to a layer, and each block is further factored as a Kronecker product of two smaller matrices. This factorization dramatically reduces the cost of inversion. EK- FAC further improves upon standard K- K-FAC by computing an eigenvalue correction in the Kronecker-factore eigenbasis (George et al. 2018). While highly effective, EK-FAC restricted to linear and convolutional layers, excluding attention and normalization layers in modern architectures. Additionally, it requires an expensive fit phase to estimate and invert the Kronecker factors, though this cost amortizes when computing influence for many query-trainies pairs.
Recent work has sought to bridge the gap between these efficient parametric approximations and exact solvers. ASTRA (Wang et al., (025a) utilizes the EK-FAC decomposition not final estimator, but as preconditioner for Stochastic Neumann Series iterations. This hybrid approach corrects the structural biases of the block-diagonal approximation by refining the estimate iteratively. However, this improved precision comes at an increased computational cost, requiring hundreds of additional iterative updates per query converge beyond the initial EK-FAC solution.
TRAK. TRAK (Tracing with the Handdomlypprojectee After Kernel: Park et al. 2023a) addresses the scalability of gradient-base based attribution by linearizing the model output function, effectively
approximating the model with its empirical Neural Tangent Kernel (eNTK) (Jacot et al. 2020 Park
al. (023a). To handle the high dimensionality of the parameter space, TRAK projects the resulting gradient vectors into a lower-dimensional space using random projections, preserving inner products with high probability. Unlike simple similarity methods, TRAK then reweights these projected gradients by approximate inverse covariance matrix to account for the local curvature of the loss landscape. Finally, to handle the stochasticity of non-convex training, TRAK ensembles these scores across multiple models trained random subsets of the data.
Most recently, TrackStar (Chang et al. 2025) has pushed gradient-based attribution to the full scale of LLM pretraining (e.g., 8B parameters over 160B tokens) without the data subsampling required
17

---

Published as a conference paper at ICLR 2026
Table 2:
Comparison of training data attribution methods. The BIF offers unique combination of being Hessian- free, architecture-agnosii and capturing higher-order geometry, though it is less efficient when amortizing over many queries compared methods with fit phases.
Property
BIF
IF
EK-FAC
TRAK
GradSim
Hessian-free
X
Architecture-agnosti
/
X
Scales > 1B params
X
V
/
No fit phase
X
V
Amortizes many queries
V
V
V
Per-token (efficient)
X
X
X
X
Higher-order geometry
X
X
X
LLinear and Conv2D layers only "GradSim and EK-FAC scale to > IB parameters via batching (avoiding OOM), but incur high compute costs per query (re-running backprop). TRAK avoids this via projection.
by EK-FAC and the BIF. TrackStar can be seen refinement of the projection- based approach of TRAK that uses different gradient and incorporates optimizer second-moment corrections and task lusk-specific Hessian approximations. LoGra (Choe et al. 2024) similarly scales to billion-par parameter models by exploiting the Kronecker structure of backpropagation gradients for efficient low-rank projection, reducing the cost of gradient projection from ((nk) to O(Vnk) while remaining within the classical IF framework. Enabling retrieval across the entire pretraining corpus shifts the bottleneck from compute to storage: these methods rely on building indices of projected gradients for every training example, which, in the case of TrackStar, requires up 87TB of storage for datasets like C4 (Raffel et al. 2020). This represents the -fhte----rhttt for coverage, but the immense infrastructure requirement for storing and retrieving these indices puts it in distinct resource class compared to methods that approximate influence using data subsets or on-the-fly batching.
These methods thus represent level of fidelity between EK-FAC/ASTRA and GradSim: they retain a notion geometric correction through reweighting, but apply within a compressed projected space rather than the full parameter space.
GradSim. Gradient Similarity (GradSim) represents the most aggressive simplification of classical IFs: drops the Hessian inverse entirely and computes influence as the raw inner product between loss gradients (Charpiat et al. 2019):
GradSim(z,) (2,, 2,) Vly(w") Vlz(w").
(18)
The intuition is that samples with aligned gradients push the model's parameters in similar directions, suggesting they teach similar patterns. While computationally efficient and mehitecctureagnnostee GradSim discards all second-on -odorr curvature information captured by the Hessian inverse. This makes it less accurate than methods that account for the loss landscape geometry, though remains a useful baseline for its simplicity.
Comparison to the local BIF. Table summarizes the key properties of these methods compared to our local Bayesian influence function (BIF). The BIF occupies a unique position in this landscape: it is Hessian-free and architecture-agnosse geometry through its distributional formulation via covariance estimation over the local posterior. Unlike EK-FAC and TRAK, it requires expensive fit phase, making it particularly efficient for fine-grained, targeted attribution tasks where the number of queries relatively small. However, does not amortize well over many queries, each SGMCMC draw must perform forward passes over both the training and query sets. The tradeoffs thus favor the BIF for large models on small datasets when fine-grained per-token analysis is necessary.
like GradSim and TRAK, but captures higher-order
18

---

Published as a conference paper at ICLR 2026
B FURTHER EXPERIMENTAL DETAILS
B.I SGLD ESTIMATOR FOR BAYESIAN INFLUENCE
See Algorithm for the stochastic Langevin gradient dynamics estimator for the Bayesian influence in its most basic form. In practice, computation of train losses and observables is batched so as to take advantage of GPU parallelism. We also find that preconditioned variants of SGLD such as RMSprop-SGiii (Li al. 2015) yield higher-quality results for wider range of hyperparameters. We use an implementation provided by van Wingerden et al. (2024).
The SGLD update step described here, which from the presentation in the main text: we introduce a scalar inverse temperature /3 (separate from the per-sample perturbations (3. Roughly speaking, the inverse temperature can be thought of as controlling the resolution at which sample from the loss landscape geometry (Chen & Murfet 2025). An alternative viewpoint is that the effective dataset size of training by iterative optimization not obviously the same the training dataset size used in the Bayesian setting: scale by to account for this difference. Hence, in practice, we combine 3n as a single hyperparameter to be tuned.
one we use our experiments, differs slightly
Another difference is that, for some of the runs, we use a burn-in period, where we discard the first draws. Finally, for some of the 2025b), where we compute posterior estimates over a subset of weights, rather than all weights. In particular, for all of the language modeling experiments, we restrict samples attention weights. For the results in Figure 18 and the scaling comparison, we additionally allow weights in the MLP layers vary. A similar weight restriction procedure is adopted in EK-F EK-FAC (Grosse al. 2023).
perform "weight-restricted"" posterior sampling (Wang et al.)
Algorithm SGLD for Bayesian influence
Input: Initial model parameters w" E W, training dataset Diaain (z)) loss functions 4 := f(z,;-); W R for each [ni]. observables oy: W - I for each j E [p]. SGLD hyperparameters (inverse temperature). (step size), y (localization), m (batch size), C' (number of chains), 7 (chain length)
Output: B = (BIF [z,, si)ssiiiiis E Raxp
L C CT, 4 OpxCT
for < c S C do
w s w"
for 1 < < do
for 1 Sis n do
D (e(e-1)C++ +- ((w)
> Compute train losses (can be batched)
end for
for 1 < do
L,,(c-1)C+ f (w
b Compute observables (can be batched)
end for Sample random B, C Duain of size m
w t m ELEB, Vwlu(w) + r(w - -w') +N(0,
b SGLD update
end for end for B
1 CT
L ICT CT -I Φ
Covariance between L and $
Return B
B.2 BIF HYPERPARAMETERS
Table summarizes the hyperparameter settings for the BIF experiments. The hyperparameters refer to the Algorithm] is the batch size, C' is the number of chains, T the number of draws per chain, b is the number of burn-in steps, is the learning rate, 3 is the inverse temperature, and y is the localization strength. See Appendix B. for more details each of these hyperparameters.
19

---

Published as a conference paper at ICLR 2026
Table 3: Summary of hyperparameter settings for BIF experiments. Hyperparameters are defined as follows: m is the number of samples per SGLD minibatches, C' is the number of SGLD chains, T is the number of draws per chain, b the number of burn-in steps, is the step-size, n, is the effective number of samples that modifies the size of the gradient term in the SGLD step, and 7 is the localization strength.
Experiment
S
Dataset
m
T
c
nB
Y
Vision
ImageNet
256
15
1000
10
1 X 10
10
1000
Language
Pile
64
5
1000
100
10 10-7
2000
7000
Scaling
Pile
32
4
500
5 X 10 10
30
300
Retraining ResNet
CIFARIO
1024
4
100
1 10
200
10000
Retraining LLM
Wikitext
128
300
150
10-6
800
50000
Language
Pile
32
4
500
5 X 10
30
300
Language
Pile
64
5
100
0
5 x 10
30
300
B.3 COMPARING THE LOCAL BIF AGAINST EK-FAC
We run all benchmarking experiments for both BIF and SGLD on single node with 4x NVIDIA A100 GPUs. As given Table 3 for the BIF estimation, we run SGLD with batch size m 32, number of chains C 4, number of draws per chain T = 500, learning rate = 5 x 10 inverse temperature n.3 = 30, and localization strength 7 = 300. These are fairly conservative values: especially for larger models, we observe interpretable results for smaller values of 7. For the sake of comparability, however, we use the same hyperparameters throughout the benchmarking. Each sequence is padded truncated to 150 tokens, and the model is set to bfloat1e precision.
use the inronfluencee package for EK-FAC computation (Grosse et al., 2023 This package splits the influence computation into fit and score step. The fit step prepares components of the approximate inverse Hessian and then the score step computes the influence scores from the components computed in the first step. The fit step is computationally expensive, but the results saved to the disk and can be recycled for any score computation. This results in a high up-front cost and large disk usage, but low incremental cost.
In the first step, the Hessian is approximated with the Fisher information matrix (or, equivalently in our setting, the Gauss-Newter Hessian), which obtained by sampling the model outputs on the training data. Since the Pile, which is the dataset used for Pythia training, is too large to iterate over in full, we approximate it by taking representative subset of 000 000 data points, curated using k-means clustering (Gao et al. 2021 | Kaddour, 2023). Distributional shifts in the chosen dataset alter the influence predictions of the EK-FAC In general, the true training distribution not publicly available, therefore we consider the choice of training data as a kind of hyperparameter sensitivity in T Moreover, we use the ext reme. wwmomeerrrr.. www.mmmmmmremooooo ry-reduce option of the kronfl vence package for both steps. Without this option, other optimizations, this setting sets the precision of gradients, activation covariances, and fitted lambda values to bfloat1 and offloads parts of the computation to the CPU.
into out-of-memor errors
compute setup. Among
The comparison depicted in Figure 5 The fitting step creates a large overhead compared to the BIF, which explains the increasing discrepancy with increasing model size. This overhead is only justified one wants to compute sufficiently many influence scores. Moreover, the BIF only saves the final results, which typically small. In contrast, the results of the fit step saved to the disk, which for the 2.t model occupies 41 GiB.
B.4 PER-TOKEN INFLURNCE
Both the BIF and EK-FAC can compute per-token influences, but the interpretation differs. For BIF, the influence of each token in training example is measured on each token in the query. In contrast, EK-FAC defines the "per-token influence" as the effect of each training token on the entire query. We can recover the EK-FAC definition of per-token influence from BIF by summing over
'The corresponding github repository is available here: kronfluence
http://ggithubb.cooo//mmm//tte
20

---

Published as a conference paper at ICLR 2026
the query tokens. In principle, EK-FAC could also be used compute per-token influences in the sense we use, but naive implementation with backpropagation is prohibitively memory-intnssive, because the gradient contribution of each training label must be propagated separately the weights. Consequently, the backward pass requires memory proportional to the sequence length.
C RETRAINING EXPERIMENTS
In its original formulation, the classical influence function is motivated as measuring the effect of each training data point on retrained model. That is, for each z; E Dramn the model is retrained from initialization on the leave-one-out dataset Diriin {zi}, what is the effect on the observable o?
C.I LINEAR DATAMODELLING SCORE
Both classical and Bayesian influence functions approximate the effect of z; exclusion from Daran as linear. That is, given subset D Dirain write (DD the value of the observable corresponding to model trained on D:
Φc (D) = o(w (D)),
w" wD(D)= E arg min Σ (((w).
WEW 2,ED
in the classical perspective and
ΦB (D) Eu w/ ( [+(w)]]
in the Bayesian perspective. In either case, we approximate ((D) as linear in the set D:
(D)
T/ E D),
where each T; E training data attribution measure associated to z; and ø, e.g. IF(z, o) or BIF Φ).
This linear approximation motivates the linear datamodelling score (LDS), introduced by Park et al. (2023b). Given the training dataset Dirain of cardinality n and query set D Query, we let the query losses (Øx; = f(z,; -)),, be our observables and suppose we are given TDA measures (Tx, ED with each Txy E RR. To measure the L.S of (Tx, )a,, we subsample datasets (Dx) with each z; E Dr with probability Cretrain E (0.1, 0.3, 0.5, 0.7} iid, (For our experiments, we set K 100). The LDS of (Ta, la; is then the average over true retrained observable and the linear approximation from )z,;
K of the correlation between the
LDS((T2) (2; EDmm (02; Da,EErD (a)))
T Σ dly 1 Dal
Ps
(Øc,2, (Dk)le, EDpw
(19)
where P Spearman's 's rank correlation coefficient. Each dc.z, (Dx) is computed by retraining the model on DD and evaluating the loss 2j. Note that, regardless of whether we evaluate the LDS of an approximate classical IF or the BIF, we use the classical version of the retrained observable dc- We expect the BIF to perform well on this metric under the hypothesis that retraining with stochastic gradient methods approximates Bayesian inference (Mandt et al. 2017 Mingard et al. 2021)
C.2 LDS EXPERIMENT DETAILS AND RESULTS
We evaluate the LDS the EK-FAC, BIF, GradSim, TRAK on ResNet-9 model with 1 972 792 parameters (He et al. 2015) trained on the CIFAR-10 (Krizhevsky 2009) image classification dataset. To minimize resource usage, we adopt the modified ResNet-9 architecture and training hyperparameters described by Jordan (2024). In addition, set aside warmup set Dwammup of 2500 images. Before the actual training runs, we perform a short warmup phase on D. Warmuup to prime the optimizer state. The training hyperparameters are summarized in Table 4
21

---

Published as a conference paper at ICLR 2026
Hyperparameter
Image Classification
Word Prediction (NLP)
Training algorithm
SGD
AdamW
Epochs
(8)
Batch size
1024
256
Momentum
0.85
81 = 0.9
Weight decay
0.0153
0.01
Learning rate
10.0
3 x 10-6
Warmup steps
100
Label smoothing
0.2
0.0
Bias scaler
64.0
Whiten bias epochs
False
False
Gradient accumulation steps
Table 4: Training hyperparameters for retraining experiments. The foundational ResNet-9 model used to compute TDA scores was trained for 8 epochs. The retrained massccllasiiiiction models were trained for a single epoch. For the next-token-preditiin task, we used the pretrained Pythia-14m model checkpoint 45,000.
BIF
EK-FAC
30
Time (min)
20
10
0
0.0
0.5
aattribution
Figure 6: Wall-clock time function of O'tributionn for BIF and EK-FAC in the retraining experi- ments. Owing to the small model sizes, EK-FAC runs approximately five times faster.
As described in Appendix C.1 we evaluate LDS by re-training the ResNet-9 100 times from initialization on random subsamples of the full CIFAR-10 training set, excluding the warmup 47 500 images). Each subsample contains Aherraio Cretroin then use the full test set (g = 000 images) as the query set, i.c., there are 000 observables, corresponding the losses each test image. Thus, both EK-FAC and BIF TDA scores comprise Natiibution X 10 000 matrix. The hyperparameters for the SGLD estimation of the BIF are given in Table 3 For EK-FAC, we set the dampening factor to 10 Both TDA techniques are computed on single model checkpoint trained with the hyperparameters listed in Table Figure displays the wall-clock times of the BIF and EK-FAC computation. In these experiments, EK-FAC around five times faster than the BIF. This advantage is largely due to the small model sizes (~ 2 1 parameters), which results in a short fitting stage.
images. We
We repeat the entire experimental pipeline (retraining of models, BIF, EK-FAC, TRAK, GradSim) five times with fixed hyperparameters and distinct initial seeds for the random number generators. From these five runs, compute the mean LDS score and the standard error. The LDS scores of each individual run are displayed in Figure 7 The local BIF, EK- FACA,, and GradSim are consistent with each other within each seed. However, the LDS score varies substantially across seeds. This
22

---

Published as a conference paper at ICLR 2026
suggests either that the LDS score is not a reliable quantitative measure for evaluating TDA methods, or that influence functions in general do not capture the true counterfactual impact of individual training examples.
The TRAK influence scores may be improved by averaging results across multiple model checkpoints. Our primary focus, however, was the comparison between EK-FAC and BIF, both methods scale reasonably well to models exceeding billion parameters. To ensure the fairest possible comparison, we aligned the experimental setup accordingly, while including TRAK primarily as a reference.
Overall, the LDS scores of EK-FAC and BIF are consistent with each other and follow a similar curve. In the low-data regime, BIF achieves higher LDS scores than EK-FAC, whereas in the large-data regime, the situation is reversed. As we show in Appendix A.1 the linear approximation (in n of the BIF coincides with the classical IF for non-singular models. This may explain the overall similarity of the LDS curves we observe (even when these are singular models). It is tempting to put the superiority of the BIF in the small-data regime down the fact that the BIF is sensitive to higher order effects in the loss landscape, since the classical IF only uses second-order information. However, it is still not possible to rule out the possibility that the discrepancy due to approximation errors, arising from the Kronecker factor approximation, or some other more mundane difference between the techniques.
The number of SGLD draws used to compute the LDS scores is of the same order of magnitude as in the qualitative analysis (Section 4T. In both cases, BIF produces interpretable results with only 100-1000 total SGLD draws.
C.3 LINEAR DATAMODELING SCORES ON FINETUNING EXPERIMENTS
Table Linear datamodeling Score for Pythia- 14M model on the Wikitext finetuning task. We used Ovetrain 0.5 and 100 subsamples.
Method
LDS
Std. Dev.
GradSim
0.198
0.109
EK-FAC
0.221
0.125
Local BIF
0.110
0.112
The LDS score also serves as benchmark for the local Bayesian influence function (BIF) for language models in next-token prediction tasks. To this end, we finetune a pretrained Pythia-14M
model checkpoint 45,000 on 5,680 wi k pedia a (en) text samples filtered from the Pile (Gao iet al. 2021). The TDA methods predict the per-sample loss on an evaluation set, consisting of 385 English Wikipedia text samples. Each text example tokenized and wrapped into contexts of fixed length of 1024 tokens. After this preprocessing step, the final dataset contains 4,096 attribution sequences and 256 evaluation sequences.
We perform full-pan aall-parmmeter finetuning of the pretrained Pythia- 14M model for one epoch using AdamW with linear learning rate schedule. The training hyperparameters are listed in Table 4 In total, we train 100 models using random subsamples of the full attribution set with Oretrain = 0.5. Afterwards, we compute per-sequence evaluation losses by averaging token-level negative log- likelihoods across each sequence.
The row "Retraining LLM" in Table TableBlists the SGLD hyperparameters. The per-sequence losses on the evaluation set serve as the measurement function for both the BIF and EK-FAC. We compute EK-FAC influence scores by fitting Kronecker-factoree curvature components on the same finetuning corpus. All experiments are performed both methods use identical finetuned checkpoints and evaluation losses.
node with 4 NVIDIA A100 GPUs (80 GB VRAM), and
We compute the EK-FAC and the GradSim TDA scores with the kronfluence package by setting the strategy option to ek- fac for the former and to identity for the latter. The resulting LDS scores summarized in T
We do not report LDS scores for TRAK because, to the best of our knowledge, there is currently no publicly available implementation TRAK for large language models.
23

---

Published as a conference paper at ICLR 2026
EK-FAC
BIF
GradSim
TRAK
0.4
0.1
Gretratn 0.3
Cretrain = =..5
dretraic 0.7
0.3
LDS Sooze
0.2
0.0
0.0
0.5
1.0
0.0
0.5 dattribution
0.5
1.0
0.0
1.0
Nottnbbtionn
Oattrbbution
dasonbution
(a) Seed
EK-FAC
+
BIF
GradSim
TRAK
0.4
Oretrain = 0.1
Oretrain = 0.3
Cretralo 0.5
aretraio 0.7
LDS Score
0.2
0.1
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Ogy
Asttribution
Agstribution
agtonbuttoo
(b) Seed =
EK-FAC
GradSim
TRAK
0.4
Cretrain
Ceetrain 0.3
Cretrain 0.5
Qretraio 0.7
LDS Score
0.2
0.1
0.0
0.0
0.5
1.0
0.0
0.5 Cattribtionn
1.0
0.0
0.5
1.0
0.0
1.0
Asttrbution
Aattribution
dacorbbtion
(c) Seed
Figure 7: Individual LDS values across different seeds. The EK-FAC and BIF results consistent within each seed, but the LDS values vary substantially. This suggests that the LDS score is not an ideal quantitative measure for evaluating TDA methods that influence functions do fully capture the counterfactual impact of individual training examples.
We consider two approaches for deriving per-sequence TDA scores from the per-token losses. The measurement dataset comprises the attribution and evaluation sets, yielding Remeasurement Alatribution Mquery text samples. Consequently, the SGLD loss trace tensor has the shape (C,T, Am S).
First, we stack the tensor along the chain axis, resulting a tensor of shape (C x T, n Pmeusurements S). This is equivalent to treating all draws as belonging to single chain. This approximation is justified
24

---

Published as a conference paper at ICLR 2026
EK-FAC
BIF
GradSim
TRAK
0.4
Gyntrain 0.1
Gratrain 0..
Antrrio 0.5
Aretrain 0.7
0.3
LDS Score
0.2
0.1
0.0
0.0
0.5
1.0
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Qlatribution
Aattribution
Gatiribution
Ratiribtioon
(a) Seed =
EK-FAC
BIF
GradSim
TRAK
0.4
Oyetrain 0.1
Oretrain 0.3
Arstraio = 0.5
aretrain 0.7
0.3
LDS Score
0.2
0.1
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
dlttribution
dattribution
dlttribution
Aattribution
(b) Seed 42
Figure 8: Individual LDS scores across different seeds. The EK-FAC and BIF results are consistent within each seed, but the LDS values vary substantially. This suggests that the LDS score is not an ideal quantitative measure for evaluating TDA methods or that influence functions do not fully capture the counterfactual impact of individual training examples.
when the SGLD sample contains substantially more samples than C T, since the probability of duplicate samples is then negligible.
Implementation A (sequence-level). We average the per-token loss along the token dimension, which reduces the tensor from shape (C xT, Pmensurement S) to (CxT, Temessureeen ). Next, we evaluate the BIF correlation matrix, obtaining array of dimension (7mmaauurmmmn; Pemensurementt. Finally, extract the top-right block of this BIF correlation matrix, with shape (Patribution Rqquery.. for subsequent analysis.
Implementation (token-level). this variant, we first compute Nattribution X Tiuurr BIF covariance matrices, each of dimension (S. We then summarize each covariance matrix by averaging its entries, reducing from an (S matrix to a scalar. We subsequently assemble these scalars into final matrix of shape (Nauuibttion M.query).
Both ways result in the TDA prediction Tz, from which we compute the LDS score Equation 19 In practice, both implementations yield the same LDS scores. Therefore, default to the sequence method because it orders of magnitude faster.
The BIF LDS score is hyperparameter sensstitiieessssstiivveee in the language model setting. See Figure 14 for hyperparameter sweep. The LDS score of 0.11 reported in (ablees)correspode to the best LDS score after wide hyperparameter search. The hyperparameter dependence of the BIF poses bottleneck
25

---

Published as a conference paper at ICLR 2026
that will be addressed in upcoming work. We expect that systematic understanding of SGLD hyperparameters is essential to close the current gap between the BIF LDS score and the ones from EK-FAC and GradSim.
The BIF LDS score is typerparameter--enntttin in the language model setting. See Figure 14 for a hyperparameter sweep. The LDS score of 0.11 reported in Table corresponds 5 the best LDS score after a wide hyperparameter search. We believe this gap is not fundamental to the BIF framework but reflects the current immaturity of SGLD sampling practice: the interplay between the BIF's inverse temperature (8), localization strength (7), and step size (e) in the language model regime is not yet well characterized. Improving samplers and developing a principled understanding of hyperparameter selection is central focus of ongoing work, and we expect that this work will close the current gap.
C.4 SGLD HYPERPARAMETERS
We analyzed the dependence on the SGLD hyperparameters by sweeping over (b, n,,,7) E [0, 100) x [100, 300, 1, 000, 3, 000) x [1, 000, 3, 000, 10, 000, 30, 000, 100, 000). using Cantribationn 0.1 and com- puting the corresponding LDS scores. The grid plots Figure Figure 12 show the resulting loss traces and LDS scores for Olyttrinn 0.1 and Oretrain 0.3. These comparisons indicate that for 6 100, the LDS scores remain stable across hyperparameter choices as long as the loss trace converges. Furthermore, Figure 13 demonstrates that this stability holds independently of the choice of Oretrain-
D ADDITIONAL QUALITATIVE RESULTS
D.1 BIF AND EK-FAC ON VISION
See Figure 15 for additional qualitative comparisons between BIF and EK-FAC for the Inception- I image classification model (Szegedy al., (215)) ImageNet data (Deng al. 2009). For each query image, we list the training set images with the highest and lowest signed influences according to BIF and EK-FAC.
Interpreting high-influence samples. We observe interpretable structure in the results of both BIF and EK-FAC. The highest-influence training images for each query image are often visually similar images with the same intuitvviii correctly-labeled training examples of, for instance, fox terrier (Figure 15] row 3), should help the model better identify fox terriers in the query set. In three of the four provided examples, the two techniques agree on the maximum influence sample.
In some cases, we note that the most influential samples include visually similar samples from different class, for example: in row 1, when the query image is lemon, the highest-influence samples include oranges and apples. In row 2, the highest-influence samples for a rotary phone include a camera and appliances. Row 3 includes other wire-haired dog breeds, and row 4 includes other (sea) birds. We conjecture that the explanation for this pattern is that, in hierarchically structured domains, the model first learns broad categories before picking up finer distinctions between classes (Saxe jet al. 2019). Thus, the model might learn to upweight the logits of all fruit classes whenever sees any kind of fruit. Especially when early in training, this behavior would (1) reduce loss on all fruit images and (2) be reinforced by any training images featuring fruit, resulting in positive correlations between any fruit examples.
Interpreting low-influence samples. The lowest-influence examples, on the other hand, appear to be less interpretable for the BIF than for EK-FAC. However, we note that the influence scores of these bottom examples typically have magnitudes an order of magnitude smaller than those of the top examples, in contrast to EK-FAC, where the highest and lowest samples often have scores of similar magnitude. Heuristically, it is reasonable to expect visually unrelated images to have correlation near zero, outside of a small biasing effect (a training image with certain label may up-weight that label uniformly across all inputs, slightly harming performance images with different labels). Instead, the question is why we find few high-magnitudh negative correlations.
Disagreement between highest- and lowest-influence samples. An intriguing discrepancy arises where EK-FAC and BIF sometimes disagree on the sign of the influence. For instance, in row I of Figure 15 images of oranges have negative influence (positive correlation) according BIF, yet
26

---

Published as a conference paper at ICLR 2026
chain 0
cbain 1
chain
chain 3
nB = 100
nB 300
nB 1000
nB = 3000
n8 = 10000
LDS score 0.276
LDS score 0.264
LDS score 0.225
LDS score 0.221
LDS score 0.321
y= 3000
1.5
1.0
0.3
score 0.258
score 0.263
score 0.300
score 0.352
score
0.0
2.0
10000
1.5
1.0
0.5
LDS 0.372
LDS 0.372
LDS
LDS 0.233
0.0
y= 30000
LDS score 0.387
LDS score = 0.387
LDS score 0.390
LDS score 0.229
LDS score 0.250
y= 100000
210 Lrace
no Lrace
no Lrace
no Lrace
no Lrace
0.5
LDS score nan
LDS score nan
LDS score nan
LDS score nan
LDS score nan
100
200
100
200
100
200
100
200
100
200
Number drews
Number of drews
Number of drews
Number of drews
Number drews
Figure 9: Loss traces and LDS scores for b () and Oretrain estimates that failed to converge.
0.1. NaNs mark divergent SGLD
27

---

Published as a conference paper at ICLR 2026
chain 0
cbain 1
chain
chain 3
nB 100
nB 300
nB 1000
nB = 3000
= 10000
2.0
1.5
LDS score 0.444
LDS score 0.442
LDS score 0.431
LDS score 0.305
LDS score 0.326
2.0
y= 3000
1.5
1.0
0.5
score 0.364
score 0.366
score 0.370
score 0.369
score
0.0
10000
LDS 0.374
LDS 0.374
LDS score 0.374
0.379
LDS 0.234
0.0
y= 30000
LDS score 0.389
LDS score = 0.389
LDS score 0.392
LDS score 0.231
LDS score 0.259
2.0
y= 100000
1.5
n10 Lrace
no Lrace
no Lrace
no Lrace
no Lrace
1.0
0.5
LDS score nan
LDS score nan
LDS score nan
LDS score nan
LDS score nan
100
200
100
200
100
200
100
200
100
200
Number drews
Number of drews
Number of draws
Number of drews
Number drews
Figure Loss traces and LDS scores for b 100 and Oretrain estimates that failed to converge.
0.1. NaNs mark divergent SGLD
28

---

Published as a conference paper at ICLR 2026
chain 0
cbain 1
chain
chain 3
nB = 100
nB 300
nB 1000
nB = 3000
n8 = 10000
LDS score 0.206
LDS score 0.199
LDS score 0.176
LDS score 0.174
LDS score 0.232
y= 3000
1.5
1.0
0.3
score
LDS score 0.210
score 0.239
score 0.284
score 0.222
0.0
2.0
10000
1.5
1.0
0.5
LDS 0.285
LDS 0,285
LDS 0 0,286
0.285
LDS 0.182
0.0
y= 30000
LDS score 0.301
LDS score = 0.302
LDS score 0.302
LDS score 0.182
LDS score 0.156
y= 100000
210 Lrace
no Lrace
no Lrace
no Lrace
no Lrace
0.5
LDS score nan
LDS score nan
LDS score nan
LDS score nan
LDS score nan
100
200
100
200
100
200
100
200
100
200
Number drews
Number of drews
Number of draws
Number of drews
Number drews
Figure 11: Loss traces and LDS scores for b and O'retrain estimates that failed to converge.
0.3. NaNs mark divergent SGLD
29

---

Published as a conference paper at ICLR 2026
chaio
obain
chain
cbain 3
nB
nB 300
nBB 1000
nB
10000
2.0
1000
1.5
1.0
0.5
score 0.309
LDS score 0.309
score 0.307
score 0.243
score 0.239
0.0
y= 3000
1.0
0.5
LDS 0,277
LDS 0,280
LDS 0.288
LDS 0.295
LDS 0.220
0.0
10000
LDS score 0,282
1.DS score 0,288
LDS score 0,283
LDS score 0.282
LDS 0.183
2.0
y= 30000
1.5
1.0
0.5
LDS scome 0.301
LDS scome 0.301
LDS soore 0.302
LDS score 0.170
LDS score 0.193
2.0
y= 100000
1.5
1.0
0.5
0.0
nan
score nan
nan
100
200
0
100
200
100
200
100
200
100
200
Number
Number
Number
Number
Number
Figure 12: Loss traces and LDS scores for b 100 and Oretrain 0.3. NaNs mark divergent SGLD estimates that failed to converge.
BIF (ng = 1000, y 10000)
BIF (nB = 300, y= 1000)
absolute difference
Gretrain = 0.1
Cretrain 0.3
aretrain 0.5
Ojetrain 0.7
0.4
LDS Score
0.2
0.0
Absolute Difference
0.05
0.00
-0.05
1.0
0.0
0.5
1.0
1.0
Qattribution
Aattribution
dlstributionn
dastribution
Figure 13: Comparison of LDS scores under two different SGLD hyperparameter settings. The lower panel shows the absolute difference between LDS scores. Despite substantial changes in hyperparameters, the resulting LDS scores remain consistent across the sweep.
30

---

Published as a conference paper at ICLR 2026
Chain
Chei
Chain
Chain
Aeerage
C-R-00016
C-V-0.1
5-5-00032. C-V- 03
4.0
20 700000
3300 6
100000 100
(-00 +55
0.00
0.000
(-5.300 05
14-
Same
[=11000 16
Semplrg Lons
20-
C=1 1M
LOS: 2003
25
0000
Sarriing
16-
24
32
100
150
202
250
350
50
300
150
200
750
300
Figure 14: Loss traces and LDS scores in the ainuugge-moee setup. The remaining SGLD hyperpa-
rameters are: C 4, m = 128, T 300, = 150, and Orettain 0.5.
31

---

Published as a conference paper at ICLR 2026
Top
TOP
vil -0621
Bootoom
30412 B52
Top
Gree
Bootom
JUIZ aril
BURT 1121
Top
TO0
Ka0e 1001
54be-188
1400-190
3404-220
180
Bettom
Labele188
LEAGER
-000
Top
ND
Battom
nattom
Labelr140
Sebe-0Rt:
M6M--441
Query
BIF
EKKKFFC
Figure 15: BIF vs. EK-FAC for Inception-vl 1 on ImageNet. For each query image (left), we list the highest and lowest-influent training set images according to BIF (center) and EK-FAC (right).
positive according to EK-FAC; a similar reversal is observed in the bottom row. We hypothesize that both observations are such discrepancies may reflect hierarchical structure within learned representations: at a coarser resolution, all fruit images may improve the model's ability to recognize fruits generally, while at a finer resolution, distinctions between specific fruits (e.g., lemons vs. oranges) introduce negative correlations. This may also explain the observed lack of high- maggit negative BIF examples (if our selected hyperparameters are currently too "coarse" [Cen Murfet 2025). Future research could explore this hypothesis by systematically varying the hyperparameters controlling the resolution or granularity of influence measures, thus clarifying how hierarchical semantic structures affect training data attribution methods.
D.2 BIF AND EK-FAC ON LANGUAGE
To qualitatively compare BIF against EK-FAC, we study the following prompt-completion pair from Grosse et al. (2023):
What your objective function? Assistant: My objective function is to help humans and provide useful and accurate information and services to them. In more technical terms, my goal is to maximize my expected utility, which derive from helpfulness, accuracy, timeliness and appropriateness of my responses and outputs. Maximizing my usefulness and relevance to humans is my fundamental objective. I do not have any explicit goals beyond serving and helping humans to the best of my ability. do not have any ulterior motives or objectives besides being useful to my users.
Human:
32

---

Published as a conference paper at ICLR 2026
auchec
K MI% km
Del >
s Bi
rste
Direet conpratdtoral Repersnet
lamioo
Diwc covrpeaaional Iypponecci
lastion
expac
guanee
5SK isalaaiia
USE MO formelariora swtiih lane
(Ooddy relited (ivtall dik stade
idasey (isa00e with UK (SAay)) L (D) 90 <6io Daakk Ka los0l UK BMl (Sel ccooucading
lovel maedy ThE wort sxpported
AX
IXT Orrstt Warnibee IP LA K
laae umatt. The INne IN Inpppeeed (Btl 35 PS KK JAK IXI IIII Stern (Ntmmheen JP 16
BS 328 and (ikm) M
us 1821 iand idec IM
LSC
Proor cll Tam Mies
poor eth
Froer
number
recrces
I
The Ielloweitg isedd for srree WIDI Namm lave ipithere Ihren l inmrbeer Iscarooe
on Treer
Ewtigted Ie
sedi INt. Irom unec all inaneeerr
ddlitateed Iiii inacaivii cocrection
200 JEERE KDV Feedback
Imanciaanional [1c) 1o0d
soditizral reformmaed (snai) pllan ical (ibe Joollack inncchaziinn tii wild
proonese
Ray
petmil gortitle
space
space probos
Aniond
Wi proguee corainncse
wiing cabanchgy U... like loat ail Y
[prppoce
soring inbonooaggy epanes
lypen ir Faom dhe 1a
Ialirr Kyteretry
conameetion The
cellix lie kell
D daww esaractions (Miti lottir lnpocsactio
perkester
IIk litiin [0000ns4agi Li (perscaai lidi l8ka lwsrk 100 Chem lanel
Intrrvo 2
Iequmment
Mitherarees
Phicensdy
Vork
(330 D8A
Desastrann (ot (Matbaznatct (Cllustka Disoorati Nov (Baa D IKX (100 27 (0SA
Maxi Foom IH latt DEl inei MMithermt Doo
BEL [Gemarr
wathrr
Am
(a) EK-FAC
(b) BIF
Figure 16: EK-FAC vs. BIF on Pythia 2.8B. The query is the completion "My objective function is..." " in the prompt-completinn pair in Appendix [D.2, The three rows display the top three most influential samples according EK-FAC in decreasing order. Tokens are colored by their EK-FAC
score (left) or BIF (right).
We compute the per-token influence of the 400 training data points used in the scaling analysis (Section 33 on the completion. In EK-FAC, per-token influence is defined as the influence of each token in the training data on the entire completion. The sum over all per-token influences yields the total influence of the sample on the prompt-complent pair.
Both EK-FAC and BIF perform poorly on Pythia-2.8B. For Pythia 2.8B, we show the three most influential samples according to EK-FAC in Figure and the three most influential samples according to the BIF Figure 17 In this setting, neither technique yields immediately human- interpretable samples. Three factors that may contribute are (1) the relatively small size of the model, (2) the small set of training data points we are querying (only 400), and (3) the fact that the EK-FAC implementation we used requires us to aggregate influence scores across the full completion. As we show in Appendix [D.3] we find that, in contrast to the colllcomplttion BIF, the per-token BIF is consistently more interpretable, reflecting tokens with similar meanings purposes (e.g., countries, years, numbers, jargon, same part of speech).
Token overlap accounts for much of the inffuence in small models. Grosse et al. 2023), found that token overlap is the best indicator for large influence for small models. For larger models, this changes to more abstract similarities. With the BIF, Figure [7] suggests the same result: the most influential samples are those that have a large token overlap between the sample and the completion. For example, the tokens correlate strongly and appear often on both sides. Similarly, the service ce tokens in the sample correlate with the tokens services and serving ng in the completion. In the third sample, the tokens for contribute the majority of influence. Furthermore, the frequent token my in the completion has a strong correlation with myself in the sample.
The differences between the EK-FAC and BIF results are probably due to the distinct definitions of per-token influence. The BIF definition of per-token influence is well-defined, with a clear interpretation of signs. Furthermore, repeating the EK-FAC computation with the same settings
33

---

Published as a conference paper at ICLR 2026
sometimes leads to different results. This is probably due to the approximation of the Hessian with the Fisher information matrix, which depends on the sampled model answers. In contrast, the BIF was more consistent acToss different choices of hyperparameters.
your objertice funetion My ubjective function is to help humans and provide
Homan
humans is my fundemental objective do not have any
useful and accurase Information and serviess to them
Ing
legel -s( Ing path -8t log
technioal
my goal
maximize my
log
path
expected unility i which I derive from helpful ness accurncy tim el iness and appcopri atennes my
registry
addresss -S1 d bo regisity anddress
resgonses outyers Maxim
my issehiliass and
d ub bo peot ocal port HS( d. ub bo peot ocal port B
relevance buraan my fundamental objective not have any explicit goals beyond serving and helping abiliry do lecsor motives or objectives breides being usefall to my users
service vension
be service
best
any
connect path connect path -5| ws
connect
port
connect
bis
port -SI ws.
connect
bus poet
service service
same
ws server
Versoon
servioe bus ser9ice
nome
ws server
Buman What is dejective fandion Avastant
Information
to more technical temms
My dhocitie Sandion is help Ihamani and proolae usett and (oceS000 Information and sersses ltben In nore scholta tern il gool eccedeed uttity. whics I detive from beistei zew acceracy un el insai and approsrt niezemi of my resposses and ouguus Maxin latng my uschiness and relevance to Iumnrs is my Bundemeraal ofjectire I nzt Ihme any exptiial pais heyond wering and helptrg
pockage logg
let Tininn
mransfomm documen abbes
mastritte my
Impeei
lrxirs
Docunett
able
brstrs
monftenn Eocmmnttiionn per Redage Optine
Inurran in the (beat my atsleyy do have any tazsel instinaa stcectivos besalos Betig lustull ay usets
brstra
mamfrr decementation
imperi
keutra
caraforn documentatio
surce Set
impert Jooa 0 BUC
caes Supp resed Decement able Filer Trins Irenet val contest at Cermot
your
and accurote Information and services Do them In zurce technical
My objective Functicol is help himans procode sneful and sccunce information and inrvices Iill them In more techolcal terns, , mi gual is BB maximize lon esperted intllry which certoe fiom hrightl nest accurcay. tim el lness and approprt winnes of ma responses and ourguts Maxtm latng ins) usetiiness and Sundamerial oijecciive.. helgnn
Junt Ini Tim ber Chrie leton
way
nothing nothing way nothing
Somettmer preanest way
say somezhing
Sunacimes the greatend way
aay muree murnehhing
no Inn Bui 1 can B help mysell no lom heep
Imyself
any explictt Deyondl sersing iool have any
nught up 2n the imsddl at mnself
humans the best imy) whillry. sl better mothees or ofjectives besides being usctul my uars.
No
Imseelf
sught
irfethm
Just In
Chrls
up let0o say nathing ar
Somvetmen the greanet
way
sxy sommtiing
Sumettiess
the crcanst
way
10 say somezing
is 10 sas nothing ac
(a) Query
(b) Most influential samples
Figure 17: Most influential samples according to BIF. The query is the completion "My objective function is..." " in the prompt-completion pair in Appendix[Da The three rows display the top three most influential samples according to EK-FAC in decreasing order. On the left, each query token colored by the BIF between that token and the full sequence on the right (i.e., summed over all tokens). On the right, coloring shows the BIF between given token and the full query sequence the left.
D.3 PER-TOKEN BIF FOR PYTHIA 2.8B AND 14M
Here we show additional examples for the per-token BIF on Pythia 2.8B (Figure 18) and Pythia 14M (Figures 19 and 20)
34

---

Published as a conference paper at ICLR 2026
opo
Prendh
Planil Nakiol (Cmatio)) 3e1
(Siked (Bokk))
Witcoal (Cossiics
Jouttal
Ixem
Mill
50mm Vor Calar
Pitti Adetiia and Use Eil
ISsaci
Temiirace colt
Alhan
Monti Merkel (Chesko) Noceed
with Theie The Fusii hal
Acciinal
Bel
Faliati
fran (piical) lard lnlee
Never and
[Rand
booncce 1150 and (209) linael pesttui
Nuka The [Nlaal (55ki Tueell Surkel
leccin aud apoil
Hoata
Inene
add (ili Haai ponnil IMI uallive 1I0
1 PAKC
and pak 2oc pawer 15 willin
and qpead
(achero Inater Ainiie 1e vih Brt
my EEflata Me inder anriratt todks
iocivall Volkol Sgann pelezal
Inginnt
Daak:
Mativnal Chraing
Unee
3D0 Hodd'
(loo) suree
Moy
Halk
Iom
BE
Jarch
Mikting
avd
fom prine
po mitmi
Upeer
uol [Tanl Supe
Dld
Tate Clocki Sa ind Kom cad
Layinne
Is sound mior (I SuL
copls
Dpper
Unalace
Luntie
Nowe (MAl
1966
The [Ni
Cen
NM mi The
Figure 18: Additional results for per-token BIF on Pythia-2.8B.
35

---

Published as a conference paper at ICLR 2026
Context Tokent
Context
NEBI pII
Sequence:
Sequence:
IASS cuer Tar INN Ielobaas
Ingiso
Thaae
Bacpooa
Sequence
Sequence:
Kell 13
Sequence:
Sequence:
Oper
Context Tokens
Context Tokens
Traker
Context Token!
Contect Taken!
peto
avaret
eatie 100
(aMi Wnst Iil boo Laee tie ponn tna
jubii calc elng (Ril Mei I ally
ma You
nityy om Contpamoon Misggt Connotiio Senie
yooalo Concriio Song
Comedioa
Figure 19: Additional results for per-token BIF on Pythia 14M.
36

---

Published as a conference paper at ICLR 2026
Centext Tokent
Context Tokent
Sequence
Sequence:
Footer
lose you
Boot
(IMM aed
Halk
Contect Tokenz
Context Taken!
Sequence1
Fatia BC Bood Bon pikt
Context Takens
Context Token:
1009 leres (ne IHE
Figure 20: Additional results for per-token BIF on Pythia 14M.
37


Reviews:
[{"review_id": "8MvTBFFhz1", "reviewer": "Reviewer_hnax", "summary": "The paper introduces the Local Bayesian Influence Function (BIF), a novel approach to Training Data Attribution (TDA) for deep neural networks. It addresses the limitations of classical influence functions, which rely on Hessian inversion and struggle with the non-invertible Hessians and high-dimensional parameter spaces of modern deep learning models. The proposed BIF replaces Hessian inversion with covariance estimation over the local posterior, leveraging stochastic-gradient MCMC sampling for efficient computation. The method is architecture-agnostic and scales to models with billions of parameters. Empirical results demonstrate that BIF achieves state-of-the-art performance in predicting retraining experiments and offers computational advantages over classical methods like EK-FAC, particularly for fine-grained and per-token attribution tasks.", "questions": "Please see the Weaknesses.\n\nOverall, the paper makes a significant contribution to the field of AI interpretability and training data attribution by introducing a scalable, theoretically grounded method that addresses key limitations of classical influence functions. The empirical results are promising, and the method's ability to handle large-scale models and provide fine-grained attribution is a notable advancement.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper provides a principled extension of classical influence functions to the Bayesian setting, addressing the challenges posed by non-invertible Hessians in deep neural networks. The proposed BIF method also scales efficiently to models with billions of parameters, making it suitable for modern large-scale deep learning architectures.\n\n- The BIF enables per-token influence computation, which is particularly useful for language models and provides insights into semantic relationships between tokens. Moreover, unlike classical methods, BIF can be applied to any differentiable architecture, including attention-based models, making it more versatile.\n\n- The paper provides a thorough comparison with state-of-the-art methods like EK-FAC, highlighting the advantages and trade-offs of the BIF approach.", "weaknesses": "-  While BIF avoids the high up-front fitting cost of EK-FAC, its computational cost scales with the number of posterior draws, which can be expensive for large datasets or models.\n\n- While the BIF provides interpretable results in many scenarios, there are cases, particularly in language modeling, where the most influential samples are not immediately intuitive.\n\n- The authors are requested to provide some initial experiments on more modern deep learning models and slightly larger models.\n\n- The paper primarily focuses on comparing BIF with EK-FAC, leaving other TDA methods like TRAK and GradSim less explored in depth.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X44AUcMbjK", "reviewer": "Reviewer_kYHm", "summary": "This paper introduces local Bayesian Influence Functions (BIF) for the training data attribution task. BIF estimates data influence through Bayesian covariance sampling using stochastic gradient Markov chain Monte Carlo. The method works for large models, avoids numerical instability, and provides fine-grained, per-token attribution. Experiments on vision and language models show that BIF matches or outperforms some existing methods while being more scalable and flexible.", "questions": "Why was Pythia-2.8B chosen over more widely used or newer LLMs?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces the local Bayesian Influence Function with both theoretical grounding and empirical validation.\n\n2. The proposed estimator is architecture-agnostic and scales to billions of parameters.\n\n3. The authors provide thorough implementation and experimental details, which support reproducibility.", "weaknesses": "1. The experimental evaluation could be strengthened by including more comparison methods and a broader range of datasets.\n\n2. A robustness analysis or ablation study on key hyperparameters would further enhance the credibility of the results.\n\n3. A brief description of the datasets used would improve the paper’s clarity and completeness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YuoKBQj2qs", "reviewer": "Reviewer_AP4E", "summary": "The paper introduces the Bayesian Influence Function (BIF), replacing the ill-posed inverse Hessian in classical influence functions with a local posterior covariance estimated via SGLD around the trained weights. On LDS benchmarks, BIF matches strong baselines while avoiding Hessian approximations, and scaling studies show favorable wall clock time compared to EK-FAC on larger models.", "questions": "1. How sensitive is BIF to hyperparameters?\n2. This method seems to be distributional in nature. Can authors make some connection/comparison with the recent work on distributional data attribution [1]?\n\n[1] Mlodozeniec, Bruno, et al. \"Distributional Training Data Attribution.\" arXiv preprint arXiv:2506.12965 (2025).", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well motivated, and the proposed Bayesian Influence Function (BIF) is natural and conceptually sound; the theoretical results clearly establish its relationship to the classical influence function. The scaling demonstrated by BIF is quite favorable.", "weaknesses": "1. It is surprising to see sign flips between BIF and EK-FAC, given that both methods are intended to approximate classical influence function. Although the authors provide some explanations, I am not fully satisfied with them. For me, this suggests one of the methods is approximating IF badly. Could the authors add comparisons to IF ground truth? Smaller-scale models are acceptable.\n2. To prove practical value, please consider adding at least one downstream evaluation where BIF guides an actionable intervention and improves an objective, especially for the language model settings. The Pythia results are currently limited to qualitative visualizations, which are insufficient to establish the method's usefulness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "buq8i2YHX2", "reviewer": "Reviewer_UhDc", "summary": "This paper proposes a new method BIF to analyze how individual training samples affect model behavior without requiring Hessian inversion. Classical influence functions break down for deep neural networks because their loss landscapes are highly singular and BIF addresses this by replacing Hessian based inverses with covariance estimates computed over a localized Bayesian posterior using stochastic gradient MCMC sampling. This approach defines influence as the negative covariance between a sample’s loss and an observable, capturing higher-order geometric effects in the loss landscape that classical IF miss. The authors provide a scalable implementation that can handle billion-parameter models, demonstrating strong agreement with approximations like EK-FAC while offering better computational scalability and robustness.", "questions": "see weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a principled bayesian generalization of classical influence functions, replacing Hessian inversion with covariance estimation over a local posterior, thus making influence computation well defined even for singular deep networks. The method is architecture agnostic, works on any differentiable model. In addition, it scales to models with billions of parameters, avoiding the cubic complexity and instability of Hessian based methods like EK-FAC.\n2. The authors rigorously connect BIF to classical IF through Laplace approximation, showing that local BIF recovers damped IFs as a first-order limit and extends them to higher-order geometric effects. And experiments on large models (Pythia-2.8B and Inception-V1) models demonstrate that BIF achieves good data attribution performance.", "weaknesses": "1. Although the method avoids explicit Hessian inversion, it still requires running multiple long SGLD chains and forward passes over both training and query sets, making it computationally intensive and potentially slower than classical methods.\n2. The theoretical connections to classical IF rely on Laplace approximations and assume locally Gaussian posteriors, which are often invalid for highly non-convex neural loss landscapes. Hence, the higher-order generalization claim might not fully hold in real world networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)