A Bayesian Approach To Analysing Training Data Attribution In Deep Learning
Authors: Elisa Nguyen
Minjoon Seo
Seong Joon Oh
Venue: NeurIPS 2023 poster (2023)
Keywords: training data attribution
interpretability
explainability
data-driven xai

Abstract: Training data attribution (TDA) techniques find influential training data for the model's prediction on the test data of interest. They approximate the impact of down- or up-weighting a particular training sample. While conceptually useful, they are hardly applicable to deep models in practice, particularly because of their sensitivity to different model initialisation. In this paper, we introduce a Bayesian perspective on the TDA task, where the learned model is treated as a Bayesian posterior and the TDA estimates as random variables. From this novel viewpoint, we observe that the influence of an individual training sample is often overshadowed by the noise stemming from model initialisation and SGD batch composition. Based on this observation, we argue that TDA can only be reliably used for explaining deep model predictions that are consistently influenced by certain training data, independent of other noise factors. Our experiments demonstrate the rarity of such noise-independent training-test data pairs but confirm their existence. We recommend that future researchers and practitioners trust TDA estimates only in such cases. Further, we find a disagreement between ground truth and estimated TDA distributions and encourage future work to study this gap. Code is provided at https://github.com/ElisaNguyen/bayesian-tda.

A Bayesian Approach To Analysing Training Data Attribution In Deep Learning
Elisa Nguyen Tübingen Al Center
Minjoon Seo KAIST AI
Seong Joon Oh Tübingen AI Center University of Tübingen
University of Tübingen
Abstract
Training data attribution (TDA) techniques find influential training data for the model's prediction on the test data of interest. They approximate the impact of down- or up- up-weighting a particular training sample. While conceptually useful, they are hardly applicable to deep models in practice, particularly because of their sensitivity to different model initialisation. In this paper, we introduce Bayesian perspective the TDA task, where the learned model is treated as a Bayesian posterior and the TDA estimates as random variables. From this novel viewpoint, observe that the influence of individual training sample is often overshadowed by the noise stemming from model initialisation and SGD batch composition. Based argue that TDA can only be reliably used for explaining
this observation, deep model predictions that are consistently influenced by certain training data, independent of other noise factors. Our experiments demonstrate the rarity of such indie-inddeendent training-test data pairs but confirm their existence. We recommend that future researchers and practitioners trust TDA estimates only in such cases. Further, we find a disagreement between ground truth and estimated TDA distributions and encourage future work to study this gap. Code provided at
com/ http://lssaaggygnnenanae caa.a//liiallygyaaaaaasssi....i.
Introduction
Understanding how machine learning models arrive at decisions is desirable for social, legal and ethical reasons, particularly for opaque deep learning models []]. One approach to explanations is the data-centric approach of training data attribution (TDA). As the name suggests, TDA finds attributing training samples for a model decision, uncovering which part of the training data is relevant. The attribution of training sample 2, 2 another sample is usually defined as the change of model loss on z when the model is retrained without
2
T := L(=; 0,) L(=;0)
where is model trained on the entire dataset D and 0, model trained on the same set without Zj. Since the direct computation of Equation iis expensive, various TDA techniques for approximating the quantity have been proposed, such as influence functions or TracIn 4 Their approximations are often based on some form of inner product between the parameter gradients VoC(2;6) and VoL(zz;)).
Knowing how training samples attribute to a model decision provides an actionable understanding of the training data distribution, especially in cases of model error. TDA methods identify the training samples that are most relevant to an error and therefore enable users to understand why the error occurred (e.g. due domain mismatch of test and training data or wrongly labelled training data) [3]. Additionally, TDA gives them the tool to address the errors by e.g. changing the model directly through the training data. Even in non-erroneous cases, understanding the attributing training
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

data may enable users affected by model decisions to contest the decisions if the attributing training data is noisy or of low quality [5]
At the same time, TDA methods, especially influence functions 3 have been criticised for their fragility when applied to deep models [6 7 [] The main reasons are model complexity and the stochasticity of deep model training. While the former poses a challenge specifically for influence functions they rely on strong convexity assumptions, the latter is more general challenge [6] 9]. The randomness inherent to the training process does not only lead to variation in the learned model parameters but also in TDA scores, which makes them untrustworthy. Hence, K Søgaard recommend using expected TDA scores for increased stability.
We argue that solely considering the expectation is not sufficient to ensure the reliability of TDA but requires inspecting the variance, too. We introduce Bayesian perspective on the TDA task, noticing that there is deterministic mapping from dataset 'D the corresponding model for deep neural networks. The learned model depends on the initialisation and batch composition in the stochastic gradient descent (SGD) optimiser. We capture the resulting randomness via Bayesian model posterior P())D) over the parameter space [10 12]. In turn, the TDA estimate (Equation is a random variable that depends on two posteriors, (DO and ((0,))P)))
This viewpoint leads to a few insights into the practical usage and evaluation of TDA techniques. We confirm quantitatively that the ground-trume truth influence T T 2) is often dominated by the noise: / /Var(T)= > E/rl. We argue that is practically difficult to apply any TDA technique on pairs (2j. 2) whose ground-truth attributions T2z,, 2) are noisy in the first place. Likewise, any evaluation of TDA methods on such high- variance pairs would not be reliable.
Nonetheless, we are optimistic that TDA techniques are useful in practice, particularly for train-test pairs with high signal-to-noise ratios: /Var(r) < TE|T]- We observe that such pairs are rare but consistently present in multiple experiments. We recommend that researchers and practitioners confine their usage to scenarios where the signal-to-noise ratios are expected to be large enough.
Our contributions are as follows: (1) Bayesian formulation of the training data attribution (TDA) task. (2) Observation that the ground-trun TDA values are often unreliable and highly variable. the TDA tools only when the expected noise level is low. (4) Experimental analysis of the contributing factors to the variance of ground-truth TDA values. (5) Observation that the TDA estimation methods capture local changes in the model with regard to the counterfactual question of "retraining without training sample 2j", ", while LOO retraining itself results in more global change through the training procedure.
(3) Recommendation for the community
2 Background
We cover the background materials for the paper, including the concept, method, and evaluation of training data attribution (TDA) methods and Bayesian deep learning.
2.1 Training data attribution (TDA)
We introduce the TDA task, a few representative TDA methods, and existing evaluation strategies.
TDA task. Given a deep model fo parametrised by 8, a training set == {-1...., sample one is interested in the impact of training sample zj on the model's 's behaviour on the test sample z. In the TDA context, one often interested in the counterfactual change in the loss value for after leave-one-out (I.00) training, when zj is excluded from the training set (Fquation []. TDA has been considered in different use cases, such as understanding the bias in word embeddings [3). fact tracing in language model outputs [14] and measuring the robustness of model predictions [5]
............ 2N }. and test
TDA methods. The conceptually most straightforward way to compute the difference due to LOO
training (Equation IT is to compute it directly. However, this is computationally expensive, as it involves the learning algorithm for obtaining 8 for every j. This gives rise to various TDA techniques that find approximate estimates r'(2,, ) of 1.O.. prominent example of such approximation the influence function (IF) method 3 based on [15]. Under strong smoothness assumptions, they

---

approximate Equation I by:
(2)
where VoL(=;0) 0) and (z]( 0) refer to the parameter gradients of fo for = and 2j respectively. Recognising the difficulty of scaling up the inverse Hessian computation H₂ and the high dimen- sionality of operations in Equation 2 subsequent papers have proposed further approximations to speed up the computation [16 17 Charpiat et al. [8]] have analysed the influence of 2j on by dropping the need to compute the Hessian and formulating influence as the loss change when an
additional training step (ATS) on 2j is taken:
T(2), == (():0+)) - ((2;
(3)
where 4+ is a learned model parameter with DD and an additional step on žj. They propose two approximations:
Grad-Dot (GD): 22), == vol(zz;0) VoL((;())
(4)
(2,, = Vol(z;;0) T Vo
Grad-Cos (GC):
(5)
[V)[(zz;0) [v
This method is closely linked to TracIn 4 which computes the Grad-Dot not just at the end of the training, but averages the regular Grad- Dot similarities throughout the model training iterations. We note later in our amalysis that within our Bayesian treatment of TDA, the TracIn method coincides conceptually with the Grad-Do crad-Dot method. In our analysis, we study the sensitivity of LOO and the above TDA methods against noise.
TDA evaluation. The primal aim of TDA methods is measure how well they approximate the ground-truth LOO values. This is often done by measuring the correlation between the estimates from each TDA method and the ground-truth LOO values (Equation ! (Pearson) correlation or a rank (Spearman) correlation over a small number of train-test sample pairs (zj, =) due to the computational burden of computing the actual LOO values, especially for larger models. Usually, a few samples are chosen for a comparison against LOO, e.g. Koh & Liang 3 report results for one z and Guo el al. [16] for samples z. some cases, the ground-truth I.OO obtained by computing the change in loss after training further from the learned model parameters,
7 9.. They use either a linear
e.g. [3] Some works have adopted indirect evaluation metries such as the retrieval performance of mislabelled or poisoned training data based on the TDA estimates 3 19 16 9 [7). In this work, we adopt the Pearson and Spearman correlation metrics and discuss ways to extend them when the target (LO0 from same initialisation) and estimates (TDA) are both random variables.
2.2 Bayesian deep learning.
Bayesian machine learning treats the learned model as posterior distribution over the parameter space, rather than a single point:
D) (00((()(()))))()
(6)
Bayesian ML nicely captures the intuition that the mapping from training set D to the learned model P(0[D) is not a deterministic mapping. especially for non-convex models like deep neural networks (DNNs). Depending on the initialisation, among other factors, DNN training almost always learns vastly different parameters.
The estimation of the true posterior is indeed difficult for complex models like DNNs. The field of Bayesian deep learning is dedicated to the interpretation of certain random elements in DNN training as sources of randomness for the approximated Bayesian posteriors. For example, if Dropout (20 is used for training a model, it may be used at test time to let users sample from the posterior distribution p(0 D)) (2I]. More generally used components like stochastic gradient descent (SGD) have also been interpreted as sources of randomness. The random walk induced by SGD iterations in the parameter space can be viewed distribution, after a slight modification of the optimisation algorithm (Stochastic Gradient Langevin Dynamics (10). Similarly, the last few iterations of the vanilla SGD iterations may also be treated as samples from the posterior, resulting in more widely applicable Bayesian methods like Stochastic Weight Averaging (SWA) [22 22 Finally, the random initialisation of DNNs has also been exploited for modelling posterior randomness; training multiple versions of the same model with different initial parameters may be interpreted as samples from the posterior [T.. We show the next section how the Bayesian viewpoint will help us model the sources of stochasticity for TDA estimates.
Markov Chain Monte-Carlo sampler from the posterior
3

---

Difference in loss (LOO)
D
Low loss
variable.
p)
Original
+(s Cle.P 4te.
C(e;
D
;
High
Counterfactual
Parameter space
C(2:
Datasets
Posterior distributions
Loss values
TDA value
Figure 1: A Bayesian interpretation of training data attribution (TDA).
A Bayesian perspective on training data attribution
Training data attribution (TDA) r(2j, 2) is defined the attribution of one training sample 2, to another sample 2 in terms of how a target metric like the loss sample [(z, 0) changes when the model is trained without 2j (Equation D We note here that according to the definition, interested in the impact of the change in the dataser from D to Dij, rather than the change in the model parameter. From Bayesian perspective, change in the training dataset leads shift in the posterior distribution, p(0 - p(0, (Dij), leading to the definition of TDA as random variable:
(7) where ~ p(0(D) and 0, ~ p(oy) This interpretation is more natural, given the non- uniqueness of the mapping from training dataset D to the optimal model parameter 0 for general, non-convex models like DNNs. Alternatively, one could treat the model built from D as a fixed variable rather than posterior TDA is applied to specific model in practice. The change of dataset from 'D to D.j however still introduces ambiguity in O.j. which is captured in the Bayesian posterior p(O,j D.j). In this study, we use the probabilistic formulation TDA in Equation
T(ZJ, 2 D) == [(2; - [(2; D) = [(2; ( L(2;
Sampling TDA values. One could plug in various Bayesian DL techniques ($2.2) to compute samples of D() which can be used get the samples of 7(z,. 2). In our work, we use the Stochastic Weight Averaging (SWA) 12 22 and Deep Ensemble (DE) [1] which are applicable to a wide class of deep models. More specifically, we obtain T samples ((1), ((1),.... ...,...,,.......
(000),...00)))) ((()) ~ p(0 either by taking the last T' model checkpoints of the SGD iterations (SWA) or by taking the last model checkpoints from 7' different model initialisations (DE). The same is done for the counterfactual posterior ( ,.... ( p(o,))Di)) This results in mixture-of-
posterior, where DE samples correspond to centroids of the distribution. Our sampling approach is thus version of stratified sampling, where the number of samples T from centroid is fixed and sampled IID.
Statistical analysis TDA.
The simplest statistics for the TDA T(z,, 2) are the mean and variance:
[)) = T Σ ((: (()) ((2;0(0)
(8)
Var(T 2)]
72 Σ (c(x) L(2; ((x:)))) (150))~E(((aa ()))
1
Our main interest lies in whether the influence of the training data 2; on the test data is statistically significant and not dominated by the inherent noise of deep model training. For this purpose, we design a Student t-test 23 for quantifying the statistical significance. Our null and alternative hypotheses are:
Hp μ ===00 =0 0
H1 # O.
(10)
We consider the test statistic based sample mean and variance:
[2)]
(11)
Vars [(2 T
Vars refers to the sample variance where denominator Equation 9 is T instead. We report the significance of the absolute TDA 1r(=j. =)l for every train-test pair (zj, =) by computing the p-value corresponding to the t-test statistie, The greater the p- p-value is, the greater the dominance of noise is for the TDA estimate.

---

0
8(2)
8(3)
80
0 (()
1>
Random batch composition
Original training
(
Random intiialiiaaion
((2)
6 1
8g
D
0 p
((4)
0 (())
Counterfactual training
g())
((3)
Last few checkpoints
0
((2)
((3)
80
(() ((2) g
(3)
(1)
(1) M
Posterior from DE- Init
Posterior from DE- Batch
Posterior from SWA
Figure 2: Sources of randomness for Bayesian posteriors. In each case, the training starts from initialisation dp. Depending whether 2; is included in the training data, has either samples from the original posterior D() or from the counterfactual posterior p(A ( [(pa)) ]. For deep ensemble [1]] the randomness stems either from random initialisation (DE-Init) or from SGD batch composition (DE- DDE-Bhtch). For stochastic weight averaging (SWA) 12 22
last few checkpoints of the training are treated as posterior samples.
TDA methods likewise estimate random quantities. Approximate TDA methods like influence functions (IF), Grad-Dot, and Grad-Cos ($2.1) also predict random quantities r2( For example,
IF predicts T(2,. 2) R (2(2) 2) == V 1 0) V (), where one may sample from the posterior p(0 D). We that IF suffers theoretical issues in its application deep models, as convexity assumptions are not met. In practice, estimation algorithms make use of a damping term ensure the positive definiteness of the inverse Hessian. Through Bayesian lens, the damping term could be seen as an isotropic Gaussian prior centred at the origin. Similar statistical analyses on the TDA estimations can performed as above, including the mean and variance computations and statistical testing for the significance of influence.
Evaluating TDA as a random variable. Previously, the LOO-based TDA values T(2j. 2) and the estimates from various approximate TDA methods (2). 2) compared via correlation measures like Pearson or Spearman. Our treatment those quantities as 1-D random variables poses a novel challenge for evaluation because there exists inborn notion of ordering among 1-D random variables. We address the challenge by examining the approximation ability of TDA methods for both the first and second moments of the true TDA values T(2,, 2). More specifically, compute the Pearson and Spearman correlation for both the mean (Equation 8) and variance (Equation between the ground-truth 7 TZz,, z) and estimated TDA r'Zzj 2) values across multiple train-test pairs (zj, 2).
4 Experiments
We introduce our experimental settings, present analyses on factors contributing to the reliability of TDA values, compare TDA methods, and draw suggestions on the evaluation practice of TDA.
4.1
Implementation details
We illustrate the specific details of our implementation. See the Appendix for further information.
TDA methods. We study different TDA methods from Bayesian perspective. We test the methods introduced in for estimating TDA: influence functions (IF) B, Grad-Dot (GD) and Grad-Cos (GC) (18). We use the PyTorch implementation of IF from Guo et al. [66 and modify it for our models. As the ground-truth target, we consider Leave-one-out training (LOO) [ For LOO, we remove of 2j from the training set 'D by zeroing out the weight for sample zj towards the loss. Additionally, we include Charpiat et al. 's [18) notion of TDA that a training data point zj attributes more if an additional training step (ATS) on it changes the test loss more significantly.
Inducing randomness posterior p(0 $2.2 we have introduced the interpretation of various elements around model training sources of randomness for Bayesian posterior. We summarise our methods for inducing randomness Figure 2 We use the notion of the Deep Ensemble (DE) [I] to sample from the posterior. In variant of DE with the initialisation as the source of randomness (DE-Init), we train each of TDE randomly initialised parameters og*) on either D or D.y, The resulting parameter sets, (()) and 0 are treated samples from respective
5

---

Table 1: Stability of TDA estimates. We report p-values for the ground-truth TDA T(22, (LOO) and the estimated TDA values (zz; 2) (rest 4 columns). The p-values are averaged across train-test pairs (2). We use the CNN model throughout.
Data
Randomness
LO0
ATS
IF
GD
GC
SWA+DE-Init DE-IInit
0.331
0.254
0.352
0.363
0,003
MNIST3
SWA+DE- DEE-Batch
0.025
0.039
0.000
0.000
0.000
SWA +DE-Init
0.692
0.437
0.575
0.587
0.356
CIFARIO
SWA+ DE-Batch
0.487
0.296
0.484
0.517
0.236
176
1226
15c
Precuescy
60k
155k
122
154
5k
40k
60k
124
54
74
20k-
8o
80
0.8
8.
0.
piaaus
p-sllc
p-salus
p-satoe
MNIST3, SWA+DE-I MNIST3, SWA+DE-B CIFARIO, SWA+DE-T CIFARIO, SWA+DE-B
Figure 3: Stability of TDA estimates per train-test pair. Distribution of p-values for ground-truth TDA (L.OO)
for different experiments.
posteriors. We also consider the batch composition in stochastic gradient descent (SGD) as the source of randomness (DE-Batch). In this case, we train from one initial parameter dn with DE different random shuffles r((() of the training sets and D.y- This results in two sets of samples from the original and counterfactual posteriors. We increase the number of samples by taking the last TswA checkpoints as the Stochastic Weight Averaging (SWA) samples [2] 22 For Grad-Dot, this coincides with the definition of TracIn
4 as we average the dot products across checkpoints. In total,
we take T TDE x TSwA= = 10 x samples from D) and p( to estimate T 2).
Datasets D. To enable an exhaustive analysis of every train-test pair (zz, z), we define smaller datasets. We use variants of MNIST [24] limited to three classes (MNIST3), and CIFARIO [25]. For MNIST3, we sample a training set of size 150 and test set of size 900, i.e. 135,000 train-test pairs. For CIFAR1O, we define the training and test set at size 500, i.e. 250,000 train-test pairs.
Models. We consider two types of image classifiers, visual transformers (VIT, 26 and convo- lutional neural networks [27]. where we primarily study a two-layer (CNN2-L). We also include a three-layer version (CNN3-L) to study the factor of model complexity. For ViT variants, instead of full finetuning, we use LoRA adapter layers (28) to minimise the number of parameters being tuned. The number of trainable parameters of ViT+LoRA (597,514) is comparable to CNN3-L (620,362).
4.2 Reliability of TDA evaluation
We assess the reliability of TDA evaluation by measuring the degrees of noise in both the ground-tree truth TDA (LOO) T(z,, 2) and the estimated TDA r'(2,,), 2). The noise level is measured with the p-value of the Student-t hypothesis testing to determine if the absolute TDA values are significantly greater than the sample noise )
We report the results in Table Generally, we observe many TDA measurements, ground-truth and estimations likewise, are unstable with non-significant p-values (> 0.05). In particular, even the ground-truth LOO shows p-values of 1 on MNIST3 and 0.692 for CIFARIO (SWA+DE-Ini)). In these cases, the noise effectively dominates the signal and any evaluation that does not consider the variance in the posterior (D) is likely to be misleading. This confirms the reports in 9 that TDA values are sensitive to model initialisation.
TDA methods often show similar levels of instability. For example, the IF attains p-values 0.352 and 0.575 on MNIST3 and CIFARIO, respectively, roughly matching the LOO case. Grad-Cos an exception: it attains lower p-values than the other TDA methods (0.003 and 0.356 for MNIST3 and CIFAR1O, respectively). We interpret this as an overconfident TDA estimation. Practitioners shall be wary of using TDA methods that are unreasonably stable when the ground-truth TDA itself is not.

---

L00
8 0.5
0.4
0.3
0.3
30
60
90
120
150
160
100
200
300
400
500
600
[P]
[P]
CNN2-L, MNIST3
CNN2-L, CIFARIO
Figure 4: Impact of training data size. Mean p-vauues of TDA methods with randomness induced by SWA+DE- Init.
Table 2: Impact model complexity. Mcan p-values the ground-truth TDA T(2,, 2) (LOO) and estimated
TDA values T' (25, 2) (the other columns) with randomness induced by SWA+DE-Batch for MNIST3 and DE-Batch for CIFARIO.
Model
Data
LOO
ATS
IF
GD
GC
CNN2-L
MNIST3
0.025
0.039
0.000
0.000
0.000
CNN3-L
MNIST3
0.370
0.368
0,464
0.470
0,005
ViT+LORA
MNIST3
0.786
0.573
0.369
0.365
0.093
CNN2-L
CIFARIO
0.623
0.374
0.535
0.534
0.314
CNN3-L
CIFARIO
0.687
0.432
0.579
0.581
0.365
ViT+LoRA
CIFARIO
0.777
0.766
0.686
0.686
0.522
4.3 Factors influencing the variability of TDA
Based on the observation $4.2 that TDA values are often dominated by noise, we delve into the factors that lead to the instability of data attributions. We inspect the contribution of model initialisation, training set size and model complexity.
Source of randomness. From Bayesian ML perspective, the stochasticity of TDA stems from the inherent uncertainty of the learned model posterior p(] We consider two sources of randomness, model initialisation (DE-Init) and SGD batch composition (DE-Batch). Results are reported in Table II For MNIST3 and CIFAR10, 10, we observe that DE-Batch introduces lower levels of noise in the TDA estimates (lower p-values). Particularly on MNIST3, both LOO and other TDA methods result in statistically significant p-values (< 0.005). This implies that almost every training data 2j is influencing every test data consistently across various batch compositions. We conclude that the greater source of variations for the TDA estimates is the model initialisation.
Training set size. We study how training set size source of noise (cf. Figure 4). We train the CNN2-L with different-size datasets of MNIST3 and CIFAR10, where vary the number of samples per class. Batches are composed differently depending on the dataset size, meaning that parameter updates are made after processing different data. In addition, complete MNIST dataset and use a subset of the training data for our experiments (cf. Appendix [B.4). The results show tendency for high variation in TDA scores with larger datasets first after which decrease in variation observed. The initial increase makes sense as the number combinations for batch composition increases with dataset size. As the batching is initialised randomly during training, batches are likely to be composed of different data for larger datasets. This leads to variation in the learned model parameters, in turn affecting the reliability of TDA. At the point of decrease, the TDA scores are rather small for all train-test pairs. The attribution of individual training samples to a model prediction is overall small in larger datasets, which leads a decrease variance.
train CNN2-L on the
Model complexity. We study how model complexity is linked to the reliability of TDA estimates. See Table We observe that, compared to the CNN models, large ViT model trained with LoRA results in dramatically greater p-values. For example, for L.O on MNIST3, the p-value increases from 0.025 (CNN2-L) to 0.786. A similar trend is observed for other TDA methods. less dramatic increase in p-values also be observed with the addition of another layer to the CNN (i.e. from
7

---

LDD
1.00
4.000
-0.03
0.04
-0
L00
1.00
0:32
0.66
0.27
0.00
L00
1.00
0.11
-0.02
-0.00
0.01
Pearson F
ATS
0.00
1.00
0.32
0.31
0.37
ATS
0.32
1.00
0.16
0.17
0.04
AT5
0.11
1.00
0.73
0.72
0.42
IF..0.03
0.32
1.00
.0.99
-3.10
IF
0.26
0.16
1.00
0.96
0:03
1F.00.02
0.73
1.00
+0.99
+ 35
0,04
100
69
0.00
0.72
1,90
0.35
0.03
1.00
0.42
0.35
0.35
1.00
8
100
A15
8
100
L0D
1.00
0.01
0.05
-0.05
0.07
L00
1.00
75
0.70
0.70
-0.05
100
1.00
0.12
0.02
00.05
0.03
Spearman's 's p
AT5
0.01
1,00
0.33
-093
-0.84
AT5
0.75
1.00
0.69
0.70
0.08
AT5
0.12
1.00
0.70
0.70
+0
IF
0.05
0.93
1.00
1.00
0.80
0.70
0.69
1.00
1.00
0.21
0.02
0.70
1:00
1.00
75
300 05
79
6D
20
GD
0.
0.18
1:00
0:75
1.00
&
L00
Mcan f
Standard deviation 5
p-valuc
Figure 5: Correlation TDA methods. Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean á. TDA standard deviation ô, and TDA p-values. All results are based on the setting: CNN2-L, MNIST3, SWA+DE-Init
CNN2-L to CNN3-L). This implies that the reliability of TDA estimates decreases with increasing model complexity. While we limit the number of trainable parameters in our VIT by using LoRA to be comparable to CNN3-L. the p-values computed from TDA estimates are significantly larger. Larger models exhibit larger parameter space that noise stemming from model initialisation or batch composition is amplified. While we fix the model initialisation and dataset size, the batch composition still varies across the model parameters f sampled from the posterior p(0) D) per model. As both the CNNs and ViT are trained with the same sampled batch compositions, we attribute the strong increase of p- value to the model complexity.
(Dis) agreemett between TDA methods
We test the reliability of different TDA methods. Ideally, all methods approximate the ground-truth TDA (LOO). Yet the results suggest that there are substantial differences among the methods. For example, Grad-Cos is much more stable than all others. Hence, we study TDA methods with respect to both their correlation with LOO and among each other using Pearson and Spearman correlation of mean and variance of the TDA distributions, as proposed in $
Figure 5 shows the correlation matrices for one experiment (all experimental results in the Appendix). The results show that viewing TDA scores distributions gives insights into the reliability of TDA methods: None of the tested TDA method's expected values f correlates with LOO. This implies that none of the TDA methods is good approximation when the random factors are considered. The poor correlation p-values of LOO indicates a disagreement the train-test pairs considered low noise. We conclude that none of the tested methods reliably capture ground-truth TDA distributions.
Interestingly, we notice stronger correlation between other methods particularly when looking at the correlations of p-values. We identify two groups based on positive correlation, i.e. ATS with IF and GD with GC. Among the two groups, there is negative correlation which indicates that methods interpret the sign of the attribution differently. Between IF, GD and GC this makes sense, as there is negative sign in the definition IF (Equation (2) which not present in GD and GC, Considering absolute correlation, IF and GD are strongly correlated which shows that the dot product is valid alternative for IF as they produce similar score distributions. The correlation between GD and GC indicates that the normalisation of gradients does have strong impact the estimated TDA.
IF, GD and GC correlate considerably with ATS, which measures how the loss model on changes after doing one additional training step zj. Practically, ATS represents the gradient update after zj. which is the same as the gradient z, itself. Therefore, it makes sense that the gradient- basedd approximation methods are close to ATS. We recognise difference in the scope LOO and ATS

---

address. LOO looks at TDA globally and encapsulates the whole training, whereas ATS considers local scope with a small model change. As IF, GD and GC correlate with ATS, we observe that they also correspond to a local change in the model, which underlines and extends the argument of [ There is a gap between LOO and IF, and more generally between the global and local view on TDA.
The TDA variance ē is noticeably well-correlatd for TDA estimators and LOO, except for GC. This implies the existence of a consistent ranking of train-test pairs with stable attribution relationships. In particular, stable train-test pairs predicted by LOO are also likely to be stable pairs for TDA methods like ATS, IF, and GD. This motivates our final analysis and recommendation for evaluation in $4.5
4.5 Considerations on TDA evaluation from a Bayesian perspective
Our analysis shows that both TDA estimates and ground-truth TDA values are affected by the noise stemming from the stochastic nature of deep model training. Hence, the practice of comparing against such a ground truth is destined to result in fragile estimates. We propose to treat TDA estimates as random variables which allows us look at the evaluation from Bayesian perspective: The comparison of TDA estimates against target TDA values is a comparison of two random variables. Since it is impossible to get rid of noise, it is better compare distributions rather than point estimates. This provides an understanding of how well methods approximate the ground-truth distribution.
We observe that p- p-values vary between individual train-test sample pairs (zj, 2); not all TDA estimates are equally affected by stochasticity. Interestingly, the presence of low-noise pairs is consistent across the majority of our experiments (ef. Figure B), with varying sizes of the low-noise fraction. We find that fixing model initialisation and a small dataset size gives rise to a larger number of low-noise pairs. We propose to focus on such low-noise pairs in TDA evaluation as their estimates are low in variance, leading to a more reliable evaluation. Identifying such pairs requires an analysis similar to this work: treating TDA values as distributions and sampling multiple times from the posterior to get an estimate of the noise. It is crucial to find low-noise pairs to base evaluations on and understand when TDA is applicable. If no low-variance pairs exist, TDA cannot be used.
5 Related work
We study the reliability of TDA methods and add to the existing body of work the fragility of TDA methods. Previous studies focused primarily on IF [29 6 & 9 7). We extend the analysis by additionally studying other TDA methods. While IFs are theoretically grounded in robust statistics 115.. they are based on two assumptions which are not always fulfilled in the context of deep learning: "wicc-diiferrnntaailiy and strict convexity of the loss (3 Zhang & Zhang [29 and Basu el al. 6 point to the fragility of the influence scores due to the non-convexity of deep learning. Particularly increasing model size is connected increased model curvature, which means that influence estimates are more fragile with larger models. They find that strong regularisation is needed improve estimation quality. Our experiments verify the observation that fragility increases with model size, which we observe across methods. We add that sources of randomness in the training process attribute to the fragility of TDA methods with increasing model size. Furthermore, related work found that the size of the training set contributes to the fragility of influence estimates. The attribution of one sample in large training set is marginal so both influence estimates and ground-truth influence scores (i.e., from retraining the model) are noisy 6 & 99. Through Bayesian lens, we connect the increased fragility with increasing dataset size to batch composition as well. Not only is the attribution of a single sample in a large dataset marginal 66 but batches have vastly different compositions in larger datasets, introducing noise. A recent work 77 states that influence functions in deep learning do not correspond to LOO and quantify gaps in the estimation stemming from model non-lincarity, A different approach in TDA 30 31 aims at predicting the expected model output given a set of data points, directly considering randomness stemming from model initialisation. K & Søgaard [9] recommend reporting expected TDA scores to increase estimation stability. This approach is closest to work but misses the consideration of variance in TDA estimates which we include by taking a Bayesian viewpoint.
In contrast to related work, we treat TDA values as distributions, which enables novel perspective on the TDA task for deep models. We highlight the importance of considering the variance when studying reliability.

---

Conclusion
We adopt a Bayesian perspective on the training data attribution (TDA) methods to study their reliability when applied to deep models, given the stochastic nature of deep model training. By modelling TDA scores as distributions, we find that randomness in the training process, particularly due parameter initialisation and batch composition, translates to variation in ground-truth TDA. We empirically observe that current estimation methods, such as influence functions, model local change in the model whereas the ground truth attribution considers. global model change. Therefore, TDA is subject to inherent variance, leading us to suggest to the community: (1) When proposing a novel TDA method, one should view TDA from Bayesian perspective and study the TDA estimates as distributions. (2) When using TDA, one should consider the variance to understand when the estimate can be trusted.
Limitations. We perform exhaustive analysis of TDA values and the estimates r' for all train- test pairs (zj, 2). Because of considerable computational costs, we have subsampled the datasets. In practice, datasets considerably larger. Moreover, for an additional hyperparameter search for model training, as the principal focus is on studying TDA methods. We choose gradient-based TDA methods but acknowledge that there exist many more, that we do not address. Hence, we encourage further study of TDA methods to fill these gaps and recommend investigating TDA from Bayesian perspective, particularly in the low-data regime.
choose simple tasks eliminate the need
Broader impact. This paper contributes to the field of data-driven XAI which aims at helping humans understand the inner workings of opaque models through data-centric approaches. Our work contributes understanding the reliability of TDA methods and rethinking their evaluation against noisy ground truth, which could help assess when TDA is appropriate and reliable.
Acknowledgments and Disclosure of Funding
Kay Choi has helped in designing Figures 1 and 2. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Elisa Nguyen. This work was supported by the Tübingen AI Center.
References
[1] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM Comput. Surv. 51(5), 2018. ISSN 0360-0300. doi: 10.1145/3236009. URL URL https: //doi. org 10. 1145/ 3286009
[2] Zayd Hammoudeh and Daniel Lowd. Training data influence analysis and estimation: A survey. ArXiv, abs/2212.0466 2022.
[3] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1885-1894, PMLR, 06-11 Aug 2017. URL,
mlr. press/ v70/koh17.. html
[4] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages (9920-19930, Curran Associates, Inc., 2020. URL http:/tcc///poocceiiiiiii neurips. cc/ cc/ pdf
paper_ filee/ielaae/eeele htt.:///e..eeeeeeeoeoe/////eeeea
[5] Jinghan Yang, Sarthak Jain, and Byron Wallace. How many and which training points would need to be removed to flip this prediction? In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2563- 2576, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. URI. attps://aclaat /ttp//aaaantooooy.oooooo org 2023 waaec-mmann..
10

---

[6] Samyadeep Basu, Phil Pope. and Soheil Feizi. Influence functions in deep learning are fragile. In International Conference on Learning Representations, 2021. URL httpps:tt//////ttt ltttps://openrevieeee htt/forrmmmndxxxkkWWWGoDDDDD
[7] Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If in- fluence functions the answer, then what is the question? In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neu- ral Information Processing Systems, volume 35, pages 17953-17967. Curran Associates, Inc., 2022. URL htttttt://ptcceediiiiaee http:://pooceeddggg.......iispie ccppaper.ppppp....p wille//papprr/222/f111. 3094400336bbbb233300666666333999 Copeeeeeeeeeepppeeooooooeeeeeeee Conference pdf
[8] Jacob Epifano, Ravichandran Ramachandran, Aaron J. Masino, and Ghulam Rasool. Revisiting the fragility of influence functions, Neural networks the official journal of the International Neural Network Society, 162:581-588 2023.
[9] Karthikeyan K and Anders Søgaard. Revisiting methods for finding influential examples. CoRR, abs/2111.0485 2021. URL https:://axiv..n
abs/2113 04683
[10] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-I1), pages
681-688, 2011.
[11] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[12] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, 34th Conference on Uncertainty in Artificial Intelligence 2018, UAT 2018, pages 876-885 885. Association For Uncertainty in Artificial Intelligence (AUAI), 2018
[13] Mare-Etienne Brunet, Colleen
Houlihan, Ashton Anderson, and Richard Zemel. Un- derstanding the origins of bias in word embeddings. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 803-811. 11..811...... 09-15 Jun 2019. URL wwt.:http/////m http::e:///rrocdinngg mmlr pwwppeeeeeeeee..e v99 0tr99//buueee bttp//tttttttttte html
[14] Ekin Akyurek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. Towards tracing knowledge in language models back to the training data. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2429-2446. Abu Dhabi, United Arab Emirates, December 2022, Association for Computational Linguistics.
URL https::/aclaatthlllleeeoooo
wwtp//22.......oooo fonflffinndnnnss emnlp. 180
[15] Frank R. Hampel. The influence curve and its role in robust estimation. Journal of the American Statistical Association, 69(34)):383-33,, 1974. doi: 10.1080001221459.1999 1974...........
[16) Han Guo, Nazneen Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. FastIF: Scalable influence functions for efficient model interpretation and debugging. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 10333-10350, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021.com 11221.emll--ame mainnllmmma..mm 808
m08..808. URL httlittts///ccaathhllle org/
[17] Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. Proceedings of the AAAI Conference on Artificial Intelligence, (6(8):8179-88 8186, Jun. 2022. doi: 10. 1ww0/660iiaiivaaaoa article/view/200
URL. httpss//ojj aaai. org/index..iniioooooooooooggmeie iindxxidxxx///// php/ /AAT//
[18] Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input similarity from the neural network perspective. In H. Wallach, Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URI htt.teeeppp//poooeeeini.ssss newrips.e....... wttpcccpppa.eee... cc/paper__. files/ .df

---

[19] Rajiv Khanna, Been Kim, Joydeep Ghosh, and Sanmi Koyejo. Interpreting black box predictions using fisher kernels. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3382-3390. PMIR, 2019.
[20] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15()):1929-1958, 2014.
[21] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference machine learning, pages 1050-1059. PMLR, 2016.
[22] Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning, Advances in neural information processing systems, 32, 2019.
[23] Student. The probable error of mean. Biometrika, pages 1-25, 1908.
[24] Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 2005.
[25] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[26]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL htttps://openreviee..t net. Saaeee/foouuunddd.cddddTTT
[27] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):22778782,,, 1998.
[28] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL
[29] Rui Zhang and Shihua Zhang. Rethinking influence functions of neural networks in the over- parameterized regime. Proceedings of the AAAI Conference on Artificial Intelligence. 36 (8):9082-9090, Jun. 2022. doi: 16.9//9a (36.8.206999009 URL https://ojjs.... aaai org/ index. pndex.php/Aie AAAI/ /ttp//rrrcceeeeee/283/3/
[30] Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Datamodels: Predicting predictions from training data. In Proceedings of the 39th International Conference Machine Learning, 2022.
[31] Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. Trak: Attributing model behavior at scale. International Conference on Machine Learning (ICML), 2023.
[32] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-the-art parameter-efffiient fine-tuning methods. https://github.. com/ huggingfacce/peft 2022.
[33] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-th the art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Association for Computational Linguistics. URL https: //ttps://www... aclweb. org/ anthology/ 2020. ewn.e-demos.........ooomo
[34] Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision, 2020.
12

---

A Model training details
We provide the source code at attpte//gghtu...o//////aararaaaa All experi- ments were run on single Nvidia 2080ti GPU.
A.1 Data sampling
We subsampled versions of the openly available MNIST 24 and CIFARIO 25 datasets. For
this, we first add an index which we use for randomly sampling a fixed number of images from each class. Table 3 includes the dataset sizes of the different experiments.
A.2 Model training
CNN2-L has two convolutional layers followed by two fully connected linear layers, with GeLU activation. We the Adam optimizer with learning rate of 0.001 and weight decay of 0.005. We use the cross-entropy loss and train the model for 15 epochs on MNIST3 and for 30 epochs on CIFARIO with a batch size of 32. CNN3-L has 3 convolutional layers followed by two fully connected linear layers with GeLU activation. The hyperparameters are the same as for CNN2-L. For training the VIT with LoRA, we use the peft [32] and HuggingFace transformers library [33] We start from the pretrained model checkpoint of 34 and finetune the LoRA model with the same hyperparameters as the CNN.
An overview of the predictive performance measured in accuracy on the subsampled training and test sets is provided in Table 3
Hint for reproducibility. In particular, we use CrossEntrropLLosssceeeeeee (reductiio- 'none)) 'nne"" during model training and also update this in the ViT training script podeling_vite vit.py. py. This is important for the LOO experiments, where we exclude a sample 2j from contributing to the training by zeroing the loss.
Table 3: Predictive performance at 95% CI across 10 runs (computed as 1.96*SE)
Experiment
Model
Data
Randomness
[Drain]
Drest
Accuracymain
Accuracytest
CNN2-L
MNIST3
SWA+DE-Init
30
900
0.0001
0.953+10.003
CNN2-L
MNIST3
SWA+DE-Init
60
900
0.985++0.007
0.970++0.004
CNN2-L
MNIST3
SWA+DE-Init
90
900
0.995+00003
0.939+000055
CNN2-L
MNIST3
SWA+DE-Init
120
900
0.99911.0011
0.941
CNN2-L
MNIST3
SWA+DE-Init
150
900
0.998110.004
0.970=0.003
CNN2-L
MNIST3
SWA+DE-Init
180
900
1.000 #0.000
0.98540.001
CNN2-L
CIFARI0
SWA+DE-Init
100
500
0.989+00020
0.260=0.010
CNN2-L
CIFAR10
SWA+DE-Init
200
500
1.000++0.000
0.328+0.007
CNN2-L
CIFAR10
SWA+DE-Init
300
500
1.0000000000
0.360=0.0055
CNN2-L
CIFAR1O
SWA+DE-Init
400
500
0.983+0.020
0.362+0.011
CNN2-L
CIFAR10 10
SWA+DE-Init
500
500
0.902
0.077
CNN2-L
CIFAR10
SWA+DE-Init
600
500
0.004
0.394+0.011
CNN2-L
MNIST3
SWA+DE-Batch
150
900
0.0022
0.975+0.002
CNN2-L
CIFARIO
SWA+DE-Batch
500
500
1.0010
0.36410.003
CNN3-L
MNIST3
SWA+DE-Init
150
900
0.99410.005
0.971
CNN3-L
CIFARIO
SWA+DE-Init
500
500
0.989100008
0.363+00011
ViT+LoRA
MNIST3
SWA+DE-Batch
150
900
0.945+0.008
0.935+00005
ViT+LoRA
CIFAR10
SWA+DE-Batte
500
500
0.93440.006
0.892+0.008
B Additional experimental results
We study the reliability TDA estimates and values through hypothesis test, where we report the v-vuus indication of the statistical significance of the TDA estimate. In this appendix,
13

---

101r8
0.6
0.70
0.5
D-value
0.4
p-value
0.3
0.60
0.2
0.55
0.1
0.50
50-5wich
50
0.0
2
3 Checkpoints
Checkpaints
MNIST3
CIFARIO
Figure P-values ground-truth TDA (LOO) with increasing number SWA samples (i.e., numher model checkpoints used as samples of 0) for the CNN trained on MNIST3 and CIFAR10.
provide a complete overview of the analyses of p-values and correlations between different TDA methods.
B.1 DE vs. DE+SWA
In our work, sample trained models 0 sampled from the posterior p(0 Concretely, train the model across 10 different random seeds and record checkpoints after each of the last five epochs of training. Each of these models represents sample. We test how the stability of the TDA values T22,,)) z) (LOO) behaves when we ensemble different numbers of checkpoints by investigating the mean p-values across all train-test pairs (zz, 2). Figure shows that higher numbers of samples increase stability, therefore we use all available samples in our subsequent analyses.
B.2
Correlation analysis on low-noise pairs
IMNIST31 150
LG0
1.00
0.09
0.10
-0.11
0.09
LO0
3.00
0.70
0.66
0.57
0.05
1D0
2.00
0:13
0.12
0.31
-0.04
0.93
0.13
0.35
1.00
1.00
0.12
0.36
GD
-0.31
-2.42
40.0
1.00
8.76
2.67
0.71
1.00
1.00
-0.14
6
0.11
3.35
0.86
1.00
0.09
-0.77
-0.44
0C
00.4
0.09
1.00
LG0
LG0
8+
1.0
1.00
0.12
40.11
0.30
0.14
1.02
3.00
0.30
0.11
0.20
-0.01
1D0
3.00
-3.04
40.00
+0.00
-0.00
ICIFARIOO 1CIFAR101=500 500=500
1.00
1.00
GD.
0.10
3.70
4.99
1.00
0.91
3.10
0.53
0.98
1.00
55
0.00
0.36
0.86
1.00
5.73
40.92
100
00.0
0.00
0.33
0.00
0.73
2.00
100
100
d
Mcan f
Standard dcviation
p-value
Figure 7: Correlation analysis of low-noise train-test pairs Spearman rank correlation coefficients for TDA scores with p < 0.05 of LOO retraining of the 2-yayer CNN models trained on the respective datasets.
We analyse the Spearman correlation between the TDA approximation methods and ground-truth TDA (LOO) for low-noise train test pairs (i.e. where p-value LOO scores < 0.05). The correlation matrices are shown in Figure 7 Within the subset of low-noise train-test pairs, the Spearman correlation the mean between LOO and the approximation methods is higher than when considering the whole dataset for both MNIST3 and CIFAR10. The correlation of & of the
14

---

low-noise analysis is similar to the results found in the correlation analysis of the whole dataset (cf. Figures [2 [5]. Therefore, we observe similar trends when considering only the low-noise train test pairs to the complete dataset: There is weak correlation between the TDA approximation methods and LOO, while the TDA approximation methods correlate strongly among themselves. This reflects the difference between the global change of retraining and the local approximations.
B.3 Mislabel identification
MNIST3
Prccioo of braning deta chocked
Fractio bushing data clecked
Peodann makkogs scorl
CIFARIO
taiiigg data
Deterministic TDA
Bayesian TDA
Figure 8: Fractions of mislabeled samples discovered with the deterministic definition of TDA (left, Equation probabilistic definition of TDA (right, Equation 7h.
A common way of evaluating TDA methods is through the auxiliary task of mislabel identification.
We perform an this experiment with CNN2-L trained on MNIST3 (() = 150) and CIFARIO (D 500). We follow the procedure from Koh & Liang [3]: First, random 10% of the datasets are mislabeled by the highest scoring incorrect label. We train the model using these mislabeled datasets (sample T 50 times from the posterior). Then, compute self-influence, which is the attribution of sample to itself, with each TDA method. The mislabeled dataset is ranked according to self-influence and the quantity of interest is the fraction of mislabeled samples found when inspecting the top X% of the ranked dataset. In the analysis, we inspect (1) the range of mislabeled fractions discovered if treat TDA deterministically. i.e. we compute the discovered fraction per T and report the range (cf. Figure & left); (2) the fraction of mislabeled samples discovered when we use the mean the TDA scores of our posterior samples (cf. Figure right). We find that deterministic TDA results in a large range of possible outcomes for identifying mislabeled samples. This means that it harder to reliably identify mislabels when TDA treated as point estimates.
B.4 Experiments on MNIST (IDI=60,000)
Table Mean p-values of ground-truth TDA (LOO) and estimated TDA values of with randomness included by SWA+DE-In for CNN2-L trained on the complete MNIST dataset.
LOO
ATS
GD
GC
0.761
0.362 0.464
0.475
0.247
In addition to our main experiments, we conduct a statistical test on the TDA scores obtained from CNN2-L model trained on the complete MNIST ((D) 60, 000) dataset (cf. Figure We base
the analysis on = 50 samples from the model posterior (10 random seeds x checkpoints) and small random subsample of data (100 training samples 10 test samples = 1000 train-test pairs) due to the high computational cost of retraining. We find high (p>0.05) variance ground-truth TDA well TDA estimates, in line with findings from 4.3 Training set size.
15

---

The results show that TDA estimates vary strongly for the small subset of train-test pairs (high p-values). We identify two main reasons: (1) MNIST is larger training set size than the initial sets we used. The attribution of one sample to model behaviour is likely to be marginal and unstable. (2) Low-noise samples exist, but they are in the minority. This experiment shows that LOO affected by the stochasticity of deep model training and TDA approximation methods fail to capture this.
B.5 All results: Stability of TDA values and estimates
Table presents the complete table of p-values of all tested TDA methods across all experiments of our work except for the experiments detailed in Appendices B.3 and [.4 Below, we display the histograms of p-values per experiment corresponding to each line the table captioned with the experiment ID. The histograms show that low-noise train-test pairs (z):2) 2) are present in all experiments involving the CNN model (i.e., experiments 1-8), where the number of pairs varies. Generally, we observe that there is no connection between the size of the dataset and the distribution of p-values. Furthermore, we notice that fixing the model initialisation (i.e., randomness induced by SWA+DE-Batch) increases the number of stable train-test pairs (ef. experiment 5 to 13, 11 to 14). However, the case of the ViT experiments, stable train-lest pairs are practically non-existent which shows that model complexity affects the stability of TDA.
Table 5: Complete list of mean p-values of TDA values for all experiments.
Experiment
ID
Model
Data
Randomness
Drrin
I.00
ATS
IF
GD
GC
CNN2-L
MNIST3
SWA+DE-
0.058
0.088
0.111
0.110
0.020
2
CNN2-L
MNIST3
SWA+DE-Inir
60
0.421
0.148
0.251
0.253
0.002
3
CNN2-L
MNIST3
SWA+DE-Inis
90
0.714
0.355
0.466
0.470
0.209
4
CNN2-L
MNIST3
SWA+DE-Init
120
0.675
0.346
0.469
0.472
0.218
5
CNN2-L
MNIST3
SWA+DE-Inir
150
0.331
0.254
0.352
0.363
0.003
6
CNN2-L
MNIST3
SWA+DE-Init
180
0.374
0.254
0,355
0.356
0.001
7
CNN2-L
CIFAR1O
SWA+DE- Init
100
0.665
0.424
0.607
0.608
0.352
8
CNN2-L
CIFARIO
SWA+DE-InG
200
0.552
0.397
0.450
0.452
0.399
CNN2-L
CIFARIO
SWA+DE-InI
0,543
0.389
0.456
0.456
0.313
10
CNN2-L.
CIFARIO
SWA+DE-Init
0.619
0.418
0.562
0.568
0.344
11
CNN2-L
CIFARIO
SWA+DE-Init
500
0.692
0.438
0.575
0.587
0.356
12
CNN2-L
CIFARIO
SWA+DE-Init
600
0.665
0.447
0.575
0.579
0.358
13
CNN2-L
MNIST3
SWA+DE-Batch
150
0.025
0.039
0.000
0.000
0.000
14
CNN2-L
CIFARIO
SWA+DE-Batch
0.623
0.374
0.535
0.534
0.314
15
CNN3-L
MNIST3
SWA+DE-Init
150
0.370
0.368
0.464
0.479
0.005
16
CNN3-L
CIFARIO
SWA+DE-Inir
500
0.687
0.432
0.579
0.581
0,365
17 ViT+LORA
MNIST3
SWA+DE-Batte Batch
150
0.786
0.573
0.369
0.365
0.093
18
ViT+LORA
CIFARIO
SWA+DE-Bateh
500
0.777
0.766
0.686
0.686
0.522
B.6
All results:
Correlation analysis
In the main body of this paper, we report the Pearson and Spearman correlation matrices for ex- periment 3 (CNN2-L trained on MNIST3 with 50 samples per class and randomness induced by SWA+DE-Init). This section presents the complete overview of correlations between TDA methods across all experiments in Figures paper hold across experiments.
27 We note that the observations and analyses in the main
16

---

1.20
30c
25k
15k
1sk
10c
2k
8:0
1.0
80
80
80
P-walue
p-valle
p-volus
p-volue
2
3
2.0-
144
12k
6c-
66-
1.5
1.30
54
Sk
5k-
2k
1e
1k-
8.0
1.0
3,
04
6.8
9.5
8.3
0.4
0.6
1.0
praalie
p-Woue
ploste
1.kk
14k
12k
1104
15k
3ok
54
8.0
1.0
3.5
0.4
0.4
5.0
8.3
0.2
0.5
8.
1.0
Praeice
p-wwue
p-vntue
10
11
21k
17k
17k
70c
Recuancy
1000
00c
F 12k
13k
g11k
60.
7k
8.0
0.4
9.5
1.0
8:3
0.8
0.5
85
0.2
0.4
8.
0.2
2.0
p-wa
P-
p-vallus
p-value
15
20k
30k
13k
122
1.0k
15k
16k
2k
8.
1.0
0.5 P-weuue
0.0
Prielue
Figure Distribution p-values for ground-Iruth (LOO) all experiments (IDs corresponding Table
17

---

100
1,00
0.09
0.07
0.00
0.05
100
1.00
0:33
0.41
0.41
0.02
100
1.00
0.07
-0.04
0.04
0.01
Pearson's S y
IF 1F--0.07
1.09
1.09
0.51
0.41
0.13
1.00
1.00
0.16
1F-
0.04
0:89
1.00
1.00
0.39
6D
0.07
0.64
1 00
1.00
0.61
60
0.41
0:13
1,00
0.30
60-
+3.0
0.
3.00
1.00
0.39
U00
A15
659
100
1.00
0.16
0.09
0.00
0.17
100
1.00
0.96
0.92
0:52
0.08
100
1.30
0.05
0.05
0.05
0.04
Spearman's p
40.29
1.9
4.09
0.52
0.44
1.00
1.00
0.23
1F--0.05
0.9
0.00
-0.90
GD-
0.09
0.93
1.00
1.00
0.57
60
0.62
0.44
1.00
1,00
0.13
60
0.05
9.92
3.00
1.00
0.90
120
Mcan f
Standard deviation &
p-value
Figure 10: Experiment 1 (ef. Table 5): Pearson and Spearman correlation cocfficients among ground-truth TDA and approximate TDA methods, We show correlations for TDA mean (i TDA standard deviation o, and TDA p-values.
100
1.00
10.08
-0:15
0.15
0.04
100
1.00
0.47
0.29
0.29
0.03
100
1.20
-.116
-3.12
0.12
0.08
Pearson's 8 y
IF-
0.15
0.32
1.09
1.09
0.18
IF
0F29
0.29
0.16
1.00
1.00
0.03
IF
0F32
0:86
1.00
1.00
0.34
0.35
1.00
6D
0.24
0.16
1.00
0.01
60
0.32
+1.00
0.34
100
100
100
1.00
0.10
-0.66
8.06
0:15
100
1.00
0.69
0.63
0.65
0.33
100
1.00
3.12
-3.15
0.15
0.13
Spearman's p
IF-
0.06
0.01
1.09
1.09
0.30
0.65
0.53
1.00
1.00
0.81
IF
0F.5
0:82
1.00
1.00
0.75
0.00
-1 00
1.00
6D
0.85
0.54
1.00
0.81
6D
0.35
1.00
1.00
0.75
100
415
8
Mcan f
Standard deviation ă
p-valuc
Figure 11: Experiment (ef. Table Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean A. TDA standard deviation à, and TDA p-values.
18

---

100
1,00
0.08
0.04
0.04
0.00
100
1.00
0.62
0.31
0.31
0.01
100
1.00
0.06
0.01
0.01
0.02
Pearson's S F
1F..0.54
0.36
1.09
1.09
0.17
0.31
0.38
1.00
1.00
0.06
IF-
0.01
0:46
1.00
1.00
0.39
6D
0.04
0.36
00
1.00
0.37
60
0.31
0.37
1.00
1,00
0.06
60
0.01
+0.40 4646
3.000
1.00
0.38
U00
100
1.00
0.10
0.06
0.06
0.01
100
1.50
0.81
0.67
0.57
0.07
100
1.30
0.08
0.04
0.04
0.01
Spearman's p
0.08
0.4b
1.9
4.99
0.57
3.27
1.00
1.00
0.25
0.04
0.41
+..00
0..82
6D-
0.06
0.46
0.99
1.00
0.70
60
0.67
8.77
1.00
1,00
0.36
6D
0.04
0.41
1.00
1.00
0.82
120
Mcan f
Standard deviation ē
p-value
Figure 12: Experiment 3 (ef. Table 5): Pearson and Spearman correlation cocfficients among ground-truth TDA and approximate TDA methods, We show correlations for TDA mean (i TDA standard deviation o, and TDA p-values.
100
1.00
10.55
-0.00
0.00
0.24
1.0
1.00
0:56
0.30
0.30
0.00
1.0
1.20
3.07
-3.05
0.05
0.02
Pearson's 8 y
IF-
0.00
0.43
1.09
1.09
0.18
IF
0F30.
0.30
0.36
1.00
1.00
0.05
0F.0
0.42
1.00
1.00
0.35
0.00
1.00
6D
0.30
0.36
1.00
0.05
65
11000
0.35
100
100
655
100
1.00
0.01
0.01
0.01
0.04
100
1.00
0.82
0.68
0.68
0.04
1.0
1.00
0.05
-8.02
0.02
0.03
Spearman's p
IF- 0.01
0.49
1.09
0.59
0.70
0.68
3.27
1.00
1.00
0.18
IF
0.02
0.39
1.00
1.00
0.83
40.01
99
1.00
0.70
6D
0.08
0.77
1.00
0.38
0.02
1,00
0.83
100
815
8
Mcan f
Standard deviation ò
p-valuc
Figure 13: Experiment (ef. Table Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean A. TDA standard deviation à, and TDA p-values.
19

---

L00
1.00
0.00
.0.03
0.04
-0.2
L00
1.00
0.32
0.26
0.27
0.00
L00
1.00
0.11
-.00
0.00
0.01
Pearson's S y
ATS..0.00
1:00
0.32
0.31
0.37
ATS
0.32
1.00
0.16
0.17
0.04
ATS-
0.11
1.00
0.73
0.72
0
03
26
0.0
IF-00
1.00
0.99
0.35
60
L00
100
A15
LO0
1.00
0.01
0.05
0.05
0.07
L00
1.00
75
0.70
0.70
0.05
L00
1.00
0.12
0.02
0.05
0.03
Spearman's p
ATS
0.01
1.00
0.93
..93
-0.84
AT5
0.75
1.00
0.69
0.70
0.08
ATS
0.12
1.00
0.70
0.70
75
0.05
1.00
1:00
1.00
0.75
L00
&
Mcan f
Standard deviation &
p-value
Figure 14: Experiment 5 (ef. Table 5) Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods, We show correlations for TDA mean (. TDA standard deviation o, and TDA p-values.
100
1.00
1014
-0.03
0.00
0.03
100
1.00
0.32
0.30
0.30
0.05
100
1.20
0.03
-3.30
0.30
0.12
Pearson's 8 y
IF-
0.03
0.30
1.09
1.09
0.30
IF-
0.30
0.16
1.00
0.39
0.03
IF
0F30
3.78
1.00
1.00
0.30
0.21
1.00
6D
0.30
0.16
1.00
0.01
60
0.30
-1.00
0.30
100
100
655
100
1.00
0.00
0.01
0.02
0:11
100
1.00
0.73
0.68
0.68
0.04
100
1.00
0.04
-3.08
0.08
0.14
Spearman's p
IF-
0.01
0.94
1.09
1.09
0.30
0.68
3.65
1.00
1.00
0.17
IF
0F.08
0.73
1.00
1.00
0.68
0.00
-1 00
1.00
6D
0.08
3.65
1.00
0.37
6D
+3
1,00
0/6
100
415
Mcan f
Standard deviation ò
p-valuc
Figure 15: Experiment (ef. Table Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean A. TDA standard deviation à, and TDA p-values.
20

---

100
1,00
0.02
0.08
0.08
0.11
100
1.00
0:46
0.11
0.11
0.04
100
1.00
3.00
03007
0.07
0.03
Pearson's S F
0F..0.58
0.47
1.09
1.09
0.52
0.11
0.28
1.00
1.00
0.26
0F-00.07
0.54
1.00
1.00
0.01
6D
0.08
10:46
00
1.00
0.61
60
0.31
0.28
1,00
0.26
60-
0.55
1.000
1.00
0.0
U00
A15
U50
100
1.00
0.07
0.05
0.06
0.07
100
1.50
0.57
0.30
0.30
0.03
100
1.30
3.03
0.06
0.05
0.08
Spearman's p
40.00
0.53
109
4.09
0.30
1.00
1.00
0.44
0.36
+..00
00.87
GD-
0.05
0.53
1.00
1.00
0.75
6D-00.3
6D
0.56
1.00
1.00
0.44
6D
0.05
0.56
1.00
1.00
0.87
120
Mcan f
Standard deviation &
p-value
Figure 16: Experiment 7 (ef. Table 5) Pearson and Spearman correlation cocfficients among ground-truth TDA and approximate TDA methods, We show correlations for TDA mean (i TDA standard deviation o, and TDA p-values.
100
1.00
0:12
-0.05
0.05
0.04
100
1.00
0.44
0.26
0.27
0.00
100
1.20
0.19
-3.04
0.04
0.00
Pearson's 8 y
IF-
0.05
0.64
1.09
1.09
0:68
0F20
0.26
0.50
1.00
1.00
0.39
IF
0.04
3:66
1.00
1.00
0.00
0.25
1.00
1.00
6D
0.27
0.50
1.00
0.34
69
-1.00
0.00
100
100
100
1.00
0.15
-0.04
0.00
0.02
100
1.00
0.54
0.45
0.45
0.01
100
1.00
0.19
-3.05
0.05
0.05
Spearman's p
IF-
0.04
0.68
1.09
1.09
0.31
IF-
0.45
0.71
1.00
1.00
0.47
IF
0.05
3.67
1.00
1.00
0.96
0.03
10/7
-1 1000
1.00 0.91
6D
0.45
0:71
1.00
0.47
6D
0.35
6.677
1,00
0.86
100
415
8
Mcan f
Standard deviation ò
p-valuc
Figure 17: Experiment 8 (ef. Table Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean A. TDA standard deviation à, and TDA p-values.
21

---

100
1,00
0.07
0.04
0.04
0.03
100
1.00
0.51
0.26
0.26
0.00
100
1.00
0.13
0.00
0.02
0.02
Pearson's S F
0F.40.34
3.60
1.09
1.09
0.52
0.26
0.41
1.00
1.00
0.28
IF-
1F-00.02
0:68
1.00
1.00
0.01
6D
0.04
0.60
1 00
1.00
60
0.26
0.41
1,00
0.28
60-
5.67
-3.00
1.00
0.1
60
U00
A15
U00
100
1.00
0.11
0.02
0.02
0.01
100
1.50
0.63
0.51
0.51
0.01
100
1.30
0:13
0.02
0.02
0.03
Spearman's p
40.02
1.99
4.09
0.51
3.70
1.00
1.00
0.40
10-00.02
0:20
0.00
-0.95
GD-00
0.02
0.69
1.00
1.00
0.59
60
0.51
3.70
1.00
1.00
0.40
6D
0.02
3.70
1.00
1.00
0.95
120
Mcan f
Standard deviation &
p-value
Figure 18: Experiment 9 (ef. Table 5): Pearson and Spearman correlation cocfficients among ground-truth TDA and approximate TDA methods, We show correlations for TDA mean (i TDA standard deviation o, and TDA p-values.
100
1.00
1004
-0.05
0.05
0.03
100
1.00
0:48
0.29
0.20
0.000
100
1.20
-3.04
0.01
-6.1
0.02
Pearson's 8 y
IF-
0.05
0.62
1.09
1.09
0.72
IF
0.20
0:35
1.00
1.00
0.28
IF
0.01
0:63
1.00
1.00
0.01
0.25
-1
1.00
0.72
6D
0.20
1.00
0.28
69
+1.00
0.1
100
100
100
1.00
0.01
-0.00
0.00
0.81
100
1.00
0.60
0.43
0.42
0.00
100
1.00
0.02
0.01
0.01
0.00
Spearman's p
IF- 0.00
0.63
1.09
1.09
0.35
IF-
0.43
3.61
1.00
1.00
0.46
IF
0.00
3.66
1.00
1.00
0.90
0.00
10/53
11 00
1.00
6D
0.42
3.60
1.00
0.45
GD
1,00
0.80
100
475
8
Mcan f
Standard deviation ò
p-valuc
Figure 19: Experiment (ef. Table 5 Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean A. TDA standard deviation à, and
TDA p-values.
22

---

100
1,00
0.03
0.05
0.05
0.04
100
1.00
0.50
0.23
0.23
0.00
100
1.00
0.04
-0.01
0.01
0.01
Pearson's S y
IF 1F..0.05
0.83
1.09
0.95
0.23
0.40
1.00
0.34
0.32
IF-
1F-00.01
0.62
1.00
0.99
0.00
6D
0.05
0.61
1.00
0.68
60
0.23
0:37
1,00
0.30
60-
0.01
03.01
-0.98
1.00
0.00
60
U00
A15
650
100
1.00
0.02
0.01
0.01
0.02
100
1.50
0.63
0.45
0.44
0.01
100
1.30
0.04
0.00
0.01
0.01
Spearman's p
IF-
40.01
1.99
4.95
0.45
1.00
0.98
10-00.00
0.63
-0.90
GD-
0.02
0.63
0.98
1.00
0.34
6D
0.44
0.63
0.88
1.00
0.48
6D
0.00
3.64
0.98
1.00
0.00
120
Mcan f
Standard deviation &
p-value
Figure 20: Experiment (cf. Table Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods, We show correlations for TDA mean A. TDA standard deviation o, and TDA p-values.
100
1.00
0002
-0.01
0.00
0.02
100
1.00
0.44
0.22
0.21
0.02
100
1.20
-3.00
-3.00
-0.00
0.01
Pearson's 8 y
IF-
0.01
0.64
1.09
1.09
0.73
IF
0F22.
0.22
0:38
1.00
1.00
0.34
IF
3:66
1.00
1.00
0.00
0.20
1.00
0.72
6D
0.21
0.37
1.00
0.31
69
-1.00
0.0
100
100
855
100
1.00
0.01
0.01
0.01
0.00
100
1.00
0.59
0.44
0.05
100
1.00
0.01
0.01
0.01
0.01
Spearman's p
IF- 0.01
0.68
1.09
1.09
0.57
IF-
0.44
3.65
1.00
1.00
0.35
IF
0.00
3.69
1.00
1.00
0.91
0..01
+067
-1 10
1.00 0.97
6D
0.44
3.64
1.00
0.54
GD -0.01
1,00
0.91
100
8
Mcan f
Standard deviation ò
p-valuc
Figure 21: Experiment (ef. Table 5 Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean A. TDA standard deviation à, and
TDA p-values.
23

---

100
1,00
0.04
0.04
0.04
0.03
100
1.00
0:48
0.21
0.21
0.00
100
1.00
0.07
-0.01
0.00
0.01
Pearson's S y
0.34
0.62
1.09
0.99
0.70
0.21
0.32
1.00
0.34
0.26
IF-
0F-00.01
0.63
1.00
0.36
0.00
6D
0.04
0.59
1.00
0.69
60
0.21
0.29
1,00
0.24
60-
-0.96
1.00
0.00
U00
A15
650
100
1.00
0.05
0.00
0.00
0.01
100
1.50
0.61
0.41
0.40
0.01
100
1.30
0.06
0.000
0.01
0.01
Spearman's p
0.00
1.99
0.41
1.00
0.96
0.48
18-00.00
3.68
+037
-0.90
6D-
0.00
0.64
0.98
1.00
0.55
6D
0.40
8.56
0.96
1.00
0.46
6D
0.00
3.67
0.57
1.00
0.89
120
Mcan f
Standard deviation &
p-value
Figure 22: Experiment 13 (cf. Table Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods, We show correlations for TDA mean A. TDA standard deviation o, and TDA p-values.
100
1.00
10.4
-0.04
0.04
0.03
100
1.00
0:48
0.23
0.21
0.000
1.0
1.20
+3.07
-3.00
0.00
0.01
Pearson's 8 y
IF-
0.04
0.62
1.09
0.59
0.70
IF
0.21
0:32
1.00
0.36
0.26
IF
0F.0
0:63
1.00
0.66
0.00
0.04
.0.49
1.00
6D
0.21
1.00
0.24
65
0.0
100
100
100
1.00
0.05
8.00
0.00
0.81
100
1.00
0.61
0.43
0.40
0.01
100
1.00
3.06
-3.00
0.61
0.01
Spearman's p
IF-
0.00
0.66
1.09
0.98
0.57
IF-
0.41
3.60
1.00
0.36
0.48
IF
0F.00
3:68
1.00
0.7
0.90
0.00
98
1.00
6D
0.40
8.56
1.00
0.46
GD 0.31
-397
1.00
0.99
100
415
8
Mcan f
Standard deviation ò
p-valuc
Figure 23: Experiment (ef. Table 5) Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean A. TDA standard deviation à, and
TDA p-values.
24

---

100
1,00
0.10
0.02
0.02
0.03
100
1,00
0.22
0.25
0.25
0.02
100
1.00
3.04
0.00
0.00
0.04
Pearson's S y
0.82
0.20
1.09
1.09
0.18
0.25
0.01
1.00
1.00
0.18
IF-
0.50
0:82
1.00
1.00
0.322
6D
0.02
+0.20
00
1.00
0.18
60
0.25
-3.01
1.00
1,00
0.20
60
00
5.80
13.00
1.00
0.32
U00
100
1.00
0.06
0.04
0.04
0.09
100
1.50
0.61
0.63
0:63
0.07
100
1.30
3.08
0.03
0.03
0.04
Spearman's p
0.24
1.99
4.09
0.53
1.00
1.00
0.39
0.25
0.78
00009
6D-
0.04
0.89
1.00
1.00
0.32
60
0:63
0:33
1.00
1,00
0.38
60
0.03
0.78
3.00
1.00
0.69
120
Mcan f
Standard deviation &
p-value
Figure 24: Experiment (ef. Table Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods, We show correlations for TDA mean Ai TDA standard deviation o, and TDA p-values.
100
1.00
0001
-0.02
0.02
0.02
100
1.00
0.54
0.26
0.26
0.02
100
1.20
+3.01
0.00
-0.00
0.01
Pearson's 8 y
IF-
0.02
0.60
1.09
1.09
0:56
0F20.26
0.26
0.43
1.00
1.00
0.34
IF
0.00
0.62
1.00
1.00
0.01
0.02
1.00
6D
0.26
0.42
1.00
0.M
69
+1.00
0.1
100
100
655
100
1.00
0.00
8.01
0.01
0.01
100
1.00
0.64
0.45
0.45
0.03
100
1.00
3.00
0.01
0.01
0.01
Spearman's p
IF-
0.01
0.63
1.09
1.09
0:33
IF-
0.45
3.63
1.00
1.00
0.48
IF
0.00
3.65
1.00
1.00
0.90
0..01
0062
1 100
1.0
6D
0.45
3.62
1.00
0.47
GD -0.01
1.00
0.80
100
415
8
Mcan f
Standard deviation ò
p-valuc
Figure 25: Experiment (ef. Table 5 Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean A. TDA standard deviation à, and
TDA p-values.
25

---

100
1,00
0.03
0.03
0.08
0.02
100
1.00
0.61
0.30
0.32
0.24
100
1.00
3.02
-0.03
0.03
0.03
Pearson's S y
1F..0.33
0.24
1.09
0.85
0.58
0.30
0.26
1.00
0.47
0.30
0.03
3.47
1.00
-084
-0.87
6D-
0.03
0.74
88
1.00
60
0.32
0.28
0.47
1,00
0.37
60
0.00
-0.84
1.00
0.88
U00
A15
8
100
1.00
0.02
0.02
0.02
0.02
100
1.50
0.65
0.44
0.43
0.26
100
1.30
3.02
0.02
0.02
0.01
Spearman's p
IF- 40.02
1.99
4.94
0.44
1.00
0.6
0.52
10.00..00
0.02
0.24
+0:1
-0.25
GD-
0.02
0.85
0.92
1.00
0.90
6D
0.43
0:40
0.63
1,00
0.61
60
0.02
3.74
0.81
1.00
0.89
120
Mcan f
Standard deviation ē
p-value
Figure 26: Experiment 17 (cf. Table Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods, We show correlations for TDA mean Ai TDA standard deviation o, and TDA p-values.
100
1.00
-0.01
0.00
0.00
100
1.50
0:73
0.38
0.35
0.06
100
1.20
0:39
-3.00
-0.00
0.00
Pearson's 8 y
IF-
0.01
0.15
1.09
0.35
0.27
0.18
0.18
1.00
0.20
0.18
0F.0
0.17
1.00
0.53
0.63
40.00
-0.15
1.00
6D
0.35
0.15
0.29
1.00
0.22
69
-3.17
-351
0:7h
100
100
1.00
0.42
-0.00
0.00
0.00
100
1.00
0.72
0.34
0:33
0.25
100
1.00
0.41
-3.00
0.00
6.00
Spearman's p
IF- 0.00
0.12
1.09
0.43
0.53
IF-
0.34
0.34
1.00
0.50
0.42
0F.00.0
0.15
1.00
0.54
0.64
0..01
.0.1
0.70
6D
0.33
0.33
0.50
1.00
0.64
GD
0.00
-3.15
-3.4
0.82
100
875
Mcan f
Standard deviation à
p-valuc
Figure 27: Experiment (cf. Table 5 Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean A. TDA standard deviation à, and
TDA p-values.
26


Reviews:
[{"review_id": "LZ8V4A65h6", "reviewer": "Reviewer_CbTs", "summary": "In this paper the authors investigate training data attribution (TDA) through a bayesian lens by explicitly considering the randomness in estimating the model parameters with and without a given training example.  To generate approximate bayesian posteriors on model parameters the authors use deep ensembles and SWA, then consider multiple different methods of TDA. They find that, using a t-test, TDA estimates often exhibit high variance, indicating high signal to noise ratio. This variability is often dominated by noise from model initialization and seems to increase with training set size and model complexity. They also show strong consistency between different groups of TDA methods. ", "questions": "- Is there a hypothesis for why so many more MNIST3 examples seem to have low p-values compared to CIFAR10 examples? It would be interesting to see whether these low p-value examples have some special properties that could be exploited to identify them efficiently.\n- It seems that the model complexity analysis should be considered relative to the dataset we are attempting to perform TDA on. Is there any way to quantify this value? Although a ViT model may exhibit high $\\tau$ variability on a simple dataset like CIFAR10 or MNIST3, it may behave differently on a much more complex  datasets like ImageNet.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The manuscript is well written and clear, and the problem setting important. Measuring the variability in downstream TDA estimates caused by variability in model parameters is an interesting perspective that adds to the growing body of literature aiming to understand various methods of TDA. The conclusions drawn by the authors regarding sources of randomness (initialization vs batching) seem to align with current intuitions around TDA estimates and present a new way to measure the quality of any proposed TDA method by considering those pairs for which the LOO estimates are statistically significant. ", "weaknesses": "One of the weaknesses is the small dataset sizes considered. The authors restrict datasets to 150 training examples and 900 test examples for MNIST and 500 train/test examples for CIFAR10. At these small dataset sizes, we might expect higher variability and correlation between training and test examples. For example, if the model only sees a single 3 written a particular way during training, then we might expect high variability in differently trained model’s predictions at a similar 3 in the test set. However, seeing many such 3s in a larger dataset should decrease the variability of posterior predictions. Although the conclusions drawn and trends observed at these small scales are interesting, it remains to be seen whether they hold for larger datasets. As a starting point, the authors should consider running the same experiments on a larger training set but consider only a small random subsample in their analysis. This should give a stronger signal for the behavior of the random variable $\\tau$ for larger datasets.\n\nThe other main set of experiments that would greatly strengthen the paper is the consideration of the downstream tasks for which TDA is useful, for example in mislabel identification. Since these tasks may not depend on exact LOO estimates [1], the inherent variability in TDA estimates may or may not be a relevant consideration. It would be interesting to see the extent to which the variance of $\\tau$ estimates makes these tasks difficult or even potentially ill-posed.\n\n[1] Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If influence functions are the answer, then what is the question?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and recommendation to accept our work. \n\nWe wish to address the remarks and questions raised by the reviewer one by one: \n\n### One of the weaknesses is the small dataset sizes considered.\n\nWe understand that considering greater dataset sizes would be desirable. However, the experiments are already computationally heavy when it comes to TDA evaluations that involve re-training of the whole network several times, each time leaving out a single training sample.  We have retrained the model #train $\\times$ #MC samples $=150\\times 50=7500$ times for the MNIST3 experiments and $500\\times 50=25000$ times for the CIFAR10 experiments. This surmounts to roughly 2 and 104 Nvidia 2080ti-hours per single TDA evaluation. This explains why TDA community has relied on apparently smaller-scale experiments: [a,b] have considered only 100 and 500 train-test pairs, respectively. In this work, we opt to work with smaller datasets and thorough consideration of all train-test pairs $(z_j,z)$. We believe that the key problem with previous TDA papers is the selection of certain train-test pairs to report the results on. Thanks to the exhaustiveness, we are capable of making conclusions based on the entire population of train-test pairs: we verified the prevalence of high-noise train-test pairs and the existence of very few low-noise pairs (Section 4.5, line 294). \n\n### It remains to be seen whether they hold for larger datasets. As a starting point, the authors should consider running the same experiments on a larger training set but consider only a small random subsample in their analysis. \nWe ran an additional experiment following the reviewer’s suggestion: We trained a model on the full MNIST dataset and defined a smaller subset of train-test pairs (100$\\times$10=1000) for studying TDA. Note that MNIST is not small for our experimental setting, where we compute LOO retraining. In fact, training the model once on one Nvidia 2080ti GPU cost around 7 min, which we did 100$\\times$50 posterior samples = 5000 times, resulting in roughly 583 GPU hours. We confirm the main findings in the smaller-scale experiments: the ground-truth TDA values tend to show high (p>0.05) variance in general. Experimental details in the global response, experiment 1. \n\n### The other main set of experiments that would greatly strengthen the paper is the consideration of the downstream tasks for which TDA is useful, for example in mislabel identification. \n\nFollowing the reviewer’s suggestion, we performed a mislabel identification experiment similar to [a] and [c]. We find that the inherent stochasticity of TDA leads to a large range in mislabel identification performance and that high variance in the TDA estimates degrades downstream task performance. Experimental details in the global response, experiment 2. \n\n### Is there a hypothesis for why so many more MNIST3 examples seem to have low p-values compared to CIFAR10 examples?\n\nOur intuition is: MNIST3 is an easier task to learn than CIFAR10, which could result in a less complex loss landscape where there may be global optima that are easier to find. If model posteriors are sampled from the same optimum there could be less variation in the TDA scores.\n\n### It would be interesting to see whether these low p-value examples have some special properties that could be exploited to identify them efficiently.\n\nWe agree that a study on the properties of low-noise train-test pairs would be interesting, which is also one of our main recommendations to the community. We do not have a strong hypothesis for the cause of this phenomenon yet, as the noise level seems independent of obvious features like classifier confidence. This is an interesting research question to be explored in the future. \n\n### It seems that the model complexity analysis should be considered relative to the dataset we are attempting to perform TDA on. Is there any way to quantify this value? Although a ViT model may exhibit high 𝜏 variability on a simple dataset like CIFAR10 or MNIST3, it may behave differently on a much more complex datasets like ImageNet.\n\nWe agree that it makes sense to match the model complexity to the data complexity. We applied LoRA finetuning to reduce the effective model complexity of a ViT down to match the CIFAR10 dataset. Though it would be interesting to perform our analysis on ViTs trained on ImageNet, this is computationally prohibitive - it would require $10,000\\times 50 = 500,000$ times retraining the ViT, which roughly corresponds to 2.4 GPU years. A realistic research scale at the moment is what we have presented.  \n\n[a] Understanding black-box predictions via influence functions (ICML 2017)\n\n[b] FastIF (EMNLP 2021)\n\n[c] Estimating training data influence by tracing gradient descent (NeurIPS 2020)\n"}, {"review_id": "IeIzuNnxYC", "reviewer": "Reviewer_fmRD", "summary": "This paper presents a Bayesian perspective on Training Data Attribution (TDA), a technique that identifies influential training data for model predictions. The authors propose treating the learned model as a Bayesian posterior and TDA estimates as random variables. This approach reveals that the influence of individual training data often gets overshadowed by noise from model initialization and batch composition. Consequently, they suggest using TDA only when certain training data consistently influences model predictions, despite noise factors.", "questions": "l 65: what do you mean by \"more global\"?", "limitations": "n.a.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Well written, easy to follow. Thanks!\n- Studies an important practical problem and provides a practically relevant recommendations.\n- Strong analysis of experiments", "weaknesses": "- The paper identifies difficulties with current practice and provides some recommendations, it does not discuss a solution or paths towards a solution of the TDA problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the encouraging review. \n\nIn the following, we answer the questions posed by the reviewer one by one:\n\n### [The paper] does not discuss a solution or path towards a solution of the TDA problem.\nOur focus is to identify issues with prior problem definition of the TDA task and to propose a novel problem definition that makes more sense in practice. In the process, we have shared interesting intuitions that may lead to effective solutions. For example, we have identified varying degrees of variance in different train-test pairs (Figure 3) and have recommended that future researchers study the low-noise train-test pairs, where the TDA estimates are stable (Section 4.5, line 298). Another path towards a more practical TDA technique is to study the factors affecting the inherent variance of TDA.\nWe will make this discussion more explicit in the final version of the paper. \n\n### Line 65: What do you mean by “more global”?\nWe categorise a TDA method as local vs global based on the expected counterfactual impact of “altering a training sample $z_j$”. For example, the leave-one-out (LOO) re-training introduces a global impact on the model parameters, as the model is trained for multiple iterations without the training sample $z_j$. On the other hand, taking a single additional training step on $z_j$ is considered local, as the expected impact is more restricted. \n\nWe understand that this is not a well-defined terminology. We will edit the sentence as follows:\n“(5) Observation that the TDA estimation methods capture local changes in the model with regard to the counterfactual question of “retraining without $z_j$”, while LOO retraining itself results in a more global change through the training procedure.”\nWe are happy to discuss this point and hope to find a phrasing that will improve the paper for better clarity here.\n"}, {"review_id": "ouaqI0gQRg", "reviewer": "Reviewer_fqXw", "summary": "This paper aims to examine training data attribution (TDA) methods from a Bayesian perspective, assuming the learned model parameters are samples from a posterior distribution. The paper illustrates how this perspective might affect TDA methods. It conducts experiments comparing and contrasting different TDA methods when run several times on model parameter samples from an approximate posterior. Experiments consider subsampled versions of MNIST and CIFAR10, and examine two model classes CNN and ViT+LoRA.\n", "questions": "Regarding the presentation of the statistical test:\n\nIn Equation 7 $\\tau$ is redefined as a random variable. The randomness stems from $\\theta$ conditioned on the observed dataset and the removal of j. (It might be helpful to indicate that with the notation.)\n\nIn Equation 10, it states that the hypotheses being tested are whether \\tau does or does not equal 0. This is unclear. I assume these hypotheses refer to $\\tau$’s mean, and that the t-test is intended to determine the probability that the samples of $\\tau$ could be generated by a random variable having mean equal to 0. Is this correct? \n\nIn Equation 11, what version of the T-test are you using? It would be helpful to cite your statistical method. Is the test designed to consider the pairing of the posterior (and perturbed posterior) samples? \n\nIn Equation 11, I assume that t should be indexed by j (and is also a function of the test sample z), but it’s unclear whether $t$ is computed for every pair of posterior (and perturbed posterior) samples. The left-most term of the numerator suggests this. If so, how are you aggregating these sets of values to get a single value per train-test pair?\n\nOther:\nIs it common to combine these approximate posterior sampling techniques? What are the implications of doing so?\n", "limitations": "The experiments in this paper consider two models. The paper notes limitations in the size of the datasets considered, explaining that it opted to spend computation time/budget on an exhaustive analysis of TDA values instead. However, if the objective was to comment on issues with TDA in practice, this choice (and limitation) is considerable. The results would be more convincing if the experiments had focused on only a subset of the test-train pairs, but considered larger (more realistic) dataset sizes, and a broader set of model architectures and/or sizes. \n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The topic is very interesting. \nAs a researcher and practitioner that has worked extensively with TDA methods, I have found myself thinking about (and navigating) the relationship between randomness in model parameters and TDA methods. \nI think the community stands to benefit from this line of investigation. \nThe paper communicates many of its ideas quite clearly. Figures 1 and 2 are especially helpful.\nAt a high level, the experimental setup is a sensible approach.\nAnonymized source code is included. \n", "weaknesses": "The paper is lacking in its theoretical treatment of the subject it aims to explore (effectively limited to Figure 1). This reduces its value to the community. Conclusions are drawn based on experiments conducted via (relatively small) samples from a few simple approximations of the posterior. The paper would benefit considerably from some mathematical analysis and discussion of how a Bayesian model or perspective might affect the validity of TDA methods, such as influence functions. For example, how does treating $\\theta$ as a random variable affect the application of the implicit function theorem used to derive influence? The Hessian of the loss is a function of $\\theta$, for some set of values in the support of $\\theta$ this Hessian is not invertible. Does this set have zero measure? Can it be safely ignored? It seems that there are numerous details that go unexamined.\n\nOverall the writing is quite clear. But crucially, I found the presentation of the test statistic (Equation 11) to be confusing. As such it was difficult to interpret (and thus to review) the subsequent experimental results. (See questions below. I continue the review assuming that the p-values quantify the probability that the samples of $\\tau$ could be generated by a random variable having mean equal to 0.)\n\nAt a high level, the experimental setup makes sense to me. However, some of the analytical steps in processing the collected data seem weak. For instance, on line 160 it is stated that “[this] treatment [...] poses a novel challenge for evaluation”. Yet if I understood correctly, the experiments yield samples from random variables that are compared in pairs, $\\tau$ and $\\tau’$. There are numerous established ways to compare two sample-sets, and to quantify the probability that they come from the same underlying distribution.  A two sample t-test may be an option.\n\nWithout much theoretical analysis to rest on, the experiments on two small datasets and two models struggle to convince me of the general claims. For example, the experiments on training set size (Section involve only three size settings, and don’t show clear trends. While the discussion on model complexity only considers two models which differ in both size and architecture.\n\nInfluence functions have been successfully used to reshape model behavior through the removal of identified sets of training instances. Several of the works cited in this paper achieve this. Therefore, the IF approximation must correlate with LOO at least on average for some types of training instances. How can this be reconciled with the results (e.g. in Figure 5)? This is not really discussed. Can they be explained via low noise pairs? \n\nMinor:\n\nLine 81: While Koh & Liang showed that IFs could be applied to neural networks, they did not derive the method. I suggest citing the original robust statistics papers on IF and/or the infinitesimal jackknife. \n\nFigures 3 and 4 would benefit from larger labels. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review and constructive suggestions.\n### The paper [...] would benefit [...] from some mathematical analysis [...] of how a Bayesian [...] perspective might affect the validity of TDA.\nOur probabilistic conversion of TDA methods does not affect theoretical soundness. We apply the original TDA algorithms on the Monte-Carlo samples of model posteriors without modification. Since the methods are identical, it is unclear which additional theoretical treatment would add value. We are happy to discuss should there be further requests.\n### Conclusions are drawn based on experiments [...] from a few simple approximations of the posterior. \nThe simplicity of posteriors is the driving force behind the successful application of Bayesian ML on complex deep learning models [a,b,c,d]. While simple posteriors like Deep Ensemble (DE) [11] and Stochastic Weight Averging (SWA) [12] do not restore the posterior perfectly, they are sufficient to support our main message: TDA is inherently stochastic. Also, we highlight that we use T=50 posterior samples. This is not a small scale in both Bayesian ML and TDA communities. DE [11] and SWA [21] used T=15 and 30, resp. Previous work on TDA has used only a few deterministic models (T=1 [e], 3 [f]).\n### How does treating θ as a random variable affect the [...] implicit function theorem [(IFT)]?\nEven in the deterministic setup, the IFT and inverse Hessians are generally inapplicable due to the non-convexity and high dimensionality of the deep learning optimisation problem. They are not novel challenges introduced by our probabilistic treatment. This is why Koh & Liang have proposed in [3] Section 4.2 (Non-convexity and non-convergence) to add a damping term λ to ensure the positive definiteness (PD) of Hessians. We used λ=3e-5 (as in [f]) and this guarantees PD Hessians at all posterior samples.\n### Analytical steps [...] seem weak. [...] There are [many] established ways to compare two sample-sets\nYes, there are many established ways to compare two random variables. However, we address a more complex problem. We need the rank correlation between two sets of random variables: $(\\tau_1, …, \\tau_N)$ vs $(\\tau_1’, …, \\tau_N’)$. $\\tau_i$ refers to the TDA value of the $i^\\text{th}$ train-test sample pair and $\\tau_i’$ refers to the estimated TDA value.\nWe are not aware of an existing tool for this problem. We have proposed to measure the Spearman’s ρ between the means and variances separately (Sec. 3), with the intuition that $\\tau’$ must also faithfully rank the train-test pairs according to the level of noise.\n### The experiments on two small datasets and two models struggle to convince me of the general claims.\nWe agree that testing on larger datasets makes our claim stronger. We were limited by computational resources (e.g. LOO for CIFAR10 of size 500 requires ~104 GPU hours). Nonetheless, the experiments support our general claim: TDA is stochastic and evaluation protocols must reflect this. Besides, TDA tends to be more stable with smaller models. We expect that noise will dominate the signal in larger settings, making the analysis less meaningful.\n### [Training set size experiments] involve only three size settings, and don’t show clear trends\nWe ran additional experiments further varying the training set size. The results show that the variance in TDA estimates remains high after certain training set size. Exp. 3 in global response. \n### [...] discussion on model complexity only considers two models which differ in both size and architecture.\nTo address the reviewer’s concern that our models are too different and few, we ran an experiment with a 3-layer CNN that matches ViT+LoRA in #trainable parameters. The results confirm that increasing model complexity decreases the statistical significance of TDA estimates. p-values for LOO with MNIST3: 0.331 (2-layer CNN) → 0.370 (3-layer CNN) → 0.786 (ViT+LoRA). Exp. 4 in global response.\n### How can [successful work with IFs] be reconciled with the results? \nWe share the reviewer’s intuition that successful related work (e.g. [3,15]) could be explained via low-noise pairs. Due to the prohibitive cost of LOO, previous work relied on a small number of train-test pairs (100 [3], 500 [15]). We internally attempted to replicate successful TDA results [3,15] but found that the impact of training stochasticity generally dominates the impact of a single training sample. This observation motivated our work.\n### Eq. 10[...] I assume these hypotheses refer to τ’s mean [...]. Is this correct?\nThe reviewer is correct. The hypothesis should read μ=0 where μ is the mean of the distribution τ(zj,z). We will correct this, thanks!\n### Eq. 11 what version of the T-test are you using?[...] Is the test designed to consider the pairing of the posterior (and perturbed posterior) samples?\nWe use the (unpaired) Student’s t-test [a] to test the statistical significance of μ>0, where μ is the mean of TDA estimates. In Eq. (9), we do not pair the indices $t$ and $t^\\prime$ for the two posteriors.\n### Is it common to combine these posterior sampling techniques?\nYes, e.g. [b] uses both DE and SWA to parametrise the posterior. This implies a mixture-of-Gaussian posterior family, where the DE models the centroids and SWA models each Gaussian component.\n### The results would be more convincing if the experiments had focused on only a subset of the test-train pairs, but considered larger [...] dataset sizes.\nWe ran an experiment with a CNN on MNIST to study TDA reliability on a set of 1000 train-test pairs. The results are in line with our submission: TDA is inherently stochastic and TDA methods fail to capture the variance in LOO. Exp.1 in global response.\n### Other\nWe will add [c], increase label sizes in Fig. 3 and 4, and include the dataset condition in Eq. 7.\\\n[a] Student (1908)\\\n[b] Wilson&Izmailov (NeurIPS 2020)\\\n[c] Hampel (1974)"}, {"review_id": "J1dT9n7RvE", "reviewer": "Reviewer_wqES", "summary": "The paper studies the challenges in measuring the performance of training data attribution (TDA) methods arising from stochasticity in training large deep neural networks. Specifically, the authors use various existing approaches to obtain many samples from the posteriors of the model weights instead of a point estimate with and without a training instance whose influence needs to be computed. The authors then compute the mean and variance of ground truth attribution/influence scores; and measure the performance of various attribution methods by looking at the correlation of these mean and variance with the ground truth scores. \n", "questions": "See Weaknesses section.", "limitations": "Yes, limitations are discussed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Thorough experiments comparing various Leave-One-Out (LOO) or attribution approximation methods including influence scores and variants (GD, GS), additional training step (ATS).\n- The paper provides further empirical evidence to previous observations [1] that common approximations used for LOO actually estimate slightly different objects and are susceptible to randomness in training arising from initialization of model weights, batch order, etc. ", "weaknesses": "- It is not clear if the phrase Bayesian perspective on TDA is useful, authors could say more on the connection between using Bayesian DL methods, Student t-test for measuring the noise in TDA estimates and Bayesian perspective on TDA.\n- In eq 8, how are the T samples from the two posteriors (with and without a training instance) paired.\n- It is also not clear if the observations made here only apply to measuring the attribution performance wrt to the ground truth attributions. That is, the main recommendation to focus on low signal to noise pairs seemly to apply only to evaluation. Did the authors perform such an evaluation? How does this recommendation affect the use of influence scores downstream applications such as correcting label mistakes, removing biased instances, etc.\n \n[1] Bae, Juhan, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B. Grosse. \"If Influence Functions are the Answer, Then What is the Question?.\" Advances in Neural Information Processing Systems 35 (2022): 17953-17967.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and questions. We hope to clarify concerns in the following. \n\n### It is not clear if the phrase Bayesian perspective on TDA is useful, authors could say more on the connection between using Bayesian DL methods, Student t-test for measuring the noise in TDA estimates and Bayesian perspective on TDA.\nTDA estimates are usually treated as deterministic point estimates, even though model training is in fact probabilistic (SGD and model initialisation are probabilistic). Bayesian ML is a framework for modelling the uncertainty in model training (cf. Section 2.2 of the submission and [a]). In our work, we borrow methods from Bayesian ML [b,c] to turn TDA estimates into random variables. The random variables are in turn studied through the proposed statistical testing. By treating TDA estimates as a probabilistic distribution, we can examine the statistical significance of TDA estimates and quantify the variance which we found to be inherent in the TDA task. \n\n### In eq 8, how are the T samples from the two posteriors (with and without a training instance) paired.\nWe don't pair the samples for the two posteriors, as you can see in Eq (9): \n$$\n\\text{Var} [\\tau (z_j, z)] = \\frac{1}{T^2}\\sum_{t,t^\\prime} \\left(\\mathcal{L}(z ;\\theta^{(t)}_{\\setminus j}) - \\mathcal{L}(z ;\\theta^{(t^\\prime)})-\\mathbb{E} [\\tau (z_j, z)]\\right)^2\n$$\n\nThe reason we use common index $t$ for the two distributions in Eq (8) is because it results in the identical output due to the linearity: \n\n$$\n\\mathbb{E} [\\tau (z_j, z)] = \\frac{1}{T}\\sum_t \\mathcal{L}(z ;\\theta^{(t)}\\_{\\setminus j}) - \\frac{1}{T}\\sum\\_{t^\\prime} \\mathcal{L}(z;\\theta^{(t^\\prime)}) = \\frac{1}{T}\\sum_t \\mathcal{L}(z ;\\theta^{(t)}_{\\setminus j}) - \\mathcal{L}(z ;\\theta^{(t)})\n$$ \n\nWe hope this brought some clarity and are happy to discuss any follow-up questions further.\n \n###  It is also not clear if the observations made here only apply to measuring the attribution performance wrt to the ground truth attributions. That is, the main recommendation to focus on low signal to noise pairs seemly to apply only to evaluation. Did the authors perform such an evaluation? \nNo, we did not make an analysis of the set of low-noise pairs in the submission but it is an interesting point. The Spearman rank correlation matrices of the means, standard deviations and p-values for the set of low-noise train-test pairs (LOO p-values <0.05) are provided in the global response PDF for two experiments (MNIST3 $|\\mathcal{D}|=150$ and CIFAR10 $|\\mathcal{D}|=500$ with CNN). \nWe can see from this analysis:\nThe approximate TDA methods do not correlate strongly with LOO. \nIn this subset, p-values correlate positively among approximate methods, underlining observations from Section 4.4 (Dis)Agreement of TDA methods: TDA estimation methods discover similar attribution including the stochastic noise. \nIn this subset, means of TDA methods correlate stronger with LOO than for all train-test pairs (e.g. for MNIST3: ATS: 0.01 -> 0.09, IF: 0.05 ->0.10, GD: -0.05 -> -0.11, GC: 0.07 -> 0.09 (Fig 5, lower left matrix -> Rebuttal PDF file, upper left)\nThis evaluation shows that none of the tested TDA methods approximate the ranking of the true change in loss, even if this change is stable wrt training process stochasticity. In the case of IF, the possible reason could lie in observations made by [d]: IFs do not correspond to pure LOO retraining, approximation gaps and solver errors lead to a different objective. A study regarding how the TDA estimations methods correspond to [d]’s objective would be interesting in the future. \n\n### How does this recommendation affect the use of influence scores downstream applications such as correcting label mistakes, removing biased instances, etc.\nThe question of how variance in TDA estimates may affect downstream tasks is intriguing. We ran an additional experiment for the downstream task of mislabel identification similar to [e] and [f] with MNIST3. We find that the inherent stochasticity of TDA leads to a large range in mislabel identification performance. High variance in the TDA estimates degrades downstream task performance. Further details in the global response, experiment 2. \n\n[a] Bayesian methods in global optimization (1991). \\\n[b] If Influence Functions are the Answer, Then What is the Question? (NeurIPS 2022).\\\n[c] Simple and scalable predictive uncertainty estimation using deep ensembles (NeurIPS 2017).\\\n[d] Averaging weights leads to wider optima and better generalization (UAI 2018).\\\n[e] Understanding black-box predictions via influence functions (ICML 2017).\\\n[f] Estimating training data influence by tracing gradient descent (NeurIPS 2020)."}]

Author rebuttal:
We thank the reviewers for their constructive comments and suggestions. Reviewers agree that we “study an important practical problem” (fmRD, CbTs) and provide “thorough experiments” (wqEs) and a “strong analysis” (fmRD). 

We have addressed individual reviewers’ comments and questions in the dedicated responses. We use the global response to show the results of experiments requested by the reviewers.

### 1: Larger dataset (fqXw, CbTs)

Summary: We perform the test for statistical significance of TDA given a model trained on MNIST ($|D|=60,000|$). We base the statistical analysis on a small random subsample and find high (p>0.05) variance in ground-truth TDA as well as TDA estimates, in line with previous findings. 

Setting: 
We train a 2-layer CNN on the full MNIST dataset and sample $T=50$ times from the posterior. For analysis of TDA methods, we uniformly sample 100 $z_j$ from the training set and 10 $z$ from the test set, resulting in 1000 train-test pairs.
Results:
The model has a predictive accuracy on the MNIST test set of $0.979 \pm 0.001$ at 95% CI. The p-values resulting from the TDA analysis are: 
LOO  |ATS  |IF   |GD   |GC   
-----|-----|-----|-----|-----
0.761|0.362|0.464|0.475|0.247

Distribution of p-values for LOO and correlation analysis is attached in the global response PDF. 

The results show that TDA estimates vary strongly for the small subset of train-test pairs (high p-values). Given our previous findings, we find two main reasons: (1) MNIST is larger in training set size than the initial sets we used. The attribution of one sample to model behavioris likely to be marginal and unstable. (2) Low-noise samples exist, but they are in the minority. This experiment shows that LOO is inherently noisy and TDA approximation methods fail to capture this. This conclusion aligns with the results in the submission. 

### 2: Mislabel identification (wqEs, CbTs)

Summary: We intentionally mislabel parts of the training dataset and aim to identify the mislabeled samples using TDA. We find that the inherent stochasticity of TDA leads to a large range in mislabel identification performances.  

Setting: 
We perform this experiment with the 2-layer CNN and MNIST3 ($|\mathcal{D}|=150$) and CIFAR10 ($|\mathcal{D}|=500$). We follow the procedure from Koh & Liang [a]: First, a random 10% of the datasets are mislabeled by the highest scoring incorrect label. We train the model using these mislabeled datasets (sample $T=50$ times from the posterior). Then, we compute self-influence, which is the attribution of a sample to itself $\tau(z_j, z_j)$ [a], with each TDA method. The mislabeled dataset is ranked according to self-influence and the quantity of interest is the fraction of mislabeled samples found when inspecting the top x% of the ranked dataset. 
In the analysis, we inspect (1) the range of mislabeled fractions discovered if we treat TDA deterministically, i.e. we compute the discovered fraction per $t\in T$ and report the range; (2) the fraction of mislabeled samples discovered when we use the mean over the TDA scores of our posterior samples. 

Results:
The results plots are visualized in the global response PDF.  

We find that deterministic TDA results in a large range of possible outcomes for identifying mislabeled samples. This means that it is harder to reliably identify mislabels when TDA is treated as point estimates. 

### 3: Training set size (fqXw) 

Summary: Ablations with training set sizes, expanding on the experiment from section 4.3 “Training set size”, show that there may be a point of “stochasticity saturation” where the increasing size of the trainset does not contribute more to the variance in TDA estimates.

Setting:
We expand the experiment with the 2-layer CNN on MNIST3 and CIFAR10 from section 4.3 Training set size by testing the model trained on MNIST3 of sizes 90, 120, 180 and CIFAR10 of sizes 300, 400, 600. 

Results:
Updated plots of Figure 4 in global response PDF. 

The results indicate that an increase in training set size does not necessarily lead to an increase in noise (i.e. the relationship is not linear). Instead, we observe that there may be a point of “stochasticity saturation” where the increasing size of the training set does not contribute more to the variance in TDA estimates (p-values stay large, but don’t increase).
We will update the discussion in paragraph 4.3 “Training set sizes” to reflect this for the final version.  

### 4: Model complexity (fqXw)
Summary: An additional experiment with a 3-layer CNN shows that model complexity is likely a factor in the reliability of TDA estimates.

Setting:
We train a 3-Layer CNN with 620,362 trainable parameters on MNIST3 ($|\mathcal{D}|=150$), CIFAR10 ($|\mathcal{D}|=500$) to analyse TDA methods. We chose this model as it is comparable to the 2-layer CNN in terms of architecture and to the ViT+LoRA model in terms of trainable parameters (597,514). 

Results:
P-values (Distributions of p-values for ground-truth TDA (LOO) in the global response PDF.)
p-values|LOO  |ATS  |IF   |GD   |GC   
--------|-----|-----|-----|-----|-----
MNIST3  |0.370|0.368|0.464|0.470|0.005
CIFAR10 |0.687|0.432|0.579|0.581|0.365

The results show that model complexity is likely a factor for the stochasticity in TDA estimates, where increasing model complexity (architecture and number of trainable params) means increasing variance, in line with previous findings. We also note that the number of low-noise train-test pairs decreases with increasing model complexity, but they still exist. 

[a] Understanding black-box predictions via influence functions (ICML 2017)


Meta-review: This paper has a very simple message that reviewers found unsurprising/intuitive, but nevertheless appreciated. Reviewers took issue with the "Bayesian" framing, and I tend to agree, as it is more of a probabilistic perspective than anything else. I am accepting the paper, but the authors should make sure to (a) appropriately reframe the paper based on reviewer comments, and (b) carefully address all of the detailed comments, especially the ones raised by fqXw.
Decision: Accept (poster)