SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We assess the reliability of TDA evaluation by measuring the degrees of noise in both the ground-truth TDA (LOO)  $\tau(z_j, z)$  and the estimated TDA  $\tau'(z_j, z)$ . The noise level is measured with the p-value of the Student-t hypothesis testing to determine if the absolute TDA values are significantly greater than the sample noise ( $\S 3$ ).
We report the results in Table 1. Generally, we observe many TDA measurements, ground-truth and estimations likewise, are unstable with non-significant p-values  $(>0.05)$ . In particular, even the ground-truth LOO shows p-values of 0.331 on MNIST3 and 0.692 for CIFAR10 (SWA+DE-Init). In these cases, the noise effectively dominates the signal and any evaluation that does not consider the variance in the posterior  $p(\theta|\mathcal{D})$  is likely to be misleading. This confirms the reports in [9] that TDA values are sensitive to model initialisation.
TDA methods often show similar levels of instability. For example, the IF attains p-values 0.352 and 0.575 on MNIST3 and CIFAR10, respectively, roughly matching the LOO case. Grad-Cos is an exception: it attains lower p-values than the other TDA methods (0.003 and 0.356 for MNIST3 and CIFAR10, respectively). We interpret this as an overconfident TDA estimation. Practitioners shall be wary of using TDA methods that are unreasonably stable when the ground-truth TDA itself is not.
CNN2-L, MNIST3
CNN2-L, CIFAR10
Figure 4: Impact of training data size. Mean p-values of TDA methods with randomness induced by SWA+DE-Init.
Table 2: Impact of model complexity. Mean p-values of the ground-truth TDA  $\tau(z_j, z)$  (LOO) and estimated TDA values  $\tau'(z_j, z)$  (the other 4 columns) with randomness induced by SWA+DE-Batch for MNIST3 and DE-Batch for CIFAR10.
Model	Data	LOO	ATS	IF	GD	GC
CNN2-L	MNIST3	0.025	0.039	0.000	0.000	0.000
CNN3-L	MNIST3	0.370	0.368	0.464	0.470	0.005
ViT+LoRA	MNIST3	0.786	0.573	0.369	0.365	0.093
CNN2-L	CIFAR10	0.623	0.374	0.535	0.534	0.314
CNN3-L	CIFAR10	0.687	0.432	0.579	0.581	0.365
ViT+LoRA	CIFAR10	0.777	0.766	0.686	0.686	0.522
4.3 Factors influencing the variability of TDA
Based on the observation in §4.2 that TDA values are often dominated by noise, we delve into the factors that lead to the instability of data attributions. We inspect the contribution of model initialisation, training set size and model complexity.
Source of randomness. From a Bayesian ML perspective, the stochasticity of TDA stems from the inherent uncertainty of the learned model posterior  $p(\theta | \mathcal{D})$ . We consider two sources of randomness, model initialisation (DE-Init) and SGD batch composition (DE-Batch). Results are reported in Table 1. For MNIST3 and CIFAR10, we observe that DE-Batch introduces lower levels of noise in the TDA estimates (lower p-values). Particularly on MNIST3, both LOO and other TDA methods result in statistically significant p-values ( $< 0.005$ ). This implies that almost every training data  $z_{j}$  is influencing every test data  $z$  consistently across various batch compositions. We conclude that the greater source of variations for the TDA estimates is the model initialisation.
Training set size. We study how training set size is a source of noise (cf. Figure 4). We train the CNN2-L with different-size datasets of MNIST3 and CIFAR10, where we vary the number of samples per class. Batches are composed differently depending on the dataset size, meaning that parameter updates are made after processing different data. In addition, we train a CNN2-L on the complete MNIST dataset and use a subset of the training data for our experiments (cf. Appendix B.4). The results show a tendency for high variation in TDA scores with larger datasets first after which a decrease in variation is observed. The initial increase makes sense as the number of combinations for batch composition increases with dataset size. As the batching is initialised randomly during training, batches are likely to be composed of different data for larger datasets. This leads to variation in the learned model parameters, in turn affecting the reliability of TDA. At the point of decrease, the TDA scores are rather small for all train-test pairs. The attribution of individual training samples to a model prediction is overall small in larger datasets, which leads to a decrease in variance.
Model complexity. We study how model complexity is linked to the reliability of TDA estimates. See Table 2. We observe that, compared to the CNN models, a large ViT model trained with LoRA results in dramatically greater p-values. For example, for LOO on MNIST3, the p-value increases from 0.025 (CNN2-L) to 0.786. A similar trend is observed for other TDA methods. A less dramatic increase in p-values can also be observed with the addition of another layer to the CNN (i.e. from
Figure 5: Correlation of TDA methods. Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values. All results are based on the setting: CNN2-L, MNIST3, SWA+DE-Init.
CNN2-L to CNN3-L). This implies that the reliability of TDA estimates decreases with increasing model complexity. While we limit the number of trainable parameters in our ViT by using LoRA to be comparable to CNN3-L, the p-values computed from TDA estimates are significantly larger. Larger models exhibit a larger parameter space so that noise stemming from model initialisation or batch composition is amplified. While we fix the model initialisation and dataset size, the batch composition still varies across the model parameters  $\theta$  sampled from the posterior  $p(\theta|\mathcal{D})$  per model. As both the CNNs and ViT are trained with the same sampled batch compositions, we attribute the strong increase of p-value to the model complexity.
4.4 (Dis)agreement between TDA methods
We test the reliability of different TDA methods. Ideally, all methods approximate the ground-truth TDA (LOO). Yet the results suggest that there are substantial differences among the methods. For example, Grad-Cos is much more stable than all others. Hence, we study TDA methods with respect to both their correlation with LOO and among each other using Pearson and Spearman correlation of mean and variance of the TDA distributions, as proposed in §3.
Figure 5 shows the correlation matrices for one experiment (all experimental results in the Appendix). The results show that viewing TDA scores as distributions gives insights into the reliability of TDA methods: None of the tested TDA method's expected values  $\hat{\mu}$  correlates with LOO. This implies that none of the TDA methods is a good approximation when the random factors are considered. The poor correlation of p-values of LOO indicates a disagreement in the train-test pairs considered low noise. We conclude that none of the tested methods reliably capture ground-truth TDA distributions.
Interestingly, we notice a stronger correlation between all other methods particularly when looking at the correlations of p-values. We identify two groups based on positive correlation, i.e. ATS with IF and GD with GC. Among the two groups, there is a negative correlation which indicates that methods interpret the sign of the attribution differently. Between IF, GD and GC this makes sense, as there is a negative sign in the definition of IF (Equation 2) which is not present in GD and GC. Considering absolute correlation, IF and GD are strongly correlated which shows that the dot product is a valid alternative for IF as they produce similar score distributions. The correlation between GD and GC indicates that the normalisation of gradients does not have a strong impact on the estimated TDA.
IF, GD and GC correlate considerably with ATS, which measures how the loss of a model on  $z$  changes after doing one additional training step on  $z_{j}$ . Practically, ATS represents the gradient update after  $z_{j}$ , which is the same as the gradient  $z_{j}$  itself. Therefore, it makes sense that the gradient-based approximation methods are close to ATS. We recognise a difference in the scope LOO and ATS
address. LOO looks at TDA globally and encapsulates the whole training, whereas ATS considers a local scope with a small model change. As IF, GD and GC correlate with ATS, we observe that they also correspond to a local change in the model, which underlines and extends the argument of [7]: There is a gap between LOO and IF, and more generally between the global and local view on TDA.
The TDA variance  $\hat{\sigma}$  is noticeably well-correlated for TDA estimators and LOO, except for GC. This implies the existence of a consistent ranking of train-test pairs with stable attribution relationships. In particular, stable train-test pairs predicted by LOO are also likely to be stable pairs for TDA methods like ATS, IF, and GD. This motivates our final analysis and recommendation for evaluation in §4.5.
4.5 Considerations on TDA evaluation from a Bayesian perspective
Our analysis shows that both TDA estimates and ground-truth TDA values are affected by the noise stemming from the stochastic nature of deep model training. Hence, the practice of comparing against such a ground truth is destined to result in fragile estimates. We propose to treat TDA estimates as random variables which allows us to look at the evaluation from a Bayesian perspective: The comparison of TDA estimates against target TDA values is a comparison of two random variables. Since it is impossible to get rid of noise, it is better to compare distributions rather than point estimates. This provides an understanding of how well methods approximate the ground-truth distribution.
We observe that p-values vary between individual train-test sample pairs  $(z_{j}, z)$ ; not all TDA estimates are equally affected by stochasticity. Interestingly, the presence of low-noise pairs is consistent across the majority of our experiments (cf. Figure 3), with varying sizes of the low-noise fraction. We find that fixing model initialisation and a small dataset size gives rise to a larger number of low-noise pairs.
We propose to focus on such low-noise pairs in TDA evaluation as their estimates are low in variance, leading to a more reliable evaluation. Identifying such pairs requires an analysis similar to this work: treating TDA values as distributions and sampling multiple times from the posterior to get an estimate of the noise. It is crucial to find low-noise pairs to base evaluations on and understand when TDA is applicable. If no low-variance pairs exist, TDA cannot be used.
5 Related work
We study the reliability of TDA methods and add to the existing body of work on the fragility of TDA methods. Previous studies focused primarily on IF [29, 6, 8, 9, 7]. We extend the analysis by additionally studying other TDA methods. While IFs are theoretically grounded in robust statistics [15], they are based on two assumptions which are not always fulfilled in the context of deep learning: Twice-differentiability and strict convexity of the loss [3]. Zhang & Zhang [29] and Basu et al. [6] point to the fragility of the influence scores due to the non-convexity of deep learning. Particularly increasing model size is connected to increased model curvature, which means that influence estimates are more fragile with larger models. They find that strong regularisation is needed to improve estimation quality. Our experiments verify the observation that fragility increases with model size, which we observe across methods. We add that sources of randomness in the training process attribute to the fragility of TDA methods with increasing model size. Furthermore, related work found that the size of the training set contributes to the fragility of influence estimates. The attribution of one sample in a large training set is marginal so both influence estimates and ground-truth influence scores (i.e., from retraining the model) are noisy [6, 8, 9]. Through a Bayesian lens, we connect the increased fragility with increasing dataset size to batch composition as well. Not only is the attribution of a single sample in a large dataset marginal [6] but batches have vastly different compositions in larger datasets, introducing noise. A recent work [7] states that influence functions in deep learning do not correspond to LOO and quantify gaps in the estimation stemming from model non-linearity. A different approach in TDA [30, 31] aims at predicting the expected model output given a set of data points, directly considering randomness stemming from model initialisation. K & Søgaard [9] recommend reporting expected TDA scores to increase estimation stability. This approach is closest to our work but misses the consideration of variance in TDA estimates which we include by taking a Bayesian viewpoint.
In contrast to related work, we treat TDA values as distributions, which enables a novel perspective on the TDA task for deep models. We highlight the importance of considering the variance when studying reliability.
We adopt a Bayesian perspective on the training data attribution (TDA) methods to study their reliability when applied to deep models, given the stochastic nature of deep model training. By modelling TDA scores as distributions, we find that randomness in the training process, particularly due to parameter initialisation and batch composition, translates to variation in ground-truth TDA. We empirically observe that current estimation methods, such as influence functions, model a local change in the model whereas the ground truth attribution considers a global model change. Therefore, TDA is subject to inherent variance, leading us to suggest to the community: (1) When proposing a novel TDA method, one should view TDA from a Bayesian perspective and study the TDA estimates as distributions. (2) When using TDA, one should consider the variance to understand when the estimate can be trusted.
Limitations. We perform an exhaustive analysis of TDA values  $\tau$  and the estimates  $\tau'$  for all train-test pairs  $(z_{j}, z)$ . Because of considerable computational costs, we have subsampled the datasets. In practice, datasets are considerably larger. Moreover, we choose simple tasks to eliminate the need for an additional hyperparameter search for model training, as the principal focus is on studying TDA methods. We choose gradient-based TDA methods but acknowledge that there exist many more, that we do not address. Hence, we encourage further study of TDA methods to fill these gaps and recommend investigating TDA from a Bayesian perspective, particularly in the low-data regime.
Broader impact. This paper contributes to the field of data-driven XAI which aims at helping humans understand the inner workings of opaque models through data-centric approaches. Our work contributes to understanding the reliability of TDA methods and rethinking their evaluation against a noisy ground truth, which could help assess when TDA is appropriate and reliable.
Acknowledgments and Disclosure of Funding
Kay Choi has helped in designing Figures 1 and 2. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Elisa Nguyen. This work was supported by the Tübingen AI Center.
[1] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM Comput. Surv., 51(5), 2018. ISSN 0360-0300. doi: 10.1145/3236009. URL https://doi.org/10.1145/3236009.
[2] Zayd Hammoudeh and Daniel Lowd. Training data influence analysis and estimation: A survey. ArXiv, abs/2212.04612, 2022.
[3] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1885-1894. PMLR, 06-11 Aug 2017. URL https://proceedings.mlr.press/v70/koh17a.html.
[4] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 19920-19930. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/e6385d39ec9394f2f3a354d9d2b88eec-Paper.pdf.
[5] Jinghan Yang, Sarthak Jain, and Byron Wallace. How many and which training points would need to be removed to flip this prediction? In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2563-2576, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. URL https://aclanthology.org/2023.eacl-main.188.
A Model training details
We provide the source code at https://github.com/ElisaNguyen/bayesian-tda. All experiments were run on a single Nvidia 2080ti GPU.
A.1 Data sampling
We use subsampled versions of the openly available MNIST [24] and CIFAR10 [25] datasets. For this, we first add an index which we use for randomly sampling a fixed number of images from each class. Table 3 includes the dataset sizes of the different experiments.
A.2 Model training
CNN2-L has two convolutional layers followed by two fully connected linear layers, with GeLU activation. We use the Adam optimizer with a learning rate of 0.001 and a weight decay of 0.005. We use the cross-entropy loss and train the model for 15 epochs on MNIST3 and for 30 epochs on CIFAR10 with a batch size of 32. CNN3-L has 3 convolutional layers followed by two fully connected linear layers with GeLU activation. The hyperparameters are the same as for CNN2-L. For training the ViT with LoRA, we use the peft [32] and HuggingFace transformers library [33]. We start from the pretrained model checkpoint of [34] and finetune the LoRA model with the same hyperparameters as the CNN.
An overview of the predictive performance measured in accuracy on the subsampled training and test sets is provided in Table 3.
Hint for reproducibility. In particular, we use CrossEntropyLoss (reduction = 'none') during model training and also update this in the ViT training script modeling_vit.py. This is important for the LOO experiments, where we exclude a sample  $z_{j}$  from contributing to the training by zeroing out the loss.
Table 3: Predictive performance at  ${95}\%$  CI across 10 runs (computed as 1.96*SE)
Experiment	Accuracytrain	Accuracytest
Model	Data	Randomness	|Dtrain|	|Dtest|
CNN2-L	MNIST3	SWA+DE-Init	30	900	0.987±0.010	0.953±0.003
CNN2-L	MNIST3	SWA+DE-Init	60	900	0.985±0.007	0.970±0.004
CNN2-L	MNIST3	SWA+DE-Init	90	900	0.995±0.003	0.939±0.005
CNN2-L	MNIST3	SWA+DE-Init	120	900	0.999±0.001	0.941±0.008
CNN2-L	MNIST3	SWA+DE-Init	150	900	0.998±0.004	0.970±0.003
CNN2-L	MNIST3	SWA+DE-Init	180	900	1.000 ± 0.000	0.985±0.001
CNN2-L	CIFAR10	SWA+DE-Init	100	500	0.989±0.020	0.260±0.010
CNN2-L	CIFAR10	SWA+DE-Init	200	500	1.000±0.000	0.328±0.007
CNN2-L	CIFAR10	SWA+DE-Init	300	500	1.000±0.000	0.360±0.005
CNN2-L	CIFAR10	SWA+DE-Init	400	500	0.983±0.020	0.362±0.011
CNN2-L	CIFAR10	SWA+DE-Init	500	500	0.992±0.014	0.377±0.012
CNN2-L	CIFAR10	SWA+DE-Init	600	500	0.994±0.005	0.394±0.011
CNN2-L	MNIST3	SWA+DE-Batch	150	900	0.993±0.002	0.975±0.002
CNN2-L	CIFAR10	SWA+DE-Batch	500	500	0.991±0.010	0.364±0.003
CNN3-L	MNIST3	SWA+DE-Init	150	900	0.994±0.005	0.971±0.008
CNN3-L	CIFAR10	SWA+DE-Init	500	500	0.989±0.008	0.363±0.011
ViT+LoRA	MNIST3	SWA+DE-Batch	150	900	0.945±0.008	0.935±0.005
ViT+LoRA	CIFAR10	SWA+DE-Batch	500	500	0.934±0.006	0.892±0.008
B Additional experimental results
We study the reliability of TDA estimates and values through a hypothesis test, where we report the p-values as an indication of the statistical significance of the TDA estimate. In this appendix, we
Figure 6: P-values of ground-truth TDA (LOO) with increasing number of SWA samples (i.e., number of model checkpoints used as samples of  $\theta$ ) for the CNN trained on MNIST3 and CIFAR10.
provide a complete overview of the analyses of p-values and correlations between different TDA methods.
B.1 DE vs. DE+SWA
In our work, we sample trained models  $\theta$  sampled from the posterior  $p(\theta | \mathcal{D})$ . Concretely, we train the model across 10 different random seeds and record the checkpoints after each of the last five epochs of training. Each of these models represents a sample. We test how the stability of the TDA values  $\tau(z_j, z)$  (LOO) behaves when we ensemble different numbers of checkpoints by investigating the mean p-values across all train-test pairs  $(z_j, z)$ . Figure 6 shows that higher numbers of samples  $\theta$  increase stability, therefore we use all available samples in our subsequent analyses.
B.2 Correlation analysis on low-noise pairs
Figure 7: Correlation analysis of low-noise train-test pairs Spearman rank correlation coefficients for TDA scores with  $p < 0.05$  of LOO retraining of the 2-layer CNN models trained on the respective datasets.
We analyse the Spearman correlation between the TDA approximation methods and ground-truth TDA (LOO) for low-noise train test pairs (i.e. where p-value of LOO scores is  $\leq 0.05$ ). The correlation matrices are shown in Figure 7. Within the subset of low-noise train-test pairs, the Spearman correlation of the mean between LOO and the approximation methods is higher than when considering the whole dataset for both MNIST3 and CIFAR10. The correlation of  $\hat{\sigma}$  of the
low-noise analysis is similar to the results found in the correlation analysis of the whole dataset (cf. Figures 12, 15). Therefore, we observe similar trends when considering only the low-noise train test pairs to the complete dataset: There is a weak correlation between the TDA approximation methods and LOO, while the TDA approximation methods correlate strongly among themselves. This reflects the difference between the global change of retraining and the local approximations.
B.3 Mislabel identification
Figure 8: Fractions of mislabeled samples discovered with the deterministic definition of TDA (left, Equation 1) vs. a probabilistic definition of TDA (right, Equation 7).
A common way of evaluating TDA methods is through the auxiliary task of mislabel identification. We perform an this experiment with CNN2-L trained on MNIST3  $(|\mathcal{D}| = 150)$  and CIFAR10  $(\mathcal{D} = 500)$ . We follow the procedure from Koh & Liang [3]: First, a random  $10\%$  of the datasets are mislabeled by the highest scoring incorrect label. We train the model using these mislabeled datasets (sample  $T = 50$  times from the posterior). Then, we compute self-influence, which is the attribution of a sample to itself, with each TDA method. The mislabeled dataset is ranked according to self-influence and the quantity of interest is the fraction of mislabeled samples found when inspecting the top  $x\%$  of the ranked dataset. In the analysis, we inspect (1) the range of mislabeled fractions discovered if we treat TDA deterministically, i.e. we compute the discovered fraction per  $t \in T$  and report the range (cf. Figure 8 left); (2) the fraction of mislabeled samples discovered when we use the mean over the TDA scores of our posterior samples (cf. Figure 8 right). We find that deterministic TDA results in a large range of possible outcomes for identifying mislabeled samples. This means that it is harder to reliably identify mislabels when TDA is treated as point estimates.
B.4 Experiments on MNIST (|D|=60,000)
Table 4: Mean p-values of ground-truth TDA (LOO) and estimated TDA values of with randomness included by SWA+DE-Init for CNN2-L trained on the complete MNIST dataset.
LOO	ATS	IF	GD	GC
0.761	0.362	0.464	0.475	0.247