SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In addition to our main experiments, we conduct a statistical test on the TDA scores obtained from a CNN2-L model trained on the complete MNIST  $(|\mathcal{D}| = 60,000)$  dataset (cf. Figure 4). We base the analysis on  $T = 50$  samples from the model posterior (10 random seeds  $\times 5$  checkpoints) and a small random subsample of data (100 training samples  $\times 10$  test samples  $= 1000$  train-test pairs) due to the high computational cost of retraining. We find high  $(\mathrm{p} > 0.05)$  variance in ground-truth TDA as well as TDA estimates, in line with findings from §4.3 Training set size.
The results show that TDA estimates vary strongly for the small subset of train-test pairs (high p-values). We identify two main reasons: (1) MNIST is larger in training set size than the initial sets we used. The attribution of one sample to model behaviour is likely to be marginal and unstable. (2) Low-noise samples exist, but they are in the minority. This experiment shows that LOO is affected by the stochasticity of deep model training and TDA approximation methods fail to capture this.
B.5 All results: Stability of TDA values and estimates
Table 5 presents the complete table of p-values of all tested TDA methods across all experiments of our work except for the experiments detailed in Appendices B.3 and B.4. Below, we display the histograms of p-values per experiment corresponding to each line in the table captioned with the experiment ID. The histograms show that low-noise train-test pairs  $(z_{j},z)$  are present in all experiments involving the CNN model (i.e., experiments 1-8), where the number of pairs varies. Generally, we observe that there is no connection between the size of the dataset and the distribution of p-values. Furthermore, we notice that fixing the model initialisation (i.e., randomness induced by SWA+DE-Batch) increases the number of stable train-test pairs (cf. experiment 5 to 13, 11 to 14). However, in the case of the ViT experiments, stable train-test pairs are practically non-existent which shows that model complexity affects the stability of TDA.
Table 5: Complete list of mean p-values of TDA values for all experiments.
Experiment	LOO	ATS	IF	GD	GC
ID	Model	Data	Randomness	|Dtrain|
1	CNN2-L	MNIST3	SWA+DE-Init	30	0.058	0.088	0.111	0.110	0.020
2	CNN2-L	MNIST3	SWA+DE-Init	60	0.421	0.148	0.251	0.253	0.002
3	CNN2-L	MNIST3	SWA+DE-Init	90	0.714	0.355	0.466	0.470	0.209
4	CNN2-L	MNIST3	SWA+DE-Init	120	0.675	0.346	0.469	0.472	0.218
5	CNN2-L	MNIST3	SWA+DE-Init	150	0.331	0.254	0.352	0.363	0.003
6	CNN2-L	MNIST3	SWA+DE-Init	180	0.374	0.254	0.355	0.356	0.001
7	CNN2-L	CIFAR10	SWA+DE-Init	100	0.665	0.424	0.607	0.608	0.352
8	CNN2-L	CIFAR10	SWA+DE-Init	200	0.552	0.397	0.450	0.452	0.399
9	CNN2-L	CIFAR10	SWA+DE-Init	300	0.543	0.389	0.456	0.456	0.313
10	CNN2-L	CIFAR10	SWA+DE-Init	400	0.619	0.418	0.562	0.568	0.344
11	CNN2-L	CIFAR10	SWA+DE-Init	500	0.692	0.438	0.575	0.587	0.356
12	CNN2-L	CIFAR10	SWA+DE-Init	600	0.665	0.447	0.575	0.579	0.358
13	CNN2-L	MNIST3	SWA+DE-Batch	150	0.025	0.039	0.000	0.000	0.000
14	CNN2-L	CIFAR10	SWA+DE-Batch	500	0.623	0.374	0.535	0.534	0.314
15	CNN3-L	MNIST3	SWA+DE-Init	150	0.370	0.368	0.464	0.479	0.005
16	CNN3-L	CIFAR10	SWA+DE-Init	500	0.687	0.432	0.579	0.581	0.365
17	ViT+LoRA	MNIST3	SWA+DE-Batch	150	0.786	0.573	0.369	0.365	0.093
18	ViT+LoRA	CIFAR10	SWA+DE-Batch	500	0.777	0.766	0.686	0.686	0.522
B.6 All results: Correlation analysis
In the main body of this paper, we report the Pearson and Spearman correlation matrices for experiment 3 (CNN2-L trained on MNIST3 with 50 samples per class and randomness induced by SWA+DE-Init). This section presents the complete overview of correlations between TDA methods across all experiments in Figures 10 - 27. We note that the observations and analyses in the main paper hold across experiments.
11
15
12
16
13
17
Figure 9: Distribution of p-values for ground-truth TDA (LOO) for all experiments (IDs corresponding to IDs in Table 5).
Figure 10: Experiment 1 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Figure 11: Experiment 2 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Figure 12: Experiment 3 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Figure 13: Experiment 4 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Mean  $\hat{\mu}$
Standard deviation  $\hat{\sigma}$
p-value
Figure 14: Experiment 5 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Mean  $\hat{\mu}$
Figure 15: Experiment 6 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Figure 16: Experiment 7 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Figure 17: Experiment 8 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Figure 18: Experiment 9 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Figure 19: Experiment 10 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Figure 20: Experiment 11 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Figure 21: Experiment 12 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Figure 22: Experiment 13 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Figure 23: Experiment 14 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Figure 24: Experiment 15 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Figure 25: Experiment 16 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Figure 26: Experiment 17 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.
Figure 27: Experiment 18 (cf. Table 5): Pearson and Spearman correlation coefficients among ground-truth TDA and approximate TDA methods. We show correlations for TDA mean  $\hat{\mu}$ , TDA standard deviation  $\hat{\sigma}$ , and TDA p-values.