SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dataset pruning We conducted a dataset pruning task in Paul et al. [46] to validate  $\mathcal{I}_{\mathrm{GEX}}$  for clean label settings. We use the same pre-training and post-hoc hyperparameters as noisy label settings for this task (except for the presence of mislabeled samples). Since the pruning capacity (i.e., the maximum number of samples to be pruned) of all datasets is not the same, we prune 40K samples for MNIST (within  $1\%$  test acc. degradation for all methods), 25K samples for CIFAR-10 and SVHN (within  $0.5\%$  test acc. degradation for F-score, EL2N, and  $\mathcal{I}_{\mathrm{GEX}}$ ), and 20K samples for CIFAR-100 (within  $4\%$  test ACC. degradation for  $\mathcal{I}_{\mathrm{GEX}}$ , F-score,  $\mathcal{I}_{\mathrm{LA}}$ ).
Separation of data sources We also conducted a data source separation task in Harutyunyan et al. [19] to validate  $\mathcal{I}_{\mathrm{GEX}}$  for clean label settings. In this task, we train a ResNet-18 on a mixed dataset of MNIST and SVHN, with 25,000 random subsamples for each training dataset. The other pre-train and post-hoc hyperparameters are the same as for noisy labels.
In this section, we conducted additional experiments for completeness. To this end, we provide an ablation study for the ensemble size of  $\mathcal{I}_{\mathrm{GEX}}$  in Appendix F.1. We also provide additional results of Sec. 5 in Appendix F.2-F.4. In Appendix F.5, we report the noisy label detection performance with cross-influence. Finally, we study a purely post-hoc  $\mathcal{I}_{\mathrm{TracInRP}}$  by using GE in Appendix F.6.
F.1 Ablation study for the ensemble size
Table 7: AUC and AP for noisy label detection with  ${10}\%$  label corruption
CIFAR-10	CIFAR-100
MGE	AUC	AP	AUC	AP
32	99.74 ± 0.02	98.31 ± 0.06	99.33 ± 0.03	96.08 ± 0.12
28	99.73 ± 0.01	98.16 ± 0.06	99.32 ± 0.03	95.92 ± 0.12
24	99.72 ± 0.01	98.08 ± 0.07	99.29 ± 0.03	95.79 ± 0.11
20	99.70 ± 0.01	97.99 ± 0.06	99.25 ± 0.04	95.59 ± 0.12
16	99.68 ± 0.02	97.86 ± 0.07	99.2 ± 0.04	95.32 ± 0.11
12	99.64 ± 0.01	97.64 ± 0.07	99.12 ± 0.05	94.94 ± 0.12
8	99.57 ± 0.02	97.23 ± 0.11	98.96 ± 0.05	94.16 ± 0.11
4	99.40 ± 0.04	96.27 ± 0.18	98.75 ± 0.07	92.99 ± 0.18
IArnoldi (NArnoldi = 100, R = 20)	61.64 ± 0.13	17.05 ± 0.18	77.20 ± 0.35	22.61 ± 0.42
ITracInRP (C = 5, R = 20)	89.56 ± 0.14	44.26 ± 0.37	74.99 ± 0.25	21.62 ± 0.26
To evaluate the effectiveness of the size of an ensemble in  $\mathcal{I}_{\mathrm{GEX}}$ , we conduct an ablation study for the number of fine-tuned models  $M_{\mathrm{GE}}$  for the noisy label detection task in Sec. 5.1 with  $10\%$  label corruption. Table 7 shows that even a small number of checkpoints (about 8) can achieve better performance of GEX than other state-of-the-art IF approximations. While we used 32 checkpoints to ensure a low variance of results, small checkpoints - as small as 4 - may be sufficient in situations requiring a low computational cost.
F.2 Results on MNIST and SVHN
Table 8: Noisy label detection performance on MNIST and SVHN
MNIST	SVHN
Detection method	AUC	AP	AUC	AP
Deep-KNN	97.28 ± 0.13	90.34 ± 0.41	93.07 ± 0.06	77.33 ± 0.36
CL	87.31 ± 0.07	58.16 ± 0.26	59.18 ± 0.21	11.04 ± 0.05
F-score	96.67 ± 0.03	57.24 ± 0.13	85.38 ± 0.07	25.42 ± 0.07
EL2N	99.99 ± 0.00	99.90 ± 0.02	99.70 ± 0.01	96.33 ± 0.04
\( \mathcal{I}_{\text{RandProj}} \)	70.90 ± 0.48	20.61 ± 0.46	58.82 ± 0.28	17.94 ± 0.19
\( \mathcal{I}_{\text{TracInRP}} \)	97.22 ± 0.05	71.22 ± 0.54	91.52 ± 0.16	47.25 ± 0.77
\( \mathcal{I}_{\text{Arnoldi}} \)	70.02 ± 0.44	19.88 ± 0.40	58.23 ± 0.48	17.71 ± 0.24
\( \mathcal{I}_{\text{GEX-lin}} \)	91.67 ± 0.23	62.93 ± 0.75	61.59 ± 0.18	18.95 ± 0.17
\( \mathcal{I}_{\text{GEX}} \)	99.99 ± 0.00	99.90 ± 0.02	99.74 ± 0.00	96.59 ± 0.06
Table 9: Relabeled test accuracy for mislabeled samples
MNIST	SVHN
Clean label acc.	98.93 ± 0.02	97.50 ± 0.01
Noisy label acc.	95.21 ± 0.04	94.84 ± 0.10
Detection method	Relabeled acc.
Deep-KNN	98.75 ± 0.04	95.15 ± 0.06
CL	64.57 ± 0.30	79.08 ± 0.05
F-score	97.75 ± 0.08	91.71 ± 0.10
EL2N	99.13 ± 0.04	95.86 ± 0.04
\( \mathcal{I}_{\text{RandProj}} \)	97.78 ± 0.07	94.89 ± 0.06
\( \mathcal{I}_{\text{TraciInR}} \)	98.08 ± 0.03	94.54 ± 0.06
\( \mathcal{I}_{\text{Arnoldi}} \)	97.69 ± 0.07	94.84 ± 0.03
\( \mathcal{I}_{\text{GEX-lin}} \)	95.37 ± 0.14	94.78 ± 0.11
\( \mathcal{I}_{\text{GEX}} \)	98.89 ± 0.03	96.32 ± 0.03
(b) SVHN
Figure 5: Data pruning results on MNIST and SVHN.
In this section, we provide the additional results of Sec. 5 for MNIST [33] and SVHN [42] with  $10\%$  label corruption. The overall tendency of MNIST and SVHN is similar to the results in Sec. 5. While EL2N is comparable (or even better) to  $\mathcal{I}_{\mathrm{GEX}}$  on MNIST, their good performance is not generalized to more scalable and practical settings in CIFAR-N [64], and ImageNet [12].
F.3 Results on DBpedia
Table 10: Noisy label detection performance on DBpedia
DBPedia
Detection method	AUC	AP
CL	86.49 ± 0.01	56.14 ± 0.04
F-score	45.18 ± 0.15	8.91 ± 0.01
EL2N	99.56 ± 0.00	95.26 ± 0.02
\(I_{RandProj}\)	98.42 ± 0.05	79.05 ± 0.90
\(I_{TracInRP}\)	98.42 ± 0.04	79.34 ± 0.60
\(I_{Arnoldi}\)	98.54 ± 0.02	80.69 ± 0.25
\(I_{GEX-lin}\)	97.10 ± 0.13	74.46 ± 1.14
\(I_{GEX}\)	99.79 ± 0.00	97.83 ± 0.03
In this section, we provide the additional results of Sec. 5.1 for DBPedia [34] with SWEM [53]. Table 10 shows that GEX better discriminates noisy labels than other methods. In addition to the robustness of various model types (including VGGNet, ResNet, ViT, and MLP-Mixer) and noise types (synthetic label noise and real-world label noise), this result indicates that GEX has consistently excellent performance for various data types.
F.4 Results on  $30\%$  label corruption
Table 11: Noisy label detection performance with  $30\%$  label corruption
CIFAR-10	CIFAR-100
Detection method	AUC	AP	AUC	AP
Deep-KNN	88.39 ± 0.57	77.94 ± 1.04	82.09 ± 0.23	64.01 ± 0.37
CL	82.28 ± 0.04	64.55 ± 0.06	78.36 ± 0.13	60.42 ± 0.18
F-score	64.40 ± 0.21	31.98 ± 0.10	53.61 ± 0.12	28.91 ± 0.05
EL2N	99.13 ± 0.01	97.81 ± 0.05	97.50 ± 0.03	93.27 ± 0.10
\( \mathcal{I}_{\text{RandProj}} \)	57.93 ± 0.51	33.36 ± 0.29	71.72 ± 0.01	46.33 ± 0.20
\( \mathcal{I}_{\text{TracInRP}} \)	84.08 ± 0.56	59.56 ± 0.85	68.30 ± 0.29	43.79 ± 0.37
\( \mathcal{I}_{\text{Arnoldi}} \)	57.26 ± 0.51	32.80 ± 0.28	69.34 ± 0.05	43.38 ± 0.23
\( \mathcal{I}_{\text{GEX-lin}} \)	58.82 ± 0.46	33.93 ± 0.26	69.36 ± 0.25	43.96 ± 0.21
\( \mathcal{I}_{\text{GEX}} \)	99.74 ± 0.01	98.19 ± 0.06	98.98 ± 0.01	97.84 ± 0.03
Table 12: Relabeled test accuracy for mislabeled samples
CIFAR-10	CIFAR-100
Clean label acc.	95.75 ± 0.06	79.08 ± 0.05
Noisy label acc.	79.81 ± 0.28	58.92 ± 0.04
Detection method	Relabeled acc.
Deep-KNN	82.57 ± 0.43	56.26 ± 0.08
CL	35.26 ± 0.02	30.25 ± 0.65
F-score	65.29 ± 0.57	44.13 ± 0.40
EL2N	88.07 ± 0.17	60.86 ± 0.13
\( I_{RandProj} \)	79.80 ± 0.26	58.88 ± 0.19
\( I_{TracInRP} \)	79.59 ± 0.23	58.50 ± 0.28
\( I_{Arnoldi} \)	80.08 ± 0.13	58.76 ± 0.32
\( I_{GEX\_lin} \)	79.72 ± 0.17	58.79 ± 0.10
\( I_{GEX} \)	89.88 ± 0.06	67.25 ± 0.07
In this section, we provide the additional results of Sec. 5.1-5.2 with  $30\%$  label corruption. Table 11-12 shows that  $\mathcal{I}_{\mathrm{GEX}}$  achieves better detection and relabeling performance than other methods on higher label noise settings. This is consistent with the empirical evaluation in Sec. 5 and shows the practicality of  $\mathcal{I}_{\mathrm{GEX}}$ .
F.5 Cross-influence results for noisy label detection
Table 13: AUC and AP for noisy label detection tasks on four datasets
MNIST	CIFAR-10	CIFAR-100	SVHN
Detection method	AUC	AP	AUC	AP	AUC	AP	AUC	AP
\( \mathcal{I}_{\text{RandProj}} \)	56.64 ± 1.92	87.42 ± 0.61	58.72 ± 0.40	36.35 ± 0.56	51.59 ± 0.47	37.51 ± 0.39	59.97 ± 1.62	37.21 ± 1.45
\( \mathcal{I}_{\text{TracInRp}} \)	57.09 ± 1.11	87.84 ± 0.31	60.74 ± 0.77	51.04 ± 1.11	51.88 ± 0.22	36.67 ± 0.35	64.10 ± 0.69	54.46 ± 1.49
\( \mathcal{I}_{\text{Arnoldi}} \)	76.09 ± 0.66	93.76 ± 0.17	77.64 ± 0.93	62.20 ± 2.00	53.64 ± 0.67	38.8 ± 0.39	80.85 ± 1.52	64.62 ± 2.84
\( \mathcal{I}_{\text{GEX}} \)	99.83 ± 0.03	99.83 ± 0.03	86.82 ± 1.24	81.98 ± 1.58	98.96 ± 0.01	97.69 ± 0.02	99.32 ± 0.19	97.51 ± 0.55
In this section, we apply the cross-influence on test dataset  $S_{\mathrm{test}}$  defined as
$$
\mathcal {I} (z, S _ {\mathrm {t e s t}}) := \frac {1}{| S _ {\mathrm {t e s t}} |} \sum_ {z ^ {\prime} \in S _ {\mathrm {t e s t}}} \mathcal {I} (z, z ^ {\prime})
$$
for the noisy label detection task in Appendix F.4. As the overfitting/memorization of mislabeled samples causes performance degradation for NNs, the estimated cross-influence values should be negative for these samples. Table 13 shows that IF approximations represent this tendency correctly. Mislabeled samples are best distinguished from clean samples using  $\mathcal{I}_{\mathrm{GEX}}$ .
F.6 TracInRP with Geometric Ensemble checkpoints
Table 14: Noisy label detection performance for  ${\mathcal{I}}_{\text{TracInRP }}$  with Geometric Ensemble checkpoints
MNIST	CIFAR-10	CIFAR-100	SVHN
Detection method	AUC	AP	AUC	AP	AUC	AP	AUC	AP
\( \mathcal{I}_{\text{RandProj}} \)	70.90 ± 0.48	20.61 ± 0.46	62.70 ± 0.19	17.90 ± 0.17	79.96 ± 0.32	26.25 ± 0.47	58.82 ± 0.28	17.94 ± 0.19
\( \mathcal{I}_{\text{TracInRP}} \)	94.37 ± 0.12	68.87 ± 0.97	89.56 ± 0.14	44.26 ± 0.37	74.99 ± 0.25	21.62 ± 0.26	91.52 ± 0.16	47.25 ± 0.77
\( \mathcal{I}_{\text{TracInRP}} \) with GE	99.78 ± 0.03	97.00 ± 0.47	90.96 ± 0.14	41.52 ± 0.20	88.84 ± 0.22	48.91 ± 0.85	85.35 ± 0.20	23.06 ± 0.23
\( \mathcal{I}_{\text{GEX}} \)	99.98 ± 0.00	99.87 ± 0.01	99.74 ± 0.02	98.31 ± 0.06	99.33 ± 0.03	96.08 ± 0.12	99.74 ± 0.00	96.59 ± 0.06
As discussed in Sec. 2,  $\mathcal{I}_{\mathrm{TracIn}}$  and  $\mathcal{I}_{\mathrm{TracInRP}}$  are not purely post-hoc as they require intermediate checkpoints during pre-training. Motivated by  $\mathcal{I}_{\mathrm{GEX}}$ , one can propose to replace the intermediate checkpoints of  $\mathcal{I}_{\mathrm{TracInRP}}$  with post-hoc checkpoints generated by Geometric Ensemble. In this section, we evaluate this setting for the noisy label detection task in Sec. 5.1. Table 14 shows that this modification consistently outperforms  $\mathcal{I}_{\mathrm{RandProj}}$ , which uses the last checkpoint  $\theta^{*}$  only. Also,  $\mathcal{I}_{\mathrm{TracInRP}}$  with GE performs better than the original  $\mathcal{I}_{\mathrm{TracInRP}}$  except SVHN. However, this improvement does not outperform  $\mathcal{I}_{\mathrm{GEX}}$ .
F.7 Further ImageNet results
Table 15: Relabeled accuracy for ImageNet [12]
ViT-S-32	Mixer-S-32
Clean acc.	67.83%	64.37%
Noisy acc.	63.42%	61.84%
Relabeled with EL2N	63.18%	63.16%
Relabeled with GEX	66.17%	63.45%
Table 16: Pruned acc. for ImageNet [12]
Mixer-S-32
Full sample acc.	67.83%
Pruned with EL2N	54.87%
Pruned with GEX	56.34%
Our first additional ImageNet experiment is the relabeling task presented in Sec. 5.2 on the ImageNet-1K environment with ViT and MLP-Mixer. To this end, we follow the relabeling process in Sec. 5.2 with the estimated influence in Table 2. As ViT and Mixer use multi-task binary classification loss instead of cross-entropy loss, the relabel function in (16) cannot be applied to this setting. To mitigate this issue, we use well-known distillation loss [21] for detected influential samples. Therefore, these influential samples receive a milder signal than from the hard label of the initial training. Table 15 presents the relabeled test accuracy for  $\mathcal{I}_{\mathrm{GEX}}$  and EL2N (the best method for noisy label detection except ours). It shows that  $\mathcal{I}_{\mathrm{GEX}}$  can detect mislabeled samples that require relabeling more accurately than EL2N. The second additional ImageNet experiment is the dataset pruning task in Sec. 5.3 on the ImageNet-1K with MLP-Mixer. For this purpose, we reproduce Mixer-S-32 and estimate the self-influence of  $\mathcal{I}_{\mathrm{GEX}}$  and EL2N score (which verified its scalability on the dataset pruning task in Sorscher et al. [58]). Then, we prune 512,466 samples ( $40\%$ ) with the smallest self-influence in ImageNet-1K and retrain neural networks with these pruned datasets. Table 16 presents the pruned test accuracy for  $\mathcal{I}_{\mathrm{GEX}}$  and EL2N. Similar to the results shown in Fig. 4,  $\mathcal{I}_{\mathrm{GEX}}$  demonstrates more effective identification of prunable samples than EL2N on the scalable ImageNet-1K dataset. Additionally, it is worth noting that EL2N cannot make use of open-source checkpoints and requires a computational cost of ( $10 \sim 20$  epochs)  $\times$  (number of checkpoints) from an initialized neural network. In summary, the better-pruned accuracy and the lower computational cost further illustrate the effectiveness of  $\mathcal{I}_{\mathrm{GEX}}$  in scalable settings.
G Limitations and Broader Impacts
While  $\mathcal{I}_{\mathrm{GEX}}$  effectively captures the multi-modal nature of  $\mathcal{I}_{\mathrm{L00}}$  's self-influence distribution, we do not delve into the theoretical guarantees, such as approximation bounds, for  $\mathcal{I}_{\mathrm{GEX}}$ . Nevertheless, we believe that this concern can be addressed in future research by making assumptions about the SGD steps used in generating checkpoints. Furthermore, although the data pruning results of  $\mathcal{I}_{\mathrm{GEX}}$  are significantly superior to other methods approximating IF, they do not outperform the performance achieved by state-of-the-art techniques like F-score and EL2N. This issue can be resolved by implementing the iterative pruning approach proposed in Harutyunyan et al. [19].
Our study examines the distributional bias present in current IF approximations and proposes an efficient solution. As a result, our approach has broad applicability in different scenarios. Specifically, our method offers enhanced efficiency in identifying mistrabeled and prunable samples compared to existing approaches. Consequently, we believe that our method can contribute to enhancing and expediting the training process of energy-intensive deep learning models.