SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[54] Thomas Simon. A multiplicative short proof for the unimodality of stable densities. Electronic Communications in Probability, 16:623-629, 2011. 16
[55] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015. 4, 20
[56] Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning, pages 5827-5837. PMLR, 2019. 2, 4, 16
[57] Nickolay Smirnov. Table for estimating the goodness of fit of empirical distributions. The annals of mathematical statistics, 19(2):279-281, 1948. 4
[58] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206.14486, 2022. 2, 9, 23
[59] Daniel Ting and Eric Brochu. Optimal subsampling with influence functions. Advances in neural information processing systems, 31, 2018. 2
[60] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34: 24261-24272, 2021. 9, 20
[61] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159, 2018. 7, 8
[62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017. 3
[63] Zifeng Wang, Hong Zhu, Zhenhua Dong, Xiuqiang He, and Shao-Lun Huang. Less is better: Unweighted data subsampling via influence function. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6340-6347, 2020. 2
[64] Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. arXiv preprint arXiv:2110.12088, 2021. 8, 20, 21
[65] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierrick Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. 3
[66] Makoto Yamazato. On strongly unimodal infinitely divisible distributions of class cme. Theory of Probability & Its Applications, 40(3):518-532, 1996. 16
[67] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization, 2017. 20
[68] Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvornek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. arXiv preprint arXiv:2203.08065, 2022. 20
Table 5: Notations used in the main paper
D	△	(true) data distribution
S := {zn : (xn, yn)}Nn=1	△	i.i.d. sampled training dataset from D
zn := (xn, yn)	△	a training instance (or sample)
xn ∈ R^D, yn ∈ R^K	△	training inputs/outputs
θ,ψ ∈ RP	△	parameter of NNs
ŷn = f(xn, θ)	△	output of NN given x, θ
l(z,θ)	△	instance-wise loss given sample z, θ
L(S,θ)	△	1/N ∑n=1N l(zn, θ)
θ* ∈ RP	△	pre-trained parameter of NN
θz ∈ RP	△	argminθ L(S, θ) - l(z, θ)/N (leave-one-out retrained parameter)
gz ∈ RP	△	∇θl(z, θ*) (gradient of instance-loss w.r.t. parameter given z, θ*)
H ∈ RP×P	△	∇θ^2L(S, θ*) (Hessian of training loss w.r.t. parameter given θ*)
ℓlinθ*(z,ψ)	△	l(z, θ*) + g_z^T(ψ - θ*) (linearization of instance-loss w.r.t. parameter)
Δℓlinθ*(z,ψ)	△	ℓlinθ*(z,ψ) - l(z, θ*)
Δℓθ*(z,ψ)	△	ℓθ*(z,ψ) - l(z, θ*)
0P ∈ RP	△	the P-dimensional zero vector
IP ∈ RP×P	△	the P-dimensional identity matrix
gcz ∈ RP	△	∇θl(z, θc) (gradient of instance-loss w.r.t. parameter for a checkpoint c)
QR ∈ RP×R	△	Random matrix sampled from N(0, 1/R) component-wisely
H = UΛU^T	△	Eigendecomposition of Hessian
H(α)	△	Damped Hessian matrix H + αIP
ΛR ∈ R^{R×R}, UR ∈ RP×R	△	Submatrices of the top-D eigenvalues/eigenvectors
λi ∈ R, ui ∈ RP	△	the i-th largest eigenvalue of H and its eigenvector
gz,i = gz^Tui ∈ R	△	i-th component of gz in the eigenspace of H
pLA(ψ)	△	N(ψ|θ*, H^-1) (the Laplace approximated posterior)
pGE(ψ)	△	1/M ∑m=1M δθ(m)(ψ) (the Geometric Ensemble distribution)
{θ(m)}Mm=1	△	checkpoints of Geometric Ensemble (GE)
Proposition B.1 (Distributional bias in bilinear self-influence). Let us assume  $g_{z}$  follows a  $P$ -dimensional stable distribution (e.g., Gaussian, Cauchy, and Lévy distribution) and  $M \in \mathbb{R}^{P \times P}$  is a positive (semi-)definite matrix. Then, self-influence in the form of  $\mathcal{I}_M(z,z) = g_z^\top Mg_z$  follows a unimodal distribution. Furthermore, if  $g_{z}$  follows a Gaussian distribution, then the self-influence follows a generalized  $\chi^2$ -distribution.
Proof. We first consider the case of gradients with stable distributions [56]. According to Corollary 1 in Mittnik et al. [41], the sum of squares of stable random variables is a stable distribution with the stability of  $\alpha /2$ . Since stable distributions are unimodal [66, 54], the resulting self-influence is also unimodal.
Now we proof the case of the Gaussian gradient. By positive semi-definite assumption, let us write the eigendecomposition of  $M$  as  $M = \sum_{i=1}^{P} \lambda_i u_i u_i^\top$  where eigenvalues are sorted in descending order. Specifically, one can compress this summation for the strictly positive eigenvalues:  $M = \sum_{i=1}^{T} \lambda_i u_i u_i^\top$  where  $T$  is the cardinality of strictly positive eigenvalues. Then the self-influence  $\mathcal{I}_M$  can be expressed as follows:
$$
\mathcal {I} _ {M} (z, z) = g _ {z} ^ {\top} M g _ {z} = \sum_ {i = 1} ^ {T} \lambda_ {i} g _ {z, i} ^ {2}
$$
where  $g_{z,i} = g_z^\top u_i$  is the  $i$ -th component of  $g_{z}$  in the eigenspace of  $M$ . As a linearly transformed Gaussian random variable follows Gaussian distribution [5], one can write  $g_{z,i}$  as follows
$$
g _ {z, i} \sim \mathcal {N} \left(g _ {z, i} \mid \mu_ {i}, \sigma_ {i} ^ {2}\right).
$$
Therefore,  $g_{z,i}$  follows following generalized  $\chi^2$ -distribution [8]
$$
g _ {z, i} ^ {2} \sim \tilde {\chi} ^ {2} (g _ {z, i} ^ {2} | \sigma_ {i} ^ {2}, 1, \mu_ {i} ^ {2}).
$$
where  $\mu_i^2$  is the non-centrality parameter and  $\sigma_i^2$  is the weight of non-central  $\chi^2$ . As a result,  $\mathcal{I}_M$  follows following generalized  $\chi^2$ -distribution
$$
\mathcal {I} _ {M} (z, z) \sim \tilde {\chi} ^ {2} (\mathcal {I} _ {M} (z, z) \mid \sigma^ {2}, \mathbf {1} _ {T}, \mu^ {2}).
$$
where  $\mu^2 = (\mu_1^2, \ldots, \mu_T^2) \in \mathbb{R}^T$ ,  $\sigma^2 = (\sigma_1^2, \ldots, \sigma_T^2) \in \mathbb{R}^T$ , and  $\mathbf{1}_T = (1, \ldots, 1) \in \mathbb{R}^T$ .
Theorem B.2 (Connection between IF and LA).  $\mathcal{I}$  in Koh and Liang [25] can be expressed as
$$
\mathcal {I} \left(z, z ^ {\prime}\right) = \mathbb {E} _ {\psi \sim p _ {\mathrm {L A}}} \left[ \Delta \ell_ {\theta^ {*}} ^ {\operatorname {l i n}} \left(z, \psi\right) \cdot \Delta \ell_ {\theta^ {*}} ^ {\operatorname {l i n}} \left(z ^ {\prime}, \psi\right) \right] \tag {17}
$$
where  $\Delta \ell_{\theta^{*}}^{\mathrm{lin}}(z,\psi)\coloneqq \ell_{\theta^{*}}^{\mathrm{lin}}(z,\psi) - \ell_{\theta^{*}}^{\mathrm{lin}}(z,\theta^{*}) = g_{z}^{\top}(\psi -\theta^{*})$  and  $p_{\mathrm{LA}}$  is the Laplace approximated posterior
Proof. One can express  $\Delta \ell_{\theta^{*}}^{\mathrm{lin}}(z,\psi)$  as follows
$$
\Delta \ell_ {\theta^ {*}} ^ {\operatorname {l i n}} (z, \psi) = \ell_ {\theta^ {*}} ^ {\operatorname {l i n}} (z, \psi) - \ell_ {\theta^ {*}} ^ {\operatorname {l i n}} (z, \theta^ {*}) = g _ {z} ^ {\top} (\psi - \theta^ {*})
$$
Therefore,  $\mathbb{E}_{\psi \sim p_{\mathrm{LA}}}\left[\Delta \ell_{\theta^{*}}^{\mathrm{lin}}(z,\psi)\cdot \Delta \ell_{\theta^{*}}^{\mathrm{lin}}(z^{\prime},\psi)\right]$  can be arranged as follows
$$
\begin{array}{l} \underset {\psi \sim p _ {\mathrm {L A}}} {\mathbb {E}} \left[ \Delta \ell_ {\theta^ {*}} ^ {\ln} (z, \psi) \cdot \Delta \ell_ {\theta^ {*}} ^ {\ln} \left(z ^ {\prime}, \psi\right) \right] = \underset {\psi \sim p _ {\mathrm {L A}}} {\mathbb {E}} \left[ g _ {z} ^ {\top} \left(\psi - \theta^ {*}\right) \left(\psi - \theta^ {*}\right) ^ {\top} g _ {z ^ {\prime}} \right] \\ = g _ {z} ^ {\top} \underset {\psi \sim p _ {\mathrm {L A}}} {\mathbb {E}} \left[ \left(\psi - \theta^ {*}\right) \left(\psi - \theta^ {*}\right) ^ {\top} \right] g _ {z ^ {\prime}} \\ = g _ {z} ^ {\top} H ^ {- 1} g _ {z ^ {\prime}} = \mathcal {I} (z, z ^ {\prime}). \\ \end{array}
$$
B.3 Proof of Proposition 4.2
Proposition B.3 (Singular Hessian for over-parameterized NNs). Let us assume a pre-trained parameter  $\theta^{*} \in \mathbb{R}^{P}$  achieves zero training loss  $L(S, \theta^{*}) = 0$  for squared error. Then,  $H$  has at least  $P - NK$  zero-eigenvalues for NNs such that  $NK < P$ . Furthermore, if  $x$  is an input of training sample  $z \in S$ , then the following holds for the eigenvectors  $\{u_{i}\}_{i = NK + 1}^{P}$
$$
g _ {z} ^ {\top} u _ {i} = \nabla_ {\hat {y}} ^ {\top} \ell \left(z, \theta^ {*}\right) \underbrace {\nabla_ {\theta} ^ {\top} f \left(x , \theta^ {*}\right) u _ {i}} _ {\mathbf {0} _ {K}} = 0 \tag {18}
$$
Proof. The Hessian matrix of training loss w.r.t. parameters can be expressed as follows [51]
$$
\begin{array}{l} H = \frac {1}{N} \sum_ {n = 1} ^ {N} \nabla_ {\theta} f (x _ {n}, \theta^ {*}) \nabla_ {\hat {y} _ {n}} ^ {2} \ell (z _ {n}, \theta^ {*}) \nabla_ {\theta} ^ {\top} f (x _ {n}, \theta^ {*}) + \nabla_ {\hat {y} _ {n}} \ell (z _ {n}, \theta^ {*}) \nabla_ {\theta} ^ {2} f (x _ {n}, \theta^ {*}) \\ = \frac {1}{N} \sum_ {n = 1} ^ {N} \nabla_ {\theta} f (x _ {n}, \theta^ {*}) \nabla_ {\hat {y} _ {n}} ^ {2} \ell (z _ {n}, \theta^ {*}) \nabla_ {\theta} ^ {\top} f (x _ {n}, \theta^ {*}) \\ = \frac {1}{N} \sum_ {n = 1} ^ {N} \nabla_ {\theta} f (x _ {n}, \theta^ {*}) \nabla_ {\theta} ^ {\top} f (x _ {n}, \theta^ {*}) \\ \end{array}
$$
where the first equality holds due to the zero training loss assumption
$$
\nabla_ {\hat {y} _ {n}} \ell (z _ {n}, \theta^ {*}) = \left(f (x _ {n}, \theta^ {*}) - y _ {n}\right) = \mathbf {0} _ {K} \quad \forall n = 1, \dots , N
$$
and the second equality holds due to the squared error assumption
$$
\nabla_ {\hat {y} _ {n}} ^ {2} \ell (z _ {n}, \theta^ {*}) = \mathbf {I} _ {K} \quad \forall n = 1, \ldots , N.
$$
Therefore,  $H$  is a sum of  $NK$  rank-one products. As a result,  $H$  has at most  $NK$  non-zero eigenvalues and at least  $P - NK$  zero-eigenvalues for over-parameterized NNs such that  $NK < P$ . As training Jacobians  $\nabla_{\theta}^{\top}f(x,\theta^{*})$  for  $n = 1,\dots ,N$  are contained in the linear span of  $\{u_i\}_{i = 1}^{NK}$ , which is orthogonal to  $\{u_i\}_{i = NK + 1}^P$ , the following holds for any training input  $x$
$$
\nabla_ {\boldsymbol {\theta}} ^ {\top} f (\boldsymbol {x}, \boldsymbol {\theta} ^ {*}) u _ {i} = \mathbf {0} _ {K}, \quad \forall u _ {i} \in \{u _ {i} \} _ {i = N K + 1} ^ {P}.
$$
B.4 Cross-entropy version of Proposition 4.2 and its proof
Proposition B.4 (Cross-entropy version of Proposition 4.2). Let us assume a pre-trained parameter  $\theta^{*}\in \mathbb{R}^{P}$  for cross-entropy loss. Then, the empirical Fisher (EF) matrix  $F = \frac{1}{N}\sum_{z\in S}g_{z}g_{z}^{\top}$  has at least  $P - N$  zero-eigenvalues for NNs such that  $N < P$ . Furthermore, the following holds for the last  $P - N$  eigenvectors  $\{u_i\}_{i = N + 1}^P$  and any training sample  $z\in S$
$$
g _ {z} ^ {\top} u _ {i} = 0 \tag {19}
$$
Proof. As  $F$  is a sum of  $N$  rank-one products. As a result,  $F$  has at most  $N$  non-zero eigenvalues and at least  $P - N$  zero-eigenvalues for over-parameterized NNs such that  $N < P$ . As gradients  $\nabla_{\theta}^{\top}\ell (z_n,\theta^*)$  for  $n = 1,\ldots ,N$  are contained in the linear span of  $\{u_i\}_{i = 1}^N$ , which is orthogonal to  $\{u_i\}_{i = N + 1}^P$ , the following holds for any training input  $x$
$$
g _ {z} u _ {i} = 0, \quad \forall u _ {i} \in \{u _ {i} \} _ {i = N + 1} ^ {P}.
$$
C Implementations
C.1 Pseudocode of LA for IF approximation
Algorithm 1  $\mathcal{I}_{\mathrm{LA}}$  with KFAC [39, 17] sub-curvature approximation
1: Input: training data  $S$ , pre-trained parameter  $\theta^{*}$ , number of LA samples  $M$ , two data samples  $z, z'$   
2:  
3: # Estimation of KFAC sub-curvature  
4: Initialize activation statistics  $A^l$  and gradient statistics  $G^l$  for each layer  $l = 1, \dots, L$   
5: for  $n = 1, \dots, N$  (This computation can be conducted batch-wisely.) do  
6: for  $l = 1, \dots, L$  do  
7: Accumulate activation statistics  $A^l$  and gradient statistics  $G^l$   
8: end for{End accumulation for layers}  
9: end for{End accumulation for training samples}  
10: Compute eigendecomposition of activation/gradient statistics:  
 $A^l = U_{Al} \Lambda_{Al} U_{Al}$ ,  $G^l = U_{Gl} \Lambda_{Gl} U_{Gl}$ .  
11:  
12: # Sample Laplace approximated posteriors  
13: for  $m = 1, \dots, M$  do  
14: Sample  $P$ -dimensional standard Gaussian random variable  $v_m \sim \mathcal{N}(v \mid 0_P, I_P)$   
15: for  $l = 1, \dots, L$  do  
16: Use reparameterization trick for each layer:  
 $\psi_m^l \gets (\theta^*)^l + U_{Gl} \left(v_m^l \odot \sqrt{(\mathrm{diag}(\Lambda_{Gl}) \mathrm{diag}(\Lambda_{Al})^\top}\right) U_{Al}^\top$  where  $\mathrm{diag}(A)$  is the diagonal components of matrix  $A$ ,  $\odot$  denotes Hadamard product of two matrices, and the square root is applied for each component of the matrix.  
17: end for  
18: end for{End LA sampling  $\{\psi_m\}_{m=1}^M$   
19:  
20: # Compute MC estimator of non-linear IF approximation (12)  
21:  $\hat{\mathcal{I}}_{\mathrm{LA}}(z, z') \gets \sum_{m=1}^{M} [\Delta \ell_{\theta^*}(z, \psi_m) \cdot \Delta \ell_{\theta^*}(z', \psi_m)]$   
22:  
23: Output:  $\hat{\mathcal{I}}_{\mathrm{LA}}(z, z')$
C.2 Pseudocode of GEX
Algorithm 2  $\mathcal{T}_{\mathrm{GEX}}$
1: Input: training data  $S$ , pre-trained parameter  $\theta^{*}$ , number of LA samples  $M$ , number of fine-tuning steps  $T$ , two data samples  $z, z'$   
2:  
3: # Generating Geometric Ensemble (GE; [16])  
4: for  $m = 1, \ldots, M$  (This computation can be parallelized for multiple devices) do  
5: Initialized the  $m$ -th checkpoint  $\theta_{m}^{0} \gets \theta^{*}$  (or  $\theta_{m-1}^{T}$ )  
6: for  $t = 1, \ldots, T$  do  
7: Apply stochastic optimization update (e.g., SGD with momentum):  $\theta_{m}^{t} \gets \theta_{m}^{t-1}$   
8: end for{End fine-tuning the  $m$ -th checkpoint  $\theta_{m}^{T}$ }  
9: end for{End generation of GE  $\{\theta_{m}^{T}\}_{m=1}^{M}$ }  
10:  
11: # Compute the non-linear IF approximation (10)  
12:  $\hat{\mathcal{I}}_{\mathrm{GEX}}(z, z') \gets \sum_{m=1}^{M} [\Delta \ell_{\theta^{*}}(z, \theta_{m}^{T}) \cdot \Delta \ell_{\theta^{*}}(z', \theta_{m}^{T})]$   
13:  
14: Output:  $\hat{\mathcal{I}}_{\mathrm{GEX}}(z, z')$
D Complexity analysis
Table 6: Comparison of computational complexity between IF approximations.  $P$  and  ${P}_{\text{Last }}$  are the entire parameter dimension and the last-layer parameter dimension,respectively.  $N$  is the number of training samples and  $R$  is the projection dimension used in  ${\mathcal{I}}_{\text{Arnoldi }}$  and  ${\mathcal{I}}_{\text{TracInRP }}$  .  $C$  is the number of checkpoints used in  ${\mathcal{I}}_{\text{TracIn }}$  and  ${\mathcal{I}}_{\text{TracInRP }}$  and  $T$  is the number of SGD steps used in  ${\mathcal{I}}_{\mathrm{{GEX}}}$  (See Appendix C).  ${M}_{\mathrm{{GE}}}$  and  ${M}_{\mathrm{{LA}}}$  are the number of samples used in Geometric Ensemble and Laplace approximation,respectively.  ${C}_{\text{forward }},{C}_{\text{backward }}$  ,and  ${C}_{\text{jvp }}$  are the constant for the single computation of forward,backward,and JVP.  ${C}_{\text{forward }}$  and  ${C}_{\text{jvp }}$  are similar for packages that offer forward-mode AD computation and  ${C}_{\text{backward }}$  is much slower than the others.
Time complexity for cache	Space complexity for cache	Time complexity for evaluation
I LiSSA	O(NLiSSA·Cbackward·P·N)	O(P·N)	O(Cbackward·P)
I Last-Layer	O(N·P3Last)	O(P2Last)	O(Cbackward·P)
I TracIn	Requires intermediate ckpts during pre-training	O(C·P)	O(C·Cbackward·P)
I TracInRP	Requires intermediate ckpts during pre-training	O(C·P)	O(C·R·Cjvp·P)
I Arnoldi	O(NArnoldi·Cbackward·P)	O(NArnoldi·P)	O(R·Cjvp·P)
I LA with KFAC	O(CKFAC·P·N)	O(MLA·P)	O(MLA·Cforward·P)
I GEX-lin	O(T·MGE·Cbackward·P)	O(MGE·P)	O(MGE·R·Cjvp·P)
I GEX	O(T·MGE·Cbackward·P)	O(MGE·P)	O(MGE·Cforward·P)
In Table 6, we compare computational complexities of various IF approximations, including  $\mathcal{I}_{\mathrm{GEX}}$ . We provide three computational complexities of each IF approximation method: time and space complexity for cache construction and the time complexity for evaluation. As discussed in Sec. 4.3, all IF approximations except  $\mathcal{I}_{\mathrm{TracIn}}$  and  $\mathcal{I}_{\mathrm{TracInRP}}$  can be implemented using only the final checkpoint, which is usually publicly released. However, the time and space complexity of cache construction are significantly different between the post-hoc methods: As mentioned in Sec. 2, the time and space complexity of  $\mathcal{I}_{\mathrm{LiSSA}}$  for cache construction are proportional to the size of the training dataset  $N$ , as  $\mathcal{I}_{\mathrm{LiSSA}}$  requires IHVP computation for each training sample. Therefore,  $\mathcal{I}_{\mathrm{LiSSA}}$  cannot be used in datasets with many training samples. On the other hand,  $\mathcal{I}_{\mathrm{Last-Layer}}$  requires high space complexity proportional to the square of the number of last-layer parameters  $P_{\mathrm{Last}}$ . As a result,  $\mathcal{I}_{\mathrm{Last-Layer}}$  cannot be applied to classification tasks with many classes (e.g., ImageNet [12]). While  $\mathcal{I}_{\mathrm{Arnoldi}}$  avoid this issue by using the principal subspace of  $H$ , we found that they suffer from the OOM issue on GPU RAM, as eigendecomposition computation (the DISTILL procedure in Schioppa et al. [52]) requires the space complexity proportional to  $N_{\mathrm{Arnoldi}}$ , the number of steps in  $\mathcal{I}_{\mathrm{Arnoldi}}$ , which is about 100-200 in Schioppa et al. [52]. However,  $\mathcal{I}_{\mathrm{GEX}}$  and  $\mathcal{I}_{\mathrm{GEX-lin}}$  does not suffer from the high time and space complexity for cache construction, as the cache construction procedure of them is just post-hoc fine-tuning in Garipov et al. [16].
For the time complexity for evaluation, we assume each method computes a single influence pair (e.g.,  $\mathcal{I}(z,z^{\prime})$  or  $\mathcal{I}(z,z)$ ). According to Table 6, all methods except  $\mathcal{I}_{\mathrm{LA}}$  and  $\mathcal{I}_{\mathrm{GEX}}$  require either backward computation ( $C_{\mathrm{backward}}$ ) or JVP computation ( $C_{\mathrm{jvp}}$ ). Among them, methods that require backward computation ( $\mathcal{I}_{\mathrm{LiSSA}}, \mathcal{I}_{\mathrm{Last-Layer}},$  and  $\mathcal{I}_{\mathrm{TracIn}}$ ) do not support batch computation for multiple samples, as mentioned in Sec. 2. Therefore, computing self-influence for all training samples in practice is inefficient. On the other hand, methods that require JVP computation ( $\mathcal{I}_{\mathrm{TracInRP}}, \mathcal{I}_{\mathrm{Arnoldi}},$  and  $\mathcal{I}_{\mathrm{GEX-lin}}$ ) are efficient for packages that offer forward-mode AD computation (e.g., JAX [6]). On the contrary,  $\mathcal{I}_{\mathrm{LA}}$  and  $\mathcal{I}_{\mathrm{GEX}}$  do not suffer from these issues, as they require only forward computation.
E Experimental settings
In the main paper, we conducted empirical studies in four different settings. We provide details on these settings in this section. We use four random seeds to compute the standard errors for all experiments in the main paper. We use 8 NVIDIA RTX 3090 GPUs for all experiments.
Two-circle classification task We use a modified two-circle classification task to study the side effects of damping and truncation in Sec. 4. We use sklearn.datasets.make_circles function to sample the training samples. We use 30 train samples for each class (circle) and add ten influential samples at the center. We use a two-layer fully connected NN with a hidden dimension of 200. We use jax.hessian function to compute the Hessian matrix of training loss and np.linalg.eigh to compute the principal subspace of  $H^{-1}$ .
Detecting and relabeling of mistrabeled samples Noisy label settings [67] are used to study the distributional bias of the standard bilinear IF approximations. We use standard datasets in the computer vision domain: MNIST [33], CIFAR-10/100 [27], and SVHN [42]. For MNIST, we use VGGNet-B [55] with batch normalization [23] after each convolution layer, similar to pytorch implementation in https://pytorch.org/vision/main/models/vgg.html. We use ResNet-18 [20] for CIFAR and SVHN. We use random label corruption for  $10\%$  of randomly selected training samples for synthetic noise and CIFAR-N [64] for real-world label noise. We use a batch size of 1024 for all datasets. For all datasets, we use the cosine learning rate annealing [36] with a linear warm-up where the peak learning rate is 0.4, and the warm-up ratio (percentage of warm-up steps to training steps) is  $10\%$ . We use 0.0005 for the L2 regularization coefficient. We use 200 training epochs for all datasets.
To verify the scalability of our method, we also use ImageNet [12] with Vision Transformer [13] and MLP-Mixer [60]. We use the "S-32" setting due to the heavy computation of ImageNet. We use random label corruption for  $10\%$  of randomly selected training samples. We use a batch size 4096 and RandomResizedCrop, following Zhuang et al. [68]. We use the cosine learning rate annealing with a linear warm-up where the peak learning is 0.003 and the warm-up step is 10,000. We use AdamW [37] with 0.3 for the weight decay coefficient. We train 300 epochs for ImageNet, following Zhuang et al. [68].
To verify the effectiveness of our method in another modality, we train SWEM [53] to DBpedia [34], following Pruthi et al. [48]. However, Pruthi et al. [48] used this setting for a clean label case study, while we used it for noisy label detection with  $10\%$  label corruption. For text representation, we use SentencePiece [28] tokenizer and apply average pooling of word embeddings. We train 60 epochs using the AdamW [37] optimizer with a peak learning rate of 0.001 and a weight decay of 0.001. We performed a grid search on the search space mentioned in [53] to find these hyperparameters, aiming to achieve the  $95.5\%$  train accuracy mentioned in [48]. We provide our results in Appendix F. Deep-KNN was omitted because it took more than 15 hours to calculate for each random seed.
We use 20 dimensions for all projection-based methods (i.e.,  $\mathcal{I}_{\mathrm{RandProj}}$ ,  $\mathcal{I}_{\mathrm{TracInRP}}$ ,  $\mathcal{I}_{\mathrm{Arnoldi}}$ ). We use 100 iterations for  $\mathcal{I}_{\mathrm{Arnoldi}}$ . Five checkpoints, evenly sampled from the latter half of training epochs, are used for  $\mathcal{I}_{\mathrm{TracInRP}}$  and Deep-KNN. We use  $10\%$  of training samples to compute the neighborhoods in Deep-KNN. For CL, we use two-fold cross-validation. Thus, CL requires at least a separate pretraining level of computational cost. We use 20 initial training epochs for EL2N with five repetitions. We use 32 samples for  $\mathcal{I}_{\mathrm{LA}}$  and  $\mathcal{I}_{\mathrm{GEX}}$ . We use a cosine learning rate of 0.05 with 800 steps for the fine-tuning procedure to generate GE. We provide an ablation study exploring the impact of ensemble size in Appendix F.