SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here, the training loss represents the negative log-likelihood  $L(S,\theta) = -\log p(S|\theta)$ , and  $Z := \int p(\theta)p(S|\theta)d\theta$  represents the evidence in Bayesian inference [5]. Similar to the IF, LA becomes computationally intractable when dealing with modern architectures due to the complexity of the Hessian matrix. To address this computational challenge, recent works have proposed various sub-curvature approximations, such as KFAC [50] and sub-network [11], which provide computationally efficient alternatives for working with LA.
According to (11), samples with a high degree of self-influence experience significant (linearized) loss changes for the parameters sampled from LA. Furthermore, Theorem 4.1 reveals that the gradients, the origin of distributional bias, arise from the linearizations in (11). Hence, to address the distributional bias of the bilinear IF approximations in Proposition 3.1, we remove the linearizations in (11). This leads us to consider a modified version of  $\mathcal{I}$ , named  $\mathcal{I}_{\mathrm{LA}}$ :
$$
\mathcal {I} _ {\mathrm {L A}} \left(z, z ^ {\prime}\right) := \mathbb {E} _ {\psi \sim p _ {\mathrm {L A}}} \left[ \Delta \ell_ {\theta^ {*}} \left(z, \psi\right) \cdot \Delta \ell_ {\theta^ {*}} \left(z ^ {\prime}, \psi\right) \right]. \tag {12}
$$
The pseudocode for computing  $\mathcal{I}_{\mathrm{LA}}$  using Kronecker-Factored Approximate Curvature (KFAC; [40]) is available in Appendix C.
While  $\mathcal{I}_{\mathrm{LA}}$  no longer theoretically produces unimodal self-influence, it turns out that it still mixes correct and mistrabeled samples, even in toy datasets. To illustrate this problem, we train a fully-connected NN on the two-circle dataset [29] with a modification as shown in Fig. 2(a): We add ten influential training samples at the center of the two-circle dataset containing 30 train samples per class (circle). These influential samples are highly susceptible (i.e., hard to recover) to leave-one-out retraining as samples of the opposite class are densely located around them. We provide other experimental details in Appendix E.
As discussed in Sec. 3, highly influential samples form a separate mode of high self-influence in the histogram of  $\mathcal{I}_{\mathrm{L00}}$  (Fig. 2(b)) and  $\mathcal{I}$  mixes typical and influential samples (Fig. 2(c)) due to the distributional bias of bilinear form. While Proposition 3.1 does not apply to  $\mathcal{I}_{\mathrm{LA}}$ , Fig. 3 shows that  $\mathcal{I}_{\mathrm{LA}}$  still fails to distinguish between correct and mislabeled samples by severely overestimating the self-influence of typical samples (Fig. 3(a)-3(b)) or squeeze them into a unimodal distribution (Fig. 3(c)), similar to  $\mathcal{I}$  (Fig. 3(d)).
4.2 Pitfalls of inverse Hessian in non-linear IF approximation
To analyze the limitation of  $\mathcal{I}_{\mathrm{LA}}$ , we reparameterize the sampled parameter from LA as follows
$$
\psi = \theta^ {*} + H ^ {- 1 / 2} v = \theta^ {*} + \sum_ {i = 1} ^ {P} \frac {v _ {i}}{\sqrt {\lambda_ {i}}} u _ {i} \tag {13}
$$
From this reparameterization, one can see that for (13) to be valid, all eigenvalues of  $H$  must be positive definite (i.e., all eigenvalues are positive), as Koh and Liang [25] assumed. However, over-parameterized NNs in practice can contain many zero eigenvalues in their Hessian.
Proposition 4.2 (Hessian singularity for over-parameterized NNs). Let us assume a pre-trained parameter  $\theta^{*} \in \mathbb{R}^{P}$  achieves zero training loss  $L(S, \theta^{*}) = 0$  for squared error. Then,  $H$  has at least  $P - NK$  zero-eigenvalues for NNs such that  $NK < P$ . Furthermore, if  $x$  is an input of training sample  $z \in S$ , then the following holds for the eigenvectors  $\{u_{i}\}_{i = NK + 1}^{P}$
$$
g _ {z} ^ {\top} u _ {i} = \nabla_ {\hat {j}} ^ {\top} \ell \left(z, \theta^ {*}\right) \underbrace {\nabla_ {\theta} ^ {\top} f \left(x , \theta^ {*}\right) u _ {i}} _ {\mathbf {0} _ {K}} = 0 \tag {14}
$$
We refer to Appendix B for the cross-entropy version of Proposition 4.2 with empirical Fisher (EF) matrix, a well-known approximation of the Hessian [31, 22], defined as  $F \coloneqq \frac{1}{N} \sum_{n=1}^{N} g_z g_z^\top$ . Note that the over-parameterization assumption ( $NK < P$ ) in Proposition 4.2 is prevalent in real-world situations, including the settings of Fig. 1-3. Also, the zero training loss assumption can be satisfied with sufficient over-parameterization [14, 2]. Accordingly, empirical evidences of Proposition 4.2 have been reported in different scenarios [51, 18].
A simple solution to mitigate this singularity is adding an isotropic matrix to  $H$ , known as the damping technique:  $H \approx H(\alpha) \coloneqq H + \alpha \mathbf{I}_P$  for  $\alpha > 0$ . Then, the modified LA sample is
$$
\psi_ {\alpha} := \theta^ {*} + (H (\alpha)) ^ {- 1 / 2} v = \theta^ {*} + \sum_ {i = 1} ^ {P} \frac {v _ {i}}{\sqrt {\lambda_ {i} + \alpha}} u _ {i}. \tag {15}
$$
Although the damping trick can make all eigenvalues positive in principle, the order of the eigenvalues remains unchanged. Therefore, the null space components  $(g_{z,i}$  for  $i = NK + 1,\dots ,P)$  in Proposition 4.2 are the most heavily weighted by  $1 / \sqrt{\alpha}$ . These heavily weighted null space components do not affect the linearized sample-loss deviations in (11) as follows
$$
\Delta \ell_ {\theta^ {*}} ^ {\mathrm {l i n}} (z, \psi_ {\alpha}) = g _ {z} ^ {\top} (\psi_ {\alpha} - \theta^ {*}) = \sum_ {i = 1} ^ {P} \frac {v _ {i}}{\sqrt {\lambda_ {i} + \alpha}} g _ {z} ^ {\top} u _ {i} = \sum_ {i = 1} ^ {N K} \frac {v _ {i}}{\sqrt {\lambda_ {i} + \alpha}} g _ {z} ^ {\top} u _ {i}
$$
by Proposition 4.2. However, this is not applicable to the sample-loss deviations in (12), since the null space components cause  $\Delta \ell_{\theta^*}(z,\psi_\alpha)$  to change rapidly (i.e., overestimating the influence of samples). This can be confirmed by observing that setting  $\alpha = 0.01$  in  $\mathcal{I}_{\mathrm{LA}}$  (Fig. 3(a)) leads to significantly overestimating self-influences. To prevent this issue, one can enlarge  $\alpha$  (i.e., decrease  $1 / \alpha$ ). However, since the scale of  $\alpha$  does not change the order of eigenvalues, the null space components still receive a higher weight than the others. Consequently,  $\mathcal{I}_{\mathrm{LA}}$  with  $\alpha = 100$  (Fig. 3(b)) does not correctly represent the multi-modality of self-influence distribution.
Another way to handle the Hessian singularity is to limit (13) to only the top- $R$  eigenvalues of Hessian, similar to  $\mathcal{I}_{\mathrm{Arnoldi}}$ . However, this approximation method may result in significant errors due to using only the smallest  $R$  eigenvalues of the inverse Hessian (i.e.,  $1 / \sqrt{\lambda_i}$ ). Consequently, even linearized sample-loss deviations (i.e.,  $\mathcal{I}$ ) with  $R = 50$  (Fig. 3(d)) suffer from severe performance degradation, compared to full Hessian with damping (Fig. 2(c)).
Compared to damping and truncation,  $\mathcal{I}_{\mathrm{GEX}}$  (Fig. 2(d)) accurately captures the bimodal self-influence distribution. This effectiveness stems from the differential impact of SGD steps on typical and influential samples [61]: Since typical samples are robust to SGD steps [35], loss deviation of these samples are small in  $\mathcal{I}_{\mathrm{GEX}}$ . In contrast, influential samples experience significant loss deviations as they are sensitive to SGD steps. As GE makes diverse predictions [16], the members of GE introduce varying levels of loss deviation for each influential sample. An ablation study exploring the impact of ensemble size is provided in Appendix F.
4.3 Practical advantages of GEX
In addition to the above benefits,  $\mathcal{I}_{\mathrm{GEX}}$  offers several implementation advantages. First, it can be obtained using only open-source final checkpoints, unlike  $\mathcal{I}_{\mathrm{TracIn}}$  and  $\mathcal{I}_{\mathrm{TracInRP}}$ . This advantage broadens the range of applicable models. Second, batch estimation is easy to implement in  $\mathcal{I}_{\mathrm{GEX}}$ . Projection-based methods like  $\mathcal{I}_{\mathrm{TracInRP}}$  and  $\mathcal{I}_{\mathrm{Arnoldi}}$  require JVP computation, which is only efficient
Table 1: Area Under Curve (AUC) and Average Precision (AP) for noisy label detection tasks on four datasets. Due to the high time complexity associated with sample-wise gradients, we do not repeatedly measure the self-influence of  $\mathcal{I}_{\mathrm{TracIn}}$ .
Synthetic label noise	Real-world label noise
CIFAR-10	CIFAR-100	CIFAR-10-N	CIFAR-100-N
Detection method	AUC	AP	AUC	AP	AUC	AP	AUC	AP
Deep-KNN	92.51 ± 0.19	69.93 ± 0.71	84.00 ± 0.14	40.17 ± 0.23	78.32 ± 0.19	72.60 ± 0.33	71.59 ± 0.21	59.76 ± 0.25
CL	57.60 ± 0.30	16.27 ± 0.20	84.16 ± 0.10	35.76 ± 0.50	75.94 ± 0.02	66.50 ± 0.09	69.49 ± 0.15	58.69 ± 0.23
F-score	73.34 ± 0.07	16.27 ± 0.09	59.18 ± 0.21	11.04 ± 0.05	69.39 ± 0.06	52.89 ± 0.06	68.95 ± 0.11	52.29 ± 0.14
EL2N	98.29 ± 0.03	95.82 ± 0.06	96.42 ± 0.05	73.82 ± 0.42	93.57 ± 0.17	91.26 ± 0.13	84.65 ± 0.08	77.26 ± 0.06
\( \mathcal{I}_{\text{RandProj}} \)	62.70 ± 0.19	17.90 ± 0.17	79.96 ± 0.32	26.25 ± 0.47	56.75 ± 0.38	45.61 ± 0.38	67.25 ± 0.09	54.14 ± 0.09
\( \mathcal{I}_{\text{TracIn}} \)	89.89	43.21	75.53	22.25	76.48	64.69	68.91	55.86
\( \mathcal{I}_{\text{TracInRP}} \)	89.56 ± 0.14	44.26 ± 0.37	74.99 ± 0.25	21.62 ± 0.26	77.24 ± 0.45	65.17 ± 0.68	69.04 ± 0.28	56.41 ± 0.31
\( \mathcal{I}_{\text{Arnoldi}} \)	61.64 ± 0.13	17.05 ± 0.18	77.20 ± 0.35	22.61 ± 0.42	56.83 ± 0.40	45.63 ± 0.40	66.57 ± 0.12	53.26 ± 0.11
\( \mathcal{I}_{\text{GEX-lin}} \)	64.11 ± 0.34	18.34 ± 0.36	76.06 ± 0.36	22.26 ± 0.47	56.88 ± 0.29	45.67 ± 0.33	65.68 ± 0.15	52.66 ± 0.13
\( \mathcal{I}_{\text{GEX}} \)	99.74 ± 0.02	98.31 ± 0.06	99.33 ± 0.03	96.08 ± 0.12	96.20 ± 0.03	94.89 ± 0.04	89.76 ± 0.01	86.30 ± 0.01
for packages that provide forward-mode auto-differentiation, such as JAX [6]. In contrast, batch computation in  $\mathcal{I}_{\mathrm{GEX}}$  necessitates only forward computations, which are efficient in almost all auto-differentiation packages. Since IF is computed for each training sample in most downstream tasks [25, 48, 52], efficient batching is critical for practical applications. As a result, we believe that this distinction will be vital for researchers and practitioners in their work. We provide the complexity analysis of  $\mathcal{I}_{\mathrm{GEX}}$  and other IF approximations in Appendix D and discuss the limitations of our method and broader impacts in Appendix G.
Here we describe experiments demonstrating the usefulness of GEX in downstream tasks. We conducted two experiments for each noisy and clean label setting: Detection (Sec. 5.1) and relabeling (Sec. 5.2) of unlabeled examples for noisy label setting and dataset pruning (Sec. 5.3) and separating data sources (Sec. 5.4) for clean label setting. We refer to Appendix E for experimental details.
5.1 Noisy label detection
In this section, we evaluate the performance of IF approximation methods for detecting noisy labels. We use self-influence as an index for the noisy label of IF approximation methods following Koh and Liang [25] and Pruthi et al. [48]. A high degree of self-influence indicates mislabeled examples since the removal of mislabeled examples will significantly change prediction or loss.
We train ResNet-18 [20] on CIFAR-10/100 [27] with  $10\%$  random label corruption for synthetic noise and CIFAR-N [64] for real-world noise. We use the "worse label" version of CIFAR-10-N since it corresponds to the highest noise level. We compare  $\mathcal{I}_{\mathrm{GEX}}$  to the following baselines:  $\mathcal{I}_{\mathrm{TracInRP}}$  with 5 checkpoints and 20 random projections,  $\mathcal{I}_{\mathrm{RandProj}}$  with 20 random projections only for the final checkpoint,  $\mathcal{I}_{\mathrm{Arnoldi}}$  with 100 iterations and 20 projections. We assess F-score [61], and EL2N [46] as they offer alternative approaches to identifying influential samples. We also evaluate an ablative version of  $\mathcal{I}_{\mathrm{GEX}}$ :  $\mathcal{I}_{\mathrm{GEX - lin}}$ , which replaces LA with GE for (11) (i.e.,  $\mathcal{I}_{\mathrm{GEX}}$  with linear sample-loss deviation). Finally, we report the performance of well-known baselines, Deep-KNN [4] and CL [43], for comparison. We provide results for cross-influence  $\mathcal{I}(z,z^{\prime})$  in case of  $z\neq z^{\prime}$  in Appendix F.
Table 1 shows that  $\mathcal{I}_{\mathrm{GEX}}$  distinguishes mislabeled samples better than other methods. Furthermore, we find that  $\mathcal{I}_{\mathrm{GEX}}$  is more effective than CL, a state-of-the-art noisy label detection technique. Since CL requires additional  $K$ -fold cross-validation for sample scoring ( $K = 2$  used in our experiments),  $\mathcal{I}_{\mathrm{GEX}}$  will be an attractive alternative in terms of both computational complexity and detection performance. Notably, results of  $\mathcal{I}_{\mathrm{GEX - lin}}$  demonstrate that removing linearization is essential for performance improvement of  $\mathcal{I}_{\mathrm{GEX}}$ .
One interesting observation in Table 1 is that  $\mathcal{I}_{\mathrm{Arnoldi}}$  did not show improvement compared to simple  $\mathcal{I}_{\mathrm{RandProj}}$ . Indeed, a similar observation was reported in Table 1 in Schioppa et al. [52]. In contrast,  $\mathcal{I}_{\mathrm{TracInRP}}$  showed improvements in CIFAR-10 and SVHN compared to  $\mathcal{I}_{\mathrm{RandProj}}$ ,  $\mathcal{I}_{\mathrm{TracInRP}}$  is computationally expensive since it requires JVP computations for multiple checkpoints. On the other hand,  $\mathcal{I}_{\mathrm{GEX}}$  can be applied in a pure post-hoc style and does not require (framework-dependent)
Table 2: Noisy label detection performance for ImageNet [12]
ViT-S-32	MLP-Mixer-S-32
Detection method	AUC	AP	AUC	AP
F-score	1.56	9.44	1.07	9.21
EL2N	87.28	38.37	88.47	41.63
\(I_{RandProj}\)	76.09	12.08	82.36	21.86
\(I_{TracInRP}\)	72.22	15.09	75.01	16.62
\(I_{Arnoldi}\)	73.95	16.09	82.16	21.83
\(I_{GEX}\)	99.39	95.22	98.73	90.65
Table 3: Relabeled test accuracy for mislabeled samples
Synthetic label noise	Real-world label noise
CIFAR-10	CIFAR-100	CIFAR-10-N	CIFAR-100-N
Clean label acc.	95.75 ± 0.06	79.08 ± 0.05	95.75 ± 0.06	79.08 ± 0.05
Noisy label acc.	90.94 ± 0.15	72.35 ± 0.17	68.63 ± 0.32	55.50 ± 0.09
Detection method	Relabeled acc.
Deep-KNN	91.58 ± 0.10	66.12 ± 0.27	69.12 ± 0.25	50.03 ± 0.19
CL	91.11 ± 0.10	72.55 ± 0.13	30.52 ± 0.02	33.17 ± 0.02
F-score	78.94 ± 0.39	58.67 ± 0.18	53.50 ± 0.28	44.34 ± 0.21
EL2N	89.40 ± 0.10	61.72 ± 0.18	72.01 ± 0.51	47.58 ± 0.22
\( \mathcal{I}_{\text{RandProj}} \)	90.94 ± 0.09	72.42 ± 0.16	68.55 ± 0.17	55.47 ± 0.08
\( \mathcal{I}_{\text{TracIn}} \)	91.24	72.07	68.36	54.87
\( \mathcal{I}_{\text{TracInRP}} \)	90.82 ± 0.06	71.70 ± 0.15	68.12 ± 0.23	55.20 ± 0.06
\( \mathcal{I}_{\text{Arnoldi}} \)	91.10 ± 0.09	72.50 ± 0.08	68.67 ± 0.02	55.37 ± 0.14
\( \mathcal{I}_{\text{GEX-lin}} \)	91.04 ± 0.16	70.08 ± 0.12	68.44 ± 0.08	55.51 ± 0.21
\( \mathcal{I}_{\text{GEX}} \)	93.54 ± 0.05	75.04 ± 0.10	73.94 ± 0.24	57.13 ± 0.10
JVP computation. Motivated by  $\mathcal{I}_{\mathrm{GEX}}$ , one can propose a purely post-hoc  $\mathcal{I}_{\mathrm{TracInRP}}$  using checkpoints generated by GE. We provide an ablation study for this setting in Appendix F.
To verify the scalability of results, we train Vision Transformer (ViT; [13]) and MLP-Mixer [60] on ImageNet [12] with  $10\%$  label corruption and evaluate the performance of IF approximation methods. Table 2 shows that the performance gap between  $\mathcal{I}_{\mathrm{GEX}}$  and other baselines is still large. Specifically, in this scenario, F-score fails since some samples have never been correctly predicted during pre-training (i.e., no forgetting events for such samples). We provide additional results for other vision datasets (MNIST [33] and SVHN [42]) and text classification settings in Appendix F.
5.2 Relabeling mislabeled samples
Following up on the detection task in Sec. 5.1, we improve classification performance by relabeling mislabeled samples following Kong et al. [26]. To this end, Otsu algorithm [44] was used to find a threshold that distinguishes between noisy and clean labels, given the self-influence of approximation methods. Since the Otsu method has no inputs other than (influence) distribution, practitioners can apply it for noisy label settings without onerous hyperparameter optimization. Following Kong et al. [26], we relabel training samples as follows:
$$
\tilde {y} _ {k} = \left\{ \begin{array}{l l} 0, & \text {i f} k = m, \\ \log_ {\varphi_ {k}} ^ {K - 1} \sqrt {1 - \varphi_ {m}}, & \text {o t h e r w i s e} \end{array} \right. \tag {16}
$$
where  $m$  is the training sample's original (corrupted) label and  $\varphi_{i}$  is the predicted probability for  $i$ -th class of the training sample. Therefore, the relabel function in (16) masks the original (noisy) label and re-distributes the remaining probabilities. We provide additional results for MNIST and SVHN in Appendix F.
Table 3 shows the relabeled test accuracy with the other settings for comparison. A critical observation in Table 3 is that not all methods achieve performance improvements with relabeling. Otsu algorithm did not find an appropriate threshold for mislabeled examples for these methods. In contrast, since  $\mathcal{I}_{\mathrm{GEX}}$  can separate mislabeled samples from the self-influence distribution, performance improvements were obtained through relabeling. We provide additional results for ImageNet in Appendix F.
5.3 Dataset pruning
We evaluate methods in the previous section on data pruning task [58] to validate the efficacy of  $\mathcal{I}_{\mathrm{GEX}}$  in clean label settings. We quantify the importance of training samples based on self-influence following Feldman and Zhang [15]: We prune low self-influence samples as they are relatively easy samples that can be generalized by learning other training samples. We use networks and datasets in Sec. 5.1 without label noise. We provide additional results for MNIST and SVHN in Appendix F.
As shown in Fig. 4,  $\mathcal{I}_{\mathrm{GEX}}$  consistently detects samples that can be pruned. It is worth noting that among the IF approximation methods considered, only  $\mathcal{I}_{\mathrm{GEX}}$  is comparable to the well-known state-of-the-art methods, F-score and EL2N. This suggests that existing IF approximation methods may experience a performance decrease in downstream tasks because of the errors caused by distributional bias.
(b) CIFAR-100
Figure 4: Data pruning results using various scoring methods. We refer to Sec. 5.3 for the details of scoring methods.
Table 4: Detection metrics for separating MNIST and SVHN
Detection method	AUC	AP
Deep-KNN	44.58 ± 0.39	48.13 ± 0.27
CL	49.88 ± 0.15	49.91 ± 0.11
F-score	67.39 ± 0.02	65.37 ± 0.02
EL2N	68.36 ± 0.33	72.20 ± 0.24
\( \mathcal{I}_{\text{RandProj}} \)	46.88 ± 1.22	56.53 ± 0.82
\( \mathcal{I}_{\text{TraciInP}} \)	56.10 ± 1.83	61.53 ± 1.65
\( \mathcal{I}_{\text{Arnoldi}} \)	38.33 ± 0.23	50.69 ± 0.11
\( \mathcal{I}_{\text{GEX lin}} \)	53.69 ± 1.52	57.88 ± 1.06
\( \mathcal{I}_{\text{GEX}} \)	73.11 ± 0.73	74.42 ± 0.46
5.4 Separation of data sources
We also evaluate IF approximations on separating heterogeneous data sources following Harutyunyan et al. [19]. One often combines multiple datasets to improve generalization. In general, these datasets differ in their informativeness. Therefore, IF can be used to determine which dataset is the most essential based on the informativeness of each dataset. In this experiment, we train ResNet-18 on a mixed dataset of MNIST and SVHN, with 25K random subsamples for each training dataset. Then, we use self-influence as an index for the informativeness following Harutyunyan et al. [19].
Table 4 shows detection metrics for MNIST and SVHN separation using various IF approximations (and other baselines in previous sections). In contrast to the noisy label detection experiment in Sec. 5.1, many IF approximations in this experiment fail to distinguish between data sources: MNIST and SVHN can only be significantly separated by  $\mathcal{I}_{\mathrm{GEX}}$ ,  $\mathcal{I}_{\mathrm{TracInRP}}$ , and F-score. Assuming only the final checkpoint is available, only  $\mathcal{I}_{\mathrm{GEX}}$  can be applied among them.
In this work, we studied the oversimplification of influence distributions due to their bilinear approximations. To mitigate this bias, we developed a non-linear IF approximation, GEX, with GE. Empirically, GEX consistently outperforms the standard IF approximations in various downstream tasks in practice. Also, GEX is user-friendly as it only requires the final checkpoint of pretraining and excludes JVP computation, often constrained by frameworks, for efficient batch estimation. With these advantages, GEX can be used as a practical tool for researchers and practitioners.
This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075 / No.2022-0-00984) and the National Research Foundation of Korea (NRF) grants (No.2018R1A5A1059921 / RS-2023-00209060) funded by the Korea government (MSIT). This work was also supported by Samsung Electronics Co., Ltd (No.IO201214-08133-01).
[1] Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization in linear time. stat, 1050:15, 2016. 3
[2] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning, pages 242-252. PMLR, 2019. 7
[3] Walter Edwin Arnoldi. The principle of minimized iterations in the solution of the matrix eigenvalue problem. Quarterly of applied mathematics, 9(1):17-29, 1951. 3
[4] Dara Bahri, Heinrich Jiang, and Maya Gupta. Deep k-nn for noisy labels. In International Conference on Machine Learning, pages 540–550. PMLR, 2020. 8