Title: GEX: A flexible method for approximating influence via Geometric Ensemble
Venue: NeurIPS 2023
Accepted: yes
Keywords: Influence Function

Geometric Ensemble

Loss Landscape
Abstract: Through a deeper understanding of predictions of neural networks, Influence Function (IF) has been applied to various tasks such as detecting and relabeling mislabeled samples, dataset pruning, and separation of data sources in practice. However, we found standard approximations of IF suffer from performance degradation due to oversimplified influence distributions caused by their bilinear approximation, suppressing the expressive power of samples with a relatively strong influence. To address this issue, we propose a new interpretation of existing IF approximations as an average relationship between two linearized losses over parameters sampled from the Laplace approximation (LA). In doing so, we highlight two significant limitations of current IF approximations: the linearity of gradients and the singularity of Hessian. Accordingly, by improving each point, we introduce a new IF approximation method with the following features: i) the removal of linearization to alleviate the bilinear constraint and ii) the utilization of Geometric Ensemble (GE) tailored for non-linear losses. Empirically, our approach outperforms existing IF approximations for downstream tasks with lighter computation, thereby providing new feasibility of low-complexity/nonlinear-based IF design.

GEX: A flexible method for approximating influence via Geometric Ensemble

Abstract

Through a deeper understanding of predictions of neural networks, Influence Function (IF) has been applied to various tasks such as detecting and relabeling mislabeled samples, dataset pruning, and separation of data sources in practice. However, we found standard approximations of IF suffer from performance degradation due to oversimplified influence distributions caused by their bilinear approximation, suppressing the expressive power of samples with a relatively strong influence. To address this issue, we propose a new interpretation of existing IF approximations as an average relationship between two linearized losses over parameters sampled from the Laplace approximation (LA). In doing so, we highlight two significant limitations of current IF approximations: the linearity of gradients and the singularity of Hessian. Accordingly, by improving each point, we introduce a new IF approximation method with the following features: i) the removal of linearization to alleviate the bilinear constraint and ii) the utilization of Geometric Ensemble (GE) tailored for non-linear losses. Empirically, our approach outperforms existing IF approximations for downstream tasks with lighter computation, thereby providing new feasibility of low-complexity/nonlinear-based IF design.

1 Introduction

In the last decade, neural networks (NNs) have made tremendous advances in various application areas [49, 24, 47]. To make reasonable predictions with NN-based systems, models must be able to explain their predictions. For example, those who doubt the model's prediction can gain insight and foresight by referencing the explanation of the model. Moreover, mission-critical areas like finance and medicine require a high degree of explainability to ensure that the predictions are not biased [30]. Understanding the mechanism of predictions also allows researchers and engineers to improve prediction quality, ensuring that NNs are performing as intended [32].

To this end, Influence Function (IF) was proposed to explain predictions of pre-trained NNs through training data [25]. Intuitively, IF measures how the leave-one-out (LOO) retraining of a training sample changes the loss of each sample. Therefore, the sign of influence determines whether the training sample is beneficial to others, and the scale of influence measures its impact. Specifically, self-influence, the increase in loss when a sample is excluded, was used to measure how much the sample is memorized [48, 15]: When LOO training is performed on a memorized training sample, its loss will increase substantially since matching its (corrupted) label will be difficult. Therefore, self-influence is used to detect mislabeled samples [48, 52] where memorization occurs. Furthermore,

recent works successfully applied IF to various downstream tasks, including dataset pruning [58] and data resampling [59, 63].

Despite its broad applicability, we found that IF and its approximations [25, 48, 52] suffer from oversimplified self-influence distributions due to their bilinear form 1 . Although these approximations are introduced to avoid prohibitive retraining costs of IF, they impose a structural constraint that self-influence becomes quadratic to gradients of pre-trained NNs. Due to this constraint, self-influence follows an unimodal distribution, as gradients of pre-trained NNs typically follow a zero-centered Stable distribution [7, 56]. Unfortunately, unimodal distributions are too restrictive for representing self-influence in real-world datasets containing mislabeled samples. While self-influence distributions estimated by LOO retraining may become bimodal depending on the proportion of (high selfinfluential) mislabeled samples, unimodal distributions cannot handle this case.

To resolve this problem, we propose a non-linear IF approximation via Geometric Ensemble (GE; [16]). Our method is motivated by a novel connection between IF approximation and linearized Laplace approximation (LA; [38]) that we discovered: IF approximations can be translated to an averaged relationship between two linearized losses over parameters sampled from LA. As linearized losses in this connection cause bilinear forms of IF approximations, we consider an IF approximation without linearization. However, we then identify an additional issue of this approximation due to the singularity of Hessian and its solutions (e.g., damping and truncation). To mitigate this issue, we propose a novel approach using GE to manage the relationship of non-linear losses more effectively. As a result, our approach, G eometric E nsemble for sample e X planation (GEX), accurately represents the multimodal nature of LOO retraining, leading to improved performance in downstream tasks across various scenarios. Furthermore, I GEX is easy to estimate as it does not require Jacobian-vector products (JVPs) for batch estimation or sub-curvature approximations like LA.

We summarize our contributions as follows:

We identify a distributional bias in commonly used IF approximations. We demonstrate how this bias results in oversimplified distributions for self-influences.

We provide a novel connection between IF approximations and LA. By identifying an inherent issue of LA, we provide a non-linear IF approximation via GE, named I GEX . Due to its non-linear nature, I GEX can express various influence distributions depending on the characteristic of the datasets.

We verify that I GEX outperforms standard IF approximations in downstream tasks, including noisy label detection, relabeling, dataset pruning, and data source separation. We also show that I GEX is competitive with well-known baselines of downstream tasks with lighter computation. 2

2 Background

Consider an independently and identically (i.i.d.) sampled training dataset S := { z n : ( x n , y n ) } N n =1 where x n ∈ R D is an input vector of n -th sample and y n ∈ R K is its label. To model the relation between inputs and outputs of training samples, we consider a neural network (NN) f : R D × R P → R K which maps input x ∈ R D and network parameter θ ∈ R P to a prediction ˆ y n := f ( x n , θ ) ∈ R K . Empirical risk minimization (ERM) solves the following optimization problem

where L ( S, θ ) := ∑ N n =1 ℓ ( z n , θ ) /N for a sample-wise loss ℓ ( z n , θ ) (e.g., cross-entropy between ˆ y n and y n ). In general, θ ∗ is trained using a stochastic optimization algorithm (e.g., stochastic gradient descent (SGD) with momentum). For the reader's convenience, we provide the notation table in Appendix A for terms used in the paper.

The Leave-One-Out (LOO) retraining effect of z ∈ S on another instance z ′ ∈ R D is defined as the difference of sample-loss z ′ between the θ ∗ and the retrained point θ ∗ z without z [9]:

where θ ∗ z := argmin θ ∈ R P L ( S, θ ) -ℓ ( z, θ ) /N . Since retraining for every pair ( z, z ′ ) is computationally intractable where we have a huge number of data points as in practice, Koh and Liang [25] proposed an efficient approximation of I LOO , named Influence Function (IF):

where g z := ∇ θ ℓ ( z, θ ∗ ) ∈ R P and H := ∇ 2 θ L ( S, θ ∗ ) ∈ R P × P by assuming the strictly convexity of ℓ (i.e., H is positive definite). Here, I can be understood as a two-step approximation of I LOO

where ℓ lin θ ∗ ( z, ψ ) := ℓ ( z, θ ∗ ) + g ⊤ z ( ψ -θ ∗ ) and ψ ∈ R P is an arbitrary vector in the parameter space R P . Here, we use the superscript lin to indicate linearization. Note that (3) applies linearization to the sample-loss z ′ and (4) approximates parameter difference as a Newton ascent term (i.e., θ ∗ z ≈ θ ∗ + H -1 g z ).

While the computation of I is cheaper than I LOO , it is still intractable to modern NNs (e.g., ResNet [20] and Transformer [62]) because of the prohibitively large Hessian. To alleviate this problem, two additional approximations are commonly used: stochastic approximation methods such as I LiSSA [1], and sub-curvature approximations such as I Last -Layer , which limits the Hessian computation only to the last-layer of NNs [10]. However, both methods also have their own problems: I LiSSA takes high time complexity since the inverse Hessian-vector product (IHVP) for each training sample needs to be computed separately, as shown in Schioppa et al. [52]. On the other hand, I Last -Layer may cause inaccurate IF approximations, as shown in Feldman and Zhang [15].

As another alternative, Pruthi et al. [48] recently proposed to exploit intermediate checkpoints during pre-training, named I TracIn :

where g c z := ∇ θ ℓ ( z, θ c ) for checkpoints θ c ( c = 1 , . . . , C ) sampled from the pre-training trajectory of θ ∗ . Here, it further simplifies the computation by assuming the expensive H -1 in (2) is an identity matrix. Instead, the performance of I TracIn is replenished by averaging over several intermediate checkpoints, which capture various local geometries of loss landscapes. In addition, Pruthi et al. [48] enhanced the efficiency of I TracIn using random projection , named I TracInRP ,

where Q R ∈ R P × R is a random projection matrix whose components are i.i.d. sampled from N (0 , 1 /R ) for R ≪ P . Note that I TracInRP is an unbiased estimator of I TracIn as E [ Q R Q ⊤ R ] = I P where I P is the identity matrix of dimension P × P . However, I TracIn and I TracInRP cannot be applied to checkpoints of the open-source community, such as TorchHub [45] and Huggingface models [65] since they only provide the final checkpoints without any intermediate results during pre-training.

On the other hand, Schioppa et al. [52] proposed to approximate IF in a purely post-hoc manner as

where Λ R ∈ R R × R is a diagonal matrix whose elements are topR eigenvalues of H and the columns of U R ∈ R P × R are corresponding eigenvectors. They use Arnoldi iteration [3] to estimate Λ R and U R . Contrary to I TracInRP with the random projection, I Arnoldi projects g z , g z ′ to the topR eigenspace of the Hessian. Due to this difference, Schioppa et al. [52] argued that I Arnoldi perform comparably to I TracInRP with less R .

Finally, it is important to note that I TracInRP and I Arnoldi can perform batch computations using Jacobian-vector products (JVPs). To be more precise, computing I and I TracIn for multiple samples requires sample-wise gradient computation for each g z , g c z , which is difficult to parallelize because of heavy memory complexity. In contrast, I TracInRP and I Arnoldi can avoid this sample-wise computation by computing JVPs for a batch at once (i.e., parallelize g c ⊤ z Q R , g ⊤ z U R for multiple z ).

0.010

0.005

0.000

0.005

10 0

10 1

10 2

10 3

10 4

10 5

Components of randomly projected grad.

proj_0 (p:0.00)

proj_1 (p:0.00)

proj_2 (p:0.00)

proj_3 (p:0.00)

proj_4 (p:0.00)

(a) Random proj. of grads.

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

1e2

10 0

10 1

10 2

10 3

10 4

AUC : 0.6908, AP : 0.1900

clean

corrupted

(b)

I

Arnoldi

0

1

2

3

4

5

6

7

8

1e4

10 0

10 1

10 2

10 3

10 4

AUC : 0.9719, AP : 0.7056

clean

corrupted

(c)

I

TracInRP

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

1e1

10 0

10 1

10 2

10 3

10 4

AUC : 0.9999, AP : 0.9991

clean

corrupted

(d)

I

GEX

3 Identifying distributional bias in bilinear influence approximations

Throughout the discussion above, IF approximations ((2), (5), (6), and (7)) are defined as bilinear forms of gradients . In this section, we demonstrate a side effect of this form on computing selfinfluence. Our key empirical observation is that gradients of pre-trained NNs follow Gaussian distributions centered at zero . To verify this, we train VGGNet [55] on MNIST [33] with 10% label corruption following Schioppa et al. [52]. Then, we project { g z } z ∈ S onto each of 5 random vectors { d i } 5 i =1 (with d i ∈ R P ), uniformly sampled on the unit sphere. If a pre-trained gradient g z follows Gaussian distribution, its projected component g ⊤ z d i will also follow Gaussian distributions for all i = 1 , . . . , 5 . We visualize histograms of projected components g ⊤ z d i for each d i , and the resulting p-values of the normality test [57] in Fig. 1(a). In agreement with our hypothesis, the randomly projected components follow Gaussian distributions with significantly low p-values. Moreover, one can observe that Gaussian distributions are centered at zero. This is because gradients at θ ∗ satisfy

by the first-order optimality condition at θ ∗ (i.e., ∇ θ L ( S, θ ∗ ) = 0 P ) where 0 P is the zero vector of dimension P . Note that a similar observation on the normality of gradient was reported in the context of differential privacy in Chen et al. [7].

Now let us write the eigendecomposition of Hessian as H = ∑ P i =1 λ i u i u ⊤ i where λ 1 , . . . , λ P > 0 are eigenvalues of H in descending order and u 1 , . . . , u P ∈ R P are corresponding eigenvectors by positive definite assumption of Koh and Liang [25]. We then arrange self-influence of I as

where g z,i := g ⊤ z u i are i -th component of g z in the eigenspace of H . Following the above discussion, one can assume that g z,i follows a Gaussian distribution. Consequently, I ( z, z ) follows a (generalized) χ 2 -distribution due to squared components in (8). The following proposition shows that this phenomenon can be generalized to any stable distribution [56] and positive definite matrix.

Proposition 3.1 (Distributional bias in bilinear self-influence) . Let us assume g z follows a P -dimensional stable distribution (e.g., Gaussian, Cauchy, and Lévy distribution) and M ∈ R P × P is a positive (semi-)definite matrix. Then, self-influence in the form of I M ( z, z ) = g ⊤ z Mg z follows a unimodal distribution. Furthermore, if g z follows a Gaussian distribution, then the self-influence follows a generalized χ 2 -distribution.

We refer to Appendix B for the proof. Self-influence distributions approximated by I Arnoldi (Fig. 1(b)) and I TracInRP (Fig. 1(c)) provide the empirical evidence of Proposition 3.1 in the noisy label setting in Fig. 1(a). In this setting, each mislabeled sample exhibits high self-influence values since predictions by incorrect labels are challenging to recover when removed and retrained. Therefore, mislabeled samples constitute a distinct mode with a greater self-influence than correctly labeled samples. Still, the distributional bias in Proposition 3.1 squeezes correct and mislabeled samples in a unimodal distribution. As a result, two types (clean and corrupted) of samples are indistinguishable in I Arnoldi and I TracInRP . Proposition 3.1 shows that this observation can be generalized to a more diverse setting (e.g., heavy-tailed distributions). In Sec. 5.1, we demonstrate that this bias occurs regardless of the dataset and the architectures of NN.

1.5

1.0

0.5

0.0

0.5

1.0

1.5

x

1.5

1.0

0.5

0.0

0.5

1.0

1.5

y

Class 0

Class 1

Typical

Influential

(a) Traininig dataset

0.0

0.5

1.0

1.5

2.0

2.5

1e

1

10 0

10 1

AUC : 1.0000, AP : 1.0000

typical

influential

(b)

I

GT

2

4

6

8

1e

3

10 0

10 1

AUC : 0.9950, AP : 0.9769

typical

influential

(c)

I

(

α

= 100

)

0

1

2

3

4

1e

2

10 0

10 1

AUC : 1.0000, AP : 1.0000

typical

influential

(d)

I

GEX

4 Geometric Ensemble for sample eXplanation

To mitigate the distributional bias in Sec. 3, we propose a flexible IF approximation method using Geometric Ensemble (GE; [16]), named Geometric Ensemble for sample eXplanataion (GEX). Here is a summary of how GEX is developed.

In Sec. 4.1, we ensure that the influence approximation is not a bilinear form for the gradient by replacing gradients in IF with sample-loss deviations. The theoretical foundation for this step is provided by our Theorem 4.1 below, which establishes a relationship between the IF and the Laplace approximation (LA; [38]). Moving on to Sec. 4.2, we modify the parameter distribution to compute the sample-loss deviation from LA to GE. This modification is necessary because GE is based on the local geometry of the loss landscape around θ ∗ , similar to LA, while avoiding overestimating loss deviations caused by the singularity of the Hessian.

4.1 GEX and its motivation

The proposed method, GEX, comprises three steps. The first step involves the collection of post-hoc checkpoints { θ m } M m =1 through multiple SGD updates on the training loss starting from θ ∗ . Then, the empirical distribution of Geometric Ensemble (GE) is computed as

where δ θ ( m ) ( · ) denotes the Dirac delta distribution at θ ( m ) . In the final step, GEX is obtained as the expectation of the product of sample-loss deviations from the pre-trained parameter as follows

where ∆ ℓ θ ∗ ( z, ψ ) := ℓ ( z, ψ ) -ℓ ( z, θ ∗ ) means the sample-loss deviations from θ ∗ . We provide the pseudocode for computing I GEX in Appendix C.

The main motivation behind I GEX in (10) is to establish a new connection between I and LA. This connection is demonstrated in Theorem 4.1, which shows that I is an expectation of the product of linearized sample-loss deviations given that parameters are sampled from LA.

Theorem 4.1 (Connection between IF and LA) . I in Koh and Liang [25] can be expressed as

where ∆ ℓ lin θ ∗ ( z, ψ ) := ℓ lin θ ∗ ( z, ψ ) -ℓ lin θ ∗ ( z, θ ∗ ) = g ⊤ z ( ψ -θ ∗ ) and p LA is the Laplace approximated posterior

2

4

6

8

1e11

10 0

2×10 0

3×10 0

4×10 0

6×10 0

AUC : 0.0700, AP : 0.0847

typical

influential

(a)

I

LA

(

α

= 0

.

01

)

1.3501.3751.4001.4251.4501.4751.5001.525

1e1

10 0

10 1

AUC : 0.0000, AP : 0.0822

typical

influential

(b)

I

LA

(

α

= 100

)

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1e6

10 0

10 1

AUC : 0.6333, AP : 0.1877

typical

influential

(c)

I

LA

(

R

= 50

)

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1e

4

10 0

10 1

AUC : 0.7233, AP : 0.2330

typical

influential

(d)

I

(

R

= 50

)

The LA was proposed to approximate the posterior distribution with a Gaussian distribution. Recently, it has gained significant attention due to its simplicity and reliable calibration performance [50, 11]. Intuitively, LA is equivalent to the second-order Taylor approximation of log-posterior at θ ∗ with Gaussian prior defined as p ( ψ ) := N ( ψ | 0 P , γ -1 I P ) :

Here, the training loss represents the negative log-likelihood L ( S, θ ) = -log p ( S | θ ) , and Z := ∫ p ( θ ) p ( S | θ ) dθ represents the evidence in Bayesian inference [5]. Similar to the IF, LA becomes computationally intractable when dealing with modern architectures due to the complexity of the Hessian matrix. To address this computational challenge, recent works have proposed various subcurvature approximations, such as KFAC [50] and sub-network [11], which provide computationally efficient alternatives for working with LA.

According to (11), samples with a high degree of self-influence experience significant (linearized) loss changes for the parameters sampled from LA. Furthermore, Theorem 4.1 reveals that the gradients, the origin of distributional bias, arise from the linearizations in (11). Hence, to address the distributional bias of the bilinear IF approximations in Proposition 3.1, we remove the linearizations in (11). This leads us to consider a modified version of I , named I LA :

The pseudocode for computing I LA using Kronecker-Factored Approximate Curvature (KFAC; [40]) is available in Appendix C.

While I LA no longer theoretically produces unimodal self-influence, it turns out that it still mixes correct and mislabeled samples, even in toy datasets. To illustrate this problem, we train a fullyconnected NN on the two-circle dataset [29] with a modification as shown in Fig. 2(a): We add ten influential training samples at the center of the two-circle dataset containing 30 train samples per class (circle). These influential samples are highly susceptible (i.e., hard to recover) to leave-oneout retraining as samples of the opposite class are densely located around them. We provide other experimental details in Appendix E.

As discussed in Sec. 3, highly influential samples form a separate mode of high self-influence in the histogram of I LOO (Fig. 2(b)) and I mixes typical and influential samples (Fig. 2(c)) due to the distributional bias of bilinear form. While Proposition 3.1 does not apply to I LA , Fig. 3 shows that I LA still fails to distinguish between correct and mislabeled samples by severely overestimating the self-influence of typical samples (Fig. 3(a)-3(b)) or squeeze them into a unimodal distribution (Fig. 3(c)), similar to I (Fig. 3(d)).

4.2 Pitfalls of inverse Hessian in non-linear IF approximation

To analyze the limitation of I LA , we reparameterize the sampled parameter from LA as follows

From this reparameterization, one can see that for (13) to be valid, all eigenvalues of H must be positive definite (i.e., all eigenvalues are positive), as Koh and Liang [25] assumed. However, over-parameterized NNs in practice can contain many zero eigenvalues in their Hessian.

Proposition 4.2 (Hessian singularity for over-parameterized NNs) . Let us assume a pre-trained parameter θ ∗ ∈ R P achieves zero training loss L ( S, θ ∗ ) = 0 for squared error. Then, H has at least P -NK zero-eigenvalues for NNs such that NK < P . Furthermore, if x is an input of training sample z ∈ S , then the following holds for the eigenvectors { u i } P i = NK +1

We refer to Appendix B for the cross-entropy version of Proposition 4.2 with empirical Fisher (EF) matrix, a well-known approximation of the Hessian [31, 22], defined as F := 1 N ∑ N n =1 g z g ⊤ z . Note that the over-parameterization assumption ( NK < P ) in Proposition 4.2 is prevalent in real-world situations, including the settings of Fig. 1- 3. Also, the zero training loss assumption can be satisfied with sufficient over-parameterization [14, 2]. Accordingly, empirical evidences of Proposition 4.2 have been reported in different scenarios [51, 18].

A simple solution to mitigate this singularity is adding an isotropic matrix to H , known as the damping technique : H ≈ H ( α ) := H + α I P for α > 0 . Then, the modified LA sample is

Although the damping trick can make all eigenvalues positive in principle, the order of the eigenvalues remains unchanged. Therefore, the null space components ( g z,i for i = NK +1 , . . . , P ) in Proposition 4.2 are the most heavily weighted by 1 / √ α . These heavily weighted null space components do not affect the linearized sample-loss deviations in (11) as follows

by Proposition 4.2. However, this is not applicable to the sample-loss deviations in (12), since the null space components cause ∆ ℓ θ ∗ ( z, ψ α ) to change rapidly (i.e., overestimating the influence of samples). This can be confirmed by observing that setting α = 0 . 01 in I LA (Fig. 3(a)) leads to significantly overestimating self-influences. To prevent this issue, one can enlarge α (i.e., decrease 1 /α ). However, since the scale of α does not change the order of eigenvalues, the null space components still receive a higher weight than the others. Consequently, I LA with α = 100 (Fig. 3(b)) does not correctly represent the multi-modality of self-influence distribution.

Another way to handle the Hessian singularity is to limit (13) to only the topR eigenvalues of Hessian, similar to I Arnoldi . However, this approximation method may result in significant errors due to using only the smallest R eigenvalues of the inverse Hessian (i.e., 1 / √ λ i ). Consequently, even linearized sample-loss deviations (i.e., I ) with R = 50 (Fig. 3(d)) suffer from severe performance degradation, compared to full Hessian with damping (Fig. 2(c)).

Compared to damping and truncation, I GEX (Fig. 2(d)) accurately captures the bimodal self-influence distribution. This effectiveness stems from the differential impact of SGD steps on typical and influential samples [61]: Since typical samples are robust to SGD steps [35], loss deviation of these samples are small in I GEX . In contrast, influential samples experience significant loss deviations as they are sensitive to SGD steps. As GE makes diverse predictions [16], the members of GE introduce varying levels of loss deviation for each influential sample. An ablation study exploring the impact of ensemble size is provided in Appendix F.

4.3 Practical advantages of GEX

In addition to the above benefits, I GEX offers several implementation advantages. First, it can be obtained using only open-source final checkpoints , unlike I TracIn and I TracInRP . This advantage broadens the range of applicable models. Second, batch estimation is easy to implement in I GEX . Projection-based methods like I TracInRP and I Arnoldi require JVP computation, which is only efficient

for packages that provide forward-mode auto-differentiation, such as JAX [6]. In contrast, batch computation in I GEX necessitates only forward computations , which are efficient in almost all auto-differentiation packages. Since IF is computed for each training sample in most downstream tasks [25, 48, 52], efficient batching is critical for practical applications. As a result, we believe that this distinction will be vital for researchers and practitioners in their work. We provide the complexity analysis of I GEX and other IF approximations in Appendix D and discuss the limitations of our method and broader impacts in Appendix G.

5 Experiments

Here we describe experiments demonstrating the usefulness of GEX in downstream tasks. We conducted two experiments for each noisy and clean label setting: Detection (Sec. 5.1) and relabeling (Sec. 5.2) of mislabeled examples for noisy label setting and dataset pruning (Sec. 5.3) and separating data sources (Sec. 5.4) for clean label setting. We refer to Appendix E for experimental details.

5.1 Noisy label detection

In this section, we evaluate the performance of IF approximation methods for detecting noisy labels. We use self-influence as an index for the noisy label of IF approximation methods following Koh and Liang [25] and Pruthi et al. [48]. A high degree of self-influence indicates mislabeled examples since the removal of mislabeled examples will significantly change prediction or loss.

̸

We train ResNet-18 [20] on CIFAR-10/100 [27] with 10% random label corruption for synthetic noise and CIFAR-N [64] for real-world noise. We use the "worse label" version of CIFAR-10-N since it corresponds to the highest noise level. We compare I GEX to the following baselines: I TracInRP with 5 checkpoints and 20 random projections, I RandProj with 20 random projections only for the final checkpoint, I Arnoldi with 100 iterations and 20 projections. We assess F -score [61], and EL2N [46] as they offer alternative approaches to identifying influential samples. We also evaluate an ablative version of I GEX : I GEX -lin , which replaces LA with GE for (11) (i.e., I GEX with linear sample-loss deviation). Finally, we report the performance of well-known baselines, Deep -KNN [4] and CL [43], for comparison. We provide results for cross-influence I ( z, z ′ ) in case of z = z ′ in Appendix F.

Table 1 shows that I GEX distinguishes mislabeled samples better than other methods. Furthermore, we find that I GEX is more effective than CL , a state-of-the-art noisy label detection technique. Since CL requires additional K -fold cross-validation for sample scoring ( K = 2 used in our experiments), I GEX will be an attractive alternative in terms of both computational complexity and detection performance. Notably, results of I GEX -lin demonstrate that removing linearization is essential for performance improvement of I GEX .

One interesting observation in Table 1 is that I Arnoldi did not show improvement compared to simple I RandProj . Indeed, a similar observation was reported in Table 1 in Schioppa et al. [52]. In contrast, I TracInRP showed improvements in CIFAR-10 and SVHN compared to I RandProj , I TracInRP is computationally expensive since it requires JVP computations for multiple checkpoints. On the other hand, I GEX can be applied in a pure post-hoc style and does not require (framework-dependent)

JVP computation. Motivated by I GEX , one can propose a purely post-hoc I TracInRP using checkpoints generated by GE. We provide an ablation study for this setting in Appendix F.

To verify the scalability of results, we train Vision Transformer (ViT; [13]) and MLP-Mixer [60] on ImageNet [12] with 10% label corruption and evaluate the performance of IF approximation methods. Table 2 shows that the performance gap between I GEX and other baselines is still large. Specifically, in this scenario, F -score fails since some samples have never been correctly predicted during pre-training (i.e., no forgetting events for such samples). We provide additional results for other vision datasets (MNIST [33] and SVHN [42]) and text classification settings in Appendix F.

5.2 Relabeling mislabeled samples

Following up on the detection task in Sec. 5.1, we improve classification performance by relabeling mislabeled samples following Kong et al. [26]. To this end, Otsu algorithm [44] was used to find a threshold that distinguishes between noisy and clean labels, given the self-influence of approximation methods. Since the Otsu method has no inputs other than (influence) distribution, practitioners can apply it for noisy label settings without onerous hyperparameter optimization. Following Kong et al. [26], we relabel training samples as follows:

where m is the training sample's original (corrupted) label and φ i is the predicted probability for i -th class of the training sample. Therefore, the relabel function in (16) masks the original (noisy) label and re-distributes the remaining probabilities. We provide additional results for MNIST and SVHN in Appendix F.

Table 3 shows the relabeled test accuracy with the other settings for comparison. A critical observation in Table 3 is that not all methods achieve performance improvements with relabeling. Otsu algorithm did not find an appropriate threshold for mislabeled examples for these methods. In contrast, since I GEX can separate mislabeled samples from the self-influence distribution, performance improvements were obtained through relabeling. We provide additional results for ImageNet in Appendix F.

5.3 Dataset pruning

We evaluate methods in the previous section on data pruning task [58] to validate the efficacy of I GEX in clean label settings. We quantify the importance of training samples based on self-influence following Feldman and Zhang [15]: We prune low self-influence samples as they are relatively easy samples that can be generalized by learning other training samples. We use networks and datasets in Sec. 5.1 without label noise. We provide additional results for MNIST and SVHN in Appendix F.

As shown in Fig. 4, I GEX consistently detects samples that can be pruned. It is worth noting that among the IF approximation methods considered, only I GEX is comparable to the well-known state-of-the-art methods, F -score and EL2N . This suggests that existing IF approximation methods may experience a performance decrease in downstream tasks because of the errors caused by distributional bias.

0

5000

10000

15000

20000

25000

Pruned data

93.0

93.5

94.0

94.5

95.0

95.5

Test accuracy (%)

Arnoldi

EL2N

F-score

GEX

Random

RandProj

TracInRP

(a) CIFAR-10

0

2500

5000

7500

10000

12500

15000

17500

20000

Pruned data

74

75

76

77

78

79

Test accuracy (%)

Arnoldi

EL2N

F-score

GEX

Random

RandProj

TracInRP

(b) CIFAR-100

5.4 Separation of data sources

We also evaluate IF approximations on separating heterogeneous data sources following Harutyunyan et al. [19]. One often combines multiple datasets to improve generalization. In general, these datasets differ in their informativeness. Therefore, IF can be used to determine which dataset is the most essential based on the informativeness of each dataset. In this experiment, we train ResNet-18 on a mixed dataset of MNIST and SVHN, with 25K random subsamples for each training dataset. Then, we use self-influence as an index for the informativeness following Harutyunyan et al. [19].

Table 4 shows detection metrics for MNIST and SVHN separation using various IF approximations (and other baselines in previous sections). In contrast to the noisy label detection experiment in Sec. 5.1, many IF approximations in this experiment fail to distinguish between data sources: MNIST and SVHN can only be significantly separated by I GEX , I TracInRP , and F -score . Assuming only the final checkpoint is available, only I GEX can be applied among them.

6 Conclusion

In this work, we studied the oversimplification of influence distributions due to their bilinear approximations. To mitigate this bias, we developed a non-linear IF approximation, GEX, with GE. Empirically, GEX consistently outperforms the standard IF approximations in various downstream tasks in practice. Also, GEX is user-friendly as it only requires the final checkpoint of pretraining and excludes JVP computation, often constrained by frameworks, for efficient batch estimation. With these advantages, GEX can be used as a practical tool for researchers and practitioners.

References

Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization in linear time. stat , 1050:15, 2016. 3

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning , pages 242-252. PMLR, 2019. 7

Walter Edwin Arnoldi. The principle of minimized iterations in the solution of the matrix eigenvalue problem. Quarterly of applied mathematics , 9(1):17-29, 1951. 3

Dara Bahri, Heinrich Jiang, and Maya Gupta. Deep k-nn for noisy labels. In International Conference on Machine Learning , pages 540-550. PMLR, 2020. 8

Christopher M. Bishop. Pattern Recognition and Machine Learning . Springer, 2006. 6, 16

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs. github , 2018. 8, 19

Xiangyi Chen, Steven Z Wu, and Mingyi Hong. Understanding gradient clipping in private sgd: Ageometric perspective. Advances in Neural Information Processing Systems , 33:13773-13782, 2020. 2, 4

William G Cochran. The distribution of quadratic forms in a normal system, with applications to the analysis of covariance. In Mathematical Proceedings of the Cambridge Philosophical Society , volume 30, pages 178-191. Cambridge University Press, 1934. 16

R Dennis Cook. Detection of influential observation in linear regression. Technometrics , 19(1): 15-18, 1977. 2

Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux - effortless bayesian deep learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021. 3

Erik Daxberger, Eric Nalisnick, James U Allingham, Javier Antoran, and Jose Miguel HernandezLobato. Bayesian deep learning via subnetwork inference. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 2510-2521. PMLR, 18-24 Jul 2021. 6

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009. 9, 19, 20, 21, 23

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020. 9, 20

Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning , pages 1675-1685. PMLR, 2019. 7

Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems , 33: 2881-2891, 2020. 1, 3, 9

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. 2, 5, 7, 18, 19

Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a kronecker factored eigenbasis. Advances in Neural Information Processing Systems , 31, 2018. 18

Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning , pages 2232-2241. PMLR, 2019. 7

Hrayr Harutyunyan, Alessandro Achille, Giovanni Paolini, Orchid Majumder, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Estimating informativeness of samples with smooth unique information. arXiv preprint arXiv:2101.06640 , 2021. 10, 20, 24

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016. 3, 8, 20

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015. 23

Alexander Immer, Matthias Bauer, Vincent Fortuin, Gunnar Rätsch, and Khan Mohammad Emtiyaz. Scalable marginal likelihood estimation for model selection in deep learning. In International Conference on Machine Learning , pages 4563-4573. PMLR, 2021. 7

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 448-456, Lille, France, 07-09 Jul 2015. PMLR. 20

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, 2021. 1

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning , pages 1885-1894. PMLR, 2017. 1, 2, 3, 4, 5, 7, 8, 16

Shuming Kong, Yanyan Shen, and Linpeng Huang. Resolving training biases via influence-based data relabeling. In International Conference on Learning Representations , 2021. 9

AKrizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto , 2009. 8, 20

Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226 , 2018. 20

Brian Kulis, Sugato Basu, Inderjit Dhillon, and Raymond Mooney. Semi-supervised graph clustering: a kernel approach. In Proceedings of the 22nd international conference on machine learning , pages 457-464, 2005. 6

Shinjini Kundu. Ai in medicine must be explainable. Nature medicine , 27(8):1328-1328, 2021. 1

Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approximation for natural gradient descent. Advances in neural information processing systems , 32, 2019. 7

Valliappa Lakshmanan, Sara Robinson, and Michael Munn. Machine learning design patterns . O'Reilly Media, 2020. 1

Y LeCun, C Cortes, and C Burges. The mnist dataset of handwritten digits (images). NYU: New York, NY, USA , 1999. 4, 9, 20, 21

Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. Dbpedia-a largescale, multilingual knowledge base extracted from wikipedia. Semantic web , 6(2):167-195, 2015. 20, 22

Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Earlylearning regularization prevents memorization of noisy labels. Advances in neural information processing systems , 33:20331-20342, 2020. 7

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016. 20

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019. 20

David J. C. MacKay. A practical bayesian framework for backpropagation networks. Neural Comput. , 4(3):448-472, may 1992. ISSN 0899-7667. doi: 10.1162/neco.1992.4.3.448. 2, 5

James Martens. New insights and perspectives on the natural gradient method. arXiv preprint arXiv:1412.1193 , 2014. 18

James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning , pages 2408-2417. PMLR, 2015. 6

Stefan Mittnik, Svetlozar T Rachev, and Jeong-Ryeol Kim. Chi-square-type distributions for heavy-tailed variates. Econometric Theory , 14(3):339-354, 1998. 16

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011 , 2011. 9, 20, 21

Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research , 70:1373-1411, 2021. 8

Nobuyuki Otsu. A threshold selection method from gray-level histograms. IEEE transactions on systems, man, and cybernetics , 9(1):62-66, 1979. 9

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, highperformance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. pdf . 3

Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems , 34:20596-20607, 2021. 8, 20

David Pfau, James S Spencer, Alexander GDG Matthews, and W Matthew C Foulkes. Ab initio solution of the many-electron schrödinger equation with deep neural networks. Physical Review Research , 2(3):033429, 2020. 1

Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems , 33: 19920-19930, 2020. 1, 2, 3, 8, 20

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 , 2021. 1

Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In International Conference on Learning Representations , 2018. 6

Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv preprint arXiv:1611.07476 , 2016. 7, 17

Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8179-8186, 2022. 1, 2, 3, 4, 8, 19

Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, and Lawrence Carin. Baseline needs more love: On simple wordembedding-based models and associated pooling mechanisms. arXiv preprint arXiv:1805.09843 , 2018. 20, 22

Thomas Simon. A multiplicative short proof for the unimodality of stable densities. Electronic Communications in Probability , 16:623-629, 2011. 16

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations , 2015. 4, 20

Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning , pages 5827-5837. PMLR, 2019. 2, 4, 16

Nickolay Smirnov. Table for estimating the goodness of fit of empirical distributions. The annals of mathematical statistics , 19(2):279-281, 1948. 4

Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206.14486 , 2022. 2, 9, 23

Daniel Ting and Eric Brochu. Optimal subsampling with influence functions. Advances in neural information processing systems , 31, 2018. 2

Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems , 34: 24261-24272, 2021. 9, 20

Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159 , 2018. 7, 8

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998-6008, 2017. 3

Zifeng Wang, Hong Zhu, Zhenhua Dong, Xiuqiang He, and Shao-Lun Huang. Less is better: Unweighted data subsampling via influence function. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 6340-6347, 2020. 2

Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. arXiv preprint arXiv:2110.12088 , 2021. 8, 20, 21

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 , 2019. 3

Makoto Yamazato. On strongly unimodal infinitely divisible distributions of class cme. Theory of Probability & Its Applications , 40(3):518-532, 1996. 16

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization, 2017. 20

Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvornek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. arXiv preprint arXiv:2203.08065 , 2022. 20

A Notations

D ≜ (true) data distribution

S := { z n : ( x n , y n ) } N n =1

≜ i.i.d. sampled training dataset from D

z n := ( x n , y n )

≜ a training instance (or sample)

x n ∈ R D , y n ∈ R K ≜

training inputs/outputs

θ, ψ ∈ R P

≜ parameter of NNs

ˆ y n = f ( x n , θ ) ≜ output of NN given x, θ

ℓ ( z, θ ) ≜ instance-wise loss given sample z, θ

L ( S, θ ) ≜ 1 N ∑ N n =1 ℓ ( z n , θ )

θ ∗ ∈ R P ≜ pre-trained parameter of NN

θ

∗

z

∈

R

P

≜

argmin

θ

L

(

S, θ

)

-

ℓ

(

z, θ

)

/N

(leave-one-out retrained parameter)

g

z

∈

R

P

≜

∇

θ

ℓ

(

z, θ

∗

)

(gradient of instance-loss w.r.t. parameter given

z, θ

∗

)

H

∈

R

P

×

P

≜

∇

2

θ

L

(

S

, θ

∗

)

(Hessian of training loss w.r.t. parameter given

θ

∗

)

ℓ

θ

∗

(

z, ψ

)

≜

ℓ

(

z, θ

∗

) +

g

⊤

z

(

ψ

-

θ

∗

)

(linearization of instance-loss w.r.t. parameter)

∆

ℓ

θ

∗

(

z, ψ

)

≜

ℓ

θ

∗

(

z, ψ

)

-

ℓ

(

z, θ

∗

)

∆ ℓ θ ∗ ( z, ψ ) ≜ ℓ θ ∗ ( z, ψ ) - ℓ ( z, θ ∗ )

0 P ∈ R P ≜ the P -dimensional zero vector

I

P

∈

R

P

×

P

≜

the

P

-dimensional identity matrix

g c z ∈ R P

≜ ∇ θ ℓ ( z, θ c ) (gradient of instance-loss w.r.t. parameter for a checkpoint c )

Q R ∈ R P × R

≜ Random matrix sampled from N (0 , 1 /R ) component-wisely

H

=

U

Λ

U

⊤

≜ Eigendecomposition of Hessian

H ( α )

≜ Damped Hessian matrix H + α I P

Λ R ∈ R R × R , U R ∈ R P × R

≜ Submatrices of the top- D eigenvalues/eigenvectors

λ i ∈ R , u i ∈ R P

≜ the i -th largest eigenvalue of H and its eigenvector

g

z,i

=

g

⊤

z

u

i

∈

R

≜

i -th component of g z in the eigenspace of H

p LA ( ψ ) ≜

N ( ψ | θ ∗ , H - 1 ) (the Laplace approximated posterior)

p GE ( ψ ) ≜

1 M ∑ M m =1 δ θ ( m ) ( ψ ) (the Geometric Ensemble distribution)

{ θ ( m ) } M m =1

≜ checkpoints of Geometric Ensemble (GE)

B Proofs

B.1 Proof of Proposition 3.1

Proposition B.1 (Distributional bias in bilinear self-influence) . Let us assume g z follows a P -dimensional stable distribution (e.g., Gaussian, Cauchy, and Lévy distribution) and M ∈ R P × P is a positive (semi-)definite matrix. Then, self-influence in the form of I M ( z, z ) = g ⊤ z Mg z follows a unimodal distribution. Furthermore, if g z follows a Gaussian distribution, then the self-influence follows a generalized χ 2 -distribution.

Proof. We first consider the case of gradients with stable distributions [56]. According to Corollary 1 in Mittnik et al. [41], the sum of squares of stable random variables is a stable distribution with the stability of α/ 2 . Since stable distributions are unimodal [66, 54], the resulting self-influence is also unimodal.

Now we proof the case of the Gaussian gradient. By positive semi-definite assumption, let us write the eigendecomposition of M as M = ∑ P i =1 λ i u i u ⊤ i where eigenvalues are sorted in descending order. Specifically, one can compress this summation for the strictly positive eigenvalues: M = ∑ T i =1 λ i u i u ⊤ i where T is the cardinality of strictly positive eigenvalues. Then the self-influence I M can be expressed as follows:

where g z,i = g ⊤ z u i is the i -th component of g z in the eigenspace of M . As a linearly transformed Gaussian random variable follows Gaussian distribution [5], one can write g z,i as follows

Therefore, g z,i follows following generalized χ 2 -distribution [8]

where µ 2 i is the non-centrality parameter and σ 2 i is the weight of non-central χ 2 . As a result, I M follows following generalized χ 2 -distribution

B.2 Proof of Theorem 4.1

Theorem B.2 (Connection between IF and LA) . I in Koh and Liang [25] can be expressed as

where ∆ ℓ lin θ ∗ ( z, ψ ) := ℓ lin θ ∗ ( z, ψ ) -ℓ lin θ ∗ ( z, θ ∗ ) = g ⊤ z ( ψ -θ ∗ ) and p LA is the Laplace approximated posterior

Proof. One can express ∆ ℓ lin θ ∗ ( z, ψ ) as follows

Therefore, E ψ ∼ p LA [ ∆ ℓ lin θ ∗ ( z, ψ ) · ∆ ℓ lin θ ∗ ( z ′ , ψ ) ] can be arranged as follows

B.3 Proof of Proposition 4.2

Proposition B.3 (Singular Hessian for over-parameterized NNs) . Let us assume a pre-trained parameter θ ∗ ∈ R P achieves zero training loss L ( S, θ ∗ ) = 0 for squared error. Then, H has at least P -NK zero-eigenvalues for NNs such that NK < P . Furthermore, if x is an input of training sample z ∈ S , then the following holds for the eigenvectors { u i } P i = NK +1

Proof. The Hessian matrix of training loss w.r.t. parameters can be expressed as follows [51]

where the first equality holds due to the zero training loss assumption

and the second equality holds due to the squared error assumption

∇

2

ˆ

y

n

ℓ

(

z

n

, θ

) =

I

K

∀

n

= 1

, . . . , N.

Therefore, H is a sum of NK rank-one products. As a result, H has at most NK non-zero eigenvalues and at least P -NK zero-eigenvalues for over-parameterized NNs such that NK < P . As training Jacobians ∇ ⊤ θ f ( x, θ ∗ ) for n = 1 , . . . , N are contained in the linear span of { u i } i NK =1 , which is orthogonal to { u i } P i = NK +1 , the following holds for any training input x

B.4 Cross-entropy version of Proposition 4.2 and its proof

Proposition B.4 (Cross-entropy version of Proposition 4.2) . Let us assume a pre-trained parameter θ ∗ ∈ R P for cross-entropy loss. Then, the empirical Fisher (EF) matrix F = 1 N ∑ z ∈ S g z g ⊤ z has at least P -N zero-eigenvalues for NNs such that N < P . Furthermore, the following holds for the last P -N eigenvectors { u i } P i = N +1 and any training sample z ∈ S

Proof. As F is a sum of N rank-one products. As a result, F has at most N non-zero eigenvalues and at least P -N zero-eigenvalues for over-parameterized NNs such that N < P . As gradients ∇ ⊤ θ ℓ ( z n , θ ∗ ) for n = 1 , . . . , N are contained in the linear span of { u i } N i =1 , which is orthogonal to { u i } P i = N +1 , the following holds for any training input x

∗

C Implementations

C.1 Pseudocode of LA for IF approximation

Algorithm 1 I LA with KFAC [39, 17] sub-curvature approximation

1: Input : training data S , pre-trained parameter θ ∗ , number of LA samples M , two data samples z, z ′

2:

3: # Estimation of KFAC sub-curvature

4: Initialize activation statistics A l and gradient statistics G l for each layer l = 1 , . . . , L

5: for n = 1 , . . . , N (This computation can be conducted batch-wisely.) do

6: for l = , 1 , . . . , L do

7: Accumulate activation statistics A l and gradient statistics G l

8: end for {End accumulation for layers}

9: end for {End accumulation for training samples}

10: Compute eigendecomposition of activation/gradient statistics:

11:

12: # Sample Laplace approximated posteriors

13: for m = 1 , . . . , M do

14: Sample P -dimensional standard Gaussian random variable v m ∼ N ( v | 0 P , I P )

15: for l = , 1 , . . . , L do

16: Use reparameterization trick for each layer:

where diag( A ) is the diagonal components of matrix A , ⊙ denotes Hadamard product of two matrices, and the square root is applied for each component of the matrix.

17: end for

18: end for {End LA sampling { ψ m } M m =1 }

19:

20: # Compute MC estimator of non-linear IF approximation (12)

22:

23: Output : ˆ I LA ( z, z ′ )

C.2 Pseudocode of GEX

Algorithm 2 I

GEX 1: Input : training data S , pre-trained parameter θ ∗ , number of LA samples M , number of fine-tuning steps T , two data samples z, z ′ 2: 3: # Generating Geometric Ensemble (GE; [16]) 4: for m = 1 , . . . , M (This computation can be parallelized for multiple devices) do 5: Initialized the m -th checkpoint θ 0 m ← θ ∗ (or θ T m -1 ) 6: for t = 1 , . . . , T do 7: Apply stochastic optimization update (e.g., SGD with momentum): θ t m ← θ t -1 m 8: end for {End fine-tuning the m -th checkpoint θ T m } 9: end for {End generation of GE { θ T m } M m =1 } 10: 11: # Compute the non-linear IF approximation (10) 12: ˆ I GEX ( z, z ′ ) ← ∑ M m =1 [ ∆ ℓ θ ∗ ( z, θ T m ) · ∆ ℓ θ ∗ ( z ′ , θ T m ) ] 13: 14: Output : ˆ I GEX ( z, z ′ )

D Complexity analysis

In Table 6, we compare computational complexities of various IF approximations, including I GEX . We provide three computational complexities of each IF approximation method: time and space complexity for cache construction and the time complexity for evaluation. As discussed in Sec. 4.3, all IF approximations except I TracIn and I TracInRP can be implemented using only the final checkpoint, which is usually publicly released. However, the time and space complexity of cache construction are significantly different between the post-hoc methods: As mentioned in Sec. 2, the time and space complexity of I LiSSA for cache construction are proportional to the size of the training dataset N , as I LiSSA requires IHVP computation for each training sample. Therefore, I LiSSA cannot be used in datasets with many training samples. On the other hand, I Last -Layer requires high space complexity proportional to the square of the number of last-layer parameters P Last . As a result, I Last -Layer cannot be applied to classification tasks with many classes (e.g., ImageNet [12]). While I Arnoldi avoid this issue by using the principal subspace of H , we found that they suffer from the OOM issue on GPU RAM, as eigendecomposition computation (the DISTILL procedure in Schioppa et al. [52]) requires the space complexity proportional to N Arnoldi , the number of steps in I Arnoldi , which is about 100-200 in Schioppa et al. [52]. However, I GEX and I GEX -lin does not suffer from the high time and space complexity for cache construction, as the cache construction procedure of them is just post-hoc fine-tuning in Garipov et al. [16].

For the time complexity for evaluation, we assume each method computes a single influence pair (e.g., I ( z, z ′ ) or I ( z, z ) ). According to Table 6, all methods except I LA and I GEX require either backward computation ( C backward ) or JVP computation ( C jvp ). Among them, methods that require backward computation ( I LiSSA , I Last -Layer , and I TracIn ) do not support batch computation for multiple samples, as mentioned in Sec. 2. Therefore, computing self-influence for all training samples in practice is inefficient. On the other hand, methods that require JVP computation ( I TracInRP , I Arnoldi , and I GEX -lin ) are efficient for packages that offer forward-mode AD computation (e.g., JAX [6]). On the contrary, I LA and I GEX do not suffer from these issues, as they require only forward computation.

E Experimental settings

In the main paper, we conducted empirical studies in four different settings. We provide details on these settings in this section. We use four random seeds to compute the standard errors for all experiments in the main paper. We use 8 NVIDIA RTX 3090 GPUs for all experiments.

Two-circle classification task We use a modified two-circle classification task to study the side effects of damping and truncation in Sec. 4. We use sklearn.datasets.make_circles function to sample the training samples. We use 30 train samples for each class (circle) and add ten influential samples at the center. We use a two-layer fully connected NN with a hidden dimension of 200. We use jax.hessian function to compute the Hessian matrix of training loss and np.linalg.eigh to compute the principal subspace of H -1 .

Detecting and relabeling of mislabeled samples Noisy label settings [67] are used to study the distributional bias of the standard bilinear IF approximations. We use standard datasets in the computer vision domain: MNIST [33], CIFAR-10/100 [27], and SVHN [42]. For MNIST, we use VGGNet-B [55] with batch normalization [23] after each convolution layer, similar to pytorch implementation in https://pytorch.org/vision/main/models/vgg.html . We use ResNet-18 [20] for CIFAR and SVHN. We use random label corruption for 10% of randomly selected training samples for synthetic noise and CIFAR-N [64] for real-world label noise. We use a batch size of 1024 for all datasets. For all datasets, we use the cosine learning rate annealing [36] with a linear warm-up where the peak learning rate is 0.4, and the warm-up ratio (percentage of warm-up steps to training steps) is 10%. We use 0.0005 for the L2 regularization coefficient. We use 200 training epochs for all datasets.

To verify the scalability of our method, we also use ImageNet [12] with Vision Transformer [13] and MLP-Mixer [60]. We use the "S-32" setting due to the heavy computation of ImageNet. We use random label corruption for 10% of randomly selected training samples. We use a batch size 4096 and RandomResizedCrop, following Zhuang et al. [68]. We use the cosine learning rate annealing with a linear warm-up where the peak learning is 0.003 and the warm-up step is 10,000. We use AdamW [37] with 0.3 for the weight decay coefficient. We train 300 epochs for ImageNet, following Zhuang et al. [68].

To verify the effectiveness of our method in another modality, we train SWEM [53] to DBpedia [34], following Pruthi et al. [48]. However, Pruthi et al. [48] used this setting for a clean label case study, while we used it for noisy label detection with 10% label corruption. For text representation, we use SentencePiece [28] tokenizer and apply average pooling of word embeddings. We train 60 epochs using the AdamW [37] optimizer with a peak learning rate of 0.001 and a weight decay of 0.001. We performed a grid search on the search space mentioned in [53] to find these hyperparameters, aiming to achieve the 95.5% train accuracy mentioned in [48]. We provide our results in Appendix F. Deep -KNN was omitted because it took more than 15 hours to calculate for each random seed.

We use 20 dimensions for all projection-based methods (i.e., I RandProj , I TracInRP , I Arnoldi ). We use 100 iterations for I Arnoldi . Five checkpoints, evenly sampled from the latter half of training epochs, are used for I TracInRP and Deep -KNN . We use 10% of training samples to compute the neighborhoods in Deep -KNN . For CL , we use two-fold cross-validation. Thus, CL requires at least a separate pretraining level of computational cost. We use 20 initial training epochs for EL2N with five repetitions. We use 32 samples for I LA and I GEX . We use a cosine learning rate of 0.05 with 800 steps for the fine-tuning procedure to generate GE. We provide an ablation study exploring the impact of ensemble size in Appendix F.

Dataset pruning We conducted a dataset pruning task in Paul et al. [46] to validate I GEX for clean label settings. We use the same pre-training and post-hoc hyperparameters as noisy label settings for this task (except for the presence of mislabeled samples). Since the pruning capacity (i.e., the maximum number of samples to be pruned) of all datasets is not the same, we prune 40K samples for MNIST (within 1% test acc. degradation for all methods), 25K samples for CIFAR-10 and SVHN (within 0.5% test acc. degradation for F -score , EL2N , and I GEX ), and 20K samples for CIFAR-100 (within 4% test ACC. degradation for I GEX , F -score , I LA ).

Sepearation of data sources We also conducted a data source separation task in Harutyunyan et al. [19] to validate I GEX for clean label settings. In this task, we train a ResNet-18 on a mixed dataset of MNIST and SVHN, with 25,000 random subsamples for each training dataset. The other pre-train and post-hoc hyperparameters are the same as for noisy labels.

F Additional results

In this section, we conducted additional experiments for completeness. To this end, we provide an ablation study for the ensemble size of I GEX in Appendix F.1. We also provide additional results of Sec. 5 in Appendix F.2- F.4. In Appendix F.5, we report the noisy label detection performance with cross-influence. Finally, we study a purely post-hoc I TracInRP by using GE in Appendix F.6.

F.1 Ablation study for the ensemble size

To evaluate the effectiveness of the size of an ensemble in I GEX , we conduct an ablation study for the number of fine-tuned models M GE for the noisy label detection task in Sec. 5.1 with 10% label corruption. Table 7 shows that even a small number of checkpoints (about 8) can achieve better performance of GEX than other state-of-the-art IF approximations. While we used 32 checkpoints to ensure a low variance of results, small checkpoints - as small as 4 - may be sufficient in situations requiring a low computational cost.

F.2 Results on MNIST and SVHN

0

5000

10000

15000

20000

25000

30000

35000

40000

Pruned data

99.62

99.64

99.66

99.68

99.70

99.72

Test accuracy (%)

Arnoldi

EL2N

F-score

GEX

Random

RandProj

TracInRP

(a) MNIST

0

5000

10000

15000

20000

25000

Pruned data

96.9

97.0

97.1

97.2

97.3

97.4

97.5

Test accuracy (%)

Arnoldi

EL2N

F-score

GEX-lin

GEX

LA-KFAC

Random

RandProj

TracInRP

(b) SVHN

In this section, we provide the additional results of Sec. 5 for MNIST [33] and SVHN [42] with 10% label corruption. The overall tendency of MNIST and SVHN is similar to the results in Sec. 5. While EL2N is comparable (or even better) to I GEX on MNIST, their good performance is not generalized to more scalable and practical settings in CIFAR-N [64], and ImageNet [12].

F.3 Results on DBpedia

In this section, we provide the additional results of Sec. 5.1 for DBPedia [34] with SWEM [53]. Table 10 shows that GEX better discriminates noisy labels than other methods. In addition to the robustness of various model types (including VGGNet, ResNet, ViT, and MLP-Mixer) and noise types (synthetic label noise and real-world label noise), this result indicates that GEX has consistently excellent performance for various data types.

F.4 Results on 30% label corruption

In this section, we provide the additional results of Sec. 5.1-5.2 with 30% label corruption. Table 11-12 shows that I GEX achieves better detection and relabeling performance than other methods on higher label noise settings. This is consistent with the empirical evaluation in Sec. 5 and shows the practicality of I GEX .

F.5 Cross-influence results for noisy label detection

In this section, we apply the cross-influence on test dataset S test defined as

for the noisy label detection task in Appendix F.4. As the overfitting/memorization of mislabeled samples causes performance degradation for NNs, the estimated cross-influence values should be negative for these samples. Table 13 shows that IF approximations represent this tendency correctly. Mislabeled samples are best distinguished from clean samples using I GEX .

F.6 TracInRP with Geometric Ensemble checkpoints

As discussed in Sec. 2, I TracIn and I TracInRP are not purely post-hoc as they require intermediate checkpoints during pre-training. Motivated by I GEX , one can propose to replace the intermediate checkpoints of I TracInRP with post-hoc checkpoints generated by Geometric Ensemble. In this section, we evaluate this setting for the noisy label detection task in Sec. 5.1. Table 14 shows that this modification consistently outperforms I RandProj , which uses the last checkpoint θ ∗ only. Also, I TracInRP with GE performs better than the original I TracInRP except SVHN. However, this improvement does not outperform I GEX .

F.7 Further ImageNet results

Our first additional ImageNet experiment is the relabeling task presented in Sec. 5.2 on the ImageNet1K environment with ViT and MLP-Mixer. To this end, we follow the relabeling process in Sec. 5.2 with the estimated influence in Table 2. As ViT and Mixer use multi-task binary classification loss instead of cross-entropy loss, the relabel function in (16) cannot be applied to this setting. To mitigate this issue, we use well-known distillation loss [21] for detected influential samples. Therefore, these influential samples receive a milder signal than from the hard label of the initial training. Table 15 presents the relabeled test accuracy for I GEX and EL2N (the best method for noisy label detection except ours). It shows that I GEX can detect mislabeled samples that require relabeling more accurately than EL2N . The second additional ImageNet experiment is the dataset pruning task in Sec. 5.3 on the ImageNet-1K with MLP-Mixer. For this purpose, we reproduce Mixer-S-32 and estimate the self-influence of I GEX and EL2N score (which verified its scalability on the dataset pruning task in Sorscher et al. [58]). Then, we prune 512,466 samples (40%) with the smallest self-influence in ImageNet-1K and retrain neural networks with these pruned datasets. Table 16 presents the pruned test accuracy for I GEX and EL2N . Similar to the results shown in Fig. 4, I GEX demonstrates more effective identification of prunable samples than EL2N on the scalable ImageNet-1K dataset. Additionally, it is worth noting that EL2N cannot make use of open-source checkpoints and requires a computational cost of ( 10 ∼ 20 epochs) × (number of checkpoints) from an initialized neural network. In summary, the better-pruned accuracy and the lower computational cost further illustrate the effectiveness of I GEX in scalable settings.

G Limitations and Broader Impacts

While I GEX effectively captures the multi-modal nature of I LOO 's self-influence distribution, we do not delve into the theoretical guarantees, such as approximation bounds, for I GEX . Nevertheless, we believe that this concern can be addressed in future research by making assumptions about the SGD steps used in generating checkpoints. Furthermore, although the data pruning results of I GEX are significantly superior to other methods approximating IF, they do not outperform the performance achieved by state-of-the-art techniques like F -score and EL2N . This issue can be resolved by implementing the iterative pruning approach proposed in Harutyunyan et al. [19].

Our study examines the distributional bias present in current IF approximations and proposes an efficient solution. As a result, our approach has broad applicability in different scenarios. Specifically, our method offers enhanced efficiency in identifying mislabeled and prunable samples compared to existing approaches. Consequently, we believe that our method can contribute to enhancing and expediting the training process of energy-intensive deep learning models.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies approximations methods for data influence. The work identifies a common theoretical drawback behind popular approximation methods to Influence Function (IF) which suppresses their expressive power and affects their performance.\n\nThis work proposes a novel interpretation of existing IF approximations as some special form of Laplace approximation (LA) and points out that the drawback is due to the linearity of gradients and the singularity of Hessian. This work proposes an original IF approximation method that circumvents these issues, which removes the linearization to ease the bilinear constraint and leverages Geometric Ensemble (GE) for non-linear losses. Both conceptually and empirically, this work demonstrates significant improvement over existing IF approximation methods. The work includes a variety of experiments including many use cases.\n\nSTRENGTHS:\nThe paper is nicely written. The narrative is smooth and the conceptual development is attractive. The problem being considered is of interest and finding of common drawback behind the mechanism of existing IF approximations is novel and insightful. Visualizations of conceptual findings are very helpful and give a straightforward presentation to help to understand.\n\nEvaluations are thorough. Various use cases are considered and the empirical performance is satisfactory. Comprehensive details for experiments are given in Appendix.\n\nWEAKNESSES:\nInfluence Function (IF) refers to the celebrated method proposed in < Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885–1894. PMLR, 2017.> that is used to estimate the quantity in Eq. (1). Equation (1) is essentially the definition for LOO. It is NOT the \"ground truth\" of Influence Function (IF). Eq. (2) is the \"Influence Function\". It is NOT named \"I_{Hess}\".\n\nThe transition to Section 4.1 needs more elaborations. I_GEX proposed in this subsection is the core contribution of the paper, yet it isn't thoroughly discussed before moving on to discussing the limitation of other methods. Similarly, an introduction to LA is much needed as it lays important grounds for the development of this work. I may consider increasing my score if this can be reasonably addressed.\n\nThough the conceptual flow of the paper is smooth, some important parts remain unclear. For example, what is Eq. 9 exactly and how is it calculated in actual implementation? How to conceptually interpret Eq. 10 and how it differs from Eq. 12?\n\nQUESTIONS:\nHow does the proposed method compare to TracIn (not TracInRP)? And how does the result compare to the actual counterfactual (LOO)?\n\nAppendix should not be submitted within the main paper, which has a page limit of 9.\n\nWhat is the \"retraining cost of IF\"? IF does not need training or \"retraining\".\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper provide a novel connection between IF approximations and LA, and introduces a new IF approximation method. The motivation takes advantage from two observations, the removal of linearization can alleviate the bilinear constraint and, the utilization of Geometric Ensemble is advantageous.  Empirical results demonstrates the advantage of the proposed method with reduced computational complexity.

STRENGTHS:
The paper provide a novel connection between IF approximations and LA, and introduces a new IF approximation method. The motivation takes advantage from two observations, the removal of linearization can alleviate the bilinear constraint and, the utilization of Geometric Ensemble is advantageous.  Empirical results demonstrates the advantage of the proposed method with reduced computational complexity. 

I enjoyed a lot reading the literature review and the analysis offered by the paper. The paper also introduced several application scenarios where the proposed method demonstrated obvious performance boost, including the noisy label identification task.

WEAKNESSES:
W1. The  method is only justified on small dataset such as MIST and SVHN. Is it possible to verify the effectiveness of the method on larger dataset such as ImageNet1K?

QUESTIONS:
Please see weakness about the scale of the training data above.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method for approximating influential examples. It treats losses as non-linear functions, addresses the singularity problem of hessians and does not require multiple checkpoints or JVP computations for the influence. The authors also highlight a connection between Influence Functions (IF) and Laplace Approximation (LA) and discuss its limitations. They propose an approximation using Geometric Ensembles (GE) instead. The authors show empirically that their approach outperforms existing IF approximation methods on downstream tasks for noisy label detection, relabeling, dataset pruning and data source separation.\n\nSTRENGTHS:\n+ The paper aims to address two important limitations of influential examples: singular nature of hessians and linearity of influence functions.\n+ The paper is well written, related work and existing limitations are properly discussed.\n+ The paper also discusses and benchmarks their approach on several important downstream tasks including noisy label detection, relabeling and dataset pruning on several datasets and models.\n\nWEAKNESSES:\n+ The authors emphasize bilinear approximation but they do not explain what bilinearity applies to in the approximations. It would be good to explain it briefly to make it clear.\n+ The overall motivation of GEX using dirac distribution in section 4.1 seems a bit unclear. It seems that GEX is highlighted by surfacing the connection between IF and Laplace Approximation (LA) and the limitations of p_{LA} but it is still unclear why GE distribution (Eq. 9) was chosen.\n+ It’s a bit challenging to follow the descriptions of Figures 2 and 3 since the authors refer both to influence and self-influence. Self-influence also means potentially mislabeled and it is unclear whether it is what the authors mean. Typical vs influential annotations on those figures are also a bit unclear. What does typical mean in this case ?\n\n**Minor comments**\n+ Figure 2: Are 2a - 2d: axes annotations are missing.  Perhaps it would be good to mention that these are histograms in advance or in figure description.\n+ Eq 9: parameter \\psi is not explained in that section.\n\nQUESTIONS:\n+ Line 167: what’s the motivation of choosing Dirac delta distribution\n+ Figure 2: Are 2a - 2d computed for self-influence or influence w.r.t. Test examples ? Have we compared I_GEX with I_Hess for influence on test examples ?  What does typical vs influential mean for self-influence ? High self-influence means that these are potentially mislabeled. So we want to say that the green ones are mislabeled ? \n+ In terms of evaluation were there any experiments made for the original tracin without random projections for the Table 1.\n\nLIMITATIONS:\nThe limitations of the work are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper identifies that existing influence functions suffer from a fundamental drawback due to their bilinear form. To address, this they propose an influence calculation that is nonlinear                                                    .

STRENGTHS:
The GEX influences seem to outperform all baselines when detecting label noise on CIFAR10-like datasets. 

The approach is motivated by principled failures of existing influence functions due to factors such as singularity of the Gaussian.

WEAKNESSES:
There is a lot of information pushed to the appendix that is reference in the main text, to the extent that it more difficult to really understand without having the appendix at hand. The current main text was too sparse for me to really understand any particular component. 

It seems the experiments largely focused on MNIST/SVHN/CIFAR10 variants. It would be nice to see some kind of breadth in application or scalability to more than CIFAR10.

QUESTIONS:
I did not find it very clear what benefit or cards the geometric ensemble brings to the table. 

It would be nice to have an actual example using only open source checkpoints, since this is one of the purported benefits of the research. 

In the limitations, the authors believe their method can expedite the training process of energy-efficient deep learning models.

LIMITATIONS:
In the appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors point out that standard approximations of Influence Function (IF) suffer from performance degradation due to oversimplified influence distributions caused by their bilinear approximation, suppressing the expressive power of samples with a relatively strong influence. Therefore, they propose a new interpretation of existing IF approximations as an average relationship between two linearized losses over parameters sampled from the Laplace approximation (LA). By doing so, they highlight two limitaions and propose GEX to improve them. The proposed GEX outperforms standard IF approximations in downstream tasks, including noisy label detection, relabeling, dataset pruning, and data source separation.\n\nSTRENGTHS:\n1. This paper is well written. The details of movtivation, methodology, experiments and proofs are described clearly.\n2. The insight of this paper is quite reasonable and the proposed approach looks novel.\n3. The proposed method is verified on a wide range of downstream tasks, including noisy label detection, relabeling, dataset pruning, and data source separation.\n\nWEAKNESSES:\n1. In table 2, the gap between GEX and the other settings is very large. It seems that the other settings are not specificly designed for this task. Is this comparison meaningful?\n2. Except table 2, the other experiments are executed on some small and relatively easy dataset, I am not sure whether the conclusions from these results are solid enough.\n\nQUESTIONS:\n1. I think nosiy label detection combined with relabeling mislabeled samples is very useful in many practical scenarios. But the experiments are only executed on CIFAR, which is a quite small dataset. What is the result if we use GEX to detection noisy labels in Imagenet and relabel them?\n2. How can this approach help some pratical scenarios?\n\nLIMITATIONS:\nSee weakness. Most experiments are executed on small and relatively easy datasets. The effectiveness of this approach on those more challenging and large benchmarks is not explored.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}