Title: Theoretical and Practical Perspectives on what Influence Functions Do
Venue: NeurIPS 2023
Accepted: yes
Keywords: Explainable AI

Influence Functions

Training Data Attribution
Abstract: Influence functions (IF) have been seen as a technique for explaining model predictions through the lens of the training data. Their utility is assumed to be in identifying training examples "responsible" for a prediction so that, for example, correcting a prediction is possible by intervening on those examples (removing or editing them) and retraining the model. However, recent empirical studies have shown that the existing methods of estimating IF predict the leave-one-out-and-retrain effect poorly. 
In order to understand the mismatch between the theoretical promise and the practical results, we analyse five assumptions made by IF methods which are problematic for modern-scale deep neural networks and which concern convexity, numeric stability, training trajectory and parameter divergence. This allows us to clarify what can be expected theoretically from IF. We show that while most assumptions can be addressed successfully, the parameter divergence poses a clear limitation on the predictive power of IF: influence fades over training time even with deterministic training. We illustrate this theoretical result with BERT and ResNet models.
Another conclusion from the theoretical analysis is that IF are still useful for model debugging and correcting even though some of the assumptions made in prior work do not hold: using natural language processing and computer vision tasks, we verify that mis-predictions can be successfully corrected by taking only a few fine-tuning steps on influential examples.

Theoretical and Practical Perspectives on what Influence Functions Do

Abstract

Influence functions (IF) have been seen as a technique for explaining model predictions through the lens of the training data. Their utility is assumed to be in identifying training examples "responsible" for a prediction so that, for example, correcting a prediction is possible by intervening on those examples (removing or editing them) and retraining the model. However, recent empirical studies have shown that the existing methods of estimating IF predict the leave-one-out-andretrain effect poorly. In order to understand the mismatch between the theoretical promise and the practical results, we analyse five assumptions made by IF methods which are problematic for modern-scale deep neural networks and which concern convexity, numeric stability, training trajectory and parameter divergence. This allows us to clarify what can be expected theoretically from IF. We show that while most assumptions can be addressed successfully, the parameter divergence poses a clear limitation on the predictive power of IF: influence fades over training time even with deterministic training. We illustrate this theoretical result with BERT and ResNet models. Another conclusion from the theoretical analysis is that IF are still useful for model debugging and correcting even though some of the assumptions made in prior work do not hold: using natural language processing and computer vision tasks, we verify that mis-predictions can be successfully corrected by taking only a few fine-tuning steps on influential examples.

1 Introduction and related work

Influence Functions (IF) [CS82] have been regarded as a tool that can trace model behavior on any example to the training examples [KL17, PLKS20, GRH + 21]. Their theoretical justification lies in the ability to predict loss changes on a specific test point when training on a perturbed loss obtained by removing or down-sampling a given training point. In the case of an undesired model behavior on a test-point, the influential training examples for that test point have been assumed to be the ones "responsible" for the prediction so that intervening on those - e.g., by removing them and then retraining the model, or by taking additional fine-tuning steps [GRH + 21] - would result in a change in the loss or prediction. It has been confirmed that for linear models it is indeed the case [KATL19].

Recently, in their extensive experiments, [BPF21] and [KS21] could not find empirical support for the claim that IF approximate the Leave-Some-Out Retraining (LSOR) effect on the loss in deep neural networks. In particular, they show that the correlation between the ranking of training examples produced by LSOR and the IF-based ranking is low and considerably affected by choice of hyperparameters. How can this discrepancy be explained? And, given that the theoretical justification for IF lacks empirical support, does it mean that IF should be abandoned as an explainability and debugging tool altogether?

In this work we clarify what question Influence Functions (IF) actually answer. We first identify assumptions which are either implicit or not investigated in prior work: these concern convexity,

numeric stability, training trajectory, and the the parameter divergence when retraining on a new loss. In principle, any of these assumptions might be problematic and be the reason why the original theoretical justification for IF is not supported empirically . However, we show how to address most of them successfully so that they cannot be the reason for the aforementioned discrepancy. Unfortunately, we also show that the parameter divergence is indeed problematic and requires to revise both theoretical and practical expectations about IF. This analysis paves the way to clarifying the question that IF can answer. As a first step in this direction, we need to distinguish two approaches to computing influence. The Hessian-based Influence Functions (HIF) [CS82, KL17] have a rigorous theoretical justification in statistics, but rely on strict convexity assumptions, which are not met in Deep Learning and which previous analyses of HIF in the Deep Learning literature still rely on [KL17, BNL + 22]. Our first contribution concerns HIF and is twofold: we prove (Theorem 1) that this unsatisfied convexity assumption is not as problematic for HIF as one may think-given a stationary point for the original loss, there is a nearby stationary point for the perturbed loss which can be approximated by using HIF. However, we point out a more serious problem with HIF: there is no guarantee that by retraining on the perturbed loss one would get to that stationary point . This observation provides an additional support to the second popular approach to IF, TracIn [PLKS20], which explicitly models the training dynamics and which additionally does not require to compute the expensive inverse Hessian vector product as it only uses gradient information. 1 Despite TracIn being grounded in the training dynamics, we unveil a hidden additive modeling assumption in [PLKS20] that prevents it from correctly modeling the (re)-training dynamics . Our second contribution is thus to provide a theoretical analysis (Theorem 2) of how training trajectories change when perturbing the loss. This is a key result as it suggests that two training trajectories differing by a small loss perturbation could diverge over time to the point of violating a first-order expansion assumption that IF make , thus making IF's predictions unreliable. Our further theoretical and empirical investigation confirms this conjecture. Therefore, what IF can do is to predict parameter changes when fine-tuning for a limited number of steps on the perturbed loss. This requires to adjust how IF are evaluated (Section 5) and applied (Section 6) . In order to confirm the conjecture and re-adjust expectations regarding IF, we need a deeper analysis of the trajectory divergence.

Our third contribution is thus to validate the conjecture on trajectory divergence and its consequences for IF. We first prove (Theorem 3), using a discrete version of Gronwall's Lemma [Gro19], an upper bound on the parameter divergence when (re)-training on a perturbed loss. We then empirically verify that the bound is sharp in Section 5 and that IF can approximate parameter changes along the perturbed trajectory only for a limited amount of time . We then empirically verify that this leads to a fading (of accuracy) of IF predictions over time. Therefore, we theoretically demonstrate and empirically verify that IF can in general answer only what happens when fine-tuning on a perturbed loss for a limited amount of time , instead of a general retraining setting.

Therefore, our new theory indicates that IF have been used incorrectly as the emphasis has been on their LSOR potential. On the positive side, it suggests an alternative way of using IF that indeed yields substantial empirical improvement. To demonstrate that, as our final contribution, in Section 6 we propose and verify a very simple method for correcting mis-predictions by taking only a few gradient steps on influential examples. Our proposal is related to model editing [DCAT21, inter alia] in that the latter also aims at changing model predictions. However, there the model is given and the modifications are done on its parameters whereas IF aim at understanding how specific training examples are responsible for the current model behavior and editing predictions through the data. We leave for future work building connections between model editing and IF.

For the case of Hessian-based influence functions, the recent work of [BNL + 22] has proposed to resolve the discrepancy between the theory and the empirical evidence by evaluating them using the proximal Bregman response function. However, [BNL + 22] relies crucially on the convexity assumption: while under this assumption there is agreement between our work and their findings (e.g. our parameter divergence drives the linearization error [BNL + 22, Sec. 4.4]), our theory applies also the the case where the Hessian is not positive definite. Moreover, our work covers also the scope of Gradient-based influence functions including TracIn [PLKS20].

2 Definition of Influence Functions and Notation

The different methods proposed to define Influence Functions share a common goal: forecasting the change in the prediction on a test example when up (or down-) weighting a training example . This is achieved by tracing the effect that re-weighting a training example has on the model parameters.

Removing or adding a training point can be modeled by a perturbation of the loss function; such a perturbation can be made smooth by modeling the weighting of a training point by a continuous parameter ε . More generally, let L ( θ ) denote the loss function where θ ∈ R N are the model parameters. We model loss perturbations by introducing a variation of the loss , which is a smooth function L ( θ, ε ) depending on an additional vector parameter ε ∈ R Q and coinciding with the original loss for ε = 0 . For example if l x denotes the loss on a given training point x , we set L ( θ, ε ) = L ( θ ) + εl x . While the scalar case Q = 1 is the one commonly considered, e.g. [KL17, PLKS20], we introduce the vector one Q > 1 which arises naturally when considering the effect of re-weighting multiple points differently, e.g. when modifying the weights in a mixture of different data-sets.

The IF method of choice then predicts what would happen if training on L ( θ, ε ) instead of L ( θ ) . The final parameters are modeled as a function θ ε of the perturbation parameter; assuming that such a function is well-defined and sufficiently smooth , one then makes a first order expansion θ ε glyph[similarequal] θ 0 + ε T ∇ ε | 0 θ ε , where ∇ ε | 0 θ ε is the Q × N -dimensional Jacobian at ε = 0 (see also Section A).

Under this first order assumption it is then straightforward to measure the change in the loss l z corresponding to a test point z :

We emphasize that (1) is general to different IF methods , which differ in the specific derivation of ∇ ε | 0 θ ε .

3 Problematic assumptions made by Influence Functions

3.1 Problematic Assumption #1: Convexity can be used to show that θ ε is a function of ε

In order to show that for each value of ε there is a single value of θ ε (so that one can model the final parameters as a function of the perturbation parameter), HIF relies on strict convexity of L . While this assumption is realistic for the statistical models considered in [CS82], this is not the case for neural networks . Even when introducing a regularization term, the loss of a neural network is not even weakly convex, and optimization methods usually converge to saddle points [DPG + 14]. To the best of our knowledge, previous analyses of HIF in the Machine Learning literature, e.g. [KL17, BPF21, BNL + 22], have relied on some form of strict convexity . In Section 4.1 we will revisit HIF and prove (Theorem 1), roughly speaking, that near a given stationary point θ 0 for the original loss L , there is a stationary point θ ε of the perturbed loss L ( θ, ε ) which can be modeled as a function of ε and such that ∇ ε | 0 θ ε is given by -H -1 θ 0 ∇ 2 ( ε,θ ) | (0 ,θ 0 ) L as in [CS82] (see Section A regarding the usage of ∇ 2 ).

3.2 Problematic Assumption #2: The model Hessian is not degenerate

As HIF requires to apply the inverse model Hessian to ∇ 2 ( ε,θ ) | (0 ,θ 0 ) L , one needs to ensure that inverting the Hessian is numerically stable . It has been empirically demonstrated [GKX19] that most eigenvalues of the Hessian tend to cluster near 0 . Numerically, this results in a considerable source of errors and instabilities when estimating HIF; while regularization can alleviate this problem, it introduces a hyper-parameter in the definition of HIF; the minimal value of such a hyper-parameter ensuring numerical stability depends on the smallest negative eigenvalue of the Hessian . Unfortunately, in realistic settings, this can be larger in absolute value than reasonable values for the regularization parameter . For example in our ResNet experiments regularization is of the order 10 -4 , while the smallest negative eigenvalue is glyph[similarequal] 5 . In Section 4.2 we discuss how the Arnoldi-based Influence Functions (abbr. ABIF) (which were introduced by [SZTS22] for computational efficiency) can be used to address such instability issues.

3.3 Problematic Assumption #3: Training trajectory can be ignored in Hessian-based Influence

Even if we solve the Problematic Assumption #1 for HIF, there is no guarantee that when actually re-training from scratch on L ( θ, ε ) one would converge to the θ ε given by Theorem 1 because the training trajectory is disregarded in HIF. As optimization is performed via some form of stochastic gradient descent, [PLKS20] propose TracIn which averages gradient dot-products across checkpoints in order to take into account the path taken by the training process. Importantly, for a single checkpoint θ 0 TracIn estimates ∇ ε θ ε as -∇ 2 ( ε,θ ) | (0 ,θ 0 ) L , so the inverse Hessian vector product does not need to be computed. While it seems that TracIn takes into account the training trajectory, in the next Assumption we identify an issue with the way it models the training trajectory.

3.4 Problematic Assumption #4: The training trajectory can be modelled additively

The analysis of TracIn in [PLKS20] is based on a first-order expansion of the final change of the loss of a test point in terms of the gradient steps across the training trajectory. While this argument seems mathematically convincing, it overlooks that if one point is removed, or slightly up-sampled / down-sampled, the subsequent training trajectory is modified . Let θ ε,t denote the value of the parameters after t steps when doing gradient descent on L ( θ, ε ) . Denoting by T the end-time, TracIn derives

where η t is the learning rate at time step t . Now, the right-hand side in formula (2) is purely additive in the time steps; and addition is commutative, so the order of the time steps does not matter . One way to see that this is problematic is by making L time-dependent so that it differs from L only at a specific time step t . In this case ∇ 2 ( ε,θ ) L would be non-zero only at t and formula (2) would consist of a single term. However, we would expect the perturbation at t to affect the following time steps , so we should have at least T -t -1 terms on the RHS for (2). In Section 4.3 we compute ∇ ε θ ε,T looking at the whole training trajectory and discover an additional first-order term that is missing from TracIn : this term models the dependency of a time step on the earlier ones . Concurrent work [GWP + 23] also criticizes the additive assumption in TracIn on empirical grounds and proposes to build (re)-training simulators which are unfortunately computationally expensive as a new simulator must be fitted on the training set for each test point.

3.5 Problematic Assumption #5: θ ε can be expanded to first order in ε

After we derive a formula for θ ε,T and ∇ ε θ ε,T we realize that the latter can grow in norm in T . However, if θ ε,T can be Taylor-expanded in ε , we need θ ε,T -θ 0 ,T to be O ( ε ) and ∇ ε θ ε,T to be O (1) . If this is not the case, the whole IF approach described in Section 2 breaks down because IF approximate the parameter change θ ε,T -θ 0 ,T using the Taylor expansion ε T ∇ ε θ ε,T , but the conditions to apply such a Taylor expansion are not satisfied .

In Section 4 we show how assumptions #1-#4 can be successfully addressed which makes them not as problematic as they may first appear. However, for assumption #5 we will see that it puts a substantial limitation on the predictive power of IF. At the same time it allows for a new, locally bound perspective on IF - that influence holds for a limited number of fine-tuning steps (Section 5). Based on this finding, in Section 6 we propose a simple approach to use IF to correct mis-predictions which is theoretically grounded and is in addition much less compute intensive than those that involve re-training (e.g. [KL17]).

4 Addressing the problematic assumptions

Proofs of all results are in the Appendix and we provide some motivation for the proof in the main text.

4.1 HIF does not need Assumption #1

Previous work [CS82, KL17, BPF21, BNL + 22] on Hessian-based Influence Functions (HIF) has assumed that L is strictly convex in order to claim that 1) the minimum is unique so that θ ε can be modeled as a function, and 2) to use the Implicit Function Theorem to differentiate through the optimality condition.

Here we will just assume that the Hessian is not singular at θ 0 ; by requiring that the final gradients do not change as we change ε we prove:

Theorem 1. Assume that ∇L is C k ( k ≥ 1 ) and let θ 0 ∈ R N ; assume that the Hessian H θ 0 = ∇ 2 θ | θ 0 L is non-singular; then there exist neighborhoods U of θ 0 and V of 0 ∈ R Q , and a C k -function Θ : V → U such that Θ(0) = θ 0 and Θ( ε ) ∈ U is the unique solution in U of the equation

Moreover, the gradient of Θ at the origin is given by:

Idea of the proof. Existence of Θ( ε ) is formulated as a solution to the local problem eq. (3); building a local solution does not require convexity if one uses the Implicit Function Theorem.

The requirement that ∇ θ L (Θ( ε ) , ε ) is constant in ε has allowed us to establish a link between the training under different losses { θ → L ( θ, ε ) } ε . If we assume that θ 0 is a stationary point , i.e. ∇ θ L ( θ 0 ) = 0 , we can strengthen the conclusions:

Corollary 1. Under the assumptions of Theorem 1:

If θ 0 is a stationary point of L , then each Θ( ε ) is a stationary point of the loss θ ↦→L ( θ, ε ) .

If θ 0 is a local (strict) minimum of L , for ε sufficiently small, Θ( ε ) is a local (strict) minimum of the loss θ ↦→L ( θ, ε ) .

Let Q = 1 (hence epsilon is a scalar) with L ( θ, ε ) = L ( θ ) + εl x ( θ ) , where l x is the loss corresponding to a specific training point x . We then obtain the classical result [CS82]:

4.2 If Assumption #2 is not satisfied, use Arnoldi-based Influence Functions

Theorem 1 requires that H θ 0 is non-singular. If the Hessian is singular, we just need to keep fixed those parameters that are responsible for the degeneracy . More precisely, we diagonalize H θ 0 ; we let P 1 be the subspace spanned by the eigenvectors corresponding to the non-zero eigenvalues and let P 0 be its orthogonal complement, that is, the kernel of H θ 0 . Up to an orthogonal transformation of the parameters, we can assume that P 1 is spanned by the first N 1 -coordinates and decompose θ = ( ϑ, ϕ ) ∈ R N 1 × R N 2 so that H ϑ 0 = ∇ 2 ϑ L (( ϑ 0 , ϕ 0 )) is non-singular. We then apply Theorem 1 to the restricted variation ( ϑ, glyph[epsilon1] ) ↦→ L (( ϑ, ϕ 0 ) , glyph[epsilon1] ) . In terms of the original parameters θ , this means that the function Θ( ε ) is constrained to lie in θ 0 + P 1 , keeping the ϕ -component constantly equal to ϕ 0 . Concretely, we can approximate P 1 using the Arnoldi iteration; therefore, we can address the failure of Assumption #2 by using Arnoldi-based Influence Functions (ABIF) [SZTS22], which approximate P 1 using the subspace spanned by the eigenvectors corresponding to the top-k (in absolute value) eigenvalues of the Hessian. The fact that ABIF stabilizes the estimation of influence scores has been observed empirically in previous work: in [SZTS22, Fig. 2] where using ABIF instead of the full Hessian improves the retrieval of mislabeled examples, and in [FLP + 23, Fig. 3] where ABIF is the best solver on the QA task and is comparable to the other solvers on the text-completion task.

4.3 The training trajectory can be traced to address Assumptions #3-#4

We need to improve our notation to correctly trace the training trajectory. The first issue is to keep track of the parameters across the time steps; the second issue are sources of non-determinism ,

e.g. batch selection or random state for dropout. When comparing training trajectories for different values of ε we want our notation to account for sources of non-determinism, as they might increase the difference between the training trajectories.

To address the first issue, we let θ ε,t be the value of the parameters after training on L ( θ, ε ) for t steps. In particular, θ ε, 0 denotes the initial value condition that we assume held fixed at θ init for different values of ε . As random state is a function of the training step (e.g. the batch to use at step t ), to address the second issue, we just need to allow both the loss and the variation to depend on the train step, denoting them by L t and L t .

To simplify the exposition and for consistency with [PLKS20] we assume that models are trained with stochastic gradient descent. Letting η t be the learning rate at step t we prove:

Theorem 2. Assume that the model is trained for T time-steps with stochastic gradient descent. Denoting by H t the Hessian ∇ 2 θ | θ 0 ,t L t , then the final parameters θ ε,T satisfy:

Idea of the proof. One can explicitly write a system of equations for θ ε,T and apply the operator ∇ ε | 0 to the solution of this system.

Note that the second term on the RHS of (6), which is missing from (2), takes into account the contribution of the earlier time steps that is missing from the analysis of [PLKS20]. A practical consequence of this second term is that the norm of ∇ glyph[epsilon1] | 0 θ ε,T might grow (in T ) more quickly than (2) would suggest: this is closely related to Assumption #5 which requires ∇ glyph[epsilon1] | 0 θ ε,T to be O (1) . In particular, for a constant learning rate, while (2) suggests a linear growth in the time step T , we will empirically verify in Section 5.1 that the growth is super-linear.

In Section K we illustrate the additional modeling error introduced by TracIn when computing the Jacobian ∇ glyph[epsilon1] | 0 θ ε,T in the case of retraining BERT with SGD.

4.4 Assumption #5 becomes problematic over time

To address Assumption #5 we need to look into the parameter divergence ‖ θ ε,T -θ 0 ,T ‖ between training on L and L . The training dynamics is a discrete version of an ODE for which uniqueness and differentiability of the solutions with respects to the initial conditions can be established by Gronwall's Lemma [Gro19]. The parameter ε can itself be considered an initial condition and we can prove a discrete version of Gronwall's Lemma to bound the parameter divergence. For simplicity of notation we prove the result for stochastic gradient descent, but we also sketch in the Appendix how to modify the argument to deal with optimizers.

Theorem 3. In the setting of Theorem 2 assume that, for t ≤ T , θ ε,t lies in a bounded region R such that

and that for θ ∈ R each loss L t ( θ, ε ) and its gradient wrt. θ are A -Lipschitz in θ . Then

Idea of the proof. If the time steps were infinitesimal, i.e. if time was made continuous, the evolution of θ ε,t would be governed by an ODE; then the bound (8) would be straightforward by applying the classical Gronwall's Lemma [Gro19]. In our case we just need to modify the ODE arguments to work with discrete time.

Note that the bound (8) is quite pessimistic as it involves an exponential of the integrated learning rate ∑ s<T η s . This means that as ∑ s<T η s increases, the parameter divergence is no longer O ( ε ) and

(a)

(b)

Difference

10-2

Difference

10-2

10-3

10-3

Parameter

10-4

ε=1e-06

ε=1e-05

Parameter

10-4

ε=1e-06

ε=1e-05

10-5

10-5

ε=1e-03

ε=1e-03

10-6

=1e-02

10-6

ε=1e-02

ε=1e-01

ε=1e-01

0.000

0.001

0.002

0.003

0.004

0.000.25

0.500.751.001.25 1.50

1.75

2.00

Integratedlearningrate

Integratedlearningrate

the crucial Assumption #5 is no longer satisfied. This observation leads to a few crucial conclusions:

An IF method can predict θ ε,t only for a limited amount of time-steps: it is therefore incorrect to evaluate IF methods on LSOR or retraining from scratch.

IF methods need to be evaluated on what they can potentially do; therefore the evaluation setup should consist of fine-tuning on the perturbed loss only a limited amount of steps with evaluation metrics being reported as a function of the step.

Applying IF for correcting mis-predictions should also involve a time-bound scenario: we propose such a method in Section 6.

Sources of non-determinism between two training runs will likely increase the parameter divergence. So one should try to reduce this with deterministic training . For example, in the case of re-weighting a point x , i.e. setting L = L + εl x , one should make sure to use the same batch B t for the loss L at time step t across training runs for different values of ε .

5 Illustrating the Theory

In this section we first demonstrate Theorem 3 empirically and then verify that the predictive power of influence scores degrades over time. Full details of our experimental setup are reported in the Appendix. We consider binary classification for nlp, where we fine-tune BERT on SST2; for computer vision we consider multi-class classification where we train from scratch ResNet on CIFAR10. All our experiments use deterministic training : the order of the training batches for the loss L is held fixed across different runs, as well are the random generators when dropout is used.

5.1 Illustrating Parameter Divergence (Theorem 3)

Theorem 3 provides an upper bound when re-training on a perturbed loss. Such a bound is rather pessimistic as it involves the integrated learning rate. We therefore investigate empirically what happens with some typical Deep Learning setups. We take an intermediate checkpoint and keep training on a new loss obtained by up-sampling 16 training points with a weight ε , that is: L t = L B t + ε · L B , where B t is the training batch for step t and B is the batch of 16 points selected for up-sampling. For each time step t we then compute ‖ θ ε,T -θ 0 ,T ‖ and then plot it against the integrated learning rate, see Figure 1.

Unfortunately, we observe that in these experiments the upper bound in Theorem 3 is matched by a lower bound with the same exponential divergence . We observe a first phase of quick divergence and then a second one in which the divergence is slower. For the second phase a linear fit of log ‖ θ ε,T -θ 0 ,T ‖ against the integrated learning rate appears to be strong: for example for BERT we obtain an R 2 of at least 0 . 9 across the different values of ε . The fitted slope, corresponding to A in Theorem 3 depends on ε and varies between 30 and 130 .

(a)

(b)

1.25

1.00

Hessian

Hessian

0.75

Tracln

0.3

Tracln

0.50

Optimizer

0.2

0.25

R(t)

R

0.00

0.1

0.25

0.0

0.50

0.75

0.1

0

200

400

600

800

1000

0

500

1000

1500

2000

2500

3000

3500

4000

Step

Step

5.2 Illustrating the fading of influence

We now verify that the predictive power of influence scores fades over time. We again fix a model checkpoint θ init and select 32 training points and 16 test points. For each training point x we retrain on L x t = L B t -1 100 l x , i.e. x has been down-sampled; for each test point z and time step t we then compute the loss difference δ ( z, x, t ) = l z,x,t -l z,t where l z,x,t is obtained when (re)-training on L x t and l z,t is obtained when training on the vanilla loss L t = L B t . Again, we have kept the order of the batches B t the same when re-training. Now, at the original checkpoint θ init we can compute the influence scores IF ( z, x ) for different methods, e.g. TracIn or HIF (using the the Conjugate Residual method 2 ). For each time step we thus have 32 × 16 values of δ ( z, x, t ) that can be linearly regressed against IF ( z, x ) ; the corresponding Pearson correlation R ( t ) then measures the predictive power of influence scores on the loss shifts when re-training . We repeat the experiments for ResNet 9 times and for BERT 25 times, with a different selection of train and test points, so that we obtain confidence intervals for the resulting time-series R ( t ) . Ideally, the theory behind an influence method predicts R ( t ) ∼ 1 . However, as discussed above, Assumption #5 is indeed problematic and, because of the parameter divergence illustrated in 5.1 we expect R ( t ) to degrade over time.

As Figure 2 demonstrates, this is indeed the case. The predictive power is high for BERT after a few re-training steps and then degrades quickly oscillating around 0 (which is covered by the confidence intervals). For ResNet, the predictive power is never as high, but it still degrades monotonically over time, as predicted by the theory . As we see in Section 6, the proponents retrieved for ResNet are less effective at correcting mis-predictions than those retrieved for BERT: we conjecture that this is related to the worse predictive power of IF in the case of ResNet. As BERT was trained with the Adam Optimizer, we also considered a variant which takes into account the optimizer's pre-conditioner by multiplying the gradients by the square root of the pre-conditioning matrix. In this case the predictive power is worse than for the vanilla version of TracIn. For BERT, more plots and a further discussion about TracIn are included in the Appendix (Section G.1). In the Appendix (Section G.2), we also illustrate the fading of influence for another NLP pre-trained model, T5. While we have illustrated the fading of influence for both BERT and ResNet, in the latter case the peak of correlation is quite low; we repeated the same experiments on the ViT (Section G.3), where we find a high correlation peak as in the case of BERT.

6 Using Influential Examples for Error Correction

[KL17] propose to correct model mis-predictions by first using influence scores to retrieve the examples most responsible for a given prediction, and, after correcting them, retraining the model . Computational considerations aside, a key conclusion from the theory in Section 3 and the empirical verification in Section 5.2 is that IF only predict influence over a limited number of training steps. In this section we demonstrate that IF can still be used to correct model mis-predictions by taking a few fine-tuning steps on influential examples .

(a)

(b)

1.0

1.0

0.9

0.8

0.8

0.7

0.6

0.6

Prop.success

Prop.success

0.5

Prop.retention

0.4

Prop.retention

Opp. success

Opp.success

0.4

Opp.retention

Opp.retention

Baseline success

0.2

Baseline success

0.3

Baseline retention

Baseline retention

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

0.0

0.2

0.4

0.6

0.8

1.0

epsilon

epsilon

Concretely, we propose to correct mis-predictions at a given test point z by first identifying a batch of influential examples B and then taking a few fine-tuning steps on the perturbed loss L = L + ε · l B . We propose two methods: (1) Proponents-correction : we identify the set B of top-k proponents for the current x and relabel them to what should be the correct prediction on x ; (2) Opponents-tuning : as opponents oppose the current prediction , we take B to be the set of top-k opponents of x .

We investigate how well these error-correction techniques work on SST2 (BERT) and CIFAR10 (ResNet). As a baseline, we randomly sample a set B of training points with the same label as the prediction on x and then set their label equal to the correct one for x . We take a maximum of 50 fine-tuning steps and take the top-50 proponents or opponents to build B . The main metric we compute is the success rate , i.e. the ratio of mis-predictions successfully corrected within the limit of 50 steps. Additionally, we report prediction retention [DCAT21] on a fixed held-out set of 50 test examples, i.e. the ratio of examples predictions which have not changed after a correction - ideally, a correction does not cause too many changes in model predictions otherwise. We experiment with different values of ε , starting from no up-sampling and gradually increasing it to when influential examples account for slightly more than half of the batch.

From Figure 3 we see that Proponents-correction and Opponents-tuning strongly outperform the baseline in binary classification (SST2). For multi-class classification (CIFAR10) Opponents-tuning is not effective, as we verified that only 54% of the retrieved opponents have the desired label; Proponents-correction still outperforms the baseline increasing on average the success rate by 2% and reducing the number of steps to take by 6%. We conjecture that for ResNet the improvement over the baseline is less than for BERT because of the worse predictive power of IF (Figure 2 (b)). In Appendix H we include additional plots showing the number of steps to correct mis-predictions as a function of the up-sampling parameter ε .

The primary goal of the experiments in this section is to verify that tuning on influential examples results in a correction more reliably and faster than on other classes of examples. Since the SST2 and CIFAR10 datasets are largely clean, for the proponents method, the training example label is flipped to an incorrect one. In Appendix I we give examples from a noisy text classification dataset illustrating the scenario that the correction-with-proponents method is supposed to address.

7 Limitations

We derive the perturbed training trajectory (Theorem 2) and the divergence of trajectories (Theorem 3) for stochastic gradient descent and, while we sketch in the Appendix the modifications needed when using other optimizers, we do not pursue this topic in detail. In Section 6 we measure prediction retention after correcting mis-predictions. While this metric is intuitive and has been used previously (e.g., [DCAT21]), we do not distinguish between semantically similar and unrelated examples and thus do not check the consistency and generalization properties of the update [MBAB22], leaving a thorough study of the correction-retention tradeoff to future work.

8 Conclusions

IF have been regarded as a tool that promises to trace model behavior to the training data. Unfortunately, recent studies have found no empirical support for such a claim as IF fail to predict the LSOR effect. This finding gives rise to the question of what IF methods really predict and whether they could be useful for model debugging. In this work we clarified which questions IF can be expected to answer. We first identified problematic assumptions made by IF methods - a priori any of these assumptions could be a reason for the observed empirical failure of IF. Thus, for each assumption we studied if it is indeed problematic. While most have turned out to be addressable in one way or another, we demonstrated that the one about parameter divergence puts a severe limitation on IF. With a deeper analysis of this assumption, we revised what can be theoretically expected from IF: IF methods are time-bound, that is, they can at most predict what happens when fine-tuning on a perturbed loss for a limited amount of time. With that, a practical usage of IF for model debugging is still possible - we proposed and empirically validated a theoretically-grounded procedure to apply IF to correct model mis-predictions.

References

[BNL + 22] Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger Grosse. If influence functions are the answer, then what is the question?, 2022.

[BPF21] Samyadeep Basu, Phil Pope, and Soheil Feizi. Influence functions in deep learning are fragile. In International Conference on Learning Representations , 2021.

[CGFT19] Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input similarity from the neural network perspective. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

[CS82] R. Cook and Weisberg S. Residuals and influence in regression . Chapman and Hall, New York, 1982.

[DCAT21] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 6491-6506, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

[DPG + 14] Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014.

[FLP + 23] Jillian Fisher, Lang Liu, Krishna Pillutla, Yejin Choi, and Zaid Harchaoui. Influence diagnostics under self-concordance. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 10028-10076. PMLR, 2023.

[GKX19] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In ICML , 2019.

[GRH + 21] Han Guo, Nazneen Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. FastIF: Scalable influence functions for efficient model interpretation and debugging. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 10333-10350, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

[Gro19] T. H. Gronwall. Note on the derivatives with respect to a parameter of the solutions of a system of differential equations. Annals of Mathematics , 20(4):292-296, 1919.

[GWP + 23] Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. Simfluence: Modeling the influence of individual training examples by simulating training runs, 2023.

[HYHI20] Kazuaki Hanawa, Sho Yokoi, Satoshi Hara, and Kentaro Inui. Evaluation of similaritybased explanations. In ICLR-21 , 2020.

[KATL19] Pang Wei Koh, Kai-Siang Ang, Hubert H. K. Teo, and Percy Liang. On the accuracy of influence functions for measuring group effects. In Proceedings of the 33rd International Conference on Neural Information Processing Systems , Red Hook, NY, USA, 2019. Curran Associates Inc.

[KL17] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1885-1894. PMLR, 06-11 Aug 2017.

[KS21] Karthikeyan K and Anders Søgaard. Revisiting methods for finding influential examples, 2021.

[MBAB22] Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

[PLKS20] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

[SZTS22] Andrea Schioppa, Polina Zablotskaia, David Vilar Torres, and Artem Sokolov. Scaling up influence functions. In AAAI-22 , 2022.

[WTD17] Ellery Wulczyn, Nithum Thain, and Lucas Dixon. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web ,WWW '17, pages 1391-1399, Republic and Canton of Geneva, CHE, 2017. International World Wide Web Conferences Steering Committee.

A Clarifications regarding the operator ∇

We spell out in further detail the way we employ the ∇ operator. For example, we say that ∇ ε | 0 θ ε is the Q × N -dimensional Jacobian at ε = 0 : concretely, ∇ ε | ε 0 f means computing the Jacobian of f wrt. ε and evaluating it at the point ε 0 ; this Jacobian is a matrix whose ( i, j ) -entry is given by:

This is easily extended to higher order derivatives denote by ∇ k ; specifically we use ∇ 2 ( ε,θ ) | (0 ,θ 0 ) L to denote the matrix whose i, j -entry is given by:

as the matrix H -1 θ 0 acts on the space of the parameters θ , the notation H -1 θ 0 ∇ 2 ( ε,θ ) | (0 ,θ 0 ) L will denote a contraction on j :

B Proof of Theorem 1

For convenience, we first recall the statement of Theorem 1.

Theorem. Assume that ∇L is C k ( k ≥ 1 ) and let θ 0 ∈ R N ; assume that the Hessian H θ 0 = ∇ 2 θ | θ 0 L is non-singular; then there exist neighborhoods U of θ 0 and V of 0 ∈ R Q , and a C k -function Θ : V → U such that Θ(0) = θ 0 and Θ( ε ) ∈ U is the unique solution in U of the equation

Moreover, the gradient of Θ at the origin is given by:

Proof. Equation (12) gives us N constraints on the N + Q variables ( θ, ε ) and we want to solve them for the first N variables θ ; to obtain the function Θ we invoke the Implicit Function Theorem, using that the Jacobian wrt. the variables we want to solve for is H θ 0 and is therefore non-singular. Finally (13) is obtained by taking the gradient of (12) wrt. ε and setting ε = 0 :

C Proof of Corollary 1

For convenience, we first recall the statement of Corollary 1.

Corollary. Under the assumptions of Theorem 1:

If θ 0 is a stationary point of L , then each Θ( ε ) is a stationary point of the loss θ ↦→L ( θ, ε ) .

If θ 0 is a local (strict) minimum of L , for ε sufficiently small, Θ( ε ) is a local (strict) minimum of the loss θ ↦→L ( θ, ε ) .

Let Q = 1 (hence epsilon is a scalar) with L ( θ, ε ) = L ( θ ) + εl x ( θ ) , where l x is the loss corresponding to a specific training point x . We then obtain the classical result [CS82] about the influence of down/up-sampling x on the training parameters:

Proof. If θ 0 is a stationary point of L , then ∇ θ L ( θ 0 , 0) = 0 in (3). This implies that ∇ θ L (Θ( ε ) , ε ) = 0 in (3), which is exactly the statement that Θ( ε ) is a stationary point of L . If θ 0 is a local (strict) minimum of L , it is not just a stationary point, but the Hessian at θ 0 is also positive definite. By continuity, for sufficiently small ε , also the Hessian ∇ 2 θ | Θ( ε ) L will be positive definite so that Θ( ε ) will be a local (strict) minimum. Finally, (14) follows from applying (3) to the variation L ( θ, ε ) = L ( θ ) + εl x ( θ ) .

D Proof of Theorem 2

For convenience, we first recall the statement of Theorem 2.

Theorem. Assume that the model is trained for T time-steps with stochastic gradient descent. Denoting by H t the Hessian ∇ 2 θ | θ 0 ,t L t , then the final parameters θ ε,T satisfy:

Proof. The parameters θ ε,t obey a recurrence relation:

then (15) follows immediately by applying ∇ ε | 0 , i.e. computing the Jacobian wrt. ε at the origin.

E Proof of Theorem 3

The first step in the proof of Theorem 3 is a discrete version of Gronwall's Lemma [Gro19].

Lemma 1. Let { u t } , { α t } and { β t } be sequences such that β t ≥ 0 and

Note that we assume that (18) holds for t ≥ 1 and we consider u 0 an initial condition. Then:

If { α t } is non-decreasing in t we then get:

Moreover, defining u t by setting (18) to be an equality, shows that (19) is sharp.

Proof. We first observe that if we set v 0 = u 0 and build v t by declaring (18) to be an equality, then v t ≥ u t for any t . This is true by induction as:

We thus focus on bounding v t +1 ; we note that

which can be solved to give

thus

then (19) follows by induction and letting t +1 = T . Finally, if { α t } is non-decreasing in t we can simply replace α s with α T obtaining (20). The sharpness follows by considering the sequence { v t } we defined in the proof.

For convenience, we first recall the statement of Theorem 3.

Theorem. In the setting of Theorem 2 assume that, for t ≤ T , θ ε,t lies in a bounded region R such that

and that for θ ∈ R each loss L t ( θ, ε ) and its gradient wrt. θ are A -Lipschitz in θ . Then

Proof. The idea is to apply Lemma 1 as we would in the case of a standard ODE. Let us compare the evolution of θ ε,t and θ 0 ,t :

which leads to

we now just need to rephrase this inequality in terms of Lemma 1:

we note that u 0 = 0 as both dynamics start at θ init and that α t is non-decreasing in t . We then have that

which is (8).

(a)

(b)

ne conf.interval

interval

0.12

0.5

Hessian

Optimizer

Tracln

0.10

0.4

conf.

0.08

h

0.3

h

0.06

0.2

8

0.04

0

0

fror

E

0.1

fror

0.02

t

0.0

0

200

400

600

800

1000

s

0.00

0

200

400

600

800

1000

Step

Step

E.1 Sketching modifications needed in the case of optimizers

The proofs of Theorems 2 and 3 were given for SGD. The argument in the case of using an optimizer would be more involved. Here we sketch the modifications needed when dealing with optimizers. An optimizer is characterized by an optimizer state , σ ε,t , which will also evolve in time. While for SGD we just considered the update rule for θ ε,t , in the case of optimizers one needs to study a joint system of update rules for the parameters and the optimizer state:

In the case of Theorem 2 one would then apply ∇ ε | 0 to the joint system to obtain the update rule. For Theorem 3 one should add additional continuity in ε and Lipschitz conditions on F and G ; one should then check that these are indeed satisfied for common optimizers like Adam or Adafactor.

F Why do we use the Conjugate Residual method?

Regarding HIF, note that usually the Conjugate Gradient method is used [KL17] for computing inverse Hessian vector products. However, in our experiments the Conjugate Gradient method always yielded a time-series of Pearson Correlations R ( t ) oscillating around 0 , thus without any predictive power . The reason is that this method assumes the Hessian to be positive-definite, which is not the case for most Neural Networks. A simple fix to the problem is to use the Conjugate Residual method which does not require the Hessian to be positive definite. We recommend to use the Conjugate Residual when applying HIF in Deep Learning; this might look like a minor technical point, but it can avoid reporting that HIF has no predictive power, when instead the issue lies in the numerical method used to compute inverse Hessian vector products .

For more discussion on the Conjugate Residual method see its Wikipedia article.

G Further empirical results on Fading of Influence scores

G.1 Fading of Influence for BERT

In the setting of Figure 2(a) we plot the distance from the confidence interval of the Pearson's r to 0 (Figure 4); once the distance becomes 0 , the predictive power of influence scores has become null.

In Figure 5 we zoom in Figure 2 (a: BERT). The fading phenomenon was non affected by checkpoint selection: in Figure 6 we consider a later checkpoint and we see the same qualitative behavior. Moreover, in Figure 6 we also consider TracIn with 3 checkpoints selected using the advice in [PLKS20] we do not see improvements and the peak is even slightly lower than for TracIn using one checkpoint. Thus, in our further experiments we have used TracIn with a single checkpoint.

In Figure 7 we zoomed in Figure 2 (b: ResNet). In this case the predictive power was never particularly high, e.g. for TracIn it quickly peaked at 0 . 3 but it takes more steps than in the case of BERT to reach 0 . We conjecture that this is in part due to the slower divergence of parameters in

Hessian

0.8

Tracln

0.6

Optimizer

0.4

R(t)

0.2

0.0

-0.2

-0.4

-0.6

0

20

40

60

80

100

Step

Hessian

1.00

Tracln

0.75

Optimizer

0.50

R(t)

0.25

0.00

-0.25

-0.50

-0.75

0

200

400

600

800

1000

Step

Hessian

0.3

Tracln

0.2

R(t)

0.1

0.0

-0.1

0

200

400

600

800

1000

Step

FadingofInfluenceovertime

Hessian

and Lossshifts

1.00

Gradient

0.75

Correlation of Influencea

0.50

0.25

0.00

-0.25

-0.50

0

50

100

150

200

250

300

350

400

Step

(a)

(b)

0.8

Hessian

Hessian

Tracln

interval

0.5

Tracln

0.6

Optimizer

Optimizer

0.4

R

+

0.4

0.2

0.3

0.2

0.0

口

0.1

-0.2

0.0

0

250

500

750

1000

1250

1500

1750

2000

0

250

500

750

1000

1250

1500

1750

2000

Step

Step

(a)

(b)

Hessian

Hessian

0.8

0.8

Tracln

Tracln

0.6

0.6

R

0.4

0.4

0.2

0.2

0.0

0.0

-0.2

-0.2

0

500

1000

1500

2000

2500

3000

3500

4000

0

50

100

150

200

250

300

350

400

Step

Step

ResNet (compare the x-axis of (a) and (b) in Figure 1) and in part due to the use of SGD. In particular, we verified that a slower fading also takes place when fine-tuning BERT with SGD (Figure 8).

G.2 Fading of Influence for T5

We consider another NLP pre- trained model fine-tuned on SST2, T5. In Figure 9 we illustrate the same effect of fading of influence for the T5 model.

G.3 Fading of Influence for the ViT

In the case of ResNet, in our fading of influence experiments (Fig. 2 (b)) the peak is particularly low. On the same task, we trained a ViT with SGD and observe that the peak is higher, similarly to what happens with BERT, see Figure 10. We conjecture that this difference is due to the different model architecture. Note that in both cases, we observe the fading of influence as predicted by the theory.

H How many steps are needed to correct mis-predictions?

In Figure 11 we plot the average and median number of steps to correct a mis-prediction as a function of ε : for any fixed ε , the more effective a correction method is, the fewer steps are needed to correct the prediction. For BERT we see that both Proponent-correction and Opponent-tuning result in a large decrease in the number of steps to take, compared with the Baseline. For ResNet, we do not plot Opponent-tuning as it performed poorly on success-rate. For ResNet the gains of Proponent-correction are smaller but consistent as ε varies.

(a)

(b)

50

Prop.mean step

50

Prop.mean step

Prop.median step

Prop.median step

40

Opp.mean step

Baseline mean step

Opp.median step

40

Baseline mean step

Baseline mean step

30

Baseline median step

30

20

20

10

10

0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

0.0

0.2

0.4

0.6

0.8

1.0

epsilon

epsilon

I Selecting mislabeled proponents for correction

For the experiments in Section 6 we use comparatively clean datasets, so the proponents we retrieve and correct are not mislabeled. In this section we use the notoriously noisy Wikipedia Toxicity Subtypes dataset [WTD17] where for many misclassified test examples we can find mislabeled proponents used during training. Table 1 presents false positives and negatives (that is, test examples incorrectly predicted to be toxic, resp. non-toxic) with a high-scoring proponent which we consider mislabeled. It is worth noting that some examples lack context necessary to confidently verify whether they are toxic or not. In total, we collected 14 such pairs, seven for mispredictions of each kind. 3

Table 2 presents the mean and the median number of steps needed to be taken on the corrected proponents to achieve a change in prediction, as well as retention once there is a change. As a point of comparison, we permute the proponents withing the false positive and false negative sets: this is a more challenging baseline than selecting examples for correction randomly. Note that the influence scores for the presented examples range from -0 . 90 to 0 . 92 , so some of the permuted proponents still have some quite influence on the test examples they get assigned to. Still, the results confirm that correcting mislabeled top-scoring proponents is the fastest way of achieving a change in model predictions, with as few as three steps required on average and with the smallest damage to predictions on other examples (measured with retention).

J Hyper-parameters

J.1 Training

ResNet was trained on a single V100 with the hyper-parameters in Table 3; training data was augmented using torchvision using the transformations transforms.RandomCrop(32, padding=4) , and transforms.RandomHorizontalFlip() .

The ViT was trained on a single V100 with the hyper-parameters in Table 6; we used the same data augmentation as in the case of ResNet.

BERT was trained on 8 TPUv3 cores using the hyper-parameters in Table 4. The best checkpoint was selected on validation-set accuracy evaluated every 500 steps; it corresponded to 6000 steps of training.

T5 was trained on a GPU V100 using the hyper-parameters in Table 5. We selected a checkpoint based on the validation set accuracy at epoch 4, where the accuracy was 92 . 9% . Note that T5 is a generative model, so at prediction time the sequence to classify is encoded with the encoder and then the decoder is used to generate a sequence that represents the desired class label. When computing influence functions, one uses the cross-entropy loss for sequences.

J.2 Correction Experiments

For the correction experiments we used ABIF [SZTS22] because of their computational efficiency as IF scores need to be computed against the whole training set; we used 32 projectors obtained with 64 Arnoldi iterations. The Arnoldi iteration can take up to 2 hours; scoring the data takes a few minutes. We used a learning rate of 10 -3 and for BERT we loaded the state of the Adam optimizer from the selected checkpoint.

For BERT we selected the checkpoint at 6000 steps which was the best on validation performance. For ResNet we selected the checkpoint after 10 epochs at which about 150 test points are incorrectly classified.

K Illustrating TracIn's modeling error

In Theorem 2 we have derived the correct formula for the Jacobian ∇ ε | 0 θ ε,T and we expect TracIn (Eq. 2) to introduce a modeling error in calculating ∇ ε | 0 θ ε,T . We can compute the difference between the ground truth θ ε,T and the first order expansion θ 0 ,T + ε T ∇ ε | 0 θ ε,T when the Jacobian is computed with TracIn (Eq. 2) or the corrected formula (Eq. 6). We can then compute the relative excess error introduced by TracIn for different values of ε . In Figure 12 we observe that except for the extremely small perturbation ε = 10 -4 , TracIn does introduce an additional relative error in modeling the change of parameters when retraining BERT with SGD. The setup of this experiment is the same as those of our parameter divergence experiments.

(%)

17.5

ε=1e-03

Tracln

15.0

ε=1e-02

ε=le-01

12.5

6

10.0

rror

7.5

e

5.0

n

t

2.5

d

0.0

0

50

100

150

200

250

300

Step

L Broader Impact Statement

We believe that our work can benefit model developers who want to debug and correct mis-predictions made by models. Our suggested error-correction procedure is much less compute intensive than previous ones using IF as it does not require model retraining. However, this debugging procedure could introduce new errors in the systems, for example if on a specific problem IF are not effective at predicting loss-shifts or if examples retrieved by IF lead to over-fitting against spurious artifacts present in such examples. Since this could have a negative impact on the users of such systems, mitigation strategies should be put in place regarding the trade-offs between the success rate of fixing specific mis-predictions and the overall retention of the system performance.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper reexamines the assumptions used in the deduction of influence function (IF) methods in order to explain the failure of IF in predicting leave-some-out-retrain performance. The authors find that all five assumptions used in the previous deduction will be violated to different degrees in practice and propose a combination of HIF and Arnoldi-based methods to overcome these violations. They find that four of the assumptions can actually be bypassed or fixed, however, the remaining challenge named *parameter divergence* seems inherent. They show that the predictive ability of IF will gradually fade over training steps due to this phenomenon and show that accounting for this effect, it is better to interpret IFs as proxies for the effect of a few fine-tuning steps.\n\nSTRENGTHS:\n* **Originality.** This paper shows an in-depth analysis of the assumptions on which IFs are based. Their analysis combines theoretical thinking well with experimental observations.\n* **Clarity.** The results are presented in a clean way and are a pleasure to read.\n* **Significance.** Although the reviewer is not an expert in the field of influence functions, the question this paper targets seems important and the paper sheds light on the real obstacles to solving this problem, as well as proposes possible ways to solve it.\n\nWEAKNESSES:\nThe reviewer thinks the paper may improve in the following aspects.\n\n* The authors argue that (1) using Arnoldi-based methods helps boost the accuracy of HIF, and (2) accounting for training trajectories can improve the estimation for IFs. The theoretical deduction of these propositions is very reasonable but it would be better to include ablation experiments or cite relevant literature to showcase the phenomenons.\n\n* The authors observe that IF methods perform poorly on Resnet but do not have any explanation for this phenomenon. It would be beneficial if the authors can investigate the reason behind the difference of IF methods performance on NLP and CV tasks.\n\nQUESTIONS:\nThe reviewer is interested in the following questions.\n * Why does the correlation between IFs and performances seem to increase before decreasing? \n * How would the authors suggest modifying the current IF estimation methods based on their observations?\n * What is the author's explanation of the rapidly increasing phase 1 in figure 1?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and the reviewer has not noticed any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInfluence functions allow one to measure the effect of up-weighting a training sample on the test loss (or functions of the model parameters) of a test example. This paper studies several of the assumptions needed for hessian-based influence functions to produce valid leave-out-one estimates of the effect on a test sample's loss. This assumptions include: convexity, stability of the hessian, and additivity of training trajectory. The paper shows that some of these assumptions are not as problematic, in practice, as previously assumed. They show that several of these assumptions can indeed be satisfied with changes to the formulation; for example, if the hessian becomes degenerate, one can approximate the hessian via arnoldi iteration. It turns out that the most problematic assumption results from a divergence between the initial set of parameters, and another obtained by re-training or even fine-tuning for longer time steps. The paper then bounds this parameter divergence using a discrete version of gronwall's lemma, and gives a set of takeaways for how to evaluate influence functions: the takeaway from this work is that influence scores only predict effect of a training sample over a small number of training steps. In the final portion of the paper, they evaluate how to use influence functions to correct errors.\n\nSTRENGTHS:\nOverall, I really enjoyed reading this paper, and found the breakdown of the assumptions really comprehensive.\n\n**Quality/Clarity**\\\nI found the paper to be of very high quality. Each assumption is stated clearly and cleanly discussed. For example, I found the resolution of assumption #1 to be interesting. Here the authors show that for hessian-based influence functions one does not need to assume strict convexity but we can relax it to assume that the final gradient steps do not change the hessian by too much. While this might seem like a simple change, it helps provide clarity to this literature. The bigger takeway from this work is that influence scores only predict effect of a training sample over a small number of training steps. This finding is important since it requires rethinking how these approaches are currently used. Overall, this paper sets out key assumptions of influence functions, discusses how to remedy violations to these assumptions, and presents experiments to corroborate the theory presented.\n\n**Significance**\\\nOne important takeaway is that, for deep models, it now makes sense that influence scores obtained via retraining have poor correlation with leave-one-out retraining scores. Secondly, another important takeaway is that one can correct errors simply be reweighting or fine-tuning on opponents or proponents. The findings from the paper should be important for practitioners and others using influence functions as part of their debugging toolbox.\n\n**Originality**\\\nThis paper mostly considers a question that was also studied by Bae et. al. (Neurips 2022), but does so in a different way and using mostly new techniques. I found the use of the discrete version of gronwall's lemma to be quite nice. I am not sure if this is new, but taken together, this work advances the understanding of influence functions quite substantially.\n\nWEAKNESSES:\nThe main weakness I have with this work are minor and listed below: \n\n**Relation with Bae et. al. from Neurips 2022**: The main related work to this one is this paper. However, this work does not do enough to sufficiently contrast their analyses with theirs. I am mostly familiar with this previous work, and understand that they address exactly the same question as well, but arrive at a different conclusion. Their finding is that hessian-based influence functions approximate what they term the \"proximal bregman response function\", and that the ability of this quantity to match a sample loo estimate is mostly affected by the linearization error (i.e., due to the taylor expansion), and 2) hessian approximation. This work seems to not agree with theirs, i.e., you suggest that hessian approximation be handled with arnoldi, and that the key discrepancy comes from parameter divergence. It would be great if the authors could spend more time in the draft to contrast this work against that one. I think this is an important issue to address in the paper.\n\nQUESTIONS:\nSome clarification questions: \n\n- For error correction, if I have k examples that I want to correct, then am I correcting each example independently or the taking gradient steps on all k examples at the same time. If it is independent, then isn't this too onerous? \n- One use of these influence scores is mislabelled training samples, how does assumption 5 affect that? When in training should the self-influence score be a useful metric for mislabeled label correction?\n- This is minor question, going from $\\theta_\\epsilon \\approx \\theta_0 + \\epsilon^\\top\\nabla \\theta_\\epsilon$ to equation 1, where did the second $\\nabla_\\theta$ come from? Is this the chain rule? I am familiar with other ways to derive influence functions.\n\nLIMITATIONS:\nThe authors discuss limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the limitations of Influence Functions (IF) based on assumptions about convexity, numerical stability, training trajectory and parameter divergence. The authors discuss each of those assumptions in detail and propose ideas to address the shortcoming of those limitations. The authors describe solutions for 4 assumptions made in the paper and discuss the limitations of the fifth assumption, parameter divergence, which is harder to address.\nApart from that the authors also illustrate the theory of parameter divergence on the example of NLP and vision classifiers. They confirm theoretical findings of fading influence over time when retraining the models on vision and test classifiers.\nThe authors also show how to perform error correction based on proponent-correction and opponent-tuning using a few fine tuning steps. They show that their approach outperforms baseline re-training procedures.\n\nSTRENGTHS:\n+ The paper is well written, especially the introduction, problem statement and the assumptions are well framed.\n+ The related work is well cited and based on the studies in the recent papers, this paper proposes to address the limitations highlighted in the literature such as the LSOR problem for influential examples.\n+ The paper studies 5 important assumptions made by IF and proposes solutions that help to alleviate the limitations posed by those assumptions. It also discusses and proposes a solution for the temporal dependency which is not accounted for in the original TracIn paper.\n\nWEAKNESSES:\n+ The main contributions of the paper seem to be around studying 5 different assumptions and proposing solutions for them. The main contribution is for assumptions 3 - 5 where the authors show that IF can predict \\theta_{\\epsilion, t} only for a limited period of time and propose incorporating temporal term into the tracin's formulation. These are important findings, however, it is not very clear whether it is a large enough contribution and whether it is completely novel since similar limitations of Tracin were also discussed in the Kelvin Guu, et.al.\n+ It looks like all the proofs corresponding to theorems mentioned in the main paper are in the appendix. Since it is not required for reviewers to read appendices it might be good to consider moving some of the important proofs or parts of them into the main paper.\n+ Section 4.2 might be difficult to follow for someone if they do not already know about Arnoldi. There is a brief description that Arnolodi approximates P1 using subspaces spanned by eigenvectors corresponding to top-k eigenvalues but I think that overall intuition of Arnoldi is not clear if someone is not familiar with that work.\n+ Overall I think that Section 4.2 might be a bit difficult to follow (e.g. how is Theorem 1 applied to restricted variation).\n+ It feels that it is somewhat difficult to follow how we arrive from Eq 7 to Eq 8. Perhaps adding more explanation in section 4.4 would be good.\n\n**Minor comments**\n\n+ The definition of C^k function on line 179 is a bit unclear. C gets used again in line 234 (C\\epsilion) which makes it a bit confusing in terms of the readability of the paper.\n+ Figure 3: X-axis is not annotated\n\nQUESTIONS:\n+ Section 6: Proponents-correction: when the set B is identified and relabeled, is it done for truly mislabeled examples ? How were those mislabeled examples sourced ?  For experiments with SST and ResNet how were set Bs curated ?\n+ Section 6: Which formulation of IF (TracIn or Influence Functions) was used in the experiments ? \n+ Section 5.2: Is R(t) pearson correlation between losses before and after re-training averaged for the same batches? Does the `Step` indicate different checkpoint on Figure 2 ?\n\nLIMITATIONS:\nThe authors discuss several limitations of their work and propose ideas to address those limitations in the future.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper verifies assumptions of Influence Functions that show discrepancies in theory and empirical results, convexity, numeric stability, training trajectory, and parameter divergence. Although the other assumptions are addressable, parameter divergence is not. Hence, the paper proposed a solution to it and also showed empirical results.

STRENGTHS:
-	Based on their observation of the issue, they proposed a simple solution to it.
-	To the reviewer, this paper reads as an instructional paper, which can contribute to the community.
-	It verified the applicability of the IF assumption one by one on convexity, numeric stability, training trajectory, and parameter divergence.

WEAKNESSES:
-	Aside from its technical contribution, there’s room where its writing can be improved. In particular, the introduction can be better structured, and there are multiple repetitive places that can be concise. 
-	In Fig 2 (1) BERT case, although the authors claimed that the predictive power degrades monotonically over time, the “fading” effect is actually not clear. There’s only one time of peak at an early step, and there's no decreasing trend in the later steps. There is no fading effect either in the Appendix (Fig. 5.)

QUESTIONS:
Do the authors have other models to show the fading effect? As BERT’s case is unclear, only one empirical case of ResNet shows it.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-21