SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Andrea Schioppa $^{1}$  Katja Filippova $^{1}$  Ivan Titov $^{2,3}$  Polina Zablotskaia $^{1}$
<sup>1</sup>Google DeepMind <sup>2</sup>University of Edinburgh <sup>3</sup>University of Amsterdam {arischioppa, katjaf, polinaz}@google.com, ititov@inf.ed.ac.uk
Influence functions (IF) have been seen as a technique for explaining model predictions through the lens of the training data. Their utility is assumed to be in identifying training examples "responsible" for a prediction so that, for example, correcting a prediction is possible by intervening on those examples (removing or editing them) and retraining the model. However, recent empirical studies have shown that the existing methods of estimating IF predict the leave-one-out-and-retrain effect poorly. In order to understand the mismatch between the theoretical promise and the practical results, we analyse five assumptions made by IF methods which are problematic for modern-scale deep neural networks and which concern convexity, numeric stability, training trajectory and parameter divergence. This allows us to clarify what can be expected theoretically from IF. We show that while most assumptions can be addressed successfully, the parameter divergence poses a clear limitation on the predictive power of IF: influence fades over training time even with deterministic training. We illustrate this theoretical result with BERT and ResNet models. Another conclusion from the theoretical analysis is that IF are still useful for model debugging and correcting even though some of the assumptions made in prior work do not hold: using natural language processing and computer vision tasks, we verify that mis-predictions can be successfully corrected by taking only a few fine-tuning steps on influential examples.
Influence Functions (IF) [CS82] have been regarded as a tool that can trace model behavior on any example to the training examples [KL17, PLKS20,  $\mathrm{GRH^{+}21}$ ]. Their theoretical justification lies in the ability to predict loss changes on a specific test point when training on a perturbed loss obtained by removing or down-sampling a given training point. In the case of an undesired model behavior on a test-point, the influential training examples for that test point have been assumed to be the ones "responsible" for the prediction so that intervening on those - e.g., by removing them and then retraining the model, or by taking additional fine-tuning steps  $[\mathrm{GRH}^{+}21]$  - would result in a change in the loss or prediction. It has been confirmed that for linear models it is indeed the case [KATL19].
Recently, in their extensive experiments, [BPF21] and [KS21] could not find empirical support for the claim that IF approximate the Leave-Some-Out Retraining (LSOR) effect on the loss in deep neural networks. In particular, they show that the correlation between the ranking of training examples produced by LSOR and the IF-based ranking is low and considerably affected by choice of hyperparameters. How can this discrepancy be explained? And, given that the theoretical justification for IF lacks empirical support, does it mean that IF should be abandoned as an explainability and debugging tool altogether?
In this work we clarify what question Influence Functions (IF) actually answer. We first identify assumptions which are either implicit or not investigated in prior work: these concern convexity,
numeric stability, training trajectory, and the parameter divergence when retraining on a new loss. In principle, any of these assumptions might be problematic and be the reason why the original theoretical justification for IF is not supported empirically. However, we show how to address most of them successfully so that they cannot be the reason for the aforementioned discrepancy. Unfortunately, we also show that the parameter divergence is indeed problematic and requires to revise both theoretical and practical expectations about IF. This analysis paves the way to clarifying the question that IF can answer. As a first step in this direction, we need to distinguish two approaches to computing influence. The Hessian-based Influence Functions (HIF) [CS82, KL17] have a rigorous theoretical justification in statistics, but rely on strict convexity assumptions, which are not met in Deep Learning and which previous analyses of HIF in the Deep Learning literature still rely on [KL17,  $\mathrm{BNL}^{+}22$ ]. Our first contribution concerns HIF and is twofold: we prove (Theorem 1) that this unsatisfied convexity assumption is not as problematic for HIF as one may think-given a stationary point for the original loss, there is a nearby stationary point for the perturbed loss which can be approximated by using HIF. However, we point out a more serious problem with HIF: there is no guarantee that by retraining on the perturbed loss one would get to that stationary point. This observation provides an additional support to the second popular approach to IF, TracIn [PLKS20], which explicitly models the training dynamics and which additionally does not require to compute the expensive inverse Hessian vector product as it only uses gradient information. Despite TracIn being grounded in the training dynamics, we unveil a hidden additive modeling assumption in [PLKS20] that prevents it from correctly modeling the (re)-training dynamics. Our second contribution is thus to provide a theoretical analysis (Theorem 2) of how training trajectories change when perturbing the loss. This is a key result as it suggests that two training trajectories differing by a small loss perturbation could diverge over time to the point of violating a first-order expansion assumption that IF make, thus making IF's predictions unreliable. Our further theoretical and empirical investigation confirms this conjecture. Therefore, what IF can do is to predict parameter changes when fine-tuning for a limited number of steps on the perturbed loss. This requires to adjust how IF are evaluated (Section 5) and applied (Section 6). In order to confirm the conjecture and re-adjust expectations regarding IF, we need a deeper analysis of the trajectory divergence.
Our third contribution is thus to validate the conjecture on trajectory divergence and its consequences for IF. We first prove (Theorem 3), using a discrete version of Gronwall's Lemma [Gro19], an upper bound on the parameter divergence when (re)-training on a perturbed loss. We then empirically verify that the bound is sharp in Section 5 and that IF can approximate parameter changes along the perturbed trajectory only for a limited amount of time. We then empirically verify that this leads to a fading (of accuracy) of IF predictions over time. Therefore, we theoretically demonstrate and empirically verify that IF can in general answer only what happens when fine-tuning on a perturbed loss for a limited amount of time, instead of a general retraining setting.
Therefore, our new theory indicates that IF have been used incorrectly as the emphasis has been on their LSOR potential. On the positive side, it suggests an alternative way of using IF that indeed yields substantial empirical improvement. To demonstrate that, as our final contribution, in Section 6 we propose and verify a very simple method for correcting mis-predictions by taking only a few gradient steps on influential examples. Our proposal is related to model editing [DCAT21, inter alia] in that the latter also aims at changing model predictions. However, there the model is given and the modifications are done on its parameters whereas IF aim at understanding how specific training examples are responsible for the current model behavior and editing predictions through the data. We leave for future work building connections between model editing and IF.
For the case of Hessian-based influence functions, the recent work of  $\left[\mathrm{BNL}^{+}22\right]$  has proposed to resolve the discrepancy between the theory and the empirical evidence by evaluating them using the proximal Bregman response function. However,  $\left[\mathrm{BNL}^{+}22\right]$  relies crucially on the convexity assumption: while under this assumption there is agreement between our work and their findings (e.g. our parameter divergence drives the linearization error  $\left[\mathrm{BNL}^{+}22, \text{Sec. 4.4}\right]$ ), our theory applies also the case where the Hessian is not positive definite. Moreover, our work covers also the scope of Gradient-based influence functions including TracIn [PLKS20].
To incorporate training dynamics, TracIn exploits multiple checkpoints which introduces a substantial overhead, hence only the latest checkpoint is often used in practice. Other gradient-based methods have been proposed in [CGFT19, HYHI20], but they lack a justification from the training dynamics perspective.
2 Definition of Influence Functions and Notation
The different methods proposed to define Influence Functions share a common goal: forecasting the change in the prediction on a test example when up (or down-) weighting a training example. This is achieved by tracing the effect that re-weighting a training example has on the model parameters.
Removing or adding a training point can be modeled by a perturbation of the loss function; such a perturbation can be made smooth by modeling the weighting of a training point by a continuous parameter  $\varepsilon$ . More generally, let  $L(\theta)$  denote the loss function where  $\theta \in \mathbf{R}^N$  are the model parameters. We model loss perturbations by introducing a variation of the loss, which is a smooth function  $\mathcal{L}(\theta, \varepsilon)$  depending on an additional vector parameter  $\varepsilon \in \mathbf{R}^Q$  and coinciding with the original loss for  $\varepsilon = 0$ . For example if  $l_x$  denotes the loss on a given training point  $x$ , we set  $\mathcal{L}(\theta, \varepsilon) = L(\theta) + \varepsilon l_x$ . While the scalar case  $Q = 1$  is the one commonly considered, e.g. [KL17, PLKS20], we introduce the vector one  $Q > 1$  which arises naturally when considering the effect of re-weighting multiple points differently, e.g. when modifying the weights in a mixture of different data-sets.
The IF method of choice then predicts what would happen if training on  $\mathcal{L}(\theta, \varepsilon)$  instead of  $L(\theta)$ . The final parameters are modeled as a function  $\theta_{\varepsilon}$  of the perturbation parameter; assuming that such a function is well-defined and sufficiently smooth, one then makes a first order expansion  $\theta_{\varepsilon} \simeq \theta_0 + \varepsilon^T \nabla_{\varepsilon|0} \theta_{\varepsilon}$ , where  $\nabla_{\varepsilon|0} \theta_{\varepsilon}$  is the  $Q \times N$ -dimensional Jacobian at  $\varepsilon = 0$  (see also Section A).
Under this first order assumption it is then straightforward to measure the change in the loss  $l_{z}$  corresponding to a test point  $z$ :
$$
l _ {z} \left(\theta_ {\varepsilon}\right) - l _ {z} \left(\theta_ {0}\right) \simeq \varepsilon^ {T} \nabla_ {\varepsilon | 0} \theta_ {\varepsilon} \nabla_ {\theta | \theta_ {0}} l _ {z}. \tag {1}
$$
We emphasize that (1) is general to different IF methods, which differ in the specific derivation of  $\nabla_{\varepsilon |0}\theta_{\varepsilon}$ .
3 Problematic assumptions made by Influence Functions
3.1 Problematic Assumption #1: Convexity can be used to show that  $\theta_{\varepsilon}$  is a function of  $\varepsilon$
In order to show that for each value of  $\varepsilon$  there is a single value of  $\theta_{\varepsilon}$  (so that one can model the final parameters as a function of the perturbation parameter), HIF relies on strict convexity of  $L$ . While this assumption is realistic for the statistical models considered in [CS82], this is not the case for neural networks. Even when introducing a regularization term, the loss of a neural network is not even weakly convex, and optimization methods usually converge to saddle points  $\mathrm{DPG}^{+}14$ . To the best of our knowledge, previous analyses of HIF in the Machine Learning literature, e.g. [KL17, BPF21,  $\mathrm{BNL}^{+}22$ ], have relied on some form of strict convexity. In Section 4.1 we will revisit HIF and prove (Theorem 1), roughly speaking, that near a given stationary point  $\theta_0$  for the original loss  $L$ , there is a stationary point  $\theta_{\varepsilon}$  of the perturbed loss  $\mathcal{L}(\theta ,\varepsilon)$  which can be modeled as a function of  $\varepsilon$  and such that  $\nabla_{\varepsilon |0}\theta_{\varepsilon}$  is given by  $-H_{\theta_0}^{-1}\nabla_{(\varepsilon ,\theta)|(0,\theta_0)}^2\mathcal{L}$  as in [CS82] (see Section A regarding the usage of  $\nabla^2$ ).
3.2 Problematic Assumption #2: The model Hessian is not degenerate
As HIF requires to apply the inverse model Hessian to  $\nabla_{(\varepsilon, \theta) \mid (0, \theta_0)}^2 \mathcal{L}$ , one needs to ensure that inverting the Hessian is numerically stable. It has been empirically demonstrated [GKX19] that most eigenvalues of the Hessian tend to cluster near 0. Numerically, this results in a considerable source of errors and instabilities when estimating HIF; while regularization can alleviate this problem, it introduces a hyper-parameter in the definition of HIF; the minimal value of such a hyper-parameter ensuring numerical stability depends on the smallest negative eigenvalue of the Hessian. Unfortunately, in realistic settings, this can be larger in absolute value than reasonable values for the regularization parameter. For example in our ResNet experiments regularization is of the order  $10^{-4}$  while the smallest negative eigenvalue is  $\simeq -5$ . In Section 4.2 we discuss how the Arnoldi-based Influence Functions (abbr. ABIF) (which were introduced by [SZTS22] for computational efficiency) can be used to address such instability issues.
3.3 Problematic Assumption #3: Training trajectory can be ignored in Hessian-based Influence
Even if we solve the Problematic Assumption #1 for HIF, there is no guarantee that when actually re-training from scratch on  $\mathcal{L}(\theta ,\varepsilon)$  one would converge to the  $\theta_{\varepsilon}$  given by Theorem 1 because the training trajectory is disregarded in HIF. As optimization is performed via some form of stochastic gradient descent, [PLKS20] propose TracIn which averages gradient dot-products across checkpoints in order to take into account the path taken by the training process. Importantly, for a single checkpoint  $\theta_0$  TracIn estimates  $\nabla_{\varepsilon}\theta_{\varepsilon}$  as  $-\nabla_{(\varepsilon ,\theta)|(0,\theta_0)}^2\mathcal{L}$ , so the inverse Hessian vector product does not need to be computed. While it seems that TracIn takes into account the training trajectory, in the next Assumption we identify an issue with the way it models the training trajectory.
3.4 Problematic Assumption #4: The training trajectory can be modelled additively
The analysis of TracIn in [PLKS20] is based on a first-order expansion of the final change of the loss of a test point in terms of the gradient steps across the training trajectory. While this argument seems mathematically convincing, it overlooks that if one point is removed, or slightly up-sampled / down-sampled, the subsequent training trajectory is modified. Let  $\theta_{\varepsilon ,t}$  denote the value of the parameters after  $t$  steps when doing gradient descent on  $\mathcal{L}(\theta ,\varepsilon)$ . Denoting by  $T$  the end-time, TracIn derives
$$
\nabla_ {\varepsilon | 0} \theta_ {\varepsilon , T} = - \sum_ {t = 0} ^ {T - 1} \eta_ {t} \nabla_ {(\varepsilon , \theta) | (0, \theta_ {0, t})} ^ {2} \mathcal {L}, \tag {2}
$$
where  $\eta_{t}$  is the learning rate at time step  $t$ . Now, the right-hand side in formula (2) is purely additive in the time steps; and addition is commutative, so the order of the time steps does not matter. One way to see that this is problematic is by making  $\mathcal{L}$  time-dependent so that it differs from  $L$  only at a specific time step  $t$ . In this case  $\nabla_{(\varepsilon ,\theta)}^2\mathcal{L}$  would be non-zero only at  $t$  and formula (2) would consist of a single term. However, we would expect the perturbation at  $t$  to affect the following time steps, so we should have at least  $T - t - 1$  terms on the RHS for (2). In Section 4.3 we compute  $\nabla_{\varepsilon}\theta_{\varepsilon ,T}$  looking at the whole training trajectory and discover an additional first-order term that is missing from TracIn: this term models the dependency of a time step on the earlier ones. Concurrent work [GWP+23] also criticizes the additive assumption in TracIn on empirical grounds and proposes to build (re)-training simulators which are unfortunately computationally expensive as a new simulator must be fitted on the training set for each test point.
3.5 Problematic Assumption #5:  $\theta_{\varepsilon}$  can be expanded to first order in  $\varepsilon$
After we derive a formula for  $\theta_{\varepsilon, T}$  and  $\nabla_{\varepsilon} \theta_{\varepsilon, T}$  we realize that the latter can grow in norm in  $T$ . However, if  $\theta_{\varepsilon, T}$  can be Taylor-expanded in  $\varepsilon$ , we need  $\theta_{\varepsilon, T} - \theta_{0, T}$  to be  $O(\varepsilon)$  and  $\nabla_{\varepsilon} \theta_{\varepsilon, T}$  to be  $O(1)$ . If this is not the case, the whole IF approach described in Section 2 breaks down because IF approximate the parameter change  $\theta_{\varepsilon, T} - \theta_{0, T}$  using the Taylor expansion  $\varepsilon^T \nabla_{\varepsilon} \theta_{\varepsilon, T}$ , but the conditions to apply such a Taylor expansion are not satisfied.
In Section 4 we show how assumptions #1–#4 can be successfully addressed which makes them not as problematic as they may first appear. However, for assumption #5 we will see that it puts a substantial limitation on the predictive power of IF. At the same time it allows for a new, locally bound perspective on IF – that influence holds for a limited number of fine-tuning steps (Section 5). Based on this finding, in Section 6 we propose a simple approach to use IF to correct mis-predictions which is theoretically grounded and is in addition much less compute intensive than those that involve re-training (e.g. [KL17]).
4 Addressing the problematic assumptions
Proofs of all results are in the Appendix and we provide some motivation for the proof in the main text.
4.1 HIF does not need Assumption #1
Previous work [CS82, KL17, BPF21,  $\mathrm{BNL}^{+}22]$  on Hessian-based Influence Functions (HIF) has assumed that  $L$  is strictly convex in order to claim that 1) the minimum is unique so that  $\theta_{\varepsilon}$  can be modeled as a function, and 2) to use the Implicit Function Theorem to differentiate through the optimality condition.
Here we will just assume that the Hessian is not singular at  $\theta_0$ ; by requiring that the final gradients do not change as we change  $\varepsilon$  we prove:
Theorem 1. Assume that  $\nabla \mathcal{L}$  is  $C^k$  ( $k \geq 1$ ) and let  $\theta_0 \in \mathbb{R}^N$ ; assume that the Hessian  $H_{\theta_0} = \nabla_{\theta|_{\theta_0}}^2 L$  is non-singular; then there exist neighborhoods  $U$  of  $\theta_0$  and  $V$  of  $0 \in \mathbb{R}^Q$ , and a  $C^k$ -function  $\Theta: V \to U$  such that  $\Theta(0) = \theta_0$  and  $\Theta(\varepsilon) \in U$  is the unique solution in  $U$  of the equation
$$
\nabla_ {\theta} \mathcal {L} (\Theta (\varepsilon), \varepsilon) = \nabla_ {\theta} \mathcal {L} (\theta_ {0}, 0). \tag {3}
$$
Moreover, the gradient of  $\Theta$  at the origin is given by:
$$
\nabla_ {\varepsilon | 0} \Theta = - H _ {\theta_ {0}} ^ {- 1} \nabla_ {(\varepsilon , \theta) | (0, \theta_ {0})} ^ {2} \mathcal {L}. \tag {4}
$$
Idea of the proof. Existence of  $\Theta (\varepsilon)$  is formulated as a solution to the local problem eq. (3); building a local solution does not require convexity if one uses the Implicit Function Theorem.
The requirement that  $\nabla_{\theta}L(\Theta (\varepsilon),\varepsilon)$  is constant in  $\varepsilon$  has allowed us to establish a link between the training under different losses  $\{\theta \to \mathcal{L}(\theta ,\varepsilon)\}_{\varepsilon}$ . If we assume that  $\theta_0$  is a stationary point, i.e.  $\nabla_{\theta}L(\theta_0) = 0$ , we can strengthen the conclusions:
Corollary 1. Under the assumptions of Theorem 1:
1. If  $\theta_0$  is a stationary point of  $L$ , then each  $\Theta(\varepsilon)$  is a stationary point of the loss  $\theta \mapsto \mathcal{L}(\theta, \varepsilon)$ .
2. If  $\theta_0$  is a local (strict) minimum of  $L$ , for  $\varepsilon$  sufficiently small,  $\Theta(\varepsilon)$  is a local (strict) minimum of the loss  $\theta \mapsto \mathcal{L}(\theta, \varepsilon)$ .
3. Let  $Q = 1$  (hence epsilon is a scalar) with  $\mathcal{L}(\theta, \varepsilon) = L(\theta) + \varepsilon l_x(\theta)$ , where  $l_x$  is the loss corresponding to a specific training point  $x$ . We then obtain the classical result [CS82]:
$$
\left. \frac {d \Theta}{d \varepsilon} \right| _ {\varepsilon = 0} = - \left(\nabla_ {\theta} ^ {2} L \left(\theta_ {0}\right)\right) ^ {- 1} \nabla_ {\theta} l _ {x} \left(\theta_ {0}\right). \tag {5}
$$
4.2 If Assumption #2 is not satisfied, use Arnoldi-based Influence Functions
Theorem 1 requires that  $H_{\theta_0}$  is non-singular. If the Hessian is singular, we just need to keep fixed those parameters that are responsible for the degeneracy. More precisely, we diagonalize  $H_{\theta_0}$ ; we let  $P_1$  be the subspace spanned by the eigenvectors corresponding to the non-zero eigenvalues and let  $P_0$  be its orthogonal complement, that is, the kernel of  $H_{\theta_0}$ . Up to an orthogonal transformation of the parameters, we can assume that  $P_1$  is spanned by the first  $N_1$ -coordinates and decompose  $\theta = (\vartheta, \varphi) \in \mathbf{R}^{N1} \times \mathbf{R}^{N2}$  so that  $H_{\vartheta_0} = \nabla_\vartheta^2 L((\vartheta_0, \varphi_0))$  is non-singular. We then apply Theorem 1 to the restricted variation  $(\vartheta, \epsilon) \mapsto L((\vartheta, \varphi_0), \epsilon)$ . In terms of the original parameters  $\theta$ , this means that the function  $\Theta(\varepsilon)$  is constrained to lie in  $\theta_0 + P_1$ , keeping the  $\varphi$ -component constantly equal to  $\varphi_0$ . Concretely, we can approximate  $P_1$  using the Arnoldi iteration; therefore, we can address the failure of Assumption #2 by using Arnoldi-based Influence Functions (ABIF) [SZTS22], which approximate  $P_1$  using the subspace spanned by the eigenvectors corresponding to the top-k (in absolute value) eigenvalues of the Hessian. The fact that ABIF stabilizes the estimation of influence scores has been observed empirically in previous work: in [SZTS22, Fig. 2] where using ABIF instead of the full Hessian improves the retrieval of unlabeled examples, and in [FLP+23, Fig. 3] where ABIF is the best solver on the QA task and is comparable to the other solvers on the text-completion task.
4.3 The training trajectory can be traced to address Assumptions #3-#4
We need to improve our notation to correctly trace the training trajectory. The first issue is to keep track of the parameters across the time steps; the second issue are sources of non-determinism,
e.g. batch selection or random state for dropout. When comparing training trajectories for different values of  $\varepsilon$  we want our notation to account for sources of non-determinism, as they might increase the difference between the training trajectories.
To address the first issue, we let  $\theta_{\varepsilon,t}$  be the value of the parameters after training on  $\mathcal{L}(\theta, \varepsilon)$  for  $t$  steps. In particular,  $\theta_{\varepsilon,0}$  denotes the initial value condition that we assume held fixed at  $\theta_{\mathrm{init}}$  for different values of  $\varepsilon$ . As random state is a function of the training step (e.g. the batch to use at step  $t$ ), to address the second issue, we just need to allow both the loss and the variation to depend on the train step, denoting them by  $L_t$  and  $\mathcal{L}_t$ .