SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To simplify the exposition and for consistency with [PLKS20] we assume that models are trained with stochastic gradient descent. Letting  $\eta_t$  be the learning rate at step  $t$  we prove:
Theorem 2. Assume that the model is trained for  $T$  time-steps with stochastic gradient descent. Denoting by  $H_{t}$  the Hessian  $\nabla_{\theta |\theta_{0,t}}^2 L_t$ , then the final parameters  $\theta_{\varepsilon ,T}$  satisfy:
$$
\nabla_ {\varepsilon | 0} \theta_ {\varepsilon , T} = - \sum_ {t = 0} ^ {T - 1} \eta_ {t} \nabla_ {(\varepsilon , \theta) \mid (0, \theta_ {0, t})} ^ {2} \mathcal {L} _ {t} - \sum_ {t = 0} ^ {T - 1} \eta_ {t} H _ {t} \nabla_ {\varepsilon | 0} \theta_ {\varepsilon , t}. \tag {6}
$$
Idea of the proof. One can explicitly write a system of equations for  $\theta_{\varepsilon, T}$  and apply the operator  $\nabla_{\varepsilon|0}$  to the solution of this system.
Note that the second term on the RHS of (6), which is missing from (2), takes into account the contribution of the earlier time steps that is missing from the analysis of [PLKS20]. A practical consequence of this second term is that the norm of  $\nabla_{\epsilon |0}\theta_{\varepsilon ,T}$  might grow (in  $T$  ) more quickly than (2) would suggest: this is closely related to Assumption #5 which requires  $\nabla_{\epsilon |0}\theta_{\varepsilon ,T}$  to be  $O(1)$ . In particular, for a constant learning rate, while (2) suggests a linear growth in the time step  $T$ , we will empirically verify in Section 5.1 that the growth is super-linear.
In Section K we illustrate the additional modeling error introduced by TracIn when computing the Jacobian  $\nabla_{\epsilon |0}\theta_{\varepsilon ,T}$  in the case of retraining BERT with SGD.
4.4 Assumption #5 becomes problematic over time
To address Assumption #5 we need to look into the parameter divergence  $\| \theta_{\varepsilon, T} - \theta_{0, T}\|$  between training on  $L$  and  $\mathcal{L}$ . The training dynamics is a discrete version of an ODE for which uniqueness and differentiability of the solutions with respects to the initial conditions can be established by Gronwall's Lemma [Gro19]. The parameter  $\varepsilon$  can itself be considered an initial condition and we can prove a discrete version of Gronwall's Lemma to bound the parameter divergence. For simplicity of notation we prove the result for stochastic gradient descent, but we also sketch in the Appendix how to modify the argument to deal with optimizers.
Theorem 3. In the setting of Theorem 2 assume that, for  $t \leq T$ ,  $\theta_{\varepsilon,t}$  lies in a bounded region  $R$  such that
$$
\sup  _ {t, \theta \in R} \| \nabla \mathcal {L} _ {t} (\theta , \varepsilon) - \nabla \mathcal {L} (\theta , 0) \| \leq C \varepsilon \tag {7}
$$
and that for  $\theta \in R$  each loss  $\mathcal{L}_t(\theta, \varepsilon)$  and its gradient wrt.  $\theta$  are  $A$ -Lipschitz in  $\theta$ . Then
$$
\left\| \theta_ {\varepsilon , T} - \theta_ {0, T} \right\| \leq C \varepsilon \sum_ {s <   T} \eta_ {s} \left(1 + \exp \left(2 A \sum_ {s <   T} \eta_ {s}\right)\right). \tag {8}
$$
Idea of the proof. If the time steps were infinitesimal, i.e. if time was made continuous, the evolution of  $\theta_{\varepsilon ,t}$  would be governed by an ODE; then the bound (8) would be straightforward by applying the classical Gronwall's Lemma [Gro19]. In our case we just need to modify the ODE arguments to work with discrete time.
Note that the bound (8) is quite pessimistic as it involves an exponential of the integrated learning rate  $\sum_{s < T} \eta_s$ . This means that as  $\sum_{s < T} \eta_s$  increases, the parameter divergence is no longer  $O(\varepsilon)$  and
(b)
Figure 1: Divergence of parameters (log-scale) as a function of the integrated learning rate. For each value of  $\epsilon$  the divergence is exponential (corresponding to a line in log-scale) with two different divergence rates, one more steep at the beginning of (re)-training. (a) BERT, (b) ResNet
the crucial Assumption #5 is no longer satisfied. This observation leads to a few crucial conclusions:
1. An IF method can predict  $\theta_{\varepsilon, t}$  only for a limited amount of time-steps: it is therefore incorrect to evaluate IF methods on LSOR or retraining from scratch.
2. IF methods need to be evaluated on what they can potentially do; therefore the evaluation setup should consist of fine-tuning on the perturbed loss only a limited amount of steps with evaluation metrics being reported as a function of the step.
3. Applying IF for correcting mis-predictions should also involve a time-bound scenario: we propose such a method in Section 6.
4. Sources of non-determinism between two training runs will likely increase the parameter divergence. So one should try to reduce this with deterministic training. For example, in the case of re-weighting a point  $x$ , i.e. setting  $\mathcal{L} = L + \varepsilon l_x$ , one should make sure to use the same batch  $B_t$  for the loss  $L$  at time step  $t$  across training runs for different values of  $\varepsilon$ .
5 Illustrating the Theory
In this section we first demonstrate Theorem 3 empirically and then verify that the predictive power of influence scores degrades over time. Full details of our experimental setup are reported in the Appendix. We consider binary classification for nlp, where we fine-tune BERT on SST2; for computer vision we consider multi-class classification where we train from scratch ResNet on CIFAR10. All our experiments use deterministic training: the order of the training batches for the loss  $L$  is held fixed across different runs, as well as the random generators when dropout is used.
5.1 Illustrating Parameter Divergence (Theorem 3)
Theorem 3 provides an upper bound when re-training on a perturbed loss. Such a bound is rather pessimistic as it involves the integrated learning rate. We therefore investigate empirically what happens with some typical Deep Learning setups. We take an intermediate checkpoint and keep training on a new loss obtained by up-sampling 16 training points with a weight  $\varepsilon$ , that is:  $\mathcal{L}_t = L_{B_t} + \varepsilon \cdot L_B$ , where  $B_t$  is the training batch for step  $t$  and  $B$  is the batch of 16 points selected for up-sampling. For each time step  $t$  we then compute  $\| \theta_{\varepsilon, T} - \theta_{0, T} \|$  and then plot it against the integrated learning rate, see Figure 1.
Unfortunately, we observe that in these experiments the upper bound in Theorem 3 is matched by a lower bound with the same exponential divergence. We observe a first phase of quick divergence and then a second one in which the divergence is slower. For the second phase a linear fit of  $\log \left\| \theta_{\varepsilon ,T} - \theta_{0,T}\right\|$  against the integrated learning rate appears to be strong: for example for BERT we obtain an  $R^2$  of at least 0.9 across the different values of  $\varepsilon$ . The fitted slope, corresponding to  $A$  in Theorem 3 depends on  $\varepsilon$  and varies between 30 and 130.
(b)
Figure 2: The predictive power of influence scores on the loss shifts degrades over time. (a) BERT, (b) ResNet. The line represents the average of the correlation  $R(t)$  across runs, with the shaded area the corresponding  $95\%$  confidence region.
5.2 Illustrating the fading of influence
We now verify that the predictive power of influence scores fades over time. We again fix a model checkpoint  $\theta_{\mathrm{init}}$  and select 32 training points and 16 test points. For each training point  $x$  we retrain on  $\mathcal{L}_t^x = L_{B_t} - \frac{1}{100} l_x$ , i.e.  $x$  has been down-sampled; for each test point  $z$  and time step  $t$  we then compute the loss difference  $\delta(z, x, t) = l_{z,x,t} - l_{z,t}$  where  $l_{z,x,t}$  is obtained when (re)-training on  $\mathcal{L}_t^x$  and  $l_{z,t}$  is obtained when training on the vanilla loss  $L_t = L_{B_t}$ . Again, we have kept the order of the batches  $B_t$  the same when re-training. Now, at the original checkpoint  $\theta_{\mathrm{init}}$  we can compute the influence scores  $IF(z, x)$  for different methods, e.g. TracIn or HIF (using the Conjugate Residual method<sup>2</sup>). For each time step we thus have  $32 \times 16$  values of  $\delta(z, x, t)$  that can be linearly regressed against  $IF(z, x)$ ; the corresponding Pearson correlation  $R(t)$  then measures the predictive power of influence scores on the loss shifts when re-training. We repeat the experiments for ResNet 9 times and for BERT 25 times, with a different selection of train and test points, so that we obtain confidence intervals for the resulting time-series  $R(t)$ . Ideally, the theory behind an influence method predicts  $R(t) \sim 1$ . However, as discussed above, Assumption #5 is indeed problematic and, because of the parameter divergence illustrated in 5.1 we expect  $R(t)$  to degrade over time.
As Figure 2 demonstrates, this is indeed the case. The predictive power is high for BERT after a few re-training steps and then degrades quickly oscillating around 0 (which is covered by the confidence intervals). For ResNet, the predictive power is never as high, but it still degrades monotonically over time, as predicted by the theory. As we see in Section 6, the proponents retrieved for ResNet are less effective at correcting mis-predictions than those retrieved for BERT: we conjecture that this is related to the worse predictive power of IF in the case of ResNet. As BERT was trained with the Adam Optimizer, we also considered a variant which takes into account the optimizer's pre-conditioner by multiplying the gradients by the square root of the pre-conditioning matrix. In this case the predictive power is worse than for the vanilla version of TracIn. For BERT, more plots and a further discussion about TracIn are included in the Appendix (Section G.1). In the Appendix (Section G.2), we also illustrate the fading of influence for another NLP pre-trained model, T5. While we have illustrated the fading of influence for both BERT and ResNet, in the latter case the peak of correlation is quite low; we repeated the same experiments on the ViT (Section G.3), where we find a high correlation peak as in the case of BERT.
6 Using Influential Examples for Error Correction
[KL17] propose to correct model mis-predictions by first using influence scores to retrieve the examples most responsible for a given prediction, and, after correcting them, retraining the model. Computational considerations aside, a key conclusion from the theory in Section 3 and the empirical verification in Section 5.2 is that IF only predict influence over a limited number of training steps. In this section we demonstrate that IF can still be used to correct model mis-predictions by taking a few fine-tuning steps on influential examples.
Further details in the Appendix.
(b)
Figure 3: Success rate and retention for error correction. (a) BERT, (b) ResNet
Concretely, we propose to correct mis-predictions at a given test point  $z$  by first identifying a batch of influential examples  $B$  and then taking a few fine-tuning steps on the perturbed loss  $\mathcal{L} = L + \varepsilon \cdot l_{B}$ . We propose two methods: (1) Proponents-correction: we identify the set  $B$  of top-k proponents for the current  $x$  and relabel them to what should be the correct prediction on  $x$ ; (2) Opponents-tuning: as opponents oppose the current prediction, we take  $B$  to be the set of top-k proponents of  $x$ .
We investigate how well these error-correction techniques work on SST2 (BERT) and CIFAR10 (ResNet). As a baseline, we randomly sample a set  $B$  of training points with the same label as the prediction on  $x$  and then set their label equal to the correct one for  $x$ . We take a maximum of 50 fine-tuning steps and take the top-50 proponents or opponents to build  $B$ . The main metric we compute is the success rate, i.e. the ratio of mis-predictions successfully corrected within the limit of 50 steps. Additionally, we report prediction retention [DCAT21] on a fixed held-out set of 50 test examples, i.e. the ratio of examples predictions which have not changed after a correction - ideally, a correction does not cause too many changes in model predictions otherwise. We experiment with different values of  $\varepsilon$ , starting from no up-sampling and gradually increasing it to when influential examples account for slightly more than half of the batch.
From Figure 3 we see that Proponents-correction and Opponents-tuning strongly outperform the baseline in binary classification (SST2). For multi-class classification (CIFAR10) Opponents-tuning is not effective, as we verified that only  $54\%$  of the retrieved opponents have the desired label; Proponents-correction still outperforms the baseline increasing on average the success rate by  $2\%$  and reducing the number of steps to take by  $6\%$ . We conjecture that for ResNet the improvement over the baseline is less than for BERT because of the worse predictive power of IF (Figure 2 (b)). In Appendix H we include additional plots showing the number of steps to correct mis-predictions as a function of the up-sampling parameter  $\varepsilon$ .
The primary goal of the experiments in this section is to verify that tuning on influential examples results in a correction more reliably and faster than on other classes of examples. Since the SST2 and CIFAR10 datasets are largely clean, for the proponents method, the training example label is flipped to an incorrect one. In Appendix I we give examples from a noisy text classification dataset illustrating the scenario that the correction-with-proponents method is supposed to address.
We derive the perturbed training trajectory (Theorem 2) and the divergence of trajectories (Theorem 3) for stochastic gradient descent and, while we sketch in the Appendix the modifications needed when using other optimizers, we do not pursue this topic in detail. In Section 6 we measure prediction retention after correcting mis-predictions. While this metric is intuitive and has been used previously (e.g., [DCAT21]), we do not distinguish between semantically similar and unrelated examples and thus do not check the consistency and generalization properties of the update [MBAB22], leaving a thorough study of the correction-retention tradeoff to future work.
IF have been regarded as a tool that promises to trace model behavior to the training data. Unfortunately, recent studies have found no empirical support for such a claim as IF fail to predict the LSOR effect. This finding gives rise to the question of what IF methods really predict and whether they could be useful for model debugging. In this work we clarified which questions IF can be expected to answer. We first identified problematic assumptions made by IF methods – a priori any of these assumptions could be a reason for the observed empirical failure of IF. Thus, for each assumption we studied if it is indeed problematic. While most have turned out to be addressable in one way or another, we demonstrated that the one about parameter divergence puts a severe limitation on IF. With a deeper analysis of this assumption, we revised what can be theoretically expected from IF: IF methods are time-bound, that is, they can at most predict what happens when fine-tuning on a perturbed loss for a limited amount of time. With that, a practical usage of IF for model debugging is still possible – we proposed and empirically validated a theoretically-grounded procedure to apply IF to correct model mis-predictions.
$\left[\mathrm{BNL}^{+}22\right]$  Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger Grosse. If influence functions are the answer, then what is the question?, 2022.
[BPF21] Samyadeep Basu, Phil Pope, and Soheil Feizi. Influence functions in deep learning are fragile. In International Conference on Learning Representations, 2021.
[CGFT19] Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input similarity from the neural network perspective. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[CS82] R. Cook and Weisberg S. Residuals and influence in regression. Chapman and Hall, New York, 1982.
[DCAT21] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6491–6506, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
$\left[\mathrm{DPG}^{+}14\right]$  Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
$\left[\mathrm{FLP}^{+}23\right]$  Jillian Fisher, Lang Liu, Krishna Pillutla, Yejin Choi, and Zaid Harchaoui. Influence diagnostics under self-concordance. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 10028-10076. PMLR, 2023.
[GKX19] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In ICML, 2019.
$\left[\mathrm{GRH}^{+}21\right]$  Han Guo, Nazneen Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. FastIF: Scalable influence functions for efficient model interpretation and debugging. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10333–10350, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[Gro19] T. H. Gronwall. Note on the derivatives with respect to a parameter of the solutions of a system of differential equations. Annals of Mathematics, 20(4):292-296, 1919.
$\left[\mathrm{GWP}^{+}23\right]$  Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. Simfluence: Modeling the influence of individual training examples by simulating training runs, 2023.
[HYHI20] Kazuaki Hanawa, Sho Yokoi, Satoshi Hara, and Kentaro Inui. Evaluation of similarity-based explanations. In ICLR-21, 2020.
[KATL19] Pang Wei Koh, Kai-Siang Ang, Hubert H. K. Teo, and Percy Liang. On the accuracy of influence functions for measuring group effects. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2019. Curran Associates Inc.
[KL17] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1885-1894. PMLR, 06-11 Aug 2017.
[KS21] Karthikeyan K and Anders Søgaard. Revisiting methods for finding influential examples, 2021.
[MBAB22] Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[PLKS20] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[SZTS22] Andrea Schioppa, Polina Zablotskaia, David Vilar Torres, and Artem Sokolov. Scaling up influence functions. In AAAI-22, 2022.
[WTD17] Ellery Wulczyn, Nithum Thain, and Lucas Dixon. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, WWW '17, pages 1391-1399, Republic and Canton of Geneva, CHE, 2017. International World Wide Web Conferences Steering Committee.
A Clarifications regarding the operator  $\nabla$
We spell out in further detail the way we employ the  $\nabla$  operator. For example, we say that  $\nabla_{\varepsilon |0}\theta_{\varepsilon}$  is the  $Q\times N$  -dimensional Jacobian at  $\varepsilon = 0$  : concretely,  $\nabla_{\varepsilon |\varepsilon_0}f$  means computing the Jacobian of  $f$  wrt.  $\varepsilon$  and evaluating it at the point  $\varepsilon_0$  ; this Jacobian is a matrix whose  $(i,j)$  -entry is given by:
$$
(\nabla_ {\varepsilon | \varepsilon_ {0}} f) _ {i, j} = \frac {\partial f _ {j}}{\partial \varepsilon_ {i}} (\varepsilon_ {0}). \tag {9}
$$
This is easily extended to higher order derivatives denote by  $\nabla^k$ ; specifically we use  $\nabla_{(\varepsilon ,\theta)|(0,\theta_0)}^2\mathcal{L}$  to denote the matrix whose  $i,j$ -entry is given by:
$$
\left(\nabla_ {(\varepsilon , \theta) | (0, \theta_ {0})} ^ {2} \mathcal {L}\right) _ {i, j} = \frac {\partial^ {2} \mathcal {L}}{\partial \varepsilon_ {i} \partial \theta_ {j}} \left(0, \theta_ {0}\right); \tag {10}
$$
as the matrix  $H_{\theta_0}^{-1}$  acts on the space of the parameters  $\theta$ , the notation  $H_{\theta_0}^{-1}\nabla_{(\varepsilon ,\theta)|(0,\theta_0)}^2\mathcal{L}$  will denote a contraction on  $j$ :
$$
\left(H _ {\theta_ {0}} ^ {- 1} \nabla_ {(\varepsilon , \theta) | (0, \theta_ {0})} ^ {2} \mathcal {L}\right) _ {i, k} = \sum_ {j} \left(H _ {\theta_ {0}} ^ {- 1}\right) _ {k, j} \frac {\partial^ {2} \mathcal {L}}{\partial \varepsilon_ {i} \partial \theta_ {j}} (0, \theta_ {0}). \tag {11}
$$
For convenience, we first recall the statement of Theorem 1.
Theorem. Assume that  $\nabla \mathcal{L}$  is  $C^k$  ( $k \geq 1$ ) and let  $\theta_0 \in \mathbb{R}^N$ ; assume that the Hessian  $H_{\theta_0} = \nabla_{\theta|_{\theta_0}}^2 L$  is non-singular; then there exist neighborhoods  $U$  of  $\theta_0$  and  $V$  of  $0 \in \mathbb{R}^Q$ , and a  $C^k$ -function  $\Theta: V \to U$  such that  $\Theta(0) = \theta_0$  and  $\Theta(\varepsilon) \in U$  is the unique solution in  $U$  of the equation
$$
\nabla_ {\theta} \mathcal {L} (\Theta (\varepsilon), \varepsilon) = \nabla_ {\theta} \mathcal {L} \left(\theta_ {0}, 0\right). \tag {12}
$$
$$
\nabla_ {\varepsilon | 0} \Theta = - H _ {\theta_ {0}} ^ {- 1} \nabla_ {(\varepsilon , \theta) | (0, \theta_ {0})} ^ {2} \mathcal {L}. \tag {13}
$$
Proof. Equation (12) gives us  $N$  constraints on the  $N + Q$  variables  $(\theta, \varepsilon)$  and we want to solve them for the first  $N$  variables  $\theta$ ; to obtain the function  $\Theta$  we invoke the Implicit Function Theorem, using that the Jacobian wrt. the variables we want to solve for is  $H_{\theta_0}$  and is therefore non-singular. Finally (13) is obtained by taking the gradient of (12) wrt.  $\varepsilon$  and setting  $\varepsilon = 0$ :
$$
\nabla_ {\theta | \theta_ {0}} ^ {2} L \cdot \nabla_ {\varepsilon | 0} \Theta + \nabla_ {(\varepsilon , \theta) | (0, \theta_ {0})} ^ {2} \mathcal {L} = 0.
$$
For convenience, we first recall the statement of Corollary 1.
Corollary. Under the assumptions of Theorem 1:
1. If  $\theta_0$  is a stationary point of  $L$ , then each  $\Theta(\varepsilon)$  is a stationary point of the loss  $\theta \mapsto \mathcal{L}(\theta, \varepsilon)$ .
2. If  $\theta_0$  is a local (strict) minimum of  $L$ , for  $\varepsilon$  sufficiently small,  $\Theta(\varepsilon)$  is a local (strict) minimum of the loss  $\theta \mapsto \mathcal{L}(\theta, \varepsilon)$ .
3. Let  $Q = 1$  (hence epsilon is a scalar) with  $\mathcal{L}(\theta, \varepsilon) = L(\theta) + \varepsilon l_x(\theta)$ , where  $l_x$  is the loss corresponding to a specific training point  $x$ . We then obtain the classical result [CS82] about the influence of down/up-sampling  $x$  on the training parameters: