SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sung-Yub Kim
Graduate School of AI, KAIST sungyub.kim@kaist.ac.kr
Kyungsu Kim
Massachusetts General Hospital and Harvard Medical School kskim.doc@gmail.com
Eunho Yang†
Graduate School of AI, KAIST and AITRICS
eunhoy@kaist.ac.kr
Through a deeper understanding of predictions of neural networks, Influence Function (IF) has been applied to various tasks such as detecting and relabeling mislabeled samples, dataset pruning, and separation of data sources in practice. However, we found standard approximations of IF suffer from performance degradation due to oversimplified influence distributions caused by their bilinear approximation, suppressing the expressive power of samples with a relatively strong influence. To address this issue, we propose a new interpretation of existing IF approximations as an average relationship between two linearized losses over parameters sampled from the Laplace approximation (LA). In doing so, we highlight two significant limitations of current IF approximations: the linearity of gradients and the singularity of Hessian. Accordingly, by improving each point, we introduce a new IF approximation method with the following features: i) the removal of linearization to alleviate the bilinear constraint and ii) the utilization of Geometric Ensemble (GE) tailored for non-linear losses. Empirically, our approach outperforms existing IF approximations for downstream tasks with lighter computation, thereby providing new feasibility of low-complexity/nonlinear-based IF design.
In the last decade, neural networks (NNs) have made tremendous advances in various application areas [49, 24, 47]. To make reasonable predictions with NN-based systems, models must be able to explain their predictions. For example, those who doubt the model's prediction can gain insight and foresight by referencing the explanation of the model. Moreover, mission-critical areas like finance and medicine require a high degree of explainability to ensure that the predictions are not biased [30]. Understanding the mechanism of predictions also allows researchers and engineers to improve prediction quality, ensuring that NNs are performing as intended [32].
To this end, Influence Function (IF) was proposed to explain predictions of pre-trained NNs through training data [25]. Intuitively, IF measures how the leave-one-out (LOO) retraining of a training sample changes the loss of each sample. Therefore, the sign of influence determines whether the training sample is beneficial to others, and the scale of influence measures its impact. Specifically, self-influence, the increase in loss when a sample is excluded, was used to measure how much the sample is memorized [48, 15]: When LOO training is performed on a memorized training sample, its loss will increase substantially since matching its (corrupted) label will be difficult. Therefore, self-influence is used to detect mislabeled samples [48, 52] where memorization occurs. Furthermore,
† Correspondence to
recent works successfully applied IF to various downstream tasks, including dataset pruning [58] and data resampling [59, 63].
Despite its broad applicability, we found that IF and its approximations [25, 48, 52] suffer from oversimplified self-influence distributions due to their bilinear form<sup>1</sup>. Although these approximations are introduced to avoid prohibitive retraining costs of IF, they impose a structural constraint that self-influence becomes quadratic to gradients of pre-trained NNs. Due to this constraint, self-influence follows an unimodal distribution, as gradients of pre-trained NNs typically follow a zero-centered Stable distribution [7, 56]. Unfortunately, unimodal distributions are too restrictive for representing self-influence in real-world datasets containing unlabeled samples. While self-influence distributions estimated by LOO retraining may become bimodal depending on the proportion of (high self-influential) unlabeled samples, unimodal distributions cannot handle this case.
To resolve this problem, we propose a non-linear IF approximation via Geometric Ensemble (GE; [16]). Our method is motivated by a novel connection between IF approximation and linearized Laplace approximation (LA; [38]) that we discovered: IF approximations can be translated to an averaged relationship between two linearized losses over parameters sampled from LA. As linearized losses in this connection cause bilinear forms of IF approximations, we consider an IF approximation without linearization. However, we then identify an additional issue of this approximation due to the singularity of Hessian and its solutions (e.g., damping and truncation). To mitigate this issue, we propose a novel approach using GE to manage the relationship of non-linear losses more effectively. As a result, our approach, Geometric Ensemble for sample eXplanation (GEX), accurately represents the multimodal nature of LOO retraining, leading to improved performance in downstream tasks across various scenarios. Furthermore,  $\mathcal{I}_{\mathrm{GEX}}$  is easy to estimate as it does not require Jacobian-vector products (JVPs) for batch estimation or sub-curvature approximations like LA.
- We identify a distributional bias in commonly used IF approximations. We demonstrate how this bias results in oversimplified distributions for self-influences.
- We provide a novel connection between IF approximations and LA. By identifying an inherent issue of LA, we provide a non-linear IF approximation via GE, named  $\mathcal{I}_{\mathrm{GEX}}$ . Due to its non-linear nature,  $\mathcal{I}_{\mathrm{GEX}}$  can express various influence distributions depending on the characteristic of the datasets.
- We verify that  $\mathcal{I}_{\mathrm{GEX}}$  outperforms standard IF approximations in downstream tasks, including noisy label detection, relabeling, dataset pruning, and data source separation. We also show that  $\mathcal{I}_{\mathrm{GEX}}$  is competitive with well-known baselines of downstream tasks with lighter computation.2
Consider an independently and identically (i.i.d.) sampled training dataset  $S \coloneqq \{z_{n} : (x_{n}, y_{n})\}_{n=1}^{N}$  where  $x_{n} \in \mathbb{R}^{D}$  is an input vector of  $n$ -th sample and  $y_{n} \in \mathbb{R}^{K}$  is its label. To model the relation between inputs and outputs of training samples, we consider a neural network (NN)  $f: \mathbb{R}^{D} \times \mathbb{R}^{P} \to \mathbb{R}^{K}$  which maps input  $x \in \mathbb{R}^{D}$  and network parameter  $\theta \in \mathbb{R}^{P}$  to a prediction  $\hat{y}_{n} \coloneqq f(x_{n}, \theta) \in \mathbb{R}^{K}$ . Empirical risk minimization (ERM) solves the following optimization problem
$$
\theta^{*}:= \operatorname *{argmin}_{\theta \in \mathbb{R}^{P}}L(S,\theta)
$$
where  $L(S,\theta)\coloneqq \sum_{n = 1}^{N}\ell (z_n,\theta) / N$  for a sample-wise loss  $\ell (z_{n},\theta)$  (e.g., cross-entropy between  $\hat{y}_n$  and  $y_{n}$ ). In general,  $\theta^*$  is trained using a stochastic optimization algorithm (e.g., stochastic gradient descent (SGD) with momentum). For the reader's convenience, we provide the notation table in Appendix A for terms used in the paper.
The Leave-One-Out (LOO) retraining effect of  $z \in S$  on another instance  $z' \in \mathbb{R}^D$  is defined as the difference of sample-loss  $z'$  between the  $\theta^{*}$  and the retrained point  $\theta_{z}^{*}$  without  $z$  [9]:
$$
\mathcal {I} _ {\mathrm {L 0 0}} \left(z, z ^ {\prime}\right) := \ell \left(z ^ {\prime}, \theta_ {z} ^ {*}\right) - \ell \left(z ^ {\prime}, \theta^ {*}\right) \tag {1}
$$
<sup>1</sup>Here, the bilinearity refers to the IF and its approximations being bilinear with respect to the sample-wise gradients. Consequently, the self-influence based on these bilinear metrics is quadratic for sample-wise gradients.
$^{2}$ Code is available at https://github.com/sungyubkim/gex.
where  $\theta_{z}^{*} := \operatorname{argmin}_{\theta \in \mathbb{R}^{P}} L(S, \theta) - \ell(z, \theta) / N$ . Since retraining for every pair  $(z, z')$  is computationally intractable where we have a huge number of data points as in practice, Koh and Liang [25] proposed an efficient approximation of  $\mathcal{I}_{\mathrm{LO0}}$ , named Influence Function (IF):
$$
\mathcal {I} \left(z, z ^ {\prime}\right) := g _ {z ^ {\prime}} ^ {\top} H ^ {- 1} g _ {z} \tag {2}
$$
where  $g_{z} \coloneqq \nabla_{\theta}\ell (z,\theta^{*}) \in \mathbb{R}^{P}$  and  $H \coloneqq \nabla_{\theta}^{2}L(S,\theta^{*}) \in \mathbb{R}^{P\times P}$  by assuming the strictly convexity of  $\ell$  (i.e.,  $H$  is positive definite). Here,  $\mathcal{I}$  can be understood as a two-step approximation of  $\mathcal{I}_{\mathrm{LO0}}$
$$
\mathcal {I} _ {\mathrm {L 0 0}} \left(z, z ^ {\prime}\right) \approx \ell_ {\theta^ {*}} ^ {\ln} \left(z ^ {\prime}, \theta_ {z} ^ {*}\right) - \ell^ {\ln} \left(z ^ {\prime}, \theta^ {*}\right) = g _ {z ^ {\prime}} ^ {\top} \left(\theta_ {z} ^ {*} - \theta^ {*}\right) \tag {3}
$$
$$
\approx g _ {z ^ {\prime}} ^ {\top} H ^ {- 1} g _ {z} = \mathcal {I} (z, z ^ {\prime}) \tag {4}
$$
where  $\ell_{\theta^*}^{\mathrm{lin}}(z,\psi)\coloneqq \ell (z,\theta^*) + g_z^\top (\psi -\theta^*)$  and  $\psi \in \mathbb{R}^{P}$  is an arbitrary vector in the parameter space  $\mathbb{R}^P$ . Here, we use the superscript lin to indicate linearization. Note that (3) applies linearization to the sample-loss  $z^{\prime}$  and (4) approximates parameter difference as a Newton ascent term (i.e.,  $\theta_z^*\approx \theta^* +H^{-1}g_z$ ).
While the computation of  $\mathcal{I}$  is cheaper than  $\mathcal{I}_{\mathrm{L00}}$ , it is still intractable to modern NNs (e.g., ResNet [20] and Transformer [62]) because of the prohibitively large Hessian. To alleviate this problem, two additional approximations are commonly used: stochastic approximation methods such as  $\mathcal{I}_{\mathrm{LiSSA}}$  [1], and sub-curvature approximations such as  $\mathcal{I}_{\mathrm{Last - Layer}}$ , which limits the Hessian computation only to the last-layer of NNs [10]. However, both methods also have their own problems:  $\mathcal{I}_{\mathrm{LiSSA}}$  takes high time complexity since the inverse Hessian-vector product (IHVP) for each training sample needs to be computed separately, as shown in Schioppa et al. [52]. On the other hand,  $\mathcal{I}_{\mathrm{Last - Layer}}$  may cause inaccurate IF approximations, as shown in Feldman and Zhang [15].
As another alternative, Pruthi et al. [48] recently proposed to exploit intermediate checkpoints during pre-training, named  $\mathcal{I}_{\mathrm{TracIn}}$ :
$$
\mathcal {I} _ {\text {T r a c I n}} \left(z, z ^ {\prime}\right) := \frac {1}{C} \sum_ {c = 1} ^ {C} g _ {z ^ {\prime}} ^ {c \top} g _ {z} ^ {c} \tag {5}
$$
where  $g_{z}^{c} \coloneqq \nabla_{\theta}\ell (z,\theta^{c})$  for checkpoints  $\theta^c$  ( $c = 1,\dots ,C$ ) sampled from the pre-training trajectory of  $\theta^{*}$ . Here, it further simplifies the computation by assuming the expensive  $H^{-1}$  in (2) is an identity matrix. Instead, the performance of  $\mathcal{I}_{\mathrm{TracIn}}$  is replenished by averaging over several intermediate checkpoints, which capture various local geometries of loss landscapes. In addition, Pruthi et al. [48] enhanced the efficiency of  $\mathcal{I}_{\mathrm{TracIn}}$  using random projection, named  $\mathcal{I}_{\mathrm{TracInRP}}$ ,
$$
\mathcal {I} _ {\text {T r a c I n R P}} \left(z, z ^ {\prime}\right) := \frac {1}{C} \sum_ {c = 1} ^ {C} g _ {z ^ {\prime}} ^ {c \top} Q _ {R} Q _ {R} ^ {\top} g _ {z} ^ {c} \tag {6}
$$
where  $Q_{R} \in \mathbb{R}^{P \times R}$  is a random projection matrix whose components are i.i.d. sampled from  $\mathcal{N}(0,1 / R)$  for  $R \ll P$ . Note that  $\mathcal{I}_{\mathrm{TracInRP}}$  is an unbiased estimator of  $\mathcal{I}_{\mathrm{TracIn}}$  as  $\mathbb{E}[Q_R Q_R^\top] = \mathbf{I}_P$  where  $\mathbf{I}_P$  is the identity matrix of dimension  $P \times P$ . However,  $\mathcal{I}_{\mathrm{TracIn}}$  and  $\mathcal{I}_{\mathrm{TracInRP}}$  cannot be applied to checkpoints of the open-source community, such as TorchHub [45] and Huggingface models [65] since they only provide the final checkpoints without any intermediate results during pre-training.
On the other hand, Schioppa et al. [52] proposed to approximate IF in a purely post-hoc manner as
$$
\mathcal {I} _ {\text {A r n o l d i}} \left(z, z ^ {\prime}\right) := g _ {z ^ {\prime}} ^ {\top} U _ {R} \Lambda_ {R} ^ {- 1} U _ {R} ^ {\top} g _ {z} \tag {7}
$$
where  $\Lambda_R \in \mathbb{R}^{R \times R}$  is a diagonal matrix whose elements are top- $R$  eigenvalues of  $H$  and the columns of  $U_R \in \mathbb{R}^{P \times R}$  are corresponding eigenvectors. They use Arnoldi iteration [3] to estimate  $\Lambda_R$  and  $U_R$ . Contrary to  $\mathcal{I}_{\mathrm{TracInRP}}$  with the random projection,  $\mathcal{I}_{\mathrm{Arnoldi}}$  projects  $g_z, g_{z'}$  to the top- $R$  eigenspace of the Hessian. Due to this difference, Schioppa et al. [52] argued that  $\mathcal{I}_{\mathrm{Arnoldi}}$  perform comparably to  $\mathcal{I}_{\mathrm{TracInRP}}$  with less  $R$ .
Finally, it is important to note that  $\mathcal{I}_{\mathrm{TracInRP}}$  and  $\mathcal{I}_{\mathrm{Arnoldi}}$  can perform batch computations using Jacobian-vector products (JVPs). To be more precise, computing  $\mathcal{I}$  and  $\mathcal{I}_{\mathrm{TracIn}}$  for multiple samples requires sample-wise gradient computation for each  $g_{z}, g_{z}^{c}$ , which is difficult to parallelize because of heavy memory complexity. In contrast,  $\mathcal{I}_{\mathrm{TracInRP}}$  and  $\mathcal{I}_{\mathrm{Arnoldi}}$  can avoid this sample-wise computation by computing JVPs for a batch at once (i.e., parallelize  $g_{z}^{c\top} Q_{R}, g_{z}^{\top} U_{R}$  for multiple  $z$ ).
(a) Random proj. of grads.
(b)  $\mathcal{I}_{\mathrm{arnoldi}}$
Figure 1: Since pre-trained gradients follow Gaussian distributions (Fig. 1(a)), quadratic self-influences follow unimodal distributions (See Proposition 3.1), even in noisy label settings (Figs. 1(b)-1(c)). This issue can be solved by removing linearization in the IF approximation (Fig. 1(d)).
(c)  $\mathcal{I}_{\mathrm{TracInRP}}$
(d)  $\mathcal{I}_{\mathrm{GEX}}$
3 Identifying distributional bias in bilinear influence approximations
Throughout the discussion above, IF approximations ((2), (5), (6), and (7)) are defined as bilinear forms of gradients. In this section, we demonstrate a side effect of this form on computing self-influence. Our key empirical observation is that gradients of pre-trained NNs follow Gaussian distributions centered at zero. To verify this, we train VGGNet [55] on MNIST [33] with  $10\%$  label corruption following Schioppa et al. [52]. Then, we project  $\{g_z\}_{z\in S}$  onto each of 5 random vectors  $\{d_i\}_{i = 1}^5$  (with  $d_{i}\in \mathbb{R}^{P}$ ), uniformly sampled on the unit sphere. If a pre-trained gradient  $g_{z}$  follows Gaussian distribution, its projected component  $g_{z}^{\top}d_{i}$  will also follow Gaussian distributions for all  $i = 1,\dots ,5$ . We visualize histograms of projected components  $g_{z}^{\top}d_{i}$  for each  $d_{i}$ , and the resulting p-values of the normality test [57] in Fig. 1(a). In agreement with our hypothesis, the randomly projected components follow Gaussian distributions with significantly low p-values. Moreover, one can observe that Gaussian distributions are centered at zero. This is because gradients at  $\theta^{*}$  satisfy
$$
\mathbb {E} _ {z \sim S} \left[ g _ {z} ^ {\top} d \right] = \frac {1}{N} \sum_ {z \in S} g _ {z} ^ {\top} d = \mathbf {0} _ {P} ^ {\top} d = 0
$$
by the first-order optimality condition at  $\theta^{*}$  (i.e.,  $\nabla_{\theta}L(S,\theta^{*}) = \mathbf{0}_{P}$ ) where  $\mathbf{0}_P$  is the zero vector of dimension  $P$ . Note that a similar observation on the normality of gradient was reported in the context of differential privacy in Chen et al. [7].
Now let us write the eigendecomposition of Hessian as  $H = \sum_{i=1}^{P} \lambda_i u_i u_i^\top$  where  $\lambda_1, \ldots, \lambda_P > 0$  are eigenvalues of  $H$  in descending order and  $u_1, \ldots, u_P \in \mathbb{R}^P$  are corresponding eigenvectors by positive definite assumption of Koh and Liang [25]. We then arrange self-influence of  $\mathcal{I}$  as
$$
\mathcal {I} (z, z) = g _ {z} ^ {\top} H ^ {- 1} g _ {z} = \sum_ {i = 1} ^ {P} \frac {\left(g _ {z , i}\right) ^ {2}}{\lambda_ {i}} \tag {8}
$$
where  $g_{z,i} \coloneqq g_z^\top u_i$  are  $i$ -th component of  $g_{z}$  in the eigenspace of  $H$ . Following the above discussion, one can assume that  $g_{z,i}$  follows a Gaussian distribution. Consequently,  $\mathcal{I}(z,z)$  follows a (generalized)  $\chi^2$ -distribution due to squared components in (8). The following proposition shows that this phenomenon can be generalized to any stable distribution [56] and positive definite matrix.
Proposition 3.1 (Distributional bias in bilinear self-influence). Let us assume  $g_{z}$  follows a  $P$ -dimensional stable distribution (e.g., Gaussian, Cauchy, and Lévy distribution) and  $M \in \mathbb{R}^{P \times P}$  is a positive (semi-)definite matrix. Then, self-influence in the form of  $\mathcal{I}_M(z,z) = g_z^\top Mg_z$  follows a unimodal distribution. Furthermore, if  $g_{z}$  follows a Gaussian distribution, then the self-influence follows a generalized  $\chi^2$ -distribution.
We refer to Appendix B for the proof. Self-influence distributions approximated by  $\mathcal{I}_{\mathrm{Arnoldi}}$  (Fig. 1(b)) and  $\mathcal{I}_{\mathrm{TracInRP}}$  (Fig. 1(c)) provide the empirical evidence of Proposition 3.1 in the noisy label setting in Fig. 1(a). In this setting, each unlabeled sample exhibits high self-influence values since predictions by incorrect labels are challenging to recover when removed and retrained. Therefore, unlabeled samples constitute a distinct mode with a greater self-influence than correctly labeled samples. Still, the distributional bias in Proposition 3.1 squeezes correct and mislabeled samples in a unimodal distribution. As a result, two types (clean and corrupted) of samples are indistinguishable in  $\mathcal{I}_{\mathrm{Arnoldi}}$  and  $\mathcal{I}_{\mathrm{TracInRP}}$ . Proposition 3.1 shows that this observation can be generalized to a more diverse setting (e.g., heavy-tailed distributions). In Sec. 5.1, we demonstrate that this bias occurs regardless of the dataset and the architectures of NN.
(a) Trainingig dataset
(b)  $\mathcal{I}_{\mathrm{GT}}$
Figure 2: Fig. 2(a): The modified two-circle dataset in Sec. 4. Here, the typical samples are the two outer circle samples with relatively high density, and influential samples correspond to the inner circle, demonstrating relatively low density. The self-influence histogram estimated with  $\mathcal{I}_{\mathrm{LO0}}$ ,  $\mathcal{I}$ , and  $\mathcal{I}_{\mathrm{GEX}}$  are provided in Fig. 2(b)-2(d). Fig. 2(b):  $\mathcal{I}_{\mathrm{LO0}}$  properly separates the influential samples from the typical samples in the modified two-circle dataset. Fig. 2(c):  $\mathcal{I}$  mixes typical and influential samples due to the distributional bias in Sec. 3. Fig. 2(d):  $\mathcal{I}_{\mathrm{GEX}}$  accurately represents the bimodal nature of  $\mathcal{I}_{\mathrm{LO0}}$ .
(c)  $\mathcal{I}(\alpha = 100)$
4 Geometric Ensemble for sample eXplanation
To mitigate the distributional bias in Sec. 3, we propose a flexible IF approximation method using Geometric Ensemble (GE; [16]), named Geometric Ensemble for sample eXplanataion (GEX). Here is a summary of how GEX is developed.
$$
\mathcal {I} \xrightarrow [ \text {S e c t i o n 4 . 1} ]{\text {D e l i n e a r i z a t i o n}} \mathcal {I} _ {\text {L A}} \xrightarrow [ \text {S e c t i o n 4 . 2} ]{\text {L A t o G E}} \mathcal {I} _ {\text {G E X}}
$$
In Sec. 4.1, we ensure that the influence approximation is not a bilinear form for the gradient by replacing gradients in IF with sample-loss deviations. The theoretical foundation for this step is provided by our Theorem 4.1 below, which establishes a relationship between the IF and the Laplace approximation (LA; [38]). Moving on to Sec. 4.2, we modify the parameter distribution to compute the sample-loss deviation from LA to GE. This modification is necessary because GE is based on the local geometry of the loss landscape around  $\theta^{*}$ , similar to LA, while avoiding overestimating loss deviations caused by the singularity of the Hessian.
4.1 GEX and its motivation
The proposed method, GEX, comprises three steps. The first step involves the collection of post-hoc checkpoints  $\{\theta^m\}_{m=1}^M$  through multiple SGD updates on the training loss starting from  $\theta^*$ . Then, the empirical distribution of Geometric Ensemble (GE) is computed as
$$
p _ {\mathrm {G E}} (\psi) := \frac {1}{M} \sum_ {m = 1} ^ {M} \delta_ {\theta^ {(m)}} (\psi). \tag {9}
$$
where  $\delta_{\theta^{(m)}}(\cdot)$  denotes the Dirac delta distribution at  $\theta^{(m)}$ . In the final step, GEX is obtained as the expectation of the product of sample-loss deviations from the pre-trained parameter as follows
$$
\mathcal {I} _ {\mathrm {G E X}} \left(z, z ^ {\prime}\right) = \mathbb {E} _ {\psi \sim p _ {\mathrm {G E}}} \left[ \Delta \ell_ {\theta^ {*}} \left(z, \psi\right) \cdot \Delta \ell_ {\theta^ {*}} \left(z ^ {\prime}, \psi\right) \right] \tag {10}
$$
where  $\Delta \ell_{\theta^{*}}(z,\psi)\coloneqq \ell (z,\psi) - \ell (z,\theta^{*})$  means the sample-loss deviations from  $\theta^{*}$ . We provide the pseudocode for computing  $\mathcal{I}_{\mathrm{GEX}}$  in Appendix C.
The main motivation behind  $\mathcal{I}_{\mathrm{GEX}}$  in (10) is to establish a new connection between  $\mathcal{I}$  and LA. This connection is demonstrated in Theorem 4.1, which shows that  $\mathcal{I}$  is an expectation of the product of linearized sample-loss deviations given that parameters are sampled from LA.
Theorem 4.1 (Connection between IF and LA).  $\mathcal{I}$  in Koh and Liang [25] can be expressed as
$$
\mathcal {I} \left(z, z ^ {\prime}\right) = \mathbb {E} _ {\psi \sim p _ {\mathrm {L A}}} \left[ \Delta \ell_ {\theta^ {*}} ^ {\operatorname {l i n}} \left(z, \psi\right) \cdot \Delta \ell_ {\theta^ {*}} ^ {\operatorname {l i n}} \left(z ^ {\prime}, \psi\right) \right] \tag {11}
$$
where  $\Delta \ell_{\theta^*}^{\mathrm{lin}}(z,\psi)\coloneqq \ell_{\theta^*}^{\mathrm{lin}}(z,\psi) - \ell_{\theta^*}^{\mathrm{lin}}(z,\theta^*) = g_z^\top (\psi -\theta^*)$  and  $p_{\mathrm{LA}}$  is the Laplace approximated posterior
$$
p _ {\mathrm {L A}} (\psi) := \mathcal {N} \left(\psi | \theta^ {*}, H ^ {- 1}\right).
$$
(a)  $\mathcal{I}_{\mathrm{LA}}(\alpha = 0.01)$
Figure 3: Damping trick for singular Hessian causes severe overestimation of self-influence (Fig. 3(a)). This issue cannot be addressed with a large damping coefficient (Fig. 3(b)). Although truncating small eigenvalues can reduce the overestimation of self-influence (Fig. 3(c)), it introduces another error that even occurs in  $\mathcal{I}$  (Fig. 3(d)).
(b)  $\mathcal{I}_{\mathrm{LA}}$  ( $\alpha = 100$ )
(c)  $\mathcal{I}_{\mathrm{LA}}(R = 50)$
(d)  $\mathcal{I}(R = 50)$
The LA was proposed to approximate the posterior distribution with a Gaussian distribution. Recently, it has gained significant attention due to its simplicity and reliable calibration performance [50, 11]. Intuitively, LA is equivalent to the second-order Taylor approximation of log-posterior at  $\theta^{*}$  with Gaussian prior defined as  $p(\psi)\coloneqq \mathcal{N}(\psi |\mathbf{0}_P,\gamma^{-1}\mathbf{I}_P)$ :
$$
\begin{array}{l} \log p (\theta | S) = \log p (S | \theta) + \log p (\theta) - \log Z \\ = - L (S, \theta) + \log p (\theta) - \log Z \\ \approx - L (S, \theta^ {*}) - (\theta - \theta^ {*}) ^ {\top} (H + \gamma \mathbf {I} _ {P}) (\theta - \theta^ {*}) / 2 - \log Z \\ \propto - (\theta - \theta^ {*}) ^ {\top} (H + \gamma \mathbf {I} _ {P}) (\theta - \theta^ {*}) / 2 \\ \end{array}
$$