SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
q _ {t} \propto e ^ {\boldsymbol {\theta} ^ {\top} \mathbf {S} _ {t} \mathbf {m} _ {t} - \frac {1}{2} \boldsymbol {\theta} ^ {\top} \mathbf {S} _ {t} \boldsymbol {\theta}} \propto (q _ {t - 1}) ^ {1 - \rho} (p _ {0}) ^ {\rho} \prod_ {j = 1} ^ {N} e ^ {\boldsymbol {\theta} ^ {\top} \hat {\mathbf {i}} _ {j, t - 1} - \frac {1}{2} \boldsymbol {\theta} ^ {\top} \hat {\mathbf {I}} _ {j, t - 1} \boldsymbol {\theta}}
$$
The prior above is Gaussian and defined using  $q_{t-1}$  and  $p_0$ . The model uses likelihoods that are Gaussian distribution with information vector  $\hat{\mathbf{i}}_{j,t-1} = \rho(\mathbf{H}_{j,t-1}\mathbf{m}_{t-1} - \hat{\mathbf{g}}_{j,t-1})$  and information matrix  $\hat{\mathbf{I}}_{j,t-1} = \rho\hat{\mathbf{H}}_{j,t-1}$ . The likelihood is allowed to be an improper distribution, meaning that its integral is not one. This is not a problem as long as  $\mathbf{S}_t$  remains positive definite. A valid  $\mathbf{S}_t$  can be ensured by either using a Generalized Gauss-Newton approximation to the Hessian [27] or by using the improved BLR of [35]. The former strategy is used in [25] to express BLR iterations as linear models and Gaussian processes. Ultimately, we want to ensure that perturbation in the approximate likelihoods in  $q_t$  yields a valid posterior and, as long as this is the case, the conjugate model can be used safely. For instance, in Thm. 4, this issue poses no problem at all.
The BLR update can be rearranged and written in a Newton-like form show below,
$$
\text {V O N :} \quad \mathbf {m} _ {t} \leftarrow \mathbf {m} _ {t - 1} - \rho \mathbf {S} _ {t} ^ {- 1} \mathbb {E} _ {q _ {t - 1}} \left[ \nabla \mathcal {L} (\boldsymbol {\theta}) \right], \quad \mathbf {S} _ {t} \leftarrow (1 - \rho) \mathbf {S} _ {t - 1} + \rho \mathbb {E} _ {q _ {t - 1}} \left[ \nabla^ {2} \mathcal {L} (\boldsymbol {\theta}) \right]. \tag {29}
$$
This is called the Variational Online Newton (VON) algorithm. A full derivation is in [27] with details on many of its variants in [28]. The simplest variant is the Online Newton (ON) algorithm, where we use the delta method,
$$
\mathbb {E} _ {q _ {t}} \left[ \nabla \mathcal {L} (\boldsymbol {\theta}) \right] \approx \nabla \mathcal {L} \left(\mathbf {m} _ {t}\right), \quad \mathbb {E} _ {q _ {t}} \left[ \nabla^ {2} \mathcal {L} (\boldsymbol {\theta}) \right] \approx \nabla^ {2} \mathcal {L} \left(\mathbf {m} _ {t}\right). \tag {30}
$$
Then denoting  $\mathbf{m}_t = \pmb{\theta}_t$ , we get the following ON update,
$$
\text {O N}: \quad \boldsymbol {\theta} _ {t} \leftarrow \boldsymbol {\theta} _ {t - 1} - \rho \mathbf {S} _ {t} ^ {- 1} \nabla \mathcal {L} \left(\boldsymbol {\theta} _ {t - 1}\right), \quad \mathbf {S} _ {t} \leftarrow (1 - \rho) \mathbf {S} _ {t - 1} + \rho \nabla^ {2} \mathcal {L} \left(\boldsymbol {\theta} _ {t - 1}\right). \tag {31}
$$
To reduce the cost, we can use a diagonal approximation  $\mathbf{S}_t = \mathrm{diag}(\mathbf{s}_t)$  where  $\mathbf{s}_t$  is a scale vector. Additionally, we can use minibatching to estimate the gradient and hessian (denoted by  $\hat{\nabla}$  and  $\hat{\nabla}^2$ ),
$$
\text {O N} (\text {d i a g o n a l} + \text {m i n i b a t c h}): \boldsymbol {\theta} _ {t} \leftarrow \boldsymbol {\theta} _ {t - 1} - \rho \mathbf {s} _ {t} ^ {- 1} \cdot \hat {\nabla} \mathcal {L} \left(\boldsymbol {\theta} _ {t - 1}\right), \tag {32}
$$
$$
\mathbf {s} _ {t} \leftarrow (1 - \rho) \mathbf {s} _ {t - 1} + \rho \operatorname {d i a g} \left(\tilde {\nabla} ^ {2} \mathcal {L} \left(\boldsymbol {\theta} _ {t - 1}\right)\right),
$$
where  $\cdot$  indicates element-wise product two vectors and  $\mathrm{diag}(\cdot)$  extracts the diagonal of a matrix.
Several optimization algorithms can be obtained as special cases from the above variants. For example, to get Newton's method, we set  $\rho = 1$  in ON to get
$$
\boldsymbol {\theta} _ {t} \leftarrow \boldsymbol {\theta} _ {t - 1} - \left[ \nabla^ {2} \mathcal {L} \left(\boldsymbol {\theta} _ {t - 1}\right) \right] ^ {- 1} \nabla \mathcal {L} \left(\boldsymbol {\theta} _ {t - 1}\right). \tag {33}
$$
RMSprop and Adam can be derived in a similar fashion [28].
In our experiments, we use the improved BLR or iBLR optimizer [35]. We use it to implement an improved version of VON [27, Eqs. 7-8] which ensures that the covariance is always positive-definite, even when the Hessian estimates are not. We use diagonal approximation  $\mathbf{S}_t = \mathrm{diag}(\pmb{\sigma}^2)^{-1}$ , momentum and minibatching as proposed in [27, 35]. For learning rate  $\alpha_t > 0$ , momentum  $\beta_1, \beta_2 \in [0,1)$  the iterations are written as follows:
iBLR:  $\mathbf{g}_t\gets \beta_1\mathbf{g}_{t - 1} + (1 - \beta_1)\widehat{\mathbf{g}}_{t - 1},$
$$
\begin{array}{l} \mathbf {h} _ {t} \leftarrow \beta_ {2} \mathbf {h} _ {t - 1} + (1 - \beta_ {2}) \widehat {\mathbf {h}} _ {t - 1} + \frac {1}{2} (1 - \beta_ {2}) ^ {2} \left(\mathbf {h} _ {t - 1} - \widehat {\mathbf {h}} _ {t - 1}\right) ^ {2} / \left(\mathbf {h} _ {t - 1} + \delta\right), \tag {34} \\ \mathbf {m} _ {t} \leftarrow \mathbf {m} _ {t - 1} - \alpha_ {t} (\mathbf {g} _ {t} + \delta \mathbf {m} _ {t - 1}) / (\mathbf {h} _ {t} + \delta), \\ \sigma_ {t} ^ {2} \leftarrow 1 / \left(N \left(\mathbf {h} _ {t} + \delta\right)\right). \\ \end{array}
$$
Here,  $\delta > 0$  is the  $L_{2}$ -regularization parameter and  $\widehat{\mathbf{g}}_{t-1} = \frac{1}{|B|} \sum_{i \in B} \mathbb{E}_{q_{t-1}(\pmb{\theta})}[\nabla \ell_i(\pmb{\theta})]$ ,  $\widehat{\mathbf{h}}_{t-1} = \frac{1}{|B|} \sum_{i \in B} \mathbb{E}_{q_{t-1}(\pmb{\theta})}[\nabla \ell_i(\pmb{\theta})(\pmb{\theta} - \mathbf{m}_{t-1}) / \pmb{\sigma}_{t-1}^2]$  denote Monte-Carlo approximations of the expected stochastic gradient and diagonal Hessian under  $q_{t-1}(\pmb{\theta}) = \mathcal{N}(\pmb{\theta} | \mathbf{m}_{t-1}, \mathrm{diag}(\pmb{\sigma}_{t-1}^2))$  and minibatch  $B$ . As suggested in [27, 35], we used the reparametrization trick to estimate the diagonal Hessian via gradients only. In practice, we approximate the expectations using a single random sample. We expect multiple samples to further improve the results.
D Proof of Thm. 2 and the Beta-Bernoulli Model
From Eq. 27, it directly follows that
$$
\tilde {\mathbf {g}} _ {j} (\boldsymbol {\lambda}) = \nabla_ {\boldsymbol {\mu}} \mathbb {E} _ {q} [ - \log \tilde {p} _ {j} ] = - \nabla_ {\boldsymbol {\mu}} \langle \widetilde {\boldsymbol {\lambda}} _ {j}, \mathbb {E} _ {q} [ \mathbf {T} (\boldsymbol {\theta}) ] \rangle = - \nabla_ {\boldsymbol {\mu}} \langle \widetilde {\boldsymbol {\lambda}} _ {j}, \boldsymbol {\mu} \rangle = - \widetilde {\boldsymbol {\lambda}} _ {j}.
$$
Using this in Eq. 6, we get the deviation given in Eq. 4.
We will now show an example on Beta-Bernoulli model, which is a conjugate model. We assume the model to be  $p(\mathcal{D},\theta)\propto p(\theta)\prod_{i}p(y_{i}|\theta)$  where the prior is  $p(\theta) = \mathrm{Beta}(\theta |\alpha_0,\beta_0)$  and likelihoods are  $p(y_i|\theta) = \mathrm{Ber}(y_i|\theta)$  with  $\mathcal{D}_i = y_i$ . This is a conjugate model and the posterior is Beta distribution, that is, it takes the same form as the prior. An expression is given below,
$$
q _ {*} = \operatorname {B e t a} (\theta | \alpha_ {*}, \beta_ {*}), \text {w h e r e} \alpha_ {*} = \alpha_ {0} + \sum_ {j = 1} ^ {N} y _ {j}, \quad \beta_ {*} = \beta_ {0} - \sum_ {j = 1} ^ {N} y _ {j} + N.
$$
The posterior for the perturbed dataset  $\mathcal{D}^{\backslash i}$  is also available in closed-form:
$$
q_{*}^{\backslash i} = \operatorname {Beta}(\theta |\alpha_{*}^{\backslash i},\beta_{*}^{\backslash i}),\text{where}\alpha_{*}^{\backslash i} = \alpha_{0} + \sum_{\substack{j = 1,\\ j\neq i}}^{N}y_{j},\qquad \beta_{*}^{\backslash i} = \beta_{0} - \sum_{\substack{j = 1,\\ j\neq i}}^{N}y_{j} + N - 1.
$$
Therefore the deviations in the posterior parameters can be simply obtained as follows:
$$
\alpha_ {*} ^ {\backslash i} - \alpha_ {*} = - y _ {i}, \quad \beta_ {*} ^ {\backslash i} - \beta_ {*} = y _ {i} - 1 \tag {35}
$$
This result can also be straightforwardly obtained using the MPE. For the Beta distribution  $q_{\lambda}(\theta) = \mathrm{Beta}(\theta|\alpha, \beta)$ , we have  $\lambda = (\alpha - 1, \beta - 1)$ , therefore  $\pmb{\lambda}_{*}^{\backslash i} - \pmb{\lambda}_{*} = (\alpha_{*}^{i} - \alpha_{*}, \beta_{*}^{\backslash i} - \beta_{*})$ . For Beta distribution,  $\mathbf{T}(\theta) = (\log \theta, \log (1 - \theta))$  and writing the likelihood in an exponential form, we get
$$
p \left(y _ {i} \mid \theta\right) = \operatorname {B e r} \left(y _ {i} \mid \theta\right) \propto \theta^ {y _ {i}} (1 - \theta) ^ {1 - y _ {i}} \propto e ^ {y _ {i} \log \theta + \left(1 - y _ {i}\right) \log (1 - \theta)},
$$
therefore  $\widetilde{\lambda}_i = (y_i, y_i - 1)$ . Setting  $\lambda_*^i - \lambda_* = -\widetilde{\lambda}_i$ , we recover the result given in Eq. 35.
E Proof of Thm. 3
For linear regression, we have
$$
\nabla \ell_ {i} (\boldsymbol {\theta}) = \mathbf {x} _ {i} \left(\mathbf {x} _ {i} ^ {\top} \boldsymbol {\theta} - y _ {i}\right), \qquad \nabla^ {2} \ell_ {i} (\boldsymbol {\theta}) = \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top}.
$$
Using these in Eq. 8, we get,
$$
\tilde {\mathbf {g}} _ {i} \left(\boldsymbol {\lambda} _ {*}\right) = \mathbb {E} _ {q} \left[ \mathbf {x} _ {i} \left(\mathbf {x} _ {i} ^ {\top} \boldsymbol {\theta} - y _ {i}\right) - \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top} \boldsymbol {\theta} _ {*}, \frac {1}{2} \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top} \right] = \left(- \mathbf {x} _ {i} y _ {i}, \frac {1}{2} \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top}\right),
$$
The natural parameter is  $\lambda_{*} = (\mathbf{H}_{*}\pmb{\theta}_{*}, - \frac{1}{2}\mathbf{H}_{*})$ . In a similar way, we can define  $q_{*}^{\backslash i}$  and its natural parameter. Using these, we can write Eq. 6 as
$$
\mathbf {H} _ {*} ^ {\backslash i} \boldsymbol {\theta} _ {*} ^ {\backslash i} - \mathbf {H} _ {*} \boldsymbol {\theta} _ {*} = - \mathbf {x} _ {i} y _ {i}, \quad - \frac {1}{2} \mathbf {H} _ {*} ^ {\backslash i} + \frac {1}{2} \mathbf {H} _ {*} = \frac {1}{2} \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top}.
$$
Substituting the second equation into the first one, we get the first equation below,
$$
\mathbf {H} _ {*} ^ {\backslash i} \pmb {\theta} _ {*} ^ {\backslash i} - (\mathbf {H} _ {*} ^ {\backslash i} + \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top}) \pmb {\theta} _ {*} = - \mathbf {x} _ {i} y _ {i} \quad \Longrightarrow \quad \pmb {\theta} _ {*} ^ {\backslash i} - \pmb {\theta} _ {*} = (\mathbf {H} _ {*} ^ {\backslash i}) ^ {- 1} \mathbf {x} _ {i} (\mathbf {x} _ {i} ^ {\top} \pmb {\theta} _ {*} - y _ {i}) = (\mathbf {H} _ {*} ^ {\backslash i}) ^ {- 1} \mathbf {x} _ {i} e _ {i}.
$$
The last equality is exactly Eq. 2. Since linear regression is a conjugate model, an alternate derivation would be to directly use the parameterization  $\tilde{\lambda}_j$  of  $\tilde{p}_i$  (derived in App. B) and plug it in Thm. 2.
F Proof of Thm. 4
For simplicity, we denote
$$
\partial \hat {\boldsymbol {\lambda}} _ {*} ^ {\epsilon_ {i} = 0} = \left. \frac {\partial \hat {\boldsymbol {\lambda}} _ {*} ^ {\epsilon_ {i}}}{\partial \epsilon_ {i}} \right| _ {\epsilon_ {i} = 0},
$$
with  $\hat{\lambda}_{*}^{\epsilon_{i}}$  as defined in Eq. 7 in the main text. For Gaussian distributions, the natural parameter comes in a pair  $\hat{\lambda}_{*}^{\epsilon_{i}} = (\mathbf{H}_{*}^{\epsilon_{i}}\pmb{\theta}_{*}^{\epsilon_{i}}, - \frac{1}{2}\mathbf{H}_{*}^{\epsilon_{i}})$ . Its derivative with respect to  $\epsilon_{i}$  at  $\epsilon_{i} = 0$  can be written as the following by using the chain rule:
$$
\partial \hat {\boldsymbol {\lambda}} _ {*} ^ {\epsilon_ {i} = 0} = \left(\mathbf {H} _ {*} \partial \boldsymbol {\theta} _ {*} ^ {\epsilon_ {i} = 0} + \partial \mathbf {H} _ {*} ^ {\epsilon_ {i} = 0} \boldsymbol {\theta} _ {*}, - \frac {1}{2} \partial \mathbf {H} _ {*} ^ {\epsilon_ {i} = 0}\right).
$$
Here, we use the fact that, as  $\epsilon_{i}\rightarrow 0$ , we have  $(\pmb{\theta}_{*}^{\epsilon_{i}},\mathbf{H}_{*}^{\epsilon_{i}})\to (\pmb{\theta}_{*},\mathbf{H}_{*})$  and also assumed that the limit of the product is equal to the product of the individual limits. Next, we need the expression for the natural gradient, for which we will use Eq. 8 but approximate the expectation by using the delta approximation  $\mathbb{E}_{q_*}[g(\pmb {\theta})]\approx g(\pmb {\theta}_*)$  for any function  $g$ , as shown below to define:
$$
\hat {\mathbf {g}} _ {i} (\boldsymbol {\lambda} _ {*}) = \left[ \nabla \ell_ {i} (\boldsymbol {\theta} _ {*}) - \nabla^ {2} \ell_ {i} (\boldsymbol {\theta} _ {*}) \boldsymbol {\theta} _ {*}, \frac {1}{2} \nabla^ {2} \ell_ {i} (\boldsymbol {\theta} _ {*}) \right]
$$
The claim is that if we set the perturbed  $\partial \hat{\pmb{\lambda}}_{*}^{\epsilon_{i} = 0} = \hat{\mathbf{g}}_{i}(\pmb{\lambda}_{*})$  we recover Eq. 3, that is, we set
$$
\mathbf {H} _ {*} \partial \boldsymbol {\theta} _ {*} ^ {\epsilon_ {i} = 0} + \partial \mathbf {H} _ {*} ^ {\epsilon_ {i} = 0} \boldsymbol {\theta} _ {*} = \nabla \ell (\boldsymbol {\theta} _ {*}) - \nabla^ {2} \ell_ {i} (\boldsymbol {\theta} _ {*}) \boldsymbol {\theta} _ {*}, \quad - \frac {1}{2} \partial \mathbf {H} _ {*} ^ {\epsilon_ {i} = 0} = \frac {1}{2} \nabla^ {2} \ell_ {i} (\boldsymbol {\theta} _ {*}).
$$
Plugging the second equation into the first, the second term cancels and we recover Eq. 3.
G Extension to Non-Differentiable Loss function
For non-differentiable cases, we can use Eq. 28 to rewrite the BLR of Eq. 29 as
$$
\mathbf {m} _ {t} \leftarrow \mathbf {m} _ {t - 1} - \rho \mathbf {S} _ {t} ^ {- 1} \nabla_ {\mathbf {m}} \mathbb {E} _ {q _ {t - 1}} [ \mathcal {L} (\boldsymbol {\theta}) ], \quad \mathbf {S} _ {t} \leftarrow (1 - \rho) \mathbf {S} _ {t - 1} + 2 \rho \nabla_ {\boldsymbol {\Sigma}} \mathbb {E} _ {q _ {t - 1}} [ \mathcal {L} (\boldsymbol {\theta}) ], \tag {36}
$$
where  $\boldsymbol{\Sigma} = \mathbf{S}^{-1}$ . Essentially, we take derivative outside the expectation instead of inside which is valid because the expectation of a non-differentiable function is still differentiable (under some regularity conditions). The same technique can be applied to Eq. 8 to get
$$
\tilde {\mathbf {g}} _ {i} (\boldsymbol {\lambda}) = \left(\nabla_ {\mathbf {m}} \mathbb {E} _ {q} [ \ell_ {i} ] - 2 \nabla_ {\boldsymbol {\Sigma}} \mathbb {E} _ {q} [ \ell_ {i} (\boldsymbol {\theta}) ] \mathbf {m}, \nabla_ {\boldsymbol {\Sigma}} \mathbb {E} _ {q} [ \ell_ {i} (\boldsymbol {\theta}) ]\right), \tag {37}
$$
and proceeding in the same fashion we can write:  $\hat{\mathbf{m}}_t^{\backslash i} - \mathbf{m}_t = (\hat{\mathbf{S}}_t^{\backslash i})^{-1}\nabla_{\mathbf{m}}\mathbb{E}_{q_t}[\ell_i(\pmb {\theta})]$ . This is the extension of Eq. 10 to non-differentiable loss functions.
H Sensitivity Measures for Sparse Variational Gaussian Processes
Sparse variational GP (SVGP) methods optimize the following variational objective to find a Gaussian posterior approximation  $q(\mathbf{u})$  over function values  $\mathbf{u} := (f(\mathbf{z}_1), f(\mathbf{z}_2), \dots, f(\mathbf{z}_M))$  where  $\mathcal{Z} := (\mathbf{z}_1, \mathbf{z}_2, \dots, \mathbf{z}_M)$  is the set of inducing inputs with  $M \ll N$ :
$$
\underline {{\mathcal {L}}} (\mathbf {m}, \boldsymbol {\Sigma}, \mathcal {Z}, \phi) := \sum_ {i = 1} ^ {N} \mathbb {E} _ {q (f _ {i})} [ \log p (y _ {i} | f _ {i}) ] - \mathbb {D} _ {\mathrm {K L}} (q (\mathbf {u}) \| p (\mathbf {u}))
$$
where  $p(\mathbf{u}) \coloneqq \mathcal{N}(\mathbf{u}|\mathbf{0},\mathbf{K}_{\mathbf{uu}})$  is the prior with  $\mathbf{K}_{\mathbf{uu}}$  as the covariance function  $\kappa (\cdot ,\cdot ')$  evaluated at  $\mathcal{Z}$ ,  $q(f_{i}) = \mathcal{N}(f_{i}|\mathbf{a}_{i}^{\top}\mathbf{m},\mathbf{a}_{i}^{\top}\boldsymbol {\Sigma}\mathbf{a}_{i} + \sigma_{i}^{2})$  is the posterior marginal of  $f_{i} = f(\mathbf{x}_{i})$  with  $\mathbf{a}_i\coloneqq \mathbf{K}_{\mathbf{uu}}^{-1}\mathbf{k}_{\mathbf{ui}}$  and  $\sigma_i^2 \coloneqq \kappa_{ii} - \mathbf{a}_i^\top \mathbf{K}_{\mathbf{uu}}\mathbf{a}_i$  as the noise variance of  $f_{i}$  conditioned on  $\mathbf{u}$ . The objective is also used to optimize hyperparameters  $\phi$  and inducing input set  $\mathcal{Z}$ .
We can optimize the objective using the BLR for which the resulting update is identical to the variational online-newton (VON) algorithm. We first write the natural gradients,
$$
\widetilde {\nabla} \mathbb {E} _ {q _ {t} (f _ {i})} [ - \log p (y _ {i} | f _ {i}) ] = \left(\left(e _ {i t} - \beta_ {i t} \mathbf {a} _ {i} ^ {\top} \mathbf {m} _ {*}\right) \mathbf {a} _ {i}, \frac {1}{2} \beta_ {i t} \mathbf {a} _ {i} \mathbf {a} _ {i} ^ {\top}\right). \tag {38}
$$
$$
e _ {i t} = \mathbb {E} _ {q _ {t} (f _ {i})} [ - \nabla_ {f _ {i}} \log p (y _ {i} | f _ {i}) ], \beta_ {i t} = \mathbb {E} _ {q _ {t} (f _ {i})} [ - \nabla_ {f _ {i}} ^ {2} \log p (y _ {i} | f _ {i}) ]
$$
We define  $\mathbf{A}$  to be a matrix with  $\mathbf{a}_i^\top$  as rows, and  $\mathbf{e}_t, \beta_t$  to be vectors of  $e_{it}, \beta_{it}$ . Using these in the VON update, we simplify as follows:
$$
\mathbf {S} _ {t + 1} = (1 - \rho) \mathbf {S} _ {t} + \rho \left[ \mathbf {A} ^ {\top} \operatorname {d i a g} \left(\boldsymbol {\beta} _ {t}\right) \mathbf {A} + \mathbf {K} _ {\mathbf {u u}} ^ {- 1} \right] \tag {39}
$$
$$
\begin{array}{l} \mathbf {m} _ {t + 1} = \mathbf {S} _ {t + 1} ^ {- 1} \left[ (1 - \rho) \mathbf {S} _ {t} \mathbf {m} _ {t} - \rho \left(\mathbf {A} ^ {\top} \mathbf {e} _ {t} - \mathbf {A} ^ {\top} \operatorname {d i a g} (\boldsymbol {\beta} _ {t}) \mathbf {A} \mathbf {m} _ {t}\right) \right] \\ = \mathbf {S} _ {t + 1} ^ {- 1} \left[ \left((1 - \rho) \mathbf {S} _ {t} + \rho \mathbf {A} ^ {\top} \operatorname {d i a g} (\boldsymbol {\beta} _ {t}) \mathbf {A}\right) \mathbf {m} _ {t} - \rho \mathbf {A} ^ {\top} \mathbf {e} _ {t} \right] \\ = \mathbf {S} _ {t + 1} ^ {- 1} \left[ \left(\mathbf {S} _ {t + 1} - \rho \mathbf {K} _ {\mathbf {u u}} ^ {- 1}\right) \mathbf {m} _ {t} - \rho \mathbf {A} ^ {\top} \mathbf {e} _ {t} \right] \tag {40} \\ = \mathbf {S} _ {t + 1} ^ {- 1} \left[ \mathbf {S} _ {t + 1} \mathbf {m} _ {t} - \rho \left(\mathbf {A} ^ {\top} \mathbf {e} _ {t} + \mathbf {K} _ {\mathbf {u u}} ^ {- 1} \mathbf {m} _ {t}\right) \right] \\ = \mathbf {m} _ {t} - \rho \mathbf {S} _ {t + 1} ^ {- 1} \left[ \mathbf {A} ^ {\top} \mathbf {e} _ {t} + \mathbf {K} _ {\mathbf {u u}} ^ {- 1} \mathbf {m} _ {t} \right]. \\ \end{array}
$$
For Gaussian likelihood, the updates in Eqs. 39 and 40 coincide with the method of [18], and for non-Gaussian likelihood they are similar to the natural-gradient method by [45], but we use the specific parameterization of [26]. An alternate update rule in terms of site parameters is given by [1] (see Eqs. 22-24).
We are now ready to write the sensitivity measure essentially substituting the gradient in Eq. 10),
$$
\mathbf {S} _ {t} ^ {- 1} \nabla_ {\mathbf {m}} \mathbb {E} _ {q _ {t} (\mathbf {u})} [ - \log p (y _ {i} | f _ {i}) ] = \mathbf {S} _ {t} ^ {- 1} \mathbf {a} _ {i} \mathbb {E} _ {q _ {t} (f _ {i})} [ - \nabla \log p (y _ {i} | f _ {i}) ] = \mathbf {S} _ {t} ^ {- 1} \mathbf {a} _ {i} e _ {i t} \tag {41}
$$
We can also see the bi-linear relationship by considering the deviation in the mean of the posterior marginal  $f_{i}(\mathbf{m}) \coloneqq \mathbf{a}_{i}^{\top}\mathbf{m}$ ,
$$
f _ {i} \left(\mathbf {m} _ {t} ^ {\backslash i}\right) - f _ {i} \left(\mathbf {m} _ {t}\right) \approx \mathbf {a} _ {i} ^ {\top} \left(\hat {\mathbf {m}} _ {t} ^ {\backslash i} - \mathbf {m} _ {t}\right) = \mathbf {a} _ {i} ^ {\top} \boldsymbol {\Sigma} _ {t} \mathbf {a} _ {i} e _ {i t} = v _ {i t} e _ {i t} \tag {42}
$$
where  $v_{it} = \mathbf{a}_i^\top \pmb{\Sigma}_t\mathbf{a}_i$  is the marginal variance of  $f_i$ .
I.1 Neural network architectures
Below, we describe different neural networks used in our experiments,
MLP (500, 300): This is a multilayer perceptron (MLP) with two hidden layers of 500 and 300 neurons and a parameter count of around 546000 (using hyperbolic-tangent activations).
MLP (32, 16): This is also an MLP with two hidden layers of 32 and 16 neurons, which accounts for around 26000 parameters (also using hyperbolic tangent activations).
LeNet5: This is a standard convolutional neural network (CNN) architecture with three convolution layers followed by two fully-connected layers, corresponding to around 62000 parameters.
CNN: This network, taken from the DeepOBS suite [46], consists of three convolution layers followed by three fully-connected layers with a parameter count of 895000.
ResNet-20: This network has around 274000 parameters. We use filter response normalization (FRN) [48] as an alternative to batch normalization.
MLP for USPS: For the experiment on binary USPS in Fig. 6(a), we use an MLP with three hidden layers of 30 neurons each and a total of around 10000 parameters.
I.2 Details of "Do estimated deviations correlate with the truth?"
In Fig. 2, we train neural network classifiers with a cross-entropy loss to obtain  $\theta_{*}$ . Due to the computational demand of per-example retraining, the removed examples are randomly subsampled from the training set. We show results over 1000 examples for MNIST and FMNIST and 100 examples for CIFAR10. In the multiclass setting, the expression yields a per-class sensitivity value. We obtain a scalar value for each example by summing over the absolute values of the per-class sensitivities. For training both the original model  $\theta_{*}$  and the perturbed models  $\theta_{*}^{i}$ , we use SGD with a momentum parameter of 0.9 and a cosine learning-rate scheduler. To obtain  $\theta_{*}^{i}$ , we retrain a model that is warmstarted at  $\theta_{*}$ . Other details regarding the training setup are given in Table 2. For all models, we do not use data augmentation during training. The resulting  $\theta_{*}$  for MNIST, FMNIST, and CIFAR10 have training accuracies of  $99.9\%$ ,  $95.0\%$ , and  $99.9\%$ , respectively. The test accuracies for these models are  $98.4\%$ ,  $91.2\%$  and  $76.7\%$ .
Dataset	Model	B	δ	E*	LR*	LR*min	E\i	LR\i	LR\min
MNIST	MLP (500, 300)	256	100	500	10-2	10-3	300	10-3	10-4
FMNIST	LeNet5	256	100	300	10-1	10-3	200	10-3	10-4
CIFAR10	CNN	512	250	500	10-2	10-4	300	10-4	10-6
Table 2: Hyperparameters for predicting true sensitivity in Fig. 2.  $B$ ,  $E$  and  $LR$  denote batch size, training epochs and learning-rates, respectively. The superscripts * and  $\backslash i$  indicate hyperparameters for training on all data and warmstarted leave-one-out retraining, respectively.  $LR_{\mathrm{min}}$  is the minimum learning-rate of the cosine scheduler.
Additional group removal experiments: We also study how the deviation for removing a group of examples in a set  $\mathcal{M}$  can be estimated using a variation of Eq. 14 for the deviation in predictions at convergence. Denoting the vector of  $f_{i}(\pmb{\theta})$  for  $i \in \mathcal{M}$  by  $\mathbf{f}_{\mathcal{M}}(\pmb{\theta})$ , we get
$$
\sigma \left( \right.\mathbf {f} _ {\mathcal {M}} \left(\boldsymbol {\theta} _ {*} ^ {\backslash \mathcal {M}}\right) - \sigma \left(\mathbf {f} _ {\mathcal {M}} \left(\boldsymbol {\theta} _ {*}\right)\right) \approx \boldsymbol {\Lambda} \left(\boldsymbol {\theta} _ {*}\right) \mathbf {V} _ {\mathcal {M}} \left(\boldsymbol {\theta} _ {*}\right) \mathbf {e} _ {\mathcal {M}} \left(\boldsymbol {\theta} _ {*}\right) \approx \sum_ {i \in \mathcal {M}} \sigma^ {\prime} \left(f _ {i *}\right) v _ {i *} e _ {i *}. \tag {43}
$$
where  $\Lambda (\pmb{\theta}_{*})$  is a diagonal matrix containing all  $\sigma^{\prime}(f_{i*})$ ,  $\mathbf{V}_{\mathcal{M}}(\pmb{\theta}_{*}) = \nabla \mathbf{f}_{\mathcal{M}}(\pmb{\theta}_{*})\mathbf{S}_{*}^{-1}\nabla \mathbf{f}_{\mathcal{M}}(\pmb{\theta}_{*})^{\top}$  is the prediction covariance of size  $M\times M$  where  $M$  is the number of examples in  $\mathcal{M}$ , and  $\mathbf{e}_{\mathcal{M}}(\pmb{\theta}_{*})$  is the vector of prediction errors. The last approximation above is done to avoid building the covariance, where we ignore the off-diagonal entries of  $\mathbf{V}_{\mathcal{M}}(\pmb{\theta}_{*})$ .