SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Vincent Adam, Paul Chang, Mohammad Emtiyaz Khan, and Arno Solin. Dual Parameterization of Sparse Variational Gaussian Processes. Advances in Neural Information Processing Systems, 2021. 20
[2] Chirag Agarwal, Daniel D'Souza, and Sara Hooker. Estimating Example Difficulty using Variance of Gradients. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022. 3
[3] Devansh Arpit, Stanisław Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A Closer Look at Memorization in Deep Networks. In International Conference on Machine Learning, 2017. 3, 6
[4] Gregor Bachmann, Thomas Hofmann, and Aurélien Lucchi. Generalization Through The Lens of Leave-One-Out Error. In International Conference on Learning Representations, 2022. 2, 7
[5] Samyadeep Basu, Phil Pope, and Soheil Feizi. Influence Functions in Deep Learning Are Fragile. In International Conference on Learning Representations, 2021. 6
[6] Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006. 5, 7
[7] R Dennis Cook. Detection of Influential Observation in Linear Regression. Technometrics, 1977. 1, 3, 5
[8] R Dennis Cook and Sanford Weisberg. Characterizations of an Empirical Influence Function for Detecting Influential Cases in Regression. Technometrics, 1980. 1, 2, 3
[9] R Dennis Cook and Sanford Weisberg. *Residuals and Influence in Regression*. Chapman and Hall, 1982. 14
[10] Corinna Cortes and Vladimir Vapnik. Support-Vector Networks. Machine learning, 1995. 4
[11] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace Redux-Effortless Bayesian Deep Learning. Advances in Neural Information Processing Systems, 2021. 8
[12] Gintare Karolina Dziugaite and Daniel M Roy. Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters Than Training Data. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, 2017. 2, 7
[13] Vitaly Feldman and Chiyuan Zhang. What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation. In Advances in Neural Information Processing Systems, 2020. 3
[14] Wing K Fung and CW Kwan. A Note on Local Influence Based on Normal Curvature. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 1997. 1, 3
[15] Ryan Giordano, Tamara Broderick, and Michael I Jordan. Covariances, robustness and variational Bayes. Journal of Machine Learning Research, 19(51), 2018. 5
[16] Satoshi Hara, Atsushi Nitanda, and Takanori Maehara. Data Cleansing for Models Trained with SGD. In Advances in Neural Information Processing Systems, 2019. 3
[17] Hrayr Harutyunyan, Alessandro Achille, Giovanni Paolini, Orchid Majumder, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Estimating Informativeness of Samples with Smooth Unique Information. In International Conference on Learning Representations, 2021. 3
A Influence Function for Linear Regression
We consider  $N$  input-output pairs  $(\mathbf{x}_i, y_i)$ . The feature matrix containing  $\mathbf{x}_i^\top$  as rows is denoted by  $\mathbf{X}$  and the output vector of length  $N$  is denoted by  $\mathbf{y}$ . The loss is  $\ell_i(\boldsymbol{\theta}) = \frac{1}{2} (y_i - f_i(\boldsymbol{\theta}))^2$  for  $f_i(\boldsymbol{\theta}) = \mathbf{x}_i^\top \boldsymbol{\theta}$ . The regularizer is assumed to be  $\mathcal{R}(\boldsymbol{\theta}) = \delta \| \boldsymbol{\theta} \|^2 / 2$ . The minimizer is given by
$$
\boldsymbol {\theta} _ {*} = \mathbf {H} _ {*} ^ {- 1} \mathbf {X} ^ {\top} \mathbf {y}. \tag {17}
$$
We define a perturbation model as follows with  $\epsilon_{i}\in \mathbb{R}$
$$
\boldsymbol {\theta} _ {*} ^ {\epsilon_ {i}} = \underset {\boldsymbol {\theta}} {\arg \min } \mathcal {L} (\boldsymbol {\theta}) - \epsilon_ {i} \ell_ {i} (\boldsymbol {\theta}).
$$
For  $\epsilon_{i} = 1$ , it corresponds to example removal. An arbitrary  $\epsilon_{i}$  simply weights the example accordingly. The solution has a closed-form expression,
$$
\boldsymbol {\theta} _ {*} ^ {\epsilon_ {i}} = \left(\mathbf {H} _ {*} - \epsilon_ {i} \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top}\right) ^ {- 1} \left(\mathbf {X} ^ {\top} \mathbf {y} - \epsilon_ {i} \mathbf {x} _ {i} y _ {i}\right). \tag {18}
$$
where  $\mathbf{H}_{*} = \sum_{i=1}^{N} \mathbf{x}_{i} \mathbf{x}_{i}^{\top} + \delta \mathbf{I}_{P}$  is the Hessian of  $\mathcal{L}(\boldsymbol{\theta})$ . We first derive a closed-form expressions for  $\pmb{\theta}_{*}^{\epsilon_{i}} - \pmb{\theta}_{*}$ , and then specialize them for different  $\epsilon_{i}$ .
A.1 Derivation of the leave-one-out (LOO) deviation
We denote  $\boldsymbol{\Sigma}_{*} = \mathbf{H}_{*}^{-1}$  and use the Sherman-Morrison formula to write
$$
\begin{array}{l} \boldsymbol {\theta} _ {*} ^ {\epsilon_ {i}} = \left(\boldsymbol {\Sigma} _ {*} + \frac {\epsilon_ {i} \boldsymbol {\Sigma} _ {*} \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top} \boldsymbol {\Sigma} _ {*}}{1 - \epsilon_ {i} \mathbf {x} _ {i} ^ {\top} \boldsymbol {\Sigma} _ {*} \mathbf {x} _ {i}}\right) \left(\mathbf {X} ^ {\top} \mathbf {y} - \epsilon_ {i} y _ {i} \mathbf {x} _ {i}\right) \\ = \boldsymbol {\Sigma} _ {*} \mathbf {X} ^ {\top} \mathbf {y} + \epsilon_ {i} \boldsymbol {\Sigma} _ {*} \mathbf {x} _ {i} \left[ \frac {\mathbf {x} _ {i} ^ {\top} \boldsymbol {\Sigma} _ {*} \mathbf {X} ^ {\top} \mathbf {y}}{1 - \epsilon_ {i} \mathbf {x} _ {i} ^ {\top} \boldsymbol {\Sigma} _ {*} \mathbf {x} _ {i}} - \frac {\epsilon_ {i} y _ {i} \mathbf {x} _ {i} ^ {\top} \boldsymbol {\Sigma} _ {*} \mathbf {x} _ {i}}{1 - \epsilon_ {i} \mathbf {x} _ {i} ^ {\top} \boldsymbol {\Sigma} _ {*} \mathbf {x} _ {i}} - y _ {i} \right] \tag {19} \\ = \boldsymbol {\theta} _ {*} + \epsilon_ {i} \boldsymbol {\Sigma} _ {*} \mathbf {x} _ {i} \left[ \frac {\mathbf {x} _ {i} ^ {\top} \boldsymbol {\theta} _ {*}}{1 - \epsilon_ {i} v _ {i}} - \frac {\epsilon_ {i} y _ {i} v _ {i}}{1 - \epsilon_ {i} v _ {i}} - y _ {i} \right] \\ = \pmb {\theta} _ {*} + \epsilon_ {i} \pmb {\Sigma} _ {*} \mathbf {x} _ {i} \left[ \frac {\mathbf {x} _ {i} ^ {\top} \pmb {\theta} _ {*} - y _ {i}}{1 - \epsilon_ {i} v _ {i}} \right] = \pmb {\theta} _ {*} + \pmb {\Sigma} _ {*} \mathbf {x} _ {i} \frac {\epsilon_ {i} e _ {i}}{1 - \epsilon_ {i} v _ {i}}. \\ \end{array}
$$
In line 3 we substitute  $v_{i} = \mathbf{x}_{i}^{\top}\pmb{\Sigma}_{*}\mathbf{x}_{i}$  and  $\pmb{\theta}_{*} = \pmb{\Sigma}_{*}\mathbf{X}^{\top}\mathbf{y}$  and in the last step we use  $e_i = \mathbf{x}_i^\top \pmb{\theta}_* - y_i$ . We define  $e_i^{\backslash i} = e_i / (1 - v_i)$  which is the prediction error of  $\pmb{\theta}_{*}^{\backslash i}$ ,
$$
e _ {i} ^ {\backslash i} = \mathbf {x} _ {i} ^ {\top} \boldsymbol {\theta} _ {*} ^ {\backslash i} - y _ {i} = \mathbf {x} _ {i} ^ {\top} \left(\boldsymbol {\theta} _ {*} + \boldsymbol {\Sigma} _ {*} \mathbf {x} _ {i} \frac {e _ {i}}{1 - v _ {i}}\right) - y _ {i} = \mathbf {x} _ {i} ^ {\top} \boldsymbol {\theta} _ {*} + \frac {v _ {i}}{1 - v _ {i}} e _ {i} - y _ {i} = \frac {e _ {i}}{1 - v _ {i}}. \tag {20}
$$
Therefore, we get the following expressions for the deviation,
$$
\boldsymbol {\theta} _ {*} ^ {\backslash i} - \boldsymbol {\theta} _ {*} = \boldsymbol {\Sigma} _ {*} \mathbf {x} _ {i} e _ {i} ^ {\backslash i}, \qquad f _ {i} (\boldsymbol {\theta} _ {*} ^ {\backslash i}) - f _ {i} (\boldsymbol {\theta} _ {*}) = v _ {i} e _ {i} ^ {\backslash i}.
$$
These expressions can be written in the form of Eq. 2 by left-multiplying with  $\mathbf{\Sigma}_{*}^{-1} = \mathbf{H}_{*}^{\backslash i} + \mathbf{x}_{i}\mathbf{x}_{i}^{\top}$
$$
(\mathbf {H} _ {*} ^ {\backslash i} + \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top}) (\pmb {\theta} _ {*} ^ {\backslash i} - \pmb {\theta} _ {*}) = \mathbf {x} _ {i} (\mathbf {x} _ {i} ^ {\top} \pmb {\theta} _ {*} ^ {\backslash i} - y _ {i}) \Rightarrow \pmb {\theta} _ {*} ^ {\backslash i} - \pmb {\theta} _ {*} = (\mathbf {H} _ {*} ^ {\backslash i}) ^ {- 1} \mathbf {x} _ {i} e _ {i}.
$$
A.2 Derivation of the infinitesimal perturbation approach
We differentiate  $\theta_{*}^{\epsilon_{i}}$  in Eq. 19 to get
$$
\frac {\partial \boldsymbol {\theta} _ {*} ^ {\epsilon_ {i}}}{\partial \epsilon_ {i}} = \boldsymbol {\Sigma} _ {*} \mathbf {x} _ {i} \frac {e _ {i}}{\left(1 - \epsilon_ {i} v _ {i}\right) ^ {2}}, \tag {21}
$$
yielding the following expressions:
$$
\left. \frac {\partial \boldsymbol {\theta} _ {*} ^ {\epsilon_ {i}}}{\partial \epsilon_ {i}} \right| _ {\epsilon_ {i} = 0} = \boldsymbol {\Sigma} _ {*} \mathbf {x} _ {i} e _ {i}, \quad \left. \frac {\partial f _ {i} \left(\boldsymbol {\theta} _ {*} ^ {\epsilon_ {i}}\right)}{\partial \epsilon_ {i}} \right| _ {\epsilon_ {i} = 0} = \mathbf {x} _ {i} ^ {\top} \left. \frac {\partial \boldsymbol {\theta} _ {*} ^ {\epsilon_ {i}}}{\partial \epsilon_ {i}} \right| _ {\epsilon_ {i} = 0} = \mathbf {x} _ {i} ^ {\top} \boldsymbol {\Sigma} _ {*} \mathbf {x} _ {i} e _ {i} = v _ {i} e _ {i}. \tag {22}
$$
The second equation in Eq. 22 follows from the chain rule. We get a bi-linear relationship of the influence measure with respect to  $v_{i}$  and prediction error  $e_{i}$ . It is also possible to evaluate Eq. 21 at  $\epsilon_{i} = 1$  representing an infinitesimal perturbation about the LOO estimate,  $\left.\partial\boldsymbol{\theta}_{*}^{\epsilon_{i}} / \partial\epsilon_{i}\right|_{\epsilon_{i}=1}=\boldsymbol{\Sigma}_{*}^{\backslash i}\mathbf{x}_{i}e_{i}^{\backslash i}$ . From this and Eq. 22, we can interpret Eq. 2 as the average derivative over the interval  $\epsilon_{i}\in [0,1]$  [9] or the derivative evaluated at some  $0 < \epsilon_{i} < 1$  (via an application of the mean value theorem) [41].
B Conjugate Exponential-Family Models
Exponential-family distributions take the following form:
$$
q = h (\boldsymbol {\theta}) \exp [ \langle \boldsymbol {\lambda}, \mathbf {T} (\boldsymbol {\theta}) \rangle - A (\boldsymbol {\lambda}) ].
$$
where  $\lambda \in \Omega$  are the natural (or canonical) parameter for which the cumulant (or log partition) function  $A(\pmb{\lambda})$  is finite, strictly convex and differentiable over  $\Omega$ . The quantity  $\mathbf{T}(\pmb{\theta})$  is the sufficient statistics,  $\langle \cdot, \cdot \rangle$  is an inner product and  $h(\pmb{\theta})$  is some function. A popular example is the Gaussian distribution, which can be rearranged to take an exponential-family form written in terms of the precision matrix  $\mathbf{S} = \pmb{\Sigma}^{-1}$ ,
$$
\begin{array}{l} \mathcal {N} (\pmb {\theta} | \mathbf {m}, \pmb {\Sigma}) = | 2 \pi \pmb {\Sigma} | ^ {- \frac {1}{2}} \exp \left[ - \frac {1}{2} (\pmb {\theta} - \mathbf {m}) ^ {\top} \pmb {\Sigma} ^ {- 1} (\pmb {\theta} - \mathbf {m}) \right] \\ = \exp \left[ \boldsymbol {\theta} ^ {\top} \mathbf {S} \mathbf {m} - \frac {1}{2} \boldsymbol {\theta} ^ {\top} \mathbf {S} \boldsymbol {\theta} - \frac {1}{2} \left(\mathbf {m} ^ {\top} \mathbf {S} \mathbf {m} + \log | 2 \pi \mathbf {S} ^ {- 1} |\right) \right]. \\ \end{array}
$$
From this, we can read-off the quantities needed to define an exponential-form,
$$
\boldsymbol {\lambda} = \left(\mathbf {S} \mathbf {m}, - \frac {1}{2} \mathbf {S}\right), \quad \mathbf {T} (\boldsymbol {\theta}) = \left(\boldsymbol {\theta}, \boldsymbol {\theta} \boldsymbol {\theta} ^ {\top}\right), \quad A (\boldsymbol {\lambda}) = \frac {1}{2} \left(\mathbf {m} ^ {\top} \mathbf {S} \mathbf {m} + \log | 2 \pi \mathbf {S} ^ {- 1} |\right), \quad h (\boldsymbol {\theta}) = 1. \tag {23}
$$
Both the natural parameter and sufficient statistics consist of two elements. The inner-product for the first element is simply a transpose to get the  $\pmb{\theta}^{\top}\mathbf{S}\mathbf{m}$  term, while for the second element it is a trace which gives  $-\mathrm{Tr}(\pmb {\theta}\pmb{\theta}^{\top}\mathbf{S} / 2) = -\frac{1}{2}\pmb{\theta}^{\top}\mathbf{S}\pmb{\theta}$ .
Conjugate Exponential-Family Models are those where both the likelihoods and prior can be expressed in terms of the same form of exponential-family distribution with respect to  $\theta$ . For instance, in linear regression, both the likelihood and prior take a Gaussian form with respect to  $\theta$ ,
$$
\begin{array}{l} \tilde {p} _ {i} = p \left(y _ {i} \mid \mathbf {x} _ {i}, \boldsymbol {\theta}\right) = \mathcal {N} \left(y _ {i} \mid \mathbf {x} _ {i} ^ {\top} \boldsymbol {\theta}, 1\right) \propto \exp \left[ \boldsymbol {\theta} ^ {\top} \mathbf {x} _ {i} y _ {i} - \frac {1}{2} \boldsymbol {\theta} ^ {\top} \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top} \boldsymbol {\theta} \right] \\ p _ {0} = p (\pmb {\theta}) = \mathcal {N} (\pmb {\theta} | 0, \mathbf {I} / \delta) \propto \exp \left[ - \frac {1}{2} \pmb {\theta} ^ {\top} (\delta \mathbf {I}) \pmb {\theta} \right]. \\ \end{array}
$$
Note that  $\tilde{p}_i$  is a distribution over  $y_i$  but it can also be expressed in an (unnormized) Gaussian form with respect to  $\theta$ . The sufficient statistics of both  $\tilde{p}_i$  and  $p_0$  correspond to those of a Gaussian distribution. Therefore, the posterior is also a Gaussian,
$$
\begin{array}{l} q _ {*} = p (\boldsymbol {\theta} | \mathcal {D}) \propto p _ {0} \tilde {p} _ {1} \tilde {p} _ {2} \dots \tilde {p} _ {N} \\ = \exp \left[ - \frac {1}{2} \boldsymbol {\theta} ^ {\top} (\delta \mathbf {I}) \boldsymbol {\theta} \right] \prod_ {i = 1} ^ {N} \exp \left[ \boldsymbol {\theta} ^ {\top} \mathbf {x} _ {i} y _ {i} - \frac {1}{2} \boldsymbol {\theta} ^ {\top} \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top} \boldsymbol {\theta} \right] \\ = \exp \left[ \boldsymbol {\theta} ^ {\top} \sum_ {i = 1} ^ {N} \mathbf {x} _ {i} y _ {i} - \frac {1}{2} \boldsymbol {\theta} ^ {\top} \left(\delta \mathbf {I} + \sum_ {i = 1} ^ {N} \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top}\right) \boldsymbol {\theta} \right] \\ = \exp \left[ \boldsymbol {\theta} ^ {\top} \mathbf {H} _ {*} \boldsymbol {\theta} _ {*} - \frac {1}{2} \boldsymbol {\theta} ^ {\top} \mathbf {H} _ {*} \boldsymbol {\theta} \right] \\ \propto \mathcal {N} (\boldsymbol {\theta} | \boldsymbol {\theta} _ {*}, \mathbf {H} _ {*} ^ {- 1}). \\ \end{array}
$$
The third line follows because  $\pmb{\theta}_{*} = \mathbf{H}_{*}^{-1}\mathbf{X}^{\top}\mathbf{y}$ , as shown in Eq. 17.
These computations can be written as conjugate-computations [26] where we simply add the natural parameters,
$$
\begin{array}{l} \tilde {p} _ {i} \propto \exp \left[ \langle \widetilde {\boldsymbol {\lambda}} _ {i}, \mathbf {T} (\boldsymbol {\theta}) \rangle \right], \text {w h e r e} \widetilde {\boldsymbol {\lambda}} _ {i} = \left(\mathbf {x} _ {i} y _ {i}, - \frac {1}{2} \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top}\right) \\ p _ {0} \propto \exp \left[ \langle \boldsymbol {\lambda} _ {0}, \mathbf {T} (\boldsymbol {\theta}) \rangle \right], \text {w h e r e} \boldsymbol {\lambda} _ {0} = (0, - \frac {1}{2} \delta \mathbf {I}) \\ \Rightarrow q _ {*} \propto \exp \left[ \left\langle \boldsymbol {\lambda} _ {*}, \mathbf {T} (\boldsymbol {\theta}) \right\rangle \right], \text {w h e r e} \boldsymbol {\lambda} _ {*} = \boldsymbol {\lambda} _ {0} + \sum_ {i = 1} ^ {N} \widetilde {\boldsymbol {\lambda}} _ {i} = \left(\mathbf {H} _ {*} \boldsymbol {\theta} _ {*}, - \frac {1}{2} \mathbf {H} _ {*}\right). \\ \end{array}
$$
In the same fashion, to remove the contributions of certain likelihoods, we can simply subtract their natural parameters from  $\lambda_{*}$ . These are the calculations which give rise to the following equation:
$$
q _ {*} ^ {\backslash \mathcal {M}} \propto \frac {q _ {*}}{\prod_ {j \in \mathcal {M}} \tilde {p} _ {j}} \quad \Longrightarrow e ^ {\langle \mathbf {T} (\boldsymbol {\theta}), \boldsymbol {\lambda} _ {*} ^ {\backslash \mathcal {M}} \rangle} \propto \frac {e ^ {\langle \mathbf {T} (\boldsymbol {\theta}) , \boldsymbol {\lambda} _ {*} \rangle}}{\prod_ {j \in \mathcal {M}} e ^ {\langle \mathbf {T} (\boldsymbol {\theta}) , \widetilde {\boldsymbol {\lambda}} _ {j} \rangle}} \quad \Longrightarrow \boldsymbol {\lambda} _ {*} ^ {\backslash \mathcal {M}} = \boldsymbol {\lambda} _ {*} - \sum_ {j \in \mathcal {M}} \widetilde {\boldsymbol {\lambda}} _ {j}.
$$
C The Bayesian Learning Rule
The Bayesian learning rule (BLR) aims to find a posterior approximation  $q(\pmb{\theta}) \approx p(\pmb{\theta}|\mathcal{D}) \propto e^{-\mathcal{L}(\pmb{\theta})}$ . Often, one considers a regular, minimal exponential-family  $q \in \mathcal{Q}$ , for example, the class of Gaussian distributions. The approximation is found by optimizing a generalized Bayesian objective,
$$
q_{*} = \operatorname *{arg  min}_{q\in \mathcal{Q}}\mathbb{E}_{q}\left[\mathcal{L}(\boldsymbol {\theta})\right] - \mathcal{H}(q).
$$
where  $\mathcal{H}(q) = \mathbb{E}_q[-\log q(\pmb{\theta})]$  is the entropy of  $q$  and  $\mathcal{Q}$  is the class of exponential family approximation. The objective is equivalent to the Evidence Lower Bound (ELBO) when  $\mathcal{L}(\pmb{\theta})$  corresponds to the negative log-joint probability of a Bayesian model; see [28, Sec 1.2].
The BLR uses natural-gradient descent to find  $q_{*}$ , where each iteration  $t$  takes the following form,
$$
\boldsymbol {\lambda} _ {t} \leftarrow \boldsymbol {\lambda} _ {t - 1} - \rho \mathbf {F} \left(\boldsymbol {\lambda} _ {t - 1}\right) ^ {- 1} \frac {\partial}{\partial \boldsymbol {\lambda}} \left[ \mathbb {E} _ {q} [ \mathcal {L} (\boldsymbol {\theta}) ] - \mathcal {H} (q) \right] \Bigg | _ {\boldsymbol {\lambda} = \boldsymbol {\lambda} _ {t - 1}} \tag {24}
$$
where  $\rho > 0$  is the learning rate. The gradient is computed with respect to  $\lambda$  (through  $q$ ), and we scale the gradient by the Fisher Information Matrix (FIM) defined as follows,
$$
\mathbf {F} (\boldsymbol {\lambda}) = \mathbb {E} _ {q} \left[ \left(\nabla_ {\boldsymbol {\lambda}} \log q\right) \left(\nabla_ {\boldsymbol {\lambda}} \log q\right) ^ {\top} \right] = \nabla_ {\boldsymbol {\lambda}} ^ {2} A (\boldsymbol {\lambda}).
$$
The second equality shows that, for exponential-family distribution, the above FIM is also the second derivative of the log-partition function  $A(\lambda)$ .
C.1 The BLR of Eq. 5
The BLR in Eq. 5 is obtained by simplifying the natural-gradient using the following identity,
$$
\mathbf {F} (\boldsymbol {\lambda}) ^ {- 1} \nabla_ {\boldsymbol {\lambda}} \mathbb {E} _ {q} (\cdot) = \left. \nabla_ {\boldsymbol {\mu}} \mathbb {E} _ {q} (\cdot) \right| _ {\boldsymbol {\mu} = \nabla_ {\boldsymbol {\lambda}} A (\boldsymbol {\lambda})} \tag {25}
$$
where  $\mu$  is the expectation parameter. The identity works because of the minimality of the exponential-family which ensures that there is a one-to-one mapping between  $\lambda$  and  $\mu$ , and also that the FIM is invertible. Using this, we can show that the natural gradient of  $\mathcal{H}(q)$  is simply equal to  $-\lambda$ ; see [28, App. B]. Defining  $\ell_0(\pmb{\theta}) = \mathcal{R}(\pmb{\theta})$ , we get the version of the BLR shown in Eq. 5,
$$
\boldsymbol {\lambda} _ {t} \leftarrow (1 - \rho) \boldsymbol {\lambda} _ {t - 1} - \rho \sum_ {j = 0} ^ {N} \tilde {\mathbf {g}} _ {j} (\boldsymbol {\lambda} _ {t - 1}), \text {w h e r e} \tilde {\mathbf {g}} _ {j} (\boldsymbol {\lambda} _ {t - 1}) = \left. \nabla_ {\boldsymbol {\mu}} \mathbb {E} _ {q} [ \ell_ {j} (\boldsymbol {\theta}) ] \right| _ {\boldsymbol {\mu} = \nabla_ {\boldsymbol {\lambda}} A (\boldsymbol {\lambda} _ {t - 1})}.
$$
C.2 The conjugate-model form of the BLR given in Eq. 5
To express the update in terms of the posterior of a conjugate model, we simply take the inner product with  $\mathbf{T}(\theta)$  and take the exponential to write the update as
$$
\underbrace {e ^ {\langle \boldsymbol {\lambda} _ {t} , \mathbf {T} (\boldsymbol {\theta}) \rangle}} _ {\propto q _ {t}} \leftarrow \left(\underbrace {e ^ {\langle \boldsymbol {\lambda} _ {t - 1} , \mathbf {T} (\boldsymbol {\theta}) \rangle}} _ {\propto q _ {t - 1}}\right) ^ {1 - \rho} \left(\underbrace {e ^ {\langle - \tilde {\mathbf {g}} _ {0} (\boldsymbol {\lambda} _ {t - 1}) , \mathbf {T} (\boldsymbol {\theta}) \rangle}} _ {\propto p _ {0}}\right) ^ {\rho} \prod_ {j = 1} ^ {N} e ^ {\langle - \rho \tilde {\mathbf {g}} _ {j} (\boldsymbol {\lambda} _ {t - 1}), \mathbf {T} (\boldsymbol {\theta}) \rangle}, \tag {26}
$$
The simplification of the second term on the left to  $p_0$  happens when  $p_0$  is a conjugate prior, that is,  $p_0 \propto \exp(\langle \pmb{\lambda}_0, \mathbf{T}(\pmb{\theta}) \rangle)$  for some  $\pmb{\lambda}_0$  (see an example in App. B where we show that  $L_2$  regularizer leads to such a choice). In such cases, we can simplify,
$$
\langle - \tilde {\mathbf {g}} _ {0} (\boldsymbol {\lambda}), \mathbf {T} (\boldsymbol {\theta}) \rangle = \left\langle \nabla_ {\boldsymbol {\mu}} \mathbb {E} _ {q} [ \log p _ {0} ], \mathbf {T} (\boldsymbol {\theta}) \right\rangle = \left\langle \nabla_ {\boldsymbol {\mu}} \left\langle \boldsymbol {\lambda} _ {0}, \boldsymbol {\mu} \right\rangle , \mathbf {T} (\boldsymbol {\theta}) \right\rangle = \left\langle \boldsymbol {\lambda} _ {0}, \mathbf {T} (\boldsymbol {\theta}) \right\rangle = \log p _ {0} + \text {c o n s t .}
$$
Using this in Eq. 26, we recover the conjugate model given in Eq. 5.
C.3 BLR for a Gaussian  $q$  and the Variational Online Newton (VON) algorithm
By choosing an appropriate form for  $q_{t}$  and making necessary approximations to  $\tilde{\mathbf{g}}_j$ , the BLR can recover many popular algorithms as special cases. We will now give a few examples for the case of a Gaussian  $q_{t} = \mathcal{N}(\pmb {\theta}|\mathbf{m}_{t},\pmb{\Sigma}_{t})$  which enables derivation of various first and second-order optimization algorithms, such as, Newton's method, RMSprop, Adam, and SGD.
As shown in Eq. 23, for a Gaussian  $\mathcal{N}(\pmb{\theta}|\mathbf{m},\pmb{\Sigma})$ , the natural parameter and sufficient statistics are shown below, along with the expectation parameters  $\pmb{\mu} = \mathbb{E}_q[\mathbf{T}(\pmb{\theta})]$ .
$$
\boldsymbol {\lambda} = (\mathbf {S m}, - \frac {1}{2} \mathbf {S}), \quad \mathbf {T} (\boldsymbol {\theta}) = (\boldsymbol {\theta}, \boldsymbol {\theta} \boldsymbol {\theta} ^ {\top}), \quad \boldsymbol {\mu} = (\mathbf {m}, \mathbf {m m} ^ {\top} + \boldsymbol {\Sigma}),
$$
Using these, we can write the natural gradients as gradients with respect to  $\pmb{\mu}$ , and then using chain-rule to express them as gradients with respect to  $\mathbf{m}$  and  $\pmb{\Sigma}$ ,
$$
\tilde {\mathbf {g}} _ {j} (\boldsymbol {\lambda}) = \nabla_ {\boldsymbol {\mu}} \mathbb {E} _ {q} [ \ell_ {j} (\boldsymbol {\theta}) ] = \left( \begin{array}{c} \nabla_ {\mathbf {m}} \mathbb {E} _ {q} [ \ell_ {j} (\boldsymbol {\theta}) ] \\ \nabla_ {\mathbf {m m} ^ {\top} + \boldsymbol {\Sigma}} \mathbb {E} _ {q} [ \ell_ {j} (\boldsymbol {\theta}) ] \end{array} \right) = \left( \begin{array}{c} \hat {\mathbf {g}} _ {j} - \hat {\mathbf {H}} _ {j} \mathbf {m} \\ \frac {1}{2} \hat {\mathbf {H}} _ {j}, \end{array} \right), \tag {27}
$$
where in the last equation we define two quantities written in terms of  $\nabla \ell_{j}(\pmb{\theta})$  and  $\nabla^2\ell_j(\pmb{\theta})$  by using Price's and Bonnet's theorem [44],
$$
\hat {\mathbf {g}} _ {j} = \nabla_ {\mathbf {m}} \mathbb {E} _ {q} [ \ell_ {j} (\boldsymbol {\theta}) ] = \mathbb {E} _ {q} [ \nabla \ell_ {j} (\boldsymbol {\theta}) ], \quad \hat {\mathbf {H}} _ {j} = 2 \nabla_ {\boldsymbol {\Sigma}} \mathbb {E} _ {q} [ \ell_ {j} (\boldsymbol {\theta}) ] = \mathbb {E} _ {q} [ \nabla^ {2} \ell_ {j} (\boldsymbol {\theta}) ]. \tag {28}
$$
Plugging these into the BLR update gives us the following update,
$$
\mathbf {S} _ {t} \mathbf {m} _ {t} \gets (1 - \rho) \mathbf {S} _ {t - 1} \mathbf {m} _ {t - 1} + \rho \sum_ {j = 0} ^ {N} \left(\hat {\mathbf {H}} _ {j, t - 1} \mathbf {m} _ {t - 1} - \hat {\mathbf {g}} _ {j, t - 1}\right), \quad \mathbf {S} _ {t} \gets (1 - \rho) \mathbf {S} _ {t - 1} + \rho \sum_ {j = 0} ^ {N} \hat {\mathbf {H}} _ {j, t - 1}
$$
where  $\hat{\mathbf{g}}_{j,t - 1}$  and  $\tilde{\mathbf{H}}_{j,t - 1}$  are quantities similar to before but now evaluated at the  $q_{t - 1}$ . The conjugate model can be written as follows,