SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Michalis K. Titsias
mtitsias@google.com
We define an optimal preconditioning for the Langevin diffusion by analytically optimizing the expected squared jumped distance. This yields as the optimal preconditioning an inverse Fisher information covariance matrix, where the covariance matrix is computed as the outer product of log target gradients averaged under the target. We apply this result to the Metropolis adjusted Langevin algorithm (MALA) and derive a computationally efficient adaptive MCMC scheme that learns the preconditioning from the history of gradients produced as the algorithm runs. We show in several experiments that the proposed algorithm is very robust in high dimensions and significantly outperforms other methods, including a closely related adaptive MALA scheme that learns the preconditioning with standard adaptive MCMC as well as the position-dependent Riemannian manifold MALA sampler.
Markov chain Monte Carlo (MCMC) is a general framework for simulating from arbitrarily complex distributions, and it has shown to be useful for statistical inference in a wide range of problems [18, 11]. The main idea of an MCMC algorithm is quite simple. Given a complex target  $\pi(x)$ , a Markov chain is constructed using a  $\pi$ -invariant transition kernel that allows to simulate dependent realizations  $x_1, x_2, \ldots$  that eventually converge to samples from  $\pi$ . These samples can be used for Monte Carlo integration by forming ergodic averages. A general way to define  $\pi$ -invariant transition kernels is the Metropolis-Hastings accept-reject mechanism in which the chain moves from state  $x_n$  to the next state  $x_{n+1}$  by first generating a candidate state  $y_n$  from a proposal distribution  $q(y_n | x_n)$  and then it sets  $x_{n+1} = y_n$  with probability  $\alpha(x_n, y_n)$ :
$$
\alpha \left(x _ {n}, y _ {n}\right) = \min  \left(1, a _ {n}\right), \quad a _ {n} = \frac {\pi \left(y _ {n}\right)}{\pi \left(x _ {n}\right)} \frac {q \left(x _ {n} \mid y _ {n}\right)}{q \left(y _ {n} \mid x _ {n}\right)}, \tag {1}
$$
or otherwise rejects  $y_{n}$  and sets  $x_{n + 1} = x_{n}$ . The choice of the proposal distribution  $q(y_{n}|x_{n})$  is crucial because it determines the mixing of the chain, i.e. the dependence of samples across time. For example, a "slowly mixing" chain even after convergence may not be useful for Monte Carlo integration since it will output a highly dependent set of samples producing ergodic estimates of very high variance. Different ways of defining  $q(y_{n}|x_{n})$  lead to common algorithms such as random walk Metropolis (RWM), Metropolis-adjusted Langevin algorithm (MALA) [40, 35] and Hamiltonian Monte Carlo (HMC) [15, 28]. Within each class of these algorithms adaptation of parameters of the proposal distribution, such as a step size, is also important and this has been widely studied in the literature by producing optimal scaling results [34, 35, 36, 22, 6, 8, 38, 7, 39, 9], and also by developing adaptive MCMC algorithms [21, 5, 37, 19, 2, 1, 4, 24]. The standard adaptive MCMC procedure in [21] uses the history of the chain to recursively compute an empirical covariance of the target  $\pi$  and build a multivariate Gaussian proposal distribution. However, this type of covariance adaptation can be too slow and not so robust in high dimensional settings [38, 3].
In this paper, we derive a fast and very robust adaptive MCMC technique in high dimensions that learns a preconditioning matrix for the MALA method, which is the standard gradient-based MCMC algorithm obtained by a first-order discretization of the continuous-time Langevin diffusion. Our first contribution is to define an optimal preconditioning by analytically optimizing a criterion on the Langevin diffusion. The criterion is the well-known expected squared jumped distance [30] which at optimum yields as a preconditioner the inverse matrix  $\mathcal{I}^{-1}$  of the following Fisher information covariance matrix  $\mathcal{I} = \mathbb{E}_{\pi(x)}\left[\nabla \log \pi(x)\nabla \log \pi(x)^{\top}\right]$ . This contradicts the common belief in adaptive MCMC that the covariance of  $\pi$  is the best preconditioner. While this is a surprising result we show that  $\mathcal{I}^{-1}$  connects with a certain quantity appearing in optimal scaling of RWM [34, 36].
Having recognized  $\mathcal{I}^{-1}$  as the optimal preconditioning we derive an easy to implement and computationally efficient adaptive MCMC algorithm that learns from the history of gradients produced as MALA runs. This method sequentially updates an empirical inverse Fisher estimate  $\hat{\mathcal{I}}_n^{-1}$  using a recursion having quadratic cost  $O(d^2)$  ( $d$  is the dimension of  $x$ ) per iteration. In practice, since for sampling we need a square root matrix of  $\hat{\mathcal{I}}_n^{-1}$  we implement the recursions over a square root matrix by adopting classical results from Kalman filtering [31, 10]. We compare our method against MALA that learns the preconditioning with standard adaptive MCMC [21], a position-dependent Riemannian manifold MALA [20] as well as simple MALA (without preconditioning) and HMC. In several experiments we show that the proposed algorithm significantly outperforms all other methods.
We consider an intractable target distribution  $\pi(x)$  with  $x \in \mathbb{R}^d$ , known up to some normalizing constant, and we assume that  $\nabla \log \pi(x) \coloneqq \nabla_x \log \pi(x)$  is well defined. A continuous time process with stationary distribution  $\pi$  is the overdamped Langevin diffusion
$$
d x _ {t} = \frac {1}{2} A \nabla \log \pi \left(x _ {t}\right) d t + \sqrt {A} d B _ {t}, \tag {2}
$$
where  $B_{t}$  denotes  $d$ -dimensional Brownian motion. This is a stochastic differential equation (SDE) that generates sample paths such that for large  $t$ ,  $x_{t} \sim \pi$ . We also incorporate a preconditioning matrix  $A$ , which is a symmetric positive definite covariance matrix, while  $\sqrt{A}$  is such that  $\sqrt{A}\sqrt{A}^{\top} = A$ .
Simulating from the SDE in (2) is intractable and the standard approach is to use a first-order Euler-Maruyama discretization combined with a Metropolis-Hastings adjustment. This leads to the so-called preconditioned Metropolis-adjusted Langevin algorithm (MALA) where at each iteration given the current state  $x_{n}$  (where  $n = 1,2,\ldots$ ) we sample  $y_{n}$  from the proposal distribution
$$
q \left(y _ {n} \mid x _ {n}\right) = \mathcal {N} \left(y _ {n} \mid x _ {n} + \frac {\sigma^ {2}}{2} A \nabla \log \pi \left(x _ {n}\right), \sigma^ {2} A\right), \tag {3}
$$
where the step size  $\sigma^2 > 0$  appears due to time discretization. We accept  $y_{n}$  with probability  $\alpha(x_n, y_n) = \min(1, a_n)$  where  $a_n$  follows the form in (1). The obvious way to compute  $a_n$  is
$$
a _ {n} = \frac {\pi (y _ {n})}{\pi (x _ {n})} \frac {q (x _ {n} | y _ {n})}{q (y _ {n} | x _ {n})} = \frac {\pi (y _ {n})}{\pi (x _ {n})} \frac {\exp \{- \frac {1}{2 \sigma^ {2}} | | x _ {n} - y _ {n} - \frac {\sigma^ {2}}{2} A \nabla \log \pi (y _ {n}) | | _ {A ^ {- 1}} ^ {2} \}}{\exp \{- \frac {1}{2 \sigma^ {2}} | | y _ {n} - x _ {n} - \frac {\sigma^ {2}}{2} A \nabla \log \pi (x _ {n}) | | _ {A ^ {- 1}} ^ {2} \}},
$$
where  $||z||_{A^{-1}}^2 = z^\top A^{-1}z$ . However, in some cases that involve high dimensional targets, this can be costly since in the ratio of proposal densities both the preconditioning matrix  $A$  and its inverse  $A^{-1}$  appear. In turns out that we can avoid  $A^{-1}$  and simplify the computation as stated below.
Proposition 1. For preconditioned MALA with proposal density given by (3) the ratio of proposals in the  $M-H$  acceptance probability can be written as
$$
\frac {q (x _ {n} | y _ {n})}{q (y _ {n} | x _ {n})} = \exp \{h (x _ {n}, y _ {n}) - h (y _ {n}, x _ {n}) \}, \quad h (z, v) = \frac {1}{2} \left(z - v - \frac {\sigma^ {2}}{4} A \nabla \log \pi (v)\right) ^ {\top} \nabla \log \pi (v).
$$
This expression does not depend on the inverse  $A^{-1}$ , and this leads to computational gains and simplified implementation that we exploit in the adaptive MCMC algorithm presented in Section 4.
The motivation behind the use of preconditioned MALA is that with a suitable preconditioner  $A$  the mixing of the chain can be drastically improved, especially for very anisotropic target distributions.
A very general way to specify  $A$  is by applying an adaptive MCMC algorithm, which learns  $A$  online. To design such an algorithm it is useful to first specify a notion of optimality. A common argument in the literature, that is used for both RWM and MALA, is that a suitable  $A$  is the unknown covariance matrix  $\Sigma$  [21, 38, 24] of the target  $\pi$ . This means that we should learn  $A$  so that to approximate  $\Sigma$ . However, this argument is rather heuristic since it is not based on an optimality criterion. One of our contributions is to specify an optimal  $A^*$  based on an optimization procedure, that we describe in Section 3. This  $A^*$  will turn out to be not the covariance matrix of the target but an inverse Fisher information matrix.
3 Optimal preconditioning using expected squared jumped distance
Preconditioning aims to improve sampling when different directions (or individual variables  $x_{i}$ ) in the state space can have different scalings under the target  $\pi$ . Here, we develop a method for selecting the preconditioning through the optimization of an objective function. This method uses the observation that an effective preconditioning correlates with large values of the global step size  $\sigma^2$  in MALA, i.e.  $\sigma^2$  is allowed to increase when preconditioning becomes effective as shown in the sampling efficiency scores in Table 1 and the corresponding estimated step sizes reported in Appendix E.1.
In our analysis we consider the rejection-free or unadjusted Langevin sampler where we discretize the time continuous Langevin diffusion in (2) with a small finite  $\delta \coloneqq \sigma^2 > 0$  so that
$$
x _ {t + \delta} - x _ {t} = \frac {\delta}{2} A \nabla \log \pi \left(x _ {t}\right) + \sqrt {A} \left(B _ {t + \delta} - B _ {t}\right), \text {w h e r e} B _ {t + \delta} - B _ {t} \sim \mathcal {N} (0, \delta I). \tag {4}
$$
We will use the expected squared jumped distance  $J(\delta, A) = \mathbb{E}[||x_{t + \delta} - x_t||^2]$  computed as follows.
Proposition 2. If  $x_{t} \sim \pi(x_{t})$  the vector  $x_{t + \delta} - x_{t}$  defined by (4) has zero mean and covariance
$$
\mathbb {E} \left[ \left(x _ {t + \delta} - x _ {t}\right) \left(x _ {t + \delta} - x _ {t}\right) ^ {\top} \right] = \frac {\delta^ {2}}{4} A \mathbb {E} _ {\pi \left(x _ {t}\right)} \left[ \nabla \log \pi \left(x _ {t}\right) \nabla \log \pi \left(x _ {t}\right) ^ {\top} \right] A + \delta A. \tag {5}
$$
Further,  $tr\left(\mathbb{E}[(x_{t + \delta} - x_t)(x_{t + \delta} - x_t)^\top ]\right) = \mathbb{E}[tr\left((x_{t + \delta} - x_t)(x_{t + \delta} - x_t)^\top\right)] = \mathbb{E}[||x_{t + \delta} - x_t||^2 ],$  which shows that  $J(\delta ,A)$  is the trace of the covariance matrix in (5).
To control discretization error we impose an upper bound constraint  $J(\delta, A) \leq \epsilon$  for a small  $\epsilon > 0$ . A preconditioning that "symmetrizes" the target can be obtained by maximizing the discretization step size  $\delta$  subject to  $J(\delta, A) \leq \epsilon$ . Since  $J(\delta, A)$  monotonically increases with  $\delta$ , the maximum  $\delta^*$  satisfies  $\min_A J(\delta^*, A) = \epsilon$ . This means that the optimal preconditioning  $A^*$  is obtained by minimizing  $J(\delta, A)$  under some global scale constraint on  $A$ , as stated next.
Proposition 3. Suppose  $A$  is a symmetric positive definite matrix satisfying  $tr(A) = c$ , with  $c > 0$  a constant. Then the objective  $J(\delta, A)$ , for any  $\delta > 0$ , is minimized for  $A^*$  given by
$$
A ^ {*} = k \mathcal {I} ^ {- 1}, \quad k = \frac {c}{\sum_ {i = 1} ^ {d} \frac {1}{\mu_ {i}}}, \quad \mathcal {I} = \mathbb {E} _ {\pi (x)} \left[ \nabla \log \pi (x) \nabla \log \pi (x) ^ {\top} \right], \tag {6}
$$
where  $\mu_{i}$  s are the eigenvalues of  $\mathcal{I}$  assumed to satisfy  $0 <   \mu_{i} <   \infty$
The positive multiplicative scalar  $k$  in (6) is not important since the specific value  $c > 0$  is arbitrary, e.g. if we choose  $c = \sum_{i=1}^{d} \frac{1}{\mu_i}$  then  $k = 1$  and  $A^* = \mathcal{I}^{-1}$ . In other words, what matters is that the optimal  $A^*$  is proportional to the inverse matrix  $\mathcal{I}^{-1}$ , so it follows the curvature of  $\mathcal{I}^{-1}$ . For a multivariate Gaussian  $\pi(x) = \mathcal{N}(x|\mu, \Sigma)$  it holds  $\mathcal{I}^{-1} = \Sigma$ , so the optimal preconditioner coincides with the covariance matrix of  $x$ . More generally though, for non-Gaussian targets this will not hold.
Connection with classical Fisher information matrix. The matrix  $\mathcal{I}$  is positive definite since it is the covariance of the gradient  $\nabla \log \pi(x) \coloneqq \nabla_x \log \pi(x)$  where  $\mathbb{E}_{\pi(x)}[\nabla \log \pi(x)] = 0$ . Also,  $\mathcal{I}$  is similar to the classical Fisher information matrix. To illustrate some differences suppose that the target  $\pi(x)$  is a Bayesian posterior  $\pi(\theta|Y) \propto p(Y|\theta)p(\theta) = p(Y,\theta)$  where  $Y$  are the observations and  $\theta \coloneqq x$  are the random parameters. The classical Fisher information is a frequentist quantity where we fix some parameters  $\theta$  and compute  $G(\theta) = \mathbb{E}_{p(Y|\theta)}[\nabla_\theta \log p(Y|\theta)\nabla_\theta \log p(Y|\theta)^\top]$  by averaging over data. In contrast,  $\mathcal{I} = \mathbb{E}_{p(\theta|Y)}[\nabla_\theta \log p(Y,\theta)\nabla_\theta \log p(Y,\theta)^\top]$  is more like a Bayesian quantity where we fix the data  $Y$  and average over the parameters  $\theta$ . Importantly,  $\mathcal{I}$  is not a function of  $\theta$  while  $G(\theta)$  is. Similarly to the classical Fisher information,  $\mathcal{I}$  also satisfies the following standard property: Given that  $\log \pi(x)$  is twice differentiable and  $\nabla_x^2 \log \pi(x)$  is the Hessian matrix,  $\mathcal{I}$  from (6) is also written as  $\mathcal{I} = -\mathbb{E}_{\pi(x)}[\nabla_x^2 \log \pi(x)]$ . Next, we refer to  $\mathcal{I}$  as the Fisher matrix.
Connection with optimal scaling. The Fisher matrix  $\mathcal{I}$  connects also with the optimal scaling result for the RWM algorithm [34, 36]. Specifically, for targets of the form  $\pi(x) = \prod_{i=1}^{d} f(x_i)$ , the RWM proposal  $q(y_n|x_n) = \mathcal{N}(y_n|x_n, (\sigma^2/d)I_d)$  and as  $d \to \infty$  the optimal parameter  $\sigma^2$  is  $\sigma^2 = \frac{2.38}{J}$  where  $J = \mathbb{E}_{f(x)}[(\frac{d\log f(x)}{dx})^2]$  is the (univariate) Fisher information for the univariate density  $f(x)$ , and the preconditioning involves as in our case the inverse Fisher  $\mathcal{I}^{-1} = \frac{1}{J} I_d$ . This result has been generalized also for heterogeneous targets in [36] where again the inverse Fisher information matrix (having now a more general diagonal form) appears as the optimal preconditioner.
4 Fisher information adaptive MALA
Armed with the previous optimality result, we wish to develop an adaptive MCMC algorithm to optimize the proposal in (3) by learning online the global variance  $\sigma^2$  and the preconditioner  $A$ . For  $\sigma^2$  we follow the standard practice to tune this parameter in order to reach an average acceptance rate around 0.574 as suggested by optimal scaling results [35, 36]. For the matrix  $A$  we want to adapt it so that approximately it becomes proportional to the inverse Fisher  $\mathcal{I}^{-1}$  from (6). We also incorporate a parametrization that helps the adaptation of  $\sigma^2$  to be more independent from the one of  $A$ . Specifically, we remove the global scale from  $A$  by defining the overall proposal as
$$
q \left(y _ {n} \mid x _ {n}\right) = \mathcal {N} \left(y _ {n} \mid x _ {n} + \frac {\sigma^ {2}}{\frac {2}{d} \operatorname {t r} (A)} A \nabla \log \pi \left(x _ {n}\right), \frac {\sigma^ {2}}{\frac {1}{d} \operatorname {t r} (A)} A\right), \tag {7}
$$
where  $\sigma^2$  is normalized by  $\frac{1}{d}\mathrm{tr}(A)$ , i.e. the average eigenvalue of  $A$ . Another way to view this is that the effective preconditioner is  $A / (\frac{1}{d}\mathrm{tr}(A))$  which has an average eigenvalue equal to one. The proposal in (7) is invariant to any scaling of  $A$ , i.e. if  $A$  is replaced by  $kA$  (with  $k > 0$ ) the proposal remains the same. Also, note that when  $A$  is the identity matrix  $I_d$  (or a multiple of identity) then  $\frac{1}{d}\mathrm{tr}(I_d) = 1$  and the above proposal reduces to standard MALA with isotropic step size  $\sigma^2$ .
It is straightforward to adapt  $\sigma^2$  towards an average acceptance rate 0.574; see pseudocode in Algorithm 1. Thus our main focus next is to describe the learning update for  $A$ , in fact eventually not for  $A$  itself but for a square root matrix  $\sqrt{A}$  which is what we need to sample from the proposal in (7).
To start with, let us simplify notation by writing the score function at the  $n$ -th MCMC iteration as  $s_n \coloneqq \nabla_{x_n} \log \pi(x_n)$ . We introduce the  $n$ -sample empirical Fisher estimate
$$
\hat {\mathcal {I}} _ {n} = \frac {1}{n} \sum_ {i = 1} ^ {n} s _ {i} s _ {i} ^ {\top} + \frac {\lambda}{n} I _ {d}, \tag {8}
$$
where  $\lambda > 0$  is a fixed damping parameter. Given that certain conditions apply [21, 38] so that the chain converges and ergodic averages converge to exact expected values,  $\hat{\mathcal{I}}_n$  is a consistent estimator satisfying  $\lim_{n\to \infty}\hat{\mathcal{I}}_n = \mathcal{I}$  since as  $n\to \infty$  the damping part  $\frac{\lambda}{n} I_d$  vanishes. Including the damping is very important since it offers a Tikhonov-like regularization, similar to ridge regression, and it ensures that for any finite  $n$  the eigenvalues of  $\hat{\mathcal{I}}_n$  are strictly positive. An estimate then for the preconditioner  $A_{n}$  can be set to be proportional to the inverse of the empirical Fisher  $\hat{\mathcal{I}}_n$ , i.e.
$$
A _ {n} \propto \left(\frac {1}{n} \sum_ {i = 1} ^ {n} s _ {i} s _ {i} ^ {\top} + \frac {\lambda}{n} I _ {d}\right) ^ {- 1} = n \left(\sum_ {i = 1} ^ {n} s _ {i} s _ {i} ^ {\top} + \lambda I _ {d}\right) ^ {- 1}. \tag {9}
$$
Since any positive multiplicative scalar in front of  $A_{n}$  plays no role, we can ignore the scalar  $n$  and define  $A_{n} = (\sum_{i=1}^{n} s_{i} s_{i}^{\top} + \lambda I_{d})^{-1}$ . Then, as MCMC iterates we can adapt  $A_{n}$  in  $O(d^{2})$  cost per iteration based on the recursion
$$
\text {I n i t i a l i z a t i o n :} A _ {1} = \left(s _ {1} s _ {1} ^ {\top} + \lambda I _ {d}\right) ^ {- 1} = \frac {1}{\lambda} \left(I _ {d} - \frac {s _ {1} s _ {1} ^ {\top}}{\lambda + s _ {1} ^ {\top} s _ {1}}\right), \tag {10}
$$
$$
I t e r a t i o n: A _ {n} = \left(A _ {n - 1} ^ {- 1} + s _ {n} s _ {n} ^ {\top}\right) ^ {- 1} = A _ {n - 1} - \frac {A _ {n - 1} s _ {n} s _ {n} ^ {\top} A _ {n - 1}}{1 + s _ {n} ^ {\top} A _ {n - 1} s _ {n}}, \tag {11}
$$
where we applied Woodbury matrix identity. This estimation in the limit can give the optimal preconditioning in the sense that under the ergodicity assumption,  $\lim_{n\to \infty}\frac{A_n}{\operatorname{tr}(A_n)} = \frac{\mathcal{I}^{-1}}{\operatorname{tr}(\mathcal{I}^{-1})}$ . In
practice we do not need to compute directly the matrix  $A_{n}$  but a square root matrix  $R_{n} \coloneqq \sqrt{A}_{n}$ , such that  $R_{n}R_{n}^{\top} = A_{n}$ , since we need a square root matrix to draw samples from the proposal in (7). To express the corresponding recursion for  $R_{n}$  we will rely on a technique that dates back to the early days of Kalman filtering [31, 10], which applied to our case gives the following result.
Proposition 4. A square root matrix  $R_{n}$ , such that  $R_{n}R_{n}^{\top} = A_{n}$ , can be computed recursively in  $O(d^{2})$  time per iteration as follows:
$$
\text {I n i t i a l i z a t i o n :} R _ {1} = \frac {1}{\sqrt {\lambda}} \left(I _ {d} - r _ {1} \frac {s _ {1} s _ {1} ^ {\top}}{\lambda + s _ {1} ^ {\top} s _ {1}}\right), \quad r _ {1} = \frac {1}{1 + \sqrt {\frac {\lambda}{\lambda + s _ {1} ^ {\top} s _ {1}}}} \tag {12}
$$
$$
I t e r a t i o n: R _ {n} = R _ {n - 1} - r _ {n} \frac {\left(R _ {n - 1} \phi_ {n}\right) \phi_ {n} ^ {\top}}{1 + \phi_ {n} ^ {\top} \phi_ {n}}, \phi_ {n} = R _ {n - 1} ^ {\top} s _ {n}, r _ {n} = \frac {1}{1 + \sqrt {\frac {1}{1 + \phi_ {n} ^ {\top} \phi_ {n}}}}. \tag {13}
$$
A way to generalize the above recursive estimation of a square root for the inverse Fisher matrix is to consider the stochastic approximation framework [33]. This requires to write an online learning update for the empirical Fisher of the form
$$
\hat {\mathcal {I}} _ {n} = \hat {\mathcal {I}} _ {n - 1} + \gamma_ {n} \left(s _ {n} s _ {n} ^ {\top} - \hat {\mathcal {I}} _ {n - 1}\right), \quad \text {i n i t i a l i z e d} \quad \hat {\mathcal {I}} _ {1} = s _ {1} s _ {1} ^ {\top} + \lambda I _ {d}, \tag {14}
$$
where the learning rates  $\gamma_{n}$  satisfy the standard conditions  $\sum_{n = 1}^{\infty}\gamma_{n} = \infty ,\sum_{n = 1}^{\infty}\gamma_{n}^{2} < \infty$  . Then, it is straightforward to generalize the recursion for the square root matrix in Proposition 4 to account for this more general case; see Appendix B. The recursion in Proposition 4 is a special case when  $\gamma_{n} = \frac{1}{n}$  . In our simulations we did not observe significant improvement by using more general learning rate sequences, and therefore in all our experiments in Section 5 we use the standard learning rate  $\gamma_{n} = \frac{1}{n}$  . Note that this learning rate is also used by other adaptive MCMC methods [21].
An adaptive algorithm that learns online from the score function vectors  $s_n$  can work well in some cases, but still it can be unstable in general. One reason is that  $s_n = \nabla \log \pi(x_n)$  will not have zero expectation when the chain is transient and states  $x_n$  are not yet draws from the stationary distribution  $\pi$ . To analyze this, note that the learning signal  $s_n$  enters in the empirical Fisher estimator  $\bar{\mathcal{I}}_n$  through the outer product  $s_n s_n^\top$  as shown by Eqs. (8) and (14). However, in the transient phase  $s_n s_n^\top$  will be biased since the expectation  $\mathbb{E}[s_n s_n^\top] = \mathbb{E}[(s_n - \mathbb{E}[s_n])(s_n - \mathbb{E}[s_n])^\top] + \mathbb{E}[s_n]\mathbb{E}[s_n]^\top \neq \mathcal{I}$ , where the expectations are taken under the marginal distribution of the chain at time  $n$ . In practice the mean vector  $\mathbb{E}[s_n]$  can take large absolute values, which can introduce significant bias through the additive term  $\mathbb{E}[s_n]\mathbb{E}[s_n]^\top$ . Thus, to reduce some bias we could track the empirical mean  $\bar{s}_n = \frac{1}{n}\sum_{i=1}^{n}s_i$  and center the signal  $s_n - \bar{s}_n$  so that the Fisher matrix is estimated by the empirical covariance  $\frac{1}{n-1}\sum_{i=1}^{n}(s_i - \bar{s}_n)(s_i - \bar{s}_n)^\top$ . The recursive estimation becomes similar to standard adaptive MCMC [21] where we recursively propagate an online empirical estimate for the mean of  $s_n$  and incorporate it into the online empirical estimate of the covariance matrix (in our case the inverse Fisher matrix); see Eq. (17) in Section 5 for the standard adaptive MCMC recursion [21] and Appendix C for our Fisher method. While this can make learning quite stable we experimentally discovered that there is another scheme, presented next in Section 4.1, that is significantly better and stable especially for very anisotropic high dimensional targets; see detailed results in Appendix E.3.
4.1 Adapting to score function increments