SlowGuess/NeurIPS_2023_OCR huggingface dataset:
which establishes  $(\mathbb{R}^{+}3)$ . Thus we have established  $(\mathbb{R}^{+}1 - 3)$  for  $Q = \Pi$ , so from Theorem 3 we have strong consistency of SIIT-MALA (i.e.  $D_{P}(P_{n,m}) \stackrel{\mathrm{a.s.}}{\to} 0$ ) provided that  $m \leq n$  with  $\log(n) = O(m^{\gamma/2})$  for some  $\gamma < 1$ . The latter condition is equivalent to  $m = \Omega((\log n)^{\delta})$  for some  $\delta > 2$ , which we used for the statement. Since  $0 \leq D_{P}(P_{n}^{\star}) \leq D_{P}(P_{n,m})$ , the strong consistency of SIIS-MALA is also established.
C Explicit Calculation of Stein Kernels
This appendix contains explicit calculations for the Langevin-Stein and KGM-Stein kernels  $k_{P}$  which are sufficient to implement Stein II-Importance Sampling and Stein II-Thinning. These calculations can also be performed using automatic differentiation, but comparison to the analytic expressions is an important step in validation of computer code.
To proceed, we observe that the diffusion Stein operator  $S_P$  in (12) applied to a matrix-valued kernel  $K$  is equivalent to the Langevin-Stein operator applied to the kernel  $C(x,y) = M(x)K(x,y)M(y)^\top$ . In the case of the Langevin-Stein and KGM-Stein kernels we have  $K(x,y) = \kappa(x,y)I$  for some  $\kappa(x,y)$  and  $M(x) = (1 + \|x - x_\star\|_\Sigma^{2})^{(s-1)/2}I$  for some  $s \in \{0,1,2,\ldots\}$ . Thus  $C(x,y) = c(x,y)I$  where
$$
c (x, y) := \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {(s - 1) / 2} \left(1 + \| y - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {(s - 1) / 2} \kappa (x, y)
$$
$$
\begin{array}{l} k _ {P} (x, y) = \nabla_ {x} \cdot \nabla_ {y} c (x, y) + [ \nabla_ {x} c (x, y) ] \cdot [ \nabla_ {y} \log p (y) ] + [ \nabla_ {y} c (x, y) ] \cdot [ \nabla_ {x} \log p (x) ] \\ + c (x, y) \left[ \nabla_ {x} \log p (x) \right] \cdot \left[ \nabla_ {y} \log p (y) \right], \\ \end{array}
$$
following the calculations in Oates et al. (2017). To evaluate the terms in this formula we start by differentiating  $c(x,y)$ , to obtain
$$
\begin{array}{l} \nabla_ {x} c (x, y) = \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {(s - 1) / 2} \left(1 + \| y - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {(s - 1) / 2} \\ \times \left[ \frac {(s - 1) \kappa (x , y) \Sigma^ {- 1} (x - x _ {\star})}{1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}} + \nabla_ {x} \kappa (x, y) \right] \\ \end{array}
$$
$$
\begin{array}{l} \nabla_ {y} c (x, y) = \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {(s - 1) / 2} \left(1 + \| y - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {(s - 1) / 2} \\ \times \left[ \frac {(s - 1) \kappa (x , y) \Sigma^ {- 1} (y - x _ {\star})}{1 + \| y - x _ {\star} \| _ {\Sigma} ^ {2}} + \nabla_ {y} \kappa (x, y) \right] \\ \end{array}
$$
$$
\begin{array}{l} \nabla_ {x} \cdot \nabla_ {y} c (x, y) = (1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {(s - 1) / 2} (1 + \| y - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {(s - 1) / 2} \\ \times \left[ \frac {(s - 1) ^ {2} \kappa (x , y) (x - x _ {\star}) ^ {\top} \Sigma^ {- 2} (y - x _ {\star})}{(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) (1 + \| y - x _ {\star} \| _ {\Sigma} ^ {2})} + \frac {(s - 1) (y - x _ {\star}) ^ {\top} \Sigma^ {- 1} \nabla_ {x} \kappa (x , y)}{(1 + \| y - x _ {\star} \| _ {\Sigma} ^ {2})} \right. \\ \left. + \frac {(s - 1) (x - x _ {\star}) ^ {\top} \Sigma^ {- 1} \nabla_ {y} \kappa (x , y)}{(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2})} + \nabla_ {x} \cdot \nabla_ {y} \kappa (x, y) \right]. \\ \end{array}
$$
These expressions involve gradients of  $\kappa (x,y)$ , and explicit formulae for these are presented for the choice of  $\kappa (x,y)$  corresponding to the Langevin-Stein kernel in Appendix C.1, and to the KGM-Stein kernel in Appendix C.2.
To implement Stein II-Thinning we require access to both  $k_{P}(x)$  and  $\nabla k_{P}(x)$ , the latter for use in the proposal distribution and acceptance probability in MALA. These quantities will now be calculated. In what follows we assume that  $\kappa (x,y)$  is continuously differentiable, so that partial derivatives with respect to  $x$  and  $y$  can be interchanged. Then
$$
\begin{array}{l} c _ {0} (x) := c (x, x) \\ = (1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {s - 1} \kappa (x, x) \\ \end{array}
$$
$$
\begin{array}{l} c _ {1} (x) := \left. \nabla_ {x} c (x, y) \right| _ {y \rightarrow x} \\ = \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {s - 1} \left[ \frac {(s - 1) \kappa (x , x) \Sigma^ {- 1} (x - x _ {\star})}{\left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right)} + \nabla_ {x} \kappa (x, y) | _ {y \rightarrow x} \right] \\ \end{array}
$$
$$
\begin{array}{l} c _ {2} (x) := \left. \nabla_ {x} \cdot \nabla_ {y} c (x, y) \right| _ {y \rightarrow x} \\ = \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {s - 1} \left[ \frac {(s - 1) ^ {2} \kappa (x , x) (x - x _ {\star}) ^ {\top} \Sigma^ {- 2} (x - x _ {\star})}{(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {2}} \right. \\ \left. \right.\left. + \frac {2 (s - 1) (x - x _ {\star}) ^ {\top} \Sigma^ {- 1} \nabla_ {x} \kappa (x , y) | _ {y \rightarrow x}}{\left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right)} + \nabla_ {x} \cdot \nabla_ {y} \kappa (x, y) | _ {y \rightarrow x} \right] \\ \end{array}
$$
$$
k _ {P} (x) := k _ {P} (x, x) = c _ {2} (x) + 2 c _ {1} (x) \cdot \nabla_ {x} \log p (x) + c _ {0} (x) \| \nabla_ {x} \log p (x) \| ^ {2}. \tag {19}
$$
Let  $[\nabla_x c_1(x)]_{i,j} = \partial_{x_i}[c_1(x)]_j$  and  $[\nabla_x^2\log p(x)]_{i,j} = \partial_{x_i}\partial_{x_j}\log p(x)$ . Now we can differentiate (19) to get
$$
\begin{array}{l} \nabla_ {x} k _ {P} (x) = \nabla_ {x} c _ {2} (x) + 2 [ \nabla_ {x} c _ {1} (x) ] [ \nabla_ {x} \log p (x) ] + 2 [ \nabla_ {x} ^ {2} \log p (x) ] c _ {1} (x) \\ + \left[ \nabla_ {x} c _ {0} (x) \right] \| \nabla_ {x} \log p (x) \| ^ {2} + 2 c _ {0} (x) \left[ \nabla_ {x} ^ {2} \log p (x) \right] \left[ \nabla_ {x} \log p (x) \right]. \tag {20} \\ \end{array}
$$
In what follows we also derive explicit formulae for  $c_{0}(x)$ ,  $c_{1}(x)$  and  $c_{2}(x)$ , and hence for  $\nabla_{x}c_{0}(x)$ ,  $\nabla_{x}c_{1}(x)$  and  $\nabla_{x}c_{2}(x)$ , for the case of the Langevin-Stein kernel in Appendix C.1, and the KGM-Stein kernel in Appendix C.2.
C.1 Explicit Formulae for the Langevin-Stein Kernel
The Langevin-Stein kernel from Appendix A.3.1 corresponds to the choice  $s = 1$  and  $\kappa(x, y)$  the inverse multi-quadratic kernel, so that
$$
\kappa (x, y) = (1 + \| x - y \| _ {\Sigma} ^ {2}) ^ {- \beta}
$$
$$
\nabla_ {x} \kappa (x, y) = - 2 \beta (1 + \| x - y \| _ {\Sigma} ^ {2}) ^ {- \beta - 1} \Sigma^ {- 1} (x - y)
$$
$$
\nabla_ {y} \kappa (x, y) = 2 \beta (1 + \| x - y \| _ {\Sigma} ^ {2}) ^ {- \beta - 1} \Sigma^ {- 1} (x - y)
$$
$$
\begin{array}{l} \nabla_ {x} \cdot \nabla_ {y} \kappa (x, y) = - 4 \beta (\beta + 1) (1 + \| x - y \| _ {\Sigma} ^ {2}) ^ {- \beta - 2} (x - y) ^ {\top} \Sigma^ {- 2} (x - y) \\ + 2 \beta \operatorname {t r} (\Sigma^ {- 1}) (1 + \| x - y \| _ {\Sigma} ^ {2}) ^ {- \beta - 1}. \\ \end{array}
$$
Evaluating on the diagonal:
$$
\kappa (x, x) = 1
$$
$$
\left. \nabla_ {x} \kappa (x, y) \right| _ {y \rightarrow x} = \left. \nabla_ {y} \kappa (x, y) \right| _ {y \rightarrow x} = 0
$$
$$
\left. \nabla_ {x} \cdot \nabla_ {y} \kappa (x, y) \right| _ {y \rightarrow x} = 2 \beta \mathrm {t r} (\Sigma^ {- 1}),
$$
so that  $c_{0}(x) = 1$ ,  $c_{1}(x) = 0$ ,  $c_{2}(x) = 2\beta \mathrm{tr}(\Sigma^{-1})$ . Differentiating these formulae,  $\nabla_{x}c_{0}(x) = 0$ ,  $\nabla_{x}c_{1}(x) = 0$ ,  $\nabla_{x}c_{2}(x) = 0$ .
C.2 Explicit Formulae for the KGM-Stein Kernel
The KGM kernel of order  $s$  from Appendix A.3.2 corresponds to the choice
$$
\kappa (x, y) = \left(1 + \| x - y \| _ {\Sigma} ^ {2}\right) ^ {- \beta} + \frac {1 + \left(x - x _ {\star}\right) ^ {\top} \Sigma^ {- 1} \left(y - x _ {\star}\right)}{\left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {s / 2} \left(1 + \| y - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {s / 2}},
$$
for which we have
$$
\begin{array}{l} \nabla_ {x} \kappa (x, y) = - 2 \beta (1 + \| x - y \| _ {\Sigma} ^ {2}) ^ {- \beta - 1} \Sigma^ {- 1} (x - y) \\ + \frac {\Sigma^ {- 1} (y - x _ {\star}) - s [ 1 + (x - x _ {\star}) ^ {\top} \Sigma^ {- 1} (y - x _ {\star}) ] \Sigma^ {- 1} (x - x _ {\star}) (1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {- 1}}{(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {s / 2} (1 + \| y - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {s / 2}} \\ \end{array}
$$
$$
\begin{array}{l} \nabla_ {y} \kappa (x, y) = 2 \beta (1 + \| x - y \| _ {\Sigma} ^ {2}) ^ {- \beta - 1} \Sigma^ {- 1} (x - y) \\ + \frac {\Sigma^ {- 1} (x - x _ {\star}) - s [ 1 + (x - x _ {\star}) ^ {\top} \Sigma^ {- 1} (y - x _ {\star}) ] \Sigma^ {- 1} (y - x _ {\star}) (1 + \| y - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {- 1}}{(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {s / 2} (1 + \| y - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {s / 2}} \\ \end{array}
$$
$$
\begin{array}{l} \nabla_ {x} \cdot \nabla_ {y} \kappa (x, y) = - 4 \beta (\beta + 1) (1 + \| x - y \| _ {\Sigma} ^ {2}) ^ {- \beta - 2} (x - y) ^ {\top} \Sigma^ {- 2} (x - y) + 2 \beta \operatorname {t r} (\Sigma^ {- 1}) (1 + \| x - y \| _ {\Sigma} ^ {2}) ^ {- \beta - 1} \\ + \frac {\left[ \begin{array}{c} \operatorname {t r} (\Sigma^ {- 1}) - s (1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {- 1} (x - x _ {\star}) ^ {\top} \Sigma^ {- 2} (x - x _ {\star}) \\ - s (1 + \| y - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {- 1} (y - x _ {\star}) ^ {\top} \Sigma^ {- 2} (y - x _ {\star}) \\ + s ^ {2} [ 1 + (x - x _ {\star}) ^ {\top} \Sigma^ {- 1} (y - x _ {\star}) ] (1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {- 1} (1 + \| y - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {- 1} \\ \times (x - x _ {\star}) ^ {\top} \Sigma^ {- 2} (y - x _ {\star}) \\ \end{array} \right]}{(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {s / 2} (1 + \| y - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {s / 2}}. \\ \end{array}
$$
$$
\kappa (x, x) = 1 + \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {- s + 1}
$$
$$
\begin{array}{l} \nabla_ {x} \kappa (x, y) | _ {y \rightarrow x} = \nabla_ {y} \kappa (x, y) | _ {y \rightarrow x} = - (s - 1) \Sigma^ {- 1} (x - x _ {\star}) (1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {- s} \\ \left. \nabla_ {x} \cdot \nabla_ {y} \kappa (x, y) \right| _ {y \rightarrow x} = 2 \beta \operatorname {t r} (\Sigma^ {- 1}) + \operatorname {t r} (\Sigma^ {- 1}) (1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {- s} \\ + s (s - 2) \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {- s - 1} \left(x - x _ {\star}\right) ^ {\top} \Sigma^ {- 2} \left(x - x _ {\star}\right) \\ \end{array}
$$
$$
c _ {0} (x) = 1 + \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {s - 1}
$$
$$
c _ {1} (x) = (s - 1) \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {s - 2} \Sigma^ {- 1} (x - x _ {\star})
$$
$$
c _ {2} (x) = \frac {[ (s - 1) ^ {2} (1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {s - 1} - 1 ] (x - x _ {\star}) ^ {\top} \Sigma^ {- 2} (x - x _ {\star})}{(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {2}} + \frac {\operatorname {t r} (\Sigma^ {- 1}) [ 1 + 2 \beta (1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {s} ]}{(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2})}.
$$
Differentiating these formulae:
$$
\begin{array}{l} \nabla_ {x} c _ {0} (x) = 2 (s - 1) \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {s - 2} \Sigma^ {- 1} (x - x _ {\star}) \\ \nabla_ {x} c _ {1} (x) = 2 (s - 1) (s - 2) \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {s - 3} \left[ \Sigma^ {- 1} \left(x - x _ {\star}\right) \right] \left[ \Sigma^ {- 1} \left(x - x _ {\star}\right) \right] ^ {\top} \\ + (s - 1) \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {s - 2} \Sigma^ {- 1} \\ \end{array}
$$
$$
\begin{array}{l} \nabla_ {x} c _ {2} (x) = 2 (s - 1) ^ {2} (s - 3) \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {s - 4} \left[ \left(x - x _ {\star}\right) ^ {\top} \Sigma^ {- 2} \left(x - x _ {\star}\right) \right] \Sigma^ {- 1} (x - x _ {\star}) \\ + 2 (s - 1) ^ {2} \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {s - 3} \Sigma^ {- 2} \left(x - x _ {\star}\right) \\ + 4 \beta \operatorname {t r} (\Sigma^ {- 1}) (s - 1) (1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}) ^ {s - 2} \Sigma^ {- 1} (x - x _ {\star}) \\ - 2 \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {- 2} \left[ \Sigma^ {- 2} \left(x - x _ {\star}\right) + \operatorname {t r} \left(\Sigma^ {- 1}\right) \Sigma^ {- 1} \left(x - x _ {\star}\right) \right] \\ + 4 \left(1 + \| x - x _ {\star} \| _ {\Sigma} ^ {2}\right) ^ {- 3} \left[ \left(x - x _ {\star}\right) ^ {\top} \Sigma^ {- 2} \left(x - x _ {\star}\right) \right] \Sigma^ {- 1} \left(x - x _ {\star}\right). \\ \end{array}
$$
These complete the analytic calculations necessary to compute the Stein kernel  $k_{P}$  and its gradient.
D Empirical Assessment
This appendix contains full details of the empirical protocols that were employed and the additional empirical results described in the main text. Appendix D.1 discusses the effect of dimension on our proposed II. Additional illustrative results from Section 3.2 are contained in Appendix D.2. The full details for how MALA was implemented are contained in Appendix D.3. An additional illustration using a generalised auto-regressive moving average (GARCH) model is presented in Appendix D.4. The full results for SIIIS-MALA are contained in Appendix D.5, and in Appendix D.6 the convergence of the sparse approximation provided by SIIT-MALA to the optimal weighted approximation is investigated. Finally, the performance of KSDs is quantified using the 1-Wasserstein divergence in Appendix D.7.
D.1 The Effect of Dimension on  $\Pi$
The improvement of Stein II-Importance Sampling over the default Stein importance sampling algorithm (i.e.  $\Pi = P$ ) can be expected to reduce as the dimension  $d$  of the target  $P$  is increased. To see this, consider the Langevin-Stein kernel
$$
k _ {P} (x) = c _ {1} + c _ {2} \| \nabla \log p (x) \| _ {\Sigma} ^ {2} \tag {21}
$$
for some  $c_{1}, c_{2} > 0$ ; see Appendix C. Taking  $P = \mathcal{N}(0, I_{d \times d})$ , for which the length scale matrix  $\Sigma$  appearing in Appendix A.3 is  $\Sigma = I_{d \times d}$ , we obtain
$$
k _ {P} (x) = c _ {1} + c _ {2} \| x \| ^ {2}.
$$
However, the sampling distribution  $\Pi$  defined in (8) depends on  $k_{P}$  only up to an unspecified normalisation constant; we may therefore equally consider the asymptotic behaviour of  $\tilde{k}_{P}(x) := k_{P}(x)/d$ . Let  $X \sim P$ . Then  $\mathbb{E}[\tilde{k}_{P}(X)] = c_{2}$  is a  $d$ -independent constant, and
$$
\left\| \tilde {k} _ {P} - \mathbb {E} [ \tilde {k} _ {P} (X) ] \right\| _ {L ^ {2} (P)} ^ {2} = \int \left[ \frac {k _ {P} (x) - \left(c _ {1} + c _ {2} d\right)}{d} \right] ^ {2} \mathrm {d} P (x) = \frac {2 c _ {2} ^ {2}}{d} \rightarrow 0
$$
as  $d\to \infty$ . This shows that  $\tilde{k}_P$  converges to a constant function in  $L^2 (P)$ , and thus for "typical" values of  $x$  in the effective support of  $P$ ,
$$
\pi (x) \propto p (x) \sqrt {\tilde {k} _ {P} (x)} \stackrel {\approx} {\propto} p (x),
$$
so that  $\Pi \approx P$  in the  $d\to \infty$  limit. This intuition is borne out in simulations involving both the Langevin-Stein kernel (as just discussed) and also the KGM3-Stein kernel. Indeed, Figure S1 shows that as the dimension  $d$  is increased, the marginal distributions of  $\Pi$  become increasingly similar to those of  $P$ .
(S1.1)
(S1.2)
Figure S1: The effect of dimension on  $\Pi$ : Here  $P$  was taken to be the standard Gaussian distribution  $\tilde{\mathcal{N}}(0, I_{d \times d})$  in  $\mathbb{R}^d$  and the proposed distribution  $\Pi$  was computed. The marginal distribution of the first component of  $\Pi$  is plotted for  $d \in \{1, 2, 10\}$ , for both (a) the Langevin-Stein kernel and (b) the KGM3-Stein kernel.
Figure S2: Assessing the performance of the sampling distributions  $\Pi$  shown in Figure 2. The mean kernel Stein discrepancy (KSD) for computation performed using the Langevin-Stein kernel (purple), the KGM3-Stein kernel (blue), and the Riemann-Stein kernel (red); in each case, KSD was computed using the same Stein kernel used to construct  $\Pi$ . Solid lines indicate the baseline case of sampling from  $P$ , while dashed lines indicate the proposed approach of sampling from  $\Pi$ . (The experiment was repeated 10 times and standard error bars are plotted.)
D.2 2D Illustration from the Main Text
Section 3.2 of the main text contained a 2-dimensional illustration of Stein II-Importance Sampling and presented the distributions  $\Pi$  corresponding to different choices of Stein kernel. Here, in Figure S2, we present the mean KSDs for Stein II-Importance Sampling performed using the Langevin-Stein kernel (purple), the KGM3-Stein kernel (blue), and the Riemann-Stein kernel (red), corresponding to the sampling distributions  $\Pi$  displayed in Figure 2 of the main text.
For this experiment, exact sampling from both  $P$  and  $\Pi$  was performed using a fine grid on which all probabilities were calculated and appropriately normalised. Results are in broad agreement with the 1-dimensional illustration contained in the main text, in the sense that in all cases Stein II-Importance Sampling provides a significant improvement over the default Stein importance sampling method with  $\Pi$  equal to  $P$ .
Algorithm 4 Adaptive MALA
Require:  $x_{0,0}$  (initial state),  $\epsilon_0$  (initial step size),  $M_0$  (initial preconditioner matrix),  $\{n_i\}_{i=0}^{h-1}$  (epoch lengths),  $\{\alpha_i\}_{i=1}^{h-1}$  (learning schedule),  $h$  (number of epochs),  $k_P$  (Stein kernel)  
1:  $\{x_{0,1}, \ldots, x_{0,n_0}\} \gets \text{MALA}(x_{0,0}, \epsilon_0, M_0, n_0, k_P)$   
2: for  $i = 1, \ldots, h-1$  do  
3:  $x_{i,0} \gets x_{i-1,n_i-1}$   
4:  $\rho_{i-1} \gets \frac{1}{n_i-1} \sum_{j=1}^{n_i-1} 1_{x_{i-1,j} \neq x_{i-1,j-1}}$   
5:  $\epsilon_i \gets \epsilon_{i-1} \exp(\rho_{i-1} - 0.57)$   
6:  $M_i \gets \alpha_i M_i + (1 - \alpha_i) \text{cov}(\{x_{i-1,1}, \ldots, x_{i-1,n_i-1}\})$   
7:  $\{x_{i,1}, \ldots, x_{i,n_i}\} \gets \text{MALA}(x_{i,0}, \epsilon_i, M_i, n_i, k_P)$   
8: end for
D.3 Implementation of MALA
For implementation of MALA in Algorithm 4 we are required to specify a step size  $\epsilon$  and a preconditioner matrix  $M$ . In general, suitable values for both of these parameters will be problem-dependent. Standard practice is to perform some form of manual or automated tuning to arrive at parameter values for which the average acceptance rate is close to 0.57, motivated by the asymptotic analysis of Roberts and Rosenthal (1998). Adaptive MCMC algorithms, which seek to optimise the parameters of MCMC algorithms such as MALA during the warm-up period, provide an appealing solution, and was the approach taken in this work.
The adaptive MALA algorithm which we used is contained in Algorithm 4, where we have let  $\mathrm{MALA}(x,\epsilon ,M,n,k_P)$  denote the output from the preconditioned MALA with initial state  $x$ , step size  $\epsilon$ , preconditioner matrix  $M$ , and chain length  $n$ , described in Algorithm 1. In Algorithm 4, we use  $\operatorname{cov}(\cdot)$  to denote the sample covariance matrix. The algorithm monitors the average acceptance rate and increases or decreases it according to whether it is below or above, respectively, the 0.57 target. For the preconditioner matrix, the sample covariance matrix of samples obtained from the penultimate tuning run of MALA is used. For all experiments that we report using MALA, we set  $\epsilon_0 = 1$ ,  $M_0 = I_d$ ,  $h = 10$ , and  $\alpha_{1} = \dots = \alpha_{9} = 0.3$ . The warm-up epoch lengths were  $n_0 = \dots = n_8 = 1,000$  and the final epoch length was  $n_9 = 10^5$ . The samples  $\{x_{h - 1,1},\ldots ,x_{h - 1,n_i - 1}\}$  from the final epoch are returned, and constituted output from MALA for our experimental assessment.
To sample from  $P$  instead of  $\Pi$ , we used Algorithm 4 we formally set  $k_{P}(x) = 1$  for all  $x \in \mathbb{R}^{d}$ , which recovers  $\Pi = P$  as the target.
D.4 Illustration on a GARCH Model
This appendix contains an additional illustrative experiment, concerning a GARCH model that is a particular instance of a model from the PosteriorDB database discussed in Section 4. The purpose of this illustration is to facilitate an empirical investigation in a slightly higher dimension  $(d = 4)$  and to explore the effect of changing the order  $s$  of the KGM-Stein kernel defined in Appendix A.3.2.
First we describe the GARCH model that was used. These models are widely-used in econometrics to describe time series data  $\{y_{t}\}_{t = 1}^{n}$  in settings where the volatility process is assumed to be time-varying (but stationary). In particular, we consider the GARCH(1,1) model
$$
y _ {t} = \phi_ {1} + a _ {t},
$$
$$
a _ {t} = \sigma_ {t} \epsilon_ {t}, \quad \epsilon_ {t} \sim \mathcal {N} (0, 1),
$$
$$
\sigma_ {t} ^ {2} = \phi_ {2} + \phi_ {3} a _ {t - 1} ^ {2} + \phi_ {4} \sigma_ {t - 1} ^ {2},
$$
where  $\phi_2 > 0, \phi_3 > 0, \phi_4 > 0$ , and  $\phi_3 + \phi_4 < 1$  are the model parameters, constrained to a subset of  $\mathbb{R}^4$ . For ease of sampling, a change of variables  $\tau : (\phi_1, \phi_2, \phi_3, \phi_4) \mapsto \theta$  is performed in such a way that the parameter  $\theta \in \mathbb{R}^4$  is unconstrained. Assuming an improper flat prior on  $\theta$ , the log-posterior density for  $\theta$  is given up to an additive constant by
$$
\log p (\theta \mid y _ {1}, \dots , y _ {n}) \stackrel {+ C} {=} \sum_ {t = 1} ^ {n} \left[ - \frac {1}{2} \log \left(\sigma_ {t} ^ {2}\right) - \frac {y _ {t} ^ {2}}{2 \sigma_ {t} ^ {2}} \right] + \log | J _ {\tau^ {- 1}} (\theta) |,
$$
where  $|J_{\tau^{-1}}(\theta)|$  is the Jacobian determinant of  $\tau^{-1}$ .
Figure S3: Illustrating the shape of  $\Pi$  based on the KGMs-Stein kernel for a GARCH(1,1) model, controlling convergence of moments up to order  $s \in \{2, 3, 4\}$ . The marginal density functions of each distribution were approximated using one-million samples obtained using MCMC.
For this illustration, real data were provided within the model description of PosteriorDB, for which the estimated maximum a posteriori parameter is  $\hat{\phi} = (5.04, 1.36, 0.53, 0.31)$ . The marginal distributions of  $\Pi$  corresponding to the KGM-Stein kernels of orders  $s \in \{2, 3, 4\}$  are compared to the marginals of  $P$  in Figure S3. It can be seen that higher orders  $s$  correspond to greater overdispersion of  $\Pi$ ; this makes intuitive sense since larger  $s$  corresponds to a more stringent KSD (controlling the convergence of moments up to order  $s$ ) which places greater emphasis on how the tails of  $P$  are approximated. Further, for the final skewed marginal of  $P$ , we note that the distribution  $\Pi$  exaggerates the skew, placing more of its mass in the tail of the direction which is positively skewed. Further discussion of skewed targets is contained in Appendix D.8.
D.5 Stein II-Importance Sampling for PosteriorDB
To introduce objectivity into our assessment, we exploited the PosteriorDB benchmark (Magnusson et al., 2022). This ongoing project is an attempt toward standardised benchmarking, consisting of a collection of posteriors to be numerically approximated. The test problems in PosteriorDB are defined in the Stan probabilistic programming language, and so BridgeStan (Roualdes et al., 2023) was used to directly access posterior densities and their gradients as required. The ambition of PosteriorDB is to provide an extensive set of benchmark tasks; at the time we conducted our research, PosteriorDB was at Version 0.4.0 and contained 149 models, of which 47 came equipped with a gold-standard sample of size  $n = 10^3$ , generated from a long run of Hamiltonian Monte Carlo (the No-U-Turn sampler in Stan). Of these 47 models, a subset of 40 were found to be compatible with BridgeStan, which was at Version 1.0.2 at the time this research was performed. The version of Stan that we used was Stanc3 Version 2.31.0 (Unix). Thus we used a total of 40 test problems for our empirical assessment.