SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For illustration, consider the univariate target  $P$  (black curve) in Figure 1.1, a 3-component Gaussian mixture model. Our recommended choice of  $\Pi$  in (8) is shown for both the Langevin-Stein kernel (purple curve) and the KGMs-Stein kernels with  $s \in \{1,3\}$  (green and blue curves). The Stein discrepancy corresponding to the Langevin-Stein kernel provides control over weak convergence (i.e. convergence of integrals of functions that are continuous and bounded), while the KGMs-Stein kernel provides additional control over the convergence of polynomial moments up to order  $s$ ; full details about the construction of Stein kernels are contained in Appendix A. The Langevin and KGM1-Stein kernels have  $k_{P}(x) \asymp x^{2}$ , while the KGM3-Stein kernel has  $k_{P}(x) \asymp x^{6}$ , in each case as  $|x| \to \infty$ , and thus greater over-dispersion results from use of the KGM3-Stein kernel. This over-dispersion is less pronounced<sup>4</sup> in higher dimensions; see Appendix D.1.
To illustrate the performance of Stein  $\Pi$ -Importance Sampling, we generated a sequence  $(x_{n})_{n\in \mathbb{N}}$  of independent samples from  $\Pi$ . For each  $n\in \{1,\dots ,100\}$ , the samples  $\{x_{1},\ldots ,x_{n}\}$  were assigned optimal weights  $w^{\star}$  by solving (3), and the associated KSD was computed. As a baseline, we performed the same calculation using independent samples from  $P$ . Figure 1.2 indicates that, for both Stein kernels, substantial improvement results from the use of samples from  $\Pi$  compared to the use of samples from  $P$ . Interestingly, the KGM3-Stein kernel demonstrated a larger improvement compared to the Langevin-Stein kernel, suggesting that the choice of  $\Pi$  may be more critical in settings where KSD enjoys a stronger form of convergence control.
To illustrate a posterior approximation task, consider a simple regression model  $y_{i} = f_{i}(x) + \epsilon_{i}$  with  $f_{i}(x) = x_{1}(1 + t_{i}x_{2})$ ,  $t_{i} = i - 5$ ,  $i = 1, \ldots, 10$ , with  $\epsilon_{i}$  independent  $\mathcal{N}(0,1)$ . The parameter  $x = (x_{1}, x_{2})$  was assigned a prior  $\mathcal{N}(0, I)$ . Data were simulated using  $x = (0, 0)$ . The posterior distribution  $P$  is depicted in the leftmost panel of Figure 2, while our choice of  $\Pi$  corresponding to the Langevin (centre left), KGM3 (centre right) and Riemann-Stein kernels (right) are also displayed. For the Langevin and KGM3 kernels, the associated  $\Pi$  target their mass toward regions where  $P$  varies the most. The reason for this behaviour is clearly seen for the Langevin-Stein kernel since
4The same holds for classical quantisation; c.f. Section 2.1.
Figure 2: Illustrating our choice of  $\Pi$  in 2D. The bivariate target  $P$  (left), together with our choice of  $\Pi$  based on the Langevin-Stein kernel (centre left), the KGM3-Stein kernel (centre right), and the Riemann-Stein kernel (right).
Algorithm 1 II-Invariant Metropolis-Adjusted Langevin Algorithm (MALA)
Require:  $x_0$  (initial state),  $\epsilon$  (step size),  $M$  (preconditioner matrix),  $n$  (chain length),  $k_P$  (Stein kernel) 1: for  $i = 1,\dots ,n$  do
$$
2: \qquad x ^ {\prime} \leftarrow \underbrace {x _ {i - 1} + \epsilon M ^ {- 1} \nabla \log p (x _ {i - 1}) + \frac {\epsilon}{2} M ^ {- 1} \nabla \log k _ {P} (x _ {i - 1})} _ {=: \nu (x _ {i - 1})} + \sqrt {2 \epsilon} M ^ {- 1 / 2} Z _ {i} \qquad \rhd Z _ {i} \stackrel {{\mathrm {I I D}}} {{\sim}} \mathcal {N} (0, I)
$$
$$
3: \quad L \leftarrow \log \left(\frac {p \left(x ^ {\prime}\right)}{p \left(x _ {i - 1}\right)}\right) + \frac {1}{2} \log \left(\frac {k _ {P} \left(x ^ {\prime}\right)}{k _ {P} \left(x _ {i - 1}\right)}\right) - \frac {1}{4 \epsilon} \| x _ {i - 1} - \nu (x ^ {\prime}) \| _ {M ^ {- 1}} ^ {2} + \frac {1}{4 \epsilon} \| x ^ {\prime} - \nu (x _ {i - 1}) \| _ {M ^ {- 1}} ^ {2}
$$
$$
4: \quad \text {i f} \log (U _ {i}) <   L \text {t h e n} x _ {i} \leftarrow x ^ {\prime}; \text {e l s e} x _ {i} \leftarrow x _ {i - 1}; \text {e n d} \text {i f} \quad \triangleright U _ {i} \stackrel {\mathrm {I I D}} {\sim} \mathcal {U} ([ 0, 1 ])
$$
Algorithm 2 Stein II-Importance Sampling (SIIIS-MALA)
Require:  $\{x_{1},\ldots ,x_{n}\}$  from Algorithm 1,  $k_{P}$  (Stein kernel)
$$
1: w ^ {\star} \in \arg \min  _ {w \in \mathbb {R} ^ {d}} \left\{\langle w, K _ {P} w \rangle : w \geq 0, 1 ^ {\top} w = 1 \right\} \quad \triangleright [ K _ {P} ] _ {i, j} = k _ {P} (x _ {i}, x _ {j})
$$
$k_{P}(x) = c_{1} + c_{2}\| \nabla \log p(x)\|^{2}$  for some  $c_{1},c_{2} > 0$ ; see Appendix C for detail. The Riemann-Stein kernel can be viewed as a preconditioned form of the Langevin-Stein kernel which takes into account the geometric structure of  $P$ ; see Appendix A for full detail<sup>5</sup>. Results in Figure S2 demonstrate that Stein  $\Pi$ -Importance Sampling improves upon the default Stein importance sampling method (i.e. with  $\Pi$  and  $P$  equal) for all choices of kernel.
An additional illustration involving a GARCH model with  $d = 4$  parameters is presented in Appendix D.4, where the effect of varying the order  $s$  of the KGM-Stein kernel is explored.
3.3 Theoretical Guarantees
The aim of this section is to establish when post-processing of  $\Pi$ -invariant MCMC produces a strongly consistent approximation of  $P$ , for our recommended choice of  $\Pi$  in (8). Our analysis focuses on MALA (Roberts and Stramer, 2002), leveraging the recent work of Durmus and Moulines (2022) to present explicit and verifiable conditions on  $P$  for our results to hold. In fact, we consider the more general preconditioned form of MALA, where the symmetric positive definite preconditioner matrix  $M$  is to be specified. Our results also allow for (optional) sparse approximation, to circumvent direct solution of (3) (c.f. Section 2.3). The resulting algorithms, which we call Stein  $\Pi$ -Importance Sampling (SIIIS-MALA) and Stein  $\Pi$ -Thinning (SIIT-MALA), are quite straight-forward and contained, respectively, in Algorithms 2 and 3. The linearly-constrained quadratic programme in Algorithm 2 was solved using the Python v3.10.4 packages qpsolvers v3.4.0 and proxsuite v0.3.7. While it is difficult to analyse the computational complexity associated with these methods, we believe they are at worst  $O(n^3)$ .
5The use of geometric information may be beneficial, in the sense that the associated diffusion process may mix more rapidly, and rapid mixing leads to sharper bounds from the perspective of convergence control (Gorham et al., 2019). However, the Riemann-Stein kernel is associated with a prohibitive computational cost; it is included here only for academic interest.
Algorithm 3 Stein II-Thinning (SIIT-MALA)
Require:  $\{x_{1},\ldots ,x_{n}\}$  from Algorithm 1,  $m$  (number of samples to retain),  $k_{P}$  (Stein kernel)
1: for  $i = 1, \ldots, m$  do
2:  $y_{i}\gets \arg \min_{y\in \{x_{1},\ldots ,x_{n}\}}\frac{1}{2} k_{P}(y) + \sum_{j = 1}^{i - 1}k_{P}(y,y_{j})$
3: end for
Let  $A \preceq B$  indicate that  $A - B$  is a positive semi-definite matrix for  $A, B \in \mathbb{R}^{n \times n}$ . For a symmetric positive definite matrix  $A$  let  $\| z \|_A \coloneqq \sqrt{z^\top A^{-1}z}$  for  $z \in \mathbb{R}^d$ . Let  $C^s(\mathbb{R}^d)$  denote the set of  $s$ -times continuously differentiable real-valued functions on  $\mathbb{R}^d$ .
Theorem 1 (Strong consistency of SIIIS- and SIIT-MALA). Let Assumption 1 hold where  $k = k_{P}$  is a Stein kernel, and let  $D_{P}: \mathcal{P}(\mathcal{X}) \to [0, \infty]$  denote the associated KSD. Assume also that
(A1)  $\nabla \log p\in C^2 (\mathbb{R}^d)$  with  $\sup_{x\in \mathbb{R}^d}\| \nabla^2\log p(x)\| <  \infty$
(A2)  $\exists b_{1} > 0, B_{1} \geq 0$  such that  $-\nabla^{2}\log p(x) \succeq b_{1}I$  for all  $\|x\| \geq B_{1}$
(A3)  $k_{P}\in C^{2}(\mathbb{R}^{d})$
(A4)  $\exists 0 < b_{2} < 2b_{1}C_{1}^{2}, B_{2} \geq 0$  such that  $\nabla^2 k_P(x) \preceq b_2I$  for all  $\|x\| \geq B_2$
Let  $P_{n}^{\star} = \sum_{i=1}^{n} w_{i}^{\star} \delta(x_{i})$  be the result of running Algorithm 2 and let  $P_{n,m} = \frac{1}{m} \sum_{i=1}^{m} \delta(y_{i})$  be the result of running Algorithm 3. Let  $m \leq n$  and  $m = \Omega((\log n)^{\delta})$  for some  $\delta > 2$ . Then there exists  $\epsilon_{0} > 0$  such that, for all step sizes  $\epsilon \in (0, \epsilon_{0})$  and all initial states  $x_{0} \in \mathbb{R}^{d}$ ,  $D_{P}(P_{n}^{\star}) \to 0$ ,  $D_{P}(P_{n,m}) \to 0$  almost surely as  $m, n \to \infty$ .
The proof is in Appendix B. Compared to earlier authors $^6$ , such as Chen et al. (2019); Riabiz et al. (2022), a major novelty here is that our assumptions are explicit and can often be verified (see also Hodgkinson et al., 2020). (A2) is strong log-concavity of  $P$  when  $B_{1} = 0$ , while for  $B_{1} > 0$  this condition is slightly stronger than the related distant dissipativity condition assumed in earlier work (Gorham and Mackey, 2017; Riabiz et al., 2022). (A4) holds for the Langevin-Stein kernel (i.e. weak convergence control) and for the KGM1-Stein kernel (i.e. weak convergence control + control over first moments), but not for the higher-order KGM-Stein kernels. Extending our proof strategy to the higher-order KGM-Stein kernels would require further research into the convergence properties of MALA, and this is expected to be difficult.
4 Benchmarking on PosteriorDB
The area of Bayesian computation has historically lacked a common set of benchmark problems, with classical examples being insufficiently difficult and case-studies being hand-picked (Chopin and Ridgway, 2017). To introduce objectivity into our assessment, we exploited the recently released PosteriorDB benchmark (Magnusson et al., 2022). This project is an attempt toward standardised benchmarking, consisting of a collection of posteriors to be numerically approximated. Here, we systematically compared the performance of SIIS-MALA against the default Stein importance sampling algorithm (i.e.  $\Pi = P$ ; denoted SIS-MALA), and also against unprocessed  $P$ -invariant MALA (i.e. uniform weights), reporting results across the breadth of PosteriorDB. The test problems in PosteriorDB are defined in the Stan probabilistic programming language, and so BridgeStan (Roualdes et al., 2023) was used to directly access posterior densities and their gradients as required. For all instances of MALA, an adaptive algorithm was used to learn a suitable preconditioner matrix  $M$  during the warm-up period; see Appendix D.3. All experiments that we report can be reproduced using code available at https://github.com/congyewang/Stein-Pi-Importance-Sampling.
Results are reported in Table 1 for  $n = 3 \times 10^{3}$  samples from MALA. These focus on the Langevin-Stein kernel, for which our theory holds, and the KGM3-Stein kernel, for which it does not. There was a significant improvement of SIIS-MALA over SIS-MALA in  $73\%$  of test problems for the Langevin-Stein kernel and in  $65\%$  of test problems for the KGM3-Stein kernel. Compared to
6These earlier results required high-level assumptions on the convergence of MCMC, for which explicit sufficient conditions had yet to be derived.
Langevin-Stein Kernel	KGM3-Stein Kernel
Task	d	MALA	SIS - MALA	STIIS - MALA	MALA	SIS - MALA	STIIS - MALA
earnings-earn_height	3	1.41	0.0674	0.0332	5.33	0.656	0.181
gp_pois_regr-gp_regr	3	0.298	0.0436	0.0373	1.22	0.385	0.223
kidiq-kidscore_momhhs	3	1.04	0.109	0.0941	4.66	0.848	0.476
kidiq-kidscore_momiq	3	5.03	0.516	0.358	25.3	4.86	1.55
mesquite-logmesquite_logvolume	3	1.10	0.179	0.156	4.97	1.70	0.844
arma-arma11	4	4.47	1.09	1.01	26.0	8.91	6.03
earnings-logearn_logheight_male	4	9.46	1.96	1.59	53.9	15.4	8.65
garch-garch11	4	0.543	0.159	0.130	4.70	1.16	1.01
kidiq-kidscore_momhsiq	4	5.21	0.982	0.897	29.3	7.25	5.05
earnings-logearn_interaction_z	5	3.09	1.36	1.33	19.3	10.4	8.94
kidiq-kidscore_interaction	5	7.74	1.65	1.79	47.8	13.2	10.1
kidiq_with_mom_work-kidscore_interaction_c	5	1.35	0.659	0.711	7.92	4.05	4.17
kidiq_with_mom_work-kidscore_interaction_c2	5	1.38	0.689	0.699	8.09	4.24	4.25
kidiq_with_mom_work-kidscore_interaction_z	5	1.11	0.500	0.499	6.62	2.63	3.25
kidiq_with_mom_work-kidscore_momwork	5	1.07	0.507	0.545	6.70	2.63	3.04
low_dim_gauss_mix-low_dim_gauss_mix	5	5.51	1.87	1.76	37.5	14.7	11.3
mesquite-logmesquite_logva	5	1.83	0.821	0.818	12.6	5.73	5.59
hmm_example-hmm_example	6	1.99	0.578	0.523	11.6	4.13	3.40
sblrc-blr	6	479	154	134	3300	1100	854
sbli-brl	6	201	66.7	60.3	1340	514	595
arK-arK	7	6.87	3.39	3.16	60.4	26.4	23.0
mesquite-logmesquite_logvash	7	1.89	1.18	1.23	15.5	8.88	10.1
bball_driver_event_0-hmm_driver_0	8	1.15	0.679	0.698	8.55	4.72	3.99
bball DRIVER_event_1-hmm_driver_1	8	42.9	11.9	12.4	285	85.6	67.8
hudson_lynx_hare-lotka_volterra	8	4.62	2.29	2.15	47.4	18.8	18.9
mesquite-logmesquite	8	1.46	1.00	1.06	13.3	8.28	9.14
mesquite-logmesquite_logvas	8	2.02	1.31	1.35	19.2	10.8	12.2
mesquite-mesquite	8	0.429	0.268	0.235	3.71	2.17	2.42
eight_schools-eight_schools_centered	10	0.526	0.100	0.182	7.53	2.15	215
eight_schools-eight_schools_noncentered	10	0.210	0.137	0.137	43.6	28.7	27.5
nes1972-nes	10	6.16	3.89	3.45	72.9	36.2	34.4
nes1976-nes	10	6.67	3.86	3.53	77.5	35.5	34.4
nes1980-nes	10	4.34	2.68	2.57	49.8	25.4	25.7
nes1984-nes	10	6.18	3.75	3.43	71.3	34.9	33.6
nes1988-nes	10	7.40	3.70	3.27	81.4	34.6	32.4
nes1992-nes	10	7.52	4.32	3.84	89.1	39.7	37.3
nes1996-nes	10	6.44	3.87	3.53	74.1	36.4	34.3
nes2000-nes	10	3.35	2.22	2.20	38.6	21.3	22.8
diamonds-diamonds	26	196	157	143	5120	2990	2620
mcycle_gp-accel_gp	66	11.3	8.25	9.79	960	623	815
Table 1: Benchmarking on PosteriorDB. Here we compared raw output from MALA with the post-processed output provided by the default Stein importance sampling method of Liu and Lee (2017) (SIS-MALA) and the proposed Stein II-Importance Sampling method (SIiIS-MALA). Here  $d = \dim(P)$  and the number of MALA samples was  $n = 3 \times 10^3$ . The Langevin and KGM3-Stein kernels were used for SIS-MALA and SIiIS-MALA and the associated KSDs are reported. Ten replicates were computed and statistically significant improvement is highlighted in bold.
unprocessed MALA, a significant improvement occurred in  $100\%$  and  $97\%$  of cases, respectively for each kernel. However, the extent of improvement decreased when the dimension  $d$  of the target increased, supporting the intuition that we set out earlier and in Appendix D.1. An in-depth breakdown of results, including varying the number  $n$  of samples that were used, and the performance SIIT-MALA, can be found in Appendices D.5 and D.6.
If  $P$  and its gradients are cheap to evaluate, the computational cost of MALA is lower than that of SIS-MALA, and one could run more iterations of MALA for an equivalent computational cost. But for more complex  $P$ , the computational cost of all algorithms will be gated by the number of times  $P$  and its gradients need to be evaluated, making the direct comparison in Table 1 meaningful. Further, if we aim for a compressed representation of  $P$ , then some form of post-processing of MALA would be required, which would then entail an additional computational cost.
Our focus is on the development of algorithms for minimisation of KSDs; the properties of KSDs themselves are out of scope for this work<sup>7</sup>. Nonetheless, there is much interest in better understanding the properties of KSDs, and we therefore also report performance of SIIIS-MALA in terms of 1-Wasserstein divergence in Appendix D.7. The main contrast between these results and the results in Table 1 is that, being score-based, KSDs suffer from the blindness to mixing proportions phenomena which has previously been documented in Wenliang and Kanagawa (2021); Koehler et al. (2022); Liu et al. (2023). Caution should therefore be taken when using algorithms based on Stein discrepancies
The interested reader is referred to Gorham and Mackey (2017); Barp et al. (2022b); Kanagawa et al. (2022).
in the context of posterior distributions with multiple high probability regions that are spatially separated. This is also a failure mode for MCMC algorithms such as MALA, and yet there are still many problems for which MALA has been successfully used.
The alternative choice  $\Pi_1$ , with  $\pi_1(x) \propto p(x)^{d/(d+1)}$ , which provides a generic form of overdispersion and is optimal for approximation in 1-Wasserstein divergence (c.f. Section 2.1), was also considered. Results in Appendix D.8 indicate that, while  $\Pi_1$  yields an improvement compared to the baseline of using  $P$  itself,  $\Pi_1$  may be less effective than our proposed  $\Pi$  when  $P$  is skewed.
This paper presented Stein II-Importance Sampling; an algorithm that is simple to implement, admits an end-to-end theoretical treatment, and achieves a significant improvement over existing post-processing methods based on KSD. On the negative side, second order derivatives of the statistical model are required, and we are ultimately bound to the performance of the KSD on which Stein II-Importance Sampling is based. Our analysis focused on MALA, but there is in principle no barrier to deriving sufficient conditions for consistent approximation that are applicable to other sampling algorithms, such as the unadjusted Langevin algorithm. Of course, it remains to be seen whether SIS-MALA or any of its variants will stand the test of time compared to continued development in MCMC methodology, but we believe this line of research merits further investigation. For models for which access to second order derivatives is impractical, our methodology and theoretical analysis are directly applicable to gradient-free KSD (Fisher and Oates, 2023), and this would be an interesting direction for future work. Similarly, alternatives to KSD that are better-suited to high-dimensional  $P$  could be considered, such as the sliced KSD of Gong et al. (2021a,b).
Acknowledgements CW was supported by the China Scholarship Council. HK and CJO were supported by EP/W019590/1. The authors are grateful to François-Xavier Briol for feedback on an earlier draft of the manuscript, and to the anonymous Reviewers for their input.
Adachi, M., Hayakawa, S., Jorgensen, M., Oberhauser, H., and Osborne, M. A. (2022). Fast Bayesian inference with batch Bayesian quadrature via kernel recombination. In Proceedings of the 35th Conference on Neural Information Processing Systems.
Agapiou, S., Papaspiliopoulos, O., Sanz-Alonso, D., and Stuart, A. M. (2017). Importance sampling: Intrinsic dimension and computational cost. Statistical Science, 32(3):405-431.
Anastasiou, A., Barp, A., Briol, F.-X., Ebner, B., Gaunt, R. E., Ghaderinezhad, F., Gorham, J., Gretton, A., Ley, C., Liu, Q., Mackey, L., Oates, C. J., Reinert, G., and Swan, Y. (2023). Stein's method meets statistics: A review of some recent developments. Statistical Science, (38):120-139.
Bach, F. (2017). On the equivalence between kernel quadrature rules and random feature expansions. The Journal of Machine Learning Research, 18(1):714-751.
Bach, F., Lacoste-Julien, S., and Obozinski, G. (2012). On the equivalence between herding and conditional gradient algorithms. In Proceedings of the 29th International Conference on Machine Learning.
Barbour, A. D. (1988). Stein's method and poisson process convergence. Journal of Applied Probability, 25(A):175-184.
Barbour, A. D. (1990). Stein's method for diffusion approximations. *Probability Theory and Related Fields*, 84(3):297-322.
Barp, A., Oates, C. J., Porcu, E., and Girolami, M. (2022a). A Riemann-Stein kernel method. Bernoulli, 28(4):2181-2208.
Barp, A., Simon-Gabriel, C.-J., Girolami, M., and Mackey, L. (2022b). Targeted separation and convergence with kernel discrepancies. arXiv preprint arXiv:2209.12835.
Roualdes, E., Ward, B., Axen, S., and Carpenter, B. (2023). BridgeStan: Efficient in-memory access to Stan programs through Python, Julia, and R. https://github.com/roualdes/bridgestan.
Shetty, A., Dwivedi, R., and Mackey, L. (2022). Distribution compression in near-linear time. In Proceedings of the 10th International Conference on Learning Representations.
Sonnleitner, M. (2022). The Power of Random Information for Numerical Approximation and Integration. PhD thesis, University of Passau.
Teymur, O., Gorham, J., Riabiz, M., and Oates, C. J. (2021). Optimal quantisation of probability measures using maximum mean discrepancy. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, pages 1027-1035.
Wenliang, L. K. and Kanagawa, H. (2021). Blindness of score-based methods to isolated components and mixing proportions. In Proceedings of "Your Model is Wrong" @ the 35th Conference on Neural Information Processing Systems.
These appendices contain supporting material for the paper *Stein II-Importance Sampling*. The mathematical background on Stein kernels is contained in Appendix A. The proof of Theorem 1 is contained in Appendix B. For implementation of Stein II-Importance Sampling without the aid of automatic differentiation, various explicit derivatives are required; the relevant calculations can be found in Appendix C. The empirical protocols and additional empirical results are presented in Appendix D.
A Mathematical Background
This appendix contains mathematical background on reproducing kernels and Stein kernels, as used in the main text. Appendix A.1 introduces matrix-valued reproducing kernels, while Appendix A.2 specialises to Stein kernels by application of a Stein operator to a matrix-valued kernel. A selection of useful Stein kernels are presented in Appendix A.3.
A.1 Matrix-Valued Reproducing Kernels
A matrix-valued kernel is a function  $K:\mathbb{R}^d\times \mathbb{R}^d\to \mathbb{R}^{d\times d}$ , that is both
1. symmetric;  $K(x,y) = K(y,x)$  for all  $x,y\in \mathbb{R}^d$ , and
2. positive semi-definite;  $\sum_{i=1}^{n} \sum_{j=1}^{n} \langle c_i, K(x_i, x_j) c_j \rangle \geq 0$  for all  $x_1, \ldots, x_n \in \mathbb{R}^d$  and  $c_1, \ldots, c_n \in \mathbb{R}^d$ .
Let  $K_{x} = K(\cdot, x)$ . For vector-valued functions  $g, g': \mathbb{R}^d \to \mathbb{R}^d$ , defined by  $g = \sum_{i=1}^{n} K_{x_i} c_i$  and  $g' = \sum_{j=1}^{m} K_{x_j'} c_i'$ , define an inner product
$$
\langle g, g ^ {\prime} \rangle_ {\mathcal {H} (K)} = \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} \left\langle c _ {i}, K \left(x _ {i}, x _ {j} ^ {\prime}\right) c _ {j} ^ {\prime} \right\rangle . \tag {9}
$$
There is a unique Hilbert space of such vector-valued functions associated to  $K$ , denoted  $\mathcal{H}(K)$ ; see Proposition 2.1 of Carmeli et al. (2006). This space is characterised as
$$
\mathcal {H} (K) = \overline {{\operatorname {s p a n}}} \{K _ {x} c: x, c \in \mathbb {R} ^ {d} \}
$$
where here the closure is taken with respect to the inner product in (9). It can be shown that  $\mathcal{H}(K)$  is in fact a reproducing kernel Hilbert space (RKHS) which satisfies the reproducing property
$$
\langle g, K _ {x} c \rangle_ {\mathcal {H} (K)} = \langle g (x), c \rangle
$$
for all  $g \in \mathcal{H}(K)$  and  $x, c \in \mathbb{R}^d$ . Matrix-valued kernels are the natural starting point for construction of KSDs, as described next.
A.2 Stein Kernels
A general construction for Stein kernels is to first identify a matrix-valued RKHS  $\mathcal{H}(K)$  and an operator  $S_P: \mathcal{H}(K) \to L^1(P)$  for which  $\int S_p h \, \mathrm{d}P = 0$  for all  $h \in \mathcal{H}(K)$ . Such an operator will be called a Stein operator. The collection  $\{S_p h : h \in \mathcal{H}(K)\}$  inherits the structure of an RKHS, whose reproducing kernel
$$
k _ {P} (x, y) = \left\langle S _ {P} K _ {x}, S _ {P} K _ {y} \right\rangle_ {\mathcal {H} (K)} \tag {10}
$$
is a Stein kernel, meaning that  $\mu_P = 0$  where  $\mu_P$  is the kernel mean embedding from (1); see Barp et al. (2022b). Explicit calculations for the Stein kernels considered in this work can be found in Appendix C.
For univariate distributions, Barbour (1988) proposed to obtain Stein operators from infinitesimal generators of  $P$ -invariant continuous-time Markov processes; see also Barbour (1990); Gotze (1991).
The approach was extended to multivariate distributions in Gorham and Mackey (2015). The starting point is the  $P$ -invariant Itô diffusion