SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For each test problem, a total of 10 replicate experiments were performed and standard errors were computed. A sampling method was defined as being significantly better for approximation of a given target, compared to all other methods considered, if had lower mean KSD and the standard error bar did not overlap with the standard error bar of any other method. Table 1 in the main text summarises the performance of SIIS-MALA, fixing the number of samples to be  $n = 3 \times 10^3$ . In this appendix, full empirical results are provided.
For sampling from MALA, we used the adaptive algorithm described in Appendix D.3 with a final epoch of length  $n_{\mathrm{max}} = 10^5$ . Then, whenever a set of  $n \ll n_{\mathrm{max}}$  consecutive samples from MALA are required for our experimental assessment, these were obtained by selecting at random a consecutive sequence of length  $n$  from the total chain of length  $10^5$ . This ensures that the performance of unprocessed MALA that we report is not negatively affected by burn-in, in so far as is practical to control.
Full results are presented in Figure S4. These results broadly support the interpretation that SIIS-MALA usually outperforms SIS-MALA, or otherwise both methods provide a similar level of performance, for the sufficiently large sample sizes  $n$  considered. The sample size threshold at which SIIS-MALA outperforms SIS-MALA appears to be dimension-dependent. A notable exception is panel 29 of Figure S4, a  $d = 10$  dimensional task for which SIIS-MALA provided a substantially worse approximation in KSD for the range of values of  $n$  considered.
Figure S4: Benchmarking on PosteriorDB. Here we compared raw output from MALA (dotted lines) with the post-processed output provided by the default Stein importance sampling method of Liu and Lee (2017) (SIS-MALA; solid lines) and the proposed Stein II-Importance Sampling method (SIIIS-MALA; dashed lines). The Langevin (purple) and KGM3-Stein kernels (blue) were used for SIS-MALA and SIIIS-MALA and the associated KSDs are reported as the number  $n$  of iterations of MALA is varied. Ten replicates were computed and standard errors were plotted. The name of each model is shown in the title of the corresponding panel, and the dimension  $d$  of the parameter vector is given in parentheses. [Langevin-Stein kernel: MALA, ——SIS-MALA, ——SIIIS-MALA. KGM3-Stein kernel: MALA, ——SIS-MALA, ——SIIIS-MALA.]
(S4.1)
(S4.2)
(S4.3)
(S4.4)
(S4.5)
(S4.6)
(S4.7)
(S4.8)
(S4.9)
(S4.10)
(S4.11)
(S4.12)
(S4.13)
(S4.14)
(S4.15)
(S4.16)
(S4.17)
(S4.18)
(S4.19)
(S4.20)
(S4.21)
(S4.22)
(S4.23)
(S4.24)
(S4.25)
(S4.26)
(S4.27)
(S4.28)
(S4.29)
(S4.30)
(S4.31)
(S4.32)
(S4.33)
(S4.34)
(S4.35)
(S4.36)
(S4.37)
(S4.38)
(S4.39)
(S4.40)
D.6 Stein II-Thinning for PosteriorDB
The results presented in the main text concerned  $n = 3 \times 10^3$  samples from MALA, which is near the limit at which the optimal weights  $w^{\star}$  can be computed in a few seconds on a laptop PC. For larger values of  $n$ , sparse approximation methods are likely to require. In the main text we presented Stein II-Thinning, which employs a greedy optimisation perspective to obtain a sparse approximation to the optimal weights at cost  $O(m^{2}n)$ , where  $m$  are the number of greedy iterations performed. Explicit and verifiable conditions for the strong consistency of the resulting SIIT-MALA algorithm were established in Section 3.3. The purpose of this appendix is to empirically explore the convergence of SIIT-MALA using the PosteriorDB test bed.
In the experiments we report the number of MALA samples was fixed to  $n = 10^3$  and the number of greedy iterations was varied from  $m = 1$  to  $m = 10^3$ . The results, in Figure S5, indicate that for most models in PosteriorDB the minimum value of KSD is approximately reached when  $m$  is anywhere from  $\frac{n}{10}$  to  $\frac{n}{2}$ , representing a modest but practically significant reduction in computational cost compared to SIIIS-MALA. This agrees with the qualitative findings reported in the original Stein thinning paper of Riabiz et al. (2022).
Figure S5: Benchmarking on PosteriorDB. Here we investigate the convergence of the sparse approximation provided by the proposed Stein II-Thinning method (SIIT-MALA). The Langevin (purple) and KGM3-Stein kernels (blue) were used for SIIT-MALA and the associated KSDs are reported as the number  $m$  of iterations of Stein thinning is varied. Ten replicates were computed and standard errors were plotted. The name of each model is shown in the title of the corresponding panel, and the dimension  $d$  of the parameter vector is given in parentheses. [Langevin-Stein kernel:——SIIT-MALA. KGM3-Stein kernel:——SIIT-MALA.]
(S5.1)
(S5.2)
(S5.3)
(S5.4)
(S5.5)
(S5.6)
(S5.7)
(S5.8)
(S5.9)
(S5.10)
(S5.11)
(S5.12)
(S5.13)
(S5.14)
(S5.15)
(S5.16)
(S5.17)
(S5.18)
(S5.19)
(S5.20)
(S5.21)
(S5.22)
(S5.23)
(S5.24)
(S5.25)
(S5.26)
(S5.27)
(S5.28)
(S5.29)
(S5.30)
(S5.31)
(S5.32)
(S5.33)
(S5.34)
(S5.35)
(S5.36)
(S5.37)
(S5.38)
D.7 Performance of Stein Discrepancies
The properties of Stein discrepancies was out of scope for this work. Nonetheless, there is much interest in better understanding the properties of KSDs, and in this appendix the performance of SIIS-MALA in terms of 1-Wasserstein divergence is reported. This was made possible since PosteriorDB supplies a set of posterior samples obtained from a long run of Hamiltonian Monte Carlo (the No-U-Turn sampler in Stan) which we treat as a gold standard.
Full results are presented in Figure S6 and Table 2. Broadly speaking, in most cases the minimisation of KSD seems to be associated with minimisation of 1-Wasserstein distance, and in particular a significant improvement of SIIIS-MALA over SIS-MALA is reported for  $\approx 63\%$  of tasks in the PosteriorDB benchmark. However there are some scenarios for which minimisation of KSD is loosely, if at all, related to minimisation of 1-Wasserstein divergence. In these cases, we attribute this performance to a combination of two factors: First, the blindness to mixing proportions phenomena, described in Wenliang and Kanagawa (2021); Koehler et al. (2022); Liu et al. (2023), which is a pathology of KSDs in general. Second, the Langevin-Stein kernel cannot be expected to control convergence in 1-Wasserstein, since convergence in 1-Wasserstein is equivalent to weak convergence plus convergence of the first moment. Focusing therefore on the KGM3-Stein kernel only, it is encouraging to note that SIIIS-MALA outperforms SIS-MALA on  $83\%$  of tasks in PosteriorDB in the 1-Wasserstein metric, as shown in Table 2. However, it is interesting to observe that MALA performed well in the 1-Wasserstein sense across the PosteriorDB test bed.
The development of improved Stein discrepancies is an active area of research, and we emphasise that the methodology developed in this work can be applied to any KSDs, including potentially KSDs with better or more direct control over standard notions of convergence (such as 1-Wasserstein) that in the future may be developed.
Figure S6: Performance of Stein discrepancies on PosteriorDB. Here we compared raw output from MALA (dotted lines) with the post-processed output provided by the default Stein importance sampling method of Liu and Lee (2017) (SIS-MALA; solid lines) and the proposed Stein II-Importance Sampling method (SIIS-MALA; dashed lines). The Langevin (purple) and KGM3-Stein kernels (blue) were used for SIS-MALA and SIIS-MALA, and the 1-Wasserstein divergence is reported as the number  $n$  of iterations of MALA is varied. Ten replicates were computed and standard errors were plotted. The name of each model is shown in the title of the corresponding panel, and the dimension  $d$  of the parameter vector is given in parentheses. [Legend: ……Raw MALA. Langevin-Stein kernel: ——SIS-MALA, —— —— —— SIIIS-MALA. KGM3-Stein kernel: —— SIS-MALA, —— —— —— SIIIS-MALA.]
(S6.1)
(S6.2)
(S6.3)
(S6.4)
(S6.5)
(S6.6)
(S6.7)
(S6.8)
(S6.9)
(S6.10)
(S6.11)
(S6.12)
(S6.31)
(S6.32)
(S6.33)
(S6.34)
(S6.35)
(S6.36)
D.8 Investigation for a Skewed Target
This final appendix contrasts the 1-Wasserstein optimal sampling distribution  $\Pi_1$  (c.f. Section 2.1), with the choice of  $\Pi$  that we recommended in (8). In particular, we focus on the KGM3-Stein kernel under a heavily skewed  $P$ , for which  $\Pi_1$  and  $\Pi$  can be markedly different.
For this investigation a bivariate skew-normal target was constructed, where the density is given by  $p(x_1,x_2) = 4\phi (x_1)\Phi (6x_1)\phi (x_2)\Phi (-3x_2)$ , with  $\phi$  and  $\Phi$  respectively denoting the density and distribution functions of a standard Gaussian. The density  $p$  of  $P$ , together with the marginal densities of  $\Pi_{1}$  and  $\Pi$ , are plotted in Figure S7. It can be seen that, while both  $\Pi_{1}$  and  $\Pi$  are over-dispersed with respect to  $P$ , our recommended  $\Pi$  assigns proportionally more mass to the tail that is positively skewed.
The performance of Stein II-Importance Sampling based on  $\Pi_1$  and  $\Pi$  is compared in Figure S8. Though both choices lead to an improvement relative to Stein importance sampling algorithm with  $\Pi = P$ , the use of  $\Pi$  leads to a significant further reduction (on average) in KSD compared to  $\Pi_1$ . Based on our investigations, this finding seems general; the use of  $\Pi_1$  does not realise the full potential of Stein II-Importance sampling when the target is skewed.
Langevin-Stein Kernel	KGM3-Stein Kernel
Task	d	MALA	SIS - MALA	SI11S - MALA	MALA	SIS - MALA	SI11S - MALA
earnings-earn_height	3	6420.0	7230.0	9440.0	6420.0	7280.0	7390.0
gp_pois_regr-gp_regr	3	0.0779	0.0758	0.0724	0.0779	0.0865	0.0778
kidiq-kidscore_momhps	3	0.282	0.581	0.508	0.282	0.950	0.604
kidiq-kidscore_momiq	3	0.826	2.19	2.01	0.826	2.53	1.82
mesquite-logmesquite_logvolume	3	0.0236	0.0253	0.0238	0.0236	0.0311	0.0234
arma-arma11	4	0.0127	0.0132	0.0131	0.0127	0.0144	0.0133
earnings-logearn_logheight_male	4	1.46	1.75	1.34	1.46	1.56	0.725
garch-garch11	4	0.260	0.241	0.243	0.260	0.365	0.263
kidiq-kidscore_momhsiq	4	1.86	2.39	1.72	1.86	2.51	1.54
earnings-logearn_interaction_z	5	0.0245	0.0240	0.0240	0.0245	0.0252	0.0251
kidiq-kidscore_interaction	5	13.9	14.5	15.3	13.9	14.4	20.4
kidiq_withMom_work-kidscore_interaction_c	5	0.289	0.251	0.237	0.289	0.584	0.371
kidiq_withMom_work-kidscore_interaction_c2	5	0.258	0.269	0.252	0.258	0.638	0.430
kidiq_withMom_work-kidscore_interaction_z	5	0.914	0.904	0.889	0.914	1.46	1.17
kidiq_withMom_work-kidscoreMom_work	5	1.01	1.05	1.06	1.01	1.74	1.43
low_dim_gauss_mix-low_dim_gauss_mix	5	0.0215	0.0206	0.0207	0.0215	0.0214	0.0212
mesquite-logmesquite_logva	5	0.0715	0.0672	0.0681	0.0715	0.0833	0.0775
hmm_example-hmm_example	6	0.0708	0.102	0.105	0.0708	0.161	0.12
sblrc-blr	6	0.0613	0.0614	0.0581	0.0613	0.0613	0.0605
sbli-rblr	6	0.0729	0.0733	0.0843	0.0729	0.0722	0.0926
arK-arK	7	0.0544	0.0533	0.0525	0.0544	0.0618	0.0557
mesquite-logmesquite_logvash	7	0.165	0.158	0.161	0.165	0.180	0.171
bball_driver_event_0-hmm_driver_0	8	0.203	0.183	0.186	0.203	0.242	0.216
bball DRIVER_event_1-hmm_driver_1	8	0.812	0.825	0.811	0.812	0.814	0.754
hudson_lynx_hare-lotka_volterra	8	0.113	0.112	0.111	0.113	0.135	0.120
mesquite-logmesquite	8	0.212	0.204	0.209	0.212	0.214	0.212
mesquite-logmesquite_logvas	8	0.197	0.192	0.196	0.197	0.208	0.203
mesquite-mesquite	8	123.0	122.0	114.0	123.0	123.0	121.0
eight_schools-eight_schools_centered	10	9.17	9.50	8.14	9.17	7.00	10.3
eight_schools-eight_schools_noncentered	10	16.8	16.7	16.7	16.8	16.7	16.7
nes1972-nes	10	0.193	0.189	0.172	0.193	0.198	0.172
nes1976-nes	10	0.190	0.189	0.179	0.190	0.202	0.177
nes1980-nes	10	0.229	0.228	0.222	0.229	0.256	0.233
nes1984-nes	10	0.202	0.198	0.197	0.202	0.210	0.184
nes1988-nes	10	0.212	0.210	0.185	0.212	0.214	0.185
nes1992-nes	10	0.172	0.169	0.159	0.172	0.182	0.168
nes1996-nes	10	0.191	0.187	0.173	0.191	0.199	0.179
nes2000-nes	10	0.275	0.273	0.274	0.275	0.306	0.284
diamonds-diamonds	26	0.648	0.647	0.643	0.648	0.648	0.658
mcycle_gp-accel_gp	66	15.5	15.5	15.6	15.5	15.5	15.8
Table 2: Benchmarking on PosteriorDB. Here we compared raw output from MALA with the post-processed output provided by the default Stein importance sampling method of Liu and Lee (2017) (SIS-MALA) and the proposed Stein II-Importance Sampling method (SIiIS-MALA). Here  $d = \dim(P)$  and the number of MALA samples was  $n = 3 \times 10^3$ . The Langevin and KGM3-Stein kernels were used for SIS-MALA and SIiIS-MALA and the associated 1-Wasserstein distances are reported. Ten replicates were computed and statistically significant improvement is highlighted in bold.
Figure S7: Comparing the proposed distribution  $\Pi$  (KGM3; based on the KGM3-Stein kernel) to  $\Pi_1$  (1Wass.; the optimal choice for 1-Wasserstein quantisation from Section 2.1) for a bivariate skew-normal target ( $d = 2$ ). The marginal density functions of each distribution were approximated using  $10^6$  samples from MCMC.
Figure S8: Comparing the performance of using the proposed distribution  $\Pi$  (KGM3; based on the KGM3-Stein kernel) to  $\Pi_1$  (1Wass.; the optimal choice for 1-Wasserstein quantisation from Section 2.1) for a bivariate skew-normal target ( $d = 2$ ). The mean kernel Stein discrepancy (KSD) for Stein II-Importance Sampling was estimated; in each case, the KSD based on the KGM3-Stein kernel was computed. Solid lines indicate the baseline case of sampling from  $P$ , while dashed lines indicate sampling from  $\Pi$ . (The experiment was repeated 10 times and standard error bars are plotted.)