SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The posterior distribution of the weights  $\pmb{\theta}$  is sampled with both MALA and NUTS using 48 independent chains, of respectively  $10^{4}$  and  $10^{5}$  iterations, and four step sizes  $\varepsilon$  are considered along with three thinning sizes  $m$ . Each MCMC sample is post-processed with the two Stein thinning algorithms. For a new input  $\mathbf{x}^{\star}$ , the resulting thinned samples are used to approximate the posterior predictive distribution  $p(Y = 1|\mathbf{x}^{\star},\mathcal{D}_N)$ , defined by  $\int p(Y = 1|\mathbf{x}^{\star},\pmb {\theta})p(\pmb {\theta}|\mathcal{D}_N)d\pmb {\theta}$ . The per
formance of Stein thinning algorithms are assessed using the standard AUC metric for classification problems, estimated with 10-fold cross-validation and 10 repetitions for uncertainties. Table 1 gathers the results for five public datasets from the UCI repository [Dua and Graff, 2017], and described in Appendix A.3, where the best AUC obtained for each algorithm over the four MCMC step sizes are reported. Clearly, regularized Stein thinning significantly improves the performance of Bayesian logistic regression.
Stein thinning has raised a high interest in recent years, as a powerful tool to post-process MCMC outputs, by the greedy minimization of the kernelized Stein discrepancy. Unfortunately, empirical studies have shown that KSD-based algorithms suffer from strong pathologies. We have conducted an in-depth theoretical analysis to identify the mechanisms at stake. From this understanding, we propose an improved Stein thinning algorithm relying on entropic regularization and Laplacian correction. This approach exhibits relevant theoretical properties regarding pathologies, as well as highly improved empirical performance. Finally, the analysis of these regularization terms for other types of KSD-based algorithms, such a KSD descent, seems a promising route for future work.
Table 1: AUCs obtained with NUTS sampler for Stein Thinning (ST) and Regularized Stein Thinning (RST).
Dataset	m = 50	m = 300
ST	RST	ST	RST
Breast W.	0.88 (0.02)	0.96 (0.00)	0.93 (0.01)	0.96 (0.00)
Diabetes	0.52 (0.01)	0.50 (0.02)	0.53 (0.02)	0.57 (0.02)
Haberman	0.51 (0.04)	0.53 (0.02)	0.53 (0.03)	0.58 (0.02)
Liver	0.53 (0.04)	0.69 (0.01)	0.61 (0.04)	0.70 (0.01)
Sonar	0.80 (0.02)	0.81 (0.01)	0.81 (0.01)	0.81 (0.01)
C. M Bishop and N.M. Nasrabadi. Pattern Recognition and Machine Learning, volume 4. Springer, 2006.
S. Brooks, A. Gelman, G. Jones, and X-L. Meng. Handbook of Markov Chain Monte Carlo. CRC press, 2011.
W.Y. Chen, L. Mackey, J. Gorham, F. Briol, and C. Oates. Stein points. In International Conference on Machine Learning, pages 844-853. PMLR, 2018.
W.Y. Chen, A. Barp, F-X. Briel, J. Gorham, M. Girolami, L. Mackey, and C. Oates. Stein point markov chain monte carlo. In International Conference on Machine Learning, pages 1011-1021. PMLR, 2019.
N. Chopin and G. Ducrocq. Fast compression of MCMC output. Entropy, 23:1017, 2021.
K. Chwialkowski, H. Strathmann, and A. Gretton. A kernel test of goodness of fit. In International Conference on Machine Learning, pages 2606-2615. PMLR, 2016.
D. Dua and C. Graff. UCI machine learning repository, 2017. URL http://archive.ics.uci.edu/ml.
E. Fong, S. Lyddon, and C. Holmes. Scalable nonparametric sampling from multimodal posteriors with the posterior bootstrap. In International Conference on Machine Learning, pages 1952-1962. PMLR, 2019.
D. Garreau, W. Jitkrittum, and M. Kanagawa. Large sample analysis of the median heuristic. arXiv preprint arXiv:1707.07269, 2017.
A. Gelman, J.B. Carlin, H.S. Stern, and D.B. Rubin. Bayesian Data Analysis. Chapman and Hall/CRC, 1995.
S. Gershman, M. Hoffman, and D. Blei. Nonparametric variational inference. arXiv preprint arXiv:1206.4665, 2012.
J. Gorham and L. Mackey. Measuring sample quality with stein's method. Advances in Neural Information Processing Systems, 28, 2015.
J. Gorham and L. Mackey. Measuring sample quality with kernels. In International Conference on Machine Learning, pages 1292-1301. PMLR, 2017.
P.J. Green, K. Łatuszyński, M. Pereyra, and C.P. Robert. Bayesian computation: a summary of the current state, and samples backwards and forwards. Statistics and Computing, 25:835-862, 2015.
A. Gretton, K. Borgwardt, M. Rasch, B. Scholkopf, and A. Smola. A kernel method for the two-sample-problem. Advances in Neural Information Processing Systems, 19, 2006.
H. Haario, E. Saksman, and J. Tamminen. Adaptive proposal distribution for random walk metropolis algorithm. Computational Statistics, 14:375-395, 1999.
A. Korba, P-C. Aubin-Frankowski, S. Majewski, and P. Ablin. Kernel stein discrepancy descent. In International Conference on Machine Learning, pages 5719-5730. PMLR, 2021.
Q. Liu and D. Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. In Advances in neural information processing systems, pages 2378-2386, 2016.
Q. Liu, J. Lee, and M. Jordan. A kernelizedstein discrepancy for goodness-of-fit tests. In International Conference on Machine Learning, pages 276-284. PMLR, 2016.
X. Liu, A.B. Duncan, and A. Gandy. Using perturbation to improve goodness-of-fit tests based on kernelized stein discrepancy. arXiv preprint arXiv:2304.14762, 2023.
C. Oates, A. Barp, and M. Girolami. Posterior integration on a riemannian manifold. arXiv preprint arXiv:1712.01793, 2017.
E. Pompe, C. Holmes, and K. Latuszyński. A framework for adaptive mcmc targeting multimodal distributions. The Annals of Statistics, 48:2930-2952, 2020.
Yixuan Qiu and Xiao Wang. Efficient multimodal sampling via tempered distribution flow. Journal of the American Statistical Association, 0(0):1-15, 2023. doi: 10.1080/01621459.2023.2198059.
M. Riabiz, W.Y. Chen, J. Cockayne, P. Swietach, S.A. Niederer, L. Mackey, and C.J. Oates. Optimal thinning of MCMC output. Journal of the Royal Statistical Society: Series B, in press, 2022.
CP.P Robert and G. Casella. Monte Carlo Statistical Methods, volume 2. Springer, 1999.
D. Sejdinovic, B. Sriperumbudur, A. Gretton, and K. Fukumizu. Equivalence of distance-based and rkhs-based statistics in hypothesis testing. The Annals of Statistics, pages 2263-2291, 2013.
L.F. South, M. Riabiz, O. Teymur, and C.J. Oates. Postprocessing of mcmc. Annual Review of Statistics and Its Application, 9:529-555, 2022.
C. Stein. A bound for the error in the normal approximation to the distribution of a sum of dependent random variables. In Proceedings of the sixth Berkeley Symposium on Mathematical Statistics and Probability, Volume 2: Probability Theory, volume 6, pages 583-603. University of California Press, 1972.
L.K. Wenliang and H. Kanagawa. Blindness of score-based methods to isolated components and mixing proportions. arXiv preprint arXiv:2008.10087, 2020.
A.1 Illustration of Theorem 2.3
To better illustrate Theorem 2.3, we run an additional experiment, where  $p$  is still defined as in Figure 7 from Example 2 recalled below, with unbalanced mode weights of 0.2 and 0.8. The density  $q$  is distributed as  $p$ , but each mode is truncated outside a circle of two standard deviation radius, and  $q$  has weight  $w$ . Next, for various values of  $w \in [0.1, 0.9]$ , we draw two samples of size  $n = 3000$  from  $p$  and  $q$ , and compute  $\mathrm{KSD}(\mathbb{P}, \mathbb{Q}_w)$  (with 30 repetitions for each  $w$  value). The result is displayed in Figure 8, and shows that the optimal weight is close to  $1/2$ , as predicted by Theorem 2.3, since  $|\mathrm{KSD}^2(\mathbb{P}, \mathbb{Q}_L) / \mathrm{KSD}^2(\mathbb{P}, \mathbb{Q}_R) - 1|$  is estimated as 0.01 in this case, implying that  $|w^\star - 1/2| < 0.005$ .
Example 2. Let the density  $p$  be a Gaussian mixture model of two components, respectively centered in  $(-\mu, \mathbf{0}_{d-1})$  and  $(\mu, \mathbf{0}_{d-1})$ , of weights  $w$  and  $1 - w$ , and of variance  $\sigma^2 \mathbf{I}_d$ . The initial particles  $\{\mathbf{x}_i\}_{i=1}^n$  are drawn from  $p$ . The KSD thinning algorithm selects  $m < n$  points to approximate  $p$ .
Figure 7: Illustration of Pathology I with the Gaussian mixture of Example 2 ( $d = 2$ ,  $\mu = 3$ ,  $\sigma = 1$ ,  $w = 0.2$ ,  $n = 3000$ ,  $m = 300$ ). Initial particles are in black, and the Stein thinning output is red.
A.2 Gaussian and Banana-shaped Mixtures
This appendix gathers additional results and details for the Gaussian and banana-shaped mixtures, as well as the MMD distance used to evaluate thinning performance, and the regularization parameter  $\lambda$ .
Gaussian mixture. The second batch of experiments in Section 4 considers a  $d$ -dimensional Gaussian mixture of four modes of equal weight, with  $d \geq 2$ , illustrated in Figure 9. The center of modes are chosen as  $(-2,0)$ ,  $(2,0)$ ,  $(-3,4)$ , and  $(3,4)$ , and null values for the higher dimension coordinates. The first two modes have an identity covariance matrix, while the remaining two modes have a diagonal covariance matrix with variance equal to 2. The results for regularized Stein thinning and the original Stein thinning are provided in Figure 10 for MALA sampler, and in Figure 11 for NUTS sampler. In both figures, the three tested step size  $\varepsilon$  are displayed, with a small impact on the resulting performance. Figures 9, 10, and 11 show the high performance improvement of regularized Stein thinning over the original algorithm.
Regularization parameter  $\lambda$ . Figure 12 displays the MMD obtained with regularization parameters  $\lambda$ , set as  $\lambda = 1/m^2$  and  $\lambda = 1/\log(m)$ . These results should be compared with the ones shown
Figure 8:  $\mathrm{KSD}(\mathbb{P},\mathbb{Q}_w)$  for  $p$  as defined in Example 2 with  $\mu = 3$ ,  $\sigma = 1$ ,  $w_{p} = 0.2$ , and  $q$  a truncation of  $p$  and with weight  $w$ . The KSD is estimated with  $n = 3000$  and 30 repetitions for each  $w$  value.
Figure 9: (Gaussian mixture with  $d = 2$ , MALA) First two panels: solutions obtained with Stein thinning and regularized Stein thinning with contour lines of the target distribution. Last panel: heatmap of the Laplacian correction  $\Delta^{+}\log(p)$ .
in Figures 10 and 11, which were obtained with a regularization parameter  $\lambda = 1 / m$ . These additional experiments show the importance of choosing the regularization parameter as  $\lambda = O(1 / m)$ , as suggested by Theorem 3.6. Indeed, slower rates of  $\lambda$  give poor quality samples, and faster rates than  $\lambda = O(1 / m)$  tend to remove the effect of the entropic regularization, and we then recover similar performance than the original Stein thinning. On the other hand,  $\lambda = 1 / m$  provides a high improvement over the standard thinning, as shown in Figures 10 and 11.
Banana-shaped mixture with t-tails. The first batch of experiments in Section 4 considers a banana-shaped mixture with t-tails, defined as follows. Let  $\varphi: \mathbb{R}^d \to \mathbb{R}^d$  be the transformation defined by  $\varphi_i(x) = x_i$  if  $i \neq 2$ , and  $\varphi_2(x) = x_2 + bx_1^2 - 100b$ . Let  $\mathbf{Z}$  be a random variable that follows the multivariate t-Student distribution with degrees of freedom 7. Then, the random variable  $\mathbf{X} = \varphi(\mathbf{Z}) + \boldsymbol{\mu}$  follows a t-banana-shaped distribution centered at  $\boldsymbol{\mu}$ . We consider a mixture of two t-banana-shaped distributions centered in  $\mathbf{0}_d$  and  $(0,8,\mathbf{0}_{d-2})$ , with weights  $w_1 = 0.25$  and  $w_2 = 0.75$ , respectively, which is illustrated in Figure 13 for  $d = 10$ . The results for regularized Stein thinning and the original Stein thinning are provided in Figure 14 for MALA sampler, and in Figure 15 for NUTS sampler. In both figures, the three tested step size  $\varepsilon$  are displayed, which confirms the higher performance of regularized Stein thinning. Figures 13, 14, and 15 show the high performance improvement of regularized Stein thinning over the original algorithm.
Figure 10: (Gaussian mixture, MALA) Graphs of the MMD distance with respect to the thinning size  $m$  (with  $d = 2$ ) and with respect to  $d$  (with  $m = 300$ ) for various step sizes  $\varepsilon$ .
Figure 11: (Gaussian mixture, NUTS) Graphs of the MMD distance with respect to the thinning size  $m$  (with  $d = 2$ ) and with respect to  $d$  (with  $m = 300$ ) for various step sizes  $\varepsilon$ .
A.3 Bayesian Logistic Regression
This appendix gathers additional results for Bayesian logistic regression. In particular, Table 2 provides a description of the tested datasets. Table 3 gives the resulting AUC, for  $m = 50, 100, 300$ , using NUTS or MALA sampler. We recall that only the best AUC over the four tested MCMC step size  $\varepsilon$  is reported.
Recall that the Bayesian logistic regression defines the probability that  $Y_{i}$  is of class 1 as  $p(Y_{i} = 1|\mathbf{X}_{i},\boldsymbol {\beta},\boldsymbol{\beta}_{0}) = 1 / (1 + \exp (-\beta_{0} - \boldsymbol{\beta}^{T}\mathbf{X}_{i}))$ , for some vector of parameters  $\pmb{\theta} = (\beta_0,\pmb {\beta})\in \mathbb{R}^{d + 1}$ . The prior distributions of the weight vector  $\pmb{\theta}$  is assumed to be Gaussian,  $p(\beta^{(j)}|\gamma^{(j)}) = \mathcal{N}(\beta^{(j)}|0,1 / \gamma^{(j)})$ , and a Gamma prior with parameters  $(a,b)$  is chosen for the precision  $\gamma^{(j)}$ . Upon marginalizing, it is found that  $\beta^{(j)}$  is distributed as the non-standardized t-distribution Student-t  $(2a,0,b / a)$  [Bishop and Nasrabadi, 2006]. Following [Fong et al., 2019], the hyperparameters are chosen as  $a = b = 1$ .
Figure 12: (Gaussian mixture, MALA) Graphs of the MMD distance with respect to the thinning size  $m$  (with  $d = 10$ ) for various step sizes  $\varepsilon$ . For the first row, we set  $\lambda = 1 / m^2$ , and we observe that the effect of entropic regularization almost vanishes, since the performance is close to the original Stein thinning. For the second row, we set  $\lambda = 1 / \log(m)$ , violating the convergence assumption, and resulting in bad thinned samples.
Figure 13: (t-banana-shaped mixture with  $d = 10$ , MALA) First two panels: solutions obtained with Stein thinning and regularized Stein thinning with contour lines of the target distribution. Last panel: heatmap of the Laplacian correction  $\Delta^{+}\log(p)$  for  $x^{(3)} = \ldots = x^{(10)} = 0$ .
Table 2: Description of UCI datasets
Dataset	Sample size	Dimension
Breast Wisconsin	569	30
Diabetes	768	8
Haberman	306	3
Liver Disorders	345	6
Sonar	208	60
Figure 14: (Mixture of t-banana-shaped distributions, MALA) Graphs of the MMD distance with respect to the thinning size  $m$  (with  $d = 2$ ) and with respect to  $d$  (with  $m = 300$ ) for various step sizes  $\varepsilon$ .
Figure 15: (Mixture of t-banana-shaped distributions, NUTS) Graphs of the MMD distance with respect to the thinning size  $m$  (with  $d = 2$ ) and with respect to  $d$  (with  $m = 300$ ) for various step sizes  $\varepsilon$ .
B Proof of Theorem 2.3
Assumption 2.2. For distant bimodal mixture distributions  $q$  and  $p$  satisfying Assumption 2.1, and for  $\eta \in (0,1)$ , we have  $\left|\mathrm{KSD}^2(\mathbb{P},\mathbb{Q}_L) / \mathrm{KSD}^2(\mathbb{P},\mathbb{Q}_R) - 1\right| < \eta$ .
Theorem 2.3. Let  $k_{p}$  be the Stein kernel associated with the radial kernel  $k(\mathbf{x},\mathbf{x}^{\prime}) = \phi (\| \mathbf{x} - \mathbf{x}^{\prime}\|_{2} / \ell)$ , where  $\mathbf{x},\mathbf{x}^{\prime}\in \mathbb{R}^{d}$ ,  $\ell >0$ , and  $\phi \in C^2 (\mathbb{R})$ , such that  $\phi (z)\to 0$ ,  $\phi '(z)\to 0$ , and  $\phi ''(z)\to 0$  for  $z\rightarrow \infty$ . Let  $p$  and  $q$  be two bimodal mixture distributions satisfying Assumptions 2.1 and 2.2, for any  $\eta \in (0,1)$ . We define  $w^{\star}$  as the optimal mixture weight of  $q$  with respect to the KSD distance, i.e.,  $w^{\star} = \operatorname *{argmin}_{w\in [0,1]}\mathrm{KSD}(\mathbb{P},\mathbb{Q}_w)$ . Then, for  $\mu$  large enough, we have  $|w^{\star} - \frac{1}{2}| < \frac{\eta}{2(1 - \eta)}$ .
Table 3: AUCs obtained by Stein Thinning (ST) and Regularized Stein Thinning (RST). A 10-fold cross-validation is performed and the experiments are repeated 10 times to provide uncertainties.
NUTS Sampler
m = 50	m = 100	m = 300
Dataset	ST	RST	ST	RST	ST	RST
Breast W. Diabetes	0.88 (0.020)	0.96 (0.004)	0.91 (0.023)	0.96 (0.003)	0.93 (0.008)	0.96 (0.004)
0.52 (0.009)	0.50 (0.019)	0.52 (0.021)	0.55 (0.018)	0.53 (0.015)	0.57 (0.019)
Haberman Liver Sonar	0.51 (0.038)	0.53 (0.023)	0.54 (0.033)	0.58 (0.035)	0.53 (0.034)	0.58 (0.017)
0.53 (0.044)	0.69 (0.014)	0.56 (0.038)	0.70 (0.013)	0.61 (0.039)	0.70 (0.011)
0.80 (0.021)	0.81 (0.007)	0.81 (0.009)	0.82 (0.011)	0.81 (0.011)	0.81 (0.009)
MALA Sampler
m = 50	m = 100	m = 300
Dataset	ST	RST	ST	RST	ST	RST
Breast W. Diabetes	0.68 (0.044)	0.93 (0.010)	0.72 (0.048)	0.93 (0.007)	0.72 (0.037)	0.88 (0.026)
0.51 (0.012)	0.48 (0.010)	0.53 (0.028)	0.51 (0.014)	0.53 (0.016)	0.56 (0.016)
Haberman Liver Sonar	0.52 (0.034)	0.60 (0.027)	0.53 (0.024)	0.58 (0.017)	0.55 (0.024)	0.61 (0.013)
0.54 (0.033)	0.70 (0.008)	0.55 (0.034)	0.69 (0.005)	0.57 (0.024)	0.62 (0.032)
0.80 (0.019)	0.80 (0.019)	0.80 (0.010)	0.80 (0.010)	0.81 (0.013)	0.80 (0.010)
Lemma 1. If  $k_{p}$  is the Stein kernel associated with the radial kernel  $k(\mathbf{x},\mathbf{y}) = \phi (\| \mathbf{x} - \mathbf{y}\| _2 / \ell)$ , where  $\ell >0$ ,  $\phi \in \mathcal{C}^2 (\mathbb{R})$ , and  $\mathbf{x},\mathbf{y}\in \mathbb{R}^{d}$  such that  $s_p(\mathbf{x}),s_p(\mathbf{y}) < s_0$ , then we have
$$
\begin{array}{l} \left| k _ {p} (\mathbf {x}, \mathbf {y}) \right| \leq \frac {d - 1}{\ell \| \mathbf {x} - \mathbf {y} \| _ {2}} \phi^ {\prime} (\| \mathbf {x} - \mathbf {y} \| _ {2} / \ell) + \frac {1}{\ell^ {2}} \phi^ {\prime \prime} (\| \mathbf {x} - \mathbf {y} \| _ {2} / \ell) + \frac {2 s _ {0}}{\ell} \phi^ {\prime} (\| \mathbf {x} - \mathbf {y} \| _ {2} / \ell) \\ + s _ {0} ^ {2} \phi (\left\| \mathbf {x} - \mathbf {y} \right\| _ {2} / \ell). \\ \end{array}
$$
Proof of Theorem B. We consider the mixture distributions  $p$  and  $q$  satisfying Assumption 2.1, for  $\mu > 0$  and  $r > 0$ , and assume that Assumption 2.2 is satisfied for  $\eta \in (0,1)$ . More precisely, we denote by  $q_{L}$  the distribution of the left mode of the mixture  $q$ , and similarly,  $q_{R}$  is the distribution of the right mode of  $q$ . The probability measures  $\mathbb{Q}_L$  and  $\mathbb{Q}_R$  respectively admits the densities  $q_{L}$  and  $q_{R}$ .
By definition of the KSD, we can write
$$
\mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {w}) = \int k _ {p} (\mathbf {x}, \mathbf {x} ^ {\prime}) q (\mathbf {x}) q (\mathbf {x} ^ {\prime}) d \mathbf {x} d \mathbf {x} ^ {\prime}.
$$
Additionally, given the above notations,  $q$  takes the form  $q = wq_{L} + (1 - w)q_{R}$ . Then, we can develop the KSD expression to get
$$
\begin{array}{l} \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {w}) = \int k _ {p} (\mathbf {x}, \mathbf {x} ^ {\prime}) (w q _ {L} (\mathbf {x}) + (1 - w) q _ {R} (\mathbf {x})) (w q _ {L} (\mathbf {x} ^ {\prime}) + (1 - w) q _ {R} (\mathbf {x} ^ {\prime})) d \mathbf {x} d \mathbf {x} ^ {\prime} \\ = w ^ {2} \int k _ {p} (\mathbf {x}, \mathbf {x} ^ {\prime}) q _ {L} (\mathbf {x}) q _ {L} \left(\mathbf {x} ^ {\prime}\right) d \mathbf {x} d \mathbf {x} ^ {\prime} + (1 - w) ^ {2} \int k _ {p} (\mathbf {x}, \mathbf {x} ^ {\prime}) q _ {R} (\mathbf {x}) q _ {R} \left(\mathbf {x} ^ {\prime}\right) d \mathbf {x} d \mathbf {x} ^ {\prime} \\ + 2 w (1 - w) \int k _ {p} (\mathbf {x}, \mathbf {x} ^ {\prime}) q _ {L} (\mathbf {x}) q _ {R} (\mathbf {x} ^ {\prime}) d \mathbf {x} d \mathbf {x} ^ {\prime}, \\ \end{array}
$$
where the last term follows from the symmetry of  $k_{p}$ . Finally, we have
$$
\begin{array}{l} \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {w}) = w ^ {2} \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {L}) + (1 - w) ^ {2} \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {R}) \\ + 2 w (1 - w) \int k _ {p} (\mathbf {x}, \mathbf {x} ^ {\prime}) q _ {L} (\mathbf {x}) q _ {R} (\mathbf {x} ^ {\prime}) d \mathbf {x} d \mathbf {x} ^ {\prime}, \\ \end{array}
$$
and we denote by  $\Delta_{L,R}$  the last term of this equation, which now writes
$$
\mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {w}) = w ^ {2} \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {L}) + (1 - w) ^ {2} \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {R}) + 2 w (1 - w) \Delta_ {L, R}. \tag {4}
$$
We first focus on the last term  $\Delta_{L,R}$  of this expression, which can be shown to be arbitrarily small when  $\mu$  gets large. According to Assumption 2.1, the distance between the centers of the two modes is  $2\mu$ , and both  $q_{L}$  and  $q_{R}$  have a compact support included in a ball of radius  $r$ . Consequently, for  $\mathbf{x}, \mathbf{x}' \in \mathbb{R}^d$  such that  $q_{L}(\mathbf{x}) > 0$  and  $q_{R}(\mathbf{x}') > 0$ , then  $\|\mathbf{x} - \mathbf{x}'\|_2 > 2(\mu - r)$ . Additionally, since the score  $s_p$  is continuous,  $s_p$  is bounded on a compact set, and it exists  $s_0 > 0$  such that  $s_p(\mathbf{x}) < s_0$  and  $s_p(\mathbf{x}') < s_0$ . Then, from Lemma 1, we have
$$
\begin{array}{l} | k _ {p} (\mathbf {x}, \mathbf {x} ^ {\prime}) | \leq \frac {d - 1}{2 \ell (\mu - r)} \phi^ {\prime} (\| \mathbf {x} - \mathbf {x} ^ {\prime} \| _ {2} / \ell) + \frac {1}{\ell^ {2}} \phi^ {\prime \prime} (\| \mathbf {x} - \mathbf {x} ^ {\prime} \| _ {2} / \ell) + \frac {2 s _ {0}}{\ell} \phi^ {\prime} (\| \mathbf {x} - \mathbf {x} ^ {\prime} \| _ {2} / \ell) \\ + s _ {0} ^ {2} \phi (\| \mathbf {x} - \mathbf {x} ^ {\prime} \| _ {2} / \ell), \\ \end{array}
$$
and since  $q_{L}(\mathbf{x})q_{R}(\mathbf{x}^{\prime}) = 0$  for  $\| \mathbf{x} - \mathbf{x}^{\prime}\|_{2} < 2(\mu -r)$  , we get
$$
\Delta_ {L, R} \leq \sup _ {z > 2 (\mu - r) / \ell} \left\{\frac {d - 1}{2 \ell (\mu - r)} \phi^ {\prime} (z) + \frac {1}{\ell^ {2}} \phi^ {\prime \prime} (z) + \frac {2 s _ {0}}{\ell} \phi^ {\prime} (z) + s _ {0} ^ {2} \phi (z) \right\}.
$$
By assumption,  $\phi(z) \to 0$ ,  $\phi'(z) \to 0$ , and  $\phi''(z) \to 0$  for  $z \to \infty$ , and then, we have
$$
\lim _ {\mu \to \infty} \Delta_ {L, R} = 0.
$$
Next, we reorder the terms of equation (4) to get a second-order polynomial in  $w$  as follows
$$
\begin{array}{l} \operatorname {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {w}) = w ^ {2} \left[ \operatorname {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {L}) + \operatorname {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {R}) - 2 \Delta_ {L, R} \right] \\ - 2 w \left[ \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {R}) - \Delta_ {L, R} \right] + \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {R}). \\ \end{array}
$$
Notice that the coefficient of  $w^2$  is  $\mathrm{KSD}^2(\mathbb{P}, \mathbb{Q}_{1/2}) / 4$ , and is therefore positive. Then,  $\mathrm{KSD}^2(\mathbb{P}, \mathbb{Q}_w)$  admits a unique minimum with respect to  $w$ , given by
$$
w ^ {\star} = \frac {\mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R}) - \Delta_ {L , R}}{\mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {L}) + \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R}) - 2 \Delta_ {L , R}}.
$$
We rewrite  $w^{\star}$  as follows,