SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The proofs of Theorem 2.4 and Corollary 2.5, reported in Appendix C, are built on the idea that the KSD of the empirical law of  $\{\mathbf{x}_i\}_{i=1}^n$ , has a bias of the form  $\sum_{i=1}^{m} \|s_p(\mathbf{x}_i)\|_2^2 / m^2$ . Consequently, when  $m$  is small, the bias has a strong influence on KSD estimates, which favor samples concentrated in regions of low score norm, as stationary points of  $p$ . This mechanism is illustrated in Figure 2 and Corollary 2.6 for Gaussian mixtures. In this case, Stein thinning aligns a large number of particles around the line of saddle points defined by  $x^{(1)} = 0$ , an area of low probability of the targeted mixture distribution, because of the variations of the score function, if the sample size  $m$  is small enough. From another perspective, for any sample size  $m$ , it exists a Gaussian mixture with  $\mu/\sigma$  large enough, such that Pathology II occurs. Therefore, Pathology II can appear for arbitrarily large samples  $m$ , depending on the target distribution properties.
Corollary 2.6 (KSD spurious minimum for Gaussian mixtures). Let  $k_{p}$  be the Stein kernel associated with the IMQ kernel with  $\ell > 0$ ,  $\beta \in (0,1)$ , and  $c = 1$ . Let the density  $p$  be a Gaussian mixture model of two components with equal weights, respectively centered in  $(-\mu, \mathbf{0}_{d-1})$  and  $(\mu, \mathbf{0}_{d-1})$ , of variance  $\sigma^2 \mathbf{I}_{\mathbf{d}}$ , and let  $\nu = \mu / \sigma$ . If  $\nu > 1$  and  $0 \leq s_0 < [\nu \sqrt{\nu^2 - 1} - \ln(\nu + \sqrt{\nu^2 - 1})] / \mu$ , then for any  $\{\mathbf{x}_i\}_{i=1}^m \subset \mathcal{M}_{s_0}$  of empirical measure  $\mathbb{Q}_m$ , we have
(i)  $\mathrm{KSD}^2\big(\mathbb{P},\mathbb{Q}_m\big) <   \mathbb{E}\big[\mathrm{KSD}^2\big(\mathbb{P},\mathbb{P}_m\big)\big]ifm$  and  $s_0$  satisfy  $m <   1 + \frac{\mathbb{E}[||s_p(\mathbf{X})||_2^2] - s_0^2}{2\beta d / \ell^2 + 2\beta s_0 / \ell + s_n^2},$
(ii) there exists three disjoint intervals  $I_{-\mu}, I_0, I_\mu \subset \mathbb{R}$ , respectively centered around  $-\mu$ ,  $0$ , and  $\mu$ , such that  $x_1^{(1)}, \ldots, x_m^{(1)} \in I_{-\mu} \cup I_0 \cup I_\mu$ .
3 Regularized Stein Thinning
Stein thinning suffers from two main pathologies, analyzed in Section 2. In a word, Pathology I comes from the insensitivity of the score to the relative weights of distant modes, whereas Pathology II originates from the variations of the score norm, which do not differentiate local minimum from local maximum of the target distribution. We propose to regularize the KSD distance to fix these two problems, using terms that are highly sensitive to the type of stationary point and the relative weights of modes. The proposed algorithm is first introduced in Subsection 3.1, then theoretical properties are discussed in Subsection 3.2, and finally the good empirical performance will be shown in Section 4.
Entropic regularization. In order to compensate the blindness of the KSD to mode proportions in multimodal distributions, we introduce the following entropic regularized KSD, denoted by  $\mathrm{KSD}_{\lambda}$ , and defined as  $\mathrm{KSD}_{\lambda}^{2}(\mathbb{P},\mathbb{Q}) = \mathbb{E}[k_{p}(\mathbf{Z},\mathbf{Z}^{\prime})] - \lambda \mathbb{E}[\log (p(\mathbf{Z}))]$ , where  $\mathbf{Z}$  and  $\mathbf{Z}^{\prime}$  have probability law  $\mathbb{Q}$ , and  $\mathbb{P}$  admits the density  $p$ . In our Bayesian setting,  $\mathbb{E}[\log (p(\mathbf{Z}))]$  is known up to an additive constant since the normalization factor of  $p$  is not tractable. However, it is possible to use  $\mathrm{KSD}_{\lambda}^{2}(\mathbb{P},\mathbb{Q})$  as the objective function of the Stein thinning algorithm, as the greedy selection of particles to optimize this quantity does not rely on the unknown additive constant. The main idea of this entropic regularization is that  $-\log (p(\mathbf{x}))$  takes higher values in modes of smaller probability, and therefore provides the relative mode weight information, which is missing in the KSD distance. More precisely, modes with smaller weights take smaller density values, and are therefore more penalized than modes of higher weights. Therefore, with such entropic penalization, regularized Stein thinning tends to select particles in modes of higher weights more frequently than in modes of smaller weights, and we recover appropriate proportions.
Laplacian correction. Chen et al. [2018] and Riabiz et al. [2022] have noticed that the term  $k_{p}(\mathbf{x}_{i},\mathbf{x}_{i})$ , which naturally appears in the empirical kernelized Stein discrepancy with the Langevin operator, can be interpreted as a regularization term. For example, Stein thinning does not select particles in the burn-in period of an MCMC output thanks to this regularization. However, this term  $k_{p}(\mathbf{x}_{i},\mathbf{x}_{i})$  is also responsible for Pathology II, of samples concentrated in stationary points of  $p$ , as shown in Theorem 2.4. Therefore, we add a second regularization term to compensate the weaknesses of  $k_{p}(\mathbf{x}_{i},\mathbf{x}_{i})$ , by penalizing particles located at local minimum and saddle points of the density  $p$ . Such points are located in areas of convexity of the target distribution, which can thus be detected with the positive values of the Laplacian of the density. Therefore, using the truncated Laplacian operator  $\Delta^{+}f(\mathbf{x}) = \sum_{j = 1}^{d}\left(\partial^{2}f(\mathbf{x}) / \partial x^{(j)2}\right)^{+}$  for a function  $f\in \mathcal{C}^2 (\mathbb{R}^d)$ , we propose the L-KSD estimate with a Laplacian correction for densities  $p\in \mathcal{C}^2 (\mathbb{R}^d)$ , defined by
$$
\mathrm {L - K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {m}) = \frac {1}{m ^ {2}} \sum_ {i \neq j} ^ {m} k _ {p} (\mathbf {x} _ {i}, \mathbf {x} _ {j}) + \frac {1}{m ^ {2}} \sum_ {i = 1} ^ {m} \left[ k _ {p} (\mathbf {x} _ {i}, \mathbf {x} _ {i}) + \Delta^ {+} \log (p (\mathbf {x} _ {i})) \right].
$$
Regularized Stein thinning. Overall, we obtain the following estimate for the entropic regularized KSD with Laplacian correction  $\mathrm{L - KSD}_{\lambda}^{2}(\mathbb{P},\mathbb{Q}_{m}) = \mathrm{L - KSD}^{2}(\mathbb{P},\mathbb{Q}_{m}) - \frac{\lambda}{m}\sum_{i = 1}^{m}\log (p(\mathbf{x}_{i}))$ . Then, at each iteration  $t\in \{1,\ldots ,m\}$ , the regularized Stein thinning selects the particle index  $\pi_t\in \{1,\dots ,n\}$  to greedily minimize
$$
k _ {p} (\mathbf {x} _ {\pi_ {t}}, \mathbf {x} _ {\pi_ {t}}) + \Delta^ {+} \log (p (\mathbf {x} _ {\pi_ {t}})) - \lambda t \log (p (\mathbf {x} _ {\pi_ {t}})) + 2 \sum_ {j = 1} ^ {t - 1} k _ {p} (\mathbf {x} _ {\pi_ {j}}, \mathbf {x} _ {\pi_ {t}}).
$$
Finally, Figure 3 illustrates the performance of regularized Stein thinning to fix the two pathologies analyzed in Section 2, in the case of Example 1 with Gaussian mixtures. Indeed, the top panel of Figure 3 shows that the majority of particles are selected in the right mode, as expected from the target distribution with  $w = 0.2$ . More precisely, an average proportion of 0.11 of the particles are located in the left mode over 100 repetitions of the procedure (0.89 in the right mode), with a standard deviation of 0.03. For the value choice of  $\lambda$ , we refer to the next subsection and the experimental Section 4. On the bottom panel of Figure 3, we observe that no particle is now selected on the line  $x^{(1)} = 0$ , as expected from the target Gaussian mixture distribution.
Remark 3.1. The truncated Laplacian operator is simply given by the trace of the Hessian matrix, where negative components are set to 0. It follows that the computational cost the regularized algorithm is similar to the original Stein thinning.
Remark 3.2. The Laplacian correction of  $k_{p}$  introduces second-order derivatives of  $p$  in the Stein discrepancy, and therefore enables to differentiate local minimum and saddle points of the density  $p$  from its local maximum. A natural approach to introduce second-order derivatives of  $p$  in KSD estimates, is to define the Stein discrepancy using second-order operators. A Laplacian Stein operator [Oates et al., 2017] is derived in Appendix G, but experiments show that this strategy is not efficient to fix Pathologies I & II.
Figure 3: Pathologies fixed by the regularized Stein thinning.
3.2 Theoretical properties
This subsection is dedicated to the theoretical analysis of regularized Stein thinning. First, we show that the proposed algorithm now enjoys good properties regarding Pathologies I and II, and thus mitigates the identified problems of the original Stein thinning. Secondly, we extend the convergence analysis of Riabiz et al. [2022] for the post-treatment of MCMC output, to show the convergence of the empirical law output by regularized Stein thinning towards the target probability measure.
Entropic regularization. In the previous section, Theorem 2.3 highlights how Pathology I of mode proportion blindness originates from the score insensitivity to mode weights. On the other hand, the entropic regularization is directly built on the target density, and therefore strongly depends on the mode weights. In the same setting of Assumption 2.1, required for Pathology I to occur with the original algorithm, the following Theorem 3.3 shows that the entropic regularized KSD is minimized for the appropriate target weight, with the suitable regularization strength  $\lambda$ . Notice that Theorem 3.3, proved in Appendix D, is valid if  $\mathbb{E}[\log(p(\mathbf{Z}_L))] \neq \mathbb{E}[\log(p(\mathbf{Z}_R))]$  with  $\mathbf{Z}_L \sim \mathbb{Q}_L$  and  $\mathbf{Z}_R \sim \mathbb{Q}_R$ , otherwise the impact of the entropic regularization on  $w_{\lambda}^{\star}$  vanishes. However, as  $w_p \neq 1/2$  is required in Assumption 2.1 for Pathology I to occur,  $p$  is asymmetric, and  $\mathbb{E}[\log(p(\mathbf{Z}_L))] = \mathbb{E}[\log(p(\mathbf{Z}_R))]$  is only possible in very specific cases. Theorem 3.3 clearly shows that the regularized entropic KSD is sensitive to the weights of distant modes. Efficient strategies to choose the regularization strength will be first discussed in the asymptotic analysis below, and then in the experiments of the next section.
Theorem 3.3. Let  $k_{p}$  be the Stein kernel associated with the radial kernel  $k(\mathbf{x},\mathbf{x}^{\prime}) = \phi (\| \mathbf{x} - \mathbf{x}^{\prime}\|_{2} / \ell)$ , where  $\mathbf{x},\mathbf{x}'\in \mathbb{R}^d$ ,  $\ell >0$ , and  $\phi \in \mathcal{C}^2 (\mathbb{R})$ . Let  $p$  and  $q$  be two bimodal mixture distributions satisfying Assumption 2.1. We define  $w_{\lambda}^{\star}$  as the optimal mixture weight of  $q$  with respect to the entropic regularized KSD distance, i.e.,  $w_{\lambda}^{\star} = \operatorname *{argmin}_{w\in [0,1]}\mathrm{KSD}_{\lambda}(\mathbb{P},\mathbb{Q}_w)$ . If  $\mathbb{E}[\log (p(\mathbf{Z}_L))]\neq \mathbb{E}[\log (p(\mathbf{Z}_R))]$
where  $\mathbf{Z}_L\sim \mathbb{Q}_L$  and  $\mathbf{Z}_R\sim \mathbb{Q}_R$ , it exists  $\lambda \in \mathbb{R}$  such that  $w_{\lambda}^{\star} = w_{p}$ .
Laplacian correction. First, we stress that the L-KSD is a strongly consistent estimate of the KSD distance, where the proof follows from the law of large numbers. Therefore, the Laplacian correction introduced in the L-KSD estimate does not undermine the good asymptotic properties of the KSD distance. Secondly, the following theorem shows that samples concentrated in local minimum or saddle points of the target distribution and of low density values, are well identified by the L-KSD as samples of worse quality than those truly sampled from the target. Consequently, the Laplacian correction fixes Pathology II, previously formalized in Theorem 2.4.
Theorem 3.4. Let  $k_{p}$  be the Stein kernel associated with the IMQ kernel with  $\ell > 0$ ,  $\beta \in (0,1)$ , and  $c = 1$ . For  $m \geq 2$ , let  $\{\mathbf{x}_i\}_{i=1}^m \subset \mathbb{R}^d$  be a set of points located at  $\mathbf{x}_0$ , a local minimum or saddle point of  $p$ , and of empirical measure  $\mathbb{Q}_m$ . Then, we have  $\mathrm{L - KSD}^2(\mathbb{P}, \mathbb{Q}_m) > \mathbb{E}[\mathrm{L - KSD}^2(\mathbb{P}, \mathbb{P}_m)]$ , if the density at  $\mathbf{x}_0$  satisfies  $p(\mathbf{x}_0) < \Delta^+ p(\mathbf{x}_0) / (\mathbb{E}[||s_p(\mathbf{X})||_2^2] + \mathbb{E}[\Delta^+ \log p(\mathbf{X})])$ .
Convergence of regularized Stein thinning. While regularized Stein thinning fixes finite sample size pathologies, the asymptotic properties of Stein thinning are also preserved. Indeed, if the initial set of particles is drawn from a different distribution than the target using a Markov chain Monte Carlo, Theorem 3.6 states that the empirical measure of the sample obtained with regularized Stein Thinning, converges towards the target measure  $\mathbb{P}$ , and thus extends the results of Riabiz et al. [2022]. Notice that the weak convergence of a sequence of probability measure is denoted by  $\Rightarrow$ , and that distantly dissipative distributions are defined in Definition 1.1. The required assumption below, essentially states mild integrability conditions, and enforces that the MCMC output is not too far from a sample drawn from  $p$ —see Appendix F for additional details.
Assumption 3.5. Let  $\mathbb{Q}$  be a probability distribution on  $\mathbb{R}^d$ , such that  $\mathbb{P}$  is absolutely continuous with respect to  $\mathbb{Q}$ . Let  $\{\mathbf{Z}_i\}_{i\in \mathbb{N}}\subset \mathbb{R}^d$  be a  $\mathbb{Q}$ -invariant, time-homogeneous Markov chain, generated using a  $V$ -uniformly ergodic transition kernel, such that  $V(\mathbf{x})\geq \frac{d\mathbb{P}}{d\mathbb{Q}}\sqrt{2\beta d / \ell^2 + \|s_p(\mathbf{x})\|_2^2}$ . Suppose that, for some  $\gamma >0$ ,  $\sup_{i\in \mathbb{N}}\mathbb{E}[e^{\gamma |\log (p(\mathbf{Z}_i))|}] < \infty$ ,  $\sup_{i\in \mathbb{N}}\mathbb{E}[e^{\gamma \Delta^{+}\log p(\mathbf{Z}_i)}] < \infty$ ,
$$
\sup_{i\in \mathbb{N}}\mathbb{E}\Big[e^{\gamma \max (1,\frac{d\mathbb{P}}{d\mathbb{Q}} (\mathbf{Z}_{i})^{2})(\frac{2\beta d}{\ell^{2}} + \| s_{p}(\mathbf{Z}_{i})\|_{2}^{2})}\Big] <   \infty ,\sup_{i\in \mathbb{N}}\mathbb{E}\Big[\frac{d\mathbb{P}}{d\mathbb{Q}} (\mathbf{Z}_{i})\sqrt{\frac{2\beta d}{\ell^{2}} + \|s_{p}(\mathbf{Z}_{i})\|_{2}^{2}} V(\mathbf{Z}_{i})\Big] <   \infty .
$$
Theorem 3.6. Let  $\mathbb{P}$  be a distantly dissipative probability measure, that admits the density  $p\in$ $\mathcal{C}^2 (\mathbb{R}^d)$ ,  $k_{p}$  be the Stein kernel associated with the IMQ kernel where  $\ell ,c > 0,\beta \in (0,1)$ . Let  $\{\mathbf{Z}_i\}_{i\in \mathbb{N}}\subset \mathbb{R}^d$  be a Markov chain satisfying Assumption 3.5,  $\pi$  be the index sequence of length  $m_{n}$  generated by regularized Stein thinning, and  $\mathbb{Q}_{m_n}$  be the empirical measure of  $\{\mathbf{Z}_{\pi_i}\}_{i = 1}^{m_n}$ . If  $\log (n)^{\alpha} < m_n < n$ , with any  $\alpha >1$ , and  $\lambda_{m_n} = o(\log (m_n) / m_n)$ , then we have almost surely  $\mathbb{Q}_{m_n}\underset {n\to \infty}{\Longrightarrow}\mathbb{P}$ .
Theorem 3.6, proved in Appendix F, provides us with interesting insights about the entropic regularization strength  $\lambda$ . We already know that  $\lambda$  should be chosen with a rate at least as fast as  $O(1 / m)$ , to avoid the introduction of a higher a bias in the L-KSD than the original KSD. Indeed, for a sample drawn from the true target distribution  $p$ , this bias  $\mathbb{E}[\mathrm{L - KSD}^2 (\mathbb{P},\mathbb{P}_m)]$  takes the form  $\mathbb{E}[k_p(\mathbf{X},\mathbf{X})] / m + \mathbb{E}[\Delta^{+}\log (p(\mathbf{X}))] / m - \lambda \mathbb{E}[\log (p(\mathbf{X}))]$ . Therefore, for slower rates of  $\lambda$  than  $O(1 / m)$ , trivial samples concentrated at a local maximum of the target  $p$ , can have smaller L-KSD than samples drawn from  $p$ . Then, Theorem 3.6 states that, for our ultimate application of MCMC post-processing, the Stein thinning sample distribution converges towards the target for such  $\lambda$  rates of  $O(1 / m)$  or faster. In practice, in our Bayesian setting, it is not possible to fine tune this parameter  $\lambda$  because no metric is available to assess the Stein thinning quality for various values of  $\lambda$ , as already mentioned in the case of the bandwidth parameter  $\ell$ . In addition, we cannot theoretically determine which exact range of values of  $\lambda$  leads to good thinned samples in a finite sample regime. However, we will see in the experiments of the following section that both slower and faster  $\lambda$  rates than  $O(1 / m)$  lead to samples of degraded quality. Therefore, we set  $\lambda = 1 / m$  in the regularized Stein thinning, to ensure good empirical performance and the algorithm convergence.
4 Empirical Assessment
This section shows how regularized Stein thinning outperforms the original algorithm through three batches of experiments: mixtures of standard distributions using exact or MCMC sampling, and Bayesian logistic regression on real datasets. For the experiments considered in Sections 4.2 and 4.3, two Metropolis-Hastings samplers are considered with the Metropolis-Adjusted Langevin Algorithm (MALA) and the No-U-Turn sampler (NUTS). We use the IMQ kernel with  $\ell$  set with the median heuristic,  $\beta = 1/2$ , and  $c = 1$ , as recommended in Chen et al. [2018], Riabiz et al. [2022]. We also set the regularization parameter with the default value of  $\lambda = 1/m$ . Notice that additional experiments are provided in Appendix A, and that the code is available at https://gitlab.com/drti/kernax.
When the target distribution is known, the efficiency of the Stein thinning algorithms are assessed by computing the MMD distance (see Equation (1)) between a large sample drawn from the target distribution and the thinned samples. More specifically, we use the following closed-form expression of the MMD [Gretton et al., 2006] with  $\mathbf{X}$ ,  $\mathbf{X}' \sim \mathbb{P}$  and  $\mathbf{Z}$ ,  $\mathbf{Z}' \sim \mathbb{Q}$ ,
$$
\operatorname {M M D} _ {k} ^ {2} (\mathbb {P}, \mathbb {Q}) = \mathbb {E} [ k (\mathbf {X}, \mathbf {X} ^ {\prime}) ] + \mathbb {E} [ k (\mathbf {Z}, \mathbf {Z} ^ {\prime}) ] - 2 \mathbb {E} [ k (\mathbf {X}, \mathbf {Z}) ], \tag {3}
$$
where the kernel function  $k$  is chosen as the distance-induced kernel studied by Sejdinovic et al. [2013] and given by  $k(\mathbf{x},\mathbf{x}^{\prime}) = \| \mathbf{x}\|_{2} + \| \mathbf{x}^{\prime}\|_{2} - \| \mathbf{x} - \mathbf{x}^{\prime}\|_{2}$ , for  $\mathbf{x},\mathbf{x}^{\prime}\in \mathbb{R}^{d}$ . In this setting, the MMD reduces to the well known energy distance, as shown by Sejdinovic et al. [2013].
4.1 Gaussian mixtures with exact sampling
As a first batch of experiments, we build on Example 1 and consider more complicated two-dimensional Gaussian mixtures to further illustrate the correction of Pathologies I & II. The first Gaussian mixture is made of four modes located at  $\pmb{\mu}_1 = (-3,3)$ ,  $\pmb{\mu}_2 = (-3,3)$ ,  $\pmb{\mu}_3 = (3,3)$ , and  $\pmb{\mu}_4 = (3,-3)$ , and with weights  $w_1 = w_2 = 0.1$ , and  $w_3 = w_4 = 0.4$ , respectively. The second mixture is taken from [Qiu and Wang, 2023]. It is made of 6 equally weighted Gaussian distributions centered at  $\pmb{\mu}_i = (3\cos(2\pi(i-1)/6), 3\sin(2\pi(i-1)/6))$ , for  $i = 1,\ldots,6$ . For both experiments, we rely on exact Monte Carlo sampling to generate  $n = 3000$  observations, and apply Stein thinning and its regularized variant to select  $m = 300$  particles. The observed samples and the selected particles are shown in Figure 4. The first example shows that vanilla Stein thinning does not capture the right proportions, while the regularized variant appropriately penalizes modes with lower weights. The second example illustrates Pathology II, which is corrected by the regularized Stein thinning.
Figure 4: Gaussian mixtures with exact Monte Carlo sampling. Solutions (red dots) obtained by Stein thinning and its regularized variant.
4.2 Banana-shaped and Gaussian mixtures with MCMC sampling
We consider a mixture of two distant modes of  $d$ -dimensional banana-shaped distributions with t-tails and unbalanced weights [Haario et al., 1999, Pompe et al., 2020], illustrated in Figure 5, and precisely defined in Appendix A.2. We sample this target banana mixture with both MALA and NUTS using three different step sizes  $\varepsilon$  and  $10^{5}$  iterations. The generated samples are post-processed with the Stein thinning and regularized Stein thinning algorithms, and their performances are compared with the MMD between the post-processed samples and large samples drawn from the known target banana mixture. This experiment is run for various thinning sizes  $m$  and dimensions  $d$ , with 20 repetitions to quantify uncertainties. The results obtained with the MALA sampler are shown in Figure 6: the regularized Stein thinning clearly generates samples of higher quality than the vanilla Stein thinning. Additionally, an example of post-processed MALA output is depicted in Figure 5, together with a heatmap of the Laplacian correction. On the left panel of Figure 5, we see that pathologies are especially strong in this experiment, with a large number of particles lying between the two modes in a region of low probability. On the right panels, we observe that regularized Stein thinning fix pathologies. Similar results were obtained with NUTS and are reported in Appendix A.2 for brevity. Next, we conduct the same experiments for a  $d$ -dimensional Gaussian mixture of four modes with different variances, as detailed in Appendix A.2. Again, Figure 6 shows the better
Figure 5: t-banana-shaped mixture ( $d = 10$ ). From left to right: solutions obtained with standard and regularized Stein thinning with contour lines of  $p$ , and heatmap of the Laplacian correction.
Figure 6: (MALA) Graphs of the MMD with respect to the thinning size  $m$  (for  $d = 2$ ) and with respect to  $d$  (for  $m = 300$ ). Left two panels: banana mixture. Right two panels: Gaussian mixture.
performance of regularized Stein thinning. Besides, we take advantage of this last experiment to explore other regularization rates than our default  $\lambda = 1 / m$ . Figure 12 in Appendix A.2 shows that a slower rate of  $\lambda = 1 / \log(m)$ , which violates the convergence assumptions of Theorem 3.6, has significantly worse performance than the original Stein thinning. On the other hand, with a faster rate than  $1 / m$  such as  $1 / m^2$ , the effect of the entropic regularization disappears, and we recover similar results than the original Stein thinning. This supports that the default value of  $\lambda = 1 / m$  is an efficient heuristic, since slower and faster rates of  $\lambda$  strongly degrade the algorithm performance.
4.3 Bayesian logistic regression
We now compare the two Stein thinning algorithms in the Bayesian logistic regression setting for binary classification, since such problem usually involves multimodal posterior—see, e.g., Gershman et al. [2012], Liu and Wang [2016], Fong et al. [2019], Korba et al. [2021]. Given a dataset  $\mathcal{D}_N = \{(X_i,Y_i)\}_{i=1}^N$  made of  $N$  pairs of features  $\mathbf{X}_i \in \mathbb{R}^d$  and labels  $Y_i \in \{0,1\}$ , the probability that  $Y_i$  is of class 1 is given by  $p(Y_i = 1 | X_i, \beta, \beta_0) = 1 / (1 + \exp(-\beta_0 - \boldsymbol{\beta}^T X_i))$ , for some parameters  $\boldsymbol{\theta} = (\beta_0, \beta) \in \mathbb{R}^{d+1}$ . The prior distributions of the weight vector  $\boldsymbol{\theta}$  is assumed to be Gaussian,  $p(\beta^{(j)} | \gamma^{(j)}) = \mathcal{N}(\beta^{(j)} | 0, 1 / \gamma^{(j)})$ , and a Gamma prior with parameters  $(a, b)$  is chosen for the precision  $\gamma^{(j)}$ . Following [Fong et al., 2019], the hyperparameters are chosen as  $a = b = 1$ .