SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} w ^ {\star} = \frac {1 / 2 \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R}) + 1 / 2 \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {L}) - \Delta_ {L , R} + 1 / 2 \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R}) - 1 / 2 \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {L})}{\mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {L}) + \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R}) - 2 \Delta_ {L , R}} \\ = \frac {1}{2} + \frac {1}{2} \frac {\mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R}) - \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {L})}{\mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {L}) + \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R}) - 2 \Delta_ {L , R}} \\ = \frac {1}{2} + \frac {1}{2} \frac {1 - \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {L}) / \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R})}{1 + \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {L}) / \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R}) - 2 \Delta_ {L , R} / \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R})} \\ = \frac {1}{2} + \frac {1}{2} \frac {1 - \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {L}) / \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R})}{2 (1 - \Delta_ {L , R} / \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R})) + (\mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {L}) / \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R}) - 1)}. \\ \end{array}
$$
We can deduce the following bound
$$
\left| w ^ {\star} - \frac {1}{2} \right| \leq \frac {1}{2} \frac {\left| \mathrm {K S D} ^ {2} \left(\mathbb {P} , \mathbb {Q} _ {L}\right) / \mathrm {K S D} ^ {2} \left(\mathbb {P} , \mathbb {Q} _ {R}\right) - 1 \right|}{\left| 2 \left(1 - \Delta_ {L , R} / \mathrm {K S D} ^ {2} \left(\mathbb {P} , \mathbb {Q} _ {R}\right)\right) + \left(\mathrm {K S D} ^ {2} \left(\mathbb {P} , \mathbb {Q} _ {L}\right) / \mathrm {K S D} ^ {2} \left(\mathbb {P} , \mathbb {Q} _ {R}\right) - 1\right) \right|}. \tag {5}
$$
According to Assumption 2.2, with  $0 < \eta < 1$
$$
\left| \frac {\mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {L})}{\mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R})} - 1 \right| <   \eta ,
$$
which gives an upper bound for the numerator of the right hand side of inequality (5). Additionally, for  $\mu$  large enough,  $\Delta_{R,L}$  is arbitrarily small, and in particular, we can have  $\Delta_{R,L} < \mathrm{KSD}^2 (\mathbb{P},\mathbb{Q}_R) / 2$ , and then  $2(1 - \Delta_{L,R} / \mathrm{KSD}^2 (\mathbb{P},\mathbb{Q}_R)) > 1$ . Next, we use the triangle inequality to get
$$
\begin{array}{l} \left. \left| 2 \left(1 - \Delta_ {L, R} / \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {R})\right) + \left(\mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {L}) / \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {R}) - 1\right) \right| \right. \\ \geq 2 \left(1 - \Delta_ {L, R} / \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {R})\right) - | \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {L}) / \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {R}) - 1 | \\ \geq 1 - \eta , \\ \end{array}
$$
where the last inequality is obtained using  $2(1 - \Delta_{L,R} / \mathrm{KSD}^2 (\mathbb{P},\mathbb{Q}_R)) > 1$  for  $\mu$  large enough, and Assumption 2.2 again. Finally, this lower bound on the denominator and the upper bound on the numerator combined with inequality (5) give
$$
\left| w ^ {\star} - \frac {1}{2} \right| <   \frac {\eta}{2 (1 - \eta)}.
$$
Proof of Lemma 1. We consider  $\mathbf{x},\mathbf{y}\in \mathbb{R}^d$ , such that  $\mathbf{x}\neq \mathbf{y}$ ,  $s_p(\mathbf{x}) < s_0$  and  $s_p(\mathbf{y}) < s_0$ . From Equation (2) of the main article, we derive the Stein kernel obtained for a radial kernel  $\phi (\| \mathbf{x} - \mathbf{y}\| _2 / \ell)$  where  $\ell >0$  and  $\phi \in \mathcal{C}^2 (\mathbb{R})$ , and get
$$
\begin{array}{l} k _ {p} (\mathbf {x}, \mathbf {y}) = \frac {1 - d}{\ell \| \mathbf {x} - \mathbf {y} \| _ {2}} \phi^ {\prime} (\| \mathbf {x} - \mathbf {y} \| _ {2} / \ell) - \frac {1}{\ell^ {2}} \phi^ {\prime \prime} (\| \mathbf {x} - \mathbf {y} \| _ {2} / \ell) \\ - \left(s _ {p} (\mathbf {x}) - s _ {p} (\mathbf {y})\right) \cdot \left(\mathbf {x} - \mathbf {y}\right) \frac {\phi^ {\prime} \left(\| \mathbf {x} - \mathbf {y} \| _ {2} / \ell\right)}{\ell \| \mathbf {x} - \mathbf {y} \| _ {2}} + \left(s _ {p} (\mathbf {x}) \cdot s _ {p} (\mathbf {y})\right) \phi \left(\| \mathbf {x} - \mathbf {y} \| _ {2} / \ell\right). \\ \end{array}
$$
Using Cauchy-Schwartz inequality, we have  $s_p(\mathbf{x}) \cdot s_p(\mathbf{y}) \leq s_0^2$ , and
$$
\frac {(s _ {p} (\mathbf {x}) - s _ {p} (\mathbf {y})) \cdot (\mathbf {x} - \mathbf {y})}{\ell \| \mathbf {x} - \mathbf {y} \| _ {2}} \leq \frac {2 s _ {0}}{\ell}.
$$
Overall, we obtain the following bound
$$
\begin{array}{l} \left| k _ {p} (\mathbf {x}, \mathbf {y}) \right| \leq \frac {1 - d}{\ell \| \mathbf {x} - \mathbf {y} \| _ {2}} \phi^ {\prime} \left(\| \mathbf {x} - \mathbf {y} \| _ {2} / \ell\right) + \frac {1}{\ell^ {2}} \phi^ {\prime \prime} \left(\| \mathbf {x} - \mathbf {y} \| _ {2} / \ell\right) + \frac {2 s _ {0}}{\ell} \phi^ {\prime} \left(\| \mathbf {x} - \mathbf {y} \| _ {2} / \ell\right) \\ + s _ {0} ^ {2} \phi (\left\| \mathbf {x} - \mathbf {y} \right\| _ {2} / \ell). \\ \end{array}
$$
C Proofs of Theorem 2.4, Corollary 2.5, and Corollary 2.6
Theorem 2.4 (KSD spurious minimum). Let  $k_{p}$  be the Stein kernel associated with the IMQ kernel with  $\ell > 0$ ,  $\beta \in (0,1)$ , and  $c = 1$ . Let  $\{\mathbf{x}_i\}_{i=1}^m \subset \mathcal{M}_{s_0} = \{\mathbf{x} \in \mathbb{R}^d : \| s_p(\mathbf{x}) \|_2 \leq s_0\}$  be a fixed set of points of empirical measure  $\mathbb{Q}_m = \frac{1}{m} \sum_{i=1}^m \delta(\mathbf{x}_i)$ , with  $s_0 \geq 0$  and  $m \geq 2$ . We have  $\mathrm{KSD}^2(\mathbb{P}, \mathbb{Q}_m) < \mathbb{E}[\mathrm{KSD}^2(\mathbb{P}, \mathbb{P}_m)]$ , if the score threshold  $s_0$  and the sample size  $m$  are small enough to satisfy  $m < 1 + (\mathbb{E}[|s_p(\mathbf{X})|^2] - s_0^2) / (2\beta d / \ell^2 + 2\beta s_0 / \ell + s_0^2)$ .
Corollary 2.6 (KSD spurious minimum for Gaussian mixtures). Let  $k_{p}$  be the Stein kernel associated with the IMQ kernel with  $\ell > 0$ ,  $\beta \in (0,1)$ , and  $c = 1$ . Let the density  $p$  be a Gaussian mixture model of two components with equal weights, respectively centered in  $(-\mu, \mathbf{0}_{d-1})$  and  $(\underline{\mu}, \mathbf{0}_{d-1})$ , of variance  $\sigma^2 \mathbf{I}_{\mathbf{d}}$ , and let  $\nu = \mu / \sigma$ . If  $\nu > 1$  and  $0 \leq s_0 < [\nu \sqrt{\nu^2 - 1} - \ln(\nu + \sqrt{\nu^2 - 1})] / \mu$ , then for any  $\{\mathbf{x}_i\}_{i=1}^m \subset \mathcal{M}_{s_0}$  of empirical measure  $\mathbb{Q}_m$ , we have
(i)  $\mathrm{KSD}^2\big(\mathbb{P},\mathbb{Q}_m\big) <   \mathbb{E}\big[\mathrm{KSD}^2\big(\mathbb{P},\mathbb{P}_m\big)\big]ifm$  and  $s_0$  satisfy  $m <   1 + \frac{\mathbb{E}[||s_p(\mathbf{X})||_2^2] - s_0^2}{2\beta d / \ell^2 + 2\beta s_0 / \ell + s_0^2},$
(ii) there exists three disjoint intervals  $I_{-\mu}, I_0, I_\mu \subset \mathbb{R}$ , respectively centered around  $-\mu$ ,  $0$ , and  $\mu$ , such that  $x_1^{(1)}, \ldots, x_m^{(1)} \in I_{-\mu} \cup I_0 \cup I_\mu$ .
Lemma 2. Let  $k_{p}$  be the Stein kernel associated to the IMQ kernel, with parameters  $\ell > 0$ ,  $\beta \in (0,1)$  and  $c = 1$ . For  $s_0 \geq \min_{\mathbf{x} \in \mathbb{R}^d} \| s(\mathbf{x}) \|_2$ , and  $\mathbf{x}, \mathbf{y} \in \mathcal{M}_{s_0}$ , we have
$$
k _ {p} (\mathbf {x}, \mathbf {y}) \leq \frac {2 \beta d}{\ell^ {2}} + \frac {2 \beta s _ {0}}{\ell} + s _ {0} ^ {2}.
$$
Proof of Theorem 2.4. By definition of the kernelized Stein discrepancy between the target distribution  $\mathbb{P}$  and the empirical measure  $\mathbb{P}_m = \frac{1}{m}\sum_{i = 1}^{m}\delta (\mathbf{X}_i)$ , we have
$$
\mathbb {E} [ \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {P} _ {m}) ] = \frac {1}{m ^ {2}} \sum_ {i, j = 1} ^ {m} \mathbb {E} [ k _ {p} (\mathbf {X} _ {i}, \mathbf {X} _ {j}) ]
$$
In what follows,  $k_{p}$  denotes the Stein kernel obtained for the inverse multi-quadratic kernel function, i.e.,
$$
\begin{array}{l} k _ {p} (\mathbf {x}, \mathbf {y}) = - \frac {4 \beta (\beta + 1)}{\ell^ {4}} \| \mathbf {x} - \mathbf {y} \| _ {2} ^ {2} (1 + \| \mathbf {x} - \mathbf {y} \| _ {2} ^ {2} / \ell^ {2}) ^ {- \beta - 2} \\ + \frac {2 \beta}{\ell^ {2}} (d + (s _ {p} (\mathbf {x}) - s _ {p} (\mathbf {y})) \cdot (\mathbf {x} - \mathbf {y})) (1 + \| \mathbf {x} - \mathbf {y} \| _ {2} ^ {2} / \ell^ {2}) ^ {- \beta - 1} \\ + \left(s _ {p} (\mathbf {x}) \cdot s _ {p} (\mathbf {y})\right) \left(1 + \| \mathbf {x} - \mathbf {y} \| _ {2} ^ {2} / \ell^ {2}\right) ^ {- \beta}. \\ \end{array}
$$
Given that  $\mathbf{X}_i$  and  $\mathbf{X}_j$  are independent random variables that follow the distribution  $\mathbb{P}$ , one has  $\mathbb{E}[k_p(\mathbf{X}_i, \mathbf{X}_j)] = 0$  for any  $i \neq j$ . Using this property and the closed-form expression of the Stein kernel  $k_p$ , it is found that
$$
\begin{array}{l} \mathbb {E} \left[ \mathrm {K S D} ^ {2} \left(\mathbb {P}, \mathbb {P} _ {m}\right) \right] = \frac {1}{m ^ {2}} \sum_ {i = 1} ^ {m} \mathbb {E} \left[ k _ {p} \left(\mathbf {X} _ {i}, \mathbf {X} _ {i}\right) \right] \\ = \frac {1}{m} \mathbb {E} \left[ k _ {p} (\mathbf {X}, \mathbf {X}) \right] \\ = \frac {2 \beta d}{m \ell^ {2}} + \frac {\mathbb {E} [ \| s _ {p} (\mathbf {X}) \| _ {2} ^ {2} ]}{m}, \\ \end{array}
$$
where  $\mathbf{X} \sim \mathbb{P}$ . On the other hand, the kernelized Stein discrepancy between the target  $\mathbb{P}$  and the empirical distribution  $\mathbb{Q}_m = \frac{1}{m} \sum_{i=1}^{m} \delta(\mathbf{x}_i)$  is given by
$$
\begin{array}{l} \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {m}) = \frac {1}{m ^ {2}} \sum_ {i, j = 1} ^ {m} k _ {p} (\mathbf {x} _ {i}, \mathbf {x} _ {j}) \\ = \frac {1}{m ^ {2}} \sum_ {i = 1} ^ {m} k _ {p} (\mathbf {x} _ {i}, \mathbf {x} _ {i}) + \frac {1}{m ^ {2}} \sum_ {i \neq j} ^ {m} k _ {p} (\mathbf {x} _ {i}, \mathbf {x} _ {j}) \\ = \frac {2 \beta d}{m \ell^ {2}} + \frac {1}{m ^ {2}} \sum_ {i = 1} ^ {m} \| s _ {p} (\mathbf {x} _ {i}) \| _ {2} ^ {2} + \frac {1}{m ^ {2}} \sum_ {i \neq j} ^ {m} k _ {p} (\mathbf {x} _ {i}, \mathbf {x} _ {j}). \\ \end{array}
$$
Next, it can be shown that the difference  $m\big(\mathrm{KSD}^2 (\mathbb{P},\mathbb{Q}_m) - \mathbb{E}[\mathrm{KSD}^2 (\mathbb{P},\mathbb{P}_m)]\big)$  takes the form
$$
m \big (\mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {m}) - \mathbb {E} [ \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {P} _ {m}) ] \big) = \frac {1}{m} \sum_ {i = 1} ^ {m} \| s _ {p} (\mathbf {x} _ {i}) \| _ {2} ^ {2} + \frac {1}{m} \sum_ {i \neq j} ^ {m} k _ {p} (\mathbf {x} _ {i}, \mathbf {x} _ {j}) - \mathbb {E} [ \| s _ {p} (\mathbf {X}) \| _ {2} ^ {2} ].
$$
Since  $\mathbf{x}_i, \mathbf{x}_j \in \mathcal{M}_{s_0}$ , we can use Lemma 2 to bound the terms  $k_p(\mathbf{x}_i, \mathbf{x}_j)$ , and then obtain
$$
m \big (\mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {m}) - \mathbb {E} [ \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {P} _ {m}) ] \big) \leq s _ {0} ^ {2} + (m - 1) \Big (\frac {2 \beta d}{\ell^ {2}} + \frac {2 \beta s _ {0} c _ {1}}{\ell} + s _ {0} ^ {2} \Big) - \mathbb {E} [ \| s _ {p} (\mathbf {X}) \| _ {2} ^ {2} ],
$$
where the right hand side is always negative if
$$
m <   1 + \frac {\mathbb {E} [ \| s _ {p} (\mathbf {X}) \| _ {2} ^ {2} ] - s _ {0} ^ {2}}{2 \beta d / \ell^ {2} + 2 \beta s _ {0} / \ell + s _ {0} ^ {2}}.
$$
Proof of Lemma 2. The Stein kernel  $k_{p}$  obtained for the inverse multi-quadratic kernel function, with parameters  $\ell > 0$ ,  $\beta \in (0,1)$ , and  $c = 1$ , is given by
$$
\begin{array}{l} k _ {p} (\mathbf {x}, \mathbf {y}) = - \frac {4 \beta (\beta + 1)}{\rho^ {4}} \| \mathbf {x} - \mathbf {y} \| _ {2} ^ {2} (1 + \| \mathbf {x} - \mathbf {y} \| _ {2} ^ {2} / \ell^ {2}) ^ {- \beta - 2} \\ + \frac {2 \beta}{\ell^ {2}} (d + (s _ {p} (\mathbf {x}) - s _ {p} (\mathbf {y})) \cdot (\mathbf {x} - \mathbf {y})) (1 + \| \mathbf {x} - \mathbf {y} \| _ {2} ^ {2} / \ell^ {2}) ^ {- \beta - 1} \\ + \left(s _ {p} (\mathbf {x}) \cdot s _ {p} (\mathbf {y})\right) \left(1 + \| \mathbf {x} - \mathbf {y} \| _ {2} ^ {2} / \ell^ {2}\right) ^ {- \beta}. \\ \end{array}
$$
Since the first term is always negative and  $(1 + \| \mathbf{x} - \mathbf{y}\| _2^2 /\ell^2)^{-\alpha}\leq 1$  for  $\alpha = \beta ,\beta +1$  , we obtain
$$
\begin{array}{l} k _ {p} (\mathbf {x}, \mathbf {y}) \leq \frac {2 \beta d}{\ell^ {2}} + \frac {2 \beta}{\ell^ {2}} | (s _ {p} (\mathbf {x}) - s _ {p} (\mathbf {y})) \cdot (\mathbf {x} - \mathbf {y}) | (1 + \| \mathbf {x} - \mathbf {y} \| _ {2} ^ {2} / \ell^ {2}) ^ {- \beta - 1} + | (s _ {p} (\mathbf {x}) \cdot s _ {p} (\mathbf {y})) | \\ \leq \frac {2 \beta d}{\ell^ {2}} + \frac {2 \beta}{\ell} \big | (s _ {p} (\mathbf {x}) - s _ {p} (\mathbf {y})) \cdot \frac {\mathbf {x} - \mathbf {y}}{\| \mathbf {x} - \mathbf {y} \| _ {2}} \big | \frac {\| \mathbf {x} - \mathbf {y} \| _ {2} / \ell}{(1 + \| \mathbf {x} - \mathbf {y} \| _ {2} ^ {2} / \ell^ {2}) ^ {\beta + 1}} + | (s _ {p} (\mathbf {x}) \cdot s _ {p} (\mathbf {y})) |. \\ \end{array}
$$
We define the function  $g$  for  $z \geq 0$  as
$$
g (z) = \frac {z}{(1 + z ^ {2}) ^ {\beta + 1}},
$$
and a simple function analysis shows that
$$
\frac{c_{1}}{2}\stackrel {\text{def}}{=}\sup_{z\geq 0}g_{1}(z) = \frac{1}{\sqrt{2\beta + 1}}\Bigl(\frac{2\beta + 1}{2\beta + 2}\Bigr)^{\beta +1}.
$$
Since  $\beta \in (0,1)$ , we have  $c_{1} \leq 1$ . We combine the last two inequalities for  $k_{p}(\mathbf{x},\mathbf{y})$  and  $c_{1}$  to get
$$
k _ {p} (\mathbf {x}, \mathbf {y}) \leq \frac {2 \beta d}{\ell^ {2}} + \frac {\beta}{\ell} \left| \left(s _ {p} (\mathbf {x}) - s _ {p} (\mathbf {y})\right) \cdot \frac {\mathbf {x} - \mathbf {y}}{\| \mathbf {x} - \mathbf {y} \| _ {2}} \right| + \left| \left(s _ {p} (\mathbf {x}) \cdot s _ {p} (\mathbf {y})\right) \right|.
$$
We can apply Cauchy-Schwartz inequality, and since  $\mathbf{x},\mathbf{y}\in \mathcal{M}_{s_0}$ , we get
$$
\left| \left(s _ {p} (\mathbf {x}) - s _ {p} (\mathbf {y})\right) \cdot \frac {\mathbf {x} - \mathbf {y}}{\| \mathbf {x} - \mathbf {y} \| _ {2}} \right| \leq \| \left(s _ {p} (\mathbf {x}) - s _ {p} (\mathbf {y})\right) \| _ {2} \frac {\| \mathbf {x} - \mathbf {y} \| _ {2}}{\| \mathbf {x} - \mathbf {y} \| _ {2}} \leq 2 s _ {0},
$$
$$
\left| \left(s _ {p} (\mathbf {x}) \cdot s _ {p} (\mathbf {y})\right) \right| \leq s _ {0} ^ {2}.
$$
Overall, we obtain
Proof of Corollary 2.5. As minimum and saddle points are stationary points of  $p$ , we have
$$
\{\mathbf {x} _ {i} \} _ {i = 1} ^ {m} \subset \mathcal {M} _ {0},
$$
and we can apply Theorem 2.4 for  $s_0 = 0$  to get the final result.
Proof of Corollary 2.6. Let the density  $p$  be a Gaussian mixture model of two components with equal weights, respectively centered in  $(-\mu, \mathbf{0}_{d-1})$  and  $(\mu, \mathbf{0}_{d-1})$ , and of variance  $\sigma^2 \mathbf{I}_d$ , and let  $\nu = \mu / \sigma$ . We assume that  $\nu > 1$  and  $0 \leq s_0 < \left[ \nu \sqrt{\nu^2 - 1} - \ln(\nu + \sqrt{\nu^2 - 1}) \right] / \mu$ . Then, according to Theorem 2.4, for any  $\{\mathbf{x}_i\}_{i=1}^m \subset \mathcal{M}_{s_0}$  of empirical measure  $Q_m = \frac{1}{m} \sum_{i=1}^m \delta(\mathbf{x}_i)$ , we have
(i)  $\mathrm{KSD}^2\big(\mathbb{P},\mathbb{Q}_m\big) <   \mathbb{E}\big[\mathrm{KSD}^2\big(\mathbb{P},\mathbb{P}_m\big)\big]$  if  $m$  and  $s_0$  satisfy  $m <   1 + \frac{\mathbb{E}[||s_p(\mathbf{X})||_2^2] - s_0^2}{2\beta d / \ell^2 + 2\beta s_0 / \ell + s_0^2}$
To prove statement (ii), we need to characterize the shape of the set  $\mathcal{M}_{s_0} \subset \mathbb{R}^d$ , given by the level lines of the squared score norm  $\| s_p(\mathbf{x})\|_2^2$ . The density  $p$  is a Gaussian mixture, i.e.,
$$
p (\mathbf {x}) = \frac {1}{2 (2 \pi) ^ {d / 2} \sigma^ {d}} e ^ {- \| \mathbf {x} ^ {(- 1)} \| _ {2} ^ {2} / 2 \sigma^ {2}} \left(e ^ {- (x ^ {(1)} + \mu) ^ {2} / 2 \sigma^ {2}} + e ^ {- (x ^ {(1)} - \mu) ^ {2} / 2 \sigma^ {2}}\right),
$$
where  $\mathbf{x}^{(-1)}$  is the vector  $\mathbf{x}$  without the first component. Then, the score is also given by an explicit formula,
$$
s _ {p} (\mathbf {x}) = \left( \begin{array}{c} - \frac {x ^ {(1)}}{\sigma^ {2}} + \frac {\mu}{\sigma^ {2}} \tanh  (\frac {\mu}{\sigma^ {2}} x ^ {(1)}) \\ - \frac {\mathbf {x} ^ {(- 1)}}{\sigma^ {2}} \end{array} \right),
$$
where  $\tanh$  is the standard hyperbolic tangent function. An important property of this score function is that the  $j$ -th component of  $s_p$  only depends on  $x^{(j)}$ , which makes  $s_p^{(j)}(\mathbf{x})$  invariant by any translation orthogonal to the  $j$ -th axis. Then, we can compute the squared score norm
$$
\| s _ {p} (\mathbf {x}) \| _ {2} ^ {2} = s _ {p} ^ {(1)} (x ^ {(1)}) ^ {2} + \frac {\| \mathbf {x} ^ {(- 1)} \| _ {2} ^ {2}}{\sigma^ {4}},
$$
Figure 16: Squared first component of the score for a Gaussian mixture with  $\mu = 3$  and  $\sigma = 1$ .
where  $s_p^{(1)}(z)^2 = \left(\frac{z}{\sigma^2} - \frac{\mu}{\sigma^2} \tanh \left(\frac{\mu}{\sigma^2} z\right)\right)^2$ . A simple function analysis of this univariate function, illustrated in Figure 16, shows that  $s_p^{(1)}(z)^2$  has two local maximum in  $z_{\max}^{-}$  and  $z_{\max}^{+}$ , and three local minimum in  $z_{\min}^{-}$ , 0, and  $z_{\min}^{+}$ , provided that  $\nu = \mu / \sigma > 1$ . We also get that  $s_p^{(1)}(z)^2$  grows to  $+\infty$  when  $x^{(1)} \to +/ - \infty$ . The extreme values are ordered as follows
$$
- \mu <   z _ {\mathrm {m i n}} ^ {-} <   z _ {\mathrm {m a x}} ^ {-} <   0 <   z _ {\mathrm {m a x}} ^ {+} <   z _ {\mathrm {m i n}} ^ {+} <   \mu .
$$
The values of  $z_{\mathrm{min}}^{-}$ ,  $z_{\mathrm{max}}^{-}$ ,  $z_{\mathrm{max}}^{+}$ , and  $z_{\mathrm{min}}^{+}$  are given by the zeros of the first derivative of  $s_p^{(1)}(z)^2$ , defined by
$$
\frac {d s _ {p} ^ {(1)} (z) ^ {2}}{d z} = 2 \Big (- \frac {1}{\sigma^ {2}} + \big (\frac {\mu}{\sigma^ {2}} \big) ^ {2} \frac {1}{\cosh (\frac {\mu}{\sigma^ {2}} z) ^ {2}} \Big) \Big (- \frac {z}{\sigma^ {2}} + \frac {\mu}{\sigma^ {2}} \tanh  \big (\frac {\mu}{\sigma^ {2}} z \big) \Big).
$$
This derivative vanishes when one of the two factors is null. Since  $\mu / \sigma > 1$ , the first term is null when
$$
\frac {\mu^ {2}}{\sigma^ {2}} \frac {1}{\cosh (\frac {\mu}{\sigma^ {2}} z) ^ {2}} = 1,
$$
$$
z _ {\max } ^ {-} = - \frac {\sigma^ {2}}{\mu} \operatorname {a r c o s h} \left(\frac {\mu}{\sigma}\right) \quad \text {a n d} \quad z _ {\max } ^ {+} = \frac {\sigma^ {2}}{\mu} \operatorname {a r c o s h} \left(\frac {\mu}{\sigma}\right).
$$
The second factor is null when
$$
\tanh  \left(\frac {\mu}{\sigma^ {2}} z\right) - \frac {z}{\mu} = 0. \tag {6}
$$
Obviously,  $z = 0$  is solution. Since  $\mu / \sigma > 1$ , equation (6) has two additional solutions. Although they do not have a closed form, we have
$$
z _ {\mathrm {m i n}} ^ {-} \in (- \mu , z _ {\mathrm {m a x}} ^ {-})
$$
$$
z _ {\mathrm {m i n}} ^ {+} \in (z _ {\mathrm {m a x}} ^ {+}, \mu).
$$
Also notice that, as  $\mu/\sigma$  gets larger,  $z_{\mathrm{min}}^{-}$  is closer to  $-\mu$ , and  $z_{\mathrm{min}}^{+}$  to  $\mu$ . For example in Figure 16, we set  $\mu/\sigma = 3$ , and we hardly see a gap between  $-\mu$  and  $z_{\mathrm{min}}^{-}$ , or  $\mu$  and  $z_{\mathrm{min}}^{+}$ .
By definition, for  $\mathbf{x} \in \mathcal{M}_{s_0}$ , we have
$$
s _ {p} ^ {(1)} (x ^ {(1)}) ^ {2} \leq \| s _ {p} (\mathbf {x}) \| _ {2} ^ {2} \leq s _ {0} ^ {2}.
$$
Therefore, given the variations of  $s_p^{(1)}(x^{(1)})^2$  detailed above and illustrated in Figure 16, if  $s_0^2 < s_p^{(1)}(z_{\mathrm{max}}^-)^2 = s_p^{(1)}(z_{\mathrm{max}}^+)^2$ , it exists three disjoint intervals  $I_{-\mu}, I_0, I_\mu \subset \mathbb{R}$ , respectively centered around  $-\mu, 0$ , and  $\mu$ , such that
$$
\mathbf {x} \in \mathcal {M} _ {s _ {0}} \Rightarrow x ^ {(1)} \in I _ {- \mu} \cup I _ {0} \cup I _ {\mu}.
$$
To conclude, we compute the value of  $s_p^{(1)}(z_{\mathrm{max}}^-)$ , that is
$$
s _ {p} ^ {(1)} \left(z _ {\max } ^ {+}\right) = - \frac {1}{\mu} \operatorname {a r c o s h} \left(\frac {\mu}{\sigma}\right) + \frac {\mu}{\sigma^ {2}} \tanh  \left(\operatorname {a r c o s h} \left(\frac {\mu}{\sigma}\right)\right).
$$
Using the formulas  $\tanh (\operatorname{arccosh}(x)) = \frac{\sqrt{x^2 - 1}}{x}$  for  $|x| > 1$ ,  $\operatorname{arccosh}(x) = \ln (x + \sqrt{x^2 - 1})$ , and with  $\nu = \mu / \sigma$ , we get
$$
\begin{array}{l} s _ {p} ^ {(1)} \left(z _ {\max } ^ {+}\right) = \sqrt {(\mu / \sigma) ^ {2} - 1} / \sigma - \ln (\mu / \sigma + \sqrt {(\mu / \sigma) ^ {2} - 1}) / \mu \\ = \left[ \nu \sqrt {\nu^ {2} - 1} - \ln \left(\nu + \sqrt {\nu^ {2} - 1}\right) \right] / \mu , \\ \end{array}
$$
which is always strictly positive since  $\nu > 1$ . By assumption,  $0 \leq s_0 < \left[\nu \sqrt{\nu^2 - 1} - \ln (\nu + \sqrt{\nu^2 - 1})\right] / \mu$ , and therefore, we have  $s_0 < s_p^{(1)}(z_{\mathrm{max}}^+) = s_p^{(1)}(z_{\mathrm{max}}^-)$ , which concludes the proof of statement (ii).
D Proof of Theorem 3.3
Theorem 3.3. Let  $k_{p}$  be the Stein kernel associated with the radial kernel  $k(\mathbf{x},\mathbf{x}^{\prime}) = \phi (\| \mathbf{x} - \mathbf{x}^{\prime}\|_{2} / \ell)$ , where  $\mathbf{x},\mathbf{x}^{\prime}\in \mathbb{R}^{d}$ ,  $\ell >0$ , and  $\phi \in C^2 (\mathbb{R})$ . Let  $p$  and  $q$  be two bimodal mixture distributions satisfying Assumption 2.1. We define  $w_{\lambda}^{\star}$  as the optimal mixture weight of  $q$  with respect to the entropic regularized KSD distance, i.e.,  $w_{\lambda}^{\star} = \operatorname *{argmin}_{w\in [0,1]}\mathrm{KSD}_{\lambda}(\mathbb{P},\mathbb{Q}_w)$ . If  $\mathbb{E}[\log (p(\mathbf{Z}_L))]\neq \mathbb{E}[\log (p(\mathbf{Z}_R))]$
Proof of Theorem 3.3. We consider the mixture distributions  $p$  and  $q$  satisfying Assumption 2.1, for  $\mu > 0$  and  $r > 0$ , and denote by  $q_{L}$  the distribution of the left mode of the mixture  $q$ , and similarly,  $q_{R}$  is the distribution of the right mode of  $q$ . The probability measures  $\mathbb{Q}_L$  and  $\mathbb{Q}_R$  respectively admits the densities  $q_{L}$  and  $q_{R}$ . As in the proof of Theorem 2.3, we get that
$$
\begin{array}{l} \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {w}) = w ^ {2} \left[ \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {L}) + \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {R}) - 2 \Delta_ {L, R} \right] \\ - 2 w \left[ \mathrm {K S D} ^ {2} \left(\mathbb {P}, \mathbb {Q} _ {R}\right) - \Delta_ {L, R} \right] + \mathrm {K S D} ^ {2} \left(\mathbb {P}, \mathbb {Q} _ {R}\right). \\ \end{array}
$$
Next, we define  $\mathbf{Z} \sim \mathbb{Q}_w$ ,  $\mathbf{Z}_L \sim \mathbb{Q}_L$ , and  $\mathbf{Z}_R \sim \mathbb{Q}_R$ , and develop the entropic regularization term,
$$
\begin{array}{l} \mathbb {E} [ \log (p (\mathbf {Z})) ] = \int \log (p (\mathbf {x})) (w q _ {L} (\mathbf {x}) + (1 - w) q _ {R} (\mathbf {x})) d \mathbf {x} \\ = w \int \log (p (\mathbf {x})) q _ {L} (\mathbf {x}) d \mathbf {x} + (1 - w) \int \log (p (\mathbf {x})) q _ {R} (\mathbf {x})) d \mathbf {x} \\ = w \mathbb {E} [ \log (p (\mathbf {Z} _ {L})) ] + (1 - w) \mathbb {E} [ \log (p (\mathbf {Z} _ {R})) ]. \\ \end{array}
$$
Combining these two results, we have
$$
\begin{array}{l} \mathrm {K S D} _ {\lambda} ^ {2} (\mathbb {P}, \mathbb {Q} _ {w}) = w ^ {2} \left[ \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {L}) + \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {R}) - 2 \Delta_ {L, R} \right] \\ - 2 w \left[ \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {R}) - \Delta_ {L, R} + \lambda / 2 \left(\mathbb {E} \left[ \log (p (\mathbf {Z} _ {L})) \right] - \mathbb {E} \left[ \log (p (\mathbf {Z} _ {R})) \right]\right) \right] \\ + \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {R}) - \lambda \mathbb {E} [ \log (p (\mathbf {Z} _ {R})) ]. \\ \end{array}
$$
We recall that  $p$  does not depend on  $w$ , but only on the fixed weight  $w_{p}$  and the two mode distributions.
Consequently,  $\mathrm{KSD}_{\lambda}^{2}(\mathbb{P},\mathbb{Q}_{w})$  is a second-order polynomial with respect to  $w$ . As for Theorem 2.3, the