SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} _ {x \sim \hat {\mu} _ {2 i}} [ h _ {x} (z) ] = \bar {R} \mathbb {E} _ {u \sim N (0, 1 / \lambda_ {1})} \left\{\tanh  \left[ \log (\rho k) \left(\langle \xi , z \rangle - (b _ {i} - 1)\right) + u \right] + 1 \right\} / 3
$$
because we have  $\langle x_1, z \rangle + x_2 = \log(\rho k) (\langle \xi, z \rangle - (b_i - 1)) + \sum_{j=1}^d u_j z_j + u_{d+1}$  for  $x \sim N([\xi^\top, (b_i - 1)]^\top, I/(2\lambda_1))$  where  $u_j \sim N(0, 1/(2\lambda_1))$  (i.i.d.) and  $\sum_{j=1}^d u_j z_j + u_{d+1}$  obeys the Gaussian distribution with mean 0 and variance  $\sum_{j=1}^d \frac{1}{2\lambda_1} \frac{1}{d} + \frac{1}{2\lambda_1} = \frac{1}{\lambda_1}$ . In the same vein, we also have
$$
\mathbb {E} _ {x \sim \hat {\mu} _ {2 i + 1}} [ h _ {x} (z) ] = - \bar {R} \mathbb {E} _ {u \sim N (0, 1 / \lambda_ {1})} \left\{\tanh  \left[ \log (\rho k) \left(\langle \xi , z \rangle - (b _ {i} + 1)\right) + u \right] + 1 \right\} / 3.
$$
Here, define  $|z| := |\{i \in \{1, \dots, k\} \mid z_i > 0\}|$  for  $z \in \{\pm 1 / \sqrt{d}\}^d$  which is the number of positive elements of  $z$  in the index set  $\{1, \dots, k\}$ . For a fixed number  $i \in \{0, \dots, k\}$ , we let
$$
f _ {1} (z; u) = \left\{\tanh  \left[ \log (\rho k) \left(\langle \xi , z \rangle - \left(b _ {i} - 1\right)\right) + u \right] + 1 \right\} / 3,
$$
$$
f _ {2} (z; u) = \left\{\tanh  \left[ \log (\rho k) \left(\langle \xi , z \rangle - \left(b _ {i} + 1\right)\right) + u \right] + 1 \right\} / 3,
$$
then we can see that
$$
f _ {1} (z; 0) = \left\{ \begin{array}{l l} O (1 / (\rho k)) & (| z | <   i), \\ 1 - O (1 / (\rho k)) & (| z | \geq i), \end{array} \right.
$$
$$
f _ {2} (z; 0) = \left\{ \begin{array}{l l} O (1 / (\rho k)) & (| z | <   i + 1), \\ 1 - O (1 / (\rho k)) & (| z | \geq i + 1), \end{array} \right.
$$
because  $\langle \xi, z \rangle - b_i = \sum_{j=1}^{k} \operatorname{sign}(z_j) b_i = 2|z| - k - b_i = 2(|z| - i)$ . Hence, we have that
$$
f (z; u) := f _ {1} (z; u) - f _ {2} (z; u) = \left\{ \begin{array}{l l} \Omega (1) & (| z | = i), \\ O (1 / (\rho k)) & (\text {o t h e r w i s e}). \end{array} \right.
$$
Then, since  $\tanh (u) + 1 = \frac{e^u - e^{-u}}{e^u + e^{-u}} + 1 = \frac{2}{1 + e^{-2u}}$ , if  $|z| = i$  and  $|u| \leq 1 / \lambda_1$ ,
$$
f (z; u) \geq \Omega (1),
$$
and if  $|z|\neq i$  and  $|u|\leq \log (\rho k) / 2$
$$
f (z; u) \leq O (1 / (\rho k)).
$$
Therefore, when  $|z| = i$
$$
\mathbb {E} _ {u \sim N (0, 1 / \lambda_ {1})} [ f (z; u) ] \geq \int_ {- 1 / \lambda_ {1}} ^ {1 / \lambda_ {1}} f (z; u) g (u) \mathrm {d} u > \Omega (1).
$$
where  $g$  is the density function of  $N(0,1 / \lambda_1)$ , and when  $|z|\neq i$ ,
$$
\begin{array}{l} \mathbb {E} _ {u \sim N (0, 1 / \lambda_ {1})} [ f (z; u) ] \leq \int_ {- \log (\rho k) / 2} ^ {\log (\rho k) / 2} f (z; u) g (u) d u + \int_ {| u | \geq \log (\rho k) / 2} f (z; u) g (u) d z \\ \leq O (1 / (\rho k)) + O \left(\frac {\exp \left(- \lambda_ {1} \log (\rho k) ^ {2} / 2\right)}{\log (\rho k)}\right) \\ = O (1 / (\rho k)), \\ \end{array}
$$
where we used the upper-tail inequality of the Gaussian distribution in the second inequality. Hence, it holds that
$$
\hat {f} _ {i} (z) := \mathbb {E} _ {x \sim \hat {\mu} _ {2 i}} [ h _ {x} (z) ] + \mathbb {E} _ {x \sim \hat {\mu} _ {2 i + 1}} [ h _ {x} (z) ] = \left\{ \begin{array}{l l} \Omega (k) & (| z | = i), \\ O (1 / \rho) & (\text {o t h e r w i s e}), \end{array} \right.
$$
because  $\bar{R} = k$ . Therefore, by taking  $\rho > 1$  sufficiently large, we also have
$$
\hat {f} (z) := \frac {1}{2 (k + 1)} \sum_ {i = 0} ^ {k} (- 1) ^ {i} \hat {f} _ {i} (z) = \left\{ \begin{array}{l l} \Omega (1) & (\mid z \mid \text {i s e v e n}), \\ - \Omega (1) & (\mid z \mid \text {i s o d d}), \end{array} \right.
$$
where the constant hidden in  $\Omega(\cdot)$  is uniform over any  $|z|$ . Hence, there exists  $c_2' > 0$  such that  $Y\hat{f}(Z) > c_2'$  almost surely. Then, if we let  $\mu_{\langle a\rangle}(A) \coloneqq \mu(aA)$  for  $a \in \mathbb{R}$ , a probability measure  $\mu$  and a measurable set  $A$ , then we can see that  $\hat{f}$  is represented as
$$
\hat {f} (\cdot) = \mathbb {E} _ {x \sim \mu^ {*}} [ h _ {x} (\cdot) ],
$$
$$
\mu^ {*} = \frac {1}{2 (k + 1)} \sum_ {i = 0} ^ {k} \left(\hat {\mu} _ {2 i, \langle (- 1) ^ {i} \rangle} + \hat {\mu} _ {2 i + 1, \langle (- 1) ^ {i} \rangle}\right).
$$
Then, by letting  $c_{2} = \ell (0) - \ell (c_{2}^{\prime})$ , we have
$$
L \left(\mu^ {*}\right) \leq \ell (0) - c _ {2}.
$$
Next, we bound the KL-divergence between  $\nu$  and  $\mu^{*}$ . Notice that the convexity of KL-divergence yields that
$$
\begin{array}{l} \mathrm {K L} (\nu , \mu^ {*}) \leq \frac {1}{2 (k + 1)} \sum_ {i = 0} ^ {k} (\mathrm {K L} (\nu , \hat {\mu} _ {2 i}) + \mathrm {K L} (\nu , \hat {\mu} _ {2 i + 1})) \\ \leq \lambda_ {1} \log (\rho k) ^ {2} [ \| \xi \| ^ {2} + \left(\max _ {i} | b _ {i} | + 1\right) ^ {2} ] + \log (1 / (2 \lambda_ {1})) + \lambda_ {1} (1 + \zeta^ {2}) \\ = O \left(\log (k) ^ {2} \left(d k + k ^ {2}\right)\right) = O \left(\log (k) ^ {2} d k\right) (= R), \\ \end{array}
$$
because  $k \leq d$ , which gives the assertion.
Proof of Proposition 4. The first assertion can be seen by  $\bar{L} (\mu_{[\lambda ]})\leq \bar{L} (\mu^{*}) + \lambda \mathrm{KL}(\nu ||\mu^{*})\leq \ell (0) - c_2 + \lambda R < \ell (0) - c_2 + c_2 / 2 = \ell (0) - c_2 / 2$ , by the assumption of  $\lambda$  such that  $\lambda < c_{2} / (2R)$ . The same argument is also applied to  $L(\mu_{[\lambda ]})$ , that is,  $L(\mu_{[\lambda ]})\leq \ell (0) - c_{2} / 2$ .
We show that the optimal solution  $f_{\mu_{[\lambda]}}$  should satisfy  $|f_{\mu_{[\lambda]}}(z)| = |f_{\mu_{[\lambda]}}(z')|$  for all  $z, z' \in \{\pm 1/\sqrt{d}\}^d$  by the symmetric property of the distribution. For that purpose, we construct a group action on  $\{-1/\sqrt{d}, 1/\sqrt{d}\}^d$ . For notational simplicity, we consider  $\mathbf{Z}^d := \{-1, +1\}^d$  instead.  $\mathbf{Z}^d$  can be equipped with a group structure where the binary operation between  $z, z' \in \mathbf{Z}^d$  is given by the element wise product  $z \cdot z' = (z_i z_i')_{i=1}^d$ . Let  $T_j: \mathbf{Z}^d \to \mathbf{Z}^d$  be a group action that flips the  $j$ -th element  $T_j z = (z_1, \ldots, -z_j, \ldots, z_d)$ . It is obvious that  $T_j$  is bijective. Then, we can show
that there exists a sequence of indices  $\sigma(1), \ldots, \sigma(2^d) \in \{1, \ldots, d\}$  such that the "orbit" generated by the chain of the group actions  $T_{\sigma(k)} \circ \cdots \circ T_{\sigma(2)} \circ T_{\sigma(1)}z$  for  $k = 1, \ldots, 2^d$  covers the entire elements of  $\mathbf{Z}^d$  for any initial state  $z \in \mathbf{Z}^d$  and the last state turns back to the initial state, that is,  $T_{\sigma(2^d)}z = z$ . This can be shown by induction. If  $d = 1$ , we just need to take  $\sigma(1) = \sigma(2) = 1$  (indeed,  $T_{\sigma(1)}z = -z$  and  $T_{\sigma(2)} \circ T_{\sigma(1)}z = z$ , which satisfies the condition). Suppose that the argument is true for  $d = 1, 2, \ldots, d' - 1$ , then we show that it is true for  $d = d'$ . Indeed, if we write  $\sigma'(1), \ldots, \sigma'(2^{d' - 1})$  be the corresponding sequence for  $d = d' - 1$ , then let
$$
\sigma (i) = \sigma^ {\prime} (i)
$$
for  $i = 1, \dots, 2^{d' - 1} - 1$ , then the corresponding orbit covers all the elements with the last element is fixed except the initial state. Then, we flip the last coordinate at the  $2^{d' - 1} + 1$ -th step as
$$
\sigma (2 ^ {d ^ {\prime} - 1}) = d,
$$
and then we "trace-back" the orbit as
$$
\sigma \left(2 ^ {d ^ {\prime} - 1} + i\right) = \sigma^ {\prime} \left(2 ^ {d ^ {\prime} - 1} - i\right),
$$
for  $i = 1, \ldots, 2^{d' - 1} - 1$ . By the construction, we notice that, after  $2^{d' - 1} + 2^{d' - 1} - 1$  steps, the state becomes
$$
T _ {\sigma \left(2 ^ {d ^ {\prime}} - 1\right)} \circ T _ {\sigma \left(2 ^ {d ^ {\prime}} - 2\right)} \circ \dots \circ T _ {\sigma (1)} z = \left[ z _ {1}, \dots , z _ {d ^ {\prime} - 1}, - z _ {d ^ {\prime}} \right] ^ {\top}.
$$
Therefore, by flipping the last coordinate again, the state can come back to the initial state, that is, by setting  $\sigma(2^{d'}) = d$ , we have  $T_{\sigma(2^{d'})} \circ \dots \circ T_{\sigma(1)}z = z$  while the orbit covers the entire elements of  $\mathbf{Z}^{d'}$ .
We can define the action  $T_{j}$  to  $\{\pm 1 / \sqrt{d}\}^{d}$  in the same manner. We note that the distribution of  $Z$  is invariant against the action of  $T_{j}$ , that is,  $P_{Z} = T_{j\#}P_{Z}$  where  $T_{j\#}$  is the push-forward induced by  $T_{j}$ . Here, let  $\hat{\mu} \coloneqq \mu_{[\lambda]}$  and  $\hat{f} = f_{\hat{\mu}}$ . We also define an "adjoint operator"  $T_{j}^{*}: \mathcal{P} \to \mathcal{P}$  for  $j \in \{k + 1, \ldots, d\}$  such that  $f_{\mu}(T_{j}z) = f_{T_{j}^{*}\mu}(z)$  for all  $z \in \{\pm 1 / \sqrt{d}\}^{d}$ . We can easily check that there exists  $T_{j}^{*}\mu$  that satisfies this condition. Indeed, we may take  $T_{j}^{*}\mu$  such that  $T_{j}^{*}\mu (A\times B\times C) = \mu (T_{j}A\times B\times C)$  where  $T_{j}A = \{T_{j}z|z \in A\}$  for any measurable set  $A \in \mathbb{B}(\mathbb{R}^d)$ ,  $B \in \mathbb{B}(\mathbb{R})$  and  $C \in \mathbb{B}(\mathbb{R})$ . Since the probability on product sets uniquely determines the probability on the Borel sigma algebra  $\mathbb{B}(\mathbb{R}^d)\times \mathbb{B}(\mathbb{R})\times \mathbb{B}(\mathbb{R}) = \mathbb{B}(\mathbb{R}^{d + 2})$ ,  $T_{j}^{*}\mu$  is uniquely determined.
Now, we consider the sequence of the group action  $T_{\sigma(1)}, \ldots, T_{\sigma(2^{d-k})}$  constructed above acting on the last  $d-k$  coordinates. Since the label is independent of the last  $d-k$  coordinates, we have  $L(T_{\sigma(1)}^* \circ \dots \circ T_{\sigma(i)}^*\hat{\mu}) = L(\hat{\mu})$  for all  $i = 1, \ldots, 2^{d-k}$ . Moreover, the symmetry of the Gaussian distribution yields that  $\mathrm{KL}(\nu, T_j^*\mu) = \mathrm{KL}(\nu, \mu)$ . Hence, if we take
$$
\hat {\mu} ^ {\prime} = \frac {1}{2 ^ {d - k}} \sum_ {i = 1} ^ {2 ^ {d - k}} T _ {\sigma (1)} ^ {*} \circ \dots \circ T _ {\sigma (i)} ^ {*} \hat {\mu},
$$
then by the convexity of the objective  $\mathcal{L}$ , we have
$$
\mathcal {L} \left(\hat {\mu} ^ {\prime}\right) \leq \frac {1}{2 ^ {d - k}} \sum_ {i = 1} ^ {2 ^ {d - k}} \mathcal {L} \left(T _ {\sigma (1)} ^ {*} \circ \dots \circ T _ {\sigma (i)} ^ {*} \hat {\mu}\right) \leq \mathcal {L} (\hat {\mu}).
$$
Then, by the strong convexity of the objective  $\mathcal{L}$  (due to the KL-divergence), we have  $\hat{\mu}^{\prime} = \hat{\mu}$ . Then, we notice that
$$
f _ {\hat {\mu}} (z) = \frac {1}{2 ^ {d - k}} \sum_ {i = 1} ^ {2 ^ {d - k}} f _ {\hat {\mu}} (T _ {\sigma (i)} \circ \dots \circ T _ {\sigma (1)} z),
$$
for any  $z \in \{\pm 1 / \sqrt{d}\}^d$ , which means that  $f_{\hat{\mu}}(z)$  is constant against any change of the last  $d - k$  coordinates.
Next, we consider to change the first  $k$  coordinates. We also construct a sequence of the group action  $T_{\sigma(1)}, \ldots, T_{\sigma(2^k)}$  acting on the first  $k$  coordinates. In this case, the action of  $T_j$  on  $z$  also flips the
corresponding label. Hence, if  $y_{(z)}$  is the label of  $z$ , then the label for  $T_{j}z$  is  $-y_{(z)} = y_{(T_{j}z)}$ . By defining the adjoint operator  $T_{j}^{*}$  as
$$
T _ {j} ^ {*} \mu (A \times B \times C) = \mu ((- T _ {j} A) \times (- B) \times (- C)),
$$
for  $j \in \{1, \dots, k\}$ , then by the symmetric shape of  $\tanh$ , we have
$$
y _ {(T _ {j} z)} f _ {\hat {\mu}} (T _ {j} z) = y _ {(z)} f _ {T _ {j} ^ {*} \hat {\mu}} (z),
$$
for any  $z \in \{\pm 1 / \sqrt{d}\}^d$ . We also have that  $\mathrm{KL}(\nu, \hat{\mu}) = \mathrm{KL}(\nu, T_j^*\hat{\mu})$  by the symmetry of the Gaussian distribution. Therefore, once again by the invariance of the  $P_Z$  against the action of  $T_j$  and the convexity of  $\mathcal{L}$ , we have that
$$
\mathcal {L} \left(\hat {\mu} ^ {\prime}\right) \leq \mathcal {L} (\hat {\mu}),
$$
$$
\hat {\mu} ^ {\prime} = \frac {1}{2 ^ {k}} \sum_ {i = 1} ^ {2 ^ {k}} T _ {\sigma (1)} ^ {*} \circ \dots \circ T _ {\sigma (i)} ^ {*} \hat {\mu}. \tag {8}
$$
By the strong convexity of  $\mathcal{L}$  due to the KL-divergence term, we have  $\mu \mu' = \hat{\mu}$ . Hence,
$$
y _ {z} f _ {\hat {\mu}} (z) = \frac {1}{2 ^ {k}} \sum_ {i = 1} ^ {2 ^ {k}} y _ {(T _ {\sigma (i)} \circ \dots \circ T _ {1} z)} f _ {\hat {\mu}} (T _ {\sigma (i)} \circ \dots \circ T _ {1} z), \tag {9}
$$
for any  $z \in \{\pm 1 / \sqrt{d}\}^d$ , which means that  $y_{z}f_{\hat{\mu}}(z)$  is constant.
Finally, combining Eq. (9) with the fact  $\bar{L} (\hat{\mu})\leq \ell (0) - c_2 / 2$  , we have that  $P(Yf_{\hat{\mu}}(Z) > 0) = 1$  and  $Yf_{\hat{\mu}}(Z)\geq c_2 / 2$  almost surely because  $\| \ell '\|_{\infty}\leq 1$
B Proofs of auxiliary lemmas
B.1 Difference between finite particle network and its integral form for the optimal solution
Lemma 2. Under Assumption 1, if  $\| z\| \leq 1$  for any  $z\in \mathrm{supp}(P_Z)$ , then it holds that
$$
\mathbb {E} _ {\mathcal {X} _ {*} \sim (\mu^ {*}) ^ {\otimes N}} \left[ \sup _ {z \in \mathrm {s u p p} (P _ {Z})} \left(\frac {1}{N} \sum_ {i = 1} ^ {N} h _ {X _ {*} ^ {i}} (z) - \int h _ {x} (z) \mathrm {d} \mu^ {*} (z)\right) ^ {2} \right] \leq \frac {1 6 \bar {R} ^ {2}}{N} \left[ 1 + 2 \left(\frac {2 \bar {R} ^ {2}}{(\lambda \lambda_ {1}) ^ {2}} + \frac {\bar {d}}{\lambda_ {1}}\right) \right].
$$
Proof. Note that the left hand side can be rewritten as
$$
\mathbb {E} _ {X _ {*} ^ {i} \sim \mu^ {*}} \left[ \right.\left.\left(\sup  _ {z \in \operatorname {s u p p} (P _ {Z})} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} h _ {X _ {*} ^ {i}} (z) - \int h _ {x} (z) \mathrm {d} \mu^ {*} (z) \right|\right) ^ {2} \right].
$$
From the standard argument of concentration inequality corresponding to the Rademacher complexity, it holds that
$$
\begin{array}{l} P \left(\sup _ {z \in \operatorname {s u p p} (P _ {Z})} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} h _ {X _ {*} ^ {i}} (z) - \int h _ {x} (z) \mathrm {d} \mu^ {*} (z) \right| \right. \\ \geq 2 \mathbb {E} _ {\epsilon , \mathcal {X} _ {*}} \left[ \sup  _ {z \in \operatorname {s u p p} (P z)} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} h _ {X _ {*} ^ {i}} (z) \right| \right] + \sqrt {\frac {2 t \bar {R} ^ {2}}{N}} \\ \leq \exp (- t), \\ \end{array}
$$
for any  $t > 0$ , where  $\epsilon = (\epsilon_i)_{i=1}^N$  is an i.i.d. sequence of the Rademacher variable,  $\mathcal{X}_* = (X_*^i)_{i=1}^N$  is an i.i.d. sequence generated from  $\mu^*$ , and the probability is taken with respect to the realization of  $\mathcal{X}_*$ . Since  $\tanh$  is Lipschitz continuous with  $|\tanh'| \leq 1$ , the contraction inequality of the Rademacher complexity yields that
$$
\mathbb {E} _ {\epsilon , \mathcal {X} _ {*}} \left[ \sup  _ {z \in \operatorname {s u p p} (P _ {Z})} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} h _ {X _ {*} ^ {i}} (z) \right| \right]
$$
$$
\begin{array}{l} \leq 2 \frac {\bar {R}}{3} \mathbb {E} _ {\epsilon , \mathcal {X} _ {*}} \left[ \sup  _ {z \in \operatorname {s u p p} (P _ {Z})} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} \left(z ^ {\top} X _ {*, 1} ^ {i} + X _ {*, 2} ^ {i}\right) \right| \right] + \frac {2 \bar {R}}{3} \mathbb {E} _ {\epsilon , \mathcal {X} _ {*}} \left[ \left| \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} \tanh  \left(X _ {*, 3} ^ {i}\right) \right| \right] \\ \leq 2 \frac {\bar {R}}{3} \sqrt {\mathbb {E} _ {\epsilon , \mathcal {X} _ {*}} \left[ \sup  _ {z \in \operatorname {s u p p} (P _ {Z})} \left(\| z \| ^ {2} + 1\right) \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} \left[ X _ {*, 1} ^ {i}; X _ {*, 2} ^ {i} \right] \right\| ^ {2} \right]} \\ + \frac {2 \bar {R}}{3} \sqrt {\mathbb {E} _ {\epsilon , \mathcal {X} _ {*}} \left[ \left| \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} \tanh  (X _ {* , 3} ^ {i}) \right| ^ {2} \right]} \\ \leq 2 \frac {\bar {R} \sqrt {2}}{3} \sqrt {\frac {1}{N} \mathbb {E} [ \| X _ {* , 1} ^ {1} \| ^ {2} + (X _ {* , 2} ^ {1}) ^ {2} ]} + \frac {2 \bar {R}}{3 \sqrt {N}}. \\ \end{array}
$$
To bound the right hand side, we evaluate the moment of  $\mu^{*}$ . Since  $\mu^{*}$  is the stationary distribution of the SDE (1). Hence, if  $X_0\sim \mu^*$ , the process  $(X_{t})_{t\geq 0}$  obeying the MFLD (1) satisfies  $X_{t}\sim \mu^{*}$  for any  $t\geq 0$ . Then, the infinitesimal generator of the MFLD gives that
$$
\begin{array}{l} 0 = \frac {\mathrm {d}}{\mathrm {d} t} \mathbb {E} [ \| X _ {t} \| ^ {2} ] = \mathbb {E} \left[ (2 X _ {t}) ^ {\top} \left(- \nabla \frac {\delta L (\mu_ {t})}{\delta \mu} (X _ {t}) - \lambda \lambda_ {1} X _ {t}\right) + 2 \lambda \bar {d} \right] \\ = - 2 \lambda \lambda_ {1} \mathbb {E} \left[ \| X _ {t} \| ^ {2} \right] + \mathbb {E} \left[ (2 X _ {t}) ^ {\top} \left(\nabla \frac {\delta L (\mu_ {t})}{\delta \mu} (X _ {t})\right) \right] + 2 \lambda \bar {d} \\ = - 2 \lambda \lambda_ {1} \mathbb {E} \left[ \| X _ {t} \| ^ {2} \right] + \mathbb {E} \left[ \lambda \lambda_ {1} \| X _ {t} \| ^ {2} + \frac {1}{\lambda \lambda_ {1}} \left\| \nabla \frac {\delta L (\mu_ {t})}{\delta \mu} (X _ {t}) \right\| ^ {2} \right] + 2 \lambda \bar {d}, \\ \end{array}
$$
$$
\mathbb {E} \left[ \| X _ {t} \| ^ {2} \right] \leq \frac {(2 \bar {R}) ^ {2}}{(\lambda \lambda_ {1}) ^ {2}} + \frac {2 \bar {d}}{\lambda_ {1}}.
$$
$$
\mathbb {E} _ {\epsilon , \mathcal {X} _ {*}} \left[ \sup  _ {z \in \operatorname {s u p p} (P _ {Z})} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} h _ {X _ {*} ^ {i}} (z) \right| \right] \leq \frac {2 \bar {R}}{3 \sqrt {N}} \left(2 \sqrt {\frac {2 \bar {R} ^ {2}}{(\lambda \lambda_ {1}) ^ {2}} + \frac {\bar {d}}{\lambda_ {1}}} + 1\right) =: \frac {\bar {D}}{\sqrt {N}}.
$$
Then, for  $x\geq 0$  , it holds that
$$
\begin{array}{l} P \left[ \right.\left.\left(\sup  _ {z \in \operatorname {s u p p} \left(P _ {Z}\right)} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} h _ {X _ {*} ^ {i}} (z) - \int h _ {x} (z) \mathrm {d} \mu^ {*} (z) \right|\right) ^ {2} \geq x \right] \\ \leq \min  \left\{1, \exp \left(- \frac {N}{4 \bar {R} ^ {2}} \left(x - \frac {8 \bar {D} ^ {2}}{N}\right)\right) \right\}. \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \right.\left.\left(\sup  _ {z \in \operatorname {s u p p} (P _ {Z})} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} h _ {X _ {*} ^ {i}} (z) - \int h _ {x} (z) \mathrm {d} \mu^ {*} (z) \right|\right) ^ {2} \right] \\ \leq \int_ {0} ^ {\infty} P \left[ \left(\sup  _ {z \in \operatorname {s u p p} (P z)} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} h _ {X _ {*} ^ {i}} (z) - \int h _ {x} (z) \mathrm {d} \mu^ {*} (z) \right|\right) ^ {2} \geq x \right] \mathrm {d} x \\ \leq \frac {8 \bar {D} ^ {2}}{N} + \int_ {0} ^ {\infty} \exp \left(- \frac {N}{4 \bar {R} ^ {2}} \bar {x}\right) \mathrm {d} \bar {x} \\ \leq \frac {8 \bar {D} ^ {2}}{N} + \frac {4 \bar {R} ^ {2}}{N} \leq \frac {1 6 \bar {R} ^ {2}}{N} \left[ 1 + 2 \left(\frac {2 \bar {R} ^ {2}}{\left(\lambda \lambda_ {1}\right) ^ {2}} + \frac {\bar {d}}{\lambda_ {1}}\right) \right]. \\ \end{array}
$$
This gives the assertion.
The proof follows the line of Lemma 5.5 of Chen et al. (2020). Let the empirical Rademacher complexity of a function class  $\tilde{\mathcal{F}}$  be
$$
\widehat {\mathrm {R a d}} (\tilde {\mathcal {F}}) := \mathbb {E} _ {\epsilon} \left[ \sup _ {\mu \in \tilde {\mathcal {F}}} \frac {1}{n} \sum_ {i = 1} ^ {n} \epsilon_ {i} f (z _ {i}) \right].
$$
We can see that  $\operatorname{Rad}(\cdot) = \mathbb{E}_{(z_i)_{i=1}^n}[\widehat{\operatorname{Rad}}(\cdot)]$ . Then, it holds that
$$
\begin{array}{l} \widehat {\operatorname {R a d}} \left(\mathcal {F} _ {M} \left(\mu^ {\circ}\right)\right) = \frac {1}{\gamma} \mathbb {E} _ {\epsilon} \left[ \sup  _ {\mu \in \mathcal {F} _ {M} \left(\mu^ {\circ}\right)} \frac {\gamma}{n} \sum_ {i = 1} ^ {n} \epsilon_ {i} \int h _ {x} \left(z _ {i}\right) \mathrm {d} \mu (x) \right] \\ \leq \frac {1}{\gamma} \left\{M + \mathbb {E} _ {\epsilon} \log \left[ \int \exp \left(\frac {\gamma}{n} \sum_ {i = 1} ^ {n} \epsilon_ {i} h _ {x} (z _ {i})\right) \mathrm {d} \mu^ {\circ} (x) \right] \right\} \\ \leq \frac {1}{\gamma} \left\{M + \log \int \mathbb {E} _ {\epsilon} \left[ \exp \left(\frac {\gamma}{n} \sum_ {i = 1} ^ {n} \epsilon_ {i} h _ {x} (z _ {i})\right) \right] \mathrm {d} \mu^ {\circ} (x) \right\}, \\ \end{array}
$$
where the first inequality is by the Donsker-Varadha duality formula of the KL-divergence (Donsker and Varadhan, 1983). The second term in the right hand side can be bounded as
$$
\mathbb {E} _ {\epsilon} \left[ \exp \left(\frac {\gamma}{n} \sum_ {i = 1} ^ {n} \epsilon_ {i} h _ {x} (z _ {i})\right) \right] \leq \mathbb {E} _ {\epsilon} \left[ \exp \left(\frac {\gamma^ {2}}{n ^ {2}} \sum_ {i = 1} ^ {n} \epsilon_ {i} ^ {2} h _ {x} (z _ {i}) ^ {2}\right) \right] \leq \exp \left(\frac {\gamma^ {2}}{n} \bar {R} ^ {2}\right).
$$
$$
\widehat {\operatorname {R a d}} \left(\mathcal {F} _ {M} \left(\mu^ {\circ}\right)\right) \leq \frac {1}{\gamma} \left\{M + \frac {\gamma^ {2}}{n} \bar {R} ^ {2} \right\}.
$$
The right hand side can be minimized by  $\gamma = \sqrt{nM} / \bar{R}$ , which yields
$$
\widehat {\mathrm {R a d}} (\mathcal {F} _ {M} (\mu^ {\circ})) \leq 2 \bar {R} \sqrt {\frac {M}{n}}.
$$
This gives the assension.
C Proofs of Theorems 1 and 2
To show the theorems, we first prepare the following uniform bound, which is crucial to obtain the fast learning rate.
Lemma 3. It holds that
$$
\begin{array}{l} \bar {L} (\mu) - L (\mu) + L \left(\mu^ {*}\right) - \bar {L} \left(\mu^ {*}\right) \\ \leq 2 \left[ \mathrm {R a d} (\mathcal {F} _ {2 \mathrm {K L} (\mu , \mu^ {*}) \vee \frac {1}{n}} (\mu^ {*})) + (2 \| f _ {\mu} - f _ {\mu^ {*}} \| _ {\infty} \vee n ^ {- 1}) \sqrt {\frac {t + \log \log_ {2} (8 n ^ {2} M \bar {R})}{n}} \right], \\ \end{array}
$$
uniformly over  $\mu \in \mathcal{P}$  with  $\mathrm{KL}(\mu, \mu^*) \leq M$  with probability  $1 - \exp(-t)$  for any  $t > 0$ .
Proof. Let  $B_{k} = M / 2^{-k}$  and  $C_k = 2\bar{R} /2^{-k'}$  for  $k = 0,1,2,\ldots$ , and  $\mathcal{F}_{k,k'}\coloneqq \{\mu \in \mathcal{P}\mid \mathrm{KL}(\mu ,\mu^{*})\leq ,\| f_{\mu} - f_{\mu^{*}}\|_{\infty}\leq C_{k}\}$ . Then, the standard concentration inequality regarding to the Rademacher complexity (Theorem 3.1 of Mohri et al. (2012)) with the contraction inequality (Theorem 11.6 of Boucheron et al. (2013) or Theorem 4.12 of Ledoux and Talagrand (1991)) yields that
$$
P \left(\sup  _ {\mu \in \mathcal {F} _ {k, k ^ {\prime}}} \bar {L} (\mu) - L (\mu) + L \left(\mu^ {*}\right) - \bar {L} \left(\mu^ {*}\right) \leq 2 \left[ \operatorname {R a d} \left(\mathcal {F} _ {B _ {k}} \left(\mu^ {*}\right)\right) + C _ {k} \sqrt {\frac {t}{n}} \right]\right) \geq 1 - \exp (- t),
$$
for each  $k, k'$ . Then, since  $\|f_{\mu} - f_{\mu'}\|_{\infty} \leq 2\bar{R}$  for any  $\mu, \mu' \in \mathcal{P}$ , by taking uniform bound for all  $k = 0, \ldots, \log_2(n^2)$  and  $k' = 0, \ldots, \log_2(2\bar{R}n)$ , we have that
$$
\begin{array}{l} P \left(\bar {L} (\mu) - L (\mu) + L \left(\mu^ {*}\right) - \bar {L} \left(\mu^ {*}\right) \leq 2 \left[ \operatorname {R a d} \left(\mathcal {F} _ {2 \mathrm {K L} \left(\mu , \mu^ {*}\right) \vee \frac {1}{n}} \left(\mu^ {*}\right)\right) + \left(2 \| f _ {\mu} - f _ {\mu^ {*}} \| _ {\infty} \vee \frac {1}{n}\right) \sqrt {\frac {t}{n}} \right], \right. \\ \left. \forall f \in \mathcal {F} _ {n} \left(\mu^ {*}\right)\right) \geq 1 - \left(2 + \log_ {2} (n M) + \log_ {2} (2 \bar {R} n)\right) \exp (- t). \\ \end{array}
$$
By resetting  $t \gets t + \log \log_2(8n^2 M\bar{R})$ , we obtain the assertion.
By Assumption 2, we have
$$
\max  \{\bar {L} (\mu_ {[ \lambda^ {(K)} ]}), L (\mu_ {[ \lambda^ {(K)} ]}) \} \leq \ell (0) - c _ {0},
$$
$$
P \left(Y f _ {\mu_ {[ \lambda} (\kappa) ]} (Z) \geq c _ {0}\right) = 1.
$$
Let  $\hat{\mu} \coloneqq \mu^{(K)}$  and  $\mu^* \coloneqq \mu_{[\lambda^{(K)}]}$ . By the optimality condition of  $\hat{\mu}$ , we have
$$
L (\hat {\mu}) + \lambda^ {(K)} \mathrm {K L} (\nu , \hat {\mu}) \leq L (\mu_ {[ \lambda^ {(K)} ]}) + \lambda^ {(K)} \mathrm {K L} (\nu , \mu_ {[ \lambda^ {(K)} ]}) + \epsilon^ {*} <   1 / 2 - 2 \delta^ {*} = c _ {0}.
$$