SlowGuess/NeurIPS_2023_OCR huggingface dataset:
for  $a_j > 0$  ( $j = 1, \dots, N$ ) where  $a_j = a_{-i}$  ( $j \neq i$ ) for some  $a_{-i} > 0$ . By taking  $a_{-i} = N t \sigma_{v,k} / \tilde{\sigma}_{v,k}$  and  $a_i = t \sigma_{v,k} / \tilde{\sigma}_{v,k}$ , the right hand side can be bounded as
$$
2 t \tilde {\sigma} _ {v, k} \sigma_ {v, k}. \tag {15}
$$
Then, we return to the right hand side of Eq. (14). First, we note that
$$
\mathbb {E} _ {\omega_ {k}} \bigg [ \left\langle - \frac {Z _ {i}}{\sqrt {2 \lambda t}}, v _ {k} ^ {i} (\mathcal {X} _ {k} ^ {\prime} - t D _ {m} F (\mathcal {X} _ {k} ^ {\prime}) + \sqrt {2 \lambda t} Z, \omega_ {k} ^ {i}) - D _ {m} ^ {i} F (\mathcal {X} _ {k} ^ {\prime} - t D _ {m} F (\mathcal {X} _ {k} ^ {\prime}) + \sqrt {2 \lambda t} Z) \right\rangle \bigg ] = 0,
$$
for fixed  $Z$  and  $\mathcal{X}_k'$ . Hence, by taking this and Eq. (15) into account, the expectation of the right hand side Eq. (14) with respect to  $\omega_k$  can be further bounded as
$$
\sqrt {2 \frac {t}{\lambda}} \tilde {\sigma} _ {v, k} \sigma_ {v, k} + \sqrt {2} Q t ^ {3 / 2} \lambda^ {- 1 / 2} \sigma_ {v, k} ^ {2}. \tag {16}
$$
(2) Next, we evaluate the second term of the right hand side of Eq. (13). We have
$$
\begin{array}{l} \int \mathbb {E} _ {\widetilde {\mathcal {X}} _ {0} | \widetilde {\mathcal {X}} _ {t}} \left[ \left\langle \int \nabla_ {i} \tilde {\mu} _ {t} ^ {(N)} (\widetilde {\mathcal {X}} _ {t} | \mathcal {X} _ {0: k} ^ {\prime}) \mathrm {d} \mu^ {(N)} (\mathcal {X} _ {0: k} ^ {\prime}), \right. \right. \\ \left. v _ {k} ^ {i} \left(\mathcal {X} _ {k}, \omega_ {k} ^ {i}\right) - \nabla \frac {\delta F \left(\mu_ {\mathcal {X} _ {k}}\right)}{\delta \mu} \left(X _ {k} ^ {i}\right) - v _ {k} ^ {i} \left(\widetilde {\mathcal {X}} _ {t} ^ {i}, \omega_ {k} ^ {i}\right) + \nabla \frac {\delta F \left(\mu_ {\widetilde {\mathcal {X}} _ {t}}\right)}{\delta \mu} \left(\widetilde {X} _ {t} ^ {i}\right) \right\rangle \Bigg ] d \widetilde {\mathcal {X}} _ {t} \\ = \mathbb {E} _ {\widetilde {\mathcal {X} _ {0}}, \widetilde {\mathcal {X} _ {t}}} \left[ \left\langle \nabla_ {i} \log \left(\tilde {\mu} _ {t} ^ {(N)} (\widetilde {\mathcal {X} _ {t}}) / p ^ {(N)} (\widetilde {\mathcal {X} _ {t}})\right), \right. \right. \\ \left. v _ {k} ^ {i} (\mathcal {X} _ {k}, \omega_ {k} ^ {i}) - \nabla \frac {\delta F (\mu_ {\mathcal {X} _ {k}})}{\delta \mu} (X _ {k} ^ {i}) - v _ {k} ^ {i} (\widetilde {\mathcal {X} _ {t}}, \omega_ {k} ^ {i}) + \nabla \frac {\delta F (\mu_ {\widetilde {\mathcal {X} _ {t}}})}{\delta \mu} (\widetilde {X} _ {t} ^ {i}) \right\rangle \Biggr ] \\ + \mathbb {E} _ {\widetilde {\mathcal {X} _ {0}}, \widetilde {\mathcal {X} _ {t}}} \left[ \left\langle \nabla_ {i} \log \left(p ^ {(N)} (\widetilde {\mathcal {X} _ {t}})\right), \right. \right. \\ \left. v _ {k} ^ {i} \left(\mathcal {X} _ {k}, \omega_ {k} ^ {i}\right) - \nabla \frac {\delta F \left(\mu_ {\mathcal {X} _ {k}}\right)}{\delta \mu} \left(X _ {k} ^ {i}\right) - v _ {k} ^ {i} \left(\widetilde {\mathcal {X} _ {t}}, \omega_ {k} ^ {i}\right) + \nabla \frac {\delta F \left(\mu_ {\widetilde {\mathcal {X} _ {t}}}\right)}{\delta \mu} \left(\widetilde {X} _ {t} ^ {i}\right) \right\rangle \Bigg ]. \tag {17} \\ \end{array}
$$
The first term in the right hand side of this inequality (Eq. (17)) can be upper bounded by
$$
\begin{array}{l} \frac {\lambda}{4} \mathbb {E} \left[ \left\| \nabla_ {i} \log \left(\tilde {\mu} _ {t} ^ {(N)} (\widetilde {\mathcal {X}} _ {t}) / p ^ {(N)} (\widetilde {\mathcal {X}} _ {t})\right) \right\| ^ {2} \right] \\ + \frac {1}{\lambda} \int \mathbb {E} _ {\widetilde {\mathcal {X} _ {0}}, \widetilde {\mathcal {X} _ {t}}} \left[ \left\| v _ {k} ^ {i} \left(\mathcal {X} _ {k}, \omega_ {k} ^ {i}\right) - \nabla \frac {\delta F (\mu_ {\mathcal {X} _ {k}})}{\delta \mu} \left(X _ {k} ^ {i}\right) - v _ {k} ^ {i} \left(\widetilde {\mathcal {X} _ {t}}, \omega_ {k} ^ {i}\right) + \nabla \frac {\delta F (\mu_ {\widetilde {\mathcal {X} _ {t}}})}{\delta \mu} \left(\widetilde {X} _ {t} ^ {i}\right) \right\| ^ {2} \right] \\ \leq \frac {\lambda}{4} \mathbb {E} \left[ \left\| \nabla_ {i} \log \left(\tilde {\mu} _ {t} ^ {(N)} (\widetilde {\mathcal {X}} _ {t}) / p ^ {(N)} (\widetilde {\mathcal {X}} _ {t})\right) \right\| ^ {2} \right] \\ + \frac {1}{\lambda} \int \mathbb {E} _ {\widetilde {\mathcal {X} _ {0}}, \widetilde {\mathcal {X} _ {t}}} \left[ \left\| v _ {k} ^ {i} ((\mathcal {X} _ {k}, \mathcal {X} _ {0: k - 1}), \omega_ {k} ^ {i}) - \nabla \frac {\delta F (\mu_ {\mathcal {X} _ {k}})}{\delta \mu} (X _ {k} ^ {i}) - v _ {k} ^ {i} ((\widetilde {\mathcal {X} _ {t}}, \mathcal {X} _ {0: k - 1}), \omega_ {k} ^ {i}) + \nabla \frac {\delta F (\mu_ {\widetilde {\mathcal {X} _ {t}}})}{\delta \mu} (\widetilde {X} _ {t} ^ {i}) \right\| ^ {2} \right], \\ \end{array}
$$
where we used Jensen's inequality. By Lemma 2 (the same bound applies to  $v_{k}^{i}$  by the same proof), we see that the second term in the right hand side is bounded by
$$
\frac {2}{\lambda} \delta_ {\eta_ {k}}. \tag {18}
$$
Finally, we evaluate the second term in the right hand side of Eq. (17). Note that
$$
\mathbb {E} _ {\widetilde {\mathcal {X} _ {0}}, \widetilde {\mathcal {X} _ {t}}} \left[ \left\langle \nabla_ {i} \log \left(p ^ {(N)} (\widetilde {\mathcal {X} _ {t}})\right), \right. \right.
$$
$$
\begin{array}{l} \left. v _ {k} ^ {i} \left(\mathcal {X} _ {k}, \omega_ {k} ^ {i}\right) - \nabla \frac {\delta F \left(\mu_ {\mathcal {X} _ {k}}\right)}{\delta \mu} \left(X _ {k} ^ {i}\right) - v _ {k} ^ {i} \left(\widetilde {\mathcal {X}} _ {t}, \omega_ {k} ^ {i}\right) + \nabla \frac {\delta F \left(\mu_ {\widetilde {\mathcal {X}} _ {t}}\right)}{\delta \mu} \left(\widetilde {X} _ {t} ^ {i}\right) \right\rangle \Bigg ] \\ = \mathbb {E} _ {\widetilde {\mathcal {X}} _ {0}, \widetilde {\mathcal {X}} _ {t}} \left[ \right.\left\langle \nabla_ {i} \log \left(p ^ {(N)} (\widetilde {\mathcal {X}} _ {t})\right) - \nabla_ {i} \log \left(p ^ {(N)} (\widetilde {\mathcal {X}} _ {0})\right) + \nabla_ {i} \log \left(p ^ {(N)} (\widetilde {\mathcal {X}} _ {0})\right), \right. \\ \left. v _ {k} ^ {i} \left(\mathcal {X} _ {k}, \omega_ {k} ^ {i}\right) - \nabla \frac {\delta F \left(\mu \mathcal {X} _ {k}\right)}{\delta \mu} \left(X _ {k} ^ {i}\right) - v _ {k} ^ {i} \left(\widetilde {\mathcal {X}} _ {t}, \omega_ {k} ^ {i}\right) + \nabla \frac {\delta F \left(\mu \widetilde {\mathcal {X}} _ {t}\right)}{\delta \mu} \left(\widetilde {X} _ {t} ^ {i}\right) \right\rangle \Bigg ] \\ \leq \frac {2 \delta_ {\eta_ {k}}}{\lambda} + \mathbb {E} _ {\widetilde {\mathcal {X} _ {0}}, \widetilde {\mathcal {X} _ {t}}} \left[ \left\langle \nabla_ {i} \log \left(p ^ {(N)} (\widetilde {\mathcal {X} _ {0}})\right), \right. \right. \\ \left. v _ {k} ^ {i} \left(\mathcal {X} _ {k}, \omega_ {k} ^ {i}\right) - \nabla \frac {\delta F \left(\mu \mathcal {X} _ {k}\right)}{\delta \mu} \left(X _ {k} ^ {i}\right) - v _ {k} ^ {i} \left(\widetilde {\mathcal {X}} _ {t}, \omega_ {k} ^ {i}\right) + \nabla \frac {\delta F \left(\mu \widetilde {\mathcal {X}} _ {t}\right)}{\delta \mu} \left(\widetilde {X} _ {t} ^ {i}\right) \right\rangle \Bigg ] , \\ \end{array}
$$
where we used Lemma 2 with the same argument as Eq. (18) and Young's inequality. Taking the expectation with respect to  $\omega_{k}^{i}$ , it holds that
$$
\mathbb {E} _ {\omega_ {k} ^ {i}} \left\{\mathbb {E} _ {\widetilde {\mathcal {X} _ {0}}, \widetilde {\mathcal {X} _ {t}}} \left[ \left\langle \nabla_ {i} \log \left(p ^ {(N)} (\widetilde {\mathcal {X} _ {0}})\right), v _ {k} ^ {i} (\mathcal {X} _ {k}, \omega_ {k} ^ {i}) - \nabla \frac {\delta F (\mu_ {\mathcal {X} _ {k}})}{\delta \mu} (X _ {k} ^ {i}) \right\rangle \right] \right\} = 0.
$$
Hence, it suffices to evaluate the term
$$
- \mathbb {E} _ {\widetilde {\mathcal {X} _ {0}}, \widetilde {\mathcal {X} _ {t}}} \left[ \left\langle \nabla_ {i} \log \left(p ^ {(N)} (\widetilde {\mathcal {X} _ {0}})\right), v _ {k} ^ {i} (\widetilde {\mathcal {X} _ {t}}, \omega_ {k} ^ {i}) - \nabla \frac {\delta F (\mu_ {\widetilde {\mathcal {X} _ {t}}})}{\delta \mu} (\widetilde {X} _ {t} ^ {i}) \right\rangle \right].
$$
Here, let  $\hat{\mathcal{X}_t} = (\hat{X}_t^i)_{i=1}^n$  be the following stochastic process:
$$
\hat {\mathcal {X}} _ {0} = \mathcal {X} _ {k}.
$$
$$
\mathrm {d} \hat {X} _ {t} ^ {i} = - \nabla \frac {\delta F (\mu_ {k})}{\delta \mu} (X _ {k} ^ {i}) \mathrm {d} t + \sqrt {2 \lambda} \mathrm {d} W _ {t} ^ {i},
$$
for  $0 \leq t \leq \eta$ , where  $(W_{t}^{i})_{t}$  is the same Brownian motion as that drives  $\widetilde{X}_{t}^{i}$ . Then, the term we are interested in can be evaluated as
$$
\begin{array}{l} - \mathbb {E} _ {\widetilde {\mathcal {X}} _ {0}, \widetilde {\mathcal {X}} _ {t}, \omega_ {k}} \left[ \left\langle \nabla_ {i} \log \left(p ^ {(N)} (\widetilde {\mathcal {X}} _ {0})\right), v _ {k} ^ {i} (\widetilde {\mathcal {X}} _ {t}, \omega_ {k} ^ {i}) - \nabla \frac {\delta F (\mu_ {\widetilde {\mathcal {X}} _ {t}})}{\delta \mu} (\widetilde {X} _ {t} ^ {i}) \right\rangle \right] \\ = - \mathbb {E} _ {\widetilde {\mathcal {X}} _ {0}, \widetilde {\mathcal {X}} _ {t}, \hat {\mathcal {X}} _ {t}, \omega_ {k}} \left[ \left\langle \nabla_ {i} \log \left(p ^ {(N)} (\widetilde {\mathcal {X}} _ {0})\right), v _ {k} ^ {i} (\widetilde {\mathcal {X}} _ {t}, \omega_ {k} ^ {i}) - \nabla \frac {\delta F (\mu \widetilde {\mathcal {X}} _ {t})}{\delta \mu} (\widetilde {X} _ {t} ^ {i}) \right. \right. \\ \left. - v _ {k} ^ {i} \left(\hat {\mathcal {X}} _ {t}, \omega_ {k} ^ {i}\right) - \nabla \frac {\delta F \left(\mu_ {\hat {\mathcal {X}} _ {t}}\right)}{\delta \mu} \left(\hat {X} _ {t} ^ {i}\right) + v _ {k} ^ {i} \left(\hat {\mathcal {X}} _ {t}, \omega_ {k} ^ {i}\right) - \nabla \frac {\delta F \left(\mu_ {\hat {\mathcal {X}} _ {t}}\right)}{\delta \mu} \left(\hat {X} _ {t} ^ {i}\right) \right\rangle \Bigg ] \\ = - \mathbb {E} _ {\widetilde {\mathcal {X}} _ {0}, \widetilde {\mathcal {X}} _ {t}, \hat {\mathcal {X}} _ {t}, \omega_ {k}} \left[ \left\langle \nabla_ {i} \log \left(p ^ {(N)} (\widetilde {\mathcal {X}} _ {0})\right), v _ {k} ^ {i} (\widetilde {\mathcal {X}} _ {t}, \omega_ {k} ^ {i}) - \nabla \frac {\delta F (\mu_ {\widetilde {\mathcal {X}} _ {t}})}{\delta \mu} (\widetilde {X} _ {t} ^ {i}) \right. \right. \\ \left. \left. - v _ {k} ^ {i} \left(\hat {\mathcal {X}} _ {t}, \omega_ {k} ^ {i}\right) - \nabla \frac {\delta F \left(\mu_ {\hat {\mathcal {X}} _ {t}}\right)}{\delta \mu} \left(\hat {X} _ {t} ^ {i}\right) \right\rangle \right] \\ \end{array}
$$
$$
\leq \mathbb {E} _ {\widetilde {\mathcal {X}} _ {0}, \widetilde {\mathcal {X}} _ {t}, \hat {\mathcal {X}} _ {t}} \left\{\left\| \nabla_ {i} \log \left(p ^ {(N)} (\widetilde {\mathcal {X}} _ {0})\right) \right\| \cdot \right.
$$
$$
\mathbb {E} _ {\omega_ {k}} \left[ \left\| v _ {k} ^ {i} \left(\widetilde {\mathcal {X} _ {t}}, \omega_ {k} ^ {i}\right) - \nabla \frac {\delta F \left(\mu_ {\widetilde {\mathcal {X} _ {t}}}\right)}{\delta \mu} \left(\widetilde {X} _ {t} ^ {i}\right) - v _ {k} ^ {i} \left(\hat {\mathcal {X} _ {t}}, \omega_ {k} ^ {i}\right) - \nabla \frac {\delta F \left(\mu_ {\hat {\mathcal {X} _ {t}}}\right)}{\delta \mu} \left(\hat {X} _ {t} ^ {i}\right) \right\| \right] \Bigg \}. \tag {19}
$$
With the same reasoning as in Eq. (16), it holds that
$$
\begin{array}{l} \mathbb {E} _ {\omega_ {k}} \left[ \left\| v _ {k} ^ {i} \left(\widetilde {\mathcal {X} _ {t}}, \omega_ {k} ^ {i}\right) - \nabla \frac {\delta F \left(\mu_ {\widetilde {\mathcal {X} _ {t}}}\right)}{\delta \mu} \left(\hat {X} _ {t} ^ {i}\right) - v _ {k} ^ {i} \left(\hat {\mathcal {X} _ {t}}, \omega_ {k} ^ {i}\right) - \nabla \frac {\delta F \left(\mu_ {\hat {\mathcal {X} _ {t}}}\right)}{\delta \mu} \left(\widetilde {X} _ {t} ^ {i}\right) \right\| \right] \\ \leq 2 t \tilde {\sigma} _ {v, k} \sigma_ {v, k} + 2 Q t ^ {2} \sigma_ {v, k} ^ {2}. \\ \end{array}
$$
Moreover, Assumption 2 and Lemm 1 yield that
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \nabla_ {i} \log \left(p ^ {(N)} (\widetilde {\mathcal {X} _ {0}})\right) \right\| \right] \leq \frac {1}{\lambda} \left(\sup  _ {\mu \in \mathcal {P}, x \in \mathbb {R} ^ {d}} \left\| \nabla \frac {\delta U (\mu)}{\delta \mu} (x) \right\| + \mathbb {E} [ \| \nabla r (\widetilde {X} _ {0} ^ {i}) \| ]\right) \\ \leq \frac {1}{\lambda} \left(R + \mathbb {E} [ \| \nabla r (\widetilde {X} _ {0} ^ {i}) - r (0) + r (0) \| ]\right) \\ \leq \frac {1}{\lambda} \left(R + \lambda_ {2} \mathbb {E} [ \| \widetilde {X} _ {0} ^ {i} \| ]\right) \\ \leq \frac {1}{\lambda} \left(R + \lambda_ {2} \sqrt {\mathbb {E} [ \| \widetilde {X} _ {0} ^ {i} \| ^ {2} ]}\right) \\ \leq \frac {1}{\lambda} \left(R + \lambda_ {2} \bar {R}\right), \\ \end{array}
$$
where the second and third inequalities are due to Assumption 2 and the last inequality is by Lemma 1. Then, the right hand side of Eq. (19) can be bounded by
$$
\frac {1}{\lambda} \left(R + \lambda_ {2} \bar {R}\right) \left(2 t \tilde {\sigma} _ {v, k} \sigma_ {v, k} + 2 Q t ^ {2} \sigma_ {v, k} ^ {2}\right).
$$
where we used Assumption 2 and Lemm 1. Hence, the second term of the right hand side of Eq. (13) (which is same as Eq. (17)) can be bounded by
$$
\frac {4}{\lambda} \delta_ {\eta_ {k}} + \frac {1}{\lambda} \left(R + \lambda_ {2} \bar {R}\right) \left(2 t \tilde {\sigma} _ {v, k} \sigma_ {v, k} + 2 Q t ^ {2} \sigma_ {v, k} ^ {2}\right).
$$
(3) By summing up the results in (1) and (2), we have that
$$
\begin{array}{l} C - (\mathbf {I}) \leq \frac {\lambda}{4} \mathbb {E} \left[ \left\| \nabla_ {i} \log \left(\tilde {\mu} _ {t} ^ {(N)} \left(\widetilde {\mathcal {X}} _ {t}\right) / p ^ {(N)} \left(\widetilde {\mathcal {X}} _ {t}\right)\right) \right\| ^ {2} \right] \\ + 4 \delta_ {\eta_ {k}} + (R + \lambda_ {2} \bar {R}) \left(1 + \sqrt {\frac {\lambda}{t}}\right) \left(2 \frac {t}{\lambda} \tilde {\sigma} _ {v, k} \sigma_ {v, k} + 2 \frac {Q t ^ {2}}{\lambda} \sigma_ {v, k} ^ {2}\right). \\ \end{array}
$$
Evaluation of  $C$  -II
Next, we evaluate the term  $C - (\Pi)$ . Here, let  $\hat{\mathcal{X}}_t = (\hat{X}_t^i)_{i=1}^n$  be the following stochastic process:
for  $0 \leq t \leq \eta$ , where  $\left(W_{t}^{i}\right)_{t}$  is the same Brownian motion as that drives  $\widetilde{X}_{t}^{i}$ . By definition, we notice that
$$
\mathrm {d} \left(\widetilde {X} _ {t} ^ {i} - \hat {X} _ {t} ^ {i}\right) = \left(\nabla \frac {\delta F (\mu_ {k})}{\delta \mu} \left(X _ {k} ^ {i}\right) - v _ {k} ^ {i}\right) \mathrm {d} t,
$$
which yields that
$$
\widetilde {X} _ {t} ^ {i} - \hat {X} _ {t} ^ {i} = t \left(\nabla \frac {\delta F (\mu_ {k})}{\delta \mu} \left(X _ {k} ^ {i}\right) - v _ {k} ^ {i}\right). \tag {20}
$$
By subtracting and adding the derivative at  $\hat{\mathcal{X}}_t$  from  $\nabla_i\log (p^{(N)}(\widetilde{\mathcal{X}}_t))$ , we have
$$
\begin{array}{l} \nabla_ {i} \log \Big (p ^ {(N)} (\widetilde {\mathcal {X} _ {t}}) \Big) = - \frac {1}{\lambda} \nabla \frac {\delta F (\mu_ {\widetilde {\mathcal {X} _ {t}}})}{\delta \mu} (\widetilde {X} _ {t} ^ {i}) \\ = - \frac {1}{\lambda} \left(\nabla \frac {\delta F (\mu_ {\widetilde {\mathcal {X} _ {t}}})}{\delta \mu} (\widetilde {X} _ {t} ^ {i}) - \nabla \frac {\delta F (\mu_ {\hat {\mathcal {X}} _ {t}})}{\delta \mu} (\hat {X} _ {t} ^ {i})\right) - \frac {1}{\lambda} \nabla \frac {\delta F (\mu_ {\hat {\mathcal {X}} _ {t}})}{\delta \mu} (\hat {X} _ {t} ^ {i}). \\ \end{array}
$$
By Assumptions 1 and 2, the first two terms in the right hand side can be bounded as
$$
\left\| \nabla \frac {\delta F (\mu_ {\widehat {\mathcal {X} _ {t}}})}{\delta \mu} \left(\widetilde {X} _ {t} ^ {i}\right) - \nabla \frac {\delta F (\mu_ {\widehat {\mathcal {X} _ {t}}})}{\delta \mu} \left(\hat {X} _ {t} ^ {i}\right) \right\|
$$
$$
\begin{array}{l} = \left\| \nabla \frac {\delta U (\mu_ {\widetilde {\mathcal {X} _ {t}}})}{\delta \mu} \left(\widetilde {X} _ {t} ^ {i}\right) + \nabla r \left(\widetilde {X} _ {t} ^ {i}\right) - \nabla \frac {\delta U (\mu_ {\hat {\mathcal {X}} _ {t}})}{\delta \mu} \left(\hat {X} _ {t} ^ {i}\right) - \nabla r \left(\hat {X} _ {t} ^ {i}\right) \right\| \\ \leq L \big (W _ {2} (\mu_ {\widetilde {\mathcal {X}} _ {t}}, \mu_ {\hat {\mathcal {X}} _ {t}}) + \| \widetilde {X} _ {t} ^ {i} - \hat {X} _ {t} ^ {i} \| \big) + \lambda_ {2} \| \widetilde {X} _ {t} ^ {i} - \hat {X} _ {t} ^ {i} \| \\ \leq t \left(L + \lambda_ {2}\right) \left(\sqrt {\frac {1}{N} \sum_ {i = 1} ^ {N} \left\| v _ {k} ^ {i} - \nabla \frac {\delta F (\mu_ {k})}{\delta \mu} \left(X _ {k} ^ {i}\right) \right\| ^ {2}} + \left\| v _ {k} ^ {i} - \nabla \frac {\delta F (\mu_ {k})}{\delta \mu} \left(X _ {k} ^ {i}\right) \right\|\right), \tag {21} \\ \end{array}
$$
where the first inequality follows from
$$
\begin{array}{l} \| \nabla r (x) - \nabla r \left(x ^ {\prime}\right) \| = \left\| \int_ {0} ^ {1} \left[ \nabla \nabla^ {\top} r (\theta x + (1 - t) x) \right] \left(x - x ^ {\prime}\right) d \theta \right\| \\ \leq \lambda_ {2} \int_ {0} ^ {1} \| x - x ^ {\prime} \| \mathrm {d} \theta = \lambda_ {2} \| x - x ^ {\prime} \| \tag {22} \\ \end{array}
$$
by Assumption 1 and we used Eq. (20) in the last inequality. Therefore, by noticing  $\widetilde{\mathcal{X}_0} = \mathcal{X}_k$ , we can obtain the following evaluation:
$$
\begin{array}{l} \sum_ {i = 1} ^ {N} C - (\mathrm {I I}) \\ = \sum_ {i = 1} ^ {N} \left| \mathbb {E} _ {v _ {k} ^ {i}} \left[ \mathbb {E} _ {\widetilde {\mathcal {X}} _ {t}, \mathcal {X} _ {0: k}} \left[ \left\langle \nabla_ {i} \log \left(p ^ {(N)} (\widetilde {\mathcal {X}} _ {t})\right), v _ {k} ^ {i} \left(\mathcal {X} _ {0: k}\right) - \nabla \frac {\delta F \left(\mu_ {\widetilde {\mathcal {X}} _ {0}}\right)}{\delta \mu} \left(\widetilde {X} _ {0} ^ {i}\right) \right\rangle \right] \right] \right| \\ \leq \frac {2 (L + \lambda_ {2}) ^ {2}}{\lambda} t \sum_ {i = 1} ^ {N} \mathbb {E} _ {\left(v _ {k} ^ {i}\right) _ {i = 1} ^ {N}, \mathcal {X} _ {0: k}} \left[ \left\| v _ {k} ^ {i} \left(\mathcal {X} _ {0: k}\right) - \nabla \frac {\delta F \left(\mu_ {\widetilde {\mathcal {X} _ {0}}}\right)}{\delta \mu} \left(\widetilde {X} _ {0} ^ {i}\right) \right\| ^ {2} \right] \\ \leq t N \frac {2 (L + \lambda_ {2}) ^ {2}}{\lambda} \sigma_ {v, k} ^ {2}. \\ \end{array}
$$
Combining the bounds on  $C-(I)$  and  $C-(II)$ :
$$
\begin{array}{l} \sum_ {i = 1} ^ {N} \lambda C \leq \lambda \sum_ {i = 1} ^ {N} (C - (\mathrm {I}) + C - (\mathrm {I I})) \\ \leq \frac {\lambda^ {2}}{4} \sum_ {i = 1} ^ {N} \mathbb {E} \left[ \left\| \nabla_ {i} \log \left(\tilde {\mu} _ {t} ^ {(N)} (\widetilde {\mathcal {X}} _ {t}) / p ^ {(N)} (\widetilde {\mathcal {X}} _ {t})\right) \right\| ^ {2} \right] \\ + N \left\{4 \delta_ {\eta_ {k}} + (R + \lambda_ {2} \bar {R}) \left(1 + \sqrt {\frac {\lambda}{t}}\right) \left(2 t \tilde {\sigma} _ {v, k} \sigma_ {v, k} + 2 Q t ^ {2} \sigma_ {v, k} ^ {2}\right) + t [ 2 (L + \lambda_ {2}) ^ {2} \sigma_ {v, k} ^ {2} ] \right\}. \\ \end{array}
$$
B.3 Putting Things Together
By Gronwall lemma (see Mischler (2019), for example), we arrive at
$$
\begin{array}{l} \frac {1}{N} \mathbb {E} _ {\omega_ {0: k}} \left[ \mathcal {F} ^ {N} \left(\mu_ {k + 1} ^ {(N)}\right) \right] - \mathcal {F} \left(\mu^ {*}\right) \\ \leq \exp (- \lambda \alpha \eta_ {k} / 2) \left(\frac {1}{N} \mathbb {E} _ {\omega_ {0: k - 1}} [ \mathcal {F} ^ {N} (\mu_ {k} ^ {(N)}) ] - \mathcal {F} (\mu^ {*})\right) + \eta_ {k} \left(\delta_ {\eta_ {k}} + \frac {C _ {\lambda}}{N}\right) + \sigma_ {v, k} ^ {2} \eta_ {k}. \\ \end{array}
$$
In addition, if Assumption 4 is satisfied, we have
$$
\begin{array}{l} \frac {1}{N} \mathbb {E} _ {\omega_ {0: k}} [ \mathcal {F} ^ {N} (\mu_ {k + 1} ^ {(N)}) ] - \mathcal {F} (\boldsymbol {\mu} ^ {*}) \\ \leq \exp (- \lambda \alpha \eta_ {k} / 2) \left(\frac {1}{N} \mathbb {E} _ {\omega_ {0: k - 1}} [ \mathcal {F} ^ {N} (\mu_ {k} ^ {(N)}) ] - \mathcal {F} (\mu^ {*})\right) + \eta_ {k} \left(\delta_ {\eta_ {k}} + \frac {C _ {\lambda}}{N}\right) \\ + \left[ 4 \eta_ {k} \delta_ {\eta_ {k}} + (R + \lambda_ {2} \bar {R}) \left(1 + \sqrt {\frac {\lambda}{\eta_ {k}}}\right) (\eta_ {k} ^ {2} \tilde {\sigma} _ {v, k} \sigma_ {v, k} + Q \eta_ {k} ^ {3} \sigma_ {v, k} ^ {2}) + \eta_ {k} ^ {2} (L + \lambda_ {2}) ^ {2} \sigma_ {v, k} ^ {2} \right]. \\ \end{array}
$$
C Proof of Theorem 3: F-MFLD and SGD-MFLD
Applying the Gronwall lemma (Mischler, 2019) with Theorem 2, we have that
$$
\begin{array}{l} \frac {1}{N} \mathbb {E} _ {\omega_ {0: k - 1}} \left[ \mathcal {F} ^ {N} \left(\mu_ {k} ^ {(N)}\right) \right] - \mathcal {F} \left(\mu^ {*}\right) \\ \leq \exp \left(- \lambda \alpha \sum_ {j = 0} ^ {k - 1} \eta_ {j} / 2\right) \Delta_ {0} + \sum_ {i = 0} ^ {k - 1} \exp \left(- \lambda \alpha \sum_ {j = i} ^ {k - 1} \eta_ {j} / 2\right) \left[ \Upsilon_ {i} + \eta_ {i} \left(\delta_ {\eta_ {i}} + \frac {C _ {\lambda}}{N}\right) \right]. \tag {23} \\ \end{array}
$$
Here, remember that Assumption 5 yields that
$$
\sigma_ {v, k} ^ {2} \leq \frac {R ^ {2}}{B}, \quad \tilde {\sigma} _ {v, k} ^ {2} \leq \frac {R ^ {2}}{B}.
$$
Under Assumption 5, we can easily check that Assumption 4 holds with  $Q = R$ . When  $\eta_k = \eta$  for all  $k \geq 0$ , we have a uniform upper bound of  $\Upsilon_k$  as
$$
\Upsilon_ {k} \leq \left\{ \begin{array}{l l} 4 \eta \delta_ {\eta} + \left[ R + \lambda_ {2} \bar {R} + (L + \lambda_ {2}) ^ {2} \right] \left(1 + \sqrt {\frac {\lambda}{\eta}}\right) \eta^ {2} \frac {R ^ {2}}{B} \\ \hskip 1 4. 2 2 6 3 7 8 p t + \left(R + \lambda_ {2} \bar {R}\right) R \left(1 + \sqrt {\frac {\lambda}{\eta}}\right) \eta^ {3} \frac {R ^ {2}}{B}, & \text {w i t h A s s u m p t i o n 5 - (i i)}, \\ \frac {R ^ {2}}{B} \eta , & \text {w i t h o u t A s s u m p t i o n 5 - (i i)}. \end{array} \right.
$$
We denote the right hand side as  $\tilde{\Upsilon}$ . Then we have
$$
\begin{array}{l} \frac {1}{N} \mathbb {E} _ {\omega_ {0: k - 1}} [ \mathcal {F} ^ {N} (\mu_ {k} ^ {(N)}) ] - \mathcal {F} (\boldsymbol {\mu} ^ {*}) \\ \leq \exp (- \lambda \alpha \eta k / 2) \Delta_ {0} + \sum_ {i = 0} ^ {k - 1} \exp (- \lambda \alpha (k - 1 - i) \eta / 2) \left[ \bar {\Upsilon} + \eta \left(\delta_ {\eta} + \frac {C _ {\lambda}}{N}\right) \right] \\ \leq \exp (- \lambda \alpha \eta k / 2) \Delta_ {0} + \frac {1 - \exp (- \lambda \alpha k \eta / 2)}{1 - \exp (- \lambda \alpha \eta / 2)} \left[ \bar {\Upsilon} + \eta \left(\delta_ {\eta} + \frac {C _ {\lambda}}{N}\right) \right] \\ \leq \exp (- \lambda \alpha \eta k / 2) \Delta_ {0} + \frac {4}{\lambda \alpha \eta} \left[ \bar {\Upsilon} + \eta \left(\delta_ {\eta} + \frac {C _ {\lambda}}{N}\right) \right] \quad (\because \lambda \alpha \eta / 2 \leq 1 / 2) \\ = \exp (- \lambda \alpha \eta k / 2) \Delta_ {0} + \frac {4}{\lambda \alpha} \bar {L} ^ {2} C _ {1} (\lambda \eta + \eta^ {2}) + \frac {4}{\lambda \alpha \eta} \bar {\Upsilon} + \frac {4 C _ {\lambda}}{\lambda \alpha N}. \\ \end{array}
$$
Then, by taking
$$
k \geq \frac {2}{\lambda \alpha \eta} \log (\Delta_ {0} / \epsilon),
$$
then the right hand side can be bounded as
$$
\begin{array}{l} \frac {1}{N} \mathbb {E} \left[ \mathcal {F} ^ {N} \left(\mu_ {k} ^ {(N)}\right) \right] - \mathcal {F} \left(\mu^ {*}\right) \\ \leq \epsilon + \frac {4}{\lambda \alpha} \bar {L} ^ {2} C _ {1} (\lambda \eta + \eta^ {2}) + \frac {4}{\lambda \alpha \eta} \bar {\Upsilon} + \frac {4 C _ {\lambda}}{\lambda \alpha N}. \\ \end{array}
$$
(1) Hence, without Assumption 5-(ii), if we take
$$
\eta \leq \frac {\lambda \alpha \epsilon}{8 \bar {L} ^ {2}} (C _ {1} \lambda) ^ {- 1} \wedge \frac {1}{8 \bar {L}} \sqrt {\frac {2 \lambda \alpha \epsilon}{C _ {1}}}, \quad B \geq 4 R ^ {2} / (\lambda \alpha \epsilon)
$$
$$
\frac {1}{N} \mathbb {E} \left[ \mathcal {F} ^ {N} \left(\mu_ {k} ^ {(N)}\right) \right] - \mathcal {F} \left(\mu^ {*}\right) \leq 3 \epsilon + \frac {4 C _ {\lambda}}{\lambda \alpha N}.
$$
We can easily check that the number of iteration  $k$  satisfies
$$
k = O \left(\frac {\bar {L} ^ {2}}{\lambda \alpha \epsilon} \lambda + \frac {\bar {L}}{\sqrt {\lambda \alpha \epsilon}}\right) \frac {1}{\lambda \alpha} \log (\epsilon^ {- 1}),
$$
in this setting.
(2) On the other hand, under Assumption 5-(ii), if we take
$$
\eta \leq \frac {\lambda \alpha \epsilon}{4 0 \bar {L} ^ {2}} \left(C _ {1} \lambda\right) ^ {- 1} \wedge \frac {1}{\bar {L}} \sqrt {\frac {\lambda \alpha \epsilon}{4 0 C _ {1}}} \wedge 1,
$$