SlowGuess/NeurIPS_2023_OCR huggingface dataset:
It is worth noting that the Wasserstein contraction bound we obtained in Lemma E.1 in the non-convex case differs from the one we obtained in the strongly-convex case (Lemma D.1) in the sense that the right hand side of (E.3) is no longer  $\| \theta - \tilde{\theta} \|$  so that Lemma 2.1 is not directly applicable. Instead, in the following, we will provide a modification of Lemma 2.1, which will be used in proving Theorem 3.3 in this paper. The definitions of the notations used in the following lemma can be found in Section 2.2.
Lemma E.5. Assume that there exist some  $\rho \in [0,1)$  and  $C\in (0,\infty)$  such that
$$
\sup  _ {\theta , \tilde {\theta} \in \mathbb {R} ^ {d}: \theta \neq \tilde {\theta}} \frac {\mathcal {W} _ {1} \left(P ^ {n} (\theta , \cdot) , P ^ {n} (\tilde {\theta} , \cdot)\right)}{d _ {\psi} \left(\delta_ {\theta} , \delta_ {\tilde {\theta}}\right)} \leq C \rho^ {n}, \tag {E.34}
$$
$$
(\hat {P} \hat {V}) (\theta) \leq \delta \hat {V} (\theta) + L. \tag {E.35}
$$
$$
\mathcal {W} _ {1} \left(p _ {n}, \hat {p} _ {n}\right) \leq C \left(\rho^ {n} d _ {\psi} \left(p _ {0}, \hat {p} _ {0}\right) + \left(1 - \rho^ {n}\right) \frac {\gamma \kappa}{1 - \rho}\right), \tag {E.36}
$$
$$
\gamma := \sup  _ {\theta \in \mathbb {R} ^ {d}} \frac {d _ {\psi} (\delta_ {\theta} P , \delta_ {\theta} \hat {P})}{\hat {V} (\theta)}, \quad \kappa := \max  \left\{\int_ {\mathbb {R} ^ {d}} \hat {V} (\theta) d \hat {p} _ {0} (\theta), \frac {L}{1 - \delta} \right\}. \tag {E.37}
$$
Proof. The proof is based on the modification of the proof of Lemma 2.1 (Theorem 3.1 in [RS18]). By induction we have
$$
\tilde {p} _ {n} - p _ {n} = \left(\tilde {p} _ {0} - p _ {0}\right) P ^ {n} + \sum_ {i = 0} ^ {n - 1} \tilde {p} _ {i} (\tilde {P} - P) P ^ {n - i - 1}, \quad n \in \mathbb {N}. \tag {E.38}
$$
$$
d _ {\psi} \left(\tilde {p} _ {i} P, \tilde {p} _ {i} \tilde {P}\right) \leq \int_ {\mathbb {R} ^ {d}} d _ {\psi} \left(\delta_ {\theta} P, \delta_ {\theta} \tilde {P}\right) d \tilde {p} _ {i} (\theta) \leq \gamma \int_ {\mathbb {R} ^ {d}} \tilde {V} (\theta) d \tilde {p} _ {i} (\theta). \tag {E.39}
$$
Moreover, for any  $i = 0,1,2,\ldots$
$$
\int_ {\mathbb {R} ^ {d}} \tilde {V} (\theta) d \tilde {p} _ {i} (\theta) = \int_ {\mathbb {R} ^ {d}} \tilde {P} ^ {i} \tilde {V} (\theta) d \tilde {p} _ {0} (\theta) \leq \delta^ {i} \tilde {p} _ {0} (\tilde {V}) + \frac {L (1 - \delta^ {i})}{1 - \delta} \leq \max  \left\{\tilde {p} _ {0} (\tilde {V}), \frac {L}{1 - \delta} \right\}, \tag {E.40}
$$
so that we obtain  $d_{\psi}(\tilde{p}_iP,\tilde{p}_i\tilde{P})\leq \gamma \kappa$  . Therefore, we have
$$
\mathcal {W} _ {1} \left(\tilde {p} _ {i} \tilde {P} P ^ {n - i - 1}, \tilde {p} _ {i} P P ^ {n - i - 1}\right) \leq C \rho^ {n - i - 1} d _ {\psi} \left(\tilde {p} _ {i} P, \tilde {p} _ {i} \tilde {P}\right) \leq C \rho^ {n - i - 1} \gamma \kappa . \tag {E.41}
$$
By the triangle inequality of the Wasserstein distance, we have
$$
\begin{array}{l} \mathcal {W} _ {1} \left(p _ {n}, \tilde {p} _ {n}\right) \leq \mathcal {W} _ {1} \left(p _ {0} P ^ {n}, \tilde {p} _ {0} P ^ {n}\right) + \sum_ {i = 0} ^ {n - 1} \mathcal {W} _ {1} \left(\tilde {p} _ {i} \tilde {P} P ^ {n - i - 1}, \tilde {p} _ {i} P P ^ {n - i - 1}\right) \\ \leq C \rho^ {n} d _ {\psi} \left(p _ {0}, \tilde {p} _ {0}\right) + C \sum_ {i = 0} ^ {n - 1} \rho^ {n - i - 1} \gamma \kappa . \tag {E.42} \\ \end{array}
$$
Lemma E.6. Under Assumption 3.3, we have
$$
\left\| \theta_ {*} \right\| \leq \frac {\sup  _ {x \in \mathcal {X}} \| \nabla f (0 , x) \| + \sqrt {\sup  _ {x \in \mathcal {X}} \| \nabla f (0 , x) \| ^ {2} + 4 m K}}{2 m}, \tag {E.43}
$$
$$
\left\| \hat {\theta} _ {*} \right\| \leq \frac {\sup  _ {x \in \mathcal {X}} \| \nabla f (0 , x) \| + \sqrt {\sup  _ {x \in \mathcal {X}} \| \nabla f (0 , x) \| ^ {2} + 4 m K}}{2 m}. \tag {E.44}
$$
Proof. By Assumption 3.3, we have
$$
\begin{array}{l} \left\langle \nabla \hat {F} (0, X _ {n}) - \nabla \hat {F} (\theta_ {*}, X _ {n}), 0 - \theta_ {*} \right\rangle = - \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \nabla f (0, x _ {i}), \theta_ {*} \rangle \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \left\langle \nabla f (0, x _ {i}) - \nabla f \left(\theta_ {*}, x _ {i}\right), 0 - \theta_ {*} \right\rangle \\ \geq m \| \theta_ {*} \| ^ {2} - K, \tag {E.45} \\ \end{array}
$$
$$
m \| \theta_ {*} \| ^ {2} - K \leq \sup  _ {x \in \mathcal {X}} \| \nabla f (0, x) \| \cdot \| \theta_ {*} \|, \tag {E.46}
$$
$$
\left\| \theta_ {*} \right\| \leq \frac {\sup  _ {x \in \mathcal {X}} \| \nabla f (0 , x) \| + \sqrt {\sup  _ {x \in \mathcal {X}} \| \nabla f (0 , x) \| ^ {2} + 4 m K}}{2 m}. \tag {E.47}
$$
Similarly, one can show that
$$
\left\| \hat {\theta} _ {*} \right\| \leq \frac {\sup  _ {x \in \mathcal {X}} \| \nabla f (0 , x) \| + \sqrt {\sup  _ {x \in \mathcal {X}} \| \nabla f (0 , x) \| ^ {2} + 4 m K}}{2 m}. \tag {E.48}
$$
E.1 Proof of Theorem 3.3
Before going to the proof, let us restate the full version of Theorem 3.3 that we provide below.
Theorem E.1 (Complete Theorem 3.3). Let  $\theta_0 = \hat{\theta}_0 = \theta$ . Assume that Assumption 3.1, Assumption 3.3 and Assumption 3.4 hold. We also assume that  $\eta < \min \left\{\frac{1}{m}, \frac{m}{K_1^2 + 64D^2K_2^2}\right\}$  and  $\sup_{x \in \mathcal{X}} \| x \| \leq D$  for some  $D < \infty$  and  $\sup_{x \in \mathcal{X}} \| \nabla f(0,x) \| \leq E$  for some  $E < \infty$ . For any  $\hat{\eta} \in (0,1)$ . Define  $M > 0$  so that  $\int_{\| \theta_1 - \theta_* \| \leq M} p(\theta_*, \theta_1) d\theta_1 \geq \sqrt{\hat{\eta}}$  and any  $R > \frac{2K_0}{m}$  where  $K_0$  is defined in (E.2) so that
$$
\inf  _ {\theta , \theta_ {1} \in \mathbb {R} ^ {d}: V (\theta) \leq R, \| \theta_ {1} - \theta_ {*} \| \leq M} \frac {p \left(\theta , \theta_ {1}\right)}{p \left(\theta_ {*} , \theta_ {1}\right)} \geq \sqrt {\hat {\eta}}. \tag {E.49}
$$
$$
\begin{array}{l} \mathcal {W} _ {1} \left(\nu_ {k}, \hat {\nu} _ {k}\right) \\ \leq \frac {1 - \bar {\eta} ^ {k}}{2 \sqrt {\psi (1 + \psi)} (1 - \bar {\eta})} \\ \end{array}
$$
$$
\cdot \frac {2 b}{n} \max  \left\{\psi \left(4 + 8 \eta^ {2} K _ {1} ^ {2}\right), \right.
$$
$$
\begin{array}{l} 1 + \psi \left(1 + \eta^ {2} \sigma^ {2} + 1 6 (1 + 2 \eta^ {2} K _ {1} ^ {2}) \left(\frac {E + \sqrt {E ^ {2} + 4 m K}}{2 m}\right) ^ {2} \right. \\ \left. + 4 \eta^ {2} \left(2 E ^ {2} + 2 K _ {1} ^ {2} \left(\frac {E + \sqrt {E ^ {2} + 4 m K}}{2 m}\right) ^ {2}\right)\right) \Bigg \} \\ \cdot \max  \left\{1 + 2 \theta^ {2} + 2 \left(\frac {E + \sqrt {E ^ {2} + 4 m K}}{2 m}\right) ^ {2}, \right. \\ \left. 2 - \frac {\eta}{m} K _ {1} ^ {2} - \frac {5 6 \eta}{m} D ^ {2} K _ {2} ^ {2} + \frac {6 4 \eta}{m} D ^ {2} K _ {2} ^ {2} \left(\frac {E + \sqrt {E ^ {2} + 4 m K}}{2 m}\right) ^ {2} + \frac {2 K}{m} + \frac {\eta}{m} \sigma^ {2} \right\}, \tag {E.50} \\ \end{array}
$$
where for any  $\eta_0\in (0,\hat{\eta})$  and  $\gamma_0\in \left(1 - m\eta +\frac{2\eta K_0}{R},1\right)$  one can choose  $\psi = \frac{\eta_0}{\eta K_0}$  and  $\bar{\eta} = (1 - (\hat{\eta} -\eta_0))\vee \frac{2 + R\psi\gamma_0}{2 + R\psi}$  and  $d_{\psi}$  is the weighted total variation distance defined in Section B.1.
Proof. By applying Lemma E.1, Lemma E.3, Lemma E.4 and Lemma E.5, which is a modification of Lemma 2.1, we obtain
$$
\begin{array}{l} \mathcal {W} _ {1} \left(\nu_ {k}, \hat {\nu} _ {k}\right) \\ \leq \frac {1 - \bar {\eta} ^ {k}}{2 \sqrt {\psi (1 + \psi)} (1 - \bar {\eta})} \\ \cdot \frac {2 b}{n} \max  \left\{\psi (4 + 8 \eta^ {2} K _ {1} ^ {2}), \right. \\ \left. 1 + \psi \left(1 + \eta^ {2} \sigma^ {2} + (4 + 8 \eta^ {2} K _ {1} ^ {2}) \| \theta_ {*} - \hat {\theta} _ {*} \| ^ {2} + 4 \eta^ {2} \sup  _ {x \in \mathcal {X}} \| \nabla f (\theta_ {*}, x) \| ^ {2}\right) \right\} \\ \cdot \max  \left\{1 + \| \theta - \hat {\theta} _ {*} \| ^ {2}, 2 - \frac {\eta}{m} K _ {1} ^ {2} - \frac {5 6 \eta}{m} D ^ {2} K _ {2} ^ {2} + \frac {6 4 \eta}{m} D ^ {2} K _ {2} ^ {2} \| \hat {\theta} _ {*} \| ^ {2} + \frac {2 K}{m} + \frac {\eta}{m} \sigma^ {2} \right\}, \tag {E.51} \\ \end{array}
$$
Finally, let us notice that  $\| \theta_{*} - \hat{\theta}_{*}\|^{2}\leq 2\| \theta_{*}\|^{2} + 2\| \hat{\theta}_{*}\|^{2}$  and  $\| \theta -\hat{\theta}_*\| ^2\leq 2\| \theta \| ^2 +2\| \hat{\theta}_*\| ^2$  and for every  $x\in \mathcal{X}$ ,
$$
\left\| \nabla f \left(\theta_ {*}, x\right) \right\| ^ {2} \leq 2 \left\| \nabla f (0, x) \right\| ^ {2} + 2 \left\| \nabla f \left(\theta_ {*}, x\right) - \nabla f (0, x) \right\| ^ {2} \leq 2 \left\| \nabla f (0, x) \right\| ^ {2} + 2 K _ {1} ^ {2} \left\| \theta_ {*} \right\| ^ {2}. \tag {E.52}
$$
By applying Lemma E.6, we complete the proof.
E.2 Proof of Corollary 3.5
Proof. Under our assumptions, the noise  $\xi_{k}$  are i.i.d. Gaussian  $\mathcal{N}(0,\Sigma)$  so that  $\mathbb{E}[\| \xi_1\|^2 ] = \mathrm{tr}(\Sigma) = \sigma^2$ . Moreover, we have  $\Sigma \prec I_d$ . Then  $p(\theta_{*},\theta_{1})$  is the probability density function of
$$
\theta_ {*} - \frac {\eta}{b} \sum_ {i \in \Omega_ {1}} \nabla f \left(\theta_ {*}, x _ {i}\right) + \eta \xi_ {1}. \tag {E.53}
$$
$$
\int_ {\left\| \theta_ {1} - \theta_ {*} \right\| \leq M} p \left(\theta_ {*}, \theta_ {1}\right) d \theta_ {1} = \mathbb {P} \left(\left\| - \frac {\eta}{b} \sum_ {i \in \Omega_ {1}} \nabla f \left(\theta_ {*}, x _ {i}\right) + \eta \xi_ {1} \right\| \leq M\right). \tag {E.54}
$$
Notice that for any  $\Omega_1$
$$
\left\| \frac {\eta}{b} \sum_ {i \in \Omega_ {1}} \nabla f \left(\theta_ {*}, x _ {i}\right) \right\| \leq \eta \sup  _ {x \in \mathcal {X}} \| \nabla f \left(\theta_ {*}, x\right) \|. \tag {E.55}
$$
$$
\begin{array}{l} \int_ {\left\| \theta_ {1} - \theta_ {*} \right\| \leq M} p \left(\theta_ {*}, \theta_ {1}\right) d \theta_ {1} = 1 - \mathbb {P} \left(\left\| - \frac {\eta}{b} \sum_ {i \in \Omega_ {1}} \nabla f \left(\theta_ {*}, x _ {i}\right) + \eta \xi_ {1} \right\| > M\right) \\ \geq 1 - \mathbb {P} \left(\| \eta \xi_ {1} \| > M - \eta \sup  _ {x \in \mathcal {X}} \| \nabla f (\theta_ {*}, x) \|\right) \\ = 1 - \mathbb {P} \left(\| \xi_ {1} \| > \frac {M}{\eta} - \sup  _ {x \in \mathcal {X}} \| \nabla f (\theta_ {*}, x) \|\right). \tag {E.56} \\ \end{array}
$$
Since  $\xi \sim \mathcal{N}(0,\Sigma)$  and  $\Sigma \prec I_d$ , for any  $\gamma \leq \frac{1}{2}$ , we have
$$
\mathbb {E} \left[ e ^ {\gamma \| \xi_ {1} \| ^ {2}} \right] = \frac {1}{\sqrt {\det \left(I _ {d} - 2 \gamma \Sigma\right)}}.
$$
By Chebychev's inequality, letting  $\gamma = \frac{1}{2}$ , for any  $M \geq \eta \sup_{x \in \mathcal{X}} \| \nabla f(\theta_*, x) \|$ , we get
$$
\int_ {\left\| \theta_ {1} - \theta_ {*} \right\| \leq M} p \left(\theta_ {*}, \theta_ {1}\right) d \theta_ {1} \geq 1 - \frac {1}{\sqrt {\det  \left(I _ {d} - \Sigma\right)}} \exp \left(- \frac {1}{2} \left(\frac {M}{\eta} - \sup  _ {x \in \mathcal {X}} \| \nabla f \left(\theta_ {*}, x\right) \|\right) ^ {2}\right).
$$
Next, for any  $\theta, \theta_1 \in \mathbb{R}^d$  such that  $\| \theta_1 - \theta_* \| \leq M$  and  $\| \theta - \theta_* \|^2 \leq \frac{2K_0}{m} (1 + \epsilon) - 1$ , we have
$$
\frac {p \left(\theta , \theta_ {1}\right)}{p \left(\theta_ {*}, \theta_ {1}\right)} = \frac {\mathbb {E} _ {\Omega_ {1}} \left[ p _ {\Omega_ {1}} \left(\theta , \theta_ {1}\right) \right]}{\mathbb {E} _ {\Omega_ {1}} \left[ p _ {\Omega_ {1}} \left(\theta_ {*}, \theta_ {1}\right) \right]}, \tag {E.57}
$$
where  $\mathbb{E}_{\Omega_1}$  denotes the expectation w.r.t.  $\Omega_{1}$ , and  $p_{\Omega_1}$  denotes the probability density function conditional on  $\Omega_{1}$ . For any given  $\Omega_{1}$ , we can compute that
$$
\begin{array}{l} \frac {p _ {\Omega_ {1}} (\theta , \theta_ {1})}{p _ {\Omega_ {1}} (\theta_ {*} , \theta_ {1})} \\ = \exp \left\{- \frac {1}{2 \eta^ {2}} \left(\theta_ {1} - \mu (\theta)\right) ^ {\top} \Sigma^ {- 1} \left(\theta_ {1} - \mu (\theta)\right) + \frac {1}{2 \eta^ {2}} \left(\theta_ {1} - \mu \left(\theta_ {*}\right)\right) ^ {\top} \Sigma^ {- 1} \left(\theta_ {1} - \mu \left(\theta_ {*}\right)\right) \right\}, \tag {E.58} \\ \end{array}
$$
$$
\mu (\theta) := \theta - \frac {\eta}{b} \sum_ {i \in \Omega_ {1}} \nabla f \left(\theta , x _ {i}\right), \quad \mu \left(\theta_ {*}\right) := \theta_ {*} - \frac {\eta}{b} \sum_ {i \in \Omega_ {1}} \nabla f \left(\theta_ {*}, x _ {i}\right). \tag {E.59}
$$
Therefore, for any  $\theta, \theta_1 \in \mathbb{R}^d$  such that  $\| \theta_1 - \theta_* \| \leq M$  and  $\| \theta - \theta_* \|^2 \leq \frac{2K_0}{m} (1 + \epsilon) - 1$ , we have
$$
\begin{array}{l} \frac {p _ {\Omega_ {1}} (\theta , \theta_ {1})}{p _ {\Omega_ {1}} (\theta_ {*}, \theta_ {1})} \geq \exp \left\{- \frac {1}{2 \eta^ {2}} \| \mu (\theta) - \mu (\theta_ {*}) \| \cdot \| \Sigma^ {- 1} \| \left(\| \theta_ {1} - \mu (\theta) \| + \| \theta_ {1} - \mu (\theta_ {*}) \|\right) \right\} \\ \geq \exp \left\{- \frac {1}{2 \eta^ {2}} \| \mu (\theta) - \mu \left(\theta_ {*}\right) \| \cdot \| \Sigma^ {- 1} \| \left(\| \mu (\theta) - \mu \left(\theta_ {*}\right) \| + 2 \| \theta_ {1} - \mu \left(\theta_ {*}\right) \|\right) \right\}. \tag {E.60} \\ \end{array}
$$
We can further compute that
$$
\left\| \theta_ {1} - \mu \left(\theta_ {*}\right) \right\| \leq \left\| \theta_ {1} - \theta_ {*} \right\| + \left\| \frac {\eta}{b} \sum_ {i \in \Omega_ {1}} \nabla f \left(\theta_ {*}, x _ {i}\right) \right\| \leq M + \eta \sup  _ {x \in \mathcal {X}} \| \nabla f \left(\theta_ {*}, x\right) \|, \tag {E.61}
$$
$$
\begin{array}{l} \left\| \mu (\theta) - \mu \left(\theta_ {*}\right) \right\| \leq \| \theta - \theta_ {*} \| + \frac {\eta}{b} \sum_ {i \in \Omega_ {1}} \| \nabla f (\theta , x _ {i}) - \nabla f \left(\theta_ {*}, x _ {i}\right) \| \\ \leq \left(1 + K _ {1} \eta\right) \| \theta - \theta_ {*} \| \\ \leq \left(1 + K _ {1} \eta\right) \left(\frac {2 K _ {0}}{m} (1 + \epsilon) - 1\right) ^ {1 / 2}. \tag {E.62} \\ \end{array}
$$
$$
\begin{array}{l} \frac {p _ {\Omega_ {1}} (\theta , \theta_ {1})}{p _ {\Omega_ {1}} (\theta_ {*} , \theta_ {1})} \geq \exp \left\{- \frac {(1 + K _ {1} \eta) \left(\frac {2 K _ {0}}{m} (1 + \epsilon) - 1\right) ^ {1 / 2}}{2 \eta^ {2}} \| \Sigma^ {- 1} \| \right. \\ \left. \cdot \left((1 + K _ {1} \eta) \left(\frac {2 K _ {0}}{m} (1 + \epsilon) - 1\right) ^ {1 / 2} + 2 \left(M + \eta \sup  _ {x \in \mathcal {X}} \| \nabla f \left(\theta_ {*}, x\right) \|\right)\right) \right\}. \tag {E.63} \\ \end{array}
$$
Since it holds for every  $\Omega_1$ , we have
$$
\begin{array}{l} \frac {p \left(\theta , \theta_ {1}\right)}{p \left(\theta_ {* ,} \theta_ {1}\right)} \geq \exp \left\{- \frac {\left(1 + K _ {1} \eta\right) \left(\frac {2 K _ {0}}{m} (1 + \epsilon) - 1\right) ^ {1 / 2}}{2 \eta^ {2}} \| \Sigma^ {- 1} \| \right. \\ \left. \cdot \left((1 + K _ {1} \eta) \left(\frac {2 K _ {0}}{m} (1 + \epsilon) - 1\right) ^ {1 / 2} + 2 \left(M + \eta \sup  _ {x \in \mathcal {X}} \| \nabla f \left(\theta_ {*}, x\right) \|\right)\right) \right\}. \tag {E.64} \\ \end{array}
$$
$$
\begin{array}{l} \hat {\eta} \geq \left(\max  _ {M \geq \eta \sup  _ {x \in \mathcal {X}} \| \nabla f (\theta_ {*}, x) \|} \left\{\min  \left\{1 - \frac {\exp \left(- \frac {1}{2} \left(\frac {M}{\eta} - \sup  _ {x \in \mathcal {X}} \| \nabla f (\theta_ {*} , x) \|\right) ^ {2}\right)}{\sqrt {\det  (I _ {d} - \Sigma)}} \right., \right. \right. \\ \exp \left\{- \frac {(1 + K _ {1} \eta) \left(\frac {2 K _ {0}}{m} (1 + \epsilon) - 1\right) ^ {1 / 2}}{2 \eta^ {2}} \| \Sigma^ {- 1} \| \right. \\ \left. \cdot \left(\left(1 + K _ {1} \eta\right) \left(\frac {2 K _ {0}}{m} (1 + \epsilon) - 1\right) ^ {1 / 2} + 2 \left(M + \eta \sup  _ {x \in \mathcal {X}} \| \nabla f \left(\theta_ {*}, x\right) \|\right)\right) \Bigg \} \Bigg \} \Bigg \}\right) ^ {2}. \tag {E.65} \\ \end{array}
$$
F Proofs of Non-Convex Case without Additive Noise
F.1 Proof of Theorem 4.1
Proof. Let us recall that  $\theta_0 = \hat{\theta}_0 = \theta$  and for any  $k\in \mathbb{N}$
$$
\theta_ {k} = \theta_ {k - 1} - \frac {\eta}{b} \sum_ {i \in \Omega_ {k}} \nabla f \left(\theta_ {k - 1}, x _ {i}\right), \tag {F.1}
$$
$$
\hat {\theta} _ {k} = \hat {\theta} _ {k - 1} - \frac {\eta}{b} \sum_ {i \in \Omega_ {k}} \nabla f \left(\hat {\theta} _ {k - 1}, \hat {x} _ {i}\right). \tag {F.2}
$$
Thus it follows that
$$
\theta_ {k} - \hat {\theta} _ {k} = \theta_ {k - 1} - \hat {\theta} _ {k - 1} - \frac {\eta}{b} \sum_ {i \in \Omega_ {k}} \left(\nabla f \left(\theta_ {k - 1}, x _ {i}\right) - \nabla f \left(\hat {\theta} _ {k - 1}, x _ {i}\right)\right) + \frac {\eta}{b} \mathcal {E} _ {k}, \tag {F.3}
$$
$$
\mathcal {E} _ {k} := \sum_ {i \in \Omega_ {k}} \left(\nabla f \left(\hat {\theta} _ {k - 1}, \hat {x} _ {i}\right) - \nabla f \left(\hat {\theta} _ {k - 1}, x _ {i}\right)\right). \tag {F.4}
$$
$$
\begin{array}{l} \left\| \theta_ {k} - \hat {\theta} _ {k} \right\| ^ {2} = \left\| \theta_ {k - 1} - \hat {\theta} _ {k - 1} - \frac {\eta}{b} \sum_ {i \in \Omega_ {k}} \left(\nabla f \left(\theta_ {k - 1}, x _ {i}\right) - \nabla f \left(\hat {\theta} _ {k - 1}, x _ {i}\right)\right) \right\| ^ {2} + \frac {\eta^ {2}}{b ^ {2}} \| \mathcal {E} _ {k} \| ^ {2} \\ + 2 \left\langle \theta_ {k - 1} - \hat {\theta} _ {k - 1} - \frac {\eta}{b} \sum_ {i \in \Omega_ {k}} \left(\nabla f \left(\theta_ {k - 1}, x _ {i}\right) - \nabla f \left(\hat {\theta} _ {k - 1}, x _ {i}\right)\right), \frac {\eta}{b} \mathcal {E} _ {k} \right\rangle . \tag {F.5} \\ \end{array}
$$
By Assumption 3.1 and Assumption 3.3, we have
$$
\begin{array}{l} \left\| \theta_ {k - 1} - \hat {\theta} _ {k - 1} - \frac {\eta}{b} \sum_ {i \in \Omega_ {k}} \left(\nabla f \left(\theta_ {k - 1}, x _ {i}\right) - \nabla f \left(\hat {\theta} _ {k - 1}, x _ {i}\right)\right) \right\| ^ {2} \\ \leq \left(1 - 2 \eta m\right) \left\| \theta_ {k - 1} - \hat {\theta} _ {k - 1} \right\| ^ {2} + 2 \eta K + \frac {\eta^ {2}}{b ^ {2}} \left(b K _ {1} \left\| \theta_ {k - 1} - \hat {\theta} _ {k - 1} \right\|\right) ^ {2} \\ \leq (1 - \eta m) \left\| \theta_ {k - 1} - \hat {\theta} _ {k - 1} \right\| ^ {2} + 2 \eta K, \tag {F.6} \\ \end{array}
$$
provided that  $\eta \leq \frac{m}{K_1^2}$ .
Since  $X_{n}$  and  $\hat{X}_{n}$  differ by at most one element and  $\sup_{x\in \mathcal{X}}\| x\| \leq D$  for some  $D < \infty$ , we have  $x_{i} = \hat{x}_{i}$  for any  $i\in \Omega_k$  with probability  $\frac{n - b}{n}$  and  $x_{i}\neq \hat{x}_{i}$  for exactly one  $i\in \Omega_k$  with probability  $\frac{b}{n}$  and therefore
$$
\mathbb {E} \left\| \mathcal {E} _ {k} \right\| ^ {2} \leq \frac {b}{n} \mathbb {E} \left[ \left(K _ {2} 2 D \left(2 \| \hat {\theta} _ {k - 1} \| + 1\right)\right) ^ {2} \right] \leq \frac {4 D ^ {2} K _ {2} ^ {2} b}{n} \left(8 \mathbb {E} \| \hat {\theta} _ {k - 1} \| ^ {2} + 2\right), \tag {F.7}
$$
and moreover,
$$
\begin{array}{l} \mathbb {E} \left\langle \theta_ {k - 1} - \hat {\theta} _ {k - 1} - \frac {\eta}{b} \sum_ {i \in \Omega_ {k}} \left(\nabla f \left(\theta_ {k - 1}, x _ {i}\right) - \nabla f \left(\hat {\theta} _ {k - 1}, x _ {i}\right)\right), \frac {\eta}{b} \mathcal {E} _ {k} \right\rangle \\ \leq \frac {\eta}{b} \frac {b}{n} \mathbb {E} \left[ \left(1 + K _ {1} \eta\right) \left\| \theta_ {k - 1} - \hat {\theta} _ {k - 1} \right\| K _ {2} 2 D \left(2 \| \hat {\theta} _ {k - 1} \| + 1\right) \right] \\ \leq \frac {2 K _ {2} D \eta}{n} (1 + K _ {1} \eta) \mathbb {E} \left[ \left(\| \theta_ {k - 1} \| + \| \hat {\theta} _ {k - 1} \|\right) \left(2 \| \hat {\theta} _ {k - 1} \| + 1\right) \right] \\ \leq \frac {2 K _ {2} D \eta}{n} \left(1 + K _ {1} \eta\right) \left(1 + \frac {3}{2} \mathbb {E} \| \theta_ {k - 1} \| ^ {2} + \frac {7}{2} \mathbb {E} \| \hat {\theta} _ {k - 1} \| ^ {2}\right), \tag {F.8} \\ \end{array}
$$
where we used the inequality that
$$
(a + b) (2 b + 1) = 2 b ^ {2} + 2 a b + a + b \leq 1 + \frac {3}{2} a ^ {2} + \frac {7}{2} b ^ {2}, \tag {F.9}
$$
for any  $a, b \in \mathbb{R}$ . Therefore, we have
$$
\begin{array}{l} \mathbb {E} \left\| \theta_ {k} - \hat {\theta} _ {k} \right\| ^ {2} \leq (1 - \eta m) \mathbb {E} \left\| \theta_ {k - 1} - \hat {\theta} _ {k - 1} \right\| ^ {2} + 2 \eta K + \frac {4 D ^ {2} K _ {2} ^ {2} \eta^ {2}}{b n} \left(8 \mathbb {E} \| \hat {\theta} _ {k - 1} \| ^ {2} + 2\right) \\ + \frac {4 K _ {2} D \eta}{n} \left(1 + K _ {1} \eta\right) \left(1 + \frac {3}{2} \mathbb {E} \| \theta_ {k - 1} \| ^ {2} + \frac {7}{2} \mathbb {E} \| \hat {\theta} _ {k - 1} \| ^ {2}\right). \tag {F.10} \\ \end{array}
$$
In Lemma E.3, we showed that under the assumption  $\eta < \min \left\{\frac{1}{m}, \frac{m}{K_1^2 + 64D^2K_2^2}\right\}$  and  $\sup_{x \in \mathcal{X}} \| x \| \leq D$  for some  $D < \infty$ , we have that for every  $k \in \mathbb{N}$ ,
$$
\mathbb {E} \hat {V} (\hat {\theta} _ {k}) \leq (1 - \eta m) \mathbb {E} \hat {V} (\hat {\theta} _ {k - 1}) + 2 \eta m - \eta^ {2} K _ {1} ^ {2} - 5 6 \eta^ {2} D ^ {2} K _ {2} ^ {2} + 6 4 \eta^ {2} D ^ {2} K _ {2} ^ {2} \| \hat {\theta} _ {*} \| ^ {2} + 2 \eta K, \tag {F.11}
$$
where  $\hat{V} (\theta)\coloneqq 1 + \| \theta -\hat{\theta}_{*}\|^{2}$ , where  $\hat{\theta}_{*}$  is the minimizer of  $\hat{F} (\theta ,\hat{X}_n)\coloneqq \frac{1}{n}\sum_{i = 1}^{n}\nabla f(\theta ,\hat{x}_i)$ . This implies that
$$
\begin{array}{l} \mathbb {E} \left[ \hat {V} \left(\hat {\theta} _ {k}\right) \right] \\ \leq (1 - \eta m) ^ {k} \mathbb {E} \left[ \hat {V} (\hat {\theta} _ {0}) \right] + \frac {2 \eta m - \eta^ {2} K _ {1} ^ {2} - 5 6 \eta^ {2} D ^ {2} K _ {2} ^ {2} + 6 4 \eta^ {2} D ^ {2} K _ {2} ^ {2} \| \hat {\theta} _ {*} \| ^ {2} + 2 \eta K}{1 - (1 - \eta m)} \\ \leq 1 + \| \theta - \hat {\theta} _ {*} \| ^ {2} + 2 - \frac {\eta}{m} K _ {1} ^ {2} - \frac {5 6 \eta}{m} D ^ {2} K _ {2} ^ {2} + \frac {6 4 \eta}{m} D ^ {2} K _ {2} ^ {2} \| \hat {\theta} _ {*} \| ^ {2} + \frac {2 K}{m}, \tag {F.12} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \| \hat {\theta} _ {k} \| ^ {2} \leq 2 \mathbb {E} \left\| \hat {\theta} _ {k} - \hat {\theta} _ {*} \right\| ^ {2} + 2 \| \hat {\theta} _ {*} \| ^ {2} \\ \leq 2 \left\| \theta - \hat {\theta} _ {*} \right\| ^ {2} + 4 - \frac {2 \eta}{m} K _ {1} ^ {2} - \frac {1 1 2 \eta}{m} D ^ {2} K _ {2} ^ {2} + \frac {1 2 8 \eta}{m} D ^ {2} K _ {2} ^ {2} \| \hat {\theta} _ {*} \| ^ {2} + \frac {4 K}{m} + 2 \| \hat {\theta} _ {*} \| ^ {2} \\ \leq 4 \| \theta \| ^ {2} + 4 \left(\frac {E + \sqrt {E ^ {2} + 4 m K}}{2 m}\right) ^ {2} + 4 - \frac {2 \eta}{m} K _ {1} ^ {2} - \frac {1 1 2 \eta}{m} D ^ {2} K _ {2} ^ {2} \\ + \frac {1 2 8 \eta}{m} D ^ {2} K _ {2} ^ {2} \left(\frac {E + \sqrt {E ^ {2} + 4 m K}}{2 m}\right) ^ {2} + \frac {4 K}{m} + 2 \left(\frac {E + \sqrt {E ^ {2} + 4 m K}}{2 m}\right) ^ {2} =: B, \tag {F.13} \\ \end{array}
$$
where we applied Lemma E.6. Similarly, we can show that
$$
\mathbb {E} \left\| \theta_ {k} \right\| ^ {2} \leq B. \tag {F.14}
$$
Since  $\theta_0 = \hat{\theta}_0 = \theta$ , it follows from (F.10), (F.13) and (F.14) that
$$
\begin{array}{l} \mathcal {W} _ {2} ^ {2} \left(\nu_ {k}, \hat {\nu} _ {k}\right) \\ \leq \mathbb {E} \left\| \theta_ {k} - \hat {\theta} _ {k} \right\| ^ {2} \\ \leq \left(1 - (1 - \eta m) ^ {k}\right) \left(\frac {4 D ^ {2} K _ {2} ^ {2} \eta}{b n m} (8 B + 2) + \frac {4 K _ {2} D}{n m} (1 + K _ {1} \eta) \left(1 + \frac {3}{2} B + \frac {7}{2} B\right) + \frac {2 K}{m}\right), \tag {F.15} \\ \end{array}
$$
provided that  $\eta < \min \left\{\frac{m}{K_1^2}, \frac{1}{m}, \frac{m}{K_1^2 + 64D^2K_2^2}\right\}$ . This completes the proof.
G Proofs of Convex Case with Additional Geometric Structure
In the following technical lemma, we show that the  $p$ -th moment of  $\theta_{k}$  and  $\hat{\theta}_k$  can be bounded in the following sense.
Lemma G.1. Let  $\theta_0 = \hat{\theta}_0 = \theta$ . Suppose Assumption 4.1 and Assumption 4.2 hold and  $\eta \leq \frac{\mu}{K_1^2 + 2^{p + 4}D^2K_2^2}$ . Then, we have
$$
\frac {1}{k} \sum_ {i = 1} ^ {k} \mathbb {E} \| \theta_ {i - 1} - \theta_ {*} \| ^ {p} \leq \frac {\| \theta - \theta_ {*} \| ^ {2}}{k \eta \mu} + \frac {8 \eta}{\mu} D ^ {2} K _ {2} ^ {2} (2 ^ {p + 1} \| \theta_ {*} \| ^ {p} + 5), \tag {G.1}
$$
$$
\frac {1}{k} \sum_ {i = 1} ^ {k} \mathbb {E} \left\| \hat {\theta} _ {i - 1} - \hat {\theta} _ {*} \right\| ^ {p} \leq \frac {\left\| \theta - \hat {\theta} _ {*} \right\| ^ {2}}{k \eta \mu} + \frac {8 \eta}{\mu} D ^ {2} K _ {2} ^ {2} \left(2 ^ {p + 1} \| \hat {\theta} _ {*} \| ^ {p} + 5\right), \tag {G.2}
$$