SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \stackrel {(c _ {1})} {=} 2 \alpha_ {k} ^ {2} \mathbb {E} \left[ \left\| h _ {0} \left(X _ {k}, \boldsymbol {\theta} _ {k}, \lambda_ {k}\right) - h _ {0} \left(X _ {k}, \boldsymbol {\theta} _ {k}, y _ {0} (\boldsymbol {\theta} _ {k})\right) + h _ {0} \left(X _ {k}, \boldsymbol {\theta} _ {k}, y _ {0} (\boldsymbol {\theta} _ {k})\right) \right. \right. \\ \left. - h _ {0} \left(X _ {k}, \boldsymbol {\theta} _ {0} ^ {*}, y _ {0} \left(\boldsymbol {\theta} _ {0} ^ {*}\right)\right) + h _ {0} \left(X _ {k}, \boldsymbol {\theta} _ {0} ^ {*}, y _ {0} \left(\boldsymbol {\theta} _ {0} ^ {*}\right)\right) - H _ {0} \left(\boldsymbol {\theta} _ {0} ^ {*}, y _ {0} \left(\boldsymbol {\theta} _ {0} ^ {*}\right)\right) \right\rVert^ {2} \mid \mathcal {F} _ {k - \tau} \Bigg ] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(c _ {2})} {\leq} 6 \alpha_ {k} ^ {2} \mathbb {E} \left[ \left\| h _ {0} (X _ {k}, \boldsymbol {\theta} _ {k}, \lambda_ {k}) - h _ {0} (X _ {k}, \boldsymbol {\theta} _ {k}, y _ {0} (\boldsymbol {\theta} _ {k})) \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] \\ + 6 \alpha_ {k} ^ {2} \mathbb {E} \left[ \left\| h _ {0} \left(X _ {k}, \boldsymbol {\theta} _ {k}, y _ {0} (\boldsymbol {\theta} _ {k})\right) - h _ {0} \left(X _ {k}, \boldsymbol {\theta} _ {0} ^ {*}, y _ {0} \left(\boldsymbol {\theta} _ {0} ^ {*}\right)\right) \right\| ^ {2} \mid \mathcal {F} _ {k - \tau} \right] \\ + 6 \alpha_ {k} ^ {2} \mathbb {E} \left[ \left\| h _ {0} \left(X _ {k}, \boldsymbol {\theta} _ {0} ^ {*}, y _ {0} (\boldsymbol {\theta} _ {0} ^ {*})\right) - H _ {0} \left(\boldsymbol {\theta} _ {0} ^ {*}, y _ {0} (\boldsymbol {\theta} _ {0} ^ {*})\right) \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] \\ \end{array}
$$
$$
\stackrel {(c _ {3})} {\leq} 6 \alpha_ {k} ^ {2} \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] + 1 5 0 \alpha_ {k} ^ {2} \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right], \tag {37}
$$
where  $(c_{1})$  holds due to  $H_0(\pmb{\theta}_0, y_0(\pmb{\theta}_0^*)) = 0$ ,  $(c_{2})$  follows from the triangular inequality, and  $(c_{3})$  follows from the Lipschitz continuity of  $h_0(X, \pmb{\theta}, \lambda)$  in Lemma 1.
$\mathrm{Term}_2$  is bounded as
$$
\begin{array}{l} \operatorname {T e r m} _ {2} = 2 \alpha_ {k} \mathbb {E} \left[ \hat {\boldsymbol {\theta}} _ {k} ^ {\mathsf {T}} \left(h \left(X _ {k}, \boldsymbol {\theta} _ {k}, \lambda_ {k}\right) - h _ {0} \left(X _ {k}, \boldsymbol {\theta} _ {k}, \lambda_ {k}\right)\right) \mid \mathcal {F} _ {k - \tau} \right] \\ \stackrel {(c _ {4})} {\leq} \frac {\eta_ {k}}{\alpha_ {k}} \mathbb {E} \Big [ \left\| \hat {\pmb {\theta}} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \Big ] + \frac {\alpha_ {k} ^ {3}}{\eta_ {k}} \mathbb {E} \Big [ \left\| h (X _ {k}, \pmb {\theta} _ {k}, \lambda_ {k}) - h _ {0} (X _ {k}, \pmb {\theta} _ {k}, \lambda_ {k}) \right\| ^ {2} | \mathcal {F} _ {k - \tau} \Big ] \\ \stackrel {(c _ {5})} {\leq} \frac {\eta_ {k}}{\alpha_ {k}} \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {k} \right\| ^ {2} \mid \mathcal {F} _ {k - \tau} \right] + \frac {\alpha_ {k} ^ {2}}{\eta_ {k}} \mathcal {O} \left(c _ {1} ^ {3} \left(\left\| \boldsymbol {\theta} _ {0} \right\| + \left| \lambda_ {0} \right| + 1\right) ^ {3} \cdot m ^ {- 1 / 2}\right), \tag {38} \\ \end{array}
$$
where  $(c_4)$  holds due to the fact that  $2\pmb{x}^{\top}\pmb{y} \leq \| \pmb{x} \|^{2} + \| \pmb{y} \|^{2}$  and  $(c_5)$  is due to Lemma 10.
$\mathrm{Term}_3$  is bounded as
$$
\begin{array}{l} \mathbf {T e r m} _ {3} = 2 \alpha_ {k} ^ {2} \mathbb {E} \left[ \left\| h (X _ {k}, \boldsymbol {\theta} _ {k}, \lambda_ {k}) - h _ {0} (X _ {k}, \boldsymbol {\theta} _ {k}, \lambda_ {k}) \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] \\ \stackrel {(c _ {6})} {\leq} 2 \alpha_ {k} ^ {2} \mathcal {O} \left(c _ {1} ^ {3} \left(\| \boldsymbol {\theta} _ {0} \| + | \lambda_ {0} | + 1\right) ^ {3} \cdot m ^ {- 1 / 2}\right), \tag {39} \\ \end{array}
$$
where  $(c_{6})$  comes from Lemma 10. Substituting  $\mathrm{Term}_1$ ,  $\mathrm{Term}_2$ , and  $\mathrm{Term}_3$  back into (36) leads to the desired result in (34), which is
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {k + 1} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] \leq \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] - 2 \alpha_ {k} \mu_ {1} \mathbb {E} \left[ \left\| \tilde {\boldsymbol {\theta}} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] \\ + 6 \alpha_ {k} ^ {2} \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} \mid \mathcal {F} _ {k - \tau} \right] + 1 5 0 \alpha_ {k} ^ {2} \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {k} \right\| ^ {2} \mid \mathcal {F} _ {k - \tau} \right] \\ + \frac {\eta_ {k}}{\alpha_ {k}} \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {k} \right\| ^ {2} \mid \mathcal {F} _ {k - \tau} \right] + \frac {\alpha_ {k} ^ {3}}{\eta_ {k}} \mathcal {O} \left(c _ {1} ^ {3} \left(\left\| \boldsymbol {\theta} _ {0} \right\| + \left| \lambda_ {0} \right| + 1\right) ^ {3} \cdot m ^ {- 1 / 2}\right) \\ + 2 \alpha_ {k} ^ {2} \mathcal {O} \left(c _ {1} ^ {3} \left(\| \boldsymbol {\theta} _ {0} \| + | \lambda_ {0} | + 1\right) ^ {3} \cdot m ^ {- 1 / 2}\right) \\ = (1 + 1 5 0 \alpha_ {k} ^ {2} + \eta_ {k} / \alpha_ {k} - 2 \alpha_ {k} \mu_ {1}) \mathbb {E} \Big [ \left\| \hat {\boldsymbol {\theta}} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \Big ] + 6 \alpha_ {k} ^ {2} \mathbb {E} \Big [ \left\| \hat {\lambda} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \Big ] \\ + \left(\alpha_ {k} ^ {3} / \eta_ {k} + 2 \alpha_ {k} ^ {2}\right) \mathcal {O} \left(c _ {1} ^ {3} \left(\| \boldsymbol {\theta} _ {0} \| + | \lambda_ {0} | + 1\right) ^ {3} \cdot m ^ {- 1 / 2}\right). \\ \end{array}
$$
By neglecting higher order infinitesimal, we have the inequality in (34). This completes the proof.
Lemma 8. Let  $\{\pmb {\theta}_k,\lambda_k\}$  be generated by (9). Then under Lemmas 1-4, for any  $k\geq \tau$  , we have
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \hat {\lambda} _ {k + 1} \right\| ^ {2} \mid \mathcal {F} _ {k - \tau} \right] \leq (1 - 2 \eta_ {k} \mu_ {2} + \alpha_ {k} \eta_ {k} + 2 4 \alpha_ {k} ^ {2} + \frac {\eta_ {k}}{\alpha_ {k}} - \frac {2 \eta_ {k} ^ {2} \mu_ {k}}{\alpha_ {k}} + \eta_ {k} ^ {2} + \frac {2 4 \alpha_ {k} ^ {3}}{\eta_ {k}}) \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} \mid \mathcal {F} _ {k - \tau} \right] \\ + \left(6 0 0 \alpha_ {k} ^ {2} + 8 \eta_ {k} ^ {2} + \frac {8 \eta_ {k} ^ {3}}{\alpha_ {k}} + \frac {6 0 0 \alpha_ {k} ^ {3}}{\eta_ {k}}\right) \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] \\ + \frac {\eta_ {k}}{\alpha_ {k}} \mathcal {O} \left(c _ {1} ^ {3} \left(\| \boldsymbol {\theta} _ {0} \| + | \lambda_ {0} | + 1\right) ^ {3} \cdot m ^ {- 1 / 2}\right). \tag {40} \\ \end{array}
$$
Proof. According to the definition in (14), we have
$$
\begin{array}{l} \hat {\lambda} _ {k + 1} = \lambda_ {k + 1} - y _ {0} (\boldsymbol {\theta} _ {k + 1}) \\ = \hat {\lambda} _ {k} + \eta_ {k} g (\boldsymbol {\theta} _ {k}) + y _ {0} (\boldsymbol {\theta} _ {k}) - y _ {0} (\boldsymbol {\theta} _ {k + 1}), \\ \end{array}
$$
$$
\begin{array}{l} \left\| \hat {\lambda} _ {k + 1} \right\| ^ {2} = \left\| \hat {\lambda} _ {k} + \eta_ {k} g (\boldsymbol {\theta} _ {k}) + y _ {0} (\boldsymbol {\theta} _ {k}) - y _ {0} (\boldsymbol {\theta} _ {k + 1}) \right\| ^ {2} \\ = \underbrace {\left\| \hat {\lambda} _ {k} + \eta_ {k} g (\boldsymbol {\theta} _ {k}) \right\| ^ {2}} _ {\text {T e r m} _ {1}} + \underbrace {\left\| y _ {0} (\boldsymbol {\theta} _ {k}) - y _ {0} (\boldsymbol {\theta} _ {k + 1}) \right\| ^ {2}} _ {\text {T e r m} _ {2}} \\ + \underbrace {2 \left(\hat {\lambda} _ {k} + \eta_ {k} g (\boldsymbol {\theta} _ {k})\right) \left(y _ {0} (\boldsymbol {\theta} _ {k}) - y _ {0} (\boldsymbol {\theta} _ {k + 1})\right)} _ {\text {T e r m} _ {3}}. \tag {41} \\ \end{array}
$$
The second equality is due to the fact that  $\| \pmb{x} + \pmb{y}\|^2 = \| \pmb{x}\|^2 + \| \pmb{y}\|^2 + 2\pmb{x}^\top \pmb{y}$ . We next analyze the conditional expectation of each term in  $\left\|\hat{\lambda}_{k+1}\right\|^2$  on  $\mathcal{F}_{k-\tau}$ . We first focus on  $\mathrm{Term}_1$ .
$$
\mathbb {E} \left[ \operatorname {T e r m} _ {1} \mid \mathcal {F} _ {k - \tau} \right]
$$
$$
\begin{array}{l} = \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} + 2 \eta_ {k} \hat {\lambda} _ {k} g (\boldsymbol {\theta} _ {k}) + \left\| \eta_ {k} g (\boldsymbol {\theta} _ {k}) \right\| ^ {2} \mid \mathcal {F} _ {k - \tau} \right] \\ = \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} + 2 \eta_ {k} \hat {\lambda} _ {k} g _ {0} (\boldsymbol {\theta} _ {k}) + 2 \eta_ {k} \hat {\lambda} _ {k} (g (\boldsymbol {\theta} _ {k}) - g _ {0} (\boldsymbol {\theta} _ {k})) + \eta_ {k} ^ {2} \left\| g (\boldsymbol {\theta} _ {k}) - g _ {0} (\boldsymbol {\theta} _ {k}) + g _ {0} (\boldsymbol {\theta} _ {k}) \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] \\ \end{array}
$$
$$
\begin{array}{l} \leq \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] + 2 \eta_ {k} \mathbb {E} \left[ \hat {\lambda} _ {k} g _ {0} (\boldsymbol {\theta} _ {k}) | \mathcal {F} _ {k - \tau} \right] + 2 \eta_ {k} \mathbb {E} \left[ \hat {\lambda} _ {k} (g (\boldsymbol {\theta} _ {k}) - g _ {0} (\boldsymbol {\theta} _ {k})) | \mathcal {F} _ {k - \tau} \right] \\ + 2 \eta_ {k} ^ {2} \mathbb {E} \left[ \left\| g (\boldsymbol {\theta} _ {k}) - g _ {0} (\boldsymbol {\theta} _ {k}) \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] + 2 \eta_ {k} ^ {2} \mathbb {E} \left[ \left\| g _ {0} (\boldsymbol {\theta} _ {k}) \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(d _ {1})} {=} \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] + 2 \eta_ {k} \mathbb {E} \left[ \hat {\lambda} _ {k} g _ {0} (\boldsymbol {\theta} _ {k}) | \mathcal {F} _ {k - \tau} \right] + 2 \eta_ {k} \mathbb {E} \left[ \hat {\lambda} _ {k} (g (\boldsymbol {\theta} _ {k}) - g _ {0} (\boldsymbol {\theta} _ {k})) | \mathcal {F} _ {k - \tau} \right] \\ + 2 \eta_ {k} ^ {2} \mathbb {E} \left[ \left\| g (\boldsymbol {\theta} _ {k}) - g _ {0} (\boldsymbol {\theta} _ {k}) \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] + 2 \eta_ {k} ^ {2} \mathbb {E} \left[ \left\| g _ {0} (\boldsymbol {\theta} _ {k}) - g _ {0} (\boldsymbol {\theta} _ {0} ^ {*}) \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(d _ {2})} {=} \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] - 2 \eta_ {k} \mu_ {2} \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] + 8 \eta_ {k} ^ {2} \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] \\ + 2 \eta_ {k} \mathbb {E} \left[ \hat {\lambda} _ {k} \left(g \left(\boldsymbol {\theta} _ {k}\right) - g _ {0} \left(\boldsymbol {\theta} _ {k}\right)\right) \mid \mathcal {F} _ {k - \tau} \right] + 2 \eta_ {k} ^ {2} \mathbb {E} \left[ \left\| g \left(\boldsymbol {\theta} _ {k}\right) - g _ {0} \left(\boldsymbol {\theta} _ {k}\right) \right\| ^ {2} \mid \mathcal {F} _ {k - \tau} \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(d _ {3})} {\leq} \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] - 2 \eta_ {k} \mu_ {2} \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] + 8 \eta_ {k} ^ {2} \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] \\ + \alpha_ {k} \eta_ {k} \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] + (4 \eta_ {k} / \alpha_ {k} + 8 \eta_ {k} ^ {2}) \mathcal {O} \left(c _ {1} ^ {3} (\| \boldsymbol {\theta} _ {0} \| + | \lambda_ {0} | + 1) ^ {3} \cdot m ^ {- 1 / 2}\right), \\ \end{array}
$$
where  $(d_{1})$  follows from  $g_{0}(\pmb{\theta}_{0}^{*}) = 0$ ,  $(d_{2})$  holds due to Lemma 4 and the Lipschitz continuity of  $y_{0}$  in Lemma 3, and  $(d_{3})$  comes from Lemma 10. For  $\mathrm{Term}_{2}$ , we have
$$
\mathbb {E} \left[ \operatorname {T e r m} _ {2} | \mathcal {F} _ {k - \tau} \right] = \mathbb {E} \left[ \left\| y _ {0} (\boldsymbol {\theta} _ {k}) - y _ {0} (\boldsymbol {\theta} _ {k + 1}) \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right]
$$
where  $(d_4)$  is due to the fact that  $H_{0}(\pmb{\theta}_{0}^{*},y_{0}(\pmb{\theta}_{0}^{*})) = 0$ ,  $(d_5)$  holds according to  $\| \pmb{x} + \pmb{y} + \pmb{z}\|^2 \leq 3\|\pmb{x}\|^2 + 3\|\pmb{y}\|^2 + 3\|\pmb{z}\|^2$  since  $g(X_k, f(\pmb{\lambda}^*), \pmb{\lambda}^*) = \mathbf{0}$ ,  $(d_6)$  holds because of the Lipschitz continuity of  $h_0$  and  $y_0$  in Lemma 1 and Lemma 3, and  $(d_7)$  comes from Lemma 10. Next, we have the conditional expectation of  $\mathrm{Term}_3$  as
$$
\begin{array}{l} \mathbb {E} \left[ \operatorname {T e r m} _ {3} | \mathcal {F} _ {k - \tau} \right] = 2 \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} + \eta_ {k} g (\boldsymbol {\theta} _ {k}) \right\| \cdot \left\| y _ {0} (\boldsymbol {\theta} _ {k}) - y _ {0} (\boldsymbol {\theta} _ {k + 1}) \right\| | \mathcal {F} _ {k - \tau} \right] \\ \stackrel {(d _ {8})} {\leq} \frac {\eta_ {k}}{\alpha_ {k}} \operatorname {T e r m} _ {1} + \frac {\alpha_ {k}}{\eta_ {k}} \operatorname {T e r m} _ {2} \\ = \frac {\eta_ {k}}{\alpha_ {k}} \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] - \frac {2 \eta_ {k} ^ {2} \mu_ {2}}{\alpha_ {k}} \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] + \frac {8 \eta_ {k} ^ {3}}{\alpha_ {k}} \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] \\ + \eta_ {k} ^ {2} \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] + \frac {\eta_ {k}}{\alpha_ {k}} \left(4 \eta_ {k} / \alpha_ {k} + 8 \eta_ {k} ^ {2}\right) \mathcal {O} \left(c _ {1} ^ {3} \left(\| \boldsymbol {\theta} _ {0} \| + | \lambda_ {0} | + 1\right) ^ {3} \cdot m ^ {- 1 / 2}\right) \\ + \frac {2 4 \alpha_ {k} ^ {3}}{\eta_ {k}} \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] + \frac {6 0 0 \alpha_ {k} ^ {3}}{\eta_ {k}} \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] \\ + \frac {8 \alpha_ {k} ^ {3}}{\eta_ {k}} \mathcal {O} \left(c _ {1} ^ {3} \left(\| \boldsymbol {\theta} _ {0} \| + | \lambda_ {0} | + 1\right) ^ {3} \cdot m ^ {- 1 / 2}\right), \\ \end{array}
$$
where  $(d_8)$  holds because  $2\mathbf{x}^T\mathbf{y}\leq 1 / \beta \| \mathbf{x}\| ^2 +\beta \| \mathbf{y}\| ^2,\forall \beta >0$  . Summing Term, Term, and Term3 and neglecting higher order infinitesimal yield the desired result.
Now we are ready to prove the results in Theorem 2. Providing Lemma 8 and Lemma 7, if  $\frac{\eta_k}{\alpha_k}$  is non-increasing, we have the following inequality
$$
\begin{array}{l} \mathbb {E} \left[ \hat {M} (\pmb {\theta} _ {k + 1}, \lambda_ {k + 1}) \Big | \mathcal {F} _ {k - \tau} \right] = \mathbb {E} \left[ \frac {\eta_ {k}}{\alpha_ {k}} \left\| \hat {\pmb {\theta}} _ {k + 1} \right\| ^ {2} + \left\| \hat {\lambda} _ {k + 1} \right\| ^ {2} \Big | \mathcal {F} _ {k - \tau} \right] \\ \leq \frac {\eta_ {k}}{\alpha_ {k}} (1 - 2 \alpha_ {k} \mu_ {1}) \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] + \frac {6 0 0 \alpha_ {k} ^ {3}}{\eta_ {k}} \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] \\ + \frac {8 \alpha_ {k} ^ {3}}{\eta_ {k}} \mathcal {O} \left(c _ {1} ^ {3} \left(\| \boldsymbol {\theta} _ {0} \| + | \lambda_ {0} | + 1\right) ^ {3} \cdot m ^ {- 1 / 2}\right) \\ + \left(1 - 2 \eta_ {k} \mu_ {2}\right) \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} | \mathcal {F} _ {k - \tau} \right] \\ + \frac {6 0 0 \alpha_ {k} ^ {3}}{\eta_ {k}} \mathbb {E} \left[ \left\| \hat {\lambda} _ {k} \right\| ^ {2} \mid \mathcal {F} _ {k - \tau} \right]. \tag {43} \\ \end{array}
$$
Since  $(k + 1)^2\cdot \frac{\alpha_k^3}{\eta} = \frac{\alpha_0^3}{\eta_0} (k + 1)^{1 / 3}$ , multiplying both sides of (43) with  $(k + 1)^2$ , we have
$$
\begin{array}{l} (k + 1) ^ {2} \mathbb {E} \left[ \hat {M} (\boldsymbol {\theta} _ {k + 1}, \lambda_ {k + 1}) \mid \mathcal {F} _ {k - \tau} \right] \\ \leq k ^ {2} \mathbb {E} \left[ \hat {M} (\boldsymbol {\theta} _ {k}, \lambda_ {k}) \mid \mathcal {F} _ {k - \tau} \right] + \frac {6 0 0 \alpha_ {0} ^ {3}}{\eta_ {0}} (k + 1) ^ {1 / 3} \left(\left\| \hat {\boldsymbol {\theta}} _ {k} \right\| ^ {2} + \left\| \hat {\lambda} _ {k} \right\| ^ {2}\right) \\ + \frac {8 \alpha_ {0} ^ {3}}{\eta_ {0}} (k + 1) ^ {1 / 3} \mathcal {O} \left(c _ {1} ^ {3} \left(\| \boldsymbol {\theta} _ {0} \| + | \lambda_ {0} | + 1\right) ^ {3} \cdot m ^ {- 1 / 2}\right). \tag {44} \\ \end{array}
$$
Summing (44) from time step  $\tau$  to time step  $k$ , we have
$$
\begin{array}{l} (k + 1) ^ {2} \mathbb {E} \left[ \hat {M} (\boldsymbol {\theta} _ {k + 1}, \lambda_ {k + 1}) \Big | \mathcal {F} _ {k} \right] \leq \tau^ {2} \mathbb {E} \left[ \hat {M} (\boldsymbol {\theta} _ {\tau}, \lambda_ {\tau}) \right] + \frac {6 0 0 \alpha_ {0} ^ {3}}{\eta_ {0}} (k + 1) ^ {4 / 3} \left(\left\| \hat {\boldsymbol {\theta}} _ {\tau} \right\| ^ {2} + \left\| \hat {\lambda} _ {\tau} \right\| ^ {2}\right) \\ + \frac {8 \alpha_ {0} ^ {3}}{\eta_ {0}} (k + 1) ^ {4 / 3} \mathcal {O} \left(c _ {1} ^ {3} \left(\| \boldsymbol {\theta} _ {0} \| + | \lambda_ {0} | + 1\right) ^ {3} \cdot m ^ {- 1 / 2}\right) \\ \leq \tau^ {2} \mathbb {E} \left[ \hat {M} (\boldsymbol {\theta} _ {\tau}, \lambda_ {\tau}) \right] + \frac {6 0 0 \alpha_ {0} ^ {3}}{\eta_ {0}} \frac {(C _ {1} + \| \hat {\boldsymbol {\theta}} _ {0} \|) ^ {2} + (2 C _ {1} + \| \hat {\lambda} _ {0} \|) ^ {2}}{(k + 1) ^ {- 4 / 3}} \\ + \frac {8 \alpha_ {0} ^ {3}}{\eta_ {0}} \frac {\mathcal {O} \left(c _ {1} ^ {3} \left(\| \boldsymbol {\theta} _ {0} \| + | \lambda_ {0} | + 1\right) ^ {3} \cdot m ^ {- 1 / 2}\right)}{(k + 1) ^ {- 4 / 3}}, \tag {45} \\ \end{array}
$$
where the second inequality holds due to Lemma 9. Finally, dividing both sides by  $(k + 1)^2$  and moving the constant term into  $\mathcal{O}(\cdot)$  yields the results in Theorem 2.
In this part, we present several key lemmas which are needed for the major proofs. We first show the parameters update in (9) is bounded in the following lemma.
Lemma 9. The update of  $\pmb{\theta}_k$  and  $\lambda_{k}$  in (9) is bounded with respect to the initial  $\pmb{\theta}_0$  and  $\lambda_0$ , i.e.,
$$
\left\| \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} _ {0} \right\| + \left| \lambda_ {k} - \lambda_ {0} \right| \leq c _ {1} \left(\left\| \boldsymbol {\theta} _ {0} \right\| + \left| \lambda_ {0} \right| + 1\right),
$$
with  $c_{1}$  be the constant, i.e.,  $c_{1} := \frac{1}{2} + \frac{3}{2}(L_{h}'\alpha_{\tau} + L_{g}'\eta_{\tau})(L_{h}'\alpha_{\tau} + L_{g}'\eta_{\tau} + 1)$ .
Proof. Without loss of generality, we assume that
$$
L _ {h} ^ {\prime} \geq \max  \left(3, \max  _ {X \in \mathcal {X}} \| h _ {0} (X, 0, 0) \|\right), L _ {g} ^ {\prime} \geq \max  \left(2, \max  _ {X \in \mathcal {X}} \| g _ {0} (X, 0, 0) \|\right).
$$
Then based on triangular inequality and Lemmas 1-2, we have
$$
\left\| h _ {0} (X, \boldsymbol {\theta}, \lambda) \right\| \leq L _ {h} ^ {\prime} (\| \boldsymbol {\theta} \| + | \lambda | + 1), \| g _ {0} (X, \boldsymbol {\theta}, \lambda) \| \leq L _ {g} ^ {\prime} (\| \boldsymbol {\theta} \| + | \lambda | + 1), \forall \boldsymbol {\theta}, \lambda , X \in \mathcal {X}. \tag {46}
$$
Since we have  $\pmb{\theta}_{k + 1} = \pmb{\theta}_k + \alpha_k h(X_k, \pmb{\theta}_k, \lambda_k)$ , we have the following inequality due to Lipschitz continuity of  $h$  in (46)
$$
\left\| \boldsymbol {\theta} _ {k + 1} - \boldsymbol {\theta} _ {k} \right\| = \alpha_ {k} \| h \left(X _ {k}, \boldsymbol {\theta} _ {k}, \lambda_ {k}\right) \| \leq \alpha_ {k} L _ {h} ^ {\prime} \left(\left\| \boldsymbol {\theta} _ {k} \right\| + \left| \lambda_ {k} \right| + 1\right). \tag {47}
$$
$$
\left| \lambda_ {k + 1} - \lambda_ {k} \right| = \eta_ {k} \left| g \left(X _ {k}, \boldsymbol {\theta} _ {k}, \lambda_ {k}\right) \right| \leq \eta_ {k} L _ {g} ^ {\prime} \left(\left\| \boldsymbol {\theta} _ {k} \right\| + \left| \lambda_ {k} \right| + 1\right). \tag {48}
$$
Due to triangular inequality, adding (47) and (48) leads to
$$
\begin{array}{l} \left\| \boldsymbol {\theta} _ {k + 1} \right\| + \left| \lambda_ {k + 1} \right| + 1 \leq \left(L _ {h} ^ {\prime} \alpha_ {k} + L _ {g} ^ {\prime} \eta_ {k} + 1\right) \left(\left\| \boldsymbol {\theta} _ {k} \right\| + \left| \lambda_ {k} \right| + 1\right) \\ \leq \left(L _ {h} ^ {\prime} \alpha_ {0} + L _ {g} ^ {\prime} \eta_ {0} + 1\right) \left(\left\| \boldsymbol {\theta} _ {k} \right\| + \left| \lambda_ {k} \right| + 1\right), \tag {49} \\ \end{array}
$$
where the second inequality holds due to the non-increasing learning rates  $\{\alpha_k, \eta_k\}$ . Rewriting the above inequality in (49) in a recursive manner yields
$$
\left\| \boldsymbol {\theta} _ {k} \right\| + \lambda_ {k} + 1 \leq \left(L _ {h} ^ {\prime} \alpha_ {0} + L _ {g} ^ {\prime} \eta_ {0} + 1\right) ^ {k - \tau} \left(\left\| \boldsymbol {\theta} _ {\tau} \right\| + \left| \lambda_ {\tau} \right| + 1\right). \tag {50}
$$
$$
\begin{array}{l} \left\| \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} _ {k - \tau} \right\| + \left| \lambda_ {k} - \lambda_ {k - \tau} \right| \leq \sum_ {t = k - \tau} ^ {k - 1} \left\| \boldsymbol {\theta} _ {t + 1} - \boldsymbol {\theta} _ {t} \right\| + \left| \lambda_ {t + 1} - \lambda_ {t} \right| \\ \leq \left(L _ {h} ^ {\prime} \alpha_ {0} + L _ {g} ^ {\prime} \eta_ {0}\right) \sum_ {t = k - \tau} ^ {k - 1} \left(\left\| \boldsymbol {\theta} _ {t} \right\| + \left| \lambda_ {t} \right| + 1\right) \\ \leq \left(L _ {h} ^ {\prime} \alpha_ {0} + L _ {g} ^ {\prime} \eta_ {0}\right) \left(\| \boldsymbol {\theta} _ {k - \tau} \| + | \lambda_ {k - \tau} | + 1\right) \sum_ {t = k - \tau} ^ {k - 1} \left(L _ {h} ^ {\prime} \alpha_ {0} + L _ {g} ^ {\prime} \eta_ {0} + 1\right) ^ {t - \tau} \\ = \left[ \left(L _ {h} ^ {\prime} \alpha_ {0} + L _ {g} ^ {\prime} \eta_ {0} + 1\right) ^ {\tau} - 1 \right] \left(\| \boldsymbol {\theta} _ {k - \tau} \| + | \lambda_ {k - \tau} | + 1\right) \\ \leq \left(e ^ {\left(L _ {h} ^ {\prime} \alpha_ {0} + L _ {g} ^ {\prime} \eta_ {0}\right) \tau} - 1\right) \left(\| \boldsymbol {\theta} _ {k - \tau} \| + | \lambda_ {k - \tau} | + 1\right) \\ \leq 2 \left(L _ {h} ^ {\prime} \alpha_ {0} + L _ {g} ^ {\prime} \eta_ {0}\right) \tau \left(\| \boldsymbol {\theta} _ {k - \tau} \| + | \lambda_ {k - \tau} | + 1\right), \\ \end{array}
$$
where the last inequality holds when  $(L_h^\prime \alpha_0 + L_g^\prime \eta_0)\tau \leq 1 / 4$ . This implies when  $k = \tau$ , we have
$$
\left\| \boldsymbol {\theta} _ {\tau} - \boldsymbol {\theta} _ {0} \right\| + \left| \lambda_ {\tau} - \lambda_ {0} \right| \leq 2 \left(L _ {h} ^ {\prime} \alpha_ {0} + L _ {g} ^ {\prime} \eta_ {0}\right) \tau \left(\left\| \boldsymbol {\theta} _ {0} \right\| + \left| \lambda_ {0} \right| + 1\right). \tag {51}
$$