SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\stackrel {(h)} {\leq} \left(1 + \ell_ {F} ^ {2} \gamma\right) \| \lambda - \lambda^ {\prime} \| + \left(\ell_ {f} \ell_ {F, 1} + \ell_ {F} \ell_ {f, 1}\right) \gamma \| x - x ^ {\prime} \| \tag {B.55}
$$
where  $(d)$  follows from non-expansiveness of projection;  $(e)$  follows from triangle inequality,  $(f)$  follows from  $\| \nabla F_z(x)\| \leq \ell_F$  for  $x\in \{x_t'\}_{t = 1}^T$ ,  $(g)$  follows from triangle inequality; and  $(h)$  follows from  $\ell_{f,1}$ -Lipschitz continuity of  $\nabla F_z(x)\lambda^\prime$ ,  $\ell_{F,1}$ -Lipschitz continuity of  $\nabla F_z(x)$ ,  $\| \nabla F_z(x)\| \leq \ell_F$  for  $x\in \{x_t'\}_{t = 1}^T$ , and  $\| \nabla F_z(x)\lambda '\| \leq \ell_f$  for  $x\in \{x_t\}_{t = 1}^T$ .
Let  $\ell_{g,1} = \ell_f\ell_{F,1} + \ell_F\ell_{f,1}$ . Taking square on both sides of (B.55) yields
$$
\begin{array}{l} \left\| G _ {x, z _ {1}, z _ {2}} (\lambda) - G _ {x ^ {\prime}, z _ {1}, z _ {2}} \left(\lambda^ {\prime}\right) \right\| ^ {2} \\ \leq \left(\left(1 + \ell_ {F} ^ {2} \gamma\right) \| \lambda - \lambda^ {\prime} \| + \ell_ {g, 1} \gamma \| x - x ^ {\prime} \|\right) ^ {2} \\ = (1 + \ell_ {F} ^ {2} \gamma) ^ {2} \| \lambda - \lambda^ {\prime} \| ^ {2} + 2 (1 + \ell_ {F} ^ {2} \gamma) \ell_ {g, 1} \gamma \| \lambda - \lambda^ {\prime} \| \| x - x ^ {\prime} \| + \ell_ {g, 1} ^ {2} \gamma^ {2} \| x - x ^ {\prime} \| ^ {2} \\ \leq \left(1 + \ell_ {F} ^ {2} \gamma\right) ^ {2} \| \lambda - \lambda^ {\prime} \| ^ {2} + \left(1 + \ell_ {F} ^ {2} \gamma\right) \ell_ {g, 1} \gamma \left(\| \lambda - \lambda^ {\prime} \| ^ {2} + \| x - x ^ {\prime} \| ^ {2}\right) + \ell_ {g, 1} ^ {2} \gamma^ {2} \| x - x ^ {\prime} \| ^ {2} \\ = \left((1 + \ell_ {F} ^ {2} \gamma) ^ {2} + (1 + \ell_ {F} ^ {2} \gamma) \ell_ {g, 1} \gamma\right) \| \lambda - \lambda^ {\prime} \| ^ {2} + \left((1 + \ell_ {F} ^ {2} \gamma) \ell_ {g, 1} \gamma + \ell_ {g, 1} ^ {2} \gamma^ {2}\right) \| x - x ^ {\prime} \| ^ {2}. \tag {B.56} \\ \end{array}
$$
B.4.2 Growth recursion
Lemma 9 (Growth recursion with approximate expansiveness). Fix an arbitrary sequence of updates  $G_{1},\ldots ,G_{T}$  and another sequence  $G_1^\prime ,\ldots ,G_T^\prime$ . Let  $x_0 = x_0^\prime$  be a starting point in  $\Omega$  and define  $\delta_t = \| x_t' - x_t\|$  where  $x_{t},x_{t}^{\prime}$  are defined recursively through
$$
x _ {t + 1} = G _ {t} \left(x _ {t}\right), \quad x _ {t + 1} ^ {\prime} = G _ {t} ^ {\prime} \left(x _ {t} ^ {\prime}\right) \quad (t > 0).
$$
Let  $\eta_t > 0, \nu_t \geq 0$ , and  $\varsigma_t \geq 0$ . Then, for any  $p > 0$ , and  $t \in [T]$ , we have the recurrence relation (with  $\delta_0 = 0$ )
$$
\delta_ {t + 1} ^ {2} \leq \left\{ \begin{array}{l l} \eta_ {t} \delta_ {t} ^ {2} + \nu_ {t}, & G _ {t} = G _ {t} ^ {\prime} i s (\eta_ {t}, \nu_ {t}) \text {- a p p r o x i m a t e l y e x p a n s i v e i n s q u a r e}; \\ (1 + p) \min  \{\eta_ {t} \delta_ {t} ^ {2} + \nu_ {t}, \delta_ {t} ^ {2} \} + (1 + \frac {1}{p}) 4 \varsigma_ {t} ^ {2} & G _ {t} a n d G _ {t} ^ {\prime} a r e \varsigma_ {t} \text {- b o u n d e d}, \\ & G _ {t} i s (\eta_ {t}, \nu_ {t}) \text {- a p p r o x i m a t e l y e x p a n s i v e i n s q u a r e}. \end{array} \right.
$$
Proof. When  $G_{t}$  and  $G_{t}^{\prime}$  are  $\varsigma_{t}$ -bounded, we can bound  $\delta_{t+1}$  by
$$
\begin{array}{l} \delta_ {t + 1} = \left\| x _ {t + 1} - x _ {t + 1} ^ {\prime} \right\| = \left\| G _ {t} \left(x _ {t}\right) - G _ {t} ^ {\prime} \left(x _ {t} ^ {\prime}\right) \right\| \\ = \| G _ {t} \left(x _ {t}\right) - x _ {t} - G _ {t} ^ {\prime} \left(x _ {t} ^ {\prime}\right) + x _ {t} ^ {\prime} + x _ {t} - x _ {t} ^ {\prime} \| \\ \leq \left\| G _ {t} \left(x _ {t}\right) - x _ {t} \right\| + \left\| G _ {t} ^ {\prime} \left(x _ {t} ^ {\prime}\right) - x _ {t} ^ {\prime} \right\| + \left\| x _ {t} - x _ {t} ^ {\prime} \right\| \\ \leq 2 \varsigma_ {t} + \delta_ {t}. \tag {B.57} \\ \end{array}
$$
Alternatively, when  $G_{t}$  and  $G_{t}^{\prime}$  are  $\varsigma_{t}$ -bounded,  $G_{t}$  is  $(\eta_t,\nu_t)$ -approximately expansive, we have
$$
\begin{array}{l} \delta_ {t + 1} = \left\| x _ {t + 1} - x _ {t + 1} ^ {\prime} \right\| = \left\| G _ {t} \left(x _ {t}\right) - G _ {t} ^ {\prime} \left(x _ {t} ^ {\prime}\right) \right\| \\ = \| G _ {t} \left(x _ {t}\right) - G _ {t} \left(x _ {t} ^ {\prime}\right) + G _ {t} \left(x _ {t} ^ {\prime}\right) - G _ {t} ^ {\prime} \left(x _ {t} ^ {\prime}\right) \| \\ \leq \left\| G _ {t} \left(x _ {t}\right) - G _ {t} \left(x _ {t} ^ {\prime}\right) \right\| + \left\| G _ {t} \left(x _ {t} ^ {\prime}\right) - G _ {t} ^ {\prime} \left(x _ {t} ^ {\prime}\right) \right\| \\ \leq \eta_ {t} \delta_ {t} + \nu_ {t} + \| G _ {t} \left(x _ {t} ^ {\prime}\right) - x _ {t} ^ {\prime} - G _ {t} ^ {\prime} \left(x _ {t} ^ {\prime}\right) + x _ {t} ^ {\prime} \| \\ \leq \eta_ {t} \delta_ {t} + \nu_ {t} + \| G _ {t} \left(x _ {t} ^ {\prime}\right) - x _ {t} ^ {\prime} \| + \| G _ {t} ^ {\prime} \left(x _ {t} ^ {\prime}\right) - x _ {t} ^ {\prime} \| \\ \leq \eta_ {t} \delta_ {t} + \nu_ {t} + 2 \varsigma_ {t}. \tag {B.58} \\ \end{array}
$$
When  $G_{t} = G_{t}^{\prime}$ , is  $(\eta_t, \nu_t)$ -approximately expansive in square, given  $\delta_t^2$ ,  $\delta_{t + 1}^2$  can be bounded by
$$
\delta_ {t + 1} ^ {2} = \left\| x _ {t + 1} - x _ {t + 1} ^ {\prime} \right\| ^ {2} = \left\| G _ {t} \left(x _ {t}\right) - G _ {t} \left(x _ {t} ^ {\prime}\right) \right\| ^ {2} \leq \eta_ {t} \left\| x _ {t} - x _ {t} ^ {\prime} \right\| ^ {2} + \nu_ {t} = \eta_ {t} \delta_ {t} ^ {2} + \nu_ {t}. \tag {B.59}
$$
When  $G_{t}$  and  $G_{t}^{\prime}$  are  $\varsigma_{t}$ -bounded, applying (B.57), we can bound  $\delta_{t+1}^{2}$  by
$$
\delta_ {t + 1} ^ {2} \leq \left(\delta_ {t} + 2 \varsigma_ {t}\right) ^ {2} \leq (1 + p) \delta_ {t} ^ {2} + (1 + 1 / p) 4 \varsigma_ {t} ^ {2} \tag {B.60}
$$
where  $p > 0$  and the last inequality follows from  $(a + b)^2 \leq (1 + p)a^2 + (1 + 1/p)b^2$ .
Alternatively, when  $G_{t}$  and  $G_{t}^{\prime}$  are  $\varsigma_{t}$ -bounded,  $G_{t}$  is  $(\eta_t,\nu_t)$ -approximately expansive in square, the following holds
$$
\begin{array}{l} \delta_ {t + 1} ^ {2} = \left\| x _ {t + 1} - x _ {t + 1} ^ {\prime} \right\| ^ {2} = \left\| G _ {t} (x _ {t}) - G _ {t} ^ {\prime} (x _ {t} ^ {\prime}) \right\| ^ {2} \\ = \| G _ {t} \left(x _ {t}\right) - G _ {t} \left(x _ {t} ^ {\prime}\right) + G _ {t} \left(x _ {t} ^ {\prime}\right) - G _ {t} ^ {\prime} \left(x _ {t} ^ {\prime}\right) \| ^ {2} \\ \leq (1 + p) \| G _ {t} \left(x _ {t}\right) - G _ {t} \left(x _ {t} ^ {\prime}\right) \| ^ {2} + (1 + 1 / p) \| G _ {t} \left(x _ {t} ^ {\prime}\right) - G _ {t} ^ {\prime} \left(x _ {t} ^ {\prime}\right) \| ^ {2} \\ \leq (1 + p) \left(\eta_ {t} \delta_ {t} ^ {2} + \nu_ {t}\right) + (1 + 1 / p) \| G _ {t} \left(x _ {t} ^ {\prime}\right) - x _ {t} ^ {\prime} - G _ {t} ^ {\prime} \left(x _ {t} ^ {\prime}\right) + x _ {t} ^ {\prime} \| ^ {2} \\ \leq (1 + p) \left(\eta_ {t} \delta_ {t} ^ {2} + \nu_ {t}\right) + 2 (1 + 1 / p) \left(\| G _ {t} \left(x _ {t} ^ {\prime}\right) - x _ {t} ^ {\prime} \| ^ {2} + \| G _ {t} ^ {\prime} \left(x _ {t} ^ {\prime}\right) - x _ {t} ^ {\prime} \| ^ {2}\right) \\ \leq (1 + p) \left(\eta_ {t} \delta_ {t} ^ {2} + \nu_ {t}\right) + (1 + 1 / p) 4 \zeta_ {t} ^ {2}. \tag {B.61} \\ \end{array}
$$
B.4.3 Upper bound of MOL uniform stability
In Theorem 5 we bound the argument stability, which is then used to derive the MOL uniform stability and PS generalization error in Theorem 2.
Theorem 5 (Argument stability bound in strongly convex case). Suppose Assumptions 1, 2, hold. Let  $A$  be the MoDo algorithm in Algorithm 1. Choose the step sizes  $\alpha_{t} \leq \alpha \leq \min \{1 / (2\ell_{f,1}), \mu / (2\ell_{f,1}^{2})\}$ , and  $\gamma_{t} \leq \gamma \leq \min \{\frac{\mu^{2}}{120\ell_{f}^{2}\ell_{g,1}}, \frac{1}{8(3\ell_{f}^{2} + 2\ell_{g,1})}\} / T$ . Then it holds that
$$
\mathbb {E} _ {A} \left[ \| A (S) - A \left(S ^ {\prime}\right) \| ^ {2} \right] \leq \frac {4 8}{\mu n} \ell_ {f} ^ {2} \left(\alpha + \frac {1 2 + 4 M \ell_ {f} ^ {2}}{\mu n} + \frac {1 0 M \ell_ {f} ^ {4} \gamma}{\mu}\right). \tag {B.62}
$$
Proof of Theorem 5. Under Assumptions 1, 2, Lemma 1 implies that for  $\{x_{t}\}$  generated by the MoDo algorithm, and for all  $\lambda \in \Delta^{M}$ , and for all  $m\in [M]$
$$
\left\| \nabla F _ {z} \left(x _ {t}\right) \lambda \right\| \leq \ell_ {f, 1} \left(c _ {x} + c _ {x ^ {*}}\right) = \ell_ {f}. \quad \text {a n d} \quad \left\| \nabla F _ {z} \left(x _ {t}\right) \right\| \leq \left\| \nabla F _ {z} \left(x _ {t}\right) \right\| _ {\mathrm {F}} \leq \sqrt {M} \ell_ {f} = \ell_ {F}. (B. 6 3)
$$
For notation simplicity, denote  $\delta_t = \| x_t - x'_t\|$ ,  $\zeta_t = \|\lambda_t - \lambda'_t\|$ ,  $x_T = A_T(S)$  and  $x'_T = A_T(S')$ . Denote the index of the different sample in  $S$  and  $S'$  as  $i^*$ , and the set of indices selected at the  $t$ -th iteration as  $I_t$ , i.e.,  $I_t = \{i_{t,s}\}_{s=1}^3$ . When  $i^* \notin I_t$ , for any  $c_1 > 0$ , based on Lemma 8, we have
$$
\begin{array}{l} \delta_ {t + 1} ^ {2} \leq (1 - 2 \alpha_ {t} \mu + 2 \alpha_ {t} ^ {2} \ell_ {f, 1} ^ {2}) \delta_ {t} ^ {2} + 2 \alpha_ {t} \ell_ {F} \delta_ {t} \zeta_ {t + 1} + 2 \alpha_ {t} ^ {2} \ell_ {F} ^ {2} \zeta_ {t + 1} ^ {2} \\ \leq \left(1 - 2 \alpha_ {t} \mu + 2 \alpha_ {t} ^ {2} \ell_ {f, 1} ^ {2}\right) \delta_ {t} ^ {2} + \alpha_ {t} \ell_ {F} \left(c _ {1} \delta_ {t} ^ {2} + c _ {1} ^ {- 1} \zeta_ {t + 1} ^ {2}\right) + 2 \alpha_ {t} ^ {2} \ell_ {F} ^ {2} \zeta_ {t + 1} ^ {2} \\ \leq \left(1 - \alpha_ {t} \mu\right) \delta_ {t} ^ {2} + \alpha_ {t} \ell_ {F} \left(c _ {1} \delta_ {t} ^ {2} + c _ {1} ^ {- 1} \zeta_ {t + 1} ^ {2}\right) + 2 \alpha_ {t} ^ {2} \ell_ {F} ^ {2} \zeta_ {t + 1} ^ {2} \tag {B.64} \\ \end{array}
$$
where the second last inequality is due to Young's inequality; the last inequality is due to choosing  $\alpha_{t} \leq \mu / (2\ell_{f,1}^{2})$ .
When  $i^{*} \in I_{t}$ , from Lemma 7, the  $(\alpha_{t}\ell_{f})$ -boundedness of the update at  $t$ -th iteration, and Lemma 9, the growth recursion, for a given constant  $p > 0$ , we have
$$
\delta_ {t + 1} ^ {2} \leq (1 + p) \delta_ {t} ^ {2} + (1 + 1 / p) 4 \alpha_ {t} ^ {2} \ell_ {f} ^ {2}. \tag {B.65}
$$
Taking expectation of  $\delta_{t + 1}^2$  over  $I_{t}$ , we have
$$
\begin{array}{l} \mathbb {E} _ {I _ {t}} \left[ \delta_ {t + 1} ^ {2} \right] \leq \mathbb {P} (i ^ {*} \notin I _ {t}) \Big ((1 - \alpha_ {t} \mu) \delta_ {t} ^ {2} + \alpha_ {t} \ell_ {F} c _ {1} \delta_ {t} ^ {2} + (\alpha_ {t} \ell_ {F} c _ {1} ^ {- 1} + 2 \alpha_ {t} ^ {2} \ell_ {F} ^ {2}) \mathbb {E} _ {I _ {t}} \left[ \zeta_ {t + 1} ^ {2} \mid i ^ {*} \notin I _ {t} \right] \Big) \\ + \mathbb {P} (i ^ {*} \in I _ {t}) \left((1 + p) \delta_ {t} ^ {2} + (1 + 1 / p) 4 \alpha_ {t} ^ {2} \ell_ {f} ^ {2}\right) \\ \leq \left(1 - \alpha_ {t} \left(\mu - \ell_ {F} c _ {1}\right) \mathbb {P} \left(i ^ {*} \notin I _ {t}\right) + p \mathbb {P} \left(i ^ {*} \in I _ {t}\right)\right) \delta_ {t} ^ {2} \\ + \alpha_ {t} \underbrace {\left(\ell_ {F} c _ {1} ^ {- 1} + 2 \alpha \ell_ {F} ^ {2}\right)} _ {c _ {2}} \mathbb {E} _ {I _ {t}} \left[ \zeta_ {t + 1} ^ {2} \mid i ^ {*} \notin I _ {t} \right] \mathbb {P} \left(i ^ {*} \notin I _ {t}\right) + \left(1 + \frac {1}{p}\right) \mathbb {P} \left(i ^ {*} \in I _ {t}\right) 4 \alpha_ {t} ^ {2} \ell_ {f} ^ {2}. \tag {B.66} \\ \end{array}
$$
At each iteration of MoDo, we randomly select three independent samples (instead of one) from the training set  $S$ . Then the probability of selecting the different sample from  $S$  and  $S'$  at the  $t$ -th iteration,  $\mathbb{P}(i^* \in I_t)$  in the above equation, can be computed as follows
$$
\mathbb {P} \left(i ^ {*} \in I _ {t}\right) = 1 - \left(\frac {n - 1}{n}\right) ^ {3} \leq \frac {3}{n}. \tag {B.67}
$$
Consequently, the probability of selecting the same sample from  $S$  and  $S'$  at the  $t$ -th iteration is  $\mathbb{P}(i^* \notin I_t) = 1 - \mathbb{P}(i^* \in I_t)$ .
Let  $\ell_{g,1} = \ell_f\ell_{F,1} + \ell_F\ell_{f,1}$ . Recalling when  $i^{*}\notin I_{t},\zeta_{t + 1}\leq (1 + \ell_{F}^{2}\gamma_{t})\zeta_{t} + 2\gamma_{t}\ell_{g,1}\delta_{t}$  from Lemma 8, it follows that
$$
\begin{array}{l} \zeta_ {t + 1} ^ {2} \leq \left((1 + \ell_ {F} ^ {2} \gamma_ {t}) ^ {2} + (1 + \ell_ {F} ^ {2} \gamma_ {t}) \ell_ {g, 1} \gamma_ {t}\right) \zeta_ {t} ^ {2} + \left((1 + \ell_ {F} ^ {2} \gamma_ {t}) \ell_ {g, 1} \gamma_ {t} + \ell_ {g, 1} ^ {2} \gamma_ {t} ^ {2}\right) \delta_ {t} ^ {2} \\ \leq \left(1 + \underbrace {\left(3 \ell_ {F} ^ {2} + 2 \ell_ {g , 1}\right)} _ {c _ {3}} \gamma_ {t}\right) \zeta_ {t} ^ {2} + 3 \ell_ {g, 1} \gamma_ {t} \delta_ {t} ^ {2} \tag {B.68} \\ \end{array}
$$
where the last inequality follows from  $\ell_{g,1}\gamma_t\leq 1$  , and  $\ell_F^2\gamma_t\leq 1$
And since  $\zeta_t$  and  $\delta_t$  are independent of  $I_t$ , it follows that
$$
\mathbb {E} _ {I _ {t}} \left[ \zeta_ {t + 1} ^ {2} \mid i ^ {*} \notin I _ {t} \right] \leq \left(1 + c _ {3} \gamma_ {t}\right) \zeta_ {t} ^ {2} + 3 \ell_ {g, 1} \gamma_ {t} \delta_ {t} ^ {2}. \tag {B.69}
$$
Combining (B.66) and (B.69), we have
$$
\begin{array}{l} \mathbb {E} _ {I _ {t}} \left[ \delta_ {t + 1} ^ {2} \right] \leq \left(1 - \alpha_ {t} \left(\mu - \ell_ {F} c _ {1}\right) \mathbb {P} \left(i ^ {*} \notin I _ {t}\right) + p \mathbb {P} \left(i ^ {*} \in I _ {t}\right)\right) \delta_ {t} ^ {2} + \left(1 + \frac {1}{p}\right) \mathbb {P} \left(i ^ {*} \in I _ {t}\right) 4 \alpha_ {t} ^ {2} \ell_ {f} ^ {2} \\ + \alpha_ {t} c _ {2} \left(\left(1 + c _ {3} \gamma_ {t}\right) \zeta_ {t} ^ {2} + 3 \ell_ {g, 1} \gamma_ {t} \delta_ {t} ^ {2}\right) \mathbb {P} \left(i ^ {*} \notin I _ {t}\right) \tag {B.70} \\ \end{array}
$$
$$
= \Big (\eta_ {t} + p \mathbb {P} (i ^ {*} \in I _ {t}) \Big) \delta_ {t} ^ {2} + \alpha_ {t} c _ {2} \big (1 + c _ {3} \gamma_ {t} \big) \zeta_ {t} ^ {2} \mathbb {P} (i ^ {*} \notin I _ {t}) + \Big (1 + \frac {1}{p} \Big) \mathbb {P} (i ^ {*} \in I _ {t}) 4 \alpha_ {t} ^ {2} \ell_ {f} ^ {2}
$$
where we define  $\eta_t = 1 - \alpha_t (\mu - \ell_F c_1 - 3c_2 \ell_{g,1} \gamma_t) \mathbb{P}(i^* \notin I_t)$ .
While when  $i^{*} \in I_{t}$ , for a given constant  $p_2 > 0$ , we have
$$
\begin{array}{l} \zeta_ {t + 1} = \left\| \Pi_ {\Delta^ {M}} \left(\lambda_ {t} - \gamma_ {t} h _ {t, 1} \left(x _ {t}\right) ^ {\top} h _ {t, 2} \left(x _ {t}\right) \lambda_ {t}\right) - \Pi_ {\Delta^ {M}} \left(\lambda_ {t} ^ {\prime} - \gamma_ {t} h _ {t, 1} ^ {\prime} \left(x _ {t} ^ {\prime}\right) ^ {\top} h _ {t, 2} ^ {\prime} \left(x _ {t} ^ {\prime}\right) \lambda_ {t} ^ {\prime}\right) \right\| \\ \leq \left\| \lambda_ {t} - \lambda_ {t} ^ {\prime} - \gamma_ {t} \left(h _ {t, 1} \left(x _ {t}\right) ^ {\top} h _ {t, 2} \left(x _ {t}\right) \lambda_ {t} - h _ {t, 1} ^ {\prime} \left(x _ {t} ^ {\prime}\right) ^ {\top} h _ {t, 2} ^ {\prime} \left(x _ {t} ^ {\prime}\right) \lambda_ {t} ^ {\prime}\right) \right\| \\ \leq \left\| \lambda_ {t} - \lambda_ {t} ^ {\prime} \right\| + 2 \gamma_ {t} \ell_ {F} \ell_ {f} \leq \zeta_ {t} + 2 \gamma_ {t} \sqrt {M} \ell_ {f} ^ {2} \\ \end{array}
$$
$$
\zeta_ {t + 1} ^ {2} \leq \left(1 + p _ {2}\right) \zeta_ {t} ^ {2} + \left(1 + 1 / p _ {2}\right) 4 \gamma_ {t} ^ {2} M \ell_ {f} ^ {4}. \tag {B.71}
$$
Taking expectation of  $\zeta_{t + 1}^2$  over  $I_{t}$  gives
$$
\begin{array}{l} \mathbb {E} _ {I _ {t}} \left[ \zeta_ {t + 1} ^ {2} \right] = \mathbb {E} _ {I _ {t}} \left[ \zeta_ {t + 1} ^ {2} \mid i ^ {*} \in I _ {t} \right] \mathbb {P} \left(i ^ {*} \in I _ {t}\right) + \mathbb {E} _ {I _ {t}} \left[ \zeta_ {t + 1} ^ {2} \mid i ^ {*} \notin I _ {t} \right] \mathbb {P} \left(i ^ {*} \notin I _ {t}\right) \\ \leq \Big ((1 + p _ {2}) \zeta_ {t} ^ {2} + (1 + 1 / p _ {2}) 4 \gamma_ {t} ^ {2} M \ell_ {f} ^ {4} \Big) \mathbb {P} (i ^ {*} \in I _ {t}) + \Big ((1 + c _ {3} \gamma_ {t}) \zeta_ {t} ^ {2} + 3 \ell_ {g, 1} \gamma_ {t} \delta_ {t} ^ {2} \Big) \mathbb {P} (i ^ {*} \notin I _ {t}) \\ \leq \left(1 + c _ {3} \gamma_ {t} + \frac {3}{n} p _ {2}\right) \zeta_ {t} ^ {2} + \left(1 + \frac {1}{p _ {2}}\right) 4 \gamma_ {t} ^ {2} M \ell_ {f} ^ {4} \frac {3}{n} + 3 \ell_ {g, 1} \gamma_ {t} \delta_ {t} ^ {2}. \tag {B.72} \\ \end{array}
$$
Based on linearity of expectation and applying (B.72) recursively yields
$$
\begin{array}{l} \mathbb {E} \left[ \zeta_ {t + 1} ^ {2} \right] \leq \sum_ {t ^ {\prime} = 0} ^ {t} \left((1 + \frac {1}{p _ {2}}) 4 \gamma^ {2} M \ell_ {f} ^ {4} \frac {3}{n} + 3 \ell_ {g, 1} \gamma \mathbb {E} \left[ \delta_ {t ^ {\prime}} ^ {2} \right]\right) \left(\prod_ {k = t ^ {\prime} + 1} ^ {t} \left(1 + c _ {3} \gamma + \frac {3}{n} p _ {2}\right)\right) \\ = \sum_ {t ^ {\prime} = 0} ^ {t} \left((1 + \frac {1}{p _ {2}}) 4 \gamma^ {2} M \ell_ {f} ^ {4} \frac {3}{n} + 3 \ell_ {g, 1} \gamma \mathbb {E} [ \delta_ {t ^ {\prime}} ^ {2} ]\right) \left(1 + c _ {3} \gamma + \frac {3}{n} p _ {2}\right) ^ {t - t ^ {\prime}} \\ \stackrel {(a)} {\leq} \sum_ {t ^ {\prime} = 0} ^ {t} \left((1 + \frac {8 T}{n}) 4 \gamma^ {2} M \ell_ {f} ^ {4} \frac {3}{n} + 3 \ell_ {g, 1} \gamma \mathbb {E} [ \delta_ {t ^ {\prime}} ^ {2} ]\right) \left(1 + \frac {1}{2 T}\right) ^ {t - t ^ {\prime}} \\ \stackrel {(b)} {\leq} \sum_ {t ^ {\prime} = 0} ^ {t} \left((1 + \frac {8 T}{n}) 4 \gamma^ {2} M \ell_ {f} ^ {4} \frac {3}{n} + 3 \ell_ {g, 1} \gamma \mathbb {E} [ \delta_ {t ^ {\prime}} ^ {2} ]\right) e ^ {\frac {1}{2}} \\ \stackrel {(c)} {\leq} 2 \gamma \sum_ {t ^ {\prime} = 0} ^ {t} \left(\left(1 + \frac {8 T}{n}\right) 4 \gamma M \ell_ {f} ^ {4} \frac {3}{n} + 3 \ell_ {g, 1} \mathbb {E} \left[ \delta_ {t ^ {\prime}} ^ {2} \right]\right) \tag {B.73} \\ \end{array}
$$
where  $(a)$  follows from choosing  $\gamma_t \leq \gamma \leq 1/(8c_3T)$ ,  $p_2 = n/(8T)$ ,  $(b)$  follows from  $t - t' \leq T$ , and  $(1 + \frac{a}{T})^T \leq e^a$ , and the inequality  $(c)$  follows from  $e^{\frac{1}{2}} < 2$ .
Note that  $\delta_0 = 0, \zeta_1 = 0$ . Applying (B.66) at  $t = 0$  gives
$$
\mathbb {E} [ \delta_ {1} ^ {2} ] \leq \frac {3}{n} (1 + \frac {1}{p}) 4 \alpha^ {2} \ell_ {f} ^ {2}
$$
which together with (B.72) gives
$$
\mathbb {E} [ \zeta_ {2} ^ {2} ] \leq 3 \ell_ {g, 1} \gamma_ {1} \delta_ {1} ^ {2} + (1 + \frac {1}{p _ {2}}) 4 \gamma_ {1} ^ {2} M \ell_ {f} ^ {4} \frac {3}{n}.
$$
Therefore, for  $0 \leq t \leq 1$ , it satisfies that
$$
\begin{array}{l} \mathbb {E} \left[ \delta_ {t} ^ {2} \right] \leq \left(\frac {3}{n} (1 + \frac {1}{p}) 4 \alpha^ {2} \ell_ {f} ^ {2} + 2 4 M \ell_ {f} ^ {4} c _ {2} \left(\frac {8 \gamma T}{n} + \gamma\right) \frac {\alpha}{n}\right) \underbrace {\left(\sum_ {t ^ {\prime} = 0} ^ {t - 1} \left(1 - \frac {1}{2} \alpha \mu + \frac {3 p}{n}\right) ^ {t - t ^ {\prime} - 1}\right)} _ {\beta_ {t}} \\ = \left(\frac {3}{n} \left(1 + \frac {1}{p}\right) 4 \alpha^ {2} \ell_ {f} ^ {2} + 2 4 M \ell_ {f} ^ {4} c _ {2} \left(\frac {8 \gamma T}{n} + \gamma\right) \frac {\alpha}{n}\right) \beta_ {t}. \tag {B.74} \\ \end{array}
$$
Next, we will prove by induction that (B.74) also holds for  $t > 1$ .
Assuming that (B.74) holds for all  $0 \leq t \leq k \leq T - 1$ , we apply (B.70) to the case where  $t = k$  to obtain
$$
\begin{array}{l} \mathbb {E} \left[ \delta_ {k + 1} ^ {2} \right] \leq \left(\eta_ {k} + \frac {3 p}{n}\right) \mathbb {E} \left[ \delta_ {k} ^ {2} \right] + \alpha_ {k} c _ {2} \left(1 + c _ {3} \gamma_ {k}\right) \mathbb {E} \left[ \zeta_ {k} ^ {2} \right] \mathbb {P} (i ^ {*} \notin I _ {t}) + \frac {3}{n} \left(1 + \frac {1}{p}\right) 4 \alpha_ {k} ^ {2} \ell_ {f} ^ {2} \\ \stackrel {(a)} {\leq} \left(\eta_ {k} + \frac {3 p}{n}\right) \mathbb {E} [ \delta_ {k} ^ {2} ] \\ + 2 \alpha_ {k} c _ {2} \gamma \left(\sum_ {t ^ {\prime} = 1} ^ {k} \left((1 + \frac {8 T}{n}) \frac {1 2 \gamma M \ell_ {f} ^ {4}}{n} + 3 \ell_ {g, 1} \mathbb {E} [ \delta_ {t ^ {\prime}} ^ {2} ]\right)\right) \mathbb {P} (i ^ {*} \notin I _ {t}) + \frac {3}{n} \left(1 + \frac {1}{p}\right) 4 \alpha_ {k} ^ {2} \ell_ {f} ^ {2} \\ \stackrel {(b)} {\leq} \underbrace {\left(\left(\eta_ {k} + \frac {3 p}{n}\right) \beta_ {k} + 1 + 6 \alpha_ {k} c _ {2} \ell_ {g , 1} \gamma \left(\sum_ {t ^ {\prime} = 1} ^ {k} \beta_ {t ^ {\prime}}\right) \mathbb {P} \left(i ^ {*} \notin I _ {t}\right)\right)} _ {J _ {1}} \\ \times \left(\frac {3}{n} \left(1 + \frac {1}{p}\right) 4 \alpha^ {2} \ell_ {f} ^ {2} + 2 4 M \ell_ {f} ^ {4} c _ {2} \left(\frac {8 \gamma T}{n} + \gamma\right) \frac {\alpha}{n}\right) \tag {B.75} \\ \end{array}
$$
where  $(a)$  follows from (B.73), and  $(b)$  follows from (B.74) for  $0\leq t\leq k$  and that  $\gamma k\leq \gamma T\leq 1$ . The coefficient  $J_{1}$  in (B.75) can be further bounded by
$$
\begin{array}{l} J _ {1} = \left(\eta_ {k} + \frac {3 p}{n}\right) \beta_ {k} + 1 + 6 \alpha_ {k} c _ {2} \ell_ {g, 1} \gamma \left(\sum_ {t ^ {\prime} = 1} ^ {k} c _ {t ^ {\prime}}\right) \mathbb {P} \left(i ^ {*} \notin I _ {t}\right) \\ \stackrel {(c)} {\leq} \left(\eta_ {k} + \frac {3 p}{n}\right) \beta_ {k} + 1 + 6 \alpha_ {k} c _ {2} \ell_ {g, 1} k \gamma \beta_ {k} \mathbb {P} \left(i ^ {*} \notin I _ {t}\right) \\ \stackrel {(d)} {\leq} \left(1 - \alpha_ {k} \left(\mu - \ell_ {F} c _ {1} - 3 c _ {2} \ell_ {g, 1} \gamma (1 + 2 k)\right) \mathbb {P} \left(i ^ {*} \notin I _ {t}\right) + \frac {3 p}{n}\right) \beta_ {k} + 1 \\ \stackrel {(e)} {\leq} \left(1 - \frac {1}{2} \alpha \mu + \frac {3 p}{n}\right) \beta_ {k} + 1 \tag {B.76} \\ \end{array}
$$
where  $(c)$  follows from  $\beta_{t} \leq \beta_{t+1}$ ,  $\gamma_{t} \leq \gamma$  for all  $t = 0, \dots, T$ ;  $(d)$  follows from the definition of  $\eta_{k}$ ;  $(e)$  is because  $\gamma \leq \mu^{2} / (120\ell_{F}^{2}\ell_{g,1}T)$ ,  $\alpha \leq 1 / (2\ell_{f,1}) \leq 1 / (2\mu)$  and choosing  $c_{1} = \mu / (4\ell_{F})$  leads to
$$
\begin{array}{l} \ell_ {F} c _ {1} + 3 c _ {2} \ell_ {g, 1} \gamma (1 + 2 k) \gamma \leq \ell_ {F} c _ {1} + 6 \left(\ell_ {F} c _ {1} ^ {- 1} + 2 \alpha \ell_ {F} ^ {2}\right) \ell_ {g, 1} (k + 1) \gamma \\ \leq \frac {1}{4} \mu + 6 (4 \mu^ {- 1} + 2 \alpha) \ell_ {F} ^ {2} \ell_ {g, 1} \frac {k + 1}{T} \frac {\mu^ {2}}{1 2 0 \ell_ {F} ^ {2} \ell_ {g , 1}} \leq \frac {1}{2} \mu . \\ \end{array}
$$
Combining (B.75) and (B.76) implies
$$
\begin{array}{l} \mathbb {E} \left[ \delta_ {k + 1} ^ {2} \right] \leq \left(\left(1 - \frac {1}{2} \alpha \mu + \frac {3 p}{n}\right) \beta_ {k} + 1\right) \left(\frac {3}{n} (1 + \frac {1}{p}) 4 \alpha^ {2} \ell_ {f} ^ {2} + 2 4 M \ell_ {f} ^ {4} c _ {2} \left(\frac {8 \gamma T}{n} + \gamma\right) \frac {\alpha}{n}\right) \\ = c _ {k + 1} \left(\frac {3}{n} \left(1 + \frac {1}{p}\right) 4 \alpha^ {2} \ell_ {f} ^ {2} + 2 4 M \ell_ {f} ^ {4} c _ {2} \left(\frac {8 \gamma T}{n} + \gamma\right) \frac {\alpha}{n}\right) \tag {B.77} \\ \end{array}
$$
where the equality follows by the definition of  $\beta_{t}$  given in (B.74). The above statements from (B.75)-(B.77) show that if (B.74) holds for all  $t$  such that  $0\leq t\leq k\leq T - 1$ , it also holds for  $t = k + 1$ .
Therefore, we can conclude that for  $T \geq 0$ , it follows
$$
\begin{array}{l} \mathbb {E} \left[ \delta_ {T} ^ {2} \right] \leq \beta_ {T} \left(\frac {3}{n} (1 + \frac {1}{p}) 4 \alpha^ {2} \ell_ {f} ^ {2} + 2 4 M \ell_ {f} ^ {4} c _ {2} \left(\frac {8 \gamma T}{n} + \gamma\right) \frac {\alpha}{n}\right) \\ = \left(\frac {3}{n} \left(1 + \frac {1}{p}\right) 4 \alpha^ {2} \ell_ {f} ^ {2} + 2 4 M \ell_ {f} ^ {4} c _ {2} \left(\frac {8 \gamma T}{n} + \gamma\right) \frac {\alpha}{n}\right) \left(\sum_ {k = 0} ^ {T - 1} \left(1 - \frac {1}{2} \alpha \mu + \frac {3 p}{n}\right) ^ {T - k - 1}\right) \\ \end{array}
$$
$$
= \left(\frac {3}{n} \left(1 + \frac {1 2}{\alpha \mu n}\right) 4 \alpha^ {2} \ell_ {f} ^ {2} + 2 4 M \ell_ {f} ^ {4} c _ {2} \left(\frac {8 \gamma T}{n} + \gamma\right) \frac {\alpha}{n}\right) \left(\frac {1}{4} \alpha \mu\right) ^ {- 1} \left(1 - \left(1 - \frac {1}{4} \alpha \mu\right) ^ {T}\right) \tag {B.78}
$$
where the last equality follows from taking  $p = \alpha \mu n / 12$ , and compute the sum of geometric series. By plugging in  $c_{1} = \mu /(4\ell_{F})$ ,  $c_{2} = \ell_{F}c_{1}^{-1} + 2\alpha \ell_{F}^{2}$ ,  $c_{3} = 3\ell_{F}^{2} + 2\ell_{g,1}$ , we have that