SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {R} _ {t} \left(\theta_ {t} ^ {*}\right) \geq \mathcal {R} _ {t} \left(\theta_ {t - 1}\right) + \left\langle \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right), \theta_ {t} ^ {*} - \theta_ {t - 1} \right\rangle + \frac {m}{2} \left\| \theta_ {t} ^ {*} - \theta_ {t - 1} \right\| _ {2} ^ {2}. \tag {4}
$$
Further since  $\theta_t^* \coloneqq \arg \min_{\theta} \mathcal{R}_t(\theta)$ , we have that
$$
\mathcal {R} _ {t} (\theta_ {t - 1}) - \mathcal {R} _ {t} (\theta_ {t} ^ {*}) \geq \frac {m}{2} \| \theta_ {t - 1} - \theta_ {t} ^ {*} \| _ {2} ^ {2}.
$$
Putting these two together, we see that
$$
\left\langle \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right), \theta_ {t - 1} - \theta_ {t} ^ {*} \right\rangle \geq m \| \theta_ {t - 1} - \theta_ {t} ^ {*} \| _ {2} ^ {2}. \tag {5}
$$
Also, We further use the following lemma.
Lemma 17.1 (Lemma 3.11 from [9]). Let  $f: \mathbb{R}^d \to \mathbb{R}$  be a  $M$  smooth and  $m$  strongly convex function. Then for all  $x, y \in \mathbb{R}^d$ ,
$$
\langle \nabla f (x) - \nabla f (y), x - y \rangle \geq \frac {m M}{M + m} \| x - y \| _ {2} ^ {2} + \frac {1}{M + m} \| \nabla f (x) - \nabla f (y) \| _ {2} ^ {2}.
$$
By substituting  $x = \theta_{t-1}$ ,  $y = \theta_t^*$  and  $f(\cdot) = \mathcal{R}_t(\cdot)$  and by leveraging the fact that  $\nabla \mathcal{R}_t(\theta_t^*) = 0$ , we get the inequality that
$$
\langle \nabla \mathcal {R} _ {t} (\theta_ {t - 1}), \theta_ {t - 1} - \theta_ {t} ^ {*} \rangle \geq \frac {m M}{m + M} \| \theta_ {t - 1} - \theta_ {t} ^ {*} \| _ {2} ^ {2} + \frac {1}{M + m} \| \nabla \mathcal {R} _ {t} (\theta_ {t - 1}) \| _ {2} ^ {2}.
$$
Re-arranging, we see that
$$
\left\| \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right) \right\| _ {2} ^ {2} \leq (M + m) \left\langle \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right), \theta_ {t - 1} - \theta_ {t} ^ {*} \right\rangle - m M \| \theta_ {t - 1} - \theta_ {t} ^ {*} \| _ {2} ^ {2}. \tag {6}
$$
Further, using the Cauchy-schwartz inequality that  $\langle \nabla \mathcal{R}_t(\theta_{t - 1}),\theta_{t - 1} - \theta_t^*\rangle \leq \| \nabla \mathcal{R}_t(\theta_{t - 1})\| \| \theta_{t - 1} - \theta_t^*\| \leq \mathcal{D}\| \nabla \mathcal{R}_t(\theta_{t - 1})\|$ , we get from the preceding inequality that
$$
\left\| \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right) \right\| \leq (M + m) \mathcal {D}. \tag {7}
$$
18 Theorem statement when the gradients are sub-gaussian with known upper-bound
Theorem 18.1 (Formal version of Theorem 4.3). Suppose Assumption 4.2 holds. For every  $\delta \in (0,1)$ , if Algorithm 1 is run with clipping value  $\lambda \geq G + \sqrt{Tr(\Sigma)} + C\left(\sqrt{\nu_{max}(\Sigma)\ln\left(\frac{2T}{\delta}\right)}\right)$ , where  $C$  is given in Proposition 19.1 and step-size  $\eta > 0$ , then with probability at-least  $1 - \delta$ ,
$$
\begin{array}{l} R e g _ {T} \leq \frac {2 (l _ {1} + \Phi_ {T})}{\eta m} + T \left(C \sqrt {\nu_ {m a x} (\Sigma) \ln \left(\frac {6 T}{\delta}\right)} + \sqrt {T r (\Sigma)}\right) \sqrt {\frac {2 \eta}{m}} \\ + \Lambda_ {T} \left(\frac {2 \lambda}{m} + \sqrt {\frac {4 \lambda \mathcal {D}}{\eta}}\right) + 2 \sqrt {\frac {T \Phi_ {T} \left(C \sqrt {\nu_ {m a x} (\Sigma) \ln \left(\frac {2 T}{\delta}\right)} + \sqrt {T r (\Sigma)}\right)}{\eta m ^ {2}}} \tag {8} \\ \end{array}
$$
Corollary 18.2. Suppose Assumption 4.2 holds. For every  $\delta \in (0,1)$ , if Algorithm 1 is run with clipping value  $\lambda \geq G + \sqrt{Tr(\Sigma)} + C\left(\sqrt{\nu_{max}(\Sigma)\ln\left(\frac{2T}{\delta}\right)}\right)$ , where  $C$  is given in Proposition 19.1 and step-size  $\eta = \frac{1}{mT^{\alpha}}$  for  $\alpha \in [0,1)$ , then with probability at-least  $1 - \delta$ ,
$$
\begin{array}{l} R e g _ {T} \leq \underbrace {\mathcal {O} \left(\Phi_ {T} T ^ {\alpha} + \sqrt {\frac {T ^ {1 + \alpha} \Phi_ {T} \left(C \sqrt {\nu_ {m a x} (\Sigma) \ln \left(\frac {2 T}{\delta}\right)} + \sqrt {T r (\Sigma)}\right)}{m}}\right)} _ {R e g r e t d u e t o d i s t r i b u t i o n s h i f t} + \\ \underbrace {\mathcal {O} \left(\frac {T ^ {1 - \frac {\alpha}{2}}}{m} \left(\sqrt {\nu_ {m a x} (\Sigma) \ln \left(\frac {6 T}{\delta}\right)} + \sqrt {T r (\Sigma)}\right)\right)} _ {\text {R e g r e t d u e t o s t a n d a r d s t a t i s t i c a l e r r o r}} + \underbrace {\mathcal {O} \left(\Lambda_ {T} T ^ {\frac {\alpha}{2}} \sqrt {m \lambda \mathcal {D}}\right)} _ {\text {R e g r e t d u e t o a d v e r s a r i a l c o r r u p t i o n s}}. \tag {9} \\ \end{array}
$$
The above theorem immediately yields the following corollary by substituting the learning rate as  $\eta = T^{-2/3}$ .
Corollary 18.3. Suppose Assumption 4.2 holds. For every  $\delta \in (0,1)$ , if Algorithm 1 is run with clipping value  $\lambda \geq G + \sqrt{Tr(\Sigma)} + C\left(\sqrt{\nu_{max}(\Sigma)\ln\left(\frac{2T}{\delta}\right)}\right)$ , where  $C$  is given in Proposition 19.1 and step-size  $\eta = \frac{T^{-2/3}}{m}$ , then with probability at-least  $1 - \delta$ ,
$$
\begin{array}{l} R e g _ {T} \leq \underbrace {\mathcal {O} \left(\Phi_ {T} T ^ {2 / 3} + \sqrt {\frac {T ^ {5 / 3} \Phi_ {T} \left(C \sqrt {\nu_ {m a x} (\Sigma) \ln \left(\frac {2 T}{\delta}\right)} + \sqrt {T r (\Sigma)}\right)}{m}}\right)} _ {R e g r e t d u e t o d i s t r i b u t i o n s h i f t} + \\ \underbrace {\mathcal {O} \left(\frac {T ^ {2 / 3}}{m} \left(\sqrt {\nu_ {m a x} (\Sigma) \ln \left(\frac {6 T}{\delta}\right)} + \sqrt {T r (\Sigma)}\right)\right)} _ {\text {R e g r e t d u e t o s t a n d a r d s t a t i s t i c a l e r r o r}} + \underbrace {\mathcal {O} \left(\Lambda_ {T} T ^ {1 / 3} \sqrt {m \lambda \mathcal {D}}\right)} _ {\text {R e g r e t d u e t o a d v e r s a r i a l c o r r u p t i o n s}}. \tag {10} \\ \end{array}
$$
19 Proof of Theorem 18.1
The proof in this section is based on the following result on high-dimensional sub-gaussian random vectors.
Proposition 19.1 (Exercise 6.3.5 [57]). There exists an absolute constant  $C > 80$ , such that for every sub-gaussian random-vector  $Z$  with 0 mean random-vector and covariance matrix  $\Sigma$  and every  $\delta \in (0,1)$ , the following
$$
\mathbb {P} \left[ \left| \| Z \| _ {2} - \mathbb {E} [ \| Z \| _ {2} ] \right| \geq C \sqrt {\nu_ {m a x} (\Sigma) \ln \left(\frac {2}{\delta}\right)} \right] \leq \delta ,
$$
holds, where  $\nu_{\max}(\Sigma)$  is the largest eigen-value of the upper-bound covariance matrix  $\Sigma$ .
19.0.1 Notations
We follow the same proof architecture as that of [55]. We define three sequences of random variables  $(\psi_t)_{t \geq 1}$  and  $(\bar{\psi})_{t \geq 1}$  as follows.
$$
\psi_ {t} := \operatorname {c l i p} \left(\nabla \mathcal {L} \left(Z _ {t}, \theta_ {t - 1}\right), \lambda\right) - \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right),
$$
$$
\bar {\psi} _ {t} := \operatorname {c l i p} \left(\nabla \mathcal {L} \left(X _ {t}, \theta_ {t - 1}\right), \lambda\right) - \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right),
$$
$$
\widetilde {\psi} _ {t} := \nabla \mathcal {L} (Z _ {t}, \theta_ {t - 1}) - \nabla \mathcal {R} _ {t} (\theta_ {t - 1}).
$$
These are random vectors since  $Z_{t} \sim \mathbb{P}_{t}$  and both  $C_t$  and  $\theta_t$  are measurable with respect to the sigma algebra generated by  $\sigma (Z_1, \dots, Z_t, C_1, \dots, C_{t-1})$ . Clearly, for all times  $t \geq 1$ , on the event that  $C_t = 0$ ,  $\psi_t = \bar{\psi}_t$  holds almost-surely. Furthermore, from triangle inequality almost-surely for all time  $t \geq 1$ , we have
$$
\left\| \psi_ {t} \right\| _ {2} ^ {2} \leq \left\| \bar {\psi} _ {t} \right\| _ {2} ^ {2} + 2 \lambda^ {2} \mathbf {1} _ {C _ {t} \neq 0}. \tag {11}
$$
Recall from Assumption 2.5 that for every  $t$  and  $\theta \in \Theta$ , the covariance matrix of the random vector  $\nabla \mathcal{L}(Z, \theta) - \nabla \mathcal{R}_t(\theta)$  is bounded from above in the positive semi-definite sense by  $\Sigma$ . Denote by  $\mathrm{Trace}(\Sigma)$  and  $\nu_{\max}(\Sigma)$  as the trace and the highest eigen value respectively of the matrix  $\Sigma$ . Assumption 4.2 implies the following two lemmas.
Denote by the event  $\mathcal{E}_{\mathrm{no -clip}}$  as
$$
\begin{array}{l} \mathcal {E} _ {\text {n o - c l i p}} = \bigcap_ {t = 1} ^ {T} \left\{\nabla \mathcal {L} \left(Z _ {t}, \theta_ {t - 1}\right) = \operatorname {c l i p} \left(\nabla \mathcal {L} \left(Z _ {t}, \theta_ {t - 1}\right) \right. \right\}, \\ = \bigcap_ {t = 1} ^ {T} \left\{\psi_ {t} = \widetilde {\psi} _ {t} \right\}. \tag {12} \\ \end{array}
$$
19.0.2 Supporting Lemmas based on sub-gaussian concentrations
The main result of this section is stated at the end in Corollary 19.5. In order to state them, we build a series of useful bounds along the way. The following lemma bounds the estimation error of the true gradient using the sample  $Z_{t}$  at each time  $t$ .
Lemma 19.2.
$$
\sup  _ {\widetilde {\theta} _ {1}, \dots , \widetilde {\theta} _ {T} \in \Theta} \mathbb {P} _ {Z _ {1}, \dots , Z _ {T}} \left[ \sup  _ {t \in [ T ]} \| \nabla \mathcal {L} (Z _ {t}, \widetilde {\theta} _ {t}) - \nabla \mathcal {R} _ {t} (\widetilde {\theta} _ {t}) \| > C \sqrt {\nu_ {\max } (\Sigma) \ln \left(\frac {2 T}{\delta}\right)} + \sqrt {T r (\Sigma)} \right] \leq \delta .
$$
In words, the above lemma states that for any sequence of  $\widetilde{\theta}_1,\dots ,\widetilde{\theta}_T$ , the probability that the norm of the difference between  $\nabla \mathcal{L}(Z_t,\widetilde{\theta}_t)$  and its expectation  $\nabla \mathcal{R}_t(\widetilde{\theta}_t)$  where  $Z_{t}\sim \mathbb{P}_{t}$  exceeds  $C\sqrt{\nu_{max}(\Sigma)\ln\left(\frac{2T}{\delta}\right)}$  is bounded above by  $\delta$ .
Proof. Fix any deterministic  $\widetilde{\theta}_1, \dots, \widetilde{\theta}_T \in \Theta$ . Assumption 4.2 yields that for all time  $t$ ,  $\nabla \mathcal{L}(Z_t, \widetilde{\theta}_t) - \nabla \mathcal{R}_t(\widetilde{\theta}_t)$  is a 0 mean sub-gaussian random vector with covariance matrix upper-bounded in the positive definite sense by  $\Sigma$ . Observe from Jensen's inequality that for all  $t$  and  $\theta \in \theta$
$$
\mathbb {E} \left[ \| \nabla \mathcal {L} \left(Z _ {t}, \theta\right) - \nabla \mathcal {R} _ {t} (\theta) \| \right] \leq \sqrt {\mathbb {E} \left[ \| \nabla \mathcal {L} \left(Z _ {t} , \theta\right) - \nabla \mathcal {R} _ {t} (\theta) \| ^ {2} \right]} \leq \sqrt {\operatorname {T r} (\Sigma)}, \tag {13}
$$
Now, Hoeffding's inequality in Proposition 19.1 gives that
$$
\begin{array}{l} \| \nabla \mathcal {L} (Z _ {t}, \widetilde {\theta} _ {t}) - \nabla \mathcal {R} _ {t} (\widetilde {\theta} _ {t}) \| \leq C \sqrt {\nu_ {m a x} (\Sigma) \ln \left(\frac {2 T}{\delta}\right)} + \mathbb {E} [ \| \nabla \mathcal {L} (Z _ {t}, \widetilde {\theta} _ {t}) - \nabla \mathcal {R} _ {t} (\widetilde {\theta} _ {t}) \| ], \\ \leq C \sqrt {\nu_ {m a x} (\Sigma) \ln \left(\frac {2 T}{\delta}\right)} + \sqrt {\operatorname {T r} (\Sigma)} \\ \end{array}
$$
holds with probability at-least  $1 - \frac{\delta}{T}$ . Now, taking an union bound over  $t = \{1,\dots ,T\}$  gives that
$$
\mathbb {P} _ {Z _ {1}, \dots , Z _ {T}} \left[ \sup  _ {t \in [ T ]} \| \nabla \mathcal {L} (Z _ {t}, \widetilde {\theta} _ {t}) - \nabla \mathcal {R} _ {t} (\widetilde {\theta} _ {t}) \| > C \sqrt {\nu_ {\max } (\Sigma) \ln \left(\frac {2 T}{\delta}\right)} + \sqrt {\operatorname {T r a c e} (\Sigma)} \right] \leq \delta .
$$
Now, since  $\widetilde{\theta}_1,\dots ,\widetilde{\theta}_T\in \Theta$  was arbitrary, we can take a supremum on both sides and that concludes the proof.
Corollary 19.3. With probability at-least  $1 - \delta$ , for all times  $t \in \{1, \dots, T\}$ ,
$$
\| \widetilde {\psi} _ {t} \| \leq C \sqrt {\nu_ {m a x} (\Sigma) \ln \left(\frac {2 T}{\delta}\right)} + \sqrt {T r (\Sigma)}.
$$
Proof. The proof follows from applying the Lemma 19.2 to the set of random locations  $\theta_{1},\dots ,\theta_{T}$
Lemma 19.4. If Algorithm 1 is run with clipping value  $\lambda \geq G + \sqrt{Tr(\Sigma)} + C\left(\sqrt{\nu_{max}(\Sigma)\ln\left(\frac{2T}{\delta}\right)}\right)$ , where  $C$  is given in Proposition 19.1, then with probability at-least  $1 - \delta$  gradient is never clipped for any un-corrupted sample, i.e., with probability at-least  $1 - \delta$ , event  $\mathcal{E}_{no - clip}$  holds.
Proof of Lemma 19.4. From the definition, we know that the gradients are never clipped for the un-corrupted samples if  $\| \nabla \mathcal{L}(\theta_{t - 1},Z_t)\| \leq \lambda$  holds for all  $t$ .
From Corollary 19.3, we know that with probability at-least  $1 - \delta$ , for all  $t \in \{1, \dots, T\}$ , we have
$$
\left\| \nabla \mathcal {L} \left(Z _ {t}, \theta_ {t - 1}\right) - \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right) \right\| \leq C \sqrt {\nu_ {\max } (\Sigma) \ln \left(\frac {2 T}{\delta}\right)} + \sqrt {\operatorname {T r} (\Sigma)}. \tag {14}
$$
From the triangle inequality we know that
$$
\begin{array}{l} \left\| \nabla \mathcal {L} \left(Z, \theta_ {t - 1}\right) \right\| \leq \left\| \nabla \mathcal {L} \left(Z, \theta_ {t - 1}\right) - \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right) \right\| + \left\| \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right) \right\|, \\ \leq \| \nabla \mathcal {L} (Z, \theta_ {t - 1}) - \nabla \mathcal {R} _ {t} (\theta_ {t - 1}) \| + \sup  _ {\theta \in \Theta} \| \nabla \mathcal {R} _ {t} (\theta) \|, \\ = \left\| \nabla \mathcal {L} \left(Z, \theta_ {t - 1}\right) - \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right) \right\| + G. \tag {15} \\ \end{array}
$$
Substituting the bound from Equation (14) into (15), we get that for every  $t \in [T]$ ,
$$
\left\| \nabla \mathcal {L} \left(Z, \theta_ {t - 1}\right) \right\| \leq G + \sqrt {\operatorname {T r} (\Sigma)} + C \left(\sqrt {\nu_ {\max } (\Sigma) \ln \left(\frac {2 T}{\delta}\right)}\right) \leq \lambda , \tag {16}
$$
This immediately yields the following corollary.
Corollary 19.5. If Algorithm 1 is run with clipping value  $\lambda \geq G + \sqrt{Tr(\Sigma)} + C\left(\sqrt{\nu_{max}(\Sigma)\ln\left(\frac{2T}{\delta}\right)}\right)$ , where  $C$  is given in Proposition 19.1, then with probability at-least  $1 - 2\delta$ , for all times  $t \in [T]$
$$
\| \psi_ {t} \| \leq C \sqrt {\nu_ {m a x} (\Sigma) \ln \left(\frac {2 T}{\delta}\right)} + \sqrt {T r (\Sigma)}.
$$
Proof. On the event that both Corollary 19.3 and Lemma 19.4 holds, we have the result. Both those events fail to hold with probability at-most  $2\delta$ .
Denote by the event
$$
\boldsymbol {\mathcal {E}} _ {\psi} = \bigcap_ {t = 1} ^ {T} \left\{\| \psi_ {t} \| \leq C \sqrt {\nu_ {\max } (\Sigma) \ln \left(\frac {2 T}{\delta}\right)} + \sqrt {\operatorname {T r} (\Sigma)} \right\}. \tag {17}
$$
Corollary 19.5 states that  $\mathbb{P}[\pmb{\mathcal{E}}_{\psi}] \geq 1 - 2\delta$ .
19.1 Lemmas based on expanding the gradient descent recursion
Lemma 19.6. Under event  $\mathcal{E}_{\psi}$ , for every time  $t \in [T]$
$$
\begin{array}{l} \left\| \theta_ {t} ^ {*} - \theta_ {t} \right\| _ {2} ^ {2} \leq (1 - \eta m) ^ {t - 1} \left\| \theta_ {1} - \theta_ {1} ^ {*} \right\| _ {2} ^ {2} + \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {t - s} \left\| \theta_ {s} ^ {*} - \theta_ {s + 1} ^ {*} \right\| _ {2} ^ {2} + \frac {2 \eta B ^ {2}}{m} \\ - 2 \eta \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {t - s - 1} \left\langle \theta_ {s} - \theta_ {s} ^ {*}, \psi_ {s + 1} \right\rangle + 2 \eta B \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {t - s - 1} \| \theta_ {s} ^ {*} - \theta_ {s + 1} ^ {*} \| \\ + 4 \eta^ {2} \lambda^ {2} \sum_ {s = 2} ^ {t - 1} (1 - \eta m) ^ {t - s} \mathbf {1} _ {C _ {s} \neq 0} + 4 \eta \lambda \mathcal {D} \sum_ {s = 2} ^ {t - 1} (1 - \eta m) ^ {t - s} \mathbf {1} _ {C _ {s} \neq 0},, \tag {18} \\ \end{array}
$$
where  $B = C\sqrt{\nu_{max}(\Sigma)\ln\left(\frac{2T}{\delta}\right)} + \sqrt{Tr(\Sigma)}$  holds.
19.1.1 Proof of Lemma 19.6
Proof. Consider any time  $t$ . We have
$$
\begin{array}{l} \left\| \theta_ {t} - \theta_ {t} ^ {*} \right\| _ {2} ^ {2} = \left\| \mathcal {P} _ {\Theta} \left(\theta_ {t - 1} - \eta \operatorname {c l i p} \left(\mathcal {L} \left(X _ {t}, \theta_ {t - 1}\right), \lambda\right)\right) - \theta_ {t} ^ {*} \right\| _ {2} ^ {2}, (19) \\ \stackrel {(a)} {\leq} \left\| \theta_ {t - 1} - \eta \operatorname {c l i p} \left(\mathcal {L} \left(X _ {t}, \theta_ {t - 1}\right), \lambda\right) - \theta_ {t} ^ {*} \right\| _ {2} ^ {2}, (20) \\ = \| \theta_ {t - 1} - \eta (\bar {\psi} _ {t} + \nabla \mathcal {R} _ {t} (\theta_ {t - 1})) - \theta_ {t} ^ {*} \| _ {2} ^ {2}, \\ = \| \theta_ {t - 1} - \theta_ {t} ^ {*} \| _ {2} ^ {2} + \eta^ {2} \| \bar {\psi} _ {t} + \nabla \mathcal {R} _ {t} (\theta_ {t - 1}) \| _ {2} ^ {2} - 2 \eta \langle \theta_ {t - 1} - \theta_ {t} ^ {*}, \bar {\psi} _ {t} + \nabla \mathcal {R} _ {t} (\theta_ {t - 1}) \rangle , \\ \stackrel {(b)} {\leq} \| \theta_ {t - 1} - \theta_ {t} ^ {*} \| _ {2} ^ {2} + 2 \eta^ {2} \| \bar {\psi} _ {t} \| _ {2} ^ {2} + 2 \eta^ {2} \| \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right) \| _ {2} ^ {2} - 2 \eta \langle \theta_ {t - 1} - \theta_ {t} ^ {*}, \bar {\psi} _ {t} + \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right) \rangle , (21) \\ \end{array}
$$
Step  $(a)$  follows since  $\Theta$  is a convex set,  $\| \mathcal{P}_{\Theta}(\theta_t) - \theta_t^*\| \leq \| \theta_t - \theta_t^*\|$ , since  $\theta_t^* \in \Theta$ . In step  $(b)$ , we use the fact that  $\| a + b\|_2^2 \leq 2\| a\|_2^2 + 2\| b\|_2^2$ , for all  $a, b \in \mathbb{R}^d$ . Substituting Equation (6) into (21), we get that
$$
\begin{array}{l} \left\| \theta_ {t} ^ {*} - \theta_ {t} \right\| _ {2} ^ {2} \leq \left\| \theta_ {t - 1} - \theta_ {t} ^ {*} \right\| _ {2} ^ {2} + 2 \eta^ {2} \| \bar {\psi} _ {t} \| _ {2} ^ {2} - 2 \eta \langle \theta_ {t - 1} - \theta_ {t} ^ {*}, \bar {\psi} _ {t} \rangle \\ + 2 \eta^ {2} \left(\left(M + m\right) \left\langle \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right), \theta_ {t - 1} - \theta_ {t} ^ {*} \right\rangle - m M \left\| \theta_ {t - 1} - \theta_ {t} ^ {*} \right\| _ {2} ^ {2}\right) - 2 \eta \left\langle \nabla \mathcal {R} _ {t} \left(\theta_ {t - 1}\right), \theta_ {t - 1} - \theta_ {t} ^ {*} \right\rangle . \\ \end{array}
$$
Re-arranging the equation above yields
$$
\begin{array}{l} \left\| \theta_ {t} ^ {*} - \theta_ {t} \right\| _ {2} ^ {2} \leq (1 - 2 \eta^ {2} m M) \left\| \theta_ {t - 1} - \theta_ {t} ^ {*} \right\| _ {2} ^ {2} + 2 \eta^ {2} \left\| \bar {\psi} _ {t} \right\| _ {2} ^ {2} - 2 \eta \langle \theta_ {t - 1} - \theta_ {t} ^ {*}, \bar {\psi} _ {t} \rangle \\ - 2 \eta (1 - \eta ((M + m)) \langle \nabla \mathcal {R} _ {t} (\theta_ {t - 1}), \theta_ {t - 1} - \theta_ {t} ^ {*} \rangle . \\ \end{array}
$$
Further substituting Equation (5) into the display above yields that
$$
\begin{array}{l} \left\| \theta_ {t} ^ {*} - \theta_ {t} \right\| _ {2} ^ {2} \leq \left(1 - 2 \eta m + 2 \eta^ {2} m ^ {2}\right) \left\| \theta_ {t - 1} - \theta_ {t} ^ {*} \right\| _ {2} ^ {2} + 2 \eta^ {2} \left\| \bar {\psi} _ {t} \right\| _ {2} ^ {2} - 2 \eta \left\langle \theta_ {t - 1} - \theta_ {t} ^ {*}, \bar {\psi} _ {t} \right\rangle , \\ \leq (1 - \eta m) \| \theta_ {t - 1} - \theta_ {t} ^ {*} \| _ {2} ^ {2} + 2 \eta^ {2} \| \bar {\psi} _ {t} \| _ {2} ^ {2} - 2 \eta \langle \theta_ {t - 1} - \theta_ {t} ^ {*}, \bar {\psi} _ {t} \rangle , \\ \end{array}
$$
where the inequality comes from the fact that if  $\eta m < 1 \Rightarrow 2\eta m - 2\eta^2 m^2 > \eta m$ . We simplify the display above using the inequality in Equation (45) as,
$$
\begin{array}{l} \left\| \theta_ {t} ^ {*} - \theta_ {t} \right\| _ {2} ^ {2} \leq (1 - \eta m) \left\| \theta_ {t - 1} - \theta_ {t - 1} ^ {*} \right\| _ {2} ^ {2} + (1 - \eta m) \left\| \theta_ {t - 1} ^ {*} - \theta_ {t} ^ {*} \right\| _ {2} ^ {2} \\ + 2 \eta^ {2} \| \psi_ {t} \| _ {2} ^ {2} + 4 \eta^ {2} \lambda^ {2} \mathbf {1} _ {C _ {t} \neq 0} - 2 \eta \left\langle \theta_ {t - 1} - \theta_ {t} ^ {*}, \psi_ {t} \right\rangle + 2 \eta \left\langle \theta_ {t - 1} - \theta_ {t} ^ {*}, \psi_ {t} - \bar {\psi} _ {t} \right\rangle . \tag {22} \\ \end{array}
$$
Using the Cauchy-Schartz inequality that  $\langle \theta_{t - 1} - \theta_t^*,\psi_t - \bar{\psi}_t\rangle \leq \| \theta_{t - 1} - \theta_t^*\| \| \bar{\psi}_t - \psi_t\| \leq 2\lambda \| \theta_{t - 1} - \theta_t^*\| \mathbf{1}_{C_t\neq 0}$ , where the last inequality comes from the fact that for all time  $t$ $\| \bar{\psi}_t - \psi_t\| \leq 2\lambda \mathbf{1}_{C_t\neq 0}$  almost-surely. Plugging this into Equation (22) yields
$$
\begin{array}{l} \left\| \theta_ {t} ^ {*} - \theta_ {t} \right\| _ {2} ^ {2} \leq (1 - \eta m) \left\| \theta_ {t - 1} - \theta_ {t - 1} ^ {*} \right\| _ {2} ^ {2} + (1 - \eta m) \left\| \theta_ {t - 1} ^ {*} - \theta_ {t} ^ {*} \right\| _ {2} ^ {2} \\ + 2 \eta^ {2} \| \psi_ {t} \| _ {2} ^ {2} + 4 \eta^ {2} \lambda^ {2} \mathbf {1} _ {C _ {t} \neq 0} - 2 \eta \left\langle \theta_ {t - 1} - \theta_ {t} ^ {*}, \psi_ {t} \right\rangle + 4 \eta \lambda \| \theta_ {t - 1} - \theta_ {t} ^ {*} \| \mathbf {1} _ {C _ {t} \neq 0}. \tag {23} \\ \end{array}
$$
Using the fact that the diameter of the set  $\Theta$  is  $\mathcal{D}_{\Theta}$  now yields that
$$
\begin{array}{l} \left\| \theta_ {t} ^ {*} - \theta_ {t} \right\| _ {2} ^ {2} \leq (1 - \eta m) \left\| \theta_ {t - 1} - \theta_ {t - 1} ^ {*} \right\| _ {2} ^ {2} + (1 - \eta m) \left\| \theta_ {t - 1} ^ {*} - \theta_ {t} ^ {*} \right\| _ {2} ^ {2} \\ + 2 \eta^ {2} \| \psi_ {t} \| _ {2} ^ {2} + 4 \eta^ {2} \lambda^ {2} \mathbf {1} _ {C _ {t} \neq 0} - 2 \eta \left\langle \theta_ {t - 1} - \theta_ {t} ^ {*}, \psi_ {t} \right\rangle + 4 \eta \lambda \mathcal {D} \mathbf {1} _ {C _ {t} \neq 0}. \\ \end{array}
$$
Unrolling the recursion yields,
$$
\| \theta_ {t} ^ {*} - \theta_ {t} \| _ {2} ^ {2} \leq (1 - \eta m) ^ {t - 1} \| \theta_ {1} - \theta_ {1} ^ {*} \| _ {2} ^ {2} + \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s} \| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \| _ {2} ^ {2} + 2 \eta^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \| \psi_ {t - s + 1} \| _ {2} ^ {2} +
$$
$$
4 \eta^ {2} \lambda^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \mathbf {1} _ {C _ {t - s + 1} \neq 0} - 2 \eta \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \langle \theta_ {t - s} - \theta_ {t - s + 1} ^ {*}, \psi_ {t - s + 1} \rangle + 4 \eta \lambda \mathcal {D} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \mathbf {1} _ {C _ {t - s + 1} \neq 0}.
$$
Using the fact that  $\sum_{s=1}^{t-1}(1 - \eta m)^{s-1} \leq \frac{1}{\eta m}$ , we get that
$$
\begin{array}{l} \left\| \theta_ {t} ^ {*} - \theta_ {t} \right\| _ {2} ^ {2} \leq (1 - \eta m) ^ {t - 1} \left\| \theta_ {1} - \theta_ {1} ^ {*} \right\| _ {2} ^ {2} + \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s} \left\| \theta_ {t - s} ^ {*} - \theta_ {t - s + 1} ^ {*} \right\| _ {2} ^ {2} \\ + 2 \eta^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \underbrace {\| \psi_ {t - s + 1} \| _ {2} ^ {2}} _ {\text {T e r m I}} \underbrace {- 2 \eta \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \langle \theta_ {t - s} - \theta_ {t - s + 1} ^ {*} , \psi_ {t - s + 1} \rangle} _ {\text {T e r m I I}} \\ + 4 \eta^ {2} \lambda^ {2} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \mathbf {1} _ {C _ {t - s + 1} \neq 0} + 4 \eta \lambda \mathcal {D} \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {s - 1} \mathbf {1} _ {C _ {t - s + 1} \neq 0}. \tag {24} \\ \end{array}
$$
Changing the variables in the summation, the above inequality can be written as
$$
\begin{array}{l} \left\| \theta_ {t} ^ {*} - \theta_ {t} \right\| _ {2} ^ {2} \leq (1 - \eta m) ^ {t - 1} \left\| \theta_ {1} - \theta_ {1} ^ {*} \right\| _ {2} ^ {2} + \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {t - s} \left\| \theta_ {s} ^ {*} - \theta_ {s + 1} ^ {*} \right\| _ {2} ^ {2} \\ + 2 \eta^ {2} \sum_ {s = 2} ^ {t - 1} (1 - \eta m) ^ {t - s} \underbrace {\| \psi_ {s} \| _ {2} ^ {2}} _ {\text {T e r m I}} - \underbrace {- 2 \eta \sum_ {s = 1} ^ {t - 1} (1 - \eta m) ^ {t - s - 1} \langle \theta_ {s} - \theta_ {s + 1} ^ {*} , \psi_ {s + 1} \rangle} _ {\text {T e r m I I}} \\ + 4 \eta^ {2} \lambda^ {2} \sum_ {s = 2} ^ {t - 1} (1 - \eta m) ^ {t - s} \mathbf {1} _ {C _ {s} \neq 0} + 4 \eta \lambda \mathcal {D} \sum_ {s = 2} ^ {t - 1} (1 - \eta m) ^ {t - s} \mathbf {1} _ {C _ {s} \neq 0}. \tag {25} \\ \end{array}
$$