SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \langle \mathbf {w} _ {y _ {i}, r} ^ {(\widetilde {t}, 0)}, \boldsymbol {\xi} _ {i} \rangle = \langle \mathbf {w} _ {y _ {i}, r} ^ {(\widetilde {t} - 1, 0)}, \boldsymbol {\xi} _ {i} \rangle - \frac {\eta (P - 1)}{B m} \cdot \underbrace {\ell_ {i} ^ {(\widetilde {t} - 1 , b _ {i} ^ {(t - 1)})} \cdot \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2}} _ {\mathrm {I} _ {4}} \\ - \frac {\eta (P - 1)}{B m} \cdot \underbrace {\sum_ {b ^ {\prime} = 0} ^ {H - 1} \sum_ {i ^ {\prime} \in \mathcal {I} _ {\tilde {t} , b ^ {\prime}}} \ell_ {i ^ {\prime}} ^ {(\widetilde {t} - 1 , b ^ {\prime})} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {y _ {i} , r} ^ {(\widetilde {t} - 1 , b ^ {\prime})} , \pmb {\xi} _ {i ^ {\prime}} \rangle) \cdot \langle \pmb {\xi} _ {i ^ {\prime}} , \pmb {\xi} _ {i} \rangle \mathbb {1} (i ^ {\prime} \neq i)} _ {\mathrm {I} _ {5}} \\ \end{array}
$$
$$
- \frac {\eta}{B m} \cdot \underbrace {\sum_ {b ^ {\prime} = 0} ^ {H - 1} \sum_ {i ^ {\prime} \in \mathcal {I} _ {\widetilde {t} - 1 , b ^ {\prime}}} \ell_ {i ^ {\prime}} ^ {(\widetilde {t} - 1 , b ^ {\prime})} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {y _ {i} , r} ^ {(\widetilde {t} - 1 , b ^ {\prime})} , y _ {i ^ {\prime}} \boldsymbol {\mu} \rangle) \cdot \langle y _ {i ^ {\prime}} \boldsymbol {\mu} , \boldsymbol {\xi} _ {i} \rangle y _ {i ^ {\prime}} ,} _ {\mathrm {I _ {6}}}
$$
for any  $i \in S_{j,r}^{(\widetilde{t} - 1,0)}$ , where the equality holds due to  $\langle \mathbf{w}_{j,r}^{(\widetilde{t} - 1,b_i^{(\widetilde{t} - 1)})}, \pmb{\xi}_i \rangle > 0$  and  $y_i = j$ . By applying the same technique used in the proof of the fourth hypothesis, it follows that
$$
\langle \mathbf {w} _ {j, r} ^ {(\widetilde {t}, 0)}, \boldsymbol {\xi} _ {i} \rangle \geq \langle \mathbf {w} _ {j, r} ^ {(\widetilde {t} - 1, 0)}, \boldsymbol {\xi} _ {i} \rangle > 0,
$$
for any  $i \in S_{j,r}^{(\widetilde{t} - 1,0)}$ . Thus, we have  $S_{j,r}^{(0,0)} \subseteq S_{j,r}^{(\widetilde{t} - 1,0)} \subseteq S_{j,r}^{(\widetilde{t},0)}$ . And it directly follows by Lemma B.4 that  $|S_{j,r}^{(\widetilde{t},0)}| \geq n\Phi(-1)/4$ .
Proof of Proposition C.2. Our proof is based on induction. The results are obvious at iteration  $(0,0)$  as all the coefficients are zero. Suppose that the results in Proposition C.2 hold for all iterations  $(0,0) \leq (t,b) < (\widetilde{t},\widetilde{b})$ . We aim to prove that they also hold for iteration  $(\widetilde{t},\widetilde{b})$ .
Firstly, We prove that (28) exists at iteration  $(\widetilde{t},\widetilde{b})$ , i.e.,  $\underline{\rho}_{j,r,i}^{(\widetilde{t},\widetilde{b})}\geq -\beta -10\sqrt{\log(6n^2 / \delta) / d}\cdot n\alpha$  for any  $r\in [m]$ ,  $j\in \{\pm 1\}$  and  $i\in [n]$ . Notice that  $\underline{\rho}_{j,r,i}^{(\widetilde{t},\widetilde{b})} = 0$  for  $j = y_{i}$ , therefore we only need to consider the case that  $j\neq y_{i}$ . We also only need to consider the case of  $\widetilde{b} = b_i^{(\widetilde{t})} + 1$  since  $\underline{\rho}_{j,r,i}^{(\widetilde{t},\widetilde{b})}$  doesn't change in other cases according to (21).
When  $\underline{\rho}_{j,r,t}^{(\tilde{t},b_i^{(\tilde{t})})} < -0.5\beta - 5\sqrt{\log(6n^2 / \delta) / d}\cdot n\alpha$ , by (32) in Lemma C.3 we have that
$$
(P - 1) \langle \mathbf {w} _ {j, r} ^ {(\widetilde {t}, b _ {i} ^ {(t)})}, \boldsymbol {\xi} _ {i} \rangle \leq \underline {{\rho}} _ {j, r, i} ^ {(\widetilde {t}, b _ {i} ^ {(t)})} + (P - 1) \langle \mathbf {w} _ {j, r} ^ {(0, 0)}, \boldsymbol {\xi} _ {i} \rangle + 5 \sqrt {\frac {\log (6 n ^ {2} / \delta)}{d}} n \alpha <   0,
$$
$$
\begin{array}{l} \underline {{\rho}} _ {j, r, i} ^ {(\widetilde {t}, \widetilde {b})} = \underline {{\rho}} _ {j, r, i} ^ {(\widetilde {t}, b _ {i} ^ {(\widetilde {t})})} + \frac {\eta (P - 1) ^ {2}}{B m} \cdot \ell_ {i} ^ {\prime (\widetilde {t}, b _ {i} ^ {(\widetilde {t})})} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(\widetilde {t}, b _ {i} ^ {(\widetilde {t})}}, \pmb {\xi} _ {i} \rangle) \cdot \| \pmb {\xi} _ {i} \| _ {2} ^ {2}. \\ = \underline {{\rho}} _ {j, r, i} ^ {(\widetilde {t}, b _ {i} ^ {(\widetilde {t})})} \geq - \beta - 1 0 \sqrt {\frac {\log (6 n ^ {2} / \delta)}{d}} n \alpha , \\ \end{array}
$$
where the last inequality is by induction hypothesis.
When  $\underline{\rho}_{j,r,t}^{(\tilde{t},b_i^{(\tilde{t})})}\geq -0.5\beta -5\sqrt{\log(6n^2 / \delta) / d}\cdot n\alpha$  we have
$$
\begin{array}{l} \underline {{\rho}} _ {j, r, i} ^ {(\widetilde {t}, \widetilde {b})} = \underline {{\rho}} _ {j, r, i} ^ {(t, b _ {i} ^ {(\widetilde {t})})} + \frac {\eta (P - 1) ^ {2}}{B m} \cdot \ell_ {i} ^ {\prime (t, b _ {i} ^ {(\widetilde {t})})} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b _ {i} ^ {(\widetilde {t})}}, \boldsymbol {\xi} _ {i} \rangle) \cdot \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2} \\ \geq - 0. 5 \beta - 5 \sqrt {\frac {\log (6 n ^ {2} / \delta)}{d}} n \alpha - \frac {\eta (P - 1) ^ {2} \cdot 3 \sigma_ {p} ^ {2} d}{2 B m} \\ \geq - 0. 5 \beta - 1 0 \sqrt {\frac {\log (6 n ^ {2} / \delta)}{d}} n \alpha \\ \geq - \beta - 1 0 \sqrt {\frac {\log (6 n ^ {2} / \delta)}{d}} n \alpha , \\ \end{array}
$$
where the first inequality is by  $\ell_i^{\prime (t,b_i^{(t)})}\in (-1,0)$  and  $\| \pmb {\xi}_i\| _2^2\leq (3 / 2)\sigma_p^2 d$  by Lemma B.1; the second inequality is due to  $5\sqrt{\log(6n^{2} / \delta) / d}\cdot n\alpha \geq 3\eta \sigma_{p}^{2}d / (2Bm)$  by Condition 3.1.
Next we prove (27) holds for  $(\widetilde{t},\widetilde{b})$ . We only need to consider the case of  $j = y_{i}$ . Consider
$$
\begin{array}{l} | \ell_ {i} ^ {\prime (\widetilde {t}, \widetilde {b})} | = \frac {1}{1 + \exp \left\{y _ {i} \cdot \left[ F _ {+ 1} \left(\mathbf {W} _ {+ 1} ^ {(\widetilde {t} , \widetilde {b})} , \mathbf {x} _ {i}\right) - F _ {- 1} \left(\mathbf {W} _ {- 1} ^ {(\widetilde {t} , \widetilde {b})} , \mathbf {x} _ {i}\right) \right] \right\}} \\ \leq \exp \left(- y _ {i} \cdot \left[ F _ {+ 1} \left(\mathbf {W} _ {+ 1} ^ {\left(\widetilde {t}, \widetilde {b}\right)}, \mathbf {x} _ {i}\right) - F _ {- 1} \left(\mathbf {W} _ {- 1} ^ {\left(\widetilde {t}, \widetilde {b}\right)}, \mathbf {x} _ {i}\right) \right]\right) \tag {40} \\ \leq \exp (- F _ {y _ {i}} (\mathbf {W} _ {y _ {i}} ^ {(\widetilde {t}, \widetilde {b})}, \mathbf {x} _ {i}) + 0. 5), \\ \end{array}
$$
where the last inequality is by  $F_{j}(\mathbf{W}_{j}^{(\widehat{t},\widehat{b})}, \mathbf{x}_{i}) \leq 0.5$  for  $j \neq y_{i}$  according to Lemma C.4. Now recall the iterative update rule of  $\overline{\rho}_{j,r,i}^{(t,b)}$ :
$$
\overline {{\rho}} _ {j, r, i} ^ {(t, b + 1)} = \overline {{\rho}} _ {j, r, i} ^ {(t, b)} - \frac {\eta (P - 1) ^ {2}}{B m} \cdot \ell_ {i} ^ {\prime (t, b)} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\xi} _ {i} \rangle) \cdot \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2} \cdot \mathbb {1} (i \in \mathcal {I} _ {t, b}).
$$
Let  $(t_{j,r,i},b_{j,r,i})$  be the last time before  $(\widetilde{t},\widetilde{b})$  that  $\overline{\rho}_{j,r,i}^{(t_{j,r,i},b_{j,r,i})}\leq 0.5\alpha$ . Then by iterating the update rule from  $(t_{j,r,i},b_{j,r,i})$  to  $(\widetilde{t},\widetilde{b})$ , we get
$$
\begin{array}{l} \overline {{\rho}} _ {j, r, i} ^ {(\widetilde {t}, \widetilde {b})} \\ = \bar {\rho} _ {j, r, i} ^ {\left(t _ {j, r, i}, b _ {j, r, i}\right)} - \underbrace {\frac {\eta (P - 1) ^ {2}}{B m} \cdot \ell_ {i} ^ {\prime \left(t _ {j , r , i} , b _ {j , r , i}\right)} \cdot \mathbb {1} \left(\left\langle \mathbf {w} _ {j , r} ^ {\left(t _ {j, r, i}, b _ {j, r, i}\right)}, \boldsymbol {\xi} _ {i} \right\rangle \geq 0\right) \cdot \mathbb {1} \left(i \in \mathcal {I} _ {t , b}\right) \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2}} _ {\mathrm {I} _ {7}} \tag {41} \\ - \underbrace {\sum_ {(t _ {j , r , i} , b _ {j , r , i}) <   (t , b) <   (\widetilde {t} , \widetilde {b})} \frac {\eta (P - 1) ^ {2}}{B m} \cdot \ell_ {i} ^ {\prime (t , b)} \cdot \mathbb {1} (\langle \mathbf {w} _ {j , r} ^ {(t , b)} , \boldsymbol {\xi} _ {i} \rangle \geq 0) \cdot \mathbb {1} (i \in \mathcal {I} _ {t , b}) \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2}} _ {\mathrm {I} _ {8}}. \\ \end{array}
$$
We first bound  $\mathrm{I}_7$  as follows:
$$
\left| \mathrm {I} _ {7} \right| \leq \left(\eta (P - 1) ^ {2} / B m\right) \cdot \left\| \boldsymbol {\xi} _ {i} \right\| _ {2} ^ {2} \leq \left(\eta (P - 1) ^ {2} / B m\right) \cdot 3 \sigma_ {p} ^ {2} d / 2 \leq 1 \leq 0. 2 5 \alpha ,
$$
where the first inequality is by  $\ell_i^{\prime (t_{j,r,i},b_{j,r,i})}\in (-1,0)$ ; the second inequality is by Lemma B.1; the third inequality is by Condition 3.1; the last inequality is by our choice of  $\alpha = 4\log (T^{*})$  and  $T^{*}\geq e$ .
Second, we bound  $\mathrm{I}_8$ . For  $(t_{j,r,i},b_{j,r,i}) < (t,b) < (\widetilde{t},\widetilde{b})$  and  $y_{i} = j$ , we can lower bound the inner product  $\langle \mathbf{w}_{j,r}^{(t,b)},\pmb{\xi}_i\rangle$  as follows
$$
\begin{array}{l} \langle \mathbf {w} _ {j, r} ^ {(t, b)}, \pmb {\xi} _ {i} \rangle \geq \langle \mathbf {w} _ {j, r} ^ {(0, 0)}, \pmb {\xi} _ {i} \rangle + \frac {1}{P - 1} \overline {{\rho}} _ {j, r, i} ^ {(t, b)} - \frac {5}{P - 1} \sqrt {\frac {\log (6 n ^ {2} / \delta)}{d}} n \alpha \\ \geq - \frac {0 . 5}{P - 1} \beta + \frac {0 . 5}{P - 1} \alpha - \frac {5}{P - 1} \sqrt {\frac {\log (6 n ^ {2} / \delta)}{d}} n \alpha \tag {42} \\ \geq \frac {0 . 2 5}{P - 1} \alpha , \\ \end{array}
$$
where the first inequality is by (31) in Lemma C.3; the second inequality is by  $\overline{\rho}_{j,r,i}^{(t,b)} > 0.5\alpha$  and  $\langle \mathbf{w}_{j,r}^{(0,0)},\pmb {\xi}_i\rangle \geq -0.5\beta /(P - 1)$  due to the definition of  $t_{j,r,i}$  and  $\beta$ ; the last inequality is by  $\beta \leq 1 / 8\leq 0.1\alpha$  and  $5\sqrt{\log(6n^2 / \delta) / d}\cdot n\alpha \leq 0.2\alpha$  by Condition 3.1.
Thus, plugging the lower bounds of  $\langle \mathbf{w}_{j,r}^{(t,b)},\pmb{\xi}_i\rangle$  into  $\mathrm{I_8}$  gives
$$
\begin{array}{l} \left| I _ {8} \right| \leq \sum_ {\left(t _ {j, r, i}, b _ {j, r, i}\right) <   (t, b) <   (\widetilde {t}, \widetilde {b})} \frac {\eta (P - 1) ^ {2}}{B m} \cdot \exp \Big (- \frac {1}{m} \sum_ {r = 1} ^ {m} (P - 1) \sigma \big (\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\xi} _ {i} \rangle \big) + 0. 5 \Big) \\ \cdot \mathbb {1} \left(\left\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\xi} _ {i} \right\rangle \geq 0\right) \cdot \left\| \boldsymbol {\xi} _ {i} \right\| _ {2} ^ {2} \\ \leq \frac {2 \eta T ^ {*} n (P - 1) ^ {2}}{B m} \cdot \exp (- 0. 2 5 \alpha) \exp (0. 5) \cdot \frac {3 \sigma_ {p} ^ {2} d}{2} \\ \leq \frac {2 \eta T ^ {*} n (P - 1) ^ {2}}{B m} \cdot \exp (- \log (T ^ {*})) \exp (0. 5) \cdot \frac {3 \sigma_ {p} ^ {2} d}{2} \\ = \frac {2 \eta n (P - 1) ^ {2}}{B m} \cdot \frac {3 \sigma_ {p} ^ {2} d}{2} \exp (0. 5) \leq 1 \leq 0. 2 5 \alpha , \\ \end{array}
$$
where the first inequality is by (40); the second inequality is by (42); the third inequality is by  $\alpha = 4\log (T^{*})$ ; the fourth inequality is by Condition 3.1; the last inequality is by  $\log (T^{*})\geq 1$  and  $\alpha = 4\log (T^{*})$ . Plugging the bound of  $\mathrm{I}_7,\mathrm{I}_8$  into (41) completes the proof for  $\overline{\rho}$ .
For the upper bound of (29), we prove a augmented hypothesis that there exists a  $i^{*} \in [n]$  with  $y_{i^{*}} = j$  such that for  $1 \leq t \leq T^{*}$  we have that  $\gamma_{j,r}^{(t,0)} / \overline{\rho}_{j,r,i^{*}} \leq C' \hat{\gamma}$ . Recall the iterative update rule of  $\gamma_{j,r}^{(t,b)}$  and  $\overline{\rho}_{j,r,i}^{(t,b)}$ , we have
$$
\begin{array}{l} \overline {{\rho}} _ {j, r, i} ^ {(t, b + 1)} = \overline {{\rho}} _ {j, r, i} ^ {(t, b)} - \frac {\eta (P - 1) ^ {2}}{B m} \cdot \ell_ {i} ^ {\prime (t, b)} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\xi} _ {i} \rangle) \cdot \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2} \cdot \mathbb {1} (y _ {i} = j) \mathbb {1} (i \in \mathcal {I} _ {t, b}), \\ \gamma_ {j, r} ^ {(t, b + 1)} = \gamma_ {j, r} ^ {(t, b)} - \frac {\eta}{B m} \cdot \left[ \sum_ {i \in \mathcal {I} _ {t, b} \cap S _ {+}} \ell_ {i} ^ {\prime (t, b)} \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b)}, y _ {i} \cdot \boldsymbol {\mu} \rangle) \right. \\ \left. - \sum_ {i \in \mathcal {I} _ {t, b} \cap S _ {-}} \ell_ {i} ^ {\prime (t, b)} \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(t, b)}, y _ {i} \cdot \boldsymbol {\mu} \right\rangle\right) \right] \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2}. \\ \end{array}
$$
According to the fifth statement of Lemma C.8, for any  $i^{*} \in S_{j,r}^{(0,0)}$  it holds that  $j = y_{i^{*}}$  and  $\langle \mathbf{w}_{j,r}^{(t,b)}, \pmb{\xi}_{i^{*}} \rangle \geq 0$  for any  $(t,b) \leq (\widetilde{t},\widetilde{b})$ . Thus, we have
$$
\overline {{\rho}} _ {j, r, i ^ {*}} ^ {(\widetilde {t}, 0)} = \overline {{\rho}} _ {j, r, i ^ {*}} ^ {(\widetilde {t} - 1, 0)} - \frac {\eta (P - 1) ^ {2}}{B m} \cdot \ell_ {i ^ {*}} ^ {\prime (\widetilde {t} - 1, b _ {i ^ {*}} ^ {(\widetilde {t} - 1)})} \cdot \| \boldsymbol {\xi} _ {i ^ {*}} \| _ {2} ^ {2} \geq \overline {{\rho}} _ {j, r, i ^ {*}} ^ {(\widetilde {t} - 1, 0)} - \frac {\eta (P - 1) ^ {2}}{B m} \cdot \ell_ {i ^ {*}} ^ {\prime (\widetilde {t} - 1, b _ {i ^ {*}} ^ {(\widetilde {t} - 1)})} \cdot \sigma_ {p} ^ {2} d / 2.
$$
For the update rule of  $\gamma_{j,r}^{(t,b)}$ , according to Lemma C.8, we have
$$
\begin{array}{l} \sum_ {b <   H} \left| \sum_ {i \in \mathcal {I} _ {t, b} \cap S _ {+}} \ell_ {i} ^ {\prime (t, b)} \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b)}, y _ {i} \cdot \boldsymbol {\mu} \rangle) - \sum_ {i \in \mathcal {I} _ {t, b} \cap S _ {-}} \ell_ {i} ^ {\prime (t, b)} \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b)}, y _ {i} \cdot \boldsymbol {\mu} \rangle) \right| \\ \leq C _ {2} n \Big | \ell_ {i ^ {*}} ^ {\prime (\widetilde {T} - 1, b _ {i ^ {*}} ^ {(\widetilde {T} - 1)})} \Big |. \\ \end{array}
$$
$$
\begin{array}{l} \frac {\gamma_ {j , r} ^ {(\widetilde {t} , 0)}}{\overline {{\rho}} _ {j , r , i ^ {*}} ^ {(\widetilde {t} , 0)}} \leq \max \left\{\frac {\gamma_ {j , r} ^ {(\widetilde {t} - 1 , 0)}}{\overline {{\rho}} _ {j , r , i ^ {*}} ^ {(\widetilde {t} - 1 , 0)}}, \frac {C _ {2} n \ell_ {i ^ {*}} ^ {\prime (\widetilde {t} - 1 , b _ {i ^ {*}} ^ {(\widetilde {t} - 1)})} \| \boldsymbol {\mu} \| _ {2} ^ {2}}{(P - 1) ^ {2} \cdot \ell_ {i ^ {*}} ^ {\prime (\widetilde {t} - 1 , b _ {i ^ {*}} ^ {(\widetilde {t} - 1)})} \cdot \sigma_ {p} ^ {2} d / 2} \right\} \\ = \max  \left\{\frac {\gamma_ {j , r} ^ {(\widetilde {t} - 1 , 0)}}{\bar {\rho} _ {j , r , i ^ {*}} ^ {(\widetilde {t} - 1 , 0)}}, \frac {2 C _ {2} n \| \boldsymbol {\mu} \| _ {2} ^ {2}}{(P - 1) ^ {2} \sigma_ {p} ^ {2} d} \right\} \tag {43} \\ \leq \frac {2 C _ {2} n \| \pmb {\mu} \| _ {2} ^ {2}}{(P - 1) ^ {2} \sigma_ {p} ^ {2} d}, \\ \end{array}
$$
where the last inequality is by  $\gamma_{j,r}^{(\widetilde{t} -1,0)} / \overline{\rho}_{j,r,i^*}^{(\widetilde{t} -1,0)}\leq 2C_2\widehat{\gamma} = 2C_2n\| \pmb {\mu}\| _2^2 /(P - 1)^2\sigma_p^2 d.$  Therefore,
$$
\begin{array}{l} \frac {\gamma_ {j , r} ^ {(\widetilde {t} , 0)}}{\overline {{\rho}} _ {j , r , i ^ {*}} ^ {(\widetilde {t} , 0)}} \leq 2 C _ {2} \widehat {\gamma}. \end{array}
$$
For iterations other than the starting of en epoch, we have the following upper bound:
$$
\frac {\gamma_ {j , r} ^ {(\widetilde {t} , b)}}{\overline {{\rho}} _ {j , r , i ^ {*}} ^ {(\widetilde {t} , b)}} \leq \frac {2 \gamma_ {j , r} ^ {(\widetilde {t} , 0)}}{\overline {{\rho}} _ {j , r , i ^ {*}} ^ {(\widetilde {t} , 0)}} \leq 4 C _ {2} \widehat {\gamma}.
$$
Thus, by taking  $C' = 4C_2$ , we have  $\gamma_{j,r}^{(\widetilde{t},b)} / \overline{\rho}_{j,r,i^*}^{(\widetilde{t},b)} \leq C'\hat{\gamma}$ .
On the other hand, when  $(t,b) < (\frac{\log(2T^{*} / \delta)}{2c_3^2},0)$ , we have
$$
\gamma_ {j, r} ^ {(t, b)} \geq - \frac {\log (2 T ^ {*} / \delta)}{2 c _ {3} ^ {2}} \cdot \frac {\eta}{B m} \cdot n \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2} \geq - \frac {1}{1 2},
$$
where the first inequality is due to update rule of  $\gamma_{j,r}^{t,b}$ , and the second inequality is due to Condition 3.1.
When  $(t,b)\geq (\frac{\log(2T^{*} / \delta)}{2c_{3}^{2}},0)$ , According to Lemma B.6, we have
$$
\begin{array}{l} \gamma_ {j, r} ^ {(t, b)} \geq \sum_ {\left(t ^ {\prime}, b ^ {\prime}\right) <   (t, b)} \frac {\eta}{B m} \left[ \min  _ {i, b ^ {\prime}} \ell_ {i} ^ {\left(t ^ {\prime}, b ^ {\prime}\right)} \min  \left\{\left| \mathcal {I} _ {t ^ {\prime}, b ^ {\prime}} \cap S _ {+} \cap S _ {- 1} \right|, \left| \mathcal {I} _ {t ^ {\prime}, b ^ {\prime}} \cap S _ {+} \cap S _ {1} \right| \right\} \right. \\ \left. - \max  _ {i, b ^ {\prime}} \ell_ {i} ^ {\prime \left(t ^ {\prime}, b ^ {\prime}\right)} \left| \mathcal {I} _ {t ^ {\prime}, b ^ {\prime}} \cap S _ {-} \right| \right] \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2} \\ \geq \frac {\eta}{B m} \left(\sum_ {t ^ {\prime} = 0} ^ {t - 1} \left(c _ {3} c _ {4} H \frac {B}{4} \min  _ {i, b ^ {\prime}} \ell_ {i} ^ {\prime \left(t ^ {\prime}, b ^ {\prime}\right)} - n q \max  _ {i, b ^ {\prime}} \ell_ {i} ^ {\prime \left(t ^ {\prime}, b ^ {\prime}\right)} - n q \max  _ {i, b ^ {\prime}} \ell_ {i} ^ {\prime \left(t, b ^ {\prime}\right)}\right)\right) \| \boldsymbol {\mu} \| _ {2} ^ {2} \\ \geq 0, \\ \end{array}
$$
where the first inequality is due to the update rule of  $\gamma_{j,r}^{(t,b)}$ , the second inequality is due to Lemma B.6, and the third inequality is due to Condition 3.1.
C.2 Decoupling with a Two-stage Analysis
C.2.1 First Stage
Lemma C.9. There exist
$$
T _ {1} = C _ {3} \eta^ {- 1} B m (P - 1) ^ {- 2} \sigma_ {p} ^ {- 2} d ^ {- 1}, T _ {2} = C _ {4} \eta^ {- 1} B m (P - 1) ^ {- 2} \sigma_ {p} ^ {- 2} d ^ {- 1},
$$
where  $C_3 = \Theta(1)$  is a large constant and  $C_4 = \Theta(1)$  is a small constant, such that
-  $\overline{\rho}_{j,r^*,i}^{(T_1,0)} \geq 2$  for any  $r^* \in S_i^{(0,0)} = \{r \in [m] : \langle \mathbf{w}_{y_i,r}^{(0)}, \pmb{\xi}_i \rangle > 0\}$ ,  $j \in \{\pm 1\}$  and  $i \in [n]$  with  $y_i = j$ .
-  $\max_{j,r} \gamma_{j,r}^{(t,b)} = O(\widehat{\gamma})$  for all  $(t,b) \leq (T_1,0)$ .
-  $\max_{j,r,i}|\underline{\rho}_{j,r,i}^{(t,b)}| = \max \{\beta ,O\big(n\sqrt{\log(n / \delta)}\log (T^{*}) / \sqrt{d}\big)\}$  for all  $(t,b)\leq (T_1,0)$ .
-  $\min_{j,r} \gamma_{j,r}^{(t,0)} = \Omega(\widehat{\gamma})$  for all  $t \geq T_2$ .
-  $\max_{j,r} \overline{\rho}_{j,r,i}^{(T_1,0)} = O(1)$  for all  $i \in [n]$ .
Proof of Lemma C.9. By Proposition C.2, we have that  $\underline{\rho}_{j,r,i}^{(t,b)}\geq -\beta -10n\sqrt{\frac{\log(6n^2 / \delta)}{d}}\alpha$  for all  $j\in \{\pm 1\}$ $r\in [m],i\in [n]$  and  $(0,0)\leq (t,b)\leq (T^{*},0)$ . According to Lemma B.2, for  $\beta$  we have
$$
\begin{array}{l} \beta = 2 \max _ {i, j, r} \{\left| \left\langle \mathbf {w} _ {j, r} ^ {(0, 0)}, \boldsymbol {\mu} \right\rangle \right|, (P - 1) \left| \left\langle \mathbf {w} _ {j, r} ^ {(0, 0)}, \boldsymbol {\xi} _ {i} \right\rangle \right| \} \\ \leq 2 \max  \left\{\sqrt {2 \log (1 2 m / \delta)} \cdot \sigma_ {0} \| \boldsymbol {\mu} \| _ {2}, 2 \sqrt {\log (1 2 m n / \delta)} \cdot \sigma_ {0} (P - 1) \sigma_ {p} \sqrt {d} \right\} \\ = O \left(\sqrt {\log (m n / \delta)} \cdot \sigma_ {0} (P - 1) \sigma_ {p} \sqrt {d}\right), \\ \end{array}
$$
where the last equality is by the first condition of Condition 3.1. Since  $\underline{\rho}_{j,r,i}^{(t,b)}\leq 0$  , we have that
$$
\begin{array}{l} \max  _ {j, r, i} | \underline {{\rho}} _ {j, r, i} ^ {(t, b)} | = \max  _ {j, r, i} - \underline {{\rho}} _ {j, r, i} ^ {(t, b)} \\ \leq \beta + 1 0 \sqrt {\frac {\log (6 n ^ {2} / \delta)}{d}} n \alpha \\ = \max \left\{\beta , O \big (\sqrt {\log (n / \delta)} \log (T ^ {*}) \cdot n / \sqrt {d} \big) \right\}. \\ \end{array}
$$
Next, for the growth of  $\gamma_{j,r}^{(t)}$ , we have following upper bound
$$
\begin{array}{l} \gamma_ {j, r} ^ {(t, b + 1)} = \gamma_ {j, r} ^ {(t, b)} - \frac {\eta}{B m} \cdot \sum_ {i \in \mathcal {I} _ {t, b}} \ell_ {i} ^ {\prime (t, b)} \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b)}, y _ {i} \cdot \boldsymbol {\mu} \rangle) \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2} \\ \leq \gamma_ {j, r} ^ {(t, b)} + \frac {\eta}{m} \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2}, \\ \end{array}
$$
where the inequality is by  $|\ell^{\prime}|\leq 1$ . Note that  $\gamma_{j,r}^{(0,0)} = 0$  and recursively use the inequality  $tB + b$  times we have
$$
\gamma_ {j, r} ^ {(t, b)} \leq \frac {\eta (t H + b)}{m} \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2}. \tag {44}
$$
Since  $n\cdot \mathrm{SNR}^2 = n\|\pmb {\mu}\|_2^2 /((P - 1)^2\sigma_p^2 d) = \widehat{\gamma}$ , we have
$$
T _ {1} = C _ {3} \eta^ {- 1} B m (P - 1) ^ {- 2} \sigma_ {p} ^ {- 2} d ^ {- 1} = C _ {3} \eta^ {- 1} m \| \boldsymbol {\mu} \| _ {2} ^ {- 2} \hat {\gamma} B / n.
$$
$$
\gamma_ {j, r} ^ {(t)} \leq \frac {\eta (t H + b)}{m} \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2} \leq \frac {\eta n T _ {1}}{m B} \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2} \leq C _ {3} \widehat {\gamma},
$$
for all  $(0,0)\leq (t,b)\leq (T_1,0)$
For  $\overline{\rho}_{j,r,i}^{(t)}$ , recall from (20) that
$$
\overline {{\rho}} _ {y _ {i}, r, i} ^ {(t + 1, 0)} = \overline {{\rho}} _ {y _ {i}, r, i} ^ {(t, 0)} - \frac {\eta (P - 1) ^ {2}}{B m} \cdot \ell_ {i} ^ {\prime (t, b _ {i} ^ {(t)})} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {y _ {i}, r} ^ {(t, b _ {i} ^ {(t)})}, \boldsymbol {\xi} _ {i} \rangle) \cdot \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2}.
$$
According to Lemma C.8, for any  $r^* \in S_i^{(0,0)} = \{r \in [m] : \langle \mathbf{w}_{y_i,r}^{(0)}, \pmb{\xi}_i \rangle > \sigma_0 \sigma_p \sqrt{d} / \sqrt{2}\}$ , we have  $\langle \mathbf{w}_{y_i,r^*}^{(t,b)}, \pmb{\xi}_i \rangle > 0$  for all  $(0,0) \leq (t,b) \leq (T^*,0)$  and hence
$$
\bar {\rho} _ {j, r ^ {*}, i} ^ {(t + 1, 0)} = \bar {\rho} _ {j, r ^ {*}, i} ^ {(t, 0)} - \frac {\eta (P - 1) ^ {2}}{B m} \cdot \ell_ {i} ^ {\prime \left(t, b _ {i} ^ {(t)}\right)} \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2}.
$$
For each  $i$ , we denote by  $T_1^{(i)}$  the last time in the period  $[0, T_1]$  satisfying that  $\overline{\rho}_{y_i, r^*, i}^{(t, 0)} \leq 2$ . Then for  $(0, 0) \leq (t, b) < (T_1^{(i)}, 0)$ ,  $\max_{j, r} \{|\overline{\rho}_{j, r, i}^{(t, b)}|, |\underline{\rho}_{j, r, i}^{(t, b)}|\} = O(1)$  and  $\max_{j, r} \gamma_{j, r}^{(t, b)} = O(1)$ . Therefore, we know that  $F_{-1}(\mathbf{W}^{(t, b)}, \mathbf{x}_i), F_{+1}(\mathbf{W}^{(t, b)}, \mathbf{x}_i) = O(1)$ . Thus there exists a positive constant  $C$  such that  $-\ell_i'^{(t, b)} \geq C \geq C_2$  for  $0 \leq t \leq T_1^{(i)}$ .
$$
\overline {{\rho}} _ {y _ {i}, r ^ {*}, i} ^ {(t, 0)} \geq \frac {C \eta (P - 1) ^ {2} \sigma_ {p} ^ {2} d t}{2 B m}.
$$
Therefore,  $\overline{\rho}_{y_i,r^*,i}^{(t,0)}$  will reach 2 within
$$
T _ {1} = C _ {3} \eta^ {- 1} B m (P - 1) ^ {2} \sigma_ {p} ^ {- 2} d ^ {- 1}
$$
iterations for any  $r^* \in S_i^{(0,0)}$ , where  $C_3$  can be taken as  $4 / C$ .
Next, we will discuss the lower bound of the growth of  $\gamma_{j,r}^{(t,b)}$ . For  $\overline{\rho}_{j,r,i}^{(t,b)}$ , we have
$$
\begin{array}{l} \bar {\rho} _ {j, r, i} ^ {(t, b + 1)} = \bar {\rho} _ {j, r, i} ^ {(t, b)} - \frac {\eta (P - 1) ^ {2}}{B m} \cdot \ell_ {i} ^ {\prime (t, b)} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\xi} _ {i} \rangle) \cdot \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2} \cdot \mathbb {1} (y _ {i} = j) \mathbb {1} (i \in \mathcal {I} _ {t, b}) \\ \leq \overline {{\rho}} _ {j, r, i} ^ {(t, b)} + \frac {3 \eta (P - 1) ^ {2} \sigma_ {p} ^ {2} d}{2 B m}. \\ \end{array}
$$
According to (44) and  $\overline{\rho}_{j,r,i}^{(0,0)} = 0$ , it follows that
$$
\bar {\rho} _ {j, r, i} ^ {(t, b)} \leq \frac {3 \eta (P - 1) ^ {2} \sigma_ {p} ^ {2} d (t H + b)}{2 B m}, \gamma_ {j, r} ^ {(t, b)} \leq \frac {\eta (t H + b)}{m} \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2}. \tag {45}
$$
Therefore,  $\max_{j,r,i}\overline{\rho}_{j,r,i}^{(t,b)}$  will be smaller than 1 and  $\gamma_{j,r}^{(t,b)}$  smaller than  $\Theta (n\| \pmb {\mu}\| _2^2 /(P - 1)^2\sigma_p^2 d) =$ $\Theta (n\cdot \mathrm{SNR}^2) = \Theta (\hat{\gamma}) = O(1)$  within
$$
T _ {2} = C _ {4} \eta^ {- 1} B m (P - 1) ^ {- 2} \sigma_ {p} ^ {- 2} d ^ {- 1}
$$