SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} y _ {i} \cdot f \left(\mathbf {W} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)}, \mathbf {x} _ {i}\right) - y _ {k} \cdot f \left(\mathbf {W} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)}, \mathbf {x} _ {k}\right) \\ = y _ {i} \sum_ {j \in \{\pm 1 \}} j \cdot F _ {j} (\mathbf {W} _ {j} ^ {(t ^ {\prime}, b _ {1} ^ {\prime})}, \mathbf {x} _ {i}) - y _ {k} \sum_ {j \in \{\pm 1 \}} j \cdot F _ {j} (\mathbf {W} _ {j} ^ {(t ^ {\prime}, b _ {2} ^ {\prime})}, \mathbf {x} _ {k}) \\ = F _ {- y _ {k}} (\mathbf {W} _ {- y _ {k}} ^ {(t ^ {\prime}, b _ {2} ^ {\prime})}, \mathbf {x} _ {k}) - F _ {- y _ {i}} (\mathbf {W} _ {- y _ {i}} ^ {(t ^ {\prime}, b _ {1} ^ {\prime})}, \mathbf {x} _ {i}) + F _ {y _ {i}} (\mathbf {W} _ {y _ {i}} ^ {(t ^ {\prime}, b _ {1} ^ {\prime})}, \mathbf {x} _ {i}) - F _ {y _ {k}} (\mathbf {W} _ {y _ {k}} ^ {(t ^ {\prime}, b _ {2} ^ {\prime})}, \mathbf {x} _ {k}) \\ = F _ {- y _ {k}} (\mathbf {W} _ {- y _ {k}} ^ {(t ^ {\prime}, b _ {2} ^ {\prime})}, \mathbf {x} _ {k}) - F _ {- y _ {i}} (\mathbf {W} _ {- y _ {i}} ^ {(t ^ {\prime}, b _ {1} ^ {\prime})}, \mathbf {x} _ {i}) \\ + \frac {1}{m} \sum_ {r = 1} ^ {m} [ \sigma (\langle \mathbf {w} _ {y _ {i}, r} ^ {(t ^ {\prime}, b _ {1} ^ {\prime})}, y _ {i} \cdot \boldsymbol {\mu} \rangle) + (P - 1) \sigma (\langle \mathbf {w} _ {y _ {i}, r} ^ {(t ^ {\prime}, b _ {1} ^ {\prime})}, \boldsymbol {\xi} _ {i} \rangle) ] \\ - \frac {1}{m} \sum_ {r = 1} ^ {m} [ \sigma (\langle \mathbf {w} _ {y _ {k}, r} ^ {(t ^ {\prime}, b _ {2} ^ {\prime})}, y _ {k} \cdot \boldsymbol {\mu} \rangle) + (P - 1) \sigma (\langle \mathbf {w} _ {y _ {k}, r} ^ {(t ^ {\prime}, b _ {2} ^ {\prime})}, \boldsymbol {\xi} _ {k} \rangle) ] \\ = \underbrace {F _ {- y _ {k}} \left(\mathbf {W} _ {- y _ {k}} ^ {\left(t ^ {\prime} , b _ {2} ^ {\prime}\right)} , \mathbf {x} _ {k}\right) - F _ {- y _ {i}} \left(\mathbf {W} _ {- y _ {i}} ^ {\left(t ^ {\prime} , b _ {1} ^ {\prime}\right)} , \mathbf {x} _ {i}\right)} _ {\mathrm {I} _ {1}} \\ + \underbrace {\frac {1}{m} \sum_ {r = 1} ^ {m} \left[ \sigma \left(\left\langle \mathbf {w} _ {y _ {i} , r} ^ {\left(t ^ {\prime} , b _ {1} ^ {\prime}\right)} , y _ {i} \cdot \boldsymbol {\mu} \right\rangle\right) - \sigma \left(\left\langle \mathbf {w} _ {y _ {k} , r} ^ {\left(t ^ {\prime} , b _ {2} ^ {\prime}\right)} , y _ {k} \cdot \boldsymbol {\mu} \right\rangle\right) \right]} _ {\mathrm {I} _ {2}} \\ + \underbrace {\frac {1}{m} \sum_ {r = 1} ^ {m} \left[ \left(P - 1\right) \sigma \left(\left\langle \mathbf {w} _ {y _ {i} , r} ^ {\left(t ^ {\prime} , b _ {1} ^ {\prime}\right)} , \boldsymbol {\xi} _ {i} \right\rangle\right) - \left(P - 1\right) \sigma \left(\left\langle \mathbf {w} _ {y _ {k} , r} ^ {\left(t ^ {\prime} , b _ {2} ^ {\prime}\right)} , \boldsymbol {\xi} _ {k} \right\rangle\right) \right]} _ {\mathrm {I} _ {3}}, \tag {33} \\ \end{array}
$$
where all the equalities are due to the network definition. Then we bound  $\mathrm{I}_1$ ,  $\mathrm{I}_2$  and  $\mathrm{I}_3$ .
For  $|\mathrm{I}_1|$ , we have the following upper bound by Lemma C.4:
$$
\begin{array}{l} | \mathrm {I} _ {1} | \leq | F _ {- y _ {k}} (\mathbf {W} _ {- y _ {k}} ^ {(t ^ {\prime}, b _ {2} ^ {\prime})}, \mathbf {x} _ {k}) | + | F _ {- y _ {i}} (\mathbf {W} _ {- y _ {i}} ^ {(t ^ {\prime}, b _ {1} ^ {\prime})}, \mathbf {x} _ {i}) | \\ = F _ {- y _ {k}} \left(\mathbf {W} _ {- y _ {k}} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)}, \mathbf {x} _ {k}\right) + F _ {- y _ {i}} \left(\mathbf {W} _ {- y _ {i}} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)}, \mathbf {x} _ {i}\right) \\ \leq 1. \tag {34} \\ \end{array}
$$
For  $|\mathrm{I}_2|$ , we have the following upper bound:
$$
\begin{array}{l} \left| \mathrm {I} _ {2} \right| \leq \max  \left\{\frac {1}{m} \sum_ {r = 1} ^ {m} \sigma \left(\left\langle \mathbf {w} _ {y _ {i}, r} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)}, y _ {i} \cdot \boldsymbol {\mu} \right\rangle\right), \frac {1}{m} \sum_ {r = 1} ^ {m} \sigma \left(\left\langle \mathbf {w} _ {y _ {k}, r} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)}, y _ {k} \cdot \boldsymbol {\mu} \right\rangle\right) \right\} \\ \leq 3 \max  \left\{\left| \left\langle \mathbf {w} _ {y _ {i}, r} ^ {(0, 0)}, y _ {i} \cdot \boldsymbol {\mu} \right\rangle \right|, \left| \left\langle \mathbf {w} _ {y _ {k}, r} ^ {(0, 0)}, y _ {k} \cdot \boldsymbol {\mu} \right\rangle \right|, \gamma_ {j, r} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)}, \gamma_ {j, r} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)}, \mathrm {S N R} \sqrt {\frac {3 2 \log (6 n / \delta)}{d}} n \alpha \right\} \\ \leq 3 \max  \left\{\beta , C ^ {\prime} \hat {\gamma} \alpha , \mathrm {S N R} \sqrt {\frac {3 2 \log (6 n / \delta)}{d}} n \alpha \right\} \\ \leq 0. 2 5, \tag {35} \\ \end{array}
$$
where the second inequality is due to (30), the second inequality is due to the definition of  $\beta$  and (29), the third inequality is due to Condition 3.1 and (25).
For  $\mathrm{I}_3$ , we have the following upper bound
$$
\begin{array}{l} \mathrm {I} _ {3} = \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ (P - 1) \sigma \left(\left\langle \mathbf {w} _ {y _ {i}, r} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)}, \boldsymbol {\xi} _ {i} \right\rangle\right) - (P - 1) \sigma \left(\left\langle \mathbf {w} _ {y _ {k}, r} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)}, \boldsymbol {\xi} _ {k} \right\rangle\right) \right] \\ \leq \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ (P - 1) \left\langle \mathbf {w} _ {y _ {i}, r} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)}, \boldsymbol {\xi} _ {i} \right\rangle - (P - 1) \left\langle \mathbf {w} _ {y _ {k}, r} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)}, \boldsymbol {\xi} _ {k} \right\rangle \right] + 0. 2 5 \\ \leq \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {(t ^ {\prime}, b _ {1} ^ {\prime})} - \bar {\rho} _ {y _ {k}, r, k} ^ {(t ^ {\prime}, b _ {2} ^ {\prime})} + 1 0 \sqrt {\frac {\log (6 n ^ {2} / \delta)}{d}} n \alpha \right] + 0. 2 5 \\ \leq \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)} - \bar {\rho} _ {y _ {k}, r, k} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)} \right] + 0. 5, \tag {36} \\ \end{array}
$$
where the first inequality is due to Lemma C.5, the second inequality is due to Lemma C.3, the third inequality is due to  $5\sqrt{\log(6n^2 / \delta) / dn\alpha} \leq 1 / 8$  according to Condition 3.1.
Similarly, we have the following lower bound
$$
\begin{array}{l} \mathrm {I} _ {3} = \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ (P - 1) \sigma \left(\left\langle \mathbf {w} _ {y _ {i}, r} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)}, \boldsymbol {\xi} _ {i} \right\rangle\right) - (P - 1) \sigma \left(\left\langle \mathbf {w} _ {y _ {k}, r} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)}, \boldsymbol {\xi} _ {k} \right\rangle\right) \right] \\ \geq \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ (P - 1) \left\langle \mathbf {w} _ {y _ {i}, r} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)}, \boldsymbol {\xi} _ {i} \right\rangle - (P - 1) \left\langle \mathbf {w} _ {y _ {k}, r} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)}, \boldsymbol {\xi} _ {k} \right\rangle \right] - 0. 2 5 \\ \geq \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {(t ^ {\prime}, b _ {1} ^ {\prime})} - \bar {\rho} _ {y _ {k}, r, k} ^ {(t ^ {\prime}, b _ {2} ^ {\prime})} - 1 0 \sqrt {\frac {\log (6 n ^ {2} / \delta)}{d}} n \alpha \right] - 0. 2 5 \\ \geq \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)} - \bar {\rho} _ {y _ {k}, r, k} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)} \right] - 0. 5, \tag {37} \\ \end{array}
$$
By plugging (34)-(36) into (33), we have
$$
\begin{array}{l} y _ {i} \cdot f \left(\mathbf {W} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)}, \mathbf {x} _ {i}\right) - y _ {k} \cdot f \left(\mathbf {W} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)}, \mathbf {x} _ {k}\right) \leq \left| I _ {1} \right| + \left| I _ {2} \right| + I _ {3} \\ \leq \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {(t ^ {\prime}, b _ {1} ^ {\prime})} - \bar {\rho} _ {y _ {k}, r, k} ^ {(t ^ {\prime}, b _ {2} ^ {\prime})} \right] + 1. 7 5, \\ \end{array}
$$
$$
\begin{array}{l} y _ {i} \cdot f \left(\mathbf {W} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)}, \mathbf {x} _ {i}\right) - y _ {k} \cdot f \left(\mathbf {W} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)}, \mathbf {x} _ {k}\right) \geq - \left| \mathrm {I} _ {1} \right| - \left| \mathrm {I} _ {2} \right| + \mathrm {I} _ {3} \\ \geq \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)} - \bar {\rho} _ {y _ {k}, r, k} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)} \right] - 1. 7 5, \\ \end{array}
$$
$$
\left| y _ {i} \cdot f \left(\mathbf {W} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)}, \mathbf {x} _ {i}\right) - y _ {k} \cdot f \left(\mathbf {W} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)}, \mathbf {x} _ {k}\right) - \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ \bar {p} _ {y _ {i}, r, i} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)} - \bar {p} _ {y _ {k}, r, k} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)} \right] \right| \leq 1. 7 5. \tag {38}
$$
Therefore, the second condition immediately follows from the first condition.
Then, we prove the first condition holds for  $(\widetilde{t},\widetilde{b})$ . Recall from Lemma C.1 that
$$
\bar {\rho} _ {j, r, i} ^ {(t, b + 1)} = \bar {\rho} _ {j, r, i} ^ {(t, b)} - \frac {\eta (P - 1) ^ {2}}{B m} \cdot \ell_ {i} ^ {\prime (t, b)} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\xi} _ {i} \rangle) \cdot \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2} \cdot \mathbb {1} (y _ {i} = j)   \mathbb {1} (i \in \mathcal {I} _ {t, b})
$$
for all  $j \in \{\pm 1\}$ ,  $r \in [m]$ ,  $i \in [n]$ ,  $(0,0) \leq (t,b) < [T^{*},0]$ . It follows that
$$
\sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {(t, b + 1)} - \bar {\rho} _ {y _ {k}, r, k} ^ {(t, b + 1)} \right]
$$
$$
\begin{array}{l} = \sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {(t, b)} - \bar {\rho} _ {y _ {k}, r, k} ^ {(t, b)} \right] - \frac {\eta (P - 1) ^ {2}}{B m} \cdot \left(\left| \widetilde {S} _ {i} ^ {(t, b)} \right| \ell_ {i} ^ {\prime (t, b)} \cdot \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2} \mathbb {1} (i \in \mathcal {I} _ {t, b}) \right. \\ \left. - \left| \widetilde {S} _ {k} ^ {(t, b)} \right| \ell_ {k} ^ {\prime (t, b)} \cdot \left\| \boldsymbol {\xi} _ {k} \right\| _ {2} ^ {2} \mathbb {1} (k \in \mathcal {I} _ {t, b})\right), \\ \end{array}
$$
for all  $i, k \in [n]$  and  $0 \leq t \leq T^{*}$ ,  $b < H$ .
If  $\widetilde{b} \in \{1,2,\dots ,H - 1\}$ , then the first statement for  $(t^{\prime},b^{\prime}) = (\widetilde{t},\widetilde{b})$  and for the last  $(t^{\prime},b^{\prime}) < (\widetilde{t},\widetilde{b})$  are the same. Otherwise, if  $\widetilde{b} = 0$ , we consider two separate cases:  $\sum_{r = 1}^{m}\left[\overline{\rho}_{y_i,r,i}^{(\widetilde{t} -1,0)} - \overline{\rho}_{y_k,r,k}^{(\widetilde{t} -1,0)}\right] \leq 0.9\kappa$  and  $\sum_{r = 1}^{m}\left[\overline{\rho}_{y_i,r,i}^{(\widetilde{t} -1,0)} - \overline{\rho}_{y_k,r,k}^{(\widetilde{t} -1,0)}\right] > 0.9\kappa$ .
When  $\sum_{r=1}^{m}\left[\overline{\rho}_{y_i,r,i}^{(\widetilde{t}-1,0)}-\overline{\rho}_{y_k,r,k}^{(\widetilde{t}-1,0)}\right] \leq 0.9\kappa$ , we have
$$
\begin{array}{l} \sum_ {r = 1} ^ {m} \left[ \overline {{\rho}} _ {y _ {i}, r, i} ^ {(\widetilde {t}, 0)} - \overline {{\rho}} _ {y _ {k}, r, k} ^ {(\widetilde {t}, 0)} \right] \\ = \sum_ {r = 1} ^ {m} \left[ \overline {{\rho}} _ {y _ {i}, r, i} ^ {(\widetilde {t} - 1, 0)} - \overline {{\rho}} _ {y _ {k}, r, k} ^ {(\widetilde {t} - 1, 0)} \right] - \frac {\eta (P - 1) ^ {2}}{B m} \cdot \left(\left| \widetilde {S} _ {i} ^ {(\widetilde {t} - 1, b _ {i} ^ {(\widetilde {t} - 1)})} \right| \ell_ {i} ^ {\prime (\widetilde {t} - 1, b _ {i} ^ {(\widetilde {t} - 1)})} \cdot \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2} \right. \\ \left. - \left| \widetilde {S} _ {k} ^ {\left(\widetilde {t} - 1, b _ {k} ^ {\left(\widetilde {t} - 1\right)}\right)} \right| \ell_ {k} ^ {\prime \left(\widetilde {t} - 1, b _ {k} ^ {\left(\widetilde {t} - 1\right)}\right)} \cdot \| \boldsymbol {\xi} _ {k} \| _ {2} ^ {2}\right) \\ \leq \sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {(\tilde {t} - 1, 0)} - \bar {\rho} _ {y _ {k}, r, k} ^ {(\tilde {t} - 1, 0)} \right] - \frac {\eta (P - 1) ^ {2}}{B m} \cdot \left| \widetilde {S} _ {i} ^ {(\tilde {t} - 1, b _ {i} ^ {(\tilde {t} - 1)})} \right| \ell_ {i} ^ {\prime (\tilde {t} - 1, b _ {i} ^ {(\tilde {t} - 1)})} \cdot \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2} \\ \leq \sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {(\tilde {t} - 1, 0)} - \bar {\rho} _ {y _ {k}, r, k} ^ {(\tilde {t} - 1, 0)} \right] + \frac {\eta (P - 1) ^ {2}}{B} \cdot \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2} \\ <   0. 9 \kappa + 0. 1 \kappa \\ = \kappa , \\ \end{array}
$$
where the first inequality is due to  $\ell_i^{\prime}(\widetilde{t} -1,b_i^{(\widetilde{t} -1)}) < 0$ ; the second inequality is due to  $|S_{i}^{(\widetilde{t} -1,b_{i}^{(\widetilde{t} -1)})}|\leq m$  and  $-\ell_i^{\prime}(\widetilde{t} -1,b_i^{(\widetilde{t} -1)}) < 1$ ; the third inequality is due to Condition 3.1.
On the other hand, for when  $\sum_{r=1}^{m}\left[\overline{\rho}_{y_i,r,i}^{(\widetilde{t}-1,0)}-\overline{\rho}_{y_k,r,k}^{(\widetilde{t}-1,0)}\right]>0.9\kappa$ , we have from the (38) that
$$
\begin{array}{l} y _ {i} \cdot f \left(\mathbf {W} ^ {\left(\widetilde {t} - 1, b _ {i} ^ {\left(\widetilde {t} - 1\right)}\right)}, \mathbf {x} _ {i}\right) - y _ {k} \cdot f \left(\mathbf {W} ^ {\left(\widetilde {t} - 1, b _ {k} ^ {\left(\widetilde {t} - 1\right)}\right)}, \mathbf {x} _ {k}\right) \\ \geq \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {(\tilde {t} - 1, b _ {i} ^ {(\tilde {t} - 1)})} - \bar {\rho} _ {y _ {k}, r, k} ^ {(\tilde {t} - 1, b _ {k} ^ {(\tilde {t} - 1)})} \right] - 1. 7 5 \\ \geq \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {(\tilde {t} - 1, 0)} - \bar {\rho} _ {y _ {k}, r, k} ^ {(\tilde {t} - 1, 0)} \right] - 0. 1 \kappa - 1. 7 5 \\ > 0. 9 \kappa - 0. 1 \kappa - 0. 5 4 \kappa \\ = 0. 2 6 \kappa , \tag {39} \\ \end{array}
$$
where the second inequality is due to  $\kappa = 10$ . Thus, according to Lemma C.6, we have
$$
\frac {\ell_ {i} ^ {\prime} (\widetilde {t} - 1 , b _ {i} ^ {(\widetilde {t} - 1)})}{\ell_ {k} ^ {\prime} (\widetilde {t} - 1 , b _ {k} ^ {(\widetilde {t} - 1)})} \leq \exp \left(y _ {k} \cdot f (\mathbf {W} ^ {(\widetilde {t} - 1, b _ {k} ^ {(\widetilde {t} - 1)})}, \mathbf {x} _ {k}) - y _ {i} \cdot f (\mathbf {W} ^ {(\widetilde {t} - 1, b _ {i} ^ {(\widetilde {t} - 1)})}, \mathbf {x} _ {i})\right) \leq \exp (- 0. 2 6 \kappa).
$$
Since  $S_{i}^{(\widetilde{t} -1,0)}\subseteq \widetilde{S}_{i}^{(\widetilde{t} -1,b_{i}^{(\widetilde{t} -1)})}$ , we have  $\left|\widetilde{S}_k^{(\widetilde{t} -1,b_k^{(\widetilde{t} -1)})}\right|\geq 0.8\Phi (-1)m$  according to the fourth condition. Also we have that  $|S_{i}^{(\widetilde{t} -1,b_{i}^{(\widetilde{t} -1)})}| \leq m$ . It follows that
$$
\frac {\left| S _ {i} ^ {(\widetilde {t} - 1 , b _ {i} ^ {(\widetilde {t} - 1)})} \right| \ell_ {i} ^ {\prime (\widetilde {t} - 1 , b _ {i} ^ {(\widetilde {t} - 1)})}}{\left| S _ {k} ^ {(\widetilde {t} - 1 , b _ {k} ^ {(\widetilde {t} - 1)})} \right| \ell_ {k} ^ {\prime (\widetilde {t} - 1 , b _ {k} ^ {(\widetilde {t} - 1)})}} \leq \frac {\exp (- 0 . 2 6 \kappa)}{0 . 8 \Phi (- 1)} <   0. 8.
$$
According to Lemma B.1, under event  $\mathcal{E}_{\mathrm{prelim}}$ , we have
$$
\left| \left\| \boldsymbol {\xi} _ {i} \right\| _ {2} ^ {2} - d \cdot \sigma_ {p} ^ {2} \right| = O \left(\sigma_ {p} ^ {2} \cdot \sqrt {d \log (6 n / \delta)}\right), \forall i \in [ n ].
$$
Note that  $d = \Omega (\log (6n / \delta))$  from Condition 3.1, it follows that
$$
| S _ {i} ^ {(\widetilde {t}, b _ {i} ^ {(\widetilde {t} - 1)})} | (- \ell_ {i} ^ {\prime (\widetilde {t}, b _ {i} ^ {(\widetilde {t} - 1)})}) \cdot \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2} <   | S _ {k} ^ {(\widetilde {t}, b _ {k} ^ {(\widetilde {t} - 1)})} | (- \ell_ {k} ^ {\prime (\widetilde {t}, b _ {k} ^ {(\widetilde {t} - 1)})}) \cdot \| \boldsymbol {\xi} _ {k} \| _ {2} ^ {2}.
$$
$$
\sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {(\widetilde {t}, 0)} - \bar {\rho} _ {y _ {k}, r, k} ^ {(\widetilde {t}, 0)} \right] \leq \sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {(\widetilde {t} - 1, 0)} - \bar {\rho} _ {y _ {k}, r, k} ^ {(\widetilde {t} - 1, 0)} \right] \leq \kappa ,
$$
which completes the proof of the first hypothesis at iteration  $(t', b') = (\widetilde{t}, \widetilde{b})$ . Next, by applying the approximation in (38), we are ready to verify the second hypothesis at iteration  $(\widetilde{t}, \widetilde{b})$ . In fact, for any  $(t', b_1'), (t', b_2') \leq (\widetilde{t}, \widetilde{b})$ , we have
$$
\begin{array}{l} y _ {i} \cdot f \left(\mathbf {W} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)}, \mathbf {x} _ {i}\right) - y _ {k} \cdot f \left(\mathbf {W} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)}, \mathbf {x} _ {k}\right) \leq \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)} - \bar {\rho} _ {y _ {k}, r, k} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)} \right] + 1. 7 5 \\ \leq \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ \bar {\rho} _ {y _ {i}, r, i} ^ {(t ^ {\prime}, 0)} - \bar {\rho} _ {y _ {k}, r, k} ^ {(t ^ {\prime}, 0)} \right] + 0. 1 \kappa + 1. 7 5 \\ \leq C _ {1}, \\ \end{array}
$$
where the first inequality is by (38); the last inequality is by induction hypothesis and taking  $\kappa$  as 10 and  $C_1$  as 5.
And the third hypothesis directly follows by noting that, for any  $(t', b_1'), (t', b_2') \leq (\widetilde{t}, \widetilde{b})$ ,
$$
\frac {\ell_ {i} ^ {\prime \left(t ^ {\prime} , b _ {1} ^ {\prime}\right)}}{\ell_ {k} ^ {\prime \left(t ^ {\prime} , b _ {2} ^ {\prime}\right)}} \leq \exp \left(y _ {k} \cdot f \left(\mathbf {W} ^ {\left(t ^ {\prime}, b _ {1} ^ {\prime}\right)}, \mathbf {x} _ {k}\right) - y _ {i} \cdot f \left(\mathbf {W} ^ {\left(t ^ {\prime}, b _ {2} ^ {\prime}\right)}, \mathbf {x} _ {i}\right)\right) \leq \exp (C _ {1}) = C _ {2}.
$$
For the fourth hypothesis, If  $\widetilde{b} \in \{1, 2, \dots, H - 1\}$ , then the first statement for  $(t', b') = (\widetilde{t}, \widetilde{b})$  and for the last  $(t', b') < (\widetilde{t}, \widetilde{b})$  are the same. Otherwise, if  $\widetilde{b} = 0$ , according to the gradient descent rule, we have
$$
- \frac {\eta}{B m} \cdot \underbrace {\sum_ {\widetilde {b} = 0} ^ {H - 1} \sum_ {i ^ {\prime} \in \mathcal {I} _ {\widetilde {t} - 1 , b ^ {\prime}}} \ell_ {i ^ {\prime}} ^ {(\widetilde {t} - 1 , b ^ {\prime})} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {y _ {i} , r} ^ {(\widetilde {t} - 1 , b ^ {\prime})} , y _ {i ^ {\prime}} \pmb {\mu} \rangle) \cdot \langle y _ {i ^ {\prime}} \pmb {\mu} , \pmb {\xi} _ {i} \rangle y _ {i ^ {\prime}} ,} _ {I _ {6}},
$$
for any  $r \in S_i^{(\widetilde{t} - 1, 0)}$ , where the last equality is by  $\langle \mathbf{w}_{y_i, r}^{(\widetilde{t} - 1, b_i^{(\widetilde{t} - 1)})}, \pmb{\xi}_i \rangle > 0$ . Then we respectively estimate  $\mathrm{I}_4, \mathrm{I}_5, \mathrm{I}_6$ . For  $\mathrm{I}_4$ , according to Lemma B.1, we have
$$
- \mathrm {I} _ {4} \geq | \ell_ {i} ^ {(\widetilde {t} - 1, b _ {i} ^ {(\widetilde {t} - 1)})} | \cdot \sigma_ {p} ^ {2} d / 2.
$$
For  $\mathrm{I}_5$ , we have following upper bound
$$
\begin{array}{l} \left| \mathrm {I} _ {5} \right| \leq \sum_ {b ^ {\prime} = 0} ^ {H - 1} \sum_ {i ^ {\prime} \in \mathcal {I} _ {\tilde {t} - 1, b ^ {\prime}}} \left| \ell_ {i ^ {\prime}} ^ {\left(\tilde {t} - 1, b ^ {\prime}\right)} \right| \cdot \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {y _ {i}, r} ^ {\left(\tilde {t} - 1, b ^ {\prime}\right)}, \boldsymbol {\xi} _ {i ^ {\prime}} \right\rangle\right) \cdot \left| \left\langle \boldsymbol {\xi} _ {i ^ {\prime}}, \boldsymbol {\xi} _ {i} \right\rangle \right| \mathbb {1} \left(i ^ {\prime} \neq i\right) \\ \leq \sum_ {b ^ {\prime} = 0} ^ {H - 1} \sum_ {i ^ {\prime} \in \mathcal {I} _ {\tilde {t} - 1, b ^ {\prime}}} | \ell_ {i ^ {\prime}} ^ {(\widetilde {t} - 1, b ^ {\prime})} | \cdot | \langle \boldsymbol {\xi} _ {i ^ {\prime}}, \boldsymbol {\xi} _ {i} \rangle | \mathbb {1} (i ^ {\prime} \neq i) \\ \leq \sum_ {b ^ {\prime} = 0} ^ {H - 1} \sum_ {i ^ {\prime} \in \mathcal {I} _ {\tilde {t} - 1, b ^ {\prime}}} | \ell_ {i ^ {\prime}} ^ {(\tilde {t} - 1, b ^ {\prime})} | \cdot 2 \sigma_ {p} ^ {2} \cdot \sqrt {d \log (6 n ^ {2} / \delta)} \\ \leq n C _ {2} \left| \ell_ {i} ^ {\left(\widetilde {t} - 1, b _ {i} ^ {\left(\widetilde {t} - 1\right)}\right)} \right| \cdot 2 \sigma_ {p} ^ {2} \cdot \sqrt {d \log (6 n ^ {2} / \delta)}, \\ \end{array}
$$
where the first inequality is due to triangle inequality, the second inequality is due to  $\sigma'(z) \in \{0,1\}$ , the third inequality is due to Lemma B.1, the forth inequality is due to the third hypothesis at epoch  $\widetilde{t} - 1$ .
For  $\mathrm{I}_6$ , we have following upper bound
$$
\begin{array}{l} \left| \mathrm {I} _ {6} \right| \leq \sum_ {b ^ {\prime} = 0} ^ {H - 1} \sum_ {i ^ {\prime} \in \mathcal {I} _ {\hat {i} - 1, b ^ {\prime}}} \left| \ell_ {\hat {i} ^ {\prime}} ^ {\left(\tilde {t} - 1, b ^ {\prime}\right)} \right| \cdot \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {y _ {i}, r} ^ {\left(\tilde {t} - 1, b ^ {\prime}\right)}, y _ {i ^ {\prime}} \boldsymbol {\mu} \right\rangle\right) \cdot \left| \left\langle y _ {i ^ {\prime}} \boldsymbol {\mu}, \boldsymbol {\xi} _ {i} \right\rangle \right| \\ \leq \sum_ {b ^ {\prime} = 0} ^ {H - 1} \sum_ {i ^ {\prime} \in \mathcal {I} _ {\tilde {t} - 1, b ^ {\prime}}} | \ell_ {i ^ {\prime}} ^ {(\tilde {t} - 1, b ^ {\prime})} | | \langle y _ {i ^ {\prime}} \boldsymbol {\mu}, \boldsymbol {\xi} _ {i} \rangle | \\ \leq n C _ {2} \left| \ell_ {i} ^ {\left(\widetilde {t} - 1, b _ {i} ^ {\left(\widetilde {t} - 1\right)}\right)} \right| \cdot \| \boldsymbol {\mu} \| _ {2} \sigma_ {p} \sqrt {2 \log (6 n / \delta)}, \\ \end{array}
$$
where the first inequality is by triangle inequality; the second inequality is due to  $\sigma'(z) \in \{0, 1\}$ ; the third inequality is by Lemma B.1; the last inequality is due to the third hypothesis at epoch  $\tilde{t} - 1$ .
Since  $d \geq \max \{32C_2^2 n^2 \cdot \log (6n^2 / \delta), 4C_2n\|\pmb{\mu}\|_{\sigma_p}^{-1}\sqrt{2\log(6n / \delta)}\}$ , we have  $-(P - 1)\mathrm{I}_4 \geq \max \{(P - 1)|\mathrm{I}_5| / 2, |\mathrm{I}_6| / 2\}$  and hence  $-(P - 1)\mathrm{I}_4 \geq (P - 1)|\mathrm{I}_5| + |\mathrm{I}_6|$ . It follows that
$$
\langle \mathbf {w} _ {y _ {i}, r} ^ {(\widetilde {t}, 0)}, \pmb {\xi} _ {i} \rangle \geq \langle \mathbf {w} _ {y _ {i}, r} ^ {(\widetilde {t} - 1, 0)}, \pmb {\xi} _ {i} \rangle > \sigma_ {0} \sigma_ {p} \sqrt {d} / \sqrt {2},
$$
for any  $r \in S_i^{(\widetilde{t} - 1, 0)}$ . Therefore,  $S_i^{(0,0)} \subseteq S_i^{(\widetilde{t} - 1, 0)} \subseteq S_i^{(\widetilde{t}, 0)}$ . And it directly follows by Lemma B.3 that  $|S_i^{(\widetilde{t}, 0)}| \geq 0.8m\Phi(-1)$ ,  $\forall i \in [n]$ .
For the fifth hypothesis, similar to the proof of the fourth hypothesis, we also have