SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \| \boldsymbol {h} _ {t} \| \leq \frac {1}{N} \sum_ {i = 1} ^ {N} \| \mathbf {P} _ {t} - \mathbf {P} _ {t} ^ {i} \| \| \nabla_ {\boldsymbol {w}} L (\boldsymbol {w} _ {t} ^ {i}, \boldsymbol {a} _ {t} ^ {i}) \| \\ \stackrel {(i)} {\leq} 2 4 c \eta (\sqrt {I d} + I) \sqrt {\log (N d / \delta)} K _ {a} ^ {2} \cdot \frac {1}{N} \sum_ {i = 1} ^ {N} \| \nabla_ {\boldsymbol {w}} L (\boldsymbol {w} _ {t} ^ {i}, \boldsymbol {a} _ {t} ^ {i}) \| \\ \stackrel {(i i)} {\leq} 2 4 c \eta (\sqrt {I d} + I) \sqrt {\log (N d / \delta)} K _ {a} ^ {4}, \tag {D.11} \\ \end{array}
$$
where  $(i)$  holds due to (B.9) and  $(ii)$  is true because  $\| \nabla_{\boldsymbol{w}}L(\boldsymbol{w}_t^i,\boldsymbol{a}_t^i)\| \leq |(\boldsymbol{a}_t^i)^\top \boldsymbol{a}^*| \leq K_a^2$ . In addition, we also have
$$
\begin{array}{l} \boldsymbol {v} _ {t + 1} ^ {\top} \boldsymbol {w} ^ {*} = \boldsymbol {v} _ {t} ^ {\top} \boldsymbol {w} ^ {*} - \eta \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbf {P} _ {t} ^ {i} \nabla_ {\boldsymbol {w}} L (\boldsymbol {w} _ {t} ^ {i}, \boldsymbol {a} _ {t} ^ {i}; \mathbf {Z} _ {t} ^ {i}) \\ = \check {\boldsymbol {v}} _ {t + 1} ^ {\top} \boldsymbol {w} ^ {*} - \eta \boldsymbol {h} _ {t} ^ {\top} \boldsymbol {w} ^ {*} \\ = \boldsymbol {v} _ {t} ^ {\top} \boldsymbol {w} ^ {*} - \eta \left(\mathbf {P} _ {t} \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla_ {\boldsymbol {w}} L (\boldsymbol {w} _ {t}, \boldsymbol {a} _ {t}) + \boldsymbol {\xi} _ {t}\right) ^ {\top} \boldsymbol {w} ^ {*} - \eta \boldsymbol {h} _ {t} ^ {\top} \boldsymbol {w} ^ {*} \\ = \boldsymbol {v} _ {t} ^ {\top} \boldsymbol {w} ^ {*} + \frac {\eta}{2 \pi} \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {\pi - \phi_ {t} ^ {i}}{\| \boldsymbol {v} _ {t} \|} \left(\boldsymbol {a} _ {t} ^ {i}\right) ^ {\top} \boldsymbol {a} ^ {*} \sin^ {2} \phi_ {t} - \eta \boldsymbol {h} _ {t} ^ {\top} \boldsymbol {w} ^ {*} - \eta \boldsymbol {\xi} _ {t} ^ {\top} \boldsymbol {w} ^ {*}. \tag {D.12} \\ \end{array}
$$
$$
\tau_ {1, 2} = \frac {1}{\eta} \frac {2 \pi}{\pi - 1} \log \left(4 + \frac {2 0 (\pi + k - 1) \| \boldsymbol {a} ^ {*} \| ^ {2}}{\left(\mathbf {1} ^ {\top} \boldsymbol {a} ^ {*}\right) ^ {2} (\pi - g (\pi / 5))}\right) \leq \frac {1}{\eta} \frac {2 \pi}{\pi - 1} \log (4 + k ^ {2}). \tag {D.13}
$$
For any  $t_0 \leq s \leq t - 1 \leq t_0 + \tau_{1,2}$ , under the event  $\mathcal{A}_{t-1} \cap \mathcal{E}_{\mathrm{good}}$ , invoking (D.12) guarantees that,
$$
\cos \phi_ {s} = \frac {1}{\| \boldsymbol {v} _ {s} \|} \left\{\boldsymbol {v} _ {t _ {0}} ^ {\top} \boldsymbol {w} ^ {*} + \frac {\eta}{2 \pi} \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {m = t _ {0}} ^ {s - 1} \frac {\pi - \phi_ {m} ^ {i}}{\| \boldsymbol {v} _ {m} \|} (\boldsymbol {a} _ {m} ^ {i}) ^ {\top} \boldsymbol {a} ^ {*} \sin^ {2} \phi_ {m} - \eta \sum_ {m = t _ {0}} ^ {s - 1} (\boldsymbol {h} _ {m} ^ {\top} \boldsymbol {w} ^ {*} + \boldsymbol {\xi} _ {m} ^ {\top} \boldsymbol {w} ^ {*}) \right\}
$$
$$
\begin{array}{l} \stackrel {(i)} {\leq} 2 \cos \phi_ {t _ {0}} + 2 s \eta \gamma_ {a} + 2 \eta \sum_ {m = t _ {0}} ^ {s - 1} \max  _ {i} \left| \boldsymbol {a} _ {m} ^ {\top} \boldsymbol {a} ^ {*} - \left(\boldsymbol {a} _ {m} ^ {i}\right) ^ {\top} \boldsymbol {a} ^ {*} \right| + 2 \eta \left| \sum_ {m = t _ {0}} ^ {s - 1} \left(\boldsymbol {\xi} _ {m} ^ {\top} \boldsymbol {w} ^ {*} + \boldsymbol {h} _ {m} ^ {\top} \boldsymbol {w} ^ {*}\right) \right| \\ \stackrel {(i i)} {\leq} 2 \cos \phi_ {t _ {0}} + 2 \tau_ {1, 2} \eta \gamma_ {a} + 2 c \tau_ {1, 2} \eta^ {2} (I + \sqrt {k I}) \sqrt {\log (1 / \eta \delta)} K _ {a} ^ {2} + 2 \eta \max  _ {0 \leq \tau \leq \tau_ {1, 1}} \left| \sum_ {m = \tau} ^ {s - 1} (\boldsymbol {\xi} _ {m} ^ {\top} \boldsymbol {w} ^ {*} + \boldsymbol {h} _ {m} ^ {\top} \boldsymbol {w} ^ {*}) \right| \\ \stackrel {(i i i)} {\leq} 2 \cos \tilde {\phi} ^ {o} + \frac {6 4 \pi}{\pi + k - 1} \frac {(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}) ^ {2}}{k} \log (4 + k ^ {2}) + \frac {4 \pi}{\pi - 1} \log (4 + k ^ {2}) \cdot c \eta (I + \sqrt {k I}) \sqrt {\log (1 / \eta \delta)} K _ {a} ^ {2} \\ + \frac {4 \pi}{\pi - 1} \log (4 + k ^ {2}) \cdot 2 4 c \eta K _ {a} ^ {2} (\sqrt {I d} + I) \sqrt {\log (N d / \delta)} \\ + 2 c K _ {a} \sqrt {\frac {4 \pi}{\pi - 1} \log (4 + k ^ {2}) \cdot \frac {\eta \log (N d / \eta \delta)}{N}} \\ \end{array}
$$
$$
\stackrel {(i v)} {\leq} 2 \cos \tilde {\phi} ^ {o} + \frac {1 2 8 (\pi - 1)}{\pi + k - 1} \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{k} \log \left(4 + k ^ {2}\right) \leq 3 \cos \tilde {\phi} ^ {o}, \tag {D.14}
$$
where  $(i)$  holds due to  $\| \pmb{v}_s\| \geq 1 / 2$ ,  $\| \pmb{v}_{t_0}\| \geq 1 / 2$  and the definition of  $\mathcal{A}_{t - 1}$ ;  $(ii)$  comes from  $t_0\leq \tau_{1,1}$  and  $\mathcal{E}_1$ ;  $(iii)$  follows from  $\phi_{t_0}\geq \tilde{\phi}^l$ , and  $\mathcal{E}_2$ ; and  $(iv)$  holds due to the assumption (D.6). Further, under the event  $\mathcal{A}_{t - 1}\cap \mathcal{E}_{\mathrm{good}}$ , we have for any  $i\in [N]$  and  $t_0\leq s\leq t - 1\leq t_0 + \tau_{1,2} - 1$
$$
\begin{array}{l} \cos \phi_ {s} ^ {i} \leq 3 \cos \tilde {\phi} ^ {o} + 1 2 c \eta K _ {a} ^ {2} (\sqrt {I d} + I) \sqrt {\log (N d / \delta)} \\ \leq 4 \cos \tilde {\phi} ^ {o} = \frac {4}{5} \leq \cos (\pi / 5). \\ \end{array}
$$
Under the event  $\mathcal{A}_{t - 1}\cap \mathcal{E}_{\mathrm{good}}\cap \{\tau_{-}(\pi /2) - t_{0}\geq \tau_{1,2}\}$ , it implies that for any  $t_0\leq s\leq t - 1\leq t_0 + \tau_{1,2} - 1$
$$
\begin{array}{l} \frac {1}{N} \sum_ {i = 1} ^ {N} B _ {s} ^ {i} \geq \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {\eta}{2 \pi} \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{\| \mathbf {a} ^ {*} \| ^ {2}} \sum_ {l = 0} ^ {s - 1} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s - 1 - l} (\pi - g (\phi_ {l} ^ {i})) \\ \geq \frac {\pi - g (\pi / 5)}{\pi + k - 1} \frac {(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}) ^ {2}}{\| \mathbf {a} ^ {*} \| ^ {2}} \left[ 1 - \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s} \right]. \\ \end{array}
$$
Together with (D.1) and (D.3), we can guarantee that
$$
\begin{array}{l} \boldsymbol {a} _ {t} ^ {\top} \boldsymbol {a} ^ {*} = \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t - t _ {0}} \boldsymbol {a} _ {t _ {0}} ^ {\top} \boldsymbol {a} ^ {*} + \frac {1 - \left(\frac {2 \pi - \eta (k + \pi - 1)}{2 \pi - \eta (\pi - 1)}\right) ^ {t - t _ {0}}}{k} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t} A _ {t _ {0}} \\ + \frac {\eta \| \boldsymbol {a} ^ {*} \| ^ {2}}{2 \pi} \sum_ {s = t _ {0}} ^ {t - 1} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t - 1 - s} \frac {1}{N} \sum_ {i = 1} ^ {N} B _ {s} ^ {i} + S (\epsilon_ {t _ {0}: t - 1}) \\ \geq \frac {\pi - g (\pi / 5)}{\pi - 1} \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{\pi + k - 1} \left[ 1 - \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t - t _ {0}} \right] \\ - \frac {1 - \left(\frac {2 \pi - \eta (k + \pi - 1)}{2 \pi - \eta (\pi - 1)}\right) ^ {t - t _ {0}}}{k} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t - t _ {0}} [ \pi - g (\pi / 5) ] \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{\pi + k - 1} \\ - 5 \| \boldsymbol {a} ^ {*} \| ^ {2} \left(1 - \frac {\eta (\pi - 1)}{2 \pi}\right) ^ {t - t _ {0}} - 2 c \sqrt {\frac {\eta k \log (1 / \delta)}{N}} K _ {a} ^ {2}. \\ \end{array}
$$
Choosing  $t = t_0 + \tau_{1,2}$ , it holds that
$$
\begin{array}{l} \boldsymbol {a} _ {t _ {0} + \tau_ {1, 2}} ^ {\top} \boldsymbol {a} ^ {*} \geq \frac {3}{4} \frac {\pi - g (\pi / 5)}{\pi - 1} \frac {\left(\boldsymbol {1} ^ {\top} \boldsymbol {a} ^ {*}\right) ^ {2}}{\pi + k - 1} - 3 c \sqrt {\frac {\eta k \log (1 / \eta \delta)}{N}} K _ {a} ^ {2} \\ \geq \frac {\pi - g (\pi / 5)}{4 (\pi - 1)} \frac {(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}) ^ {2}}{\pi + k - 1} \\ \geq \frac {0 . 5}{4 (\pi - 1)} \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{\pi + k - 1} \\ \end{array}
$$
$$
\geq \frac {1 6 (\pi - 1) \left(\mathbf {1} ^ {\top} \boldsymbol {a} ^ {*}\right) ^ {2}}{k (\pi + k - 1)} = \gamma_ {a},
$$
where we used the assumptions (D.6) and  $k \geq 320(\pi - 1)^2$ .
Recalling that  $\mathcal{Z}_2 = \left\{\tau_{-}(\pi /2)\geq \tau_{-}(\tilde{\phi}^o) + \tau_{1,2}\right\}$ , hence we have verified that
$$
\begin{array}{l} \mathbb {P} \left(\mathcal {Z} _ {1} ^ {c} \cap \mathcal {Z} _ {2} \cap \left\{\tau_ {a} > \tau_ {1, 1} + \tau_ {1, 2} \right\}\right) \leq \mathbb {P} \left(\mathcal {Z} _ {1} ^ {c} \cap \mathcal {Z} _ {2} \cap \left\{\tau_ {a} > \tau_ {-} (\tilde {\phi} ^ {o}) + \tau_ {1, 2} \right\}\right) \\ = \mathbb {P} \left(\mathcal {Z} _ {1} ^ {c} \cap \mathcal {Z} _ {2} \cap \mathcal {A} _ {\tau_ {-} (\tilde {\phi} ^ {o}) + \tau_ {1, 2} - 1} \cap \left\{\boldsymbol {a} _ {t _ {0} + \tau_ {1, 2}} ^ {\top} \boldsymbol {a} ^ {*} \leq \gamma_ {a} \right\}\right) \\ \leq \mathbb {P} \left(\mathcal {A} _ {\tau_ {-} (\tilde {\phi} ^ {o}) + \tau_ {1, 2} - 1} \cap \mathcal {E} _ {\text {g o o d}} ^ {c}\right) \leq 5 \delta . \tag {D.15} \\ \end{array}
$$
If  $\tau_{1,2} \geq \tau_{-}(\pi/2)$ , we regard  $(\pmb{v}_{\tau_{-}(\pi/2)}, \pmb{a}_{\tau_{-}(\pi/2)})$  as the new initial point and step into the analysis of the next region.
Initial region:  $[\pi /2,\tilde{\phi}^u)$ . From now on, we assign  $t_0 = \tau_{-}(\pi /2)$  and consider the case  $\left\{\tau_{-}(\tilde{\phi}^{o})\leq \tau_{1,1}\right\} \cap \left\{t_{0} < \tau_{-}(\tilde{\phi}^{o}) + \tau_{1,2}\right\}$ . Take
$$
\tau_ {1, 3} = \frac {1}{\eta} \frac {2 \pi}{\pi - 1} \log \left(4 + \frac {2 4 (\pi - 1)}{\zeta_ {a}}\right) \leq \frac {1}{\eta} \frac {2 \pi}{\pi - 1} \log \left(4 + k ^ {2}\right), \tag {D.16}
$$
where the inequality holds due to our assumptions  $k \geq 320(\pi - 1)^2$  and (D.8). Similar to (D.14), for any  $t_0 \leq s \leq t - 1 \leq \widetilde{O}(\eta^{-2})$ , under the event  $\mathcal{A}_{t-1} \cap \mathcal{E}_{\mathrm{good}}$ , we have
$$
\begin{array}{l} \cos \phi_ {s} = \frac {1}{\| \boldsymbol {v} _ {s} \|} \left\{\boldsymbol {v} _ {t _ {0}} ^ {\top} \boldsymbol {w} ^ {*} + \frac {\eta}{2 \pi} \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {m = t _ {0}} ^ {s - 1} \frac {\pi - \phi_ {m} ^ {i}}{\| \boldsymbol {v} _ {m} \|} (\boldsymbol {a} _ {m} ^ {i}) ^ {\top} \boldsymbol {a} ^ {*} \sin^ {2} \phi_ {m} - \eta \sum_ {m = t _ {0}} ^ {s - 1} (\boldsymbol {h} _ {m} ^ {\top} \boldsymbol {w} ^ {*} + \boldsymbol {\xi} _ {m} ^ {\top} \boldsymbol {w} ^ {*}) \right\} \\ \stackrel {(i)} {\leq} 2 \tau_ {1, 3} \eta \gamma_ {a} + 2 c \tau_ {1, 3} \eta^ {2} (I + \sqrt {k I}) \sqrt {\log (1 / \eta \delta)} K _ {a} ^ {2} + 2 \eta \max  _ {\tau \leq \tau_ {1, 1} + \tau_ {1, 2}} \left| \sum_ {m = \tau} ^ {s - 1} (\boldsymbol {\xi} _ {m} ^ {\top} \boldsymbol {w} ^ {*} + \boldsymbol {h} _ {m} ^ {\top} \boldsymbol {w} ^ {*}) \right| \\ \stackrel {(i i)} {\leq} \frac {6 4 \pi}{\pi + k - 1} \frac {(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}) ^ {2}}{k} \log (4 + k ^ {2}) + \frac {4 \pi}{\pi - 1} \log (4 + k ^ {2}) \cdot c \eta (I + \sqrt {k I}) \sqrt {\log (1 / \eta \delta)} K _ {a} ^ {2} \\ + \frac {6 \pi}{\pi - 1} \log (4 + k ^ {2}) \cdot \left(2 4 c \eta K _ {a} ^ {2} (\sqrt {I d} + I) \sqrt {\log (N d / \delta)} + c K _ {a} \sqrt {\frac {\eta \log (N d / \eta \delta)}{N}}\right) \\ \leq \frac {1 2 8 \pi}{\pi + k - 1} \frac {\left(\mathbf {1} ^ {\top} \boldsymbol {a} ^ {*}\right) ^ {2}}{k} \log \left(4 + k ^ {2}\right), \tag {D.17} \\ \end{array}
$$
where  $(i)$  holds due to  $\phi_{t_0} \geq \pi / 2$  and the definition of  $\mathcal{A}_{t-1}$ ; and  $(ii)$  holds due to  $s - 1 - \tau \leq \widetilde{O}(\eta^{-2})$ . Under the event  $\mathcal{A}_{t-1} \cap \mathcal{E}_{\mathrm{good}}$ , we can guarantee that
$$
\begin{array}{l} \operatorname * {m a x} _ {i} \cos \phi_ {s} ^ {i} \leq \cos \phi_ {s} + \operatorname * {m a x} _ {i} \left| \cos \phi_ {s} - \cos \phi_ {s} ^ {i} \right| \\ \leq \frac {1 2 8 \pi}{\pi + k - 1} \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{k} \log (4 + k ^ {2}) + 1 2 c \eta K _ {a} ^ {2} (\sqrt {I d} + I) \sqrt {\log (N d / \delta)} \\ \leq \frac {1 4 4 \pi}{\pi + k - 1} \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{k} \log (4 + k ^ {2}) \\ = \cos \tilde {\phi} ^ {l}, \\ \end{array}
$$
where we used the assumption (D.6). It implies that  $\min_i\phi_s^i\geq \tilde{\phi}^l$  under the event  $\mathcal{A}_{t - 1}\cap \mathcal{E}_{\mathrm{good}}$ . In addition, we notice that  $\cos \tilde{\phi}^l = -\cos \tilde{\phi}^u$ , which implies  $\tilde{\phi}^l +\tilde{\phi}^u = \pi$  and  $\sin \tilde{\phi}^l = \sin \tilde{\phi}^u$ . Under the event  $\mathcal{A}_{t - 1}\cap \mathcal{E}_{\mathrm{good}}\cap \{t - 1\leq \min \{\tau_{-}(\tilde{\phi}^{u}),\widetilde{O} (\eta^{-2})\} \}$ , for any  $s\leq t - 1$ , we have
$$
\frac {1}{N} \sum_ {i = 1} ^ {N} \left\{g (\boldsymbol {\phi} _ {s} ^ {i}) + \frac {\eta}{2 \pi} \frac {(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}) ^ {2}}{\| \mathbf {a} ^ {*} \| ^ {2}} \sum_ {l = t _ {0}} ^ {s - 1} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s - 1 - l} (1 - g (\boldsymbol {\phi} _ {l} ^ {i})) \right\}
$$
$$
\begin{array}{l} \geq g \left(\tilde {\phi} ^ {u}\right) + \frac {\eta}{2 \pi} \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{\| \mathbf {a} ^ {*} \| ^ {2}} \sum_ {l = t _ {0}} ^ {s - 1} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s - 1 - l} \left(1 - g \left(\tilde {\phi} ^ {l}\right)\right) \\ \geq g \left(\tilde {\phi} ^ {u}\right) + \frac {\eta k}{2 \pi} \sum_ {l = t _ {0}} ^ {s - 1} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s - 1 - l} \left(1 - g \left(\tilde {\phi} ^ {l}\right)\right) \\ = g \left(\tilde {\phi} ^ {u}\right) + \frac {k \left(1 - g \left(\tilde {\phi} ^ {l}\right)\right)}{\pi + k - 1} \left[ 1 - \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s} \right] \\ \geq g \left(\tilde {\phi} ^ {u}\right) + 1 - g \left(\tilde {\phi} ^ {l}\right) \\ = (\pi - \tilde {\phi} ^ {u}) \cos \tilde {\phi} ^ {u} + \sin \tilde {\phi} ^ {u} - (\pi - \tilde {\phi} ^ {l}) \cos \tilde {\phi} ^ {l} - \sin \tilde {\phi} ^ {l} + 1 \\ = (2 \pi - \tilde {\phi} ^ {u} - \tilde {\phi} ^ {l}) \cos \tilde {\phi} ^ {u} + 1 \\ = - \frac {1 4 4 \pi^ {2}}{\pi + k - 1} \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{k} \log \left(4 + k ^ {2}\right) + 1 \geq 0, \tag {D.18} \\ \end{array}
$$
where the last inequality holds due to the condition (D.7). Together with the definition of  $B_s^i$ , we have
$$
\begin{array}{l} \frac {\mathbb {1} _ {\mathcal {A} _ {t - 1}}}{N} \sum_ {i = 1} ^ {N} B _ {s} ^ {i} = \frac {\mathbb {1} _ {\mathcal {A} _ {t - 1}}}{N} \sum_ {i = 1} ^ {N} \left\{(g \left(\phi_ {s} ^ {i}\right) - 1) + \frac {\eta}{2 \pi} \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{\| \mathbf {a} ^ {*} \| ^ {2}} \sum_ {l = 0} ^ {s - 1} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s - 1 - l} \left(\pi - g \left(\phi_ {l} ^ {i}\right)\right) \right\} \\ \geq \frac {\eta}{2 \pi} \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{\| \mathbf {a} ^ {*} \| ^ {2}} \sum_ {l = 0} ^ {s - 1} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s - 1 - l} (\pi - 1) \\ = \underbrace {\frac {\pi - 1}{\pi + k - 1} \frac {\left(\mathbf {1} ^ {\top} \boldsymbol {a} ^ {*}\right) ^ {2}}{\left\| \boldsymbol {a} ^ {*} \right\| ^ {2}} - 1} _ {\zeta_ {a}} - \frac {\pi - 1}{\pi + k - 1} \frac {\left(\mathbf {1} ^ {\top} \boldsymbol {a} ^ {*}\right) ^ {2}}{\left\| \boldsymbol {a} ^ {*} \right\| ^ {2}} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s}. \\ \end{array}
$$
We know  $\zeta_{a} > 0$  due to the condition (D.8). Under  $\mathcal{A}_{t - 1}\cap \mathcal{E}_{\mathrm{good}}\cap \{t - 1\leq \min \{\tau_{-}(\tilde{\phi}^{u}),\widetilde{O} (\eta^{-2})\} \}$ , we can guarantee
$$
\begin{array}{l} \boldsymbol {a} _ {t} ^ {\top} \boldsymbol {a} ^ {*} \geq \frac {\zeta_ {a} \| \boldsymbol {a} ^ {*} \| ^ {2}}{\pi - 1} \left[ 1 - \left(1 - \frac {\eta (\pi - 1)}{2 \pi}\right) ^ {t} \right] - 5 K _ {a} ^ {2} \left(1 - \frac {\eta (\pi - 1)}{2 \pi}\right) ^ {t} \\ - 2 c K _ {a} ^ {2} \sqrt {\frac {\eta \log (1 / \eta \delta)}{N}} - \frac {1 - \left(\frac {2 \pi - \eta (k + \pi - 1)}{2 \pi - \eta (\pi - 1)}\right) ^ {t}}{k} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t} \frac {(\pi - 1) \left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{\pi + k - 1} \\ \geq \frac {\zeta_ {a} \| \boldsymbol {a} ^ {*} \| ^ {2}}{\pi - 1} - \left(1 - \frac {\eta (\pi - 1)}{2 \pi}\right) ^ {t} \left(6 \| \boldsymbol {a} ^ {*} \| ^ {2} + \frac {\zeta_ {a} \| \boldsymbol {a} ^ {*} \| ^ {2}}{\pi - 1}\right) - 2 c K _ {a} ^ {2} \sqrt {\frac {\eta k \log (1 / \eta \delta)}{N}}, \\ \end{array}
$$
where we used  $|\pmb{a}_{t_0}^\top \pmb{a}^*| \leq \| \pmb{a}^*\|^2 \cdot K_a = 5\|\pmb{a}^*\|^2$ . Now let  $t = t_0 + \tau_{1,3}$ , under  $\mathcal{A}_{t-1} \cap \mathcal{E}_{\mathrm{good}} \cap \{t_0 + \tau_{1,3} - 1 \leq \tau_{-}(\tilde{\phi}^u)\}$ , we have
$$
\begin{array}{l} \boldsymbol {a} _ {t _ {0} + \tau_ {1, 3}} ^ {\top} \boldsymbol {a} ^ {*} \geq \frac {3}{4} \frac {\zeta_ {a} \| \boldsymbol {a} ^ {*} \| ^ {2}}{\pi - 1} - 2 c K _ {a} ^ {2} \sqrt {\frac {\eta k \log (1 / \eta \delta)}{N}} \\ \geq \frac {2 4 (\pi - 1) \left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{k (\pi + k - 1)} - 2 c K _ {a} ^ {2} \sqrt {\frac {\eta k \log (1 / \eta \delta)}{N}} \\ \geq \frac {1 6 (\pi - 1) \left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{k (\pi + k - 1)} = \gamma_ {a}, \tag {D.19} \\ \end{array}
$$
where we used the assumption (D.8) and (D.6). Since  $\mathcal{Z}_3 = t_0 + \tau_{1,3} - 1\leq \tau_{-}(\hat{\phi}^u)$ , we have
$$
\mathbb {P} \left(\mathcal {Z} _ {1} ^ {c} \cap \mathcal {Z} _ {2} ^ {c} \cap \mathcal {Z} _ {3} \cap \left\{\tau_ {a} > \tau_ {1, 1} + \tau_ {1, 2} + \tau_ {1, 3} \right\}\right) \leq 5 \delta . \tag {D.20}
$$
If  $\tau_{-}(\tilde{\phi}^{u}) < \tau_{-}(\pi /2) + \tau_{1,3}$  (that is  $\mathcal{Z}_3^c$ ), we regard  $(\pmb{v}_{\tau_{-}(\tilde{\phi}^{u})},\pmb{a}_{\tau_{-}(\tilde{\phi}^{u})})$  as the new initial point and step into the analysis of the next region.
Initial region:  $[\tilde{\phi}^u,\pi)$ . Now we assign  $t_0 = \tau_{-}(\tilde{\phi}^u)$  and choose  $\tau_{1,4} = \tau_{1,3}$ . Recall that  $\cos \tilde{\phi}^u = -\frac{144\pi}{\pi + k - 1}\frac{(\mathbf{1}^\top\mathbf{a}^*)^2}{k}\log (4 + k^2)$ . Similar to what we showed in (D.17), for any  $t_0\leq s\leq t - 1\leq$ $t_0 + \tau_{1,4} - 1$ , under the event  $\mathcal{A}_{t - 1}\cap \mathcal{E}_{\mathrm{good}}$ , we have,
$$
\begin{array}{l} \cos \phi_ {s} = \frac {1}{\| \boldsymbol {v} _ {s} \|} \left\{\boldsymbol {v} _ {t _ {0}} ^ {\top} \boldsymbol {w} ^ {*} + \frac {\eta}{2 \pi} \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {m = t _ {0}} ^ {s - 1} \frac {\pi - \phi_ {m} ^ {i}}{\| \boldsymbol {v} _ {m} \|} (\boldsymbol {a} _ {m} ^ {i}) ^ {\top} \boldsymbol {a} ^ {*} \sin^ {2} \phi_ {m} - \eta \sum_ {m = t _ {0}} ^ {s - 1} (\boldsymbol {h} _ {m} ^ {\top} \boldsymbol {w} ^ {*} + \boldsymbol {\xi} _ {m} ^ {\top} \boldsymbol {w} ^ {*}) \right\} \\ \stackrel {(i)} {\leq} \frac {1}{\| \boldsymbol {v} _ {s} \|} \left\{\frac {\cos \phi_ {t _ {0}}}{3} + \tau_ {1, 4} \eta \gamma_ {a} + c \tau_ {1, 4} \eta^ {2} (I + \sqrt {k I}) \sqrt {\log (k / \eta \delta)} K _ {a} ^ {2} + \eta \left| \sum_ {m = t _ {0}} ^ {s - 1} \left(\boldsymbol {\xi} _ {m} ^ {\top} \boldsymbol {w} ^ {*} + \boldsymbol {h} _ {m} ^ {\top} \boldsymbol {w} ^ {*}\right) \right| \right\} \\ \leq \frac {1}{\| \boldsymbol {v} _ {s} \|} \left\{\frac {\cos \tilde {\phi} ^ {u}}{3} + \frac {3 6 \pi}{\pi + k - 1} \frac {\left(\mathbf {1} ^ {\top} \boldsymbol {a} ^ {*}\right) ^ {2}}{k} \log (4 + k ^ {2}) \right\} \\ \stackrel {(i i)} {\leq} - \frac {4 \pi}{\pi + k - 1} \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{k} \log \left(4 + k ^ {2}\right). \tag {D.21} \\ \end{array}
$$
where  $(i)$  holds due to  $\| \pmb{v}_m\| \leq 3$ ,  $\phi_0 > \pi /2$  and the definition of  $\mathcal{A}_{t - 1}$ ; and  $(ii)$  holds due to  $\| \pmb{v}_s\| \leq 3$ . Applying the bound  $|\cos \phi_s - \cos \phi_s^i |$  in (D.4) and the assumption D.6, we have
$$
\begin{array}{l} \max _ {i} \cos \phi_ {s} ^ {i} \leq \cos \phi_ {s} + \max _ {i} \left| \cos \phi_ {s} - \cos \phi_ {s} ^ {i} \right| \\ \leq - \frac {4 \pi}{\pi + k - 1} \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{k} \log (4 + k ^ {2}) + 1 2 c \eta K _ {a} ^ {2} (\sqrt {I d} + I) \sqrt {\log (N d / \delta)} \leq 0, \\ \end{array}
$$
which means  $\min_{i}\phi_{s}^{i}\geq \frac{\pi}{2}$ . It follows that for any  $s\leq t - 1$
$$
\begin{array}{l} \frac {1}{N} \sum_ {i = 1} ^ {N} B _ {s} ^ {i} = \frac {1}{N} \sum_ {i = 1} ^ {N} \left\{\left(g \left(\phi_ {s} ^ {i}\right) - 1\right) + \frac {\eta}{2 \pi} \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{\| \mathbf {a} ^ {*} \| ^ {2}} \sum_ {l = t _ {0}} ^ {s - 1} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s - 1 - l} \left(\pi - g \left(\phi_ {l} ^ {i}\right)\right) \right\} \\ \geq \frac {1}{N} \sum_ {i = 1} ^ {N} \left\{- 1 + \frac {\eta}{2 \pi} \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{\| \mathbf {a} ^ {*} \| ^ {2}} \sum_ {l = t _ {0}} ^ {s - 1} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s - 1 - l} (\pi - 1) \right\} \\ = \zeta_ {a} - \frac {\pi - 1}{\pi + k - 1} \frac {\left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{\left\| \mathbf {a} ^ {*} \right\| ^ {2}} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s}. \tag {D.22} \\ \end{array}
$$
Plugging (D.22) and (D.3) into (D.1), under the event  $\mathcal{A}_{t - 1}\cap \mathcal{E}_{\mathrm{good}}$  , we can get
$$
\begin{array}{l} \boldsymbol {a} _ {t} ^ {\top} \boldsymbol {a} ^ {*} \geq \frac {\zeta_ {a} \| \boldsymbol {a} ^ {*} \| ^ {2}}{\pi - 1} \left[ 1 - \left(1 - \frac {\eta (\pi - 1)}{2 \pi}\right) ^ {t} \right] - | \boldsymbol {a} _ {0} ^ {\top} \boldsymbol {a} ^ {*} | \left(1 - \frac {\eta (\pi - 1)}{2 \pi}\right) ^ {t} \\ - \frac {1 - \left(\frac {2 \pi - \eta (k + \pi - 1)}{2 \pi - \eta (\pi - 1)}\right) ^ {t}}{k} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t} \frac {(\pi - 1) \left(\mathbf {1} ^ {\top} \boldsymbol {a} ^ {*}\right) ^ {2}}{\pi + k - 1} \\ - 2 c K _ {a} ^ {2} \sqrt {\frac {M \eta \log (1 / \eta \delta)}{N}}. \tag {D.23} \\ \end{array}
$$
Plugging  $t = t_0 + \tau_{1,4}$ , according to (D.23) and (D.19), we can guarantee that  $\phi_{\tau_a} \geq \pi / 2$  and
$$
\mathbb {P} \left(\mathcal {Z} _ {1} ^ {c} \cap \mathcal {Z} _ {2} ^ {c} \cap \mathcal {Z} _ {3} ^ {c} \cap \left\{\tau_ {a} > \sum_ {q = 1} ^ {4} \tau_ {1, q} \right\}\right) \leq 5 \delta . \tag {D.24}
$$
Conclusion. Combining (D.10), (D.15), (D.20) and (D.24), we have
$$
\begin{array}{l} \mathbb {P} \left(\tau_ {a} > \sum_ {q = 1} ^ {4} \tau_ {1, q}\right) \leq \mathbb {P} \left(\left\{\tau_ {a} > \tau_ {1, 1} \right\} \cap \mathcal {Z} _ {1}\right) + \mathbb {P} \left(\left\{\tau_ {a} > \sum_ {q = 1} ^ {4} \tau_ {1, q} \right\} \cap \mathcal {Z} _ {1} ^ {c}\right) \\ \leq 4 \delta + \mathbb {P} \left(\left\{\tau_ {a} > \sum_ {q = 1} ^ {4} \tau_ {1, q} \right\} \cap \mathcal {Z} _ {1} ^ {c} \cap \mathcal {Z} _ {2}\right) + \mathbb {P} \left(\left\{\tau_ {a} > \sum_ {q = 1} ^ {2} \tau_ {1, q} \right\} \cap \mathcal {Z} _ {1} ^ {c} \cap \mathcal {Z} _ {2} ^ {c}\right) \\ \end{array}
$$
$$
\begin{array}{l} \leq 9 \delta + \mathbb {P} \left(\left\{\tau_ {a} > \sum_ {q = 1} ^ {3} \tau_ {1, q} \right\} \cap \mathcal {Z} _ {1} ^ {c} \cap \mathcal {Z} _ {2} ^ {c} \cap \mathcal {Z} _ {3}\right) + \mathbb {P} \left(\left\{\tau_ {a} > \sum_ {q = 1} ^ {4} \tau_ {1, q} \right\} \cap \mathcal {Z} _ {1} ^ {c} \cap \mathcal {Z} _ {2} ^ {c} \cap \mathcal {Z} _ {3} ^ {c}\right) \\ \leq 1 4 \delta + \mathbb {P} \left(\left\{\tau_ {a} > \sum_ {q = 1} ^ {4} \tau_ {1, q} \right\} \cap \mathcal {Z} _ {1} ^ {c} \cap \mathcal {Z} _ {2} ^ {c} \cap \mathcal {Z} _ {3} ^ {c}\right) \\ \leq 1 9 \delta . \\ \end{array}
$$
Therefore, we have proved  $\tau_{a} \leq \sum_{q=1}^{4} \tau_{1,q}$  with high probability for  $\phi_0 \in [0, \tilde{\phi}^o)$ . The conclusion for other initial regions can be obtained in similar arguments. From the definitions in (D.9), (D.13) and (D.16), we have
$$
\sum_ {q = 1} ^ {4} \tau_ {1, q} \lesssim \eta^ {- 1} \log k.
$$
D.2 Proof of Lemma 3