SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left| w _ {1} - 1 \right| \leqslant \frac {2 0 c _ {2} ^ {3}}{\pi c _ {1} t} \quad \text {f o r} \quad t \in \mathbb {N} ^ {+}. \tag {28}
$$
Step 3: the expected loss converges to 0. We now estimate the convergence of the expected square loss when  $(\boldsymbol{w}_0, \psi_{w,0}) \in D$ . For any  $(\boldsymbol{w}, \psi_w) \in D$ , define non-negative quantities  $\Delta_{\boldsymbol{w}} = \| \boldsymbol{w} - \boldsymbol{v} \|$  and  $\Delta_{\psi} = |\psi_w - \psi_v|$ . We provide an upper bound for  $L_{\mathrm{cc}}$  by discussion.
1. Suppose  $(\pmb{w},\psi_w)\in D_1$  . Then we have
$$
\begin{array}{l} 4 \pi L _ {\mathrm {c c}} (\pmb {w}, \psi_ {w}) = \| \pmb {w} \| ^ {2} s _ {w} - 2 \| \pmb {w} \| \| \pmb {v} \| \cos \theta_ {\pmb {w}, \pmb {v}} s _ {m} + \| \pmb {v} \| ^ {2} s _ {v} \\ \leqslant \| \boldsymbol {w} \| ^ {2} (s _ {v} + s _ {\Delta}) - 2 \| \boldsymbol {w} \| \| \boldsymbol {v} \| (1 - \Delta_ {\boldsymbol {w}} ^ {2}) (s _ {v} - s _ {\Delta}) + \| \boldsymbol {v} \| ^ {2} s _ {v} \\ \leqslant 4 \left(\| \boldsymbol {w} \| ^ {2} + 2 \| \boldsymbol {w} \| \| \boldsymbol {v} \|\right) \Delta_ {\psi} + \left(s _ {v} + 2 \| \boldsymbol {w} \| \| \boldsymbol {v} \|\right) \Delta_ {\boldsymbol {w}} ^ {2} \\ \leqslant 3 2 \Delta_ {\psi} + 8 \Delta_ {\boldsymbol {w}} ^ {2}, \\ \end{array}
$$
where the first inequality holds from  $s_w \leqslant s_v + s_\Delta$ ,  $\cos \theta_{\mathbf{w},\mathbf{v}} \geqslant \sqrt{1 - \Delta_{\mathbf{w}}^2} \geqslant 1 - \Delta_{\mathbf{w}}^2$ , and  $s_m \geqslant s_v - s_\Delta$  with  $s_\Delta = 2\Delta_\psi + \sin(2\Delta_\psi)$ , the second inequality holds since  $\| \boldsymbol{w} \| - \| \boldsymbol{v} \| \leqslant \Delta_{\boldsymbol{w}}^2$  and  $s_\Delta \leqslant 4\Delta_\psi$ , and the third inequality holds based on  $\| \boldsymbol{w} \| \leqslant 2$  and  $s_v \leqslant \pi$ .
2. Suppose  $(\pmb{w}, \psi_{\pmb{w}}) \in D_2$ . Let  $\theta = \theta_{\pmb{w}, \pmb{v}}$ . Then one knows
$$
\begin{array}{l} 4 \pi L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) = \| \boldsymbol {w} \| ^ {2} s _ {w} + \| \boldsymbol {v} \| ^ {2} s _ {v} \\ - \| \boldsymbol {w} \| \| \boldsymbol {v} \| [ 2 (\psi_ {w} + \psi_ {v} - \theta) \cos \theta + \sin (2 \psi_ {w} - \theta) + \sin (2 \psi_ {v} - \theta) ] \\ = s _ {v} \left(\| \boldsymbol {w} \| - \| \boldsymbol {v} \|\right) ^ {2} + \left(\| \boldsymbol {w} \| ^ {2} - \| \boldsymbol {w} \| \| \boldsymbol {v} \| \cos \theta\right) \left(s _ {w} - s _ {v}\right) \\ + \| \boldsymbol {w} \| \| \boldsymbol {v} \| \theta \cos \theta + 2 \| \boldsymbol {w} \| \| \boldsymbol {v} \| s _ {v} (1 - \cos \theta). \\ \end{array}
$$
Then according to  $\| \pmb{w} \| - \| \pmb{v} \| \leqslant \Delta_{\pmb{w}}$ ,  $s_w - s_v \leqslant 4\Delta_\psi$ ,  $\theta \leqslant \arcsin \Delta_{\pmb{w}} \leqslant 2\Delta_{\pmb{w}}$ , and  $\cos \theta \geqslant 1 - \Delta_{\pmb{w}}^2$ , we have
$$
\begin{array}{l} 4 \pi L _ {c c} \leqslant 4 \left| \| \boldsymbol {w} \| ^ {2} - \| \boldsymbol {w} \| \| \boldsymbol {v} \| \cos \theta \right| \Delta_ {\psi} + 2 \| \boldsymbol {w} \| \| \boldsymbol {v} \| \cos \theta \Delta_ {\boldsymbol {w}} + (1 + 2 \| \boldsymbol {w} \| \| \boldsymbol {v} \|) s _ {v} \Delta_ {\boldsymbol {w}} ^ {2} \\ \leqslant 1 6 \Delta_ {\psi} + 5 \Delta_ {w}, \\ \end{array}
$$
where the second inequality hodls based on  $\| \pmb{w}\| \leqslant 2$ ,  $s_v\leqslant \pi$ , and  $\Delta_{\pmb{w}}\leqslant \sqrt{2} R = \sqrt{2} /25$
Combining the cases above, one knows from  $\Delta_{\pmb{w}} \leqslant 5/8$  that for any  $(\pmb{w}, \psi_{\pmb{w}}) \in D$ , the loss satisfies
$$
L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) \leqslant 3 2 \Delta_ {\psi} + 5 \Delta_ {\boldsymbol {w}}.
$$
Then based on  $(\pmb{w}_t,\psi_{w,t})\in D$  and Eqs. (26)-(28), we obtain from  $c_{2}\geqslant 1$  that
$$
L _ {\mathrm {c c}} (\boldsymbol {w} _ {t}, \psi_ {w, t}) \leqslant \frac {3 2 0 c _ {2} ^ {2}}{c _ {1} t} + \frac {5 c _ {2} ^ {2}}{4 \pi c _ {1} t} + \frac {1 0 0 c _ {2} ^ {3}}{\pi c _ {1} t} \leqslant \frac {4 0 0 c _ {2} ^ {3}}{c _ {1} t},
$$
which holds with probability at least  $10^{-5}$  from Lemma 19. Thus, we have completed the proof.
C.1 Optimization behaviors
The following two lemmas consider the gradient with respect to  $w_{1}$  in  $D_{1}$  and  $D_{2}$ , respectively.
Lemma 20. Let  $w_{1} = w_{1} - \eta \nabla_{w_{1}}L_{\mathrm{cc}}(\boldsymbol{w},\psi_{w})$  with  $(\boldsymbol{w},\psi_w)\in D_1$ . If  $\eta \in (0,2)$  and  $(1 - R)s_w\leqslant s_v$ , then we have
$$
\nabla_ {w _ {1}} L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) = \frac {s _ {w}}{2 \pi} (w _ {1} - 1) + \frac {1}{2 \pi} [ s _ {w} - \min  \{s _ {w}, s _ {v} \} ] a n d | w _ {1} ^ {\prime} - 1 | \leqslant R.
$$
Proof. For any  $(\boldsymbol{w}, \psi_w) \in D_1$ , one has
$$
\nabla_ {w _ {1}} L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) = \frac {s _ {w}}{2 \pi} [ w _ {1} - \min  \left\{s _ {w}, s _ {v} \right\} ] = \frac {s _ {w}}{2 \pi} (w _ {1} - 1) + r, \tag {29}
$$
where  $r$  denotes a remainder defined by  $r = \frac{1}{2\pi} [s_w - \min \{s_w,s_v\} ]$ . Then Eq. (29) implies
$$
\left| w _ {1} ^ {\prime} - 1 \right| \leqslant \left| 1 - \frac {\eta s _ {w}}{2 \pi} \right| \left| w _ {1} - 1 \right| + \left| \eta r \right| \leqslant \left(1 - \frac {\eta s _ {w}}{2 \pi}\right) R + \frac {\eta}{2 \pi} \left[ s _ {w} - \min  \left\{s _ {w}, s _ {v} \right\} \right], \tag {30}
$$
where the first inequality holds from the triangle inequality, and the second inequality holds based on  $1 - \eta s_w / (2\pi) \geqslant 0$  and  $|w_1 - 1| \leqslant R$ . We proceed to complete the proof by discussion.
- Suppose that  $\min \{s_w, s_v\} = s_w$ . Then Eq. (30) implies
$$
\left| w _ {1} ^ {\prime} - 1 \right| \leqslant \left(1 - \frac {\eta s _ {w}}{2 \pi}\right) R \leqslant R,
$$
where the second inequality holds from  $\eta > 0$  and  $s_w \geqslant 0$ .
- Suppose that  $\min \{s_w, s_v\} = s_v$ . Then one knows from Eq. (30) that
$$
\left| w _ {1} ^ {\prime} - 1 \right| \leqslant \left(1 - \frac {\eta s _ {w}}{2 \pi}\right) R + \frac {\eta \left(s _ {w} - s _ {v}\right)}{2 \pi} \leqslant R,
$$
where the second inequality holds because of  $(1 - R)s_{w}\leqslant s_{v}$
Lemma 21. Let  $w_{1} = w_{1} - \eta \nabla_{w_{1}}L_{\mathrm{cc}}(\boldsymbol{w},\psi_{w})$  with  $(\boldsymbol{w},\psi_w)\in D_2$ . If  $\eta \in (0,2)$ ,  $R\leqslant \psi_l / 20$  and  $\max \{\psi_u - \psi_v,\psi_v - \psi_l\} \leqslant 5R\psi_l / 3$ , then we have
$$
\nabla_ {w _ {1}} L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) = \frac {s _ {w} - \theta_ {\boldsymbol {w} , \boldsymbol {v}}}{2 \pi} (w _ {1} - 1) + \frac {1}{4 \pi} \left[ (s _ {w} - s _ {v}) + 2 (\theta_ {\boldsymbol {w}, \boldsymbol {v}} - \sin \theta_ {\boldsymbol {w}, \boldsymbol {v}}) \right] a n d | w _ {1} ^ {\prime} - 1 | \leqslant R.
$$
Proof. For any  $(\boldsymbol{w}, \psi_{\boldsymbol{w}}) \in D_2$ , the gradient of  $L_{\mathrm{cc}}$  with respect to  $w_{1}$  can be calculated by
$$
\nabla_ {w _ {1}} L _ {\mathrm {c c}} = \frac {s _ {w} - \theta_ {\boldsymbol {w} , \boldsymbol {v}}}{2 \pi} (w _ {1} - 1) + \frac {1}{4 \pi} \left[ \left(s _ {w} - s _ {v}\right) + 2 \left(\theta_ {\boldsymbol {w}, \boldsymbol {v}} - \sin \theta_ {\boldsymbol {w}, \boldsymbol {v}}\right) \right] = \frac {s _ {w} - \theta_ {\boldsymbol {w} , \boldsymbol {v}}}{2 \pi} (w _ {1} - 1) + r,
$$
where  $r$  denotes a remainder defined by  $r = [(s_w - s_v) + 2(\theta_{\pmb{w},\pmb{v}} - \sin \theta_{\pmb{w},\pmb{v}})] / (4\pi)$ . Then we have
$$
\left| w _ {1} ^ {\prime} - 1 \right| \leqslant \left| 1 - \frac {\eta \left(s _ {w} - \theta_ {\boldsymbol {w} , \boldsymbol {v}}\right)}{2 \pi} \right| \left| w _ {1} - 1 \right| + \left| \eta r \right| \leqslant R + \eta \left[ | r | - \frac {R \left(s _ {w} - \theta_ {\boldsymbol {w} , \boldsymbol {v}}\right)}{2 \pi} \right], \tag {31}
$$
where the first inequality holds from the triangle inequality, and the second inequality holds based on  $\eta(s_w - \theta_{\mathbf{w},\mathbf{v}}) \leqslant \eta s_w \leqslant 2\pi$  and  $|w_1 - 1| \leqslant R$ . It is observed that
$$
s _ {w} - \theta_ {\boldsymbol {w}, \boldsymbol {v}} \geqslant \frac {7}{2} \psi_ {l} - \theta_ {\boldsymbol {w}, \boldsymbol {v}} \geqslant \frac {7}{2} \psi_ {l} - 2 R, \tag {32}
$$
where the first inequality holds based on  $s_w \geqslant 2\psi_l + \sin(2\psi_l)$  and  $\sin \psi_l \geqslant 3\psi_l / 4$  for  $\psi_l \leqslant \pi / 4$ , and the second inequality holds from  $\theta_{\mathbf{w},\mathbf{v}} \leqslant \arcsin R \leqslant 2R$ . Meanwhile, one has
$$
\left| r \right| \leqslant \frac {1}{4 \pi} \left| s _ {w} - s _ {v} \right| + \frac {1}{2 \pi} \left| \theta_ {\boldsymbol {w}, \boldsymbol {v}} - \sin \theta_ {\boldsymbol {w}, \boldsymbol {v}} \right| \leqslant \frac {\operatorname* {m a x} \left\{\psi_ {u} - \psi_ {v} , \psi_ {v} - \psi_ {l} \right\}}{\pi} + \frac {2 R ^ {3}}{3 \pi}, \tag {33}
$$
where the first inequality holds from the triangle inequality, and the second inequality holds according to the 4-Lipschitzness of  $2\theta + \sin(2\theta)$ ,  $\theta - \sin \theta \leqslant \theta^3 / 6$  for any  $\theta \geqslant 0$ , and  $\theta_{w,v} \leqslant 2R$ . Substituting Eqs. (32) and (33) into Eq. (31), we obtain
$$
| w _ {1} ^ {\prime} - 1 | \leqslant R + \frac {\eta}{1 2 \pi} \left[ 1 2 \max \{\psi_ {u} - \psi_ {v}, \psi_ {v} - \psi_ {l} \} + 8 R ^ {3} + 1 2 R ^ {2} - 2 1 R \psi_ {l} \right] \leqslant R,
$$
where the second inequality holds from  $\max \{\psi_u - \psi_v, \psi_v - \psi_l\} \leqslant 5R\psi_l / 3$  and  $R \leqslant \psi_l / 20 \leqslant 1$ . Thus, we have completed the proof.
The following two lemmas focus on the gradient with respect to  $w_{2}$  in  $D_{1}$  and  $D_{2}$ , respectively.
Lemma 22. Let  $w_2' = w_2 - \eta \nabla_{w_2} L_{\mathrm{cc}}(\pmb{w}, \psi_w)$  with  $(\pmb{w}, \psi_w) \in D_1$ . If  $\eta \in (0, 2)$ , then we have
$$
\left| w _ {2} ^ {\prime} \right| \leqslant \left(1 - \frac {\eta s _ {w}}{2 \pi}\right) \left| w _ {2} \right| \quad a n d \quad \left| w _ {2} ^ {\prime} \right| \leqslant R.
$$
Proof. For any  $(\boldsymbol{w}, \psi_w) \in D_1$ , one has  $\nabla_{w_2} L_{\mathrm{cc}}(\boldsymbol{w}, \psi_w) = \frac{s_w w_2}{2\pi}$ . Thus, we have
$$
w _ {2} ^ {\prime} = \left(1 - \frac {\eta s _ {w}}{2 \pi}\right) w _ {2}. \tag {34}
$$
According to  $s_w \in [0, \pi]$  and  $\eta \in (0, 2)$ , the coefficient  $1 - \eta s_w / (2\pi)$  is positive and smaller than 1. Based on  $(\boldsymbol{w}, \psi_w) \in D_1$ , one knows  $|w_2| \leqslant R$ . Then Eq. (34) implies
$$
\left| w _ {2} ^ {\prime} \right| = \left(1 - \frac {\eta s _ {w}}{2 \pi}\right) \left| w _ {2} \right| \leqslant R,
$$
Lemma 23. Let  $w_2' = w_2 - \eta \nabla_{w_2} L_{\mathrm{cc}}(\boldsymbol{w}, \psi_w)$  with  $(\boldsymbol{w}, \psi_w) \in D_2$ . If  $R \leqslant 1/2$ ,  $\sqrt{2}R \leqslant \cos^2 \psi_v$ , and  $\eta \leqslant 3R/2$ , then we have
$$
\frac {\cos^ {2} \psi_ {v} - \sqrt {2} R}{2 \pi} \leqslant \nabla_ {w _ {2}} L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) \mathrm {s g n} (w _ {2}) \leqslant \frac {2}{3} a n d | w _ {2} ^ {\prime} | \leqslant R.
$$
Proof. For any  $(\boldsymbol{w}, \psi_{\boldsymbol{w}}) \in D_2$ , the gradient of  $L_{\mathrm{cc}}$  with respect to  $w_2$  can be calculated by
$$
\nabla_ {w _ {2}} L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) = \frac {1}{2 \pi} s _ {w} w _ {2} + \frac {1}{4 \pi} \left(\cos (2 \psi_ {w}) + \cos (2 \psi_ {v}) + \frac {2 w _ {1} ^ {2}}{\sqrt {w _ {1} ^ {2} + w _ {2} ^ {2}}}\right) \operatorname {s g n} (w _ {2}). \tag {35}
$$
Since  $(\pmb{w},\psi_w)\in D_2$ , one knows that  $|w_{1} - 1|\leqslant R$  and  $|w_{2}|\leqslant R$ . Thus, we have
$$
2 (1 - \sqrt {2} R) \leqslant \frac {2 (1 - R) ^ {2}}{\sqrt {(1 - R) ^ {2} + R ^ {2}}} \leqslant \frac {2 w _ {1} ^ {2}}{\sqrt {w _ {1} ^ {2} + w _ {2} ^ {2}}} \leqslant 2 (1 + R),
$$
where the first inequality holds because of  $R \in [0, 1/2]$ . Then we have
$$
\cos \left(2 \psi_ {w}\right) + \cos \left(2 \psi_ {v}\right) + \frac {2 w _ {1} ^ {2}}{\sqrt {w _ {1} ^ {2} + w _ {2} ^ {2}}} \leqslant 1 + \cos \left(2 \psi_ {v}\right) + 2 (1 + R) \leqslant 5, \tag {36}
$$
where the second inequality holds based on  $R \leqslant 1/2$ . Meanwhile, one has
$$
\cos (2 \psi_ {w}) + \cos (2 \psi_ {v}) + \frac {2 w _ {1} ^ {2}}{\sqrt {w _ {1} ^ {2} + w _ {2} ^ {2}}} \geqslant - 1 + \cos (2 \psi_ {v}) + 2 (1 - \sqrt {2} R) = 2 (\cos^ {2} \psi_ {v} - \sqrt {2} R). \tag {37}
$$
It is observed that  $0 \leqslant s_w |w_2| \leqslant \frac{\pi}{2}$  since  $s_w \in [0, \pi]$  and  $|w_2| \leqslant R \leqslant \frac{1}{2}$ . Then substituting Eqs. (36) and (37) into Eq. (35), we obtain
$$
\frac {\cos^ {2} \psi_ {v} - \sqrt {2} R}{2 \pi} \leqslant \nabla_ {w _ {2}} L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) \mathrm {s g n} (w _ {2}) \leqslant \frac {1}{4} + \frac {5}{4 \pi} \leqslant \frac {2}{3}.
$$
Thus, one knows from Eq. (35) that
$$
\left| w _ {2} ^ {\prime} \right| = \left| \left| w _ {2} \right| - \eta \nabla_ {w _ {2}} L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) \operatorname {s g n} (w _ {2}) \right| \leqslant \max  \left\{\left| w _ {2} \right|, \eta \nabla_ {w _ {2}} L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) \operatorname {s g n} (w _ {2}) \right\} \leqslant R,
$$
where the first inequality holds from  $|a - b| \leqslant \max \{a, b\}$  for non-negative numbers  $a$  and  $b$ , and the second inequality holds based on  $|w_2| \leqslant R$  and  $\eta \leqslant 3R / 2$ . Thus, we have completed the proof.
The following two lemmas investigate the gradient with respect to  $\psi_w$  in  $D_1$  and  $D_2$ , respectively.
Lemma 24. Let  $\psi_w' = \psi_w - \eta \nabla_{\psi_w} L_{cc}(\boldsymbol{w}, \psi_w)$  with  $(\boldsymbol{w}, \psi_w) \in D_1$ . If  $R \leqslant 1/2$ ,  $\eta \leqslant \pi (\psi_u - \psi_v) / 3$  and  $\eta \leqslant \pi (\psi_v - \psi_l) / 3$ , then we have
$$
\frac {\cos^ {2} \psi_ {u}}{4 \pi} \leqslant \mathrm {s g n} (\psi_ {w} - \psi_ {v}) \nabla_ {\psi_ {w}} L _ {\mathrm {c c}} (\pmb {w}, \psi_ {w}) \leqslant \frac {3}{\pi} a n d \psi_ {w} ^ {\prime} \in [ \psi_ {l}, \psi_ {u} ].
$$
Proof. For any  $(\boldsymbol{w}, \psi_{\boldsymbol{w}}) \in D_{1}$ , the gradient of  $L_{cc}$  with respect to  $\psi_{\boldsymbol{w}}$  can be calculated by
$$
\nabla_ {\psi_ {w}} L _ {\mathrm {c c}} (\pmb {w}, \psi_ {w}) = \left\{ \begin{array}{l l} - \frac {1}{2 \pi} [ 1 + \cos (2 \psi_ {w}) ] [ 1 - \| \pmb {w} - \pmb {v} \| ^ {2} ], & \psi_ {w} <   \psi_ {v}, \\ \frac {1}{2 \pi} [ 1 + \cos (2 \psi_ {w}) ] \| \pmb {w} \| ^ {2}, & \psi_ {w} > \psi_ {v}, \end{array} \right.
$$
where the gradient at  $\psi_w = \psi_v$  can be any subgradient. For any  $(\boldsymbol{w}, \psi_w) \in D_2$ , we have  $\psi_w \in [\psi_l, \psi_u]$ , which indicates  $2\cos^2\psi_u \leqslant 1 + \cos(2\psi_w) \leqslant 2$ . Meanwhile, all points in  $D_2$  satisfy  $1 - 2R^2 \leqslant 1 - \| \boldsymbol{w} - \boldsymbol{v} \|^2 \leqslant 1$  and  $(1 - R)^2 \leqslant \| \boldsymbol{w} \|^2 \leqslant (1 + R)^2 + R^2$ . Thus, the gradient of  $L_{\mathrm{cc}}$  with respect to  $\psi_w$  can be bounded by
$$
\frac {\cos^ {2} \psi_ {u}}{4 \pi} \leqslant \operatorname {s g n} (\psi_ {w} - \psi_ {v}) \nabla_ {\psi_ {w}} L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) \leqslant \frac {3}{\pi},
$$
where the first and second inequalities hold based on  $R \leqslant 1/2$ . Then  $\psi_w'$  satisfies
$$
\psi_ {w} ^ {\prime} = \psi_ {w} - \eta \nabla_ {\psi_ {w}} L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) \leqslant \max  \left\{\psi_ {w}, \psi_ {v} + \frac {3 \eta}{\pi} \right\} \leqslant \psi_ {u} ,
$$
where the first inequality holds from discussing the relation between  $\psi_w$  and  $\psi_v$ , and the second inequality holds based on  $\psi_w \leqslant \psi_u$  and  $\eta \leqslant \pi (\psi_u - \psi_v) / 3$ . Meanwhile, one has
$$
\psi_ {w} ^ {\prime} = \psi_ {w} - \eta \nabla_ {\psi_ {w}} L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) \geqslant \min  \left\{\psi_ {w}, \psi_ {v} - \frac {3 \eta}{\pi} \right\} \geqslant \psi_ {l},
$$
where the first inequality holds from discussing the relation between  $\psi_w$  and  $\psi_v$ , and the second inequality holds based on  $\psi_w \geqslant \psi_l$  and  $\eta \leqslant \pi (\psi_v - \psi_l) / 3$ . Thus, we have completed the proof.
Lemma 25. Let  $\psi_w' = \psi_w - \eta \nabla_{\psi_w} L_{\mathrm{cc}}(\boldsymbol{w}, \psi_w)$  with  $(\boldsymbol{w}, \psi_w) \in D_2$ . If  $R \leqslant 1/2$  and  $\arcsin R + 9\eta \leqslant \psi_u - \psi_v$ , then we have
$$
- 9 \leqslant - 2 \left(\frac {\pi}{2} - \psi_ {w}\right) ^ {2} - 2 \left(\frac {\pi}{2} - \psi_ {w}\right) | w _ {2} | \leqslant \nabla_ {\psi_ {w}} L _ {\mathrm {c c}} \leqslant - \frac {1}{4} \left(\frac {\pi}{2} - \psi_ {w}\right) ^ {2} \quad a n d \quad \psi_ {w} ^ {\prime} \in [ \psi_ {l}, \psi_ {u} ].
$$
Proof. For any  $(\boldsymbol{w}, \psi_w) \in D_1$ , the gradient of  $L_{cc}$  with respect to  $\psi_w$  can be calculated by
$$
\nabla_ {\psi_ {w}} L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) = \frac {\| \boldsymbol {w} \| ^ {2}}{2 \pi} [ 1 + \cos (2 \psi_ {w}) ] - \frac {\| \boldsymbol {w} \|}{2 \pi} [ \cos \theta_ {\boldsymbol {w}, \boldsymbol {v}} + \cos (\theta_ {\boldsymbol {w}, \boldsymbol {v}} - 2 \psi_ {w}) ].
$$
It is observed that the above expression is the same as the gradient of  $L_{\mathrm{cr}}$  with respect to  $\psi$  in Eq. (16). The only difference comes from the domain of  $\pmb{w}$ , which is  $\| \pmb{w} - \pmb{v} \| \leqslant R$  in Lemma 12 and  $\| \pmb{w} - \pmb{v} \|_{\infty} \leqslant R$  here. Then according to  $\| \pmb{x} \| \leqslant \sqrt{2} \| \pmb{x} \|_{\infty}$  in  $\mathbb{R}^2$ , one knows from Lemma 12 that
$$
- 9 \leqslant - 2 \left(\frac {\pi}{2} - \psi_ {w}\right) ^ {2} - 2 \left(\frac {\pi}{2} - \psi_ {w}\right) | w _ {2} | \leqslant \nabla_ {\psi_ {w}} L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) \leqslant - \frac {1}{4} \left(\frac {\pi}{2} - \psi_ {w}\right) ^ {2},
$$
where the first inequality holds according to  $|\pi /2 - \psi_w|\leqslant \pi /2$  and  $|w_{2}|\leqslant 1$ , and the third inequality holds based on  $R\leqslant 1 / 2$ . Then  $\psi_w^\prime$  satisfies
$$
\psi_ {w} ^ {\prime} \leqslant \psi_ {w} + 9 \eta \leqslant \psi_ {v} + \theta_ {\boldsymbol {w}, \boldsymbol {v}} + 9 \eta \leqslant \psi_ {u},
$$
where the second inequality holds from the condition  $\theta_{\pmb{w},\pmb{v}}\geqslant |\psi_w - \psi_v|$  in the definition of  $D_{2}$ , and the third inequality holds according to
$$
\theta_ {\boldsymbol {w}, \boldsymbol {v}} \leqslant \arcsin R \leqslant \psi_ {u} - \psi_ {v} - 9 \eta .
$$
Meanwhile, it is observed that the gradient is always negative, which implies  $\psi_w' \geqslant \psi_w \geqslant \psi_l$ . Thus, we have completed the proof.
C.2 Convergence Rate Lemmas
This section presents some sufficient conditions for convergence with an inversely proportional rate.
Lemma 26. Let  $f: K \to \mathbb{R}$  represent a function with a global minimum  $x^*$ , where  $K \subset \mathbb{R}$  indicates the convex domain satisfying  $B(x^*, r_1) \subset K \subset B(x^*, r_2)$ . Suppose that there exist constants  $c_1, c_3, g_l, g_u$  such that  $c_1 \leqslant r_1 / g_u$  and for any  $x \in K$ , at least one of the following holds.
1.  $|x' - x^*| \leqslant (1 - c_3\eta)|x - x^*|$  and  $(x' - x^*)(x - x^*) \geqslant 0$  with  $x' = x - \eta \nabla f(x)$  and  $\eta \in (0, c_1]$ .
2.  $g_{l} \leqslant \operatorname{sgn}(x - x^{*}) \nabla f(x) \leqslant g_{u}$  for any  $x \neq x^{*}$  and  $|\nabla f(x^{*})| \leqslant g_{u}$ .
Then for any  $c_{2} \geqslant \max \{1 / c_{3}, 2r_{2} / g_{l}, 2c_{1}g_{u} / g_{l}\}$ , the sequence  $\{x_{t}\}_{t=1}^{\infty}$  generated by gradient descent  $x_{t+1} = x_{t} - \eta_{t}\nabla f(x_{t})$  with  $x_{0} \in K$  and  $\eta_{t} = \min \{c_{1}, c_{2} / t\}$  satisfies
$$
x _ {t} \in K \quad a n d \quad | x _ {t} - x ^ {*} | \leqslant \frac {a}{t} \quad w i t h \quad a = \frac {c _ {2} ^ {2} g _ {l}}{2 c _ {1}}.
$$
Proof. Firstly, we prove  $x_{t} \in K$ . Suppose  $x_{t} \in K$  for  $t = k$ . We prove  $x_{k + 1} \in K$  by discussion.
1. If the first condition holds, then  $x_{k + 1}$  is a convex combination of  $x_{k}$  and  $x^{*}$ . Thus,  $x_{k + 1} \in K$ .
2. If the second condition holds and  $\operatorname{sgn}(x_{k+1} - x^*) = \operatorname{sgn}(x_k - x^*)$ , then  $x_{k+1}$  is a convex combination of  $x_k$  and  $x^*$ . Thus,  $x_{k+1} \in K$ .
3. If the third condition holds and  $\mathrm{sgn}(x_{k + 1} - x^{*})\neq \mathrm{sgn}(x_k - x^*)$ , then one knows from  $\eta_t\leqslant c_1$  and  $|\nabla f(x)|\leqslant g_u$  that  $|x_{k + 1} - x^{*}|\leqslant c_{1}g_{u}\leqslant r_{1}$ , where the second inequality holds based on  $c_{1}\leqslant r_{1} / g_{u}$ . Thus,  $B(x^{*},r_{1})\subset K$  leads to  $x_{k + 1}\in K$ .
Combining the cases above,  $x_0 \in K$  and mathematical induction completes the proof of  $x_t \in K$ .
Secondly, we prove  $|x_{t} - x^{*}| \leqslant a / t$ . Let  $t_0 = c_2 / c_1$ . According to  $c_2 \geqslant 2c_1g_u / g_l \geqslant 2c_1$ , one knows  $t_0 \geqslant 2$ . For  $t < t_0$ , it is observed that
$$
\left| x _ {t} - x ^ {*} \right| \leqslant r _ {2} \leqslant \frac {a}{t _ {0}} \leqslant \frac {a}{t},
$$
where the first inequality holds based on  $K \subset B(x^{*}, r_{2})$ , the second inequality holds because of  $a = c_{2}^{2} g_{l} / (2c_{1}) \geqslant r_{2} t_{0}$ . Thus, the conclusion holds for any  $t < t_{0}$ . Suppose that  $|x_{k} - x^{*}| \leqslant a / k$  holds for  $k \geqslant t_{0} - 1$ . We then prove  $|x_{k + 1} - x^{*}| \leqslant a / (k + 1)$  by discussion.
1. If the first condition holds, then we have
$$
\left| x _ {k + 1} - x ^ {*} \right| \leqslant \left(1 - \frac {c _ {2} c _ {3}}{k + 1}\right) \frac {a}{k} \leqslant \frac {a}{k + 1},
$$
where the first inequality holds based on the first condition and the induction hypothesis, and the second inequality holds from  $c_{2} \geqslant 1 / c_{3}$ . Thus, the conclusion holds for  $t = k + 1$ .
2. If the second condition holds and  $\operatorname{sgn}(x_{k+1} - x^*) = \operatorname{sgn}(x_k - x^*)$ , then one knows
$$
\left| x _ {k + 1} - x ^ {*} \right| \leqslant \frac {a}{k} - \frac {c _ {2} g _ {l}}{k + 1} \leqslant \frac {a}{k + 1},
$$
where the first inequality holds from the induction hypothesis and the second condition, and the second inequality holds because of
$$
\frac {a}{k} - \frac {c _ {2} g _ {l}}{k + 1} - \frac {a}{k + 1} = \frac {a - c _ {2} g _ {l} k}{k (k + 1)} = \frac {c _ {2} g _ {l} \left(t _ {0} / 2 - k\right)}{k (k + 1)} \leqslant 0,
$$
where the first equality holds based on  $c_2 \geqslant 1 / c_3$ , the second equality holds from the choice of  $a$  and  $t_0$ , and the first inequality holds from  $t_0 \geqslant 2$  and  $k \geqslant t_0 - 1 \geqslant t_0 / 2$ . Thus, the conclusion holds for  $t = k + 1$ .
3. If the second condition holds and  $\operatorname{sgn}(x_{k+1} - x^*) \neq \operatorname{sgn}(x_k - x^*)$ , then it is observed that
$$
\left| x _ {k + 1} - x ^ {*} \right| \leqslant \frac {c _ {2} g _ {u}}{k + 1} \leqslant \frac {a}{k + 1},
$$
where the first inequality holds from the second condition, and the second inequality holds based on  $a = c_2^2 g_l / (2c_1) \geqslant c_2 g_u$ . Thus, the conclusion holds for  $t = k + 1$ .
Combining the cases above, we have completed the proof.
Lemma 27. Let  $f: K \to \mathbb{R}$  represent a function with a global minimum  $x^{*}$ , where  $K \subset \mathbb{R}$  indicates the convex domain satisfying  $B(x^{*}, r_{1}) \subset K \subset B(x^{*}, r_{2})$ . Let  $\{\theta_{t}\}_{t=0}^{\infty}$  be a positive sequence bounded by  $\theta_{t} \leqslant a / t$ . Suppose that there exist constants  $g_{l}, g_{u}$  such that for any  $x \in K$ , the following holds
1. If  $|x_{t} - x^{*}| \geqslant \theta_{t}$ , then  $g_{l} \leqslant \operatorname{sgn}(x_{t} - x^{*}) \nabla f(x_{t}) \leqslant g_{u}$ .
2. If  $|x_{t} - x^{*}| \leqslant \theta_{t}$ , then  $|\nabla f(x_{t})| \leqslant g_{u}$ .