SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} 2 \pi \nabla_ {\psi} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) = [ 1 + \cos (2 \psi) ] w _ {1} ^ {2} - [ 1 + \cos (2 \psi) ] w _ {1} + [ 1 + \cos (2 \psi) ] w _ {2} ^ {2} - \sin (2 \psi) | w _ {2} | \tag {16} \\ = [ 1 + \cos (2 \psi) ] [ \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2} - 1 ] + [ 1 + \cos (2 \psi) ] w _ {1} - \sin (2 \psi) | w _ {2} |. \\ \end{array}
$$
Firstly, we prove the upper bound for  $\nabla_{\psi}L_{\mathrm{cr}}(\pmb {w},\psi)$  . It is observed that
$$
[ 1 + \cos (2 \psi) ] w _ {1} - \sin (2 \psi) | w _ {2} | \leqslant 2 \cos \psi (w _ {1} \sin \theta - | w _ {2} | \cos \theta) = 0,
$$
where the first inequality holds based on  $\pi /2\geqslant \psi \geqslant \pi /2 - \theta \geqslant 0$  , and the first equality holds from  $w_{1} = r\cos \theta$  and  $|w_{2}| = r\sin \theta$  . Substituting Eq. (24) into Eq. (16), we obtain
$$
2 \pi \nabla_ {\psi} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) \leqslant [ 1 + \cos (2 \psi) ] [ \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2} - 1 ] \leqslant - \frac {1 - R ^ {2}}{2} \left(\frac {\pi}{2} - \psi\right) ^ {2},
$$
where the second inequality holds according to  $1 + \cos (2\psi)\geqslant \frac{1}{2} (\pi /2 - \psi)^2$  for any  $\psi \in [0,\pi /2]$  and  $\| \pmb {w} - \pmb {v}\| \leqslant R$
Secondly, we verify the lower bound for  $\nabla_{\psi}L_{\mathrm{cr}}(\pmb {w},\psi)$ . It is observed that
$$
\begin{array}{l} 2 \pi \nabla_ {\psi} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) \geqslant - [ 1 + \cos (2 \psi) ] - \sin (2 \psi) | w _ {2} | \\ \geqslant - 2 \left(\frac {\pi}{2} - \psi\right) ^ {2} - \sin (2 \psi) | w _ {2} | \\ \geqslant - 2 \left(\frac {\pi}{2} - \psi\right) ^ {2} - 2 \left(\frac {\pi}{2} - \psi\right) | w _ {2} |, \\ \end{array}
$$
where the first inequality holds because of  $[1 + \cos(2\psi)]w_1 \geqslant 0$  and  $\|\pmb{w} - \pmb{v}\| \geqslant 0$ , the second inequality holds according to  $1 + \cos(2\psi) \leqslant 2(\pi/2 - \psi)^2$ , and the third inequality holds based on  $\sin(2\psi) \leqslant \pi - 2\psi$  for  $\psi \in [0, \pi/2]$ . Thus, we have completed the proof.
B.2 Convergence Rate Lemmas
The following lemma provides a sufficient condition for linear convergence of gradient descent.
Lemma 13. If there exist two constants  $c_{1}$  and  $c_{2}$  such that
$$
\langle \nabla f (\boldsymbol {w}), \boldsymbol {w} - \boldsymbol {v} \rangle \geqslant c _ {1} \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2} \quad a n d \quad \| \nabla f (\boldsymbol {w}) \| ^ {2} \leqslant c _ {2} \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2},
$$
then  $\pmb{w}^{\prime} = \pmb{w} - \eta \nabla f(\pmb{w})$  with  $\eta \in (0,2c_1 / c_2)$  and  $c = \sqrt{1 - 2c_1\eta + c_2\eta^2}\in (0,1)$  satisfies
$$
\left\| \boldsymbol {w} ^ {\prime} - \boldsymbol {v} \right\| \leqslant c \| \boldsymbol {w} - \boldsymbol {v} \|.
$$
Proof. It is observed that
$$
\begin{array}{l} \left\| \boldsymbol {w} ^ {\prime} - \boldsymbol {v} \right\| ^ {2} = \left\| \boldsymbol {w} - \eta \nabla f (\boldsymbol {w}) - \boldsymbol {v} \right\| ^ {2} \\ = \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2} - 2 \eta \langle \nabla f (\boldsymbol {w}), \boldsymbol {w} - \boldsymbol {v} \rangle + \eta^ {2} \| \nabla f (\boldsymbol {w}) \| ^ {2} \\ \leqslant \left(1 - 2 c _ {1} \eta + c _ {2} \eta^ {2}\right) \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2}. \\ \end{array}
$$
For  $\eta \in (0,2c_1 / c_2)$ , the coefficient  $1 - 2c_{1}\eta +c_{2}\eta^{2}$  is smaller than 1, which completes the proof. The following lemma gives a sufficient condition for convergence with an inversely proportional rate.
Lemma 14. Let  $\{a_t\}_{t=0}^{\infty} \subset [0,1/2]$  represent a real-valued sequence.
1. If  $a_{t+1} \leqslant a_t (1 - a_t)$ , then  $a_t \leqslant \frac{1}{t+1}$ .
2. If  $a_{t + 1} \geqslant a_t(1 - a_t)$ , then  $a_{t} \geqslant \frac{a_{0}}{t + 1}$ .
Proof. We prove the first conclusion by mathematical induction.
1. Base case. For  $t = 0$ , the conclusion holds from  $a_0 \leqslant 1 / 2 \leqslant 1$ .
2. Induction. Suppose that the conclusion holds for  $t = k$  with  $k \in \mathbb{N}$ . Then it is observed that
$$
a _ {t + 1} \leqslant \frac {1}{k + 1} \left(1 - \frac {1}{k + 1}\right) = \frac {k}{(k + 1) ^ {2}} \leqslant \frac {1}{k + 2},
$$
where the first inequality holds from the induction hypothesis and the monotonicity of  $x(1 - x)$  for  $x \in [0, 1/2]$ . Thus, the conclusion holds for  $t = k + 1$ .
Therefore, mathematical induction completes the proof of the first conclusion.
We proceed to verify the second conclusion by mathematical induction.
1. Base case. For  $t = 0$ , the conclusion holds from  $a_0 \geqslant a_0$ .
2. Induction. Suppose that the conclusion holds for  $t = k$  with  $k \in \mathbb{N}$ . Then one has
$$
a _ {t + 1} \geqslant \frac {a _ {0}}{k + 1} \left(1 - \frac {a _ {0}}{k + 1}\right) = \frac {a _ {0} (k + 1 - a _ {0})}{(k + 1) ^ {2}} \geqslant \frac {a _ {0}}{k + 2},
$$
where the first inequality holds from the induction hypothesis and the monotonicity of  $x(1 - x)$  for  $x \in [0, 1/2]$ , and the second inequality holds based on  $a_0 \leqslant 1/2$ . Thus, the conclusion holds for  $t = k + 1$ .
Therefore, mathematical induction completes the proof.
B.3 Technical Lemmas
We present upper bounds for some small terms used in the proof.
Lemma 15. Let  $R_{1} = \left[\left(\frac{\pi}{2} -\psi -\theta\right) - \frac{1}{2}\sin (2\psi)\right](w_{1} - 1)$ . If  $(\boldsymbol {w},\psi)\in D_2$ , then
$$
R _ {1} \leqslant \frac {1}{2} [ \sin (2 \psi) + 2 \psi ] \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2}.
$$
Proof. Let  $r = \sqrt{w_1^2 + w_2^2}$  denote the norm of  $\pmb{w}$ . Then according to the definition of  $\theta$ , one has  $w_{1} = r\cos \theta$  and  $|w_{2}| = r\sin \theta$ . Thus, we can rewrite  $R_{1}$  as
$$
R _ {1} = \left[ \left(\frac {\pi}{2} - \psi - \theta\right) - \frac {1}{2} \sin (2 \psi) \right] (r \cos \theta - 1).
$$
We provide the upper bound for  $R_{1}$  by discussion.
1. Suppose  $r\cos \theta - 1 \geqslant 0$ . Based on the definition of  $D_{2}$ , we have  $\frac{\pi}{2} - \psi - \theta \leqslant 0$ . Meanwhile,  $\psi \in [0, \pi/2]$  indicates  $\sin(2\psi) \geqslant 0$ . Thus, one knows  $R_{1} \leqslant 0$ .
2. Suppose  $r \cos \theta - 1 < 0$ .  $R_{1}$  can be rewritten as
$$
R _ {1} = \frac {1}{2} [ \sin (2 \psi) + 2 \psi ] \left(1 - 2 r \cos \theta + r ^ {2}\right) + \widetilde {R}, \tag {17}
$$
$$
\widetilde {R} = \frac {1}{2} [ \sin (2 \psi) + 2 \psi ] r (\cos \theta - r) + \left(\frac {\pi}{2} - \theta\right) (r \cos \theta - 1).
$$
If  $\cos \theta - r \leqslant 0$ , it is observed that  $\widetilde{R} \leqslant 0$  because of  $\psi, \theta \in [0, \pi/2]$  and  $r \cos \theta - 1 < 0$ . If  $\cos \theta - r > 0$ , then
$$
\widetilde {R} \leqslant \frac {\pi}{2} r (\cos \theta - r) + \left(\frac {\pi}{2} - \theta\right) (r \cos \theta - 1) = - \frac {\pi}{2} r ^ {2} + (\pi - \theta) \cos \theta r - \left(\frac {\pi}{2} - \theta\right) =: f (r),
$$
where the inequality holds since  $\sin (2\psi) + 2\psi$  is monotonically increasing. The discriminant of  $f$  is
$$
\Delta (\theta) = (\pi - \theta) ^ {2} \cos^ {2} \theta - \pi (\pi - 2 \theta) \leqslant \frac {1}{\pi^ {2}} \theta^ {2} (\pi - 2 \theta) (2 \theta - 3 \pi),
$$
where the first inequality holds since  $\cos^2\theta \leqslant 1 - 4\theta^2 /\pi^2$  on  $[0,\pi /2]$ . According to  $\theta \in [0,\pi /2]$ , one knows  $\Delta (\theta)\leqslant 0$ , which indicates  $f(r)\leqslant 0$ , and thus,  $\widetilde{R}\leqslant 0$  when  $\cos \theta -r\leqslant 0$ . Combining the cases above, we obtain  $\widetilde{R}\leqslant 0$ , which, together with Eq. (17), implies  $R_{1}\leqslant \frac{1}{2} [\sin (2\psi) + 2\psi ](1 - 2r\cos \theta +r^2)$ .
Combining the cases above, one knows
$$
R _ {1} \leqslant \frac {1}{2} [ \sin (2 \psi) + 2 \psi ] (1 - 2 r \cos \theta + r ^ {2}) = \frac {1}{2} [ \sin (2 \psi) + 2 \psi ] \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2},
$$
Lemma 16. Let  $R_2 = \left[\frac{1}{2} - \frac{1}{2}\cos(2\psi) - \frac{w_1}{w_1^2 + w_2^2}\right] |w_2|$ . If  $(\boldsymbol{w}, \psi) \in D_2$ , then
$$
R _ {2} \leqslant \frac {1}{\pi} [ \sin (2 \psi) + 2 \psi ] \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2}.
$$
Proof. Let  $r = \sqrt{w_1^2 + w_2^2}$  denote the norm of  $w$ . Then according to the definition of  $\theta$ , one has  $w_1 = r\cos \theta$  and  $|w_2| = r\sin \theta$ . Thus, we can rewrite  $R_2$  as
$$
R _ {2} = \left[ \frac {r}{2} (1 - \cos (2 \psi)) - \cos \theta \right] \sin \theta .
$$
We provide the upper bound for  $R_{2}$  by discussion.
1. Suppose  $\frac{r}{2} [1 - \cos(2\psi)] - \cos\theta \leqslant 0$ . From  $\theta \in [0, \pi/2]$ , we have  $R_2 \leqslant 0$ .
2. Suppose  $\frac{r}{2} [1 - \cos(2\psi)] - \cos\theta > 0$ . It is observed that  $r < 2\cos\theta$  since  $\| \pmb{w} - \pmb{v} \|^2 \leqslant r_0^2 < 1$  holds from the definition of  $D_2$ . Thus, the supposition indicates  $\cos\theta < \frac{r}{2} [1 - \cos(2\psi)] < [1 - \cos(2\psi)]\cos\theta$ , which, together with  $\theta \in [0, \pi/2]$ , implies  $\psi \geqslant \pi/4$ . It is observed that
$$
f (r) = \frac {1}{2} (1 - 2 r \cos \theta + r ^ {2}) - (r - \cos \theta) \sin \theta = \frac {1}{2} (r - \cos \theta - \sin \theta) ^ {2} \geqslant 0,
$$
$$
\frac {1}{\pi} [ \sin (2 \psi) + 2 \psi ] (1 - 2 r \cos \theta + r ^ {2}) \geqslant \frac {1}{2} (1 - 2 r \cos \theta + r ^ {2}) \geqslant (r - \cos \theta) \sin \theta \geqslant R _ {2},
$$
where the first inequality holds from  $\psi \geqslant \pi /4$ , and the third inequality holds because of  $\cos (2\psi)\geqslant -1$
Combining the cases above, we obtain
$$
R _ {2} \leqslant \frac {1}{\pi} [ \sin (2 \psi) + 2 \psi ] (1 - 2 r \cos \theta + r ^ {2}) = \frac {1}{\pi} [ \sin (2 \psi) + 2 \psi ] \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2},
$$
Lemma 17. Let  $T_{1} = \left([ \sin (2\psi) + 2\psi ]w_{1} - \frac{1}{2}\sin (2\psi) - \left(\frac{\pi}{2} +\psi -\theta\right) - \frac{w_{1}|w_{2}|}{w_{1}^{2} + w_{2}^{2}}\right)^{2}$ . If  $(\boldsymbol {w},\psi)\in D_2$ , then we have
$$
T _ {1} \leqslant 7 \pi [ \sin (2 \psi) + 2 \psi ] \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2}.
$$
Proof. It is observed that  $T_{1} = \left[\left[\sin (2\psi) + 2\psi\right](w_{1} - 1) + T_{11} + T_{12}\right]^{2}$  with
$$
T _ {1 1} = \frac {1}{2} \sin (2 \psi) + \left(\psi + \theta - \frac {\pi}{2}\right) \quad \text {a n d} \quad T _ {1 2} = - \frac {w _ {1} \left| w _ {2} \right|}{w _ {1} ^ {2} + w _ {2} ^ {2}}. \tag {18}
$$
Firstly, denote by  $r_0 \in (0,1)$  a parameter determined later and we calculate an upper bound for  $T_{11}$  by discussion.
1. Suppose  $|w_{1} - 1| + |w_{2}| \geqslant r_{0}$ . Then one has
$$
\left| T _ {1 1} \right| \leqslant \frac {1}{2} \sin (2 \psi) + \psi \leqslant \frac {1}{2 r _ {0}} [ \sin (2 \psi) + 2 \psi ] [ | w _ {1} - 1 | + | w _ {2} | ],
$$
where the first inequality holds from  $\theta \leqslant \frac{\pi}{2}$ .
2. Suppose  $|w_{1} - 1| + |w_{2}| \leqslant r_{0}$ . Then it is observed that  $w_{1} \geqslant 1 - r_{0} + |w_{2}| \geqslant 0$ . Thus,
$$
r = \sqrt {w _ {1} ^ {2} + w _ {2} ^ {2}} \geqslant \sqrt {(1 - r _ {0}) ^ {2} + 2 | w _ {2} | (| w _ {2} | + 1 - r _ {0})} \geqslant 1 - r _ {0},
$$
where the second inequality holds because of  $r_0 \leqslant 1$ . Then we can bound  $|w_2|$  from below as
$$
\left| w _ {2} \right| = r \sin \theta \geqslant \left(1 - r _ {0}\right) \sin \theta \geqslant \frac {1 - r _ {0}}{2} \theta , \tag {19}
$$
where the second inequality holds since  $\theta \leqslant 2\sin \theta$  for all  $\theta \in [0,\pi /2]$ . Meanwhile, we bound  $\theta$  from above as
$$
\theta \leqslant \tan \theta = \frac {\left| w _ {2} \right|}{w _ {1}} \leqslant \left(\frac {1 - r _ {0}}{\left| w _ {2} \right|} + 1\right) ^ {- 1} \leqslant \left(\frac {1 - r _ {0}}{r _ {0}} + 1\right) ^ {- 1} = r _ {0}, \tag {20}
$$
where the second inequality holds from  $w_{1} \geqslant 1 - r_{0} + |w_{2}|$ , and the third inequality holds based on  $|w_{2}| \leqslant r_{0}$ . Then we obtain an upper bound of  $T_{11}$  as follows
$$
| T _ {1 1} | \leqslant \theta \leqslant \frac {2 | w _ {2} |}{1 - r _ {0}} \leqslant \frac {4 \psi | w _ {2} |}{(1 - r _ {0}) (\pi - 2 r _ {0})} \leqslant \frac {2}{(1 - r _ {0}) (\pi - 2 r _ {0})} [ \sin (2 \psi) + 2 \psi ] [ | w _ {1} - 1 | + | w _ {2} | ],
$$
where the first inequality holds from the monotonicity of  $\frac{1}{2}\sin (2\psi) + \psi$  and  $\psi \leqslant \frac{\pi}{2}$ , the second inequality holds from Eq. (19), and the third inequality holds based on  $\psi \geqslant \frac{\pi}{2} -\theta$  and Eq. (20).
Combining the cases above, we have proven
$$
\left| T _ {1 1} \right| \leqslant \max  \left\{\frac {1}{2 r _ {0}}, \frac {2}{(1 - r _ {0}) (\pi - 2 r _ {0})} \right\} [ \sin (2 \psi) + 2 \psi ] [ | w _ {1} - 1 | + | w _ {2} | ].
$$
Choosing  $r_0 = \frac{1}{4}\left[\pi + 6 - \sqrt{p i^2 + 4\pi + 36}\right]$ , we obtain an upper bound of  $T_{11}$  as follows
$$
\left| T _ {1 1} \right| \leqslant \frac {3}{2} [ \sin (2 \psi) + 2 \psi ] [ \left| w _ {1} - 1 \right| + \left| w _ {2} \right| ]. \tag {21}
$$
Secondly, we provide an upper bound for  $T_{12}$ . We claim and prove by discussion that
$$
\left| w _ {2} \right| \leqslant 2 \sqrt {w _ {1} ^ {2} + w _ {2} ^ {2}} \left(\left| w _ {1} - 1 \right| + \left| w _ {2} \right|\right). \tag {22}
$$
1. Suppose  $w_{1} \leqslant 1 / 2$ . Then it is observed that  $|w_{1} - 1| \geqslant 1 / 2$ , which implies
$$
\left| w _ {2} \right| \leqslant \sqrt {w _ {1} ^ {2} + w _ {2} ^ {2}} \leqslant \sqrt {w _ {1} ^ {2} + w _ {2} ^ {2}} \cdot 2 \left| w _ {1} - 1 \right| \leqslant 2 \sqrt {w _ {1} ^ {2} + w _ {2} ^ {2}} \left(\left| w _ {1} - 1 \right| + \left| w _ {2} \right|\right).
$$
2. Suppose  $w_{1} \geqslant 1 / 2$ . Then one has  $\sqrt{w_1^2 + w_2^2} \geqslant 1 / 2$ , which indicates
$$
\left| w _ {2} \right| \leqslant \left| w _ {1} - 1 \right| + \left| w _ {2} \right| \leqslant 2 \sqrt {w _ {1} ^ {2} + w _ {2} ^ {2}} \left(\left| w _ {1} - 1 \right| + \left| w _ {2} \right|\right).
$$
From the definition of  $D_{2}$ , one has  $\frac{\pi}{2} \geqslant \psi \geqslant \frac{\pi}{2} - \theta \geqslant 0$ , which indicates
$$
\psi \geqslant \sin \psi \geqslant \sin \left(\frac {\pi}{2} - \theta\right) = \cos \theta = \frac {w _ {1}}{\sqrt {w _ {1} ^ {2} + w _ {2} ^ {2}}}. \tag {23}
$$
Then we obtain an upper bound of  $|T_{12}|$  as
$$
\left| T _ {1 2} \right| \leqslant \frac {2 w _ {1}}{\sqrt {w _ {1} ^ {2} + w _ {2} ^ {2}}} \left(\left| w _ {1} - 1 \right| + \left| w _ {2} \right|\right) \leqslant 2 \psi \left(\left| w _ {1} - 1 \right| + \left| w _ {2} \right|\right) \leqslant \left[ \sin (2 \psi) + 2 \psi \right] \left(\left| w _ {1} - 1 \right| + \left| w _ {2} \right|\right), \tag {24}
$$
where the first inequality holds according to Eq. (22), and the second inequality holds based on Eq. (23). Finally, combining Eqs. (21) and (24), we conclude
$$
T _ {1} \leqslant \left[ \left| \left[ \sin (2 \psi) + 2 \psi \right] \left(w _ {1} - 1\right) \right| + \max  \left\{\left| T _ {1 1} \right|, \left| T _ {1 2} \right| \right\} \right] ^ {2} \leqslant 7 \pi [ \sin (2 \psi) + 2 \psi ] \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2},
$$
where the first inequality holds based on  $T_{11} \geqslant 0$  and  $T_{12} \leqslant 0$ , and the second inequality holds because of  $\sin(2\psi) + 2\psi \leqslant \pi$  for any  $\psi \in [0, \pi/2]$ . Thus, we have completed the proof.
Lemma 18. Let  $T_{2} = \left(\left[\frac{1}{2}\cos (2\psi) - \frac{1}{2} +\frac{w_{1}^{2}}{w_{1}^{2} + w_{2}^{2}}\right]\operatorname {sgn}(w_{2}) + [\sin (2\psi) + 2\psi ]w_{2}\right)^{2}$ . If  $(\boldsymbol {w},\psi)\in D_2$ , then we have
$$
T _ {2} \leqslant 7 \pi [ \sin (2 \psi) + 2 \psi ] \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2}.
$$
Proof. From  $\cos \theta = w_1 / \sqrt{w_1^2 + w_2^2}$ , one has  $\cos (\pi - 2\theta) = 1 - 2\cos^2\theta = 1 - 2w_1^2 / (w_1^2 + w_2^2)$ . Thus, we have
$$
\left| \left[ \frac {1}{2} \cos (2 \psi) - \frac {1}{2} + \frac {w _ {1} ^ {2}}{w _ {1} ^ {2} + w _ {2} ^ {2}} \right] \operatorname {s g n} (w _ {2}) \right| = \frac {1}{2} | \cos (2 \psi) - \cos (\pi - 2 \theta) | \leqslant \psi + \theta - \frac {\pi}{2} \leqslant T _ {1 1},
$$
where the first inequality holds because of  $|\cos a - \cos b| \leqslant |a - b|$ , and the second inequality holds based on the definition of  $T_{11}$  in Eq. (18) and  $\sin(2\psi) \geqslant 0$ . Recalling the upper bound of  $T_{11}$  in Eq. (21), we obtain
$$
\begin{array}{l} T _ {2} \leqslant \left(\left| \left[ \frac {1}{2} \cos (2 \psi) - \frac {1}{2} + \frac {w _ {1} ^ {2}}{w _ {1} ^ {2} + w _ {2} ^ {2}} \right] \operatorname {s g n} (w _ {2}) \right| + | [ \sin (2 \psi) + 2 \psi ] w _ {2} |\right) ^ {2} \\ \leqslant 7 \pi [ \sin (2 \psi) + 2 \psi ] \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2}, \\ \end{array}
$$
In the main part of this section, we present the closed form of the loss, definition and properties of the ideal region, and the detailed proof of Theorem 2. Subsection C.1 provides the optimization behaviors. Subsection C.2 gives some convergence rate lemmas.
According to Lemma 8, the expected square loss  $L_{\mathrm{cc}}$  can be calculated by
$$
L _ {\mathrm {c c}} (\boldsymbol {w}, \psi_ {w}) = \frac {1}{2} B (\boldsymbol {w}, \boldsymbol {w}, \psi_ {w}, \psi_ {w}) - B (\boldsymbol {w}, \boldsymbol {v}, \psi_ {w}, \psi_ {v}) + \frac {1}{2} B (\boldsymbol {v}, \boldsymbol {v}, \psi_ {v}, \psi_ {v}). \tag {25}
$$
For  $R\in (0,1)$ $\psi_l\in [0,\delta_l]$  , and  $\psi_{u}\in [\pi /2 - \delta_{u},\pi /2]$  , define
$$
\begin{array}{l} D _ {1} = \left\{\left(\boldsymbol {w}, \psi_ {w}\right) \mid \| \boldsymbol {w} - \boldsymbol {v} \| _ {\infty} \leqslant R, \psi_ {w} \in \left[ \psi_ {l}, \psi_ {u} \right], \theta_ {\boldsymbol {w}, \boldsymbol {v}} \in \left[ 0, | \psi_ {w} - \psi_ {v} | \right] \right\}, \\ D _ {2} = \left\{\left(\boldsymbol {w}, \psi_ {w}\right) \mid \| \boldsymbol {w} - \boldsymbol {v} \| _ {\infty} \leqslant R, \psi_ {w} \in \left[ \psi_ {l}, \psi_ {u} \right], \theta_ {\boldsymbol {w}, \boldsymbol {v}} \in \left(| \psi_ {w} - \psi_ {v} |, \psi_ {w} + \psi_ {v}\right) \right\}. \\ \end{array}
$$
Let  $D = D_{1} \cup D_{2}$  indicate the ideal region, i.e.,
$$
D = \left\{\left(\boldsymbol {w}, \psi_ {w}\right) \mid \| \boldsymbol {w} - \boldsymbol {v} \| _ {\infty} \leqslant R, \psi_ {w} \in \left[ \psi_ {l}, \psi_ {u} \right], \theta_ {\boldsymbol {w}, \boldsymbol {v}} \in \left[ 0, \psi_ {w} + \psi_ {v} \right] \right\}.
$$
By spherical symmetry, we assume  $\pmb{v} = (1,0)$  without loss of generality in the rest proof. For conciseness, define  $s_w = \sin(2\psi_w) + 2\psi_w$  and  $s_v = \sin(2\psi_v) + 2\psi_v$ . The following lemma discusses the properties of the ideal region, concerning the closeness of the region under gradient descent and the probability that an initialization falls into this region.
Lemma 19. Let  $\psi_v \in [7\pi/20, 2\pi/5]$ . If we choose the parameters as
$$
R = \frac {1}{2 5}, \quad \psi_ {l} = \psi_ {v} - \frac {1 0 9}{1 0 0} R, \quad \psi_ {u} = \psi_ {v} + \frac {1 0 9}{1 0 0} R, \quad a n d \quad 0 <   \eta \leqslant \frac {1}{1 2 0} R,
$$
then all conditions in Lemmas 20-25 are satisfied. If  $\pmb{w}_0\sim \mathcal{N}(0,I_2)$  and  $\psi_{w,0}\sim \mathcal{U}(0,\pi /2)$ , then
$$
\Pr \left[ \left(\boldsymbol {w} _ {0}, \psi_ {w, 0}\right) \in D \right] \geqslant 1 0 ^ {- 5}.
$$
Proof. We first prove that all conditions in the lemmas are satisfied.
- Lemma 20. It is observed that the first condition holds from
$$
\eta \leqslant \frac {1}{1 2 0} R = \frac {1}{1 2 0} \cdot \frac {1}{2 5} <   2.
$$
According to  $\psi_u > \psi_v > \pi / 4$ , we have  $\psi_v \sin(2\psi_u) \leqslant \psi_u \sin(2\psi_v)$ , which implies
$$
s _ {v} \geqslant \frac {\psi_ {v} s _ {u}}{\psi_ {u}} = \frac {\psi_ {v} s _ {u}}{\psi_ {v} + 1 0 9 R / 1 0 0} \geqslant \frac {7 \pi s _ {u} / 2 0}{7 \pi / 2 0 + 1 0 9 R / 1 0 0} \geqslant (1 - R) s _ {u} \geqslant (1 - R) s _ {w},
$$
where the fourth inequality holds since  $s_w$  is monotonic. Thus, the second condition is satisfied.
- Lemma 21. The first condition  $\eta < 2$  has been satisfied above. It is observed that  $\psi_{l} \geqslant 7\pi / 20 - 109R / 100$ . Thus, the second condition holds from  $\psi_{l} / 20 \geqslant 7\pi / 400 - 109R / 2000 \geqslant R$ . The third condition holds since
$$
\max  \left\{\psi_ {u} - \psi_ {v}, \psi_ {v} - \psi_ {l} \right\} = \frac {1 0 9 R}{1 0 0} \leqslant \frac {5 R \psi_ {l}}{3}.
$$
- Lemma 22. The only condition  $\eta < 2$  has been satisfied.
- Lemma 23. The first condition holds because of  $R = 1 / 25 \leqslant 1 / 2$ . The second condition holds based on  $\cos^2 \psi_v \geqslant \cos^2 (2\pi / 5) \geqslant 1 / 25$ . The third condition holds from  $\eta \leqslant R / 120 \leqslant 3R / 2$ .
- Lemma 24. The first condition  $R \leqslant 1/2$  has been satisfied above. The second and third conditions hold because of
$$
\frac {\pi}{3} \min  \left\{\psi_ {u} - \psi_ {v}, \psi_ {v} - \psi_ {l} \right\} = \frac {\pi}{3} \cdot \frac {1 0 9 R}{1 0 0} \geqslant \frac {R}{1 2 0} \geqslant \eta .
$$
- Lemma 25. The first condition  $R \leqslant 1/2$  has been satisfied above. The second one holds from
$$
\arcsin R + 9 \eta \leqslant \frac {1 0 1 R}{1 0 0} + \frac {3 R}{4 0} \leqslant \frac {1 0 9 R}{1 0 0} = \psi_ {u} - \psi_ {v}.
$$
We then prove the second conclusion. Let  $p_0 = \operatorname*{Pr}[(\boldsymbol{w}_0, \psi_{w,0}) \in D]$  for simplicity. Then we have
$$
\begin{array}{l} p _ {0} = \Pr [ \psi_ {l} \leqslant \psi_ {w, 0} \leqslant \psi_ {u} ] \cdot \Pr [ 1 - R \leqslant w _ {1} \leqslant 1 + R ] \cdot \Pr [ - R \leqslant w _ {2} \leqslant R ] \\ = \frac {1 0 9 R}{5 0} \cdot \frac {1}{2} [ \operatorname {e r f} (1 + R) - \operatorname {e r f} (1 - R) ] \cdot \operatorname {e r f} (R) \\ \geqslant 1 0 ^ {- 5}, \\ \end{array}
$$
where  $\operatorname{erf}(x)$  denotes the error function. Thus, we have completed the proof.
We are now ready to prove Theorem 2.
Proof of Theorem 2. Let  $R$ ,  $\psi_l$ , and  $\psi_u$  be the same as those in Lemma 19. Suppose that  $(\pmb{w}_0, \psi_{\pmb{w},0}) \in D$ . Then Lemma 19 implies  $(\pmb{w}_t, \psi_{\pmb{w},t}) \in D$  for any  $t \in \mathbb{N}$ . The proof of convergence is divided into several stages.
Step 1:  $w_{2}$  converges to 0. In stage I, we consider the convergence of  $w_{2,t}$  when  $(\boldsymbol{w}_0, \psi_{w,0}) \in D$ . From Lemmas 22 and 23, the optimization behaviors of  $w_{2}$  is the combination of minimizing a contraction mapping or an almost absolute function. Thus, Lemma 26 with  $r_1 = r_2 = R$ ,  $c_{3} = s_{w} / (2\pi)$ ,  $g_{l} = (\cos^{2}\psi_{v} - \sqrt{2} R) / (2\pi)$ , and  $g_{u} = 2 / 3$  implies
$$
\left| w _ {2} \right| \leqslant \frac {c _ {2} ^ {2} \left(\cos^ {2} \psi_ {v} - \sqrt {2} R\right)}{4 \pi c _ {1} t} \leqslant \frac {c _ {2} ^ {2}}{4 \pi c _ {1} t} \quad \text {f o r} \quad t \in \mathbb {N} ^ {+}. \tag {26}
$$
Step 2:  $\psi_w$  converges to  $\psi_v$ . In stage II, we prove the convergence of  $\psi_{w,t}$  when  $(\pmb{w}_0,\psi_{w,0})\in D$ . From Lemmas 24 and 25, the convergence of  $\psi_w$  is limited by that of  $w_{2}$ , i.e.,  $\psi_w$  tends to the global minimum with constant-order gradient when the error of  $\psi_w$  is larger than that of  $w_{2}$ , while becomes far away from the global minimum otherwise. Then Lemma 27 with  $r_1 = r_2 = 109R / 100$ ,  $a = c_2^2 (\cos^2\psi_v - \sqrt{2} R) / (4\pi c_1)$ ,  $g_l = \cos^2\psi_u / (4\pi)$ , and  $g_u = 9$  indicates
$$
\left| \psi_ {w} - \psi_ {v} \right| \leqslant \left[ \frac {c _ {2} ^ {2} \left(\cos^ {2} \psi_ {v} - \sqrt {2} R\right)}{4 \pi c _ {1}} + 9 c _ {2} \right] \frac {1}{t} \leqslant \frac {1 0 c _ {2} ^ {2}}{c _ {1} t} \quad \text {f o r} \quad t \in \mathbb {N} ^ {+}. \tag {27}
$$
Step 3:  $w_{1}$  converges to 1. In stage III, we investigate the convergence of  $w_{1,t}$  when  $(\pmb{w}_0, \psi_{w,0}) \in D$ . From Lemmas 20 and 21, the gradient points to the global minimum with a remainder controlled by the error of  $w_{1}$  and  $\psi_w$ . Then Lemma 28 with  $d_{l} = 1 / 4$ ,  $d_u = 1 / 2$ , and  $e = 20c_2^2 / (\pi c_1)$  leads to