SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[35] Mahdi Soltanolkotabi. Learning RLUs via gradient descent. In Advances in Neural Information Processing Systems 30, pages 2007-2017, 2017.
[36] Simon S Du, Jason D Lee, and Yuandong Tian. When is a convolutional filter easy to learn? In Proceedings of the 6th International Conference on Learning Representations, 2018.
[37] Song Mei, Yu Bai, and Andrea Montanari. The landscape of empirical risk for nonconvex losses. The Annals of Statistics, 46(6A):2747-2774, 2018.
[38] Seyed Mohammadreza Mousavi Kalan, Mahdi Soltanolkotabi, and A Salman Avestimehr. Fitting RLUs via SGD and quantized SGD. In Proceedings of 2019 IEEE International Symposium on Information Theory, pages 2469-2473, 2019.
[39] Nitzan Guberman. On complex valued convolutional neural networks. arXiv:1602.09046, 2016.
[40] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pages 249-256, 2010.
[41] Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer ReLU neural networks. In Proceedings of the 35th International Conference on Machine Learning, pages 4433-4441, 2018.
[42] Zhi-Hua Zhou. Why over-parameterization of deep neural networks does not overfit? Science China Information Sciences, 64:1-3, 2021.
[43] Zhi-Hao Tan, Yi Xie, Yuan Jiang, and Zhi-Hua Zhou. Real-valued backpropagation is unsuitable for complex-valued neural networks. Advances in Neural Information Processing Systems 35, pages 34052-34063, 2022.
Appendix of “Complex-valued Neurons Can Learn More but Slower than Real-valued Neurons via Gradient Descent”
In this section, we first summarize frequently used notations in the following table.
Table 4: Frequently used notations.
Notation	Description
C^d	the d-dimensional complex space
E	expectation
I(·)	the indicator function
L	the expected square loss of learning a neuron
N(0,I)	the standard Gaussian distribution
O,Ω,Θ	asymptotic notations
Pr	probability
P_Q(x)	the projection of x on Q
R^2d	the 2d-dimensional real space
Re(z)	the real part of a complex number z
t	the iteration index of gradient descent
U(a,b)	the uniform distribution on the interval [a,b]
v	the weight vector of a learning neuron
w	the weight vector of a target neuron
x	an input vector in R^2d
xi	the i-th coordinate of x
xC	xC=(x1;...;xd)+(xd+1;...;xd)i∈C^d
X̅C	the complex conjugate of xC
θa,b	the angle between a and b
θz	the argument of a complex number z
σψ(z)	the real part of the symmetrical version of zReLU activation function
η	the step size of gradient descent
τ	theReLU activation function τ(x)=max{0,x}
ψ	the learnable parameter of the symmetrical version of zReLU activation function
∇	gradient
||·||	the 2-norm of a vector
We then give some basic lemmas that help us calculate the closed form of the expected loss.
Lemma 7. Let  $d = 1$ . For any  $\pmb{w}, \pmb{v} \in \mathbb{R}^{2d}$ , and  $a \leqslant b \leqslant a + 2\pi$ , we have
$$
\begin{array}{l} A (\boldsymbol {w}, \boldsymbol {v}, a, b) = \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {N} (\boldsymbol {0}, \mathbf {I})} \left[ \boldsymbol {w} ^ {\top} \boldsymbol {x} \cdot \boldsymbol {v} ^ {\top} \boldsymbol {x} \cdot \mathbb {I} \left(\theta_ {x} \in [ a, b ]\right) \right] \\ = \frac {\| \boldsymbol {w} \| \| \boldsymbol {v} \|}{4 \pi} \left[ 2 (b - a) \cos \theta_ {\boldsymbol {w}, \boldsymbol {v}} + \sin (\theta_ {\boldsymbol {w}} + \theta_ {\boldsymbol {v}} - 2 a) - \sin (\theta_ {\boldsymbol {w}} - \theta_ {\boldsymbol {v}} - 2 b) \right]. \\ \end{array}
$$
Proof. According to the probability density function of Gaussian distribution, we can calculate  $A$  in the polar coordinate system as
$$
\begin{array}{l} A (\boldsymbol {w}, \boldsymbol {v}, a, b) = \frac {\| \boldsymbol {w} \| \| \boldsymbol {v} \|}{2 \pi} \int_ {0} ^ {\infty} \int_ {a} ^ {b} r ^ {3} \mathrm {e} ^ {- \frac {1}{2} r ^ {2}} \cos (\theta_ {\boldsymbol {w}} - \phi) \cos (\theta_ {\boldsymbol {v}} - \phi) \mathrm {d} \phi \mathrm {d} r \\ = \frac {\| \boldsymbol {w} \| \| \boldsymbol {v} \|}{\pi} \int_ {a} ^ {b} \cos \left(\theta_ {\boldsymbol {w}} - \phi\right) \cos \left(\theta_ {\boldsymbol {v}} - \phi\right) d \phi \\ = \frac {\| \pmb {w} \| \| \pmb {v} \|}{4 \pi} \left[ 2 (b - a) \cos \theta_ {\pmb {w}, \pmb {v}} + \sin (\theta_ {\pmb {w}} + \theta_ {\pmb {v}} - 2 a) - \sin (\theta_ {\pmb {w}} - \theta_ {\pmb {v}} - 2 b) \right], \\ \end{array}
$$
where the second and third equalities hold from integrating over  $r$  and  $\phi$ , respectively. Thus, we have completed the proof.
Lemma 8. Let  $d = 1$ . For any  $\boldsymbol{w}, \boldsymbol{v} \in \mathbb{R}^{2d}$ , denote by  $\theta = \theta_{\boldsymbol{w}, \boldsymbol{v}}$  the angle between  $\boldsymbol{w}$  and  $\boldsymbol{v}$ . Then for any  $\psi_w, \psi_v \in [0, \pi/2]$ , define  $\psi_m = \min\{\psi_w, \psi_v\}$ . Then we have
$$
\begin{array}{l} B (\boldsymbol {w}, \boldsymbol {v}, \psi_ {w}, \psi_ {v}) = \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} \left[ \sigma_ {\psi_ {w}} \left(\boldsymbol {w} _ {\mathbb {C}} ^ {\top} \overline {{\boldsymbol {x}}} _ {\mathbb {C}}\right) \sigma_ {\psi_ {v}} \left(\boldsymbol {v} _ {\mathbb {C}} ^ {\top} \overline {{\boldsymbol {x}}} _ {\mathbb {C}}\right) \right] \\ = \left\{ \begin{array}{l l} \frac {\| \pmb {w} \| \| \pmb {v} \|}{2 \pi} \cos \theta_ {\pmb {w}, \pmb {v}} [ 2 \psi_ {m} + \sin (2 \psi_ {m}) ], & \theta_ {\pmb {w}, \pmb {v}} \in [ 0, | \psi_ {v} - \psi_ {w} | ], \\ \frac {\| \pmb {w} \| \| \pmb {v} \|}{4 \pi} [ 2 (\psi_ {w} + \psi_ {v} - \theta_ {\pmb {w}, \pmb {v}}) \cos \theta_ {\pmb {w}, \pmb {v}} - \sin (\theta_ {\pmb {w}, \pmb {v}} - 2 \psi_ {v}) \\ - \sin (\theta_ {\pmb {w}, \pmb {v}} - 2 \psi_ {w}) ], & \theta_ {\pmb {w}, \pmb {v}} \in [ | \psi_ {v} - \psi_ {w} |, \psi_ {v} + \psi_ {w} ], \\ 0, & \theta_ {\pmb {w}, \pmb {v}} \in [ \psi_ {v} + \psi_ {w}, \pi ]. \end{array} \right. \\ \end{array}
$$
Proof. We only consider the case of  $\psi_w \leqslant \psi_v$ . The other case  $\psi_w \geqslant \psi_v$  can be proven similarly. We prove the conclusion by discussion.
1. Suppose  $\theta_{\pmb{w},\pmb{v}}\in [0,\psi_v - \psi_w]$ . Then Lemma 7 leads to
$$
B (\boldsymbol {w}, \boldsymbol {v}, \psi_ {w}, \psi_ {v}) = A (\boldsymbol {w}, \boldsymbol {v}, \theta_ {\boldsymbol {w}} - \psi_ {w}, \theta_ {\boldsymbol {w}} + \psi_ {w}) = \frac {\| \boldsymbol {w} \| \| \boldsymbol {v} \|}{2 \pi} \cos \theta_ {\boldsymbol {w}, \boldsymbol {v}} [ 2 \psi_ {w} + \sin (2 \psi_ {w}) ].
$$
2. Suppose  $\theta_{\pmb{w},\pmb{v}} \in [\psi_v - \psi_w, \psi_v + \psi_w]$  and  $\theta_{\pmb{w}} \leqslant \theta_{\pmb{v}}$ . Then one knows from Lemma 7 that
$$
\begin{array}{l} B (\boldsymbol {w}, \boldsymbol {v}, \psi_ {w}, \psi_ {v}) = A (\boldsymbol {w}, \boldsymbol {v}, \theta_ {\boldsymbol {v}} - \psi_ {v}, \theta_ {\boldsymbol {w}} + \psi_ {w}) \\ = \frac {\| \boldsymbol {w} \| \| \boldsymbol {v} \|}{4 \pi} \left[ 2 \left(\psi_ {w} + \psi_ {v} - \theta_ {\boldsymbol {w}, \boldsymbol {v}}\right) \cos \theta_ {\boldsymbol {w}, \boldsymbol {v}} - \sin \left(\theta_ {\boldsymbol {w}, \boldsymbol {v}} - 2 \psi_ {v}\right) - \sin \left(\theta_ {\boldsymbol {w}, \boldsymbol {v}} - 2 \psi_ {w}\right) \right]. \\ \end{array}
$$
3. Suppose  $\theta_{\pmb{w},\pmb{v}} \in [\psi_v - \psi_w, \psi_v + \psi_w]$  and  $\theta_{\pmb{w}} \geqslant \theta_v$ . Based on Lemma 7, we have
$$
\begin{array}{l} B (\boldsymbol {w}, \boldsymbol {v}, \psi_ {w}, \psi_ {v}) = A (\boldsymbol {w}, \boldsymbol {v}, \theta_ {\boldsymbol {w}} - \psi_ {w}, \theta_ {\boldsymbol {v}} + \psi_ {v}) \\ = \frac {\| \boldsymbol {w} \| \| \boldsymbol {v} \|}{4 \pi} \left[ 2 \left(\psi_ {w} + \psi_ {v} - \theta_ {\boldsymbol {w}, \boldsymbol {v}}\right) \cos \theta_ {\boldsymbol {w}, \boldsymbol {v}} - \sin \left(\theta_ {\boldsymbol {w}, \boldsymbol {v}} - 2 \psi_ {v}\right) - \sin \left(\theta_ {\boldsymbol {w}, \boldsymbol {v}} - 2 \psi_ {w}\right) \right]. \\ \end{array}
$$
4. Suppose  $\theta_{\pmb{w},\pmb{v}} \in [\psi_v + \psi_w, \pi]$ . Then the support of  $\sigma_{\psi_w}(\pmb{w}_{\mathbb{C}}^{\top}\overline{\pmb{x}}_{\mathbb{C}})$  does not overlap with that of  $\sigma_{\psi_v}(\pmb{v}_{\mathbb{C}}^{\top}\overline{\pmb{x}}_{\mathbb{C}})$ , which leads to  $B(\pmb{w},\pmb{v},\psi_w,\psi_v) = 0$ .
Combining the cases above completes the proof.
In the main part of this section, we provide the closed form of the loss, definition of the ideal region, and the detailed proof of Theorem 1. Subsection B.1 presents the optimization behaviors in the ideal region. Subsection B.2 proves several convergence rate lemmas. Subsection B.3 gives some technical lemmas to bound small terms in the proof.
Let  $\pmb{w} = (w_{1}, w_{2})$ . According to the spherical symmetry, we assume  $\pmb{v} = (1, 0)$  without loss of generality. According to Lemma 8, the expected loss can be calculated by
$$
\begin{array}{l} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) = \frac {1}{2} B (\boldsymbol {w}, \boldsymbol {w}, \psi , \psi) - B (\boldsymbol {w}, \boldsymbol {v}, \psi , \pi / 2) + \frac {1}{2} B (\boldsymbol {v}, \boldsymbol {v}, \pi / 2, \pi / 2) \\ = \left\{ \begin{array}{l l} \frac {1}{4} - \frac {1}{4 \pi} [ \sin (2 \psi) + 2 \psi ] \left[ 1 - \left(w _ {1} - 1\right) ^ {2} - w _ {2} ^ {2} \right], & \theta \in [ 0, \pi / 2 - \psi ], \\ \frac {1}{4} - \frac {1}{2 \pi} \left[ \frac {1}{2} \sin (2 \psi) w _ {1} - \frac {1}{2} \cos (2 \psi) | w _ {2} | + \frac {1}{2} | w _ {2} | + (\frac {\pi}{2} + \psi - \theta) w _ {1} \right] & \\ + \frac {1}{4 \pi} [ \sin (2 \psi) + 2 \psi ] \left(w _ {1} ^ {2} + w _ {2} ^ {2}\right), & \theta \in (\pi / 2 - \psi , \pi / 2 + \psi), \\ \frac {1}{4} + \frac {1}{4 \pi} [ 2 \psi + \sin (2 \psi) ] \left(w _ {1} ^ {2} + w _ {2} ^ {2}\right), & \theta \in [ \pi / 2 + \psi , \pi ], \end{array} \right. \tag {3} \\ \end{array}
$$
where  $\theta = \theta_{\pmb{w},\pmb{v}} = \arccos (w_1 / \sqrt{w_1^2 + w_2^2})$ . For any  $R\in (0,1)$ , define
$$
\begin{array}{l} D _ {1} = \left\{ \right.\left( \right.\boldsymbol {w}, \psi) \mid \| \boldsymbol {w} - \boldsymbol {v} \| \leqslant R, \psi \in [ 0, \pi / 2 ], \theta \in [ 0, \pi / 2 - \psi ] \left. \right\}, \\ D _ {2} = \left\{\left(\boldsymbol {w}, \psi\right) \mid \| \boldsymbol {w} - \boldsymbol {v} \| \leqslant R, \psi \in [ 0, \pi / 2 ], \theta \in (\pi / 2 - \psi , \pi / 2 + \psi) \right\}. \\ \end{array}
$$
Let  $D = D_{1} \cup D_{2}$  denote the ideal region, i.e.,
$$
D = \left\{\left(\boldsymbol {w}, \psi\right) \mid \| \boldsymbol {w} - \boldsymbol {v} \| \leqslant R, \psi \in [ 0, \pi / 2 ], \theta \in [ 0, \pi / 2 + \psi ] \right\}.
$$
We are now ready to prove Theorem 1.
Proof of Theorem 1. The proof is divided into four steps.
Step 1:  $D$  is closed under gradient descent. Before considering the convergence, we prove the maintenance of inclusion by mathematical induction, i.e.,  $(\pmb{w}_0,\psi_0)\in D$  indicates  $(\pmb {w}_t,\psi_t)\in D$
1. Base case. The conclusion holds for  $t = 0$  from the condition.
2. Induction. Suppose that the conclusion holds for  $t = k$  with  $k \in \mathbb{N}$ . Then based on Lemmas 11 and 12, one knows
$$
- 6 \left(\psi^ {*} - \psi_ {k}\right) \leqslant \nabla_ {\psi} L _ {\mathrm {c r}} \left(\boldsymbol {w} _ {k}, \psi_ {k}\right) \leqslant - \frac {1 - R ^ {2}}{4 \pi} \left(\psi^ {*} - \psi_ {k}\right) ^ {2} \leqslant 0, \tag {4}
$$
where  $\psi^{*} = \pi /2$ , the first inequality holds based on the induction hypothesis and  $|w_{2,k}| \leqslant 1$ , and the third inequality holds from  $R < 1$ . Thus, the updating rule  $\psi_{k + 1} = \psi_k - \eta \nabla_\psi L_{\mathrm{cr}}(\pmb {w}_k,\psi_k)$  with  $\eta \in (0,1 / (12\pi))$  leads to
$$
\frac {\pi}{2} \geqslant \psi^ {*} - \psi_ {k} \geqslant \psi^ {*} - \psi_ {k + 1} \geqslant (1 - 6 \eta) (\psi^ {*} - \psi_ {k}) \geqslant 0, \tag {5}
$$
where the first and fourth inequalities hold from the induction hypothesis. Meanwhile, Lemmas 9 and 10 imply
$$
\left\| \boldsymbol {w} _ {k + 1} - \boldsymbol {v} \right\| \leqslant \left(1 - \frac {\eta}{2 4 \pi} \left[ \sin \left(2 \psi_ {k}\right) + 2 \psi_ {k} \right]\right) \| \boldsymbol {w} _ {k} - \boldsymbol {v} \| \leqslant R. \tag {6}
$$
Combining Eqs. (5) and (6), the conclusion holds for  $t = k + 1$ .
Therefore, mathematical induction implies  $(\pmb{w}_t,\psi_t)\in D$  when  $(\pmb {w}_0,\psi_0)\in D$
Step 2: parameters converge to the global minimum in  $D$ . The convergence process consists of two stages. In stage I, we deal with the convergence of  $\psi$  when  $(\boldsymbol{w}_0, \psi_0) \in D$ . Based on Eq. (4) and the updating rule  $\psi_{k+1} = \psi_k - \eta \nabla_\psi L_{\mathrm{cr}}(\boldsymbol{w}_k, \psi_k)$ , one knows
$$
\psi^ {*} - \psi_ {t + 1} \leqslant (\psi^ {*} - \psi_ {t}) \left[ 1 - \frac {\eta (1 - R ^ {2})}{4 \pi} (\psi^ {*} - \psi_ {t}) \right].
$$
Define  $a_{t} = \eta (1 - R^{2})(\psi^{*} - \psi_{t}) / (4\pi)$ . Then we obtain  $a_{t + 1}\leqslant a_t(1 - a_t)$ . From  $\psi^* -\psi_t\in [0,\pi /2]$  and  $\eta < 1 / (12\pi)\leqslant 4$ , one knows  $a_{t}\in [0,1 / 2]$ . Thus, applying Lemma 14 to  $a_{t}$  leads to
$$
\psi^ {*} - \psi_ {t} = \frac {4 \pi a _ {t}}{\eta (1 - R ^ {2})} \leqslant \frac {4 \pi}{\eta (1 - R ^ {2}) (t + 1)}. \tag {7}
$$
In stage II, we consider the convergence of  $\pmb{w}$  when  $(\pmb{w}_0,\psi_0)\in D$  . Based on Eq. (7), choosing  $T_{1}\geqslant 16[\eta (1 - R^{2})]^{-1}$  leads to  $\psi^{*} - \psi_{t}\leqslant \pi /4$  for any  $t\geqslant T_1$  , i.e.,  $\psi_t\geqslant \pi /4$  for any  $t\geqslant T_1$  . Thus, for any  $t\geqslant T_1$  , Eq. (6) indicates
$$
\left\| \boldsymbol {w} _ {t} - \boldsymbol {v} \right\| \leqslant \left(1 - \frac {\eta}{4 8}\right) \left\| \boldsymbol {w} _ {t - 1} - \boldsymbol {v} \right\| \leqslant \left(1 - \frac {\eta}{4 8}\right) ^ {t - T _ {1}} , \tag {8}
$$
where the first inequality holds from the monotonic increasing of  $\sin (x) + x$  and  $\psi_t\geqslant \pi /4$ , and the second inequality holds because of  $\| \pmb{w}_{T_1} - \pmb {v}\| \leqslant R < 1$ .
Step 3: the loss converges to 0 in  $D$ . We estimate the convergence of the expected loss when  $(\pmb{w}_0, \psi_0) \in D$ . For any  $(\pmb{w}, \psi) \in D$ , define non-negative quantities  $\Delta_{\pmb{w}} = \| \pmb{w} - \pmb{v} \|$  and  $\Delta_{\psi} = \psi^* - \psi$ . We provide an upper bound for  $L_{\mathrm{cr}}$  by discussion.
1. Suppose  $(\pmb{w},\psi)\in D_1$  . Then we have
$$
L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) \leqslant \frac {1}{4} - \frac {1}{2 \pi} \left(\psi^ {*} - \Delta_ {\psi} ^ {3}\right) \left(1 - \Delta_ {\boldsymbol {w}} ^ {2}\right) \leqslant \frac {1}{2 \pi} \Delta_ {\psi} ^ {3} + \frac {1}{4} \Delta_ {\boldsymbol {w}} ^ {2}, \tag {9}
$$
where the first inequality holds based on  $\sin (2\psi) + 2\psi = \sin (2\Delta_{\psi}) + 2\psi^{*} - 2\Delta_{\psi}\geqslant 2\psi^{*} - 2\Delta_{\psi}^{3}$ , and the second inequality holds from non-negative  $\Delta_{\psi}$ .
2. Suppose  $(\pmb{w},\psi)\in D_2$ . The expected loss can be rewritten as
$$
\begin{array}{l} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) = \frac {1}{4} - \frac {1}{4 \pi} [ \sin (2 \psi) + 2 \psi ] \left(1 - \Delta_ {\boldsymbol {w}} ^ {2}\right) \\ + \frac {1}{4 \pi} [ (\cos (2 \psi) - 1) | w _ {2} | + (\sin (2 \psi) + 2 \psi + 2 \theta - 2 \psi^ {*}) w _ {1} ] \\ \leqslant \frac {1}{4} - \frac {1}{2 \pi} \left(\psi^ {*} - \Delta_ {\psi} ^ {3}\right) \left(1 - \Delta_ {\boldsymbol {w}} ^ {2}\right) + \frac {1}{4 \pi} \left[ (\pi + 2 \theta - 2 \psi^ {*}) w _ {1} \right] \tag {10} \\ \leqslant \frac {1}{4} - \frac {1}{2 \pi} \left(\psi^ {*} - \Delta_ {\psi} ^ {3}\right) \left(1 - \Delta_ {\boldsymbol {w}} ^ {2}\right) + \frac {1}{2 \pi} \Delta_ {\boldsymbol {w}} \left(1 + \Delta_ {\boldsymbol {w}}\right) \\ \leqslant \frac {1}{2 \pi} \Delta_ {\psi} ^ {3} + \frac {1}{2 \pi} \Delta_ {\boldsymbol {w}} + \frac {1}{2} \Delta_ {\boldsymbol {w}} ^ {2}, \\ \end{array}
$$
where the first inequality holds from  $\pi \geqslant \sin (2\psi) + 2\psi \geqslant 2\psi^{*} - 2\Delta_{\psi}^{3}$  and  $\cos (2\psi) - 1 \leqslant 0$ , the second inequality holds based on  $\theta \leqslant \tan \theta \leqslant \Delta_{\boldsymbol{w}}$  and  $w_{1} \leqslant 1 + \Delta_{\boldsymbol{w}}$ , and the third inequality holds from  $\Delta_{\psi} \geqslant 0$ .
Combining Eqs. (9) and (10), one knows that the following holds for any  $(\pmb{w}_0, \psi_0) \in D$  and  $t \geqslant T_1$
$$
L _ {\mathrm {c r}} \left(\boldsymbol {w} _ {t}, \psi_ {t}\right) \leqslant \frac {1}{2 \pi} \Delta_ {\psi , t} ^ {3} + \Delta_ {\boldsymbol {w}, t} \leqslant \frac {3 2 \pi^ {3}}{\eta^ {3} (1 - R ^ {2}) ^ {3} t ^ {3}} + \left(1 - \frac {\eta}{4 8}\right) ^ {t - T _ {1}}, \tag {11}
$$
where the first inequality holds from  $\Delta_{\pmb{w}}^2 \leqslant \Delta_{\pmb{w}}$ , and the second inequality holds by Eqs. (7) and (8).
Step 4: initialization falls into  $D$  with constant probability. Let  $p_0 = \operatorname*{Pr}[(\boldsymbol{w}_0, \psi_0) \in D]$  for simplicity. From  $\psi_0 \sim \mathcal{U}(0, \pi/2)$ , the requirement  $\psi \in [0, \pi/2]$  is satisfied. Denote by  $p(\boldsymbol{w})$  the probability density function of  $\mathcal{N}(0, I_2)$ . Then one has
$$
p _ {0} = \Pr [ \| \boldsymbol {w} _ {0} - \boldsymbol {v} \| \leqslant R ] = \int_ {\boldsymbol {w} \in B (\boldsymbol {v}, R)} p (\boldsymbol {w}) \mathrm {d} \boldsymbol {w} \geqslant \mu (B (\boldsymbol {v}, R)) \min  _ {\boldsymbol {w} \in B (\boldsymbol {v}, R)} p (\boldsymbol {w}) \geqslant \frac {R ^ {2}}{1 6}. \tag {12}
$$
Let  $R^2 = 1 / 2$ . We obtain from Eqs. (11) and (12) that
$$
\Pr \left[ L _ {\mathrm {c r}} \left(\boldsymbol {w} _ {t}, \psi_ {t}\right) \leqslant \frac {8 0 0 0}{\eta^ {3} t ^ {3}} + \left(1 - \frac {\eta}{4 8}\right) ^ {t + 1 - 3 2 / \eta} \right] \geqslant \frac {1}{3 2},
$$
B.1 Optimization Behaviors
The following two lemmas indicate the linear convergence of  $\pmb{w}$  in  $D_{1}$  and  $D_{2}$ , respectively.
Lemma 9. Let  $\pmb{w}^{\prime} = \pmb{w} - \eta \nabla_{\pmb{w}}L_{\mathrm{cr}}(\pmb{w},\psi)$ . If  $(\pmb{w},\psi) \in D_1$  and  $\eta \in (0,4)$ , then we have
$$
\left\| \boldsymbol {w} ^ {\prime} - \boldsymbol {v} \right\| \leqslant \left(1 - \frac {\eta}{4 \pi} [ \sin (2 \psi) + 2 \psi ]\right) \| \boldsymbol {w} - \boldsymbol {v} \| .
$$
Proof. For any  $(\pmb{w},\psi)\in D_1$  , one has
$$
\langle \nabla_ {\boldsymbol {w}} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi), \boldsymbol {w} - \boldsymbol {v} \rangle = \left\langle \frac {1}{4 \pi} [ \sin (2 \psi) + 2 \psi ] (\boldsymbol {w} - \boldsymbol {v}), \boldsymbol {w} - \boldsymbol {v} \right\rangle = \frac {1}{4 \pi} [ \sin (2 \psi) + 2 \psi ] \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2}.
$$
Meanwhile,
$$
\| \nabla_ {\boldsymbol {w}} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) \| ^ {2} = \frac {1}{(4 \pi) ^ {2}} [ \sin (2 \psi) + 2 \psi ] ^ {2} \| (\boldsymbol {w} - \boldsymbol {v}) \| ^ {2}.
$$
Then according to Lemma 13 and  $\psi \in [0,\pi /2]$ , for any  $\eta \in (0,4)$ , one has
$$
\left\| \boldsymbol {w} ^ {\prime} - \boldsymbol {v} \right\| \leqslant \left(1 - \frac {\eta}{4 \pi} [ \sin (2 \psi) + 2 \psi ]\right) \| \boldsymbol {w} - \boldsymbol {v} \|,
$$
Lemma 10. Let  $\pmb{w}^{\prime} = \pmb{w} - \eta \nabla_{\pmb{w}}L_{\mathrm{cr}}(\pmb {w},\psi)$ . If  $(\pmb {w},\psi)\in D_2$  and  $\eta \in (0,1 / (12\pi))$ , then we have
$$
\left\| \boldsymbol {w} ^ {\prime} - \boldsymbol {v} \right\| \leqslant \left(1 - \frac {\eta}{2 4 \pi} [ \sin (2 \psi) + 2 \psi ]\right) \| \boldsymbol {w} - \boldsymbol {v} \| .
$$
Proof. Firstly, we prove the strong convexity in  $D_{2}$ . For any  $(\boldsymbol{w},\psi)\in D_2$ , one has
$$
\begin{array}{l} 2 \pi \langle \nabla_ {\boldsymbol {w}} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi), \boldsymbol {w} - \boldsymbol {v} \rangle \\ = - \left[ \frac {1}{2} \sin (2 \psi) + \left(\frac {\pi}{2} + \psi - \theta\right) + \frac {w _ {1} | w _ {2} |}{w _ {1} ^ {2} + w _ {2} ^ {2}} \right] \left(w _ {1} - 1\right) + \left[ \sin (2 \psi) + 2 \psi \right] w _ {1} \left(w _ {1} - 1\right) \tag {13} \\ - \left[ - \frac {1}{2} \cos (2 \psi) + \frac {1}{2} - \frac {w _ {1} ^ {2}}{w _ {1} ^ {2} + w _ {2} ^ {2}} \right] | w _ {2} | + [ \sin (2 \psi) + 2 \psi ] w _ {2} ^ {2} \\ = \left[ \sin (2 \psi) + 2 \psi \right] \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2} - R _ {1} - R _ {2}, \\ \end{array}
$$
$$
R _ {1} = \left[ \left(\frac {\pi}{2} - \psi - \theta\right) - \frac {1}{2} \sin (2 \psi) \right] (w _ {1} - 1) \quad \text {a n d} \quad R _ {2} = \left[ \frac {1}{2} - \frac {1}{2} \cos (2 \psi) - \frac {w _ {1}}{w _ {1} ^ {2} + w _ {2} ^ {2}} \right] | w _ {2} |.
$$
According to Lemmas 15 and 16, Eq. (13) can be bounded by
$$
\langle \nabla_ {\boldsymbol {w}} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi), \boldsymbol {w} - \boldsymbol {v} \rangle \geqslant \frac {1}{2 \pi} \left(\frac {1}{2} - \frac {1}{\pi}\right) [ \sin (2 \psi) + 2 \psi ] \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2} \geqslant \frac {1}{1 2 \pi} [ \sin (2 \psi) + 2 \psi ] \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2}. \tag {14}
$$
Secondly, we provide an upper bound of gradient in  $D_2$ . For any  $(\boldsymbol{w}, \psi) \in D_2$ , the gradient satisfies
$$
4 \pi^ {2} \| \nabla_ {\boldsymbol {w}} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) \| ^ {2} = T _ {1} + T _ {2},
$$
$$
T _ {1} = \left(\left[ \sin (2 \psi) + 2 \psi \right] w _ {1} - \frac {1}{2} \sin (2 \psi) - \left(\frac {\pi}{2} + \psi - \theta\right) - \frac {w _ {1} \left| w _ {2} \right|}{w _ {1} ^ {2} + w _ {2} ^ {2}}\right) ^ {2},
$$
$$
T _ {2} = \left(\left[ \frac {1}{2} \cos (2 \psi) - \frac {1}{2} + \frac {w _ {1} ^ {2}}{w _ {1} ^ {2} + w _ {2} ^ {2}} \right] \mathrm {s g n} (w _ {2}) + [ \sin (2 \psi) + 2 \psi ] w _ {2}\right) ^ {2}.
$$
From Lemmas 17 and 18, one knows
$$
\left\| \nabla_ {\boldsymbol {w}} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) \right\| ^ {2} \leqslant \left[ \sin (2 \psi) + 2 \psi \right] \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2}. \tag {15}
$$
Finally, based on Eqs. (14) and (15) and Lemma 13, we conclude
$$
\| \boldsymbol {w} ^ {\prime} - \boldsymbol {v} \| \leqslant \sqrt {1 - \left(\frac {1}{6 \pi} - \eta\right) \eta [ \sin (2 \psi) + 2 \psi ]} \| \boldsymbol {w} - \boldsymbol {v} \| \leqslant \left(1 - \frac {\eta}{2 4 \pi} [ \sin (2 \psi) + 2 \psi ]\right) \| \boldsymbol {w} - \boldsymbol {v} \|,
$$
where the first inequality holds based on  $\sqrt{1 - x} \leqslant 1 - x / 2$  for any  $x \in [0,1]$  and  $\eta \in (0,1 / (12\pi))$ . Thus, we have completed the proof.
The following two lemmas depict the gradient with respect to  $\psi$  in  $D_{1}$  and  $D_{2}$ , respectively.
Lemma 11. Let  $\psi' = \psi - \eta \nabla_{\psi} L_{\mathrm{cr}}(\pmb{w}, \psi)$ . If  $(\pmb{w}, \psi) \in D_1$ , then
$$
- \frac {1}{\pi} \left(\frac {\pi}{2} - \psi\right) ^ {2} \leqslant \nabla_ {\psi} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) \leqslant - \frac {1 - R ^ {2}}{4 \pi} \left(\frac {\pi}{2} - \psi\right) ^ {2}.
$$
Proof. For any  $(\boldsymbol{w}, \psi) \in D_1$ , one has
$$
\nabla_ {\psi} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) = - \frac {1}{2 \pi} [ \cos (2 \psi) + 1 ] (1 - \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2}).
$$
For any  $\psi \in [0, \pi/2]$ , we have  $\frac{1}{2} (\pi/2 - \psi)^2 \leqslant \cos(2\psi) + 1 \leqslant 2(\pi/2 - \psi)^2$ . Meanwhile, one has  $0 \leqslant \| \pmb{w}_t - \pmb{v} \| \leqslant R$ . Thus, the gradient with respect to  $\psi$  can be bounded by
$$
- \frac {1}{\pi} \left(\frac {\pi}{2} - \psi\right) ^ {2} \leqslant \nabla_ {\psi} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) \leqslant - \frac {1 - R ^ {2}}{4 \pi} \left(\frac {\pi}{2} - \psi\right) ^ {2},
$$
which completes the proof of the lower bound.
Lemma 12. If  $(\boldsymbol{w}, \psi) \in D_2$ , then
$$
- 2 \left(\frac {\pi}{2} - \psi\right) ^ {2} - 2 \left(\frac {\pi}{2} - \psi\right) | w _ {2} | \leqslant \nabla_ {\psi} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) \leqslant - \frac {1 - R ^ {2}}{2} \left(\frac {\pi}{2} - \psi\right) ^ {2}.
$$
Proof. The gradient of  $L_{\mathrm{cr}}$  with respect to  $\psi$  in  $D_2$  can be calculated by