SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $c_{1} > 0$ , and  $c_{2} \geqslant \max \{2r_{2} / g_{l}, 2c_{1}\}$ . Suppose that the sequence  $\{x_{t}\}_{t=1}^{\infty}$  generated by gradient descent  $x_{t+1} = x_{t} - \eta_{t}\nabla f(x_{t})$  with  $x_{0} \in K$  and  $\eta_{t} = \min \{c_{1}, c_{2} / t\}$  satisfies  $x_{t} \in K$  for any  $t \in \mathbb{N}^{+}$ . Then the following holds for any  $t \in \mathbb{N}^{+}$
$$
\left| x _ {t} - x ^ {*} \right| \leqslant \frac {b}{t} \quad \text {w i t h} \quad b = \max  \left\{2 a + c _ {2} g _ {u}, \frac {c _ {2} ^ {2} g _ {l}}{2 c _ {1}} \right\}.
$$
Proof. Let  $t_0 = 2b / (c_2g_l) \geqslant c_2 / c_1 \geqslant 2$ . For any  $0 < t < t_0$ , it is observed that
$$
\left| x _ {t} - x ^ {*} \right| \leqslant r _ {2} \leqslant \frac {c _ {2} g _ {l}}{2} = \frac {b}{t _ {0}} \leqslant \frac {b}{t}.
$$
Thus, the conclusion holds for  $0 < t < t_0$ . Suppose that  $|x_k - x^*| \leqslant b / k$  holds for  $k \geqslant t_0 - 1$ . We then prove  $|x_{k + 1} - x^{*}| \leqslant b / (k + 1)$  by discussion.
1. If the first condition holds and  $\operatorname{sgn}(x_{k+1} - x^*) = \operatorname{sgn}(x_k - x^*)$ , then we have
$$
\left| x _ {k + 1} - x ^ {*} \right| \leqslant \left| x _ {k} - x ^ {*} \right| - \eta_ {k + 1} g _ {l} \leqslant \frac {b}{k} - \frac {c _ {2} g _ {l}}{k + 1} \leqslant \frac {b}{k + 1},
$$
where the second inequality holds from the induction hypothesis, and the third inequality holds based on  $b = c_2 g_l t_0 / 2$  and  $t_0 / 2 \leqslant t_0 - 1 \leqslant k$ . Thus, the conclusion holds for  $t = k + 1$ .
2. If the first condition holds and  $\operatorname{sgn}(x_{k+1} - x^*) \neq \operatorname{sgn}(x_k - x^*)$ , then we have
$$
\left| x _ {k + 1} - x ^ {*} \right| \leqslant \eta_ {k + 1} g _ {u} \leqslant \frac {c _ {2} g _ {u}}{k + 1} \leqslant \frac {b}{k + 1},
$$
which implies that the conclusion holds for  $t = k + 1$ .
3. If the second condition holds, then one knows
$$
\left| x _ {k + 1} - x ^ {*} \right| \leqslant \left| x _ {k} - x ^ {*} \right| + \eta_ {k + 1} g _ {u} \leqslant \frac {a}{k} + \frac {c _ {2} g _ {u}}{k + 1} \leqslant \frac {b}{k + 1},
$$
where the second inequality holds based on  $|x_{k + 1} - x^{*}| \leqslant \theta_{k + 1} \leqslant a / (k + 1)$ , and the third inequality holds because of  $b \geqslant 2a + c_2g_u$ . Thus, the conclusion holds for  $t = k + 1$ .
Lemma 28. Let  $f: K \to \mathbb{R}$  represent a function with a global minimum  $x^{*}$ , where  $K \subset \mathbb{R}$  indicates the convex domain satisfying  $K \subset B(x^{*}, R)$ . Let  $\{x_{t}\}_{t=1}^{\infty}$  denote the sequence generated by gradient descent  $x_{t+1} = x_{t} - \eta_{t} \nabla f(x_{t})$  with  $x_{0} \in K$  and  $\eta_{t} = \min\{c_{1}, c_{2}/t\}$ , satisfying  $x_{t} \in K$  for  $t \in \mathbb{N}^{+}$ . Suppose that the gradient satisfies  $\nabla f(x_{t}) = d(x_{t} - x^{*}) + r_{t}$ , where  $d_{l} \leqslant d \leqslant d_{u}$  and  $|r_{t}| \leqslant e/t$ . If  $c_{1} \leqslant 1/d_{u}$  and  $c_{2} \geqslant 2/d_{l}$ , then we have
$$
\left| x _ {t} - x ^ {*} \right| \leqslant \frac {c}{t} \quad \text {w i t h} \quad c = \max  \left\{\frac {c _ {2} R}{c _ {1}}, c _ {2} e \right\}.
$$
Proof. Let  $t_0 = c_2 / c_1$ . We prove the conclusion by mathematical induction.
1. Base case. For  $0 < t \leqslant t_0$ , it is observed that
$$
\left| x _ {t} - x ^ {*} \right| \leqslant R \leqslant \frac {c}{t _ {0}} \leqslant \frac {c}{t}.
$$
Thus, the conclusion holds for  $0 < t \leqslant t_0$ .
2. Induction. Suppose that  $|x_{k} - x^{*}| \leqslant c / k$  holds for  $k \geqslant t_0 - 1$ . Then we have
$$
\left| x _ {k + 1} - x ^ {*} \right| = \left| \left(1 - d \eta_ {k}\right) \left(x _ {k} - x ^ {*}\right) - \eta_ {k} r _ {k} \right| \leqslant \left(1 - d \eta_ {k}\right) \left| x _ {k} - x ^ {*} \right| + \eta_ {k} \left| r _ {k} \right|,
$$
where the first inequality holds based on  $d\eta_{k}\leqslant c_{1}d_{u}\leqslant 1$  . Then the induction hypothesis leads to
$$
\left| x _ {k + 1} - x ^ {*} \right| \leqslant \left(1 - \frac {2}{k}\right) \frac {c}{k} + \frac {c _ {2} e}{k ^ {2}} \leqslant \frac {c}{k + 1},
$$
where the first inequality holds according to  $c_{2}d_{l}\geqslant 2$ , and the second inequality holds based on  $c\geqslant c_2e$ . Thus, the conclusion holds for  $t = k + 1$
We begin the proof with two lemmas. For any non-zero vector  $\mathbf{a}$  in  $\mathbb{R}^2$  and  $\theta \in [0,\pi ]$ , define  $S(\pmb {a},\theta) = \{\pmb {x}\in \mathbb{R}^2\mid \theta_{\pmb{x}}\in [\theta_{\pmb{a}} - \theta ,\theta_{\pmb{a}} + \theta ]\}$  as the sector region with central angle  $2\theta$  that is symmetric with respect to  $\pmb{a}$ . Let  $\mathcal{N}_{\pmb{a},\theta}$  represent the truncated standard Gaussian distribution on  $S(\pmb {a},\theta)$ , of which the probability density function is
$$
p (\boldsymbol {x}) = \left\{ \begin{array}{l l} \frac {1}{2 \theta} \mathrm {e} ^ {- \frac {1}{2} \| \boldsymbol {x} \| ^ {2}}, & \boldsymbol {x} \in S (\boldsymbol {a}, \theta), \\ 0, & \text {o t h e r w i s e}. \end{array} \right.
$$
The following lemma provides a lower bound for the expected squared inner product on  $S(\pmb{a}, \theta)$ .
Lemma 29. Let  $d = 1$ . For any  $\pmb{w} \in \mathbb{R}^{2d}$ , non-zero  $\pmb{a} \in \mathbb{R}^{2d}$ , and  $\theta \in [0, \pi/2]$ , we have
$$
\mathbb {E} _ {\boldsymbol {x} \sim \mathcal {N} _ {\boldsymbol {a}, \theta}} \left[ \left(\boldsymbol {w} ^ {\top} \boldsymbol {x}\right) ^ {2} \right] \geqslant \frac {\theta^ {2}}{3} \| \boldsymbol {w} \| ^ {2}.
$$
Proof. Let  $\theta_{\mathbf{w}}$  indicate the phase of  $\mathbf{w}$ , i.e.,  $\mathbf{w} = \| \mathbf{w}\| (\sin \theta_{\mathbf{w}} + \cos \theta_{\mathbf{w}}\mathrm{i})$ . Then calculating the expectation in the polar coordinate system leads to
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {N} _ {\boldsymbol {a}, \theta}} \left[ \left(\boldsymbol {w} ^ {\top} \boldsymbol {x}\right) ^ {2} \right] = \frac {\| \boldsymbol {w} \| ^ {2}}{2 \theta} \int_ {0} ^ {+ \infty} \int_ {\theta_ {\boldsymbol {a}} - \theta} ^ {\theta_ {\boldsymbol {a}} + \theta} r ^ {3} \left(\cos \theta_ {\boldsymbol {w}} \cos \phi + \sin \theta_ {\boldsymbol {w}} \sin \phi\right) ^ {2} e ^ {- \frac {1}{2} r ^ {2}} d \phi d r \tag {38} \\ = \frac {\| \pmb {w} \| ^ {2}}{\theta} \left[ \theta + \frac {1}{2} \sin (2 \theta) \cos (2 \theta_ {\pmb {a} , \pmb {w}}) \right], \\ \end{array}
$$
where the second equality holds based on integrating over  $r$  and  $\phi$  separately, and the identity  $\cos (\theta_{\mathbf{a}} - \theta_{\mathbf{w}}) = \cos \theta_{\mathbf{a},\mathbf{w}}$ . The expectation in Eq. (38) can be further bounded by
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {N} _ {\boldsymbol {a}, \theta}} \left[ \left(\boldsymbol {w} ^ {\top} \boldsymbol {x}\right) ^ {2} \right] = \| \boldsymbol {w} \| ^ {2} \left[ \left(1 - \frac {1}{2 \theta} \sin (2 \theta)\right) + \frac {1}{\theta} \sin (2 \theta) \cos^ {2} \theta_ {\boldsymbol {a}, \boldsymbol {w}} \right] \\ \geqslant \left(1 - \frac {1}{2 \theta} \sin (2 \theta)\right) \| \boldsymbol {w} \| ^ {2} \\ \geqslant \frac {\theta^ {2}}{3} \| \boldsymbol {w} \| ^ {2}, \\ \end{array}
$$
where the first inequality holds according to  $\theta \in [0,\pi /2]$ , and the second inequality holds because of  $\sin (x)\leqslant x - x^3 /12$  for all  $\theta \in [0,\pi /2]$ . Thus, we have completed the proof.
The following lemma provides a lower bound for expressing a complex-valued vector with four real-valued vectors under a symmetric constant.
Lemma 30. Let  $\mathbf{v}_k\in \mathbb{R}^d$  with  $k\in [4]$  and  $\pmb {v}\in \mathbb{R}^d$  . If  $\pmb {v}_1 + \pmb {v}_3 = \pmb {v}_2 + \pmb {v}_4$  , then we have
$$
\sum_ {k = 1} ^ {4} \left\| \boldsymbol {v} _ {i} - \boldsymbol {v} \cdot \mathbb {I} (k = 1) \right\| ^ {2} \geqslant \frac {1}{4} \| \boldsymbol {v} \| ^ {2}.
$$
Proof. According to the generalized mean inequality, one knows
$$
\sum_ {k = 1} ^ {4} \| \boldsymbol {v} _ {i} - \boldsymbol {v} \cdot \mathbb {I} (k = 1) \| ^ {2} \geqslant \frac {1}{4} \left(\sum_ {k = 1} ^ {4} \| \boldsymbol {v} _ {i} - \boldsymbol {v} \cdot \mathbb {I} (k = 1) \|\right) ^ {2} \geqslant \frac {1}{4} \| (\boldsymbol {v} _ {1} - \boldsymbol {v}) - \boldsymbol {v} _ {2} + \boldsymbol {v} _ {3} - \boldsymbol {v} _ {4} \| ^ {2} = \frac {1}{4} \| \boldsymbol {v} \| ^ {2},
$$
where the second inequality holds because of the triangle inequality, and the first equality holds based on the condition  $\pmb{v}_1 + \pmb{v}_3 = \pmb{v}_2 + \pmb{v}_4$ . Thus, we have completed the proof.
We are now ready to prove Theorem 4.
Proof of Theorem 4. We define  $\mathcal{N}_{\alpha, \mathbf{W}} = \sum_{i=1}^{n} \alpha_i \tau(\boldsymbol{w}_i^\top \boldsymbol{x})$  for simplicity. From  $d = 1$ , the weight vector  $\boldsymbol{w}_i$  is a 2-dimensional real-valued vector. Let  $\theta_{\boldsymbol{w}_i} = \arctan(w_{i,1}^{-1} w_{i,2}) \in (-\psi, 2\pi - \psi]$  denote the phase of  $\boldsymbol{w}_i$ . We assume  $\theta_\boldsymbol{v} = 0$  without loss of generality. Denote by  $\Theta_{\mathbf{W}}$  the  $\pi/2$ -symmetrical phase set induced from  $\mathbf{W}$  and  $\psi$ , i.e.,
$$
\Theta_ {\mathbf {W}} = \left\{\theta_ {\boldsymbol {w} _ {i}} + \frac {(j - 1) \pi}{2} \mid i \in [ n ], j \in [ 4 ] \right\} \cup \left\{i \psi + \frac {(j - 1) \pi}{2} \mid i \in \{- 1, + 1 \}, j \in [ 4 ] \right\}.
$$
It is observed that there is an integer  $m \leqslant n + 2$  such that  $|\Theta_{\mathbf{W}}| = 4m$ . We sort all phases in  $\Theta_{\mathbf{W}}$  as
$$
\Theta_ {\mathbf {W}} = \left\{\theta_ {i} \right\} _ {i = 1} ^ {4 m} \quad \text {w i t h} \quad - \psi <   \theta_ {1} <   \dots <   \theta_ {4 m} = 2 \pi - \psi .
$$
Let  $\mathcal{N}_{\beta, \mathbf{U}}$  represent an arbitrary two-layer RVNN with weight phases from  $\Theta_{\mathbf{W}}$ , i.e.,
$$
\mathcal {N} _ {\boldsymbol {\beta}, \mathbf {U}} (\boldsymbol {x}) = \sum_ {i = 1} ^ {4 m} \beta_ {i} \tau \left(\boldsymbol {u} _ {i} ^ {\top} \boldsymbol {x}\right) \quad \text {w i t h} \quad \theta_ {\boldsymbol {u} _ {i}} = \theta_ {i}.
$$
It is observed that  $\mathcal{N}_{\beta, \mathbf{U}}$  degenerates to  $\mathcal{N}_{\alpha, \mathbf{W}}$  with suitable parameters. Thus, the expected square loss  $L_{\mathrm{rc}}$  can be bounded as
$$
\begin{array}{l} L _ {\mathrm {r c}} (\boldsymbol {\alpha}, \mathbf {W}) \geqslant \frac {1}{2} \inf  _ {\boldsymbol {\beta}, \mathbf {U}} \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} \left[ \left(\mathcal {N} _ {\boldsymbol {\beta}, \mathbf {U}} (\boldsymbol {x}) - \sigma_ {\psi} \left(\boldsymbol {v} _ {\mathbb {C}} ^ {\top} \overline {{\boldsymbol {x}}} _ {\mathbb {C}}\right)\right) ^ {2} \right] \\ = \frac {1}{2} \inf  _ {\boldsymbol {\beta}, \mathbf {U}} \sum_ {i = 1} ^ {4 m} \frac {\Delta \theta_ {i}}{\pi} \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {N} \left(\boldsymbol {a} _ {i}, \Delta \theta_ {i}\right)} \left[ \left(\mathcal {N} _ {\boldsymbol {\beta}, \mathbf {U}} (\boldsymbol {x}) - \sigma_ {\psi} \left(\boldsymbol {v} _ {\mathbb {C}} ^ {\top} \overline {{\boldsymbol {x}}} _ {\mathbb {C}}\right)\right) ^ {2} \right], \tag {39} \\ \end{array}
$$
where  $\Delta \theta_{i} = (\theta_{i} - \theta_{i - 1}) / 2$  and  $\pmb{a}_{i} = \mathrm{e}^{(\theta_{i} - \Delta \theta_{i})\mathrm{i}}$  with  $\theta_0 = \theta_{4(n + 1)}$ . The indices can be divided into  $m$  groups as  $\mathcal{I}_i = \{i + (k - 1)m\mid k\in [4]\}$  with  $i\in [m]$ . Denote by  $i_{\psi}$  the index of  $\psi$ , i.e.,  $\theta_{i_{\psi}} = \psi$ . Then Eq. (39) becomes
$$
\begin{array}{l} L _ {\mathrm {r c}} (\boldsymbol {\alpha}, \mathbf {W}) \geqslant \frac {1}{2} \inf  _ {\boldsymbol {\beta}, \mathbf {U}} \sum_ {i = 1} ^ {m} \frac {\Delta \theta_ {i}}{\pi} \sum_ {j \in \mathcal {I} _ {i}} \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {N} (\boldsymbol {a} _ {j}, \Delta \theta_ {j})} \left[ \left(\mathcal {N} _ {\boldsymbol {\beta}, \mathbf {U}} (\boldsymbol {x}) - \sigma_ {\psi} (\boldsymbol {v} _ {\mathbb {C}} ^ {\top} \overline {{\boldsymbol {x}}} _ {\mathbb {C}})\right) ^ {2} \right] \\ = \frac {1}{2} \inf  _ {\boldsymbol {\beta}, \mathbf {U}} \sum_ {i = 1} ^ {m} \frac {\Delta \theta_ {i}}{\pi} \sum_ {j \in \mathcal {I} _ {i}} \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {N} (\boldsymbol {a} _ {j}, \Delta \theta_ {j})} \left[ \left(\left(\boldsymbol {v} _ {j} - \boldsymbol {v} \cdot \mathbb {I} (j \leqslant i _ {\psi})\right) ^ {\top} \boldsymbol {x}\right) ^ {2} \right], \\ \end{array}
$$
where the first inequality holds since  $\Delta \theta_{j}$  remains the same in  $\mathcal{I}_i$ , the second inequality holds based on the activation regions of ReLU and zReLU, and the definition of  $\pmb{v}_{j}$  as follows
$$
\boldsymbol {v} _ {j} = \sum_ {l = j - m} ^ {j + m - 1} \beta_ {\phi (l)} \boldsymbol {u} _ {\phi (l)} \quad \text {w i t h} \quad \phi (l) = \left\{ \begin{array}{l l} l + 4 m, & l \leqslant 0, \\ l, & 0 <   l \leqslant 4 m, \\ l - 4 m, & l > 4 m. \end{array} \right. \tag {41}
$$
Applying Lemma 29 to Eq. (40), we obtain
$$
\begin{array}{l} L _ {\mathrm {r c}} (\boldsymbol {\alpha}, \mathbf {W}) \geqslant \frac {1}{2} \inf  _ {\boldsymbol {\beta}, \mathbf {U}} \sum_ {i = 1} ^ {m} \frac {\Delta \theta_ {i}}{\pi} \sum_ {j \in \mathcal {I} _ {i}} \frac {(\Delta \theta_ {j}) ^ {2}}{3} \| \boldsymbol {v} _ {j} - \boldsymbol {v} \cdot \mathbb {I} (j \leqslant i _ {\psi}) \| ^ {2} \\ \geqslant \frac {1}{2} \inf _ {\boldsymbol {\beta}, \mathbf {U}} \sum_ {i = \max \{1, i _ {\psi} - m + 1 \}} ^ {\min \{i _ {\psi}, m \}} \frac {(\Delta \theta_ {i}) ^ {3}}{3 \pi} \sum_ {k = 1} ^ {4} \| \pmb {v} _ {i, k} - \pmb {v} \cdot \mathbb {I} (k = 1) \| ^ {2}, \\ \end{array}
$$
where the second inequality holds based on the definition of  $\pmb{v}_{i,k} = \pmb{v}_{i + (k - 1)(n + 1)}$  and  $\Delta \theta_{j} = \Delta \theta_{i}$  for any  $j \in \mathcal{I}_i$ . Based on Eq. (41), one has  $\pmb{v}_{i,1} + \pmb{v}_{i,3} = \pmb{v}_{i,2} + \pmb{v}_{i,4}$ . Then Lemma 30 implies
$$
\begin{array}{l} L _ {\mathrm {r c}} (\boldsymbol {\alpha}, \mathbf {W}) \geqslant \frac {1}{2} \inf  _ {\boldsymbol {\beta}, \mathbf {U}} \sum_ {i = \max  \{1, i _ {\psi} - m + 1 \}} ^ {\min  \left\{i _ {\psi}, m \right\}} \frac {\left(\Delta \theta_ {i}\right) ^ {3}}{3 \pi} \cdot \frac {1}{4} \| \boldsymbol {v} \| ^ {2} \\ \geqslant \frac {\| \boldsymbol {v} \| ^ {2}}{2 4 \pi (\max  \{1 , i _ {\psi} - m + 1 \} - \min  \{i _ {\psi} , m \}) ^ {2}} \left(\sum_ {i = \max  \{1, i _ {\psi} - m + 1 \}} ^ {\min  \{i _ {\psi}, m \}} \Delta \theta_ {i}\right) ^ {3} \\ \geqslant \frac {\| \pmb {v} \| ^ {2} \min \{2 \psi , \pi - 2 \psi \} ^ {3}}{2 4 \pi (n + 2) ^ {2}}, \\ \end{array}
$$
where the second inequality holds based on the generalized mean inequality, and the third one holds from  $\max \{1,i_{\psi} - m + 1\} -\min \{i_{\psi},m\} \leqslant m\leqslant n + 2$ . Thus, we have completed the proof.
E Proof of Theorem 6
We begin with a lemma providing a lower bound for convergence.
Lemma 31. If there exists a constant  $c$  such that
$$
\langle \nabla f (\boldsymbol {w}), \boldsymbol {w} - \boldsymbol {v} \rangle \leqslant c \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2},
$$
then  $\pmb{w}^{\prime} = \pmb{w} - \eta \nabla f(\pmb{w})$  with  $\eta \in (0,1 / (2c))$  satisfies
$$
\left\| \boldsymbol {w} ^ {\prime} - \boldsymbol {v} \right\| \geqslant \sqrt {1 - 2 c \eta} \left\| \boldsymbol {w} - \boldsymbol {v} \right\|.
$$
Proof. From the updating rule, it is observed that
$$
\left\| \boldsymbol {w} ^ {\prime} - \boldsymbol {v} \right\| ^ {2} \geqslant \left\| \boldsymbol {w} - \boldsymbol {v} \right\| ^ {2} - 2 \eta \langle \boldsymbol {w} - \boldsymbol {v}, \nabla f (\boldsymbol {w}) \rangle \geqslant (1 - 2 c \eta) \left\| \boldsymbol {w} - \boldsymbol {v} \right\| ^ {2},
$$
We then prove Theorem 6.
Proof of Theorem 6. Denote by  $R = \| \pmb{w}_0 - \pmb{v} \|$ . The convergence analysis consists of several stages.
Stage 1: the error of  $\psi$  decreases below a threshold fast. By the same arguments as those in the proof of Theorem 1,  $\eta \in (0,1 / (12\pi))$  indicates  $(\boldsymbol{w}_t,\psi_t)\in D$  for any  $t\in \mathbb{N}$ . Recalling the convergence of  $\psi$  in Eq. (7), we have  $\psi_t\geqslant \pi /4$  when  $t\geqslant \lceil 16\eta^{-1}(1 - R^2)^{-1}\rceil$ . From Eq. (4), one knows  $\nabla_{\psi}L_{\mathrm{cr}}(\boldsymbol{w}_t,\psi_t)\geqslant -6(\psi^* -\psi_t)$ . Then we have
$$
\langle \nabla_ {\psi} L _ {\mathrm {c r}} (\boldsymbol {w} _ {t}, \psi_ {t}), \psi^ {*} - \psi_ {t} \rangle \geqslant - 6 (\psi^ {*} - \psi_ {t}) ^ {2}.
$$
Then we obtain from  $\eta \in (0,1 / 12)$  and Lemma 31 that
$$
\psi^ {*} - \psi_ {t} \geqslant (1 - 1 2 \eta) ^ {t / 2} \left(\psi^ {*} - \psi_ {0}\right). \tag {42}
$$
Thus, one has
$$
(1 - 1 2 \eta) ^ {t / 2} (\psi^ {*} - \psi_ {0}) \leqslant \psi^ {*} - \psi_ {t} \leqslant \frac {\pi}{4} \quad \text {w i t h} \quad t \geqslant T _ {1} = 1 6 \eta^ {- 1} (1 - R ^ {2}) ^ {- 1}.
$$
Step 2: both errors of  $w$  and  $\psi$  decrease below small constants fast. Based on Eq. (8), we have
$$
\left\| \boldsymbol {w} _ {t} - \boldsymbol {v} \right\| \leqslant \left(1 - \frac {\eta}{4 8}\right) ^ {t - T _ {1}} \quad \text {f o r} \quad t \geqslant T _ {1}, \tag {43}
$$
which, together with Eqs. (7) and (42), implies that
$$
(1 - 1 2 \eta) ^ {t / 2} \left(\psi^ {*} - \psi_ {0}\right) \leqslant \psi^ {*} - \psi_ {t} \leqslant \frac {1}{3 8 4} \quad \text {a n d} \quad | w _ {2} | \leqslant \| \boldsymbol {w} _ {t} - \boldsymbol {v} \| \leqslant \frac {1}{3 8 4}, \tag {44}
$$
with  $t \geqslant T_2 = \max \left\{T_1 + \frac{\ln 384}{\ln(1 + \eta / 48)}, \frac{3200\pi}{\eta(1 - R^2)}\right\}$ .
Step 3:  $w$  converges faster than  $\psi$ . For any  $t \geqslant T_2$ , Lemmas 11 and 12 imply
$$
\langle \nabla_ {\psi} L _ {\mathrm {c r}} (\pmb {w} _ {t}, \psi_ {t}), \psi_ {t} - \psi^ {*} \rangle \leqslant 2 (\psi^ {*} - \psi_ {t}) ^ {3} + 2 (\psi^ {*} - \psi_ {t}) ^ {2} | w _ {2, t} | \leqslant \frac {1}{9 6} (\psi^ {*} - \psi_ {t}) ^ {2},
$$
where the second inequality holds based on Eq. (44). Then Lemma 31 indicates
$$
\psi^ {*} - \psi_ {t + 1} \geqslant \sqrt {1 - \eta / 4 8} (\psi^ {*} - \psi_ {t}) \quad \text {f o r} \quad t \geqslant T _ {2},
$$
which, together with Eq. (43), indicates
$$
\left| w _ {w, t} \right| \leqslant \left\| \boldsymbol {w} _ {t} - \boldsymbol {v} \right\| \leqslant \psi^ {*} - \psi_ {t} \quad \text {w i t h} \quad t \geqslant T _ {3} = 2 T _ {1} + \frac {T _ {2} \ln (1 - 1 2 \eta) + 2 \ln \left(\psi^ {*} - \psi_ {0}\right)}{\ln (1 - \eta / 4 8)}. \tag {45}
$$
Step 4:  $\psi$  converges with an inversely proportional rate. For any  $t \geqslant T_3$ , it is observed from Lemmas 11, 12, and Eq. (45) that
$$
\nabla_ {\psi} L _ {\mathrm {c r}} \left(\boldsymbol {w} _ {t}, \psi_ {t}\right) \geqslant - 4 \left(\psi^ {*} - \psi\right) ^ {2}.
$$
Let  $a_{t} = 4\eta (\psi^{*} - \psi_{t})$ . Then the updating rule implies  $a_{t + 1}\geqslant a_t(1 - a_t)$ . Choosing  $\eta \in (0,1 / (4\pi))$  guarantees  $a_{t}\in [0,1 / 2]$ . Then Lemma 14 indicates
$$
\psi^ {*} - \psi_ {t} \geqslant \frac {\left(1 - 1 2 \eta\right) ^ {T _ {3} / 2} \left(\psi^ {*} - \psi_ {0}\right)}{t - T _ {3} + 1} \quad \text {f o r} \quad t \geqslant T _ {3}. \tag {46}
$$
Step 5: the loss converges to 0 with an inversely proportional rate. Define non-negative quantities  $\Delta_{\boldsymbol{w}} = \| \boldsymbol{w} - \boldsymbol{v}\|$  and  $\Delta_{\psi} = \psi^{*} - \psi$ . We provide a lower bound for  $L_{\mathrm{cr}}$  by discussion.
$$
L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) \geqslant \frac {1}{4} - \frac {1}{8 \pi} (4 \psi^ {*} - \Delta_ {\psi} ^ {3}) (1 - \Delta_ {\boldsymbol {w}} ^ {2}) = \frac {1}{8 \pi} \Delta_ {\psi} ^ {3} + \frac {1}{8 \pi} \Delta_ {\boldsymbol {w}} ^ {2} (2 \pi - \Delta_ {\psi} ^ {3}) \geqslant \frac {1}{8 \pi} \Delta_ {\psi} ^ {3}, \tag {47}
$$
where the first inequality holds based on  $\sin (2\psi) + 2\psi = \sin (2\Delta_{\psi}) + 2\psi^{*} - 2\Delta_{\psi}\leqslant 2\psi^{*} - \Delta_{\psi}^{3} / 2$  for any  $\psi \in [0,\pi /2]$ , and the second inequality holds from  $\Delta_{\psi}\leqslant \pi /2$
$$
\begin{array}{l} L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) = \frac {1}{4} - \frac {1}{4 \pi} [ \sin (2 \psi) + 2 \psi ] (1 - \Delta_ {\boldsymbol {w}} ^ {2}) \\ + \frac {1}{4 \pi} [ (\cos (2 \psi) - 1) | w _ {2} | + (\sin (2 \psi) + 2 \psi + 2 \theta - 2 \psi^ {*}) w _ {1} ] \\ \geqslant \frac {1}{4} - \frac {1}{8 \pi} \left(4 \psi^ {*} - \Delta_ {\psi} ^ {3}\right) \left(1 - \Delta_ {\boldsymbol {w}} ^ {2}\right) + \frac {1}{4 \pi} \left[ \left(\cos (2 \psi) - 1\right) | w _ {2} | \right] \tag {48} \\ \geqslant \frac {1}{4} - \frac {1}{8 \pi} (4 \psi^ {*} - \Delta_ {\psi} ^ {3}) (1 - \Delta_ {\boldsymbol {w}} ^ {2}) - \frac {1}{2 \pi} \Delta_ {\boldsymbol {w}} \\ \geqslant \frac {1}{8 \pi} \Delta_ {\psi} ^ {3} - \frac {1}{2 \pi} \Delta_ {\boldsymbol {w}}, \\ \end{array}
$$
where the first inequality holds from  $\sin (2\psi) + 2\psi \leqslant 2\psi^{*} - \Delta_{\psi}^{3} / 2$  and  $\sin (2\psi) + 2\psi +2\theta -2\psi^{*}\geqslant 0$ , the second inequality holds based on  $\cos (2\psi) - 1\geqslant -2$  and  $|w_{2}|\leqslant \Delta_{\pmb{w}}$
Combining Eqs. (47) and (48), one knows that the following holds for any  $(\boldsymbol{w}_0, \psi_0) \in D$  and  $t \geqslant T_3$
$$
L _ {\mathrm {c r}} (\boldsymbol {w} _ {t}, \psi_ {t}) \geqslant \frac {1}{8 \pi} \Delta_ {\psi , t} ^ {3} - \frac {1}{2 \pi} \Delta_ {\boldsymbol {w}, t} \geqslant \frac {(1 - 1 2 \eta) ^ {3 T _ {3} / 2} (\psi^ {*} - \psi_ {0}) ^ {3}}{8 \pi (t - T _ {3} + 1) ^ {3}} - \frac {1}{2 \pi} \left(1 - \frac {\eta}{4 8}\right) ^ {t - T _ {3}},
$$
where the second inequality holds from Eqs. (43) and (46). Thus, we have completed the proof.
F Simulation Experiments
Experimental settings. A training set of size 7,000 and a test set of size 3,000 are generated by a randomly initialized target neuron (can be a real-valued or a complex-valued neuron). After random initialization, a complex-valued neuron and a real-valued neuron are trained by gradient descent with
the empirical mean square loss and a learning rate of 0.1 for 100 epochs (or 300 epochs when the loss does not converge).
Experimental results. It should be noticed that a complex-valued neuron cannot always learn a target neuron. From the theoretical formulation, our convergence rate holds with a small constant probability. From the loss landscape, there exist constant pieces in the parameter space, i.e., the complex-valued neuron does not learn anything after initialization. Thus, we cannot expect a complex-valued neuron to learn a target neuron all the time. In the experiments, we train the complex-valued neuron with several random initializations and find that our theoretical conclusions occur in experiments. This phenomenon verifies our theories and also motivates a novel learning algorithm for CVNNs, as discussed in the conclusion part.