SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We summarize the proof idea of Theorem 2 as follows. The overall procedure is similar to that of Theorem 1 but every step is different and more challenging because of non-smoothness and more spurious stationary points. Firstly, we identify an ideal region with nice gradient properties: the gradient with respect to  $\boldsymbol{w}_{\perp}$ , the weight component perpendicular to  $\boldsymbol{v}$ , points to the global minimum  $\mathbf{0}$  and maintains constant order. The gradient  $\nabla_{\psi_w}$  is bounded and points towards  $\psi_v$  when the angle  $\theta_{\boldsymbol{w},\boldsymbol{v}}$  is small enough. Meanwhile, the gradient with respect to  $\boldsymbol{w}_v$  performs like a contraction mapping with fixed point  $[1 - \Theta (\psi_v\psi_w^{-1})]v$  and Lipschitz constant  $1 - \Theta (\psi)$ , i.e., there exists a deviation of the fixed point from the global minimum. Based on these observations, we then prove the convergence with three stages, as demonstrated in Fig 1(b): In Stage I,  $\boldsymbol{w}_{\perp}$ , the weight component perpendicular to  $\boldsymbol{v}$ , converges to 0 with an inversely proportional rate, and  $\psi_w$  and  $\boldsymbol{w}_v$  remain in the ideal region. Thus, the angle  $\theta_{\boldsymbol{w},\boldsymbol{v}}$  decreases with an inversely proportional rate. When  $\theta_{\boldsymbol{w},\boldsymbol{v}}$  declines below some threshold, we come to Stage II where phase  $\psi_w$  converges to  $\psi_v$  with rate  $O(t^{-1})$ . As  $\psi_w$  approaches  $\psi_v$ , the fixed point becomes close to  $v$  and we step into Stage III where  $w$  converges to  $v$  with the same rate as  $\psi_w$ . Finally, we estimate the order of loss and provide a lower bound of the probability of falling into the ideal region with Gaussian initialization to complete the proof. We provide detailed proofs in Appendix C.
4.3 Finite-Width RVNNs Cannot Learn a Single Non-degenerate Complex-valued Neuron
We then study learning one complex-valued neuron with zReLU activation using real-valued neurons. Since a complex-valued neuron has more parameters than a real-valued neuron, it is unfair to learn a complex-valued neuron with a single real-valued neuron. Thus, we consider the problem of learning a complex-valued neuron with a two-layer RVNN. A two-layer RVNN with  $n$  hidden neurons can be represented by  $\boldsymbol{x} \rightarrow \boldsymbol{\alpha}^{\top} \tau(\mathbf{W} \boldsymbol{x})$ , where  $\mathbf{W} \in \mathbb{R}^{n \times 2d}$  and  $\boldsymbol{\alpha} \in \mathbb{R}^{n}$  indicate weight parameters of the network, and  $\tau$  is the ReLU activation function applied componentwisely. We still focus on the expected square loss, which takes the form
$$
L _ {\mathrm {r c}} (\pmb {\alpha}, \mathbf {W}) = \frac {1}{2} \mathbb {E} _ {\pmb {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} \left[ \left(\pmb {\alpha} ^ {\top} \tau (\mathbf {W} \pmb {x}) - \sigma_ {\psi} (\pmb {v} _ {\mathbb {C}} ^ {\top} \overline {{\pmb {x}}} _ {\mathbb {C}})\right) ^ {2} \right],
$$
where we abbreviate the phase parameter  $\psi_v$  as  $\psi$  since RVNN has no phase parameter. We are mainly interested in learning a non-degenerate complex-valued neuron, which is distinct from a real-valued neuron and defined as follows.
Definition 3. A complex-valued neuron is a non-degenerate one if  $\psi \notin \{0, \pi/2\}$  and  $\pmb{v} \neq \mathbf{0}$ .
For a complex-valued neuron with phase  $\psi = 0$  or  $\pmb{v} = \mathbf{0}$ , the zReLU activation function always outputs 0. Then the complex-valued neuron is equivalent to a real-valued neuron with all zero weights. For a complex-valued with phase  $\psi = \pi /2$ , the zReLU activation function is equivalent to the ReLU activation function. Thus, a non-degenerate complex-valued neuron is a non-real-valued neuron. Then we present the third theorem for complex-valued neuron learning.
Theorem 4. Let  $d = 1$ . For any non-degenerate complex-valued neuron with phase  $\psi \in (0, \pi/2)$  and non-zero weight vector  $\mathbf{v} \in \mathbb{C}^d$ , denote by  $L_{\mathrm{rc}}$  the expected square loss of learning this complex-valued neuron using a one-hidden-layer RVNN with  $n$  hidden neurons. Then the loss satisfies
$$
L _ {\mathrm {r c}} (\boldsymbol {\alpha}, \mathbf {W}) \geqslant \frac {\| \boldsymbol {v} \| ^ {2} \min  \{2 \psi , \pi - 2 \psi \} ^ {3}}{2 4 \pi (n + 2) ^ {2}} > 0.
$$
Theorem 4 provides a positive lower bound for the expected squared loss of approximating a nondegenerate complex-valued neuron using a two-layer RVNN with a fixed number of hidden neurons. This positive lower bound indicates that there always remains a positive gap between the target non-degenerate complex-valued neuron and the two-layer RVNN of fixed width no matter how the parameters of the RVNN are learned. Thus, a finite-width RVNN cannot learn a single non-degenerate complex-valued neuron.
(a) The loss in one sector.
(b) The loss in four sectors.
Figure 2: An illustration of the proof idea of Theorem 4. Shaded areas represent sectors with infinite radii. (a) The expectation  $\mathbb{E}_{\boldsymbol{x} \in S}[(\boldsymbol{w}^\top \boldsymbol{x})^2]$  in the sector  $S$  equals the sum of the expectations on three subareas  $A_1, A_2$ , and  $A_3$ . The minimum expectation on a subarea can be bounded by  $\Omega(\theta^2 \| \boldsymbol{w} \|^2)$ . (b) The expected loss of learning a complex-valued neuron using four symmetric real-valued neurons in four symmetric sectors can be bounded by  $\Omega(\theta^2 \| \boldsymbol{v} \|^2)$ , where  $\boldsymbol{w}_i$  indicates the weight vector of the  $i$ -th real-valued neuron, and  $\boldsymbol{v}$  denotes that of a complex-valued neuron.
The lower bound decreases at rate  $\Theta (\| \pmb{v} \|^{2} \min \{2\psi, \pi - 2\psi\}^{3} n^{-2})$ . The norm term  $\| \pmb{v} \|$  depicts the magnitude of the problem, which affects the expected square loss quadratically from the homogeneity of zReLU. In the extreme case of  $\pmb{v} = \mathbf{0}$ , a trivial real-valued neuron with  $\pmb{w} = \mathbf{0}$  reaches the lower bound 0. Meanwhile, the lower bound possesses a positive relation with a phase-dependent term  $\{2\psi, \pi - 2\psi\}$ . Intuitively, this term indicates the difference between a complex-valued neuron and a real-valued neuron. A real-valued neuron corresponds to  $\psi = 0$  or  $\psi = \pi / 2$  and this term measures the distance between the phase of a complex-valued neuron and a real-valued one. Finally, the lower bound decreases with a rate inversely proportional to the square of hidden size  $n$ . We conjecture that this dependence is tight and cannot be improved: a two-layer RVNN with  $n$  neurons divides the space into  $n$  pieces, in each of which RVNN acts as a linear function. Choosing the  $n$  weight vectors of RVNN suitably, the difference between the RVNN and the complex-valued neuron remains small (of order  $n^{-1}$ ) in each piece, which leads to the expected loss of order  $O(n^{-2})$ .
The conditions in Theorem 4 are made for conciseness of proof and we believe the conclusion holds in more general cases. The dimension  $d = 1$  corresponds to the intrinsic dimension of expressing a complex-valued neuron because of the rotational invariance of the inner product and the spherical symmetry of Gaussian distributions. Thus, additional dimensions contain no information and cannot improve the efficiency of approximation when  $d > 1$ . It is necessary to consider non-degenerate complex-valued neurons since degenerate complex-valued neurons are equivalent to real-valued ones.
We provide the central proof idea of Theorem 4 as follows. It is observed that the expected square loss  $L_{\mathrm{rc}}$  is a piecewise quadratic function and each piece forms a sector centered at the origin with infinite radius. In each piece,  $L_{\mathrm{rc}}$  takes the form  $\mathbb{E}[(\boldsymbol{w}^\top \boldsymbol{x})^2]$ . The proof mainly consists of two steps: we obtain a lower bound of  $L_{\mathrm{rc}}$  in a sector and then sum over all sectors with suitable weights and order. Firstly, we consider the expected loss in a sector with a small central angle  $\theta$ , as shown in Fig. 2(a). We divide the sector into three identical subareas  $A_1$ ,  $A_2$ , and  $A_3$ . Then at least one of  $A_1$  and  $A_3$  remains  $\theta / 6$  away from the vertical direction of  $\boldsymbol{w}$ , which leads to a lower bound  $\Omega(\theta^2 \| \boldsymbol{w} \|^2)$ . Secondly, we consider the loss in four rotationally symmetric sectors, as shown in Fig. 2(b), where  $\boldsymbol{v}$  represents a complex-valued neuron,  $\boldsymbol{w}_i$  indicates a real-valued neuron, and the expression in each sector implies the activated neurons. It is observed that at least one sector possesses a weight vector with norm  $\Omega(\| \boldsymbol{v} \|)$ , no matter how we choose the real-valued neurons. Thus, the overall loss is bounded by  $\Omega(\theta^2 \| \boldsymbol{v} \|^2)$ . Finally, we take the weight  $\alpha$  into consideration and sum over all sectors. For RVNN with  $n$  neurons, the best choice of  $\theta = \Theta(n^{-1})$  arrives at the lower bound  $\Omega(n^{-2} \| \boldsymbol{v} \|^2)$ . Detailed proofs are provided in Appendix D.
Summary and simulation experiments. We summarize the main conclusions of this section in Table 2. Both a real-valued neuron and a complex-valued neuron succeed in learning functions expressed by any one real-valued neuron. But difference occurs when learning those expressed by any non-degenerate complex-valued neuron: A complex-valued neuron can efficiently learn functions expressed by any one complex-valued neuron, but a two-layer RVNN with finite width cannot learn a single non-degenerate complex-valued neuron. Such a disagreement demonstrates that a complex-valued neuron possesses more powerful learning capability, which profits from the consideration of phase information in complex-valued operations. Our theoretical conclusions are based on the setting of low-dimensional inputs and no bias term, and the simulation results in Fig. 3 verify and extend these discoveries in more general settings. Details about the simulation experiments are available in Appendix F.
Table 2: A complex-valued neuron can learn more than a real-valued neuron.
Target	Real-valued Neuron	Complex-valued Neuron
Real-valued Neuron	O(e-ct) [19]	O(t-3) (Theorem 1)
Complex-valued Neuron	Cannot Learn (Theorem 4)	O(t-1) (Theorem 2)
(a)  $d = 1$  and no bias term.
(b)  $d = 5$  and with bias terms.
Figure 3: The test error of learning a complex-valued neuron. In both the theoretical setting (Fig. 3a) and more general settings (Fig. 3b), complex-valued neurons have vanishing errors, while real-valued neurons converge to positive errors.
5 Complex-valued Neurons Learn Slower
In this section, we demonstrate that complex-valued neurons learn slower than real-valued neurons. To arrive at this conclusion, we first rephrase the linear convergence of learning functions expressed by any one real-valued neuron using real-valued neurons. Then we prove that a complex-valued neuron learns the same class of functions at an exponentially slower rate.
We concentrate on learning one real-valued neuron  $\pmb{x} \rightarrow \tau(\pmb{v}^\top \pmb{x})$  with  $\| \pmb{v} \| = 1$ . When learning one real-valued neuron using a real-valued neuron, the expected square loss in Eq. (1) possesses the following simple closed form [41]
$$
L _ {\mathrm {r r}} (\boldsymbol {w}) = \frac {1}{4} \| \boldsymbol {w} \| ^ {2} - \frac {1}{2 \pi} \| \boldsymbol {w} \| [ \sin \theta_ {\boldsymbol {w}, \boldsymbol {v}} + (\pi - \theta_ {\boldsymbol {w}, \boldsymbol {v}}) \cos \theta_ {\boldsymbol {w}, \boldsymbol {v}} ] + \frac {1}{4}.
$$
It is widely known that a real-valued neuron learns a real-valued neuron with high probability and linear convergence rate [19], as reformulated by the following lemma.
Lemma 5. [19, Theorem 6.4] Suppose that the weight vector  $\boldsymbol{w} \in \mathbb{R}^{2d}$  is initialized by a Gaussian distribution  $\mathcal{N}(0, \mathbf{I} / (2d))$ . Let  $L_{\mathrm{rr}}$  denote the expected square loss of learning a real-valued neuron using a real-valued neuron. Then there exist constants  $c_1, c_2$  such that gradient descent with suitable step size satisfies
$$
\Pr \left[ L _ {\text {r r}} \left(\boldsymbol {w} _ {t}\right) \leqslant \mathrm {e} ^ {- c _ {1} t} \right] \geqslant 1 - \mathrm {e} ^ {- c _ {2} d}.
$$
Figure 4: A demonstration of the convergence stages of Theorem 6. The horizontal axis represents the iteration index of gradient descent. The black dotted line is the separation of convergence stages.
Recalling the expected loss of learning one real-valued neuron with a complex-valued neuron in Eq. (2), then we present the fourth theorem for complex-valued neuron learning, which provides a lower bound for the convergence rate.
Theorem 6. Let  $d = 1$ . Suppose that  $\| \pmb{w}_0 - \pmb{v} \| < 1$ . Let  $\{(w_t, \psi_t)\}_{t=0}^{\infty}$  denote the parameter sequence of the complex-valued neuron generated by projected gradient descent when optimizing  $L_{\mathrm{cr}}$ , the expected loss of learning a real-valued neuron using a complex-valued neuron. If the step size  $\eta_t = \eta \in (0, 1/(12\pi))$ , then we have
$$
L _ {\mathrm {c r}} \left(\boldsymbol {w} _ {t}, \psi_ {t}\right) \geqslant \frac {(1 - 1 2 \eta) ^ {3 T _ {3} / 2} \left(\psi^ {*} - \psi_ {0}\right) ^ {3}}{8 \pi (t - T _ {3} + 1) ^ {3}} - \frac {1}{2 \pi} \left(1 - \frac {\eta}{4 8}\right) ^ {t - T _ {3}},
$$
where  $\psi^{*} = \pi /2$ , and  $T_{3}$  is a constant dependent on  $\| \pmb{w}_0 - \pmb{v}\|$ ,  $\eta$ , and  $\psi^{*} - \psi_{0}$ .
Theorem 6 presents a lower bound for the expected loss of learning one real-valued neuron with a complex-valued neuron. It is observed that the negative term in the lower bound becomes 0 exponentially fast as  $t$  increases, and the positive term decreases with order  $\Omega(t^{-3})$ . Thus, the expected loss possesses a lower bound  $\Omega(t^{-3})$  since the positive term dominates the loss when  $t$  grows sufficiently large. This lower bound matches the upper bound in Theorem 1. Thus,  $O(t^{-3})$  becomes the utmost limit of learning with a complex-valued neuron via gradient descent, i.e., we cannot expect a complex-valued neuron to learn faster than this utmost limit.
The conditions in Theorem 6 are technical and reasonable. The condition on  $\pmb{w}_0$  is made for the conciseness of proof and can be removed. It is observed that  $(\pmb{w},\psi) = (\pmb{v},\psi^{*})$  is the unique global minimum with  $L_{\mathrm{cr}} = 0$ . Meanwhile, it is easy to verify that the loss goes to infinity when  $\| \pmb {w}\| \to \infty$ . Thus, if we aim to obtain a small loss, the parameter sequence must fall into a small neighborhood of the global minimum, which is depicted by condition  $\| \pmb {w}_0 - \pmb {v}\| \leqslant R < 1$ . The condition  $\psi_0\neq \psi^*$  holds with probability 1 when we initialize  $\psi_0$  with a continuous distribution. This condition is necessary to obtain a meaningful lower bound since the numerator of the positive term equals 0 when  $\psi_0 = \psi^*$ . We emphasize that this condition is essential and cannot be removed because a complex-valued neuron with  $\psi_0 = \psi^*$  is equivalent to a real-valued neuron, which enjoys linear convergence as stated in Lemma 5. The condition  $d = 1$  corresponds to the simplest optimization problem of learning one real-valued neuron with a complex-valued neuron since high-dimensional optimization brings more difficulties. Thus, we cannot expect a complex-valued neuron to learn a real-valued neuron with a convergence rate faster than  $O(t^{-3})$  in a higher dimension.
We summarize the proof idea of Theorem 6 as follows. The gradient with respect to  $\psi$  possesses the order  $(\psi^{*} - \psi)^{2} + (\psi^{*} - \psi)\theta_{w,v}$ . The key intuition is that  $\psi$  converges fast to the global minimum when  $\theta_{w,v}$  remains large, but  $\theta_{w,v}$  diminishes as  $w$  converges to the global minimum  $v$ . The detailed proofs are complicated and consist of several stages, depicting the entangled convergence between  $w$  and  $\psi$  as shown in Figure 4. In Stage I,  $\psi$  increases above a positive constant, which is a necessary condition for fast convergence of  $w$  in Stage II. When the distance between  $w$  and  $v$  declines below a threshold, the angle  $\theta_{w,v}$  becomes small. Then we enter Stage III, where  $w$  converges faster than  $\psi$ . Stage IV begins when  $\psi^{*} - \psi$  dominates  $\theta_{w,v}$ . Then the gradient degenerates to order  $(\psi^{*} - \psi)^{2}$ , which implies a lower bound of convergence  $\psi^{*} - \psi = \Omega (t^{-1})$ . Finally, estimating the loss around the global minimum leads to the conclusion. Detailed proofs are provided in Appendix E.
Summary and simulation experiments. Table 3 summarizes the conclusions in this section, which shows that a complex-valued neuron learns slower than a real-valued one. A complex-valued neuron is more flexible since it can learn the phase. But this flexibility becomes redundant and slows down the convergence when learning a phase-independent function. Our theories are based on the setting of low-dimensional inputs and no bias term, and the simulation results in Fig. 3 verify and extend these discoveries in more general settings. Details about the experiments are available in Appendix F.
Table 3: A real-valued neuron learns faster than a complex-valued neuron.
Target	Real-valued Neuron	Complex-valued Neuron
Real-valued Neuron	O(e-ct) (Lemma 5)	Ω(t-3) (Theorem 6)
(b)  $d = 5$  and with bias terms.
Figure 5: The test error of learning a real-valued neuron. In both the theoretical setting (Fig. 5a) and more general settings (Fig. 5b), a complex-valued neuron learns a real-valued neuron slower.
6 Conclusions and Prospects
In this paper, we investigate the problem of learning a single neuron using another neuron by optimizing the expected square loss via gradient descent. Firstly, we prove that a complex-valued neuron can efficiently learn functions expressed by any one real-valued neuron and any one complex-valued neuron with convergence rate  $O(t^{-3})$  and  $O(t^{-1})$ , respectively, where  $t$  denotes the iteration index of gradient descent. Meanwhile, two-layer RVNNs with finite width cannot learn a single nondegenerate complex-valued neuron in a strong sense that there always exists a positive gap between a two-layer RVNN of fixed width and a non-degenerate complex-valued neuron. These conclusions suggest that complex-valued neurons can learn more than real-valued neurons since CVNNs benefit from the phase parameter, which helps CVNNs learn phase information more efficiently. Secondly, we provide a convergence lower bound  $\Omega (t^{-3})$ , which matches the upper bound, for learning one real-valued neuron with a complex-valued neuron. This conclusion, together with the well-known linear convergence of learning one real-valued neuron with a real-valued neuron, implies that complex-valued neurons learn slower than real-valued neurons in phase-independent tasks. This phenomenon captures the additional price for learning simpler tasks with more complicated models, where the redundant phase consideration exponentially slows down the convergence.
Our study serves as a preliminary attempt to compare the learning process of artificial neural networks with different functional operations. In the future, it is important to extend our theoretical results to more general settings, such as cases of high-dimensional inputs, equipped with bias terms, and over-parameterized architectures [42]. Meanwhile, it is prospective to investigate complex-valued neuron learning from finite samples and derive a high-probability convergence condition. Since the empirical loss is a piecewise constant function with respect to the learnable phase parameter, it might be necessary to explore new learning algorithms, which is also encouraged by the neural tangent kernel aspect [43]. Besides, it is promising to consider the more practical and challenging procedure of learning general functions with deep architectures.
This research was supported by NSFC (61921006) and Collaborative Innovation Center of Novel Software Technology and Industrialization. We would like to thank the anonymous reviewers for their invaluable suggestions.
[1] Zhimian Zhang, Haipeng Wang, Feng Xu, and Ya-Qiu Jin. Complex-valued convolutional neural network and its application in polarimetric SAR image classification. IEEE Transactions on Geoscience and Remote Sensing, 55(12):7177-7188, 2017.
[2] Izhak Shafran, Tom Bagby, and RJ Skerry-Ryan. Complex evolution recurrent neural networks (ceRNNs). In Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 5854-5858, 2018.
[3] Shanshan Wang, Huitao Cheng, Leslie Ying, Taohui Xiao, Ziwen Ke, Hairong Zheng, and Dong Liang. DeepcomplexMRI: Exploiting deep residual network for fast parallel MR imaging with complex convolution. Magnetic Resonance Imaging, 68:136-147, 2020.
[4] Joshua Bassey, Lijun Qian, and Xianfang Li. A survey of complex-valued neural networks. arXiv:2101.12249, 2021.
[5] ChiYan Lee, Hideyuki Hasegawa, and Shangce Gao. Complex-valued neural networks: A comprehensive survey. IEEE/CAA Journal of Automatica Sinica, 9(8):1406-1426, 2022.
[6] Taehwan Kim and Tülay Adali. Universal approximation of fully complex feed-forward neural networks. In Proceedings of 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing, pages 973-976, 2002.
[7] Felix Voigtlaender. The universal approximation theorem for complex-valued neural networks. Applied and Computational Harmonic Analysis, 64:33-61, 2023.
[8] Paul Geuchen, Thomas Jahn, and Hannes Matt. Universal approximation with complex-valued deep narrow neural networks. arXiv:2305.16910, 2023.
[9] Bo Zhou and Qiankun Song. Boundedness and complete stability of complex-valued neural networks with time delay. IEEE Transactions on Neural Networks and Learning Systems, 24(8):1227-1238, 2013.
[10] Tohru Nitta. Local minima in hierarchical structures of complex-valued neural networks. Neural Networks, 43:1-7, 2013.
[11] Jin-Hui Wu, Shao-Qun Zhang, Yuan Jiang, and Zhi-Hua Zhou. Theoretical exploration of flexible transmitter model. IEEE Transactions on Neural Networks and Learning Systems, 2023. doi: 10.1109/TNNLS.2022.3195909.
[12] Shao-Qun Zhang, Wei Gao, and Zhi-Hua Zhou. Towards understanding theoretical advantages of complex-reaction networks. Neural Networks, 151:80-93, 2022.
[13] Eran Malach and Shai Shalev-Shwartz. Is deeper better only when shallow is good? In Advances in Neural Information Processing Systems 32, pages 6426-6435, 2019.
[14] Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. The connection between approximation, depth separation and learnability in neural networks. In Proceedings of the 34th Conference on Learning Theory, pages 3265-3295, 2021.
[15] Peter Auer, Mark Herbster, and Manfred K Warmuth. Exponentially many local minima for single neurons. Advances in Neural Information Processing Systems 8, pages 316-322, 1995.
[16] Spencer Frei, Yuan Cao, and Quanquan Gu. Agnostic learning of a single neuron with gradient descent. Advances in Neural Information Processing Systems 33, pages 5417-5428, 2020.