SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jin-Hui Wu, Shao-Qun Zhang, Yuan Jiang, Zhi-Hua Zhou
National Key Laboratory for Novel Software Technology, Nanjing University, China
School of Artificial Intelligence, Nanjing University, China
{wujh,zhangsq,jiangy,zhouzh}@lamda.nju.edu.cn
Complex-valued neural networks potentially possess better representations and performance than real-valued counterparts when dealing with some complicated tasks such as acoustic analysis, radar image classification, etc. Despite empirical successes, it remains unknown theoretically when and to what extent complex-valued neural networks outperform real-valued ones. We take one step in this direction by comparing the learnability of real-valued neurons and complex-valued neurons via gradient descent. We show that a complex-valued neuron can efficiently learn functions expressed by any one real-valued neuron and any one complex-valued neuron with convergence rate  $O(t^{-3})$  and  $O(t^{-1})$  where  $t$  is the iteration index of gradient descent, respectively, whereas a two-layer real-valued neural network with finite width cannot learn a single non-degenerate complex-valued neuron. We prove that a complex-valued neuron learns a real-valued neuron with rate  $\Omega(t^{-3})$ , exponentially slower than the  $O(\mathrm{e}^{-ct})$  rate of learning one real-valued neuron using a real-valued neuron with a constant  $c$ . We further verify and extend these results via simulation experiments in more general settings.
Complex-valued neural networks (CVNNs) utilize neuron models and operations in the complex-valued domain and are good at handling many complicated scenarios. Pioneering works successfully apply CVNNs to various areas, such as synthetic aperture radar image classification [1], acoustic analysis [2], and magnetic resonance image reconstruction [3]. In these applications, input signals naturally contain phase information. CVNNs seem more suitable than real-valued neural networks (RVNNs) in phase-dependent tasks since empirical experiments and intuitive explanations suggest that CVNNs can possess better data representations of phase information and grasp the phase-rotational dynamics more accurately [4, 5].
Beyond the seemingly promising performance of CVNNs, many efforts have been devoted to the theoretical understanding of CVNNs. Most existing works demonstrate some desirable properties of CVNNs such as universal approximation [6, 7], the minimum width for universal approximation [8], boundedness and complete stability [9], most critical points not being local minimum [10], and local-minimum-free conditions [11]. Some recent works demonstrate the approximation advantage of CVNNs in phase-invariant tasks by proving that neuromorphic networks with complex-valued operations can approximate radial functions with exponentially fewer parameters than RVNNs [11, 12]. However, those studies do not explain why CVNNs outperform RVNNs mostly in phase-dependent tasks, particularly when considering the fact that there are functions that can be efficiently approximated but not efficiently learned with gradient methods [13, 14]. Indeed, the general functional difference between CVNNs and RVNNS remains unknown.
In this paper, we take one step towards understanding when and to what extent one can benefit from common learning paradigms using CVNNs rather than RVNNs. More specifically, we attempt to identify the superiority and inferiority of CVNNs through the following fundamental questions
When CVNNs outperform RVNNs via gradient descent?
Can we learn everything with CVNNs without paying additional price?
We theoretically study the above two questions by focusing on learning a single neuron by optimizing the expected square loss via gradient descent under the setting of low-dimensional inputs and no bias term. Learning a single neuron, a simple and widely investigated learning problem [15, 16, 17, 18], is helpful to understand the difference between learning RVNNs and learning CVNNs since the neural operations inside a fundamental neuron model include the key factors of neural network learning. Furthermore, we conduct simulation experiments to verify our theories and extend them to more general settings of high-dimensional inputs and with bias terms. Our contributions are summarized in Table 1 and further explained as follows.
- Complex-valued neurons can learn more than real-valued neurons. We prove that using gradient descent, a single complex-valued neuron can efficiently learn functions expressed by any one real-valued neuron and any one complex-valued neuron with convergence rate  $O(t^{-3})$  and  $O(t^{-1})$  in Theorems 1 and 2, respectively. In contrast, we show the lower bound of expressing a non-degenerate complex-valued neuron with a two-layer RVNN in Theorem 4, which implies that a two-layer RVNN with finite width cannot learn a single non-degenerate complex-valued neuron. These results provide positive responses to the first question from at least two perspectives. Firstly, CVNNs outperform RVNNs when dealing with phase-sensitive tasks. Secondly, CVNN is always a conservative choice when we are unwilling to take the risk of failure.
- Complex-valued neurons learn slower than real-valued neurons. We present a lower bound  $\Omega(t^{-3})$  for learning functions expressed by any one real-valued neuron using a complex-valued neuron via gradient descent in Theorem 6. This conclusion, together with the well-known linear convergence of learning functions expressed by any one real-valued neuron using a real-valued neuron [19], implies that CVNNs suffer from slower convergence than RVNNs when handling simple phase-independent tasks. This phenomenon answers the second question and reveals the additional price for learning everything with CVNNs.
Table 1: A summary of our contributions. The first column lists the target neurons. The second and third columns represent the convergence rates of learning the target neurons using real-valued neurons and complex-valued neurons via gradient descent, respectively.
Target Neurons	Real-valued Neurons	Complex-valued Neurons
Real-valued Neuron	O(e-ct) [19]	Θ(t-3) (Theorems 1 and 6)
Complex-valued Neuron	Cannot Learn (Theorem 4)	O(t-1) (Theorem 2)
The rest of this paper is organized as follows. Section 2 introduces related works. Section 3 details our settings and notations. Section 4 demonstrates that complex-valued neurons can learn more than real-valued neurons. Section 5 proves that complex-valued neurons learn slower than real-valued neurons. Section 6 concludes our work with prospects.
Complex-valued Neural Networks. CVNNs originate in the 1990s when parameters of networks and the commonly used back-propagation algorithm are generalized to the complex-valued domain [20, 21, 22]. The motivation of CVNNs is at least threefold. From the representation perspective, CVNNs consider the phase information and model complex-valued problems more efficiently and properly than RVNNs [23, 24, 25]. From the computation perspective, a complex-valued neuron is capable of solving the exclusive-or problem and the detection of symmetry, whereas a real-valued neuron cannot [26]. From the biological perspective, the recently proposed flexible transmitter neuron [27], which has a natural complex implementation, formulates the communication between pre-synapse and post-synapse precisely rather than considering only the pre-synapse in traditional MP neuron [28]. CVNNs achieve better performance in versatile applications, especially those with naturally phase-related signals, such as radio frequency signals [29], sonar signals [30], and audio signals [31]. We refer to two surveys for more detailed discussions [4, 5].
From the aspect of theories, several works provide preliminary support for CVNNs by proving fundamental properties of CVNNs, such as shallow CVNNs are universal approximators [6, 7], most critical points are not spurious local minimum [10, 11], and CVNNs are bounded and completely stable [9]. These theoretical insights only consider CVNNs without comparison with RVNNs. Another line of research verifies the superiority of CVNNs by comparing the approximation complexity of RVNNs and CVNNs and finding that CVNNs can express radial functions more efficiently [11, 12]. This line of work only takes approximation into account and does not explicitly consider learning processes, which is of more interest in practice. This work takes the first step toward analyzing and comparing the learning behaviors of CVNNs and RVNNs.
Neuron Learning. Neuron learning is the simplest case of neural network learning, and existing works mainly focus on learning real-valued neurons. Some studies demonstrate the possibility of learning one real-valued neuron or a network using meticulously designed algorithms [32, 33, 34]. Later, researchers investigate the learnability of neurons using standard gradient methods. An exponential convergence rate is established for learning one real-valued neuron with a real-valued neuron under different assumptions [19, 35, 36, 37, 38]. We consider the problem of learning between one real-valued neuron and one complex-valued neuron, as well as learning one complex-valued neuron using a complex-valued neuron. The heterogeneity between real-valued and complex-valued neurons makes the analysis of optimization behaviors more complicated.
Notations. Suppose that the input dimension is an even number. For any vector  $\pmb{x} \in \mathbb{R}^{2d}$ , we denote  $x_{i}$  as the  $i$ -th coordinate of  $\pmb{x}$ . Let  $\pmb{x}_{\mathbb{C}} = (x_{1}; \ldots; x_{d}) + (x_{d+1}; \ldots; x_{2d})$ $\mathrm{i} \in \mathbb{C}^d$  be the folded complex-valued representation of  $\pmb{x}$ , and  $\overline{\pmb{x}}_{\mathbb{C}} = (x_{1}; \ldots; x_{d}) - (x_{d+1}; \ldots; x_{2d})$ $\mathrm{i}$  is the complex conjugate of  $\pmb{x}_{\mathbb{C}}$ . For any two vectors  $\pmb{w}, \pmb{v} \in \mathbb{R}^{2d}$ ,  $\theta_{\pmb{w},\pmb{v}} = \operatorname{arccos}(\pmb{w}^\top \pmb{v} \| \pmb{w} \|^{-1} \| \pmb{v} \|^{-1}) \in [0, \pi]$  denotes the angle between  $\pmb{w}$  and  $\pmb{v}$ . For any  $x \in \mathbb{R}$ ,  $\tau(x) = \max\{0, x\}$  indicates the ReLU activation function. Let  $\operatorname{Re}(z)$  denote the real part of a complex number  $z$ . For any  $z \in \mathbb{C}$  and  $\psi \in [0, \pi/2]$ ,  $\sigma_{\psi}(z)$  denotes the real part of the symmetrical version of zReLU activation function [39], i.e.,
$$
\sigma_ {\psi} (z) = \left\{ \begin{array}{l l} \operatorname {R e} (z), & \theta_ {z} \in [ - \psi , \psi ], \\ 0, & \text {o t h e r w i s e}, \end{array} \right.
$$
where  $\theta_z$  represents the argument of  $z$ . For any proposition  $p$ , we use  $\mathbb{I}(p)$  to represent the indicator function of  $p$ , i.e.,  $\mathbb{I}(p) = 1$  if  $p$  is true and  $\mathbb{I}(p) = 0$  otherwise. A table of frequently used notations is provided at the beginning of Appendix A.
Learning a Single Neuron. We consider learning a target neuron with a learning neuron. A neuron generally takes the form  $\pmb{x} \rightarrow \sigma_{\psi}(\pmb{w}; \pmb{x})$ , where the weight  $\pmb{w} \in \mathbb{R}^{2d}$  and phase  $\psi \in [0, \pi/2]$  indicate learnable parameters, and we omit the bias term for technical reasons. This general formulation includes a real-valued neuron with ReLU activation  $\pmb{x} \rightarrow \tau(\pmb{w}^{\top} \pmb{x})$  and a complex-valued neuron with zReLU activation  $\pmb{x} \rightarrow \sigma_{\psi}(\pmb{w}_{\mathbb{C}}^{\top} \overline{\pmb{x}}_{\mathbb{C}})$  as special cases. For any target neuron with parameters  $(\pmb{v}, \psi_v)$ , the learning process consists of finding a neuron with parameters  $(\pmb{w}, \psi_w)$  to minimize the expected square loss
$$
L (\boldsymbol {w}, \psi_ {\boldsymbol {w}}) = \frac {1}{2} \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {N} (\boldsymbol {0}, \mathbf {I})} \left[ \left(\sigma_ {\psi_ {\boldsymbol {w}}} (\boldsymbol {w}; \boldsymbol {x}) - \sigma_ {\psi_ {\boldsymbol {v}}} (\boldsymbol {v}; \boldsymbol {x})\right) ^ {2} \right], \tag {1}
$$
where the learnable parameter  $\psi_w$  occurs only when the learning neuron is complex-valued, and the input  $\pmb{x}$  follows the Gaussian distribution  $\mathcal{N}(\mathbf{0},\mathbf{I})$ .
Learning Algorithm and Initialization. We utilize the projected gradient descent as the learning algorithm, where the projection guarantees the constraint on phase  $\psi \in [0,\pi /2]$ . To minimize a function  $f(\pmb {x})$  with an initialization  $\pmb{x}_0$ , projected gradient descent iteratively updates weights along the negative gradient direction and projects the updated weights onto the constraint set, i.e.,  $\pmb{x}_{t + 1} = P_{\mathcal{Q}}(\pmb {x}_t - \eta_t\nabla_\pmb {x}f(\pmb {x}_t))$ , where  $\eta_t$  represents the step size,  $\mathcal{Q}$  denotes the constraint set, and  $P_{\mathcal{Q}}$  indicates the projection operator defined by  $P_{\mathcal{Q}}(\pmb {x}_0) = \arg \min_{\pmb {x}\in \mathcal{Q}}\| \pmb {x} - \pmb {x}_0\|$ . We initialize weights of neurons with Gaussian distribution, which includes most standard initialization schemes in practice [40]. The learnable parameter of the zReLU activation is initialized with  $\mathcal{U}(0,\pi /2)$ , i.e., the uniform distribution on  $[0,\pi /2]$ .
4 Complex-valued Neurons Can Learn More
In this section, we provide theoretical support for the learning advantage of complex-valued neurons by providing two positive learning scenarios for complex-valued neurons and one negative learning result for real-valued neurons. This section is organized as follows. Subsections 4.1 and 4.2 confirm the learning power of complex-valued neurons, by verifying that a complex-valued neuron can efficiently learn functions expressed by any one real-valued neuron and any one complex-valued neuron, respectively. Subsection 4.3 points out the limited learning capability of real-valued neurons, by proving that a two-layer RVNN with finite width cannot learn a single non-degenerate complex-valued neuron.
4.1 Learning One Real-valued Neuron with a Complex-valued Neuron
We first investigate the case of learning one real-valued neuron with ReLU activation using a complex-valued neuron with zReLU activation, where the expected square loss in Eq. (1) becomes
$$
L _ {\mathrm {c r}} (\boldsymbol {w}, \psi) = \frac {1}{2} \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {N} (\boldsymbol {0}, \mathbf {I})} \left[ \left(\sigma_ {\psi} \left(\boldsymbol {w} _ {\mathbb {C}} ^ {\top} \overline {{\boldsymbol {x}}} _ {\mathbb {C}}\right) - \tau \left(\boldsymbol {v} ^ {\top} \boldsymbol {x}\right)\right) ^ {2} \right], \tag {2}
$$
where we abbreviate the phase parameter  $\psi_w$  as  $\psi$  since the target real-valued neuron does not have a phase parameter,  $\pmb{w} \in \mathbb{R}^{2d}$  and  $\pmb{v} \in \mathbb{R}^{2d}$  represent the weight vectors of the complex-valued neuron and the real-valued neuron, respectively. We assume  $\| \pmb{v} \| = 1$  without loss of generality. Then we present the first theorem for complex-valued neuron learning.
Theorem 1. Let  $d = 1$ . Suppose that  $w_0 \sim \mathcal{N}(0, I_2)$  and  $\psi_0 \sim \mathcal{U}(0, \pi/2)$ . Let  $\{(w_t, \psi_t)\}_{t=0}^{\infty}$  denote the parameter sequence of the complex-valued neuron generated by projected gradient descent when optimizing  $L_{\mathrm{cr}}$ , the expected loss of learning a real-valued neuron using a complex-valued neuron. If the step size  $\eta_t = \eta \in (0, 1/(12\pi))$ , then we have
$$
\Pr \left[ L _ {\mathrm {c r}} (\boldsymbol {w} _ {t}, \psi_ {t}) \leqslant \frac {8 0 0 0}{\eta^ {3} t ^ {3}} + \left(1 - \frac {\eta}{4 8}\right) ^ {t + 1 - 3 2 / \eta} \right] \geqslant \frac {1}{3 2}.
$$
Theorem 1 shows that a complex-valued neuron can efficiently learn the functions expressed by any one real-valued neuron with convergence rate  $O(t^{-3})$  using projected gradient descent. It should be mentioned that we do not attempt to decrease the large constants in the theorem, as they do not hurt the constant probability and convergence rate.
The constant probability, rather than high probability, comes from the intrinsic difference between real-valued neurons and complex-valued neurons. A real-valued neuron activates half of the phase domain, whereas a complex-valued neuron may only activate a small part as controlled by the parameter  $\psi$ , which makes the expected loss a piecewise function. When the initialization of  $\boldsymbol{w}$  falls into the opposite direction of  $\boldsymbol{v}$  and  $\psi$  is small, the activated regions of the real-valued and complex-valued neurons are not overlapped. Such a bad initialization happens with a constant probability and encourages the complex-valued neuron to decrease phase to minimize the loss. As a result, the phase of the complex-valued neuron will shrink to zero, which leads to a constant expected square loss and the failure of learning.
Challenges. Although  $(\boldsymbol{w}, \psi) = (\boldsymbol{v}, \pi/2)$  is an obvious global minimum of the expected loss with  $L_{\mathrm{cr}} = 0$ , the convergence conclusion in Theorem 1 is non-trivial. As one can see in the proof, the landscape of the expected loss possesses a stationary point  $(\boldsymbol{w}, \psi) = \mathbf{0}$ . If we initialize  $\boldsymbol{w} = -k\boldsymbol{v}$  with  $k > 0$ , then it is easy to verify that  $\boldsymbol{w}$  converges to  $\mathbf{0}$  and  $\psi$  decreases to 0 when the step size is sufficiently small. This implies that the landscape is not convex and the spurious stationary point is an attractor. The existence of this spurious stationary point becomes a critical obstacle in the proof and provides another reason for the hardness of a high-probability conclusion.
The proof idea of Theorem 1 mainly consists of estimating the first-order derivatives and finding an ideal region with both constant probability and convergence guarantees. We provide a proof sketch as follows. Firstly, we analyze the optimization behaviors of  $\pmb{w}$  and  $\psi$  in all pieces of the loss function separately. Then we identify an ideal region with desirable gradient properties: the gradient  $\nabla_{\psi}L_{\mathrm{cr}}(\pmb{w},\psi)$  can be bounded by  $O((\psi -\pi /2)^2)$ , which implies that  $\psi -\pi /2$  decreases with an inversely propositional rate. Meanwhile, gradient descent on  $\pmb{w}$  performs like a contraction mapping with fixed point  $\pmb{v}$  and Lipschitz constant  $1 - \Theta (\psi)$ , i.e.,  $\pmb{w}$  converges to  $\pmb{v}$  linearly when  $\psi$
(a) Convergence stages of Theorem 1.
(b) Convergence stages of Theorem 2.
Figure 1: Subfigures (a) and (b) demonstrate the convergence stages of Theorems 1 and 2, respectively. The horizontal axis represents the iteration index of gradient descent. The black dotted line denotes the separation of convergence stages.
is large enough. Based on these observations, our convergence analysis consists of two stages, as shown in Fig. 1(a). In Stage I, the phase  $\psi$  converges towards the global minimum, and the weight  $w$  remains in the ideal region. When the phase grows above some threshold, one enters Stage II where the weight converges linearly and the phase maintains its slow convergence rate. Finally, we estimate the order of loss and provide a lower bound of the probability of falling into the ideal region with Gaussian initialization to complete the proof. Detailed proofs are available in Appendix B.
4.2 Learning One Complex-valued Neuron with a Complex-valued Neuron
We proceed to consider learning one complex-valued neuron using a complex-valued neuron. In this case, the expected square loss in Eq. (1) can be rewritten as
$$
L _ {\mathrm {c c}} (\pmb {w}, \psi) = \frac {1}{2} \mathbb {E} _ {\pmb {x} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} \left[ \left(\sigma_ {\psi} (\pmb {w} _ {\mathbb {C}} ^ {\top} \overline {{\pmb {x}}} _ {\mathbb {C}}) - \sigma_ {\phi} (\pmb {v} _ {\mathbb {C}} ^ {\top} \overline {{\pmb {x}}} _ {\mathbb {C}})\right) ^ {2} \right],
$$
where  $(\pmb{v},\phi)$  denotes the parameter of the target complex-valued neuron, and  $(\pmb{w},\psi)$  is the learnable parameter. Without loss of generality, we still assume  $\| \pmb{v}\| = 1$ . Here, we use gradient descent with vanishing step size  $x_{t + 1} = x_t - \eta_t\nabla f(x_t)$ , where the positive step size  $\eta_t$  satisfies  $\eta_t\to 0$  as  $t\to \infty$ . Then we present the second theorem for complex-valued neuron learning.
Theorem 2. Let  $d = 1$ , and  $\psi_v \in [7\pi / 20, 2\pi / 5]$ . Suppose that  $\mathbf{w}_0 \sim \mathcal{N}(\mathbf{0}, I_2)$  and  $\psi_{w,0} \sim \mathcal{U}(0, \pi / 2)$ . Let  $\{(\mathbf{w}_t, \psi_{w,t})\}_{t=0}^{\infty}$  denote the parameter sequence of the complex-valued neuron generated by projected gradient descent when optimizing  $L_{\mathrm{cc}}$ , the expected loss of learning a complex-valued neuron using a complex-valued neuron. If we utilize vanishing step size  $\eta_t = \min \{c_1, c_2 / t\}$  with  $c_1 \leqslant 1 / 3000$  and  $c_2 \geqslant 20$ , then
$$
\Pr \left[ L _ {\mathrm {c c}} (\boldsymbol {w} _ {t}, \psi_ {w, t}) \leqslant \frac {4 0 0 c _ {2} ^ {3}}{c _ {1} t} \right] \geqslant 1 0 ^ {- 5}.
$$
Theorem 2 demonstrates that a complex-valued neuron can efficiently learn functions expressed by any one complex-valued neuron with convergence rate  $O(t^{-1})$  and constant probability.
Challenges. It is observed that the  $O(t^{-1})$  convergence rate in Theorem 2 is slower than the  $O(t^{-3})$  convergence rate in Theorem 1. The deceleration of convergence comes from the intrinsic difficulties of learning functions expressed by any one complex-valued neuron. These difficulties become the main challenges in the analysis and can be understood from at least two perspectives. Firstly, there emerge new spurious stationary points. As one can see in the proof, the gradient with respect to  $\psi_w$  becomes 0 once  $\psi_w$  reaches  $\pi / 2$  and  $w$  is close to  $v$ , i.e.,  $(w, \psi_w) = (v, \pi / 2)$  is a spurious stationary point. Secondly, the landscape of the loss function is no longer smooth. For both  $w$  and  $\psi_w$ , the local landscape around the global minimum is roughly an absolute function, which declares the non-smoothness of the loss and the failure of gradient descent with a constant step size.
To overcome these obstacles, we apply mild conditions and slight modifications to guarantee convergence and maintain the generality of our conclusion. We separate the phase of the target complex
valued neuron far from 0 and  $\pi /2$  in consideration of spurious local stationary points: As  $\psi_v$  becomes closer to 0, it is more likely to obtain an initialization of the learning neuron that does not overlap with the target neuron. Then we will take the risk of falling into the spurious local minimum  $(\boldsymbol{w},\psi_w) = (\mathbf{0},0)$ . As  $\psi_v$  approaches  $\pi /2$ , we are confronted by another spurious stationary point  $\psi_w = \pi /2$ . We utilize gradient descent with a vanishing step size to cope with the non-smoothness of the loss function since a constant step size inevitably suffers from oscillation.