Title: Complex-valued Neurons Can Learn More but Slower than Real-valued Neurons via Gradient Descent
Venue: NeurIPS 2023
Accepted: yes
Keywords: Complex-valued Neural Networks; Learning Neurons; Real-valued Neural Networks; Convergence Rate
Abstract: Complex-valued neural networks potentially possess better representations and performance than real-valued counterparts when dealing with some complicated tasks such as acoustic analysis, radar image classification, etc. Despite empirical successes, it remains unknown theoretically when and to what extent complex-valued neural networks outperform real-valued ones. We take one step in this direction by comparing the learnability of real-valued neurons and complex-valued neurons via gradient descent. We show that a complex-valued neuron can efficiently learn functions expressed by any one real-valued neuron and any one complex-valued neuron with convergence rate $O(t^{-3})$ and $O(t^{-1})$ where $t$ is the iteration index of gradient descent, respectively, whereas a two-layer real-valued neural network with finite width cannot learn a single non-degenerate complex-valued neuron. We prove that a complex-valued neuron learns a real-valued neuron with rate $\Omega (t^{-3})$, exponentially slower than the $O(\mathrm{e}^{- c t})$ rate of learning one real-valued neuron using a real-valued neuron with a constant $c$. We further verify and extend these results via simulation experiments in more general settings.

Complex-valued Neurons Can Learn More but Slower than Real-valued Neurons via Gradient Descent

Abstract

Complex-valued neural networks potentially possess better representations and performance than real-valued counterparts when dealing with some complicated tasks such as acoustic analysis, radar image classification, etc. Despite empirical successes, it remains unknown theoretically when and to what extent complexvalued neural networks outperform real-valued ones. We take one step in this direction by comparing the learnability of real-valued neurons and complex-valued neurons via gradient descent. We show that a complex-valued neuron can efficiently learn functions expressed by any one real-valued neuron and any one complexvalued neuron with convergence rate O ( t -3 ) and O ( t -1 ) where t is the iteration index of gradient descent, respectively, whereas a two-layer real-valued neural network with finite width cannot learn a single non-degenerate complex-valued neuron. We prove that a complex-valued neuron learns a real-valued neuron with rate Ω( t -3 ) , exponentially slower than the O (e -ct ) rate of learning one real-valued neuron using a real-valued neuron with a constant c . We further verify and extend these results via simulation experiments in more general settings.

1 Introduction

Complex-valued neural networks (CVNNs) utilize neuron models and operations in the complexvalued domain and are good at handling many complicated scenarios. Pioneering works successfully apply CVNNs to various areas, such as synthetic aperture radar image classification [1], acoustic analysis [2], and magnetic resonance image reconstruction [3]. In these applications, input signals naturally contain phase information. CVNNs seem more suitable than real-valued neural networks (RVNNs) in phase-dependent tasks since empirical experiments and intuitive explanations suggest that CVNNs can possess better data representations of phase information and grasp the phase-rotational dynamics more accurately [4, 5].

Beyond the seemingly promising performance of CVNNs, many efforts have been devoted to the theoretical understanding of CVNNs. Most existing works demonstrate some desirable properties of CVNNs such as universal approximation [6, 7], the minimum width for universal approximation [8], boundedness and complete stability [9], most critical points not being local minimum [10], and local-minimum-free conditions [11]. Some recent works demonstrate the approximation advantage of CVNNs in phase-invariant tasks by proving that neuromorphic networks with complex-valued operations can approximate radial functions with exponentially fewer parameters than RVNNs [11, 12]. However, those studies do not explain why CVNNs outperform RVNNs mostly in phasedependent tasks, particularly when considering the fact that there are functions that can be efficiently approximated but not efficiently learned with gradient methods [13, 14]. Indeed, the general functional difference between CVNNs and RVNNS remains unknown.

In this paper, we take one step towards understanding when and to what extent one can benefit from common learning paradigms using CVNNs rather than RVNNs. More specifically, we attempt to identify the superiority and inferiority of CVNNs through the following fundamental questions

When CVNNs outperform RVNNs via gradient descent?

Can we learn everything with CVNNs without paying additional price?

We theoretically study the above two questions by focusing on learning a single neuron by optimizing the expected square loss via gradient descent under the setting of low-dimensional inputs and no bias term. Learning a single neuron, a simple and widely investigated learning problem [15, 16, 17, 18], is helpful to understand the difference between learning RVNNs and learning CVNNs since the neural operations inside a fundamental neuron model include the key factors of neural network learning. Furthermore, we conduct simulation experiments to verify our theories and extend them to more general settings of high-dimensional inputs and with bias terms. Our contributions are summarized in Table 1 and further explained as follows.

Complex-valued neurons can learn more than real-valued neurons. We prove that using gradient descent, a single complex-valued neuron can efficiently learn functions expressed by any one real-valued neuron and any one complex-valued neuron with convergence rate O ( t -3 ) and O ( t -1 ) in Theorems 1 and 2, respectively. In contrast, we show the lower bound of expressing a non-degenerate complex-valued neuron with a two-layer RVNN in Theorem 4, which implies that a two-layer RVNN with finite width cannot learn a single non-degenerate complex-valued neuron. These results provide positive responses to the first question from at least two perspectives. Firstly, CVNNs outperform RVNNs when dealing with phase-sensitive tasks. Secondly, CVNN is always a conservative choice when we are unwilling to take the risk of failure.

Complex-valued neurons learn slower than real-valued neurons. We present a lower bound Ω( t -3 ) for learning functions expressed by any one real-valued neuron using a complex-valued neuron via gradient descent in Theorem 6. This conclusion, together with the well-known linear convergence of learning functions expressed by any one real-valued neuron using a real-valued neuron [19], implies that CVNNs suffer from slower convergence than RVNNs when handling simple phase-independent tasks. This phenomenon answers the second question and reveals the additional price for learning everything with CVNNs.

The rest of this paper is organized as follows. Section 2 introduces related works. Section 3 details our settings and notations. Section 4 demonstrates that complex-valued neurons can learn more than real-valued neurons. Section 5 proves that complex-valued neurons learn slower than real-valued neurons. Section 6 concludes our work with prospects.

2 Related Works

Complex-valued Neural Networks. CVNNs originate in the 1990s when parameters of networks and the commonly used back-propagation algorithm are generalized to the complex-valued domain [20, 21, 22]. The motivation of CVNNs is at least threefold. From the representation perspective, CVNNs consider the phase information and model complex-valued problems more efficiently and properly than RVNNs [23, 24, 25]. From the computation perspective, a complex-valued neuron is capable of solving the exclusive-or problem and the detection of symmetry, whereas a real-valued neuron cannot [26]. From the biological perspective, the recently proposed flexible transmitter neuron [27], which has a natural complex implementation, formulates the communication between pre-synapse and post-synapse precisely rather than considering only the pre-synapse in traditional MP neuron [28]. CVNNs achieve better performance in versatile applications, especially those with naturally phase-related signals, such as radio frequency signals [29], sonar signals [30], and audio signals [31]. We refer to two surveys for more detailed discussions [4, 5].

From the aspect of theories, several works provide preliminary support for CVNNs by proving fundamental properties of CVNNs, such as shallow CVNNs are universal approximators [6, 7], most critical points are not spurious local minimum [10, 11], and CVNNs are bounded and completely stable [9]. These theoretical insights only consider CVNNs without comparison with RVNNs. Another line of research verifies the superiority of CVNNs by comparing the approximation complexity of RVNNs and CVNNs and finding that CVNNs can express radial functions more efficiently [11, 12]. This line of work only takes approximation into account and does not explicitly consider learning processes, which is of more interest in practice. This work takes the first step toward analyzing and comparing the learning behaviors of CVNNs and RVNNs.

Neuron Learning. Neuron learning is the simplest case of neural network learning, and existing works mainly focus on learning real-valued neurons. Some studies demonstrate the possibility of learning one real-valued neuron or a network using meticulously designed algorithms [32, 33, 34]. Later, researchers investigate the learnability of neurons using standard gradient methods. An exponential convergence rate is established for learning one real-valued neuron with a real-valued neuron under different assumptions [19, 35, 36, 37, 38]. We consider the problem of learning between one real-valued neuron and one complex-valued neuron, as well as learning one complex-valued neuron using a complex-valued neuron. The heterogeneity between real-valued and complex-valued neurons makes the analysis of optimization behaviors more complicated.

3 Preliminaries

Notations. Suppose that the input dimension is an even number. For any vector x ∈ R 2 d , we denote x i as the i -th coordinate of x . Let x C = ( x 1 ; . . . ; x d ) + ( x d +1 ; . . . ; x 2 d )i ∈ C d be the folded complex-valued representation of x , and x C = ( x 1 ; . . . ; x d ) -( x d +1 ; . . . ; x 2 d )i is the complex conjugate of x C . For any two vectors w , v ∈ R 2 d , θ w , v = arccos( w glyph[latticetop] v ‖ w ‖ -1 ‖ v ‖ -1 ) ∈ [0 , π ] denotes the angle between w and v . For any x ∈ R , τ ( x ) = max { 0 , x } indicates the ReLU activation function. Let Re( z ) denote the real part of a complex number z . For any z ∈ C and ψ ∈ [0 , π/ 2] , σ ψ ( z ) denotes the real part of the symmetrical version of zReLU activation function [39], i.e.,

where θ z represents the argument of z . For any proposition p , we use I ( p ) to represent the indicator function of p , i.e., I ( p ) = 1 if p is true and I ( p ) = 0 otherwise. A table of frequently used notations is provided at the beginning of Appendix A.

Learning a Single Neuron. We consider learning a target neuron with a learning neuron. A neuron generally takes the form x → σ ψ ( w ; x ) , where the weight w ∈ R 2 d and phase ψ ∈ [0 , π/ 2] indicate learnable parameters, and we omit the bias term for technical reasons. This general formulation includes a real-valued neuron with ReLU activation x → τ ( w glyph[latticetop] x ) and a complex-valued neuron with zReLU activation x → σ ψ ( w glyph[latticetop] C x C ) as special cases. For any target neuron with parameters ( v , ψ v ) , the learning process consists of finding a neuron with parameters ( w , ψ w ) to minimize the expected square loss

where the learnable parameter ψ w occurs only when the learning neuron is complex-valued, and the input x follows the Gaussian distribution N ( 0 , I ) .

Learning Algorithm and Initialization. We utilize the projected gradient descent as the learning algorithm, where the projection guarantees the constraint on phase ψ ∈ [0 , π/ 2] . To minimize a function f ( x ) with an initialization x 0 , projected gradient descent iteratively updates weights along the negative gradient direction and projects the updated weights onto the constraint set, i.e., x t +1 = P Q ( x t -η t ∇ x f ( x t )) , where η t represents the step size, Q denotes the constraint set, and P Q indicates the projection operator defined by P Q ( x 0 ) = arg min x ∈Q ‖ x -x 0 ‖ . We initialize weights of neurons with Gaussian distribution, which includes most standard initialization schemes in practice [40]. The learnable parameter of the zReLU activation is initialized with U (0 , π/ 2) , i.e., the uniform distribution on [0 , π/ 2] .

4 Complex-valued Neurons Can Learn More

In this section, we provide theoretical support for the learning advantage of complex-valued neurons by providing two positive learning scenarios for complex-valued neurons and one negative learning result for real-valued neurons. This section is organized as follows. Subsections 4.1 and 4.2 confirm the learning power of complex-valued neurons, by verifying that a complex-valued neuron can efficiently learn functions expressed by any one real-valued neuron and any one complex-valued neuron, respectively. Subsection 4.3 points out the limited learning capability of real-valued neurons, by proving that a two-layer RVNN with finite width cannot learn a single non-degenerate complexvalued neuron.

4.1 Learning One Real-valued Neuron with a Complex-valued Neuron

We first investigate the case of learning one real-valued neuron with ReLU activation using a complexvalued neuron with zReLU activation, where the expected square loss in Eq. (1) becomes

where we abbreviate the phase parameter ψ w as ψ since the target real-valued neuron does not have a phase parameter, w ∈ R 2 d and v ∈ R 2 d represent the weight vectors of the complex-valued neuron and the real-valued neuron, respectively. We assume ‖ v ‖ = 1 without loss of generality. Then we present the first theorem for complex-valued neuron learning.

Theorem 1. Let d = 1 . Suppose that w 0 ∼ N (0 , I 2 ) and ψ 0 ∼ U (0 , π/ 2) . Let { ( w t , ψ t ) } ∞ t =0 denote the parameter sequence of the complex-valued neuron generated by projected gradient descent when optimizing L cr , the expected loss of learning a real-valued neuron using a complex-valued neuron. If the step size η t = η ∈ (0 , 1 / (12 π )) , then we have

Theorem 1 shows that a complex-valued neuron can efficiently learn the functions expressed by any one real-valued neuron with convergence rate O ( t -3 ) using projected gradient descent. It should be mentioned that we do not attempt to decrease the large constants in the theorem, as they do not hurt the constant probability and convergence rate.

The constant probability, rather than high probability, comes from the intrinsical difference between real-valued neurons and complex-valued neurons. A real-valued neuron activates half of the phase domain, whereas a complex-valued neuron may only activate a small part as controlled by the parameter ψ , which makes the expected loss a piecewise function. When the initialization of w falls into the opposite direction of v and ψ is small, the activated regions of the real-valued and complex-valued neurons are not overlapped. Such a bad initialization happens with a constant probability and encourages the complex-valued neuron to decrease phase to minimize the loss. As a result, the phase of the complex-valued neuron will shrink to zero, which leads to a constant expected square loss and the failure of learning.

Challenges. Although ( w , ψ ) = ( v , π/ 2) is an obvious global minimum of the expected loss with L cr = 0 , the convergence conclusion in Theorem 1 is non-trivial. As one can see in the proof, the landscape of the expected loss possesses a stationary point ( w , ψ ) = 0 . If we initialize w = -k v with k > 0 , then it is easy to verify that w converges to 0 and ψ decreases to 0 when the step size is sufficiently small. This implies that the landscape is not convex and the spurious stationary point is an attractor. The existence of this spurious stationary point becomes a critical obstacle in the proof and provides another reason for the hardness of a high-probability conclusion.

The proof idea of Theorem 1 mainly consists of estimating the first-order derivatives and finding an ideal region with both constant probability and convergence guarantees. We provide a proof sketch as follows. Firstly, we analyze the optimization behaviors of w and ψ in all pieces of the loss function separately. Then we identify an ideal region with desirable gradient properties: the gradient ∇ ψ L cr ( w , ψ ) can be bounded by O (( ψ -π/ 2) 2 ) , which implies that ψ -π/ 2 decreases with an inversely propositional rate. Meanwhile, gradient descent on w performs like a contraction mapping with fixed point v and Lipschitz constant 1 -Θ( ψ ) , i.e., w converges to v linearly when ψ

(a) Convergence stages of Theorem 1.

(b) Convergence stages of Theorem 2.

Ilw-vll

--lyw-π/2|

Stage I

Stage II

t

Ilw -vll

---w

Stage I

Stage II

Stage IIl

t

is large enough. Based on these observations, our convergence analysis consists of two stages, as shown in Fig. 1(a). In Stage I, the phase ψ converges towards the global minimum, and the weight w remains in the ideal region. When the phase grows above some threshold, one enters Stage II where the weight converges linearly and the phase maintains its slow convergence rate. Finally, we estimate the order of loss and provide a lower bound of the probability of falling into the ideal region with Gaussian initialization to complete the proof. Detailed proofs are available in Appendix B.

4.2 Learning One Complex-valued Neuron with a Complex-valued Neuron

We proceed to consider learning one complex-valued neuron using a complex-valued neuron. In this case, the expected square loss in Eq. (1) can be rewritten as

where ( v , φ ) denotes the parameter of the target complex-valued neuron, and ( w , ψ ) is the learnable parameter. Without loss of generality, we still assume ‖ v ‖ = 1 . Here, we use gradient descent with vanishing step size x t +1 = x t -η t ∇ f ( x t ) , where the positive step size η t satisfies η t → 0 as t →∞ . Then we present the second theorem for complex-valued neuron learning.

Theorem 2. Let d = 1 , and ψ v ∈ [7 π/ 20 , 2 π/ 5] . Suppose that w 0 ∼ N ( 0 , I 2 ) and ψ w, 0 ∼ U (0 , π/ 2) . Let { ( w t , ψ w,t ) } ∞ t =0 denote the parameter sequence of the complex-valued neuron generated by projected gradient descent when optimizing L cc , the expected loss of learning a complexvalued neuron using a complex-valued neuron. If we utilize vanishing step size η t = min { c 1 , c 2 /t } with c 1 glyph[lessorequalslant] 1 / 3000 and c 2 glyph[greaterorequalslant] 20 , then

Theorem 2 demonstrates that a complex-valued neuron can efficiently learn functions expressed by any one complex-valued neuron with convergence rate O ( t -1 ) and constant probability.

Challenges. It is observed that the O ( t -1 ) convergence rate in Theorem 2 is slower than the O ( t -3 ) convergence rate in Theorem 1. The deceleration of convergence comes from the intrinsic difficulties of learning functions expressed by any one complex-valued neuron. These difficulties become the main challenges in the analysis and can be understood from at least two perspectives. Firstly, there emerge new spurious stationary points. As one can see in the proof, the gradient with respect to ψ w becomes 0 once ψ w reaches π/ 2 and w is close to v , i.e., ( w , ψ w ) = ( v , π/ 2) is a spurious stationary point. Secondly, the landscape of the loss function is no longer smooth. For both w and ψ w , the local landscape around the global minimum is roughly an absolute function, which declares the non-smoothness of the loss and the failure of gradient descent with a constant step size.

To overcome these obstacles, we apply mild conditions and slight modifications to guarantee convergence and maintain the generality of our conclusion. We separate the phase of the target complex-

valued neuron far from 0 and π/ 2 in consideration of spurious local stationary points: As ψ v becomes closer to 0 , it is more likely to obtain an initialization of the learning neuron that does not overlap with the target neuron. Then we will take the risk of falling into the spurious local minimum ( w , ψ w ) = ( 0 , 0) . As ψ v approaches π/ 2 , we are confronted by another spurious stationary point ψ w = π/ 2 . We utilize gradient descent with a vanishing step size to cope with the non-smoothness of the loss function since a constant step size inevitably suffers from oscillation.

We summarize the proof idea of Theorem 2 as follows. The overall procedure is similar to that of Theorem 1 but every step is different and more challenging because of non-smoothness and more spurious stationary points. Firstly, we identify an ideal region with nice gradient properties: the gradient with respect to w ⊥ , the weight component perpendicular to v , points to the global minimum 0 and maintains constant order. The gradient ∇ ψ w is bounded and points towards ψ v when the angle θ w , v is small enough. Meanwhile, the gradient with respect to w v performs like a contraction mapping with fixed point [1 -Θ( ψ v ψ -1 w )] v and Lipschitz constant 1 -Θ( ψ ) , i.e., there exists a deviation of the fixed point from the global minimum. Based on these observations, we then prove the convergence with three stages, as demonstrated in Fig 1(b): In Stage I, w ⊥ , the weight component perpendicular to v , converges to 0 with an inversely proportional rate, and ψ w and w v remain in the ideal region. Thus, the angle θ w , v decreases with an inversely proportional rate. When θ w , v declines below some threshold, we come to Stage II where phase ψ w converges to ψ v with rate O ( t -1 ) . As ψ w approaches ψ v , the fixed point becomes close to v and we step into Stage III where w converges to v with the same rate as ψ w . Finally, we estimate the order of loss and provide a lower bound of the probability of falling into the ideal region with Gaussian initialization to complete the proof. We provide detailed proofs in Appendix C.

4.3 Finite-Width RVNNs Cannot Learn a Single Non-degenerate Complex-valued Neuron

We then study learning one complex-valued neuron with zReLU activation using real-valued neurons. Since a complex-valued neuron has more parameters than a real-valued neuron, it is unfair to learn a complex-valued neuron with a single real-valued neuron. Thus, we consider the problem of learning a complex-valued neuron with a two-layer RVNN. A two-layer RVNN with n hidden neurons can be represented by x → α glyph[latticetop] τ ( W x ) , where W ∈ R n × 2 d and α ∈ R n indicate weight parameters of the network, and τ is the ReLU activation function applied componentwisely. We still focus on the expected square loss, which takes the form

where we abbreviate the phase parameter ψ v as ψ since RVNN has no phase parameter. We are mainly interested in learning a non-degenerate complex-valued neuron, which is distinct from a real-valued neuron and defined as follows.

glyph[negationslash]

Definition 3. A complex-valued neuron is a non-degenerate one if ψ / ∈ { 0 , π/ 2 } and v = 0 .

For a complex-valued neuron with phase ψ = 0 or v = 0 , the zReLU activation function always outputs 0 . Then the complex-valued neuron is equivalent to a real-valued neuron with all zero weights. For a complex-valued with phase ψ = π/ 2 , the zReLU activation function is equivalent to the ReLU activation function. Thus, a non-degenerate complex-valued neuron is a non-real-valued neuron. Then we present the third theorem for complex-valued neuron learning.

Theorem 4. Let d = 1 . For any non-degenerate complex-valued neuron with phase ψ ∈ (0 , π/ 2) and non-zero weight vector v ∈ C d , denote by L rc the expected square loss of learning this complexvalued neuron using a one-hidden-layer RVNN with n hidden neurons. Then the loss satisfies

Theorem 4 provides a positive lower bound for the expected squared loss of approximating a nondegenerate complex-valued neuron using a two-layer RVNN with a fixed number of hidden neurons. This positive lower bound indicates that there always remains a positive gap between the target non-degenerate complex-valued neuron and the two-layer RVNN of fixed width no matter how the parameters of the RVNN are learned. Thus, a finite-width RVNN cannot learn a single non-degenerate complex-valued neuron.

(a) The loss in one sector.

(b) The loss in four sectors.

W

θ

A3

A2

A1

S

S2

W1+W2

W1

S1

W4 + W1 - v

S3

θ

W3

W4

The lower bound decreases at rate Θ( ‖ v ‖ 2 min { 2 ψ, π -2 ψ } 3 n -2 ) . The norm term ‖ v ‖ depicts the magnitude of the problem, which affects the expected square loss quadratically from the homogeneity of zReLU. In the extreme case of v = 0 , a trivial real-valued neuron with w = 0 reaches the lower bound 0 . Meanwhile, the lower bound possesses a positive relation with a phase-dependent term { 2 ψ, π -2 ψ } . Intuitively, this term indicates the difference between a complex-valued neuron and a real-valued neuron. A real-valued neuron corresponds to ψ = 0 or ψ = π/ 2 and this term measures the distance between the phase of a complex-valued neuron and a real-valued one. Finally, the lower bound decreases with a rate inversely proportional to the square of hidden size n . We conjecture that this dependence is tight and cannot be improved: a two-layer RVNN with n neurons divides the space into n pieces, in each of which RVNN acts as a linear function. Choosing the n weight vectors of RVNN suitably, the difference between the RVNN and the complex-valued neuron remains small (of order n -1 ) in each piece, which leads to the expected loss of order O ( n -2 ) .

The conditions in Theorem 4 are made for conciseness of proof and we believe the conclusion holds in more general cases. The dimension d = 1 corresponds to the intrinsic dimension of expressing a complex-valued neuron because of the rotational invariance of the inner product and the spherical symmetry of Gaussian distributions. Thus, additional dimensions contain no information and cannot improve the efficiency of approximation when d > 1 . It is necessary to consider non-degenerate complex-valued neurons since degenerate complex-valued neurons are equivalent to real-valued ones.

We provide the central proof idea of Theorem 4 as follows. It is observed that the expected square loss L rc is a piecewise quadratic function and each piece forms a sector centered at the origin with infinite radius. In each piece, L rc takes the form E [( w glyph[latticetop] x ) 2 ] . The proof mainly consists of two steps: we obtain a lower bound of L rc in a sector and then sum over all sectors with suitable weights and order. Firstly, we consider the expected loss in a sector with a small central angle θ , as shown in Fig. 2(a). We divide the sector into three identical subareas A 1 , A 2 , and A 3 . Then at least one of A 1 and A 3 remains θ/ 6 away from the vertical direction of w , which leads to a lower bound Ω( θ 2 ‖ w ‖ 2 ) . Secondly, we consider the loss in four rotationally symmetric sectors, as shown in Fig. 2(b), where v represents a complex-valued neuron, w i indicates a real-valued neuron, and the expression in each sector implies the activated neurons. It is observed that at least one sector possesses a weight vector with norm Ω( ‖ v ‖ ) , no matter how we choose the real-valued neurons. Thus, the overall loss is bounded by Ω( θ 2 ‖ v ‖ 2 ) . Finally, we take the weight α into consideration and sum over all sectors. For RVNN with n neurons, the best choice of θ = Θ( n -1 ) arrives at the lower bound Ω( n -2 ‖ v ‖ 2 ) . Detailed proofs are provided in Appendix D.

Summary and simulation experiments. We summarize the main conclusions of this section in Table 2. Both a real-valued neuron and a complex-valued neuron succeed in learning functions expressed by any one real-valued neuron. But difference occurs when learning those expressed by any non-degenerate complex-valued neuron: A complex-valued neuron can efficiently learn functions expressed by any one complex-valued neuron, but a two-layer RVNN with finite width cannot learn a single non-degenerate complex-valued neuron. Such a disagreement demonstrates that a complex-valued neuron possesses more powerful learning capability, which profits from the consideration of phase information in complex-valued operations. Our theoretical conclusions are based on the setting of low-dimensional inputs and no bias term, and the simulation results in Fig. 3 verify and extend these discoveries in more general settings. Details about the simulation experiments are available in Appendix F.

(a)

d

= 1

and no bias term.

(b)

d

= 5

and with bias terms.

0.3

Real Neuron

Error

0.2

Test

0.1

0.0

0

20

40

60

80

100

Number of Epochs

0.4

Real Neuron

0.3

Error

0.2

Test

0.1

0.0

0

100

200

300

Number of Epochs

5 Complex-valued Neurons Learn Slower

In this section, we demonstrate that complex-valued neurons learn slower than real-valued neurons. To arrive at this conclusion, we first rephrase the linear convergence of learning functions expressed by any one real-valued neuron using real-valued neurons. Then we prove that a complex-valued neuron learns the same class of functions at an exponentially slower rate.

We concentrate on learning one real-valued neuron x → τ ( v glyph[latticetop] x ) with ‖ v ‖ = 1 . When learning one real-valued neuron using a real-valued neuron, the expected square loss in Eq. (1) possesses the following simple closed form [41]

It is widely known that a real-valued neuron learns a real-valued neuron with high probability and linear convergence rate [19], as reformulated by the following lemma.

Lemma 5. [19, Theorem 6.4] Suppose that the weight vector w ∈ R 2 d is initialized by a Gaussian distribution N (0 , I / (2 d )) . Let L rr denote the expected square loss of learning a real-valued neuron using a real-valued neuron. Then there exist constants c 1 , c 2 such that gradient descent with suitable step size satisfies

Ilw-vll

w-

StageI

StageIl

Stage Il

StageIV

t

Recalling the expected loss of learning one real-valued neuron with a complex-valued neuron in Eq. (2), then we present the fourth theorem for complex-valued neuron learning, which provides a lower bound for the convergence rate.

Theorem 6. Let d = 1 . Suppose that ‖ w 0 -v ‖ < 1 . Let { ( w t , ψ t ) } ∞ t =0 denote the parameter sequence of the complex-valued neuron generated by projected gradient descent when optimizing L cr , the expected loss of learning a real-valued neuron using a complex-valued neuron. If the step size η t = η ∈ (0 , 1 / (12 π )) , then we have

where ψ ∗ = π/ 2 , and T 3 is a constant dependent on ‖ w 0 -v ‖ , η , and ψ ∗ -ψ 0 .

Theorem 6 presents a lower bound for the expected loss of learning one real-valued neuron with a complex-valued neuron. It is observed that the negative term in the lower bound becomes 0 exponentially fast as t increases, and the positive term decreases with order Ω( t -3 ) . Thus, the expected loss possesses a lower bound Ω( t -3 ) since the positive term dominates the loss when t grows sufficiently large. This lower bound matches the upper bound in Theorem 1. Thus, O ( t -3 ) becomes the utmost limit of learning with a complex-valued neuron via gradient descent, i.e., we cannot expect a complex-valued neuron to learn faster than this utmost limit.

glyph[negationslash]

The conditions in Theorem 6 are technical and reasonable. The condition on w 0 is made for the conciseness of proof and can be removed. It is observed that ( w , ψ ) = ( v , ψ ∗ ) is the unique global minimum with L cr = 0 . Meanwhile, it is easy to verify that the loss goes to infinity when ‖ w ‖ → ∞ . Thus, if we aim to obtain a small loss, the parameter sequence must fall into a small neighborhood of the global minimum, which is depicted by condition ‖ w 0 -v ‖ glyph[lessorequalslant] R < 1 . The condition ψ 0 = ψ ∗ holds with probability 1 when we initialize ψ 0 with a continuous distribution. This condition is necessary to obtain a meaningful lower bound since the numerator of the positive term equals 0 when ψ 0 = ψ ∗ . We emphasize that this condition is essential and cannot be removed because a complex-valued neuron with ψ 0 = ψ ∗ is equivalent to a real-valued neuron, which enjoys linear convergence as stated in Lemma 5. The condition d = 1 corresponds to the simplest optimization problem of learning one real-valued neuron with a complex-valued neuron since high-dimensional optimization brings more difficulties. Thus, we cannot expect a complex-valued neuron to learn a real-valued neuron with a convergence rate faster than O ( t -3 ) in a higher dimension.

We summarize the proof idea of Theorem 6 as follows. The gradient with respect to ψ possesses the order ( ψ ∗ -ψ ) 2 +( ψ ∗ -ψ ) θ w , v . The key intuition is that ψ converges fast to the global minimum when θ w , v remains large, but θ w , v diminishes as w converges to the global minimum v . The detailed proofs are complicated and consist of several stages, depicting the entangled convergence between w and ψ as shown in Figure 4. In Stage I, ψ increases above a positive constant, which is a necessary condition for fast convergence of w in Stage II. When the distance between w and v declines below a threshold, the angle θ w , v becomes small. Then we enter Stage III, where w converges faster than ψ . Stage IV begins when ψ ∗ -ψ dominates θ w , v . Then the gradient degenerates to order ( ψ ∗ -ψ ) 2 , which implies a lower bound of convergence ψ ∗ -ψ = Ω( t -1 ) . Finally, estimating the loss around the global minimum leads to the conclusion. Detailed proofs are provided in Appendix E.

Summary and simulation experiments. Table 3 summarizes the conclusions in this section, which shows that a complex-valued neuron learns slower than a real-valued one. A complex-valued neuron is more flexible since it can learn the phase. But this flexibility becomes redundant and slows down the convergence when learning a phase-independent function. Our theories are based on the setting of low-dimensional inputs and no bias term, and the simulation results in Fig. 3 verify and extend these discoveries in more general settings. Details about the experiments are available in Appendix F.

(a)

d

= 1

and no bias term.

(b)

d

= 5

and with bias terms.

0.5

RealNeuron

0.4

ComplexNeuron

Error

0.3

t

司

0.2

0.1

0.0

0

20

40

60

80

100

NumberofEpochs

0.5

RealNeuron

0.4

ComplexNeuron

Error

0.3

t

0.2

司

0.1

0.0

0

20

40

60

80

100

NumberofEpochs

6 Conclusions and Prospects

In this paper, we investigate the problem of learning a single neuron using another neuron by optimizing the expected square loss via gradient descent. Firstly, we prove that a complex-valued neuron can efficiently learn functions expressed by any one real-valued neuron and any one complexvalued neuron with convergence rate O ( t -3 ) and O ( t -1 ) , respectively, where t denotes the iteration index of gradient descent. Meanwhile, two-layer RVNNs with finite width cannot learn a single nondegenerate complex-valued neuron in a strong sense that there always exists a positive gap between a two-layer RVNN of fixed width and a non-degenerate complex-valued neuron. These conclusions suggest that complex-valued neurons can learn more than real-valued neurons since CVNNs benefit from the phase parameter, which helps CVNNs learn phase information more efficiently. Secondly, we provide a convergence lower bound Ω( t -3 ) , which matches the upper bound, for learning one real-valued neuron with a complex-valued neuron. This conclusion, together with the well-known linear convergence of learning one real-valued neuron with a real-valued neuron, implies that complexvalued neurons learn slower than real-valued neurons in phase-independent tasks. This phenomenon captures the additional price for learning simpler tasks with more complicated models, where the redundant phase consideration exponentially slows down the convergence.

Our study serves as a preliminary attempt to compare the learning process of artificial neural networks with different functional operations. In the future, it is important to extend our theoretical results to more general settings, such as cases of high-dimensional inputs, equipped with bias terms, and over-parameterized architectures [42]. Meanwhile, it is prospective to investigate complex-valued neuron learning from finite samples and derive a high-probability convergence condition. Since the empirical loss is a piecewise constant function with respect to the learnable phase parameter, it might be necessary to explore new learning algorithms, which is also encouraged by the neural tangent kernel aspect [43]. Besides, it is promising to consider the more practical and challenging procedure of learning general functions with deep architectures.

Acknowledgments

This research was supported by NSFC (61921006) and Collaborative Innovation Center of Novel Software Technology and Industrialization. We would like to thank the anonymous reviewers for their invaluable suggestions.

References

Zhimian Zhang, Haipeng Wang, Feng Xu, and Ya-Qiu Jin. Complex-valued convolutional neural network and its application in polarimetric SAR image classification. IEEE Transactions on Geoscience and Remote Sensing , 55(12):7177-7188, 2017.

Izhak Shafran, Tom Bagby, and RJ Skerry-Ryan. Complex evolution recurrent neural networks (ceRNNs). In Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing , pages 5854-5858, 2018.

Shanshan Wang, Huitao Cheng, Leslie Ying, Taohui Xiao, Ziwen Ke, Hairong Zheng, and Dong Liang. DeepcomplexMRI: Exploiting deep residual network for fast parallel MR imaging with complex convolution. Magnetic Resonance Imaging , 68:136-147, 2020.

Joshua Bassey, Lijun Qian, and Xianfang Li. A survey of complex-valued neural networks. arXiv:2101.12249 , 2021.

ChiYan Lee, Hideyuki Hasegawa, and Shangce Gao. Complex-valued neural networks: A comprehensive survey. IEEE/CAA Journal of Automatica Sinica , 9(8):1406-1426, 2022.

Taehwan Kim and Tülay Adali. Universal approximation of fully complex feed-forward neural networks. In Proceedings of 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing , pages 973-976, 2002.

Felix Voigtlaender. The universal approximation theorem for complex-valued neural networks. Applied and Computational Harmonic Analysis , 64:33-61, 2023.

Paul Geuchen, Thomas Jahn, and Hannes Matt. Universal approximation with complex-valued deep narrow neural networks. arXiv:2305.16910 , 2023.

Bo Zhou and Qiankun Song. Boundedness and complete stability of complex-valued neural networks with time delay. IEEE Transactions on Neural Networks and Learning Systems , 24(8):1227-1238, 2013.

Tohru Nitta. Local minima in hierarchical structures of complex-valued neural networks. Neural Networks , 43:1-7, 2013.

Jin-Hui Wu, Shao-Qun Zhang, Yuan Jiang, and Zhi-Hua Zhou. Theoretical exploration of flexible transmitter model. IEEE Transactions on Neural Networks and Learning Systems , 2023. doi: 10.1109/TNNLS.2022.3195909.

Shao-Qun Zhang, Wei Gao, and Zhi-Hua Zhou. Towards understanding theoretical advantages of complex-reaction networks. Neural Networks , 151:80-93, 2022.

Eran Malach and Shai Shalev-Shwartz. Is deeper better only when shallow is good? In Advances in Neural Information Processing Systems 32 , pages 6426-6435, 2019.

Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. The connection between approximation, depth separation and learnability in neural networks. In Proceedings of the 34th Conference on Learning Theory , pages 3265-3295, 2021.

Peter Auer, Mark Herbster, and Manfred K Warmuth. Exponentially many local minima for single neurons. Advances in Neural Information Processing Systems 8 , pages 316-322, 1995.

Spencer Frei, Yuan Cao, and Quanquan Gu. Agnostic learning of a single neuron with gradient descent. Advances in Neural Information Processing Systems 33 , pages 5417-5428, 2020.

Ilias Diakonikolas, Surbhi Goel, Sushrut Karmalkar, Adam R Klivans, and Mahdi Soltanolkotabi. Approximation schemes for ReLU regression. In Proceedings of the 33rd Annual Conference on Learning Theory , pages 1452-1485, 2020.

Ilias Diakonikolas, Daniel Kane, Lisheng Ren, and Yuxin Sun. SQ lower bounds for learning single neurons with Massart noise. Advances in Neural Information Processing Systems 35 , pages 24006-24018, 2022.

Gilad Yehudai and Ohad Shamir. Learning a single neuron with gradient methods. In Proccedings of the 33rd Conference on Learning Theory , pages 3756-3786, 2020.

Henry Leung and Simon Haykin. The complex backpropagation algorithm. IEEE Transactions on Signal Processing , 39(9):2101-2104, 1991.

Nevio Benvenuto and Francesco Piazza. On the complex backpropagation algorithm. IEEE Transactions on Signal Processing , 40(4):967-969, 1992.

George M Georgiou and Cris Koutsougeras. Complex domain backpropagation. IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing , 39(5):330-334, 1992.

Paolo Arena, Luigi Fortuna, R Re, and Maria Gabriella Xibilia. Multilayer perceptrons to approximate complex valued functions. International Journal of Neural Systems , 6(4):435-446, 1995.

Md Faijul Amin, Ramasamy Savitha, Muhammad Ilias Amin, and Kazuyuki Murase. Complexvalued functional link network design by orthogonal least squares method for function approximation problems. In Proceedings of 2011 International Joint Conference on Neural Networks , pages 1489-1496, 2011.

Akira Hirose and Shotaro Yoshida. Generalization characteristics of complex-valued feedforward neural networks in relation to signal coherence. IEEE Transactions on Neural Networks and Learning Systems , 23(4):541-551, 2012.

Tohru Nitta. Solving the XOR problem and the detection of symmetry using a single complexvalued neuron. Neural Networks , 16(8):1101-1105, 2003.

Shao-Qun Zhang and Zhi-Hua Zhou. Flexible transmitter network. Neural Computation , 33(11):2951-2970, 2021.

Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics , 5(4):115-133, 1943.

Junming Zhang and Yan Wu. A new method for automatic sleep stage classification. IEEE Transactions on Biomedical Circuits and Systems , 11(5):1097-1110, 2017.

Jingkun Gao, Bin Deng, Yuliang Qin, Hongqiang Wang, and Xiang Li. Enhanced radar imaging using a complex-valued convolutional neural network. IEEE Geoscience and Remote Sensing Letters , 16(1):35-39, 2018.

Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, Joao Felipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher J Pal. Deep complex networks. In Proceedings of the 6th International Conference on Learning Representations , 2018.

Adam Tauman Kalai and Ravi Sastry. The isotron algorithm: High-dimensional isotonic regression. In Proceedings of the 22nd Conference on Learning Theory , 2009.

Sham M Kakade, Varun Kanade, Ohad Shamir, and Adam Kalai. Efficient learning of generalized linear and single index models with isotonic regression. In Advances in Neural Information Processing Systems 24 , pages 927-935, 2011.

Surbhi Goel, Varun Kanade, Adam Klivans, and Justin Thaler. Reliably learning the ReLU in polynomial time. In Proceedings of the 30th Conference on Learning Theory , pages 1004-1042, 2017.

Mahdi Soltanolkotabi. Learning ReLUs via gradient descent. In Advances in Neural Information Processing Systems 30 , pages 2007-2017, 2017.

Simon S Du, Jason D Lee, and Yuandong Tian. When is a convolutional filter easy to learn? In Proceedings of the 6th International Conference on Learning Representations , 2018.

Song Mei, Yu Bai, and Andrea Montanari. The landscape of empirical risk for nonconvex losses. The Annals of Statistics , 46(6A):2747-2774, 2018.

Seyed Mohammadreza Mousavi Kalan, Mahdi Soltanolkotabi, and A Salman Avestimehr. Fitting ReLUs via SGD and quantized SGD. In Proceedings of 2019 IEEE International Symposium on Information Theory , pages 2469-2473, 2019.

Nitzan Guberman. On complex valued convolutional neural networks. arXiv:1602.09046 , 2016.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics , pages 249-256, 2010.

Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer ReLU neural networks. In Proceedings of the 35th International Conference on Machine Learning , pages 4433-4441, 2018.

Zhi-Hua Zhou. Why over-parameterization of deep neural networks does not overfit? Science China Information Sciences , 64:1-3, 2021.

Zhi-Hao Tan, Yi Xie, Yuan Jiang, and Zhi-Hua Zhou. Real-valued backpropagation is unsuitable for complex-valued neural networks. Advances in Neural Information Processing Systems 35 , pages 34052-34063, 2022.

Appendix of 'Complex-valued Neurons Can Learn More but Slower than Real-valued Neurons via Gradient Descent'

A Preliminaries

In this section, we first summarize frequently used notations in the following table.

We then give some basic lemmas that help us calculate the closed form of the expected loss.

Lemma 7. Let d = 1 . For any w , v ∈ R 2 d , and a glyph[lessorequalslant] b glyph[lessorequalslant] a +2 π , we have

Proof. According to the probability density function of Gaussian distribution, we can calculate A in the polar coordinate system as

where the second and third equalities hold from integrating over r and φ , respectively. Thus, we have completed the proof.

Lemma 8. Let d = 1 . For any w , v ∈ R 2 d , denote by θ = θ w , v the angle between w and v . Then for any ψ w , ψ v ∈ [0 , π/ 2] , define ψ m = min { ψ w , ψ v } . Then we have

Proof. We only consider the case of ψ w glyph[lessorequalslant] ψ v . The other case ψ w glyph[greaterorequalslant] ψ v can be proven similarly. We prove the conclusion by discussion.

Suppose θ w , v ∈ [0 , ψ v -ψ w ] . Then Lemma 7 leads to

Suppose θ w , v ∈ [ ψ v -ψ w , ψ v + ψ w ] and θ w glyph[lessorequalslant] θ v . Then one knows from Lemma 7 that B ( w , v , ψ w , ψ v ) = A ( w , v , θ v -ψ v , θ w + ψ w )

Suppose θ w , v ∈ [ ψ v -ψ w , ψ v + ψ w ] and θ w glyph[greaterorequalslant] θ v . Based on Lemma 7, we have

Suppose θ w , v ∈ [ ψ v + ψ w , π ] . Then the support of σ ψ w ( w glyph[latticetop] C x C ) does not overlap with that of σ ψ v ( v glyph[latticetop] C x C ) , which leads to B ( w , v , ψ w , ψ v ) = 0 .

Combining the cases above completes the proof.

B Proof of Theorem 1

In the main part of this section, we provide the closed form of the loss, definition of the ideal region, and the detailed proof of Theorem 1. Subsection B.1 presents the optimization behaviors in the ideal region. Subsection B.2 proves several convergence rate lemmas. Subsection B.3 gives some technical lemmas to bound small terms in the proof.

Let w = ( w 1 , w 2 ) . According to the spherical symmetry, we assume v = (1 , 0) without loss of generality. According to Lemma 8, the expected loss can be calculated by

where θ = θ w , v = arccos( w 1 / √ w 2 1 + w 2 2 ) . For any R ∈ (0 , 1) , define

Let D = D 1 ∪ D 2 denote the ideal region, i.e.,

We are now ready to prove Theorem 1.

Proof of Theorem 1. The proof is divided into four steps.

Step 1: D is closed under gradient descent. Before considering the convergence, we prove the maintenance of inclusion by mathematical induction, i.e., ( w 0 , ψ 0 ) ∈ D indicates ( w t , ψ t ) ∈ D .

Base case. The conclusion holds for t = 0 from the condition.

Induction. Suppose that the conclusion holds for t = k with k ∈ N . Then based on Lemmas 11 and 12, one knows

where ψ ∗ = π/ 2 , the first inequality holds based on the induction hypothesis and | w 2 ,k | glyph[lessorequalslant] 1 , and the third inequality holds from R < 1 . Thus, the updating rule ψ k +1 = ψ k -η ∇ ψ L cr ( w k , ψ k ) with η ∈ (0 , 1 / (12 π )) leads to

where the first and fourth inequalities hold from the induction hypothesis. Meanwhile, Lemmas 9 and 10 imply

Combining Eqs. (5) and (6), the conclusion holds for t = k +1 .

Therefore, mathematical induction implies ( w t , ψ t ) ∈ D when ( w 0 , ψ 0 ) ∈ D .

Step 2: parameters converge to the global minimum in D . The convergence process consists of two stages. In stage I, we deal with the convergence of ψ when ( w 0 , ψ 0 ) ∈ D . Based on Eq. (4) and the updating rule ψ k +1 = ψ k -η ∇ ψ L cr ( w k , ψ k ) , one knows

Define a t = η (1 -R 2 )( ψ ∗ -ψ t ) / (4 π ) . Then we obtain a t +1 glyph[lessorequalslant] a t (1 -a t ) . From ψ ∗ -ψ t ∈ [0 , π/ 2] and η < 1 / (12 π ) glyph[lessorequalslant] 4 , one knows a t ∈ [0 , 1 / 2] . Thus, applying Lemma 14 to a t leads to

In stage II, we consider the convergence of w when ( w 0 , ψ 0 ) ∈ D . Based on Eq. (7), choosing T 1 glyph[greaterorequalslant] 16 glyph[ceilingleft] η (1 -R 2 ) glyph[ceilingright] -1 leads to ψ ∗ -ψ t glyph[lessorequalslant] π/ 4 for any t glyph[greaterorequalslant] T 1 , i.e., ψ t glyph[greaterorequalslant] π/ 4 for any t glyph[greaterorequalslant] T 1 . Thus, for any t glyph[greaterorequalslant] T 1 , Eq. (6) indicates

where the first inequality holds from the monotonic increasing of sin( x ) + x and ψ t glyph[greaterorequalslant] π/ 4 , and the second inequality holds because of ‖ w T 1 -v ‖ glyph[lessorequalslant] R < 1 .

Step 3: the loss converges to 0 in D . We estimate the convergence of the expected loss when ( w 0 , ψ 0 ) ∈ D . For any ( w , ψ ) ∈ D , define non-negative quantities ∆ w = ‖ w -v ‖ and ∆ ψ = ψ ∗ -ψ . We provide an upper bound for L cr by discussion.

Suppose ( w , ψ ) ∈ D 1 . Then we have

where the first inequality holds based on sin(2 ψ )+2 ψ = sin(2∆ ψ )+2 ψ ∗ -2∆ ψ glyph[greaterorequalslant] 2 ψ ∗ -2∆ 3 ψ , and the second inequality holds from non-negative ∆ ψ .

Suppose ( w , ψ ) ∈ D 2 . The expected loss can be rewritten as

where the first inequality holds from π glyph[greaterorequalslant] sin(2 ψ ) + 2 ψ glyph[greaterorequalslant] 2 ψ ∗ -2∆ 3 ψ and cos(2 ψ ) -1 glyph[lessorequalslant] 0 , the second inequality holds based on θ glyph[lessorequalslant] tan θ glyph[lessorequalslant] ∆ w and w 1 glyph[lessorequalslant] 1 + ∆ w , and the third inequality holds from ∆ ψ glyph[greaterorequalslant] 0 .

Combining Eqs. (9) and (10), one knows that the following holds for any ( w 0 , ψ 0 ) ∈ D and t glyph[greaterorequalslant] T 1

where the first inequality holds from ∆ 2 glyph[lessorequalslant] ∆

Step 4: initialization falls into D with constant probability. Let p 0 = Pr[( w 0 , ψ 0 ) ∈ D ] for simplicity. From ψ 0 ∼ U (0 , π/ 2) , the requirement ψ ∈ [0 , π/ 2] is satisfied. Denote by p ( w ) the probability density function of N (0 , I ) . Then one has w w , and the second inequality holds by Eqs. (7) and (8). 2

Let R 2 = 1 / 2 . We obtain from Eqs. (11) and (12) that

which completes the proof.

B.1 Optimization Behaviors

The following two lemmas indicate the linear convergence of w in D 1 and D 2 , respectively.

Lemma 9. Let w ′ = w -η ∇ w L cr ( w , ψ ) . If ( w , ψ ) ∈ D 1 and η ∈ (0 , 4) , then we have

Proof. For any ( w , ψ ) ∈ D 1 , one has

Meanwhile,

Then according to Lemma 13 and ψ ∈ [0 , π/ 2] , for any η ∈ (0 , 4) , one has

which completes the proof.

Lemma 10. Let w ′ = w -η ∇ w L cr ( w , ψ ) . If ( w , ψ ) ∈ D 2 and η ∈ (0 , 1 / (12 π )) , then we have

Proof. Firstly, we prove the strong convexity in D 2 . For any ( w , ψ ) ∈ D 2 , one has

where

According to Lemmas 15 and 16, Eq. (13) can be bounded by

Secondly, we provide an upper bound of gradient in D 2 . For any ( w , ψ ) ∈ D 2 , the gradient satisfies

where

From Lemmas 17 and 18, one knows

Finally, based on Eqs. (14) and (15) and Lemma 13, we conclude

where the first inequality holds based on √ 1 -x glyph[lessorequalslant] 1 -x/ 2 for any x ∈ [0 , 1] and η ∈ (0 , 1 / (12 π )) . Thus, we have completed the proof.

The following two lemmas depict the gradient with respect to ψ in D 1 and D 2 , respectively.

Lemma 11. Let ψ ′ = ψ -η ∇ ψ L cr ( w , ψ ) . If ( w , ψ ) ∈ D 1 , then

Proof. For any ( w , ψ ) ∈ D 1 , one has

For any ψ ∈ [0 , π/ 2] , we have 1 2 ( π/ 2 -ψ ) 2 glyph[lessorequalslant] cos(2 ψ ) + 1 glyph[lessorequalslant] 2( π/ 2 -ψ ) 2 . Meanwhile, one has 0 glyph[lessorequalslant] ‖ w t -v ‖ glyph[lessorequalslant] R . Thus, the gradient with respect to ψ can be bounded by

which completes the proof of the lower bound.

Lemma 12. If ( w , ψ ) ∈ D 2 , then

Proof. The gradient of L cr with respect to ψ in D 2 can be calculated by

Firstly, we prove the upper bound for ∇ ψ L cr ( w , ψ ) . It is observed that

where the first inequality holds based on π/ 2 glyph[greaterorequalslant] ψ glyph[greaterorequalslant] π/ 2 -θ glyph[greaterorequalslant] 0 , and the first equality holds from w 1 = r cos θ and | w 2 | = r sin θ . Substituting Eq. (24) into Eq. (16), we obtain

where the second inequality holds according to 1 + cos(2 ψ ) glyph[greaterorequalslant] 1 2 ( π/ 2 -ψ ) 2 for any ψ ∈ [0 , π/ 2] and ‖ w -v ‖ glyph[lessorequalslant] R .

Secondly, we verify the lower bound for ∇ ψ L cr ( w , ψ ) . It is observed that

where the first inequality holds because of [1 + cos(2 ψ )] w 1 glyph[greaterorequalslant] 0 and ‖ w -v ‖ glyph[greaterorequalslant] 0 , the second inequality holds according to 1 + cos(2 ψ ) glyph[lessorequalslant] 2( π/ 2 -ψ ) 2 , and the third inequality holds based on sin(2 ψ ) glyph[lessorequalslant] π -2 ψ for ψ ∈ [0 , π/ 2] . Thus, we have completed the proof.

B.2 Convergence Rate Lemmas

The following lemma provides a sufficient condition for linear convergence of gradient descent. Lemma 13. If there exist two constants c 1 and c 2 such that

Proof. It is observed that

For η ∈ (0 , 2 c 1 /c 2 ) , the coefficient 1 -2 c 1 η + c 2 η 2 is smaller than 1 , which completes the proof. The following lemma gives a sufficient condition for convergence with an inversely proportional rate. Lemma 14. Let { a t } ∞ t =0 ⊂ [0 , 1 / 2] represent a real-valued sequence.

If a t +1 glyph[lessorequalslant] a t (1 -a t ) , then a t glyph[lessorequalslant] 1 t +1 .

If a t +1 glyph[greaterorequalslant] a t (1 -a t ) , then a t glyph[greaterorequalslant] a 0 t +1 .

Proof. We prove the first conclusion by mathematical induction.

Base case. For t = 0 , the conclusion holds from a 0 glyph[lessorequalslant] 1 / 2 glyph[lessorequalslant] 1 .

Induction. Suppose that the conclusion holds for t = k with k ∈ N . Then it is observed that

where the first inequality holds from the induction hypothesis and the monotonicity of x (1 -x ) for x ∈ [0 , 1 / 2] . Thus, the conclusion holds for t = k +1 .

Therefore, mathematical induction completes the proof of the first conclusion.

We proceed to verify the second conclusion by mathematical induction.

Base case. For t = 0 , the conclusion holds from a 0 glyph[greaterorequalslant] a 0 .

Induction. Suppose that the conclusion holds for t = k with k ∈ N . Then one has

where the first inequality holds from the induction hypothesis and the monotonicity of x (1 -x ) for x ∈ [0 , 1 / 2] , and the second inequality holds based on a 0 glyph[lessorequalslant] 1 / 2 . Thus, the conclusion holds for t = k +1 .

Therefore, mathematical induction completes the proof.

B.3 Technical Lemmas

We present upper bounds for some small terms used in the proof.

Proof. Let r = √ w 2 1 + w 2 2 denote the norm of w . Then according to the definition of θ , one has w 1 = r cos θ and | w 2 | = r sin θ . Thus, we can rewrite R 1 as

We provide the upper bound for R 1 by discussion.

Suppose r cos θ -1 glyph[greaterorequalslant] 0 . Based on the definition of D 2 , we have π 2 -ψ -θ glyph[lessorequalslant] 0 . Meanwhile, ψ ∈ [0 , π/ 2] indicates sin(2 ψ ) glyph[greaterorequalslant] 0 . Thus, one knows R 1 glyph[lessorequalslant] 0 .

Suppose r cos θ -1 < 0 . R 1 can be rewritten as

where

If cos θ -r glyph[lessorequalslant] 0 , it is observed that ˜ R glyph[lessorequalslant] 0 because of ψ, θ ∈ [0 , π/ 2] and r cos θ -1 < 0 . If cos θ -r > 0 , then

where the inequality holds since sin(2 ψ ) + 2 ψ is monotonically increasing. The discriminant of f is

where the first inequality holds since cos 2 θ glyph[lessorequalslant] 1 -4 θ 2 /π 2 on [0 , π/ 2] . According to θ ∈ [0 , π/ 2] , one knows ∆( θ ) glyph[lessorequalslant] 0 , which indicates f ( r ) glyph[lessorequalslant] 0 , and thus, ˜ R glyph[lessorequalslant] 0 when cos θ -r glyph[lessorequalslant] 0 . Combining the cases above, we obtain ˜ R glyph[lessorequalslant] 0 , which, together with Eq. (17), implies R 1 glyph[lessorequalslant] 1 2 [sin(2 ψ ) + 2 ψ ](1 -2 r cos θ + r 2 ) .

Combining the cases above, one knows

Proof. Let r = √ w 2 1 + w 2 2 denote the norm of w . Then according to the definition of θ , one has w 1 = r cos θ and | w 2 | = r sin θ . Thus, we can rewrite R 2 as

We provide the upper bound for R 2 by discussion.

Suppose r 2 [1 -cos(2 ψ )] -cos θ glyph[lessorequalslant] 0 . From θ ∈ [0 , π/ 2] , we have R 2 glyph[lessorequalslant] 0 .

, then

Suppose r 2 [1 -cos(2 ψ )] -cos θ > 0 . It is observed that r < 2 cos θ since ‖ w -v ‖ 2 glyph[lessorequalslant] r 2 0 < 1 holds from the definition of D 2 . Thus, the supposition indicates cos θ < r 2 [1 -cos(2 ψ )] < [1 -cos(2 ψ )] cos θ , which, together with θ ∈ [0 , π/ 2] , implies ψ glyph[greaterorequalslant] π/ 4 . It is observed that

which indicates

where the first inequality holds from ψ glyph[greaterorequalslant] π/ 4 , and the third inequality holds because of cos(2 ψ ) glyph[greaterorequalslant] -1 .

Combining the cases above, we obtain

which completes the proof.

∈ D 2 , then we have

Proof. It is observed that T 1 = [ [sin(2 ψ ) + 2 ψ ]( w 1 -1) + T 11 + T 12 ] 2 with

Firstly, denote by r 0 ∈ (0 , 1) a parameter determined later and we calculate an upper bound for T 11 by discussion.

Suppose | w 1 -1 | + | w 2 | glyph[greaterorequalslant] r 0 . Then one has

where the first inequality holds from θ glyph[lessorequalslant] π 2 .

Suppose | w 1 -1 | + | w 2 | glyph[lessorequalslant] r 0 . Then it is observed that w 1 glyph[greaterorequalslant] 1 -r 0 + | w 2 | glyph[greaterorequalslant] 0 . Thus,

where the second inequality holds because of r 0 glyph[lessorequalslant] 1 . Then we can bound | w 2 | from below as

where the second inequality holds since θ glyph[lessorequalslant] 2 sin θ for all θ ∈ [0 , π/ 2] . Meanwhile, we bound θ from above as

where the second inequality holds from w 1 glyph[greaterorequalslant] 1 -r 0 + | w 2 | , and the third inequality holds based on | w 2 | glyph[lessorequalslant] r 0 . Then we obtain an upper bound of T 11 as follows

where the first inequality holds from the monotonicity of 1 2 sin(2 ψ ) + ψ and ψ glyph[lessorequalslant] π 2 , the second inequality holds from Eq. (19), and the third inequality holds based on ψ glyph[greaterorequalslant] π 2 -θ and Eq. (20).

Combining the cases above, we have proven

Choosing r 0 = 1 4 [ π +6 -√ pi 2 +4 π +36 ] , we obtain an upper bound of T 11 as follows

Secondly, we provide an upper bound for T 12 . We claim and prove by discussion that

Suppose w 1 glyph[lessorequalslant] 1 / 2 . Then it is observed that | w 1 -1 | glyph[greaterorequalslant] 1 / 2 , which implies

From the definition of D 2 , one has π 2 glyph[greaterorequalslant] ψ glyph[greaterorequalslant] π 2 -θ glyph[greaterorequalslant] 0 , which indicates

Then we obtain an upper bound of | T 12 | as

where the first inequality holds according to Eq. (22), and the second inequality holds based on Eq. (23). Finally, combining Eqs. (21) and (24), we conclude

where the first inequality holds based on T 11 glyph[greaterorequalslant] 0 and T 12 glyph[lessorequalslant] 0 , and the second inequality holds because of sin(2 ψ ) + 2 ψ glyph[lessorequalslant] π for any ψ ∈ [0 , π/ 2] . Thus, we have completed the proof.

Proof. From cos θ = w 1 / √ w 2 1 + w 2 2 , one has cos( π -2 θ ) = 1 -2 cos 2 θ = 1 -2 w 2 1 / ( w 2 1 + w 2 2 ) . Thus, we have

where the first inequality holds because of | cos a -cos b | glyph[lessorequalslant] | a -b | , and the second inequality holds based on the definition of T 11 in Eq. (18) and sin(2 ψ ) glyph[greaterorequalslant] 0 . Recalling the upper bound of T 11 in Eq. (21), we obtain

which completes the proof.

C Proof of Theorem 2

In the main part of this section, we present the closed form of the loss, definition and properties of the ideal region, and the detailed proof of Theorem 2. Subsection C.1 provides the optimization behaviors. Subsection C.2 gives some convergence rate lemmas.

According to Lemma 8, the expected square loss L cc can be calculated by

(25) For R ∈ (0 , 1) , ψ l ∈ [0 , δ l ] , and ψ u ∈ [ π/ 2 -δ u , π/ 2] , define D 1 = { ( w , ψ w ) | ‖ w -v ‖ ∞ glyph[lessorequalslant] R,ψ w ∈ [ ψ l , ψ u ] , θ w , v ∈ [0 , | ψ w -ψ v | ] } , D 2 = { ( w , ψ w ) | ‖ w -v ‖ ∞ glyph[lessorequalslant] R,ψ w ∈ [ ψ l , ψ u ] , θ w , v ∈ ( | ψ w -ψ v | , ψ w + ψ v ) } . Let D = D 1 ∪ D 2 indicate the ideal region, i.e., D = { ( w , ψ w ) | ‖ w -v ‖ ∞ glyph[lessorequalslant] R,ψ w ∈ [ ψ l , ψ u ] , θ w , v ∈ [0 , ψ w + ψ v ] } .

By spherical symmetry, we assume v = (1 , 0) without loss of generality in the rest proof. For conciseness, define s w = sin(2 ψ w ) + 2 ψ w and s v = sin(2 ψ v ) + 2 ψ v . The following lemma discusses the properties of the ideal region, concerning the closeness of the region under gradient descent and the probability that an initialization falls into this region.

Lemma 19. Let ψ v ∈ [7 π/ 20 , 2 π/ 5] . If we choose the parameters as

then all conditions in Lemmas 20-25 are satisfied. If w 0 ∼ N (0 , I 2 ) and ψ w, 0 ∼ U (0 , π/ 2) , then

Proof. We first prove that all conditions in the lemmas are satisfied.

Lemma 20. It is observed that the first condition holds from

According to ψ u > ψ v > π/ 4 , we have ψ v sin(2 ψ u ) glyph[lessorequalslant] ψ u sin(2 ψ v ) , which implies

where the fourth inequality holds since s w is monotonic. Thus, the second condition is satisfied.

Lemma 21. The first condition η < 2 has been satisfied above. It is observed that ψ l glyph[greaterorequalslant] 7 π/ 20 -109 R/ 100 . Thus, The second condition holds from ψ l / 20 glyph[greaterorequalslant] 7 π/ 400 -109 R/ 2000 glyph[greaterorequalslant] R . The third condition holds since

Lemma 22. The only condition η < 2 has been satisfied.

Lemma 23. The first condition holds because of R = 1 / 25 glyph[lessorequalslant] 1 / 2 . The second condition holds based on cos 2 ψ v glyph[greaterorequalslant] cos 2 (2 π/ 5) glyph[greaterorequalslant] 1 / 25 . The third condition holds from η glyph[lessorequalslant] R/ 120 glyph[lessorequalslant] 3 R/ 2 .

Lemma 24. The first condition R glyph[lessorequalslant] 1 / 2 has been satisfied above. The second and third conditions hold because of

Lemma 25. The first condition R glyph[lessorequalslant] 1 / 2 has been satisfied above. The second one holds from

We then prove the second conclusion. Let p 0 = Pr[( w 0 , ψ w, 0 ) ∈ D ] for simplicity. Then we have

where erf( x ) denotes the error function. Thus, we have completed the proof.

We are now ready to prove Theorem 2.

Proof of Theorem 2. Let R , ψ l , and ψ u be the same as those in Lemma 19. Suppose that ( w 0 , ψ w, 0 ) ∈ D . Then Lemma 19 implies ( w t , ψ w,t ) ∈ D for any t ∈ N . The proof of convergence is divided into several stages.

Step 1: w 2 converges to 0 . In stage I, we consider the convergence of w 2 ,t when ( w 0 , ψ w, 0 ) ∈ D . From Lemmas 22 and 23, the optimization behaviors of w 2 is the combination of minimizing a contraction mapping or an almost absolute function. Thus, Lemma 26 with r 1 = r 2 = R , c 3 = s w / (2 π ) , g l = (cos 2 ψ v -√ 2 R ) / (2 π ) , and g u = 2 / 3 implies

Step 2: ψ w converges to ψ v . In stage II, we prove the convergence of ψ w,t when ( w 0 , ψ w, 0 ) ∈ D . From Lemmas 24 and 25, the convergence of ψ w is limited by that of w 2 , i.e., ψ w tends to the global minimum with constant-order gradient when the error of ψ w is larger than that of w 2 , while becomes far away from the global minimum otherwise. Then Lemma 27 with r 1 = r 2 = 109 R/ 100 , √

Step 3: w 1 converges to 1 . In stage III, we investigate the convergence of w 1 ,t when ( w 0 , ψ w, 0 ) ∈ D . From Lemmas 20 and 21, the gradient points to the global minimum with a remainder controlled by the error of w 1 and ψ w . Then Lemma 28 with d l = 1 / 4 , d u = 1 / 2 , and e = 20 c 2 2 / ( πc 1 ) leads to

Step 3: the expected loss converges to 0 . We now estimate the convergence of the expected square loss when ( w 0 , ψ w, 0 ) ∈ D . For any ( w , ψ w ) ∈ D , define non-negative quantities ∆ w = ‖ w -v ‖ and ∆ ψ = | ψ w -ψ v | . We provide an upper bound for L cc by discussion.

Suppose ( w , ψ w ) ∈ D 1 . Then we have

where the first inequality holds from s w glyph[lessorequalslant] s v + s ∆ , cos θ w , v glyph[greaterorequalslant] √ 1 -∆ 2 w glyph[greaterorequalslant] 1 -∆ 2 w , and s m glyph[greaterorequalslant] s v -s ∆ with s ∆ = 2∆ ψ +sin(2∆ ψ ) , the second inequality holds since |‖ w ‖-‖ v ‖| glyph[lessorequalslant] ∆ 2 w and s ∆ glyph[lessorequalslant] 4∆ ψ , and the third inequality holds based on ‖ w ‖ glyph[lessorequalslant] 2 and s v glyph[lessorequalslant] π .

Suppose ( w , ψ w ) ∈ D 2 . Let θ = θ w , v . Then one knows

Then according to |‖ w ‖ - ‖ v ‖| glyph[lessorequalslant] ∆ w , s w -s v glyph[lessorequalslant] 4∆ ψ , θ glyph[lessorequalslant] arcsin ∆ w glyph[lessorequalslant] 2∆ w , and cos θ glyph[greaterorequalslant] 1 -∆ 2 w , we have

where the second inequality hodls based on ‖ w ‖ glyph[lessorequalslant] 2 , s v glyph[lessorequalslant] π , and ∆ w glyph[lessorequalslant] √ 2 R = √ 2 / 25 . Combining the cases above, one knows from ∆ w glyph[lessorequalslant] 5 / 8 that for any ( w , ψ w ) ∈ D , the loss satisfies

Then based on ( w t , ψ w,t ) ∈ D and Eqs. (26)-(28), we obtain from c 2 glyph[greaterorequalslant] 1 that

which holds with probability at least 10 -5 from Lemma 19. Thus, we have completed the proof.

C.1 Optimization behaviors

The following two lemmas consider the gradient with respect to w 1 in D 1 and D 2 , respectively. Lemma20. Let w 1 = w 1 -η ∇ w 1 L cc ( w , ψ w ) with ( w , ψ w ) ∈ D 1 . If η ∈ (0 , 2) and (1 -R ) s w glyph[lessorequalslant] s v , then we have

Proof. For any ( w , ψ w ) ∈ D 1 , one has

where r denotes a remainder defined by r = 1 2 π [ s w -min { s w , s v } ] . Then Eq. (29) implies

where the first inequality holds from the triangle inequality, and the second inequality holds based on 1 -ηs w / (2 π ) glyph[greaterorequalslant] 0 and | w 1 -1 | glyph[lessorequalslant] R . We proceed to complete the proof by discussion.

Suppose that min { s w , s v } = s w . Then Eq. (30) implies

where the second inequality holds from η > 0 and s w glyph[greaterorequalslant] 0 .

Suppose that min { s w , s v } = s v . Then one knows from Eq. (30) that

where the second inequality holds because of (1 -R ) s w glyph[lessorequalslant] s v .

Combining the cases above completes the proof.

Lemma 21. Let w 1 = w 1 -η ∇ w 1 L cc ( w , ψ w ) with ( w , ψ w ) ∈ D 2 . If η ∈ (0 , 2) , R glyph[lessorequalslant] ψ l / 20 and max { ψ u -ψ v , ψ v -ψ l } glyph[lessorequalslant] 5 Rψ l / 3 , then we have

Proof. For any ( w , ψ w ) ∈ D 2 , the gradient of L cc with respect to w 1 can be calculated by

where r denotes a remainder defined by r = [( s w -s v ) + 2( θ w , v -sin θ w , v )] / (4 π ) . Then we have

where the first inequality holds from the triangle inequality, and the second inequality holds based on η ( s w -θ w , v ) glyph[lessorequalslant] ηs w glyph[lessorequalslant] 2 π and | w 1 -1 | glyph[lessorequalslant] R . It is observed that

where the first inequality holds based on s w glyph[greaterorequalslant] 2 ψ l +sin(2 ψ l ) and sin ψ l glyph[greaterorequalslant] 3 ψ l / 4 for ψ l glyph[lessorequalslant] π/ 4 , and the second inequality holds from θ w , v glyph[lessorequalslant] arcsin R glyph[lessorequalslant] 2 R . Meanwhile, one has

where the first inequality holds from the triangle inequality, and the second inequality holds according to the 4 -Lipschitzness of 2 θ +sin(2 θ ) , θ -sin θ glyph[lessorequalslant] θ 3 / 6 for any θ glyph[greaterorequalslant] 0 , and θ w , v glyph[lessorequalslant] 2 R . Substituting Eqs. (32) and (33) into Eq. (31), we obtain

where the second inequality holds from max { ψ u -ψ v , ψ v -ψ l } glyph[lessorequalslant] 5 Rψ l / 3 and R glyph[lessorequalslant] ψ l / 20 glyph[lessorequalslant] 1 . Thus, we have completed the proof.

The following two lemmas focus on the gradient with respect to w 2 in D 1 and D 2 , respectively. Lemma 22. Let w ′ 2 = w 2 -η ∇ w 2 L cc ( w , ψ w ) with ( w , ψ w ) ∈ D 1 . If η ∈ (0 , 2) , then we have

Proof. For any ( w , ψ w ) ∈ D 1 , one has ∇ w 2 L cc ( w , ψ w ) = s w w 2 2 π . Thus, we have

According to s w ∈ [0 , π ] and η ∈ (0 , 2) , the coefficient 1 -ηs w / (2 π ) is positive and smaller than 1. Based on ( w , ψ w ) ∈ D 1 , one knows | w 2 | glyph[lessorequalslant] R . Then Eq. (34) implies

Lemma 23. Let w ′ 2 = w 2 -η ∇ w 2 L cc ( w , ψ w ) with ( w , ψ w ) ∈ D 2 . If R glyph[lessorequalslant] 1 / 2 , √ 2 R glyph[lessorequalslant] cos 2 ψ v , and η glyph[lessorequalslant] 3 R/ 2 , then we have which completes the proof.

Proof. For any ( w , ψ w ) ∈ D 2 , the gradient of L cc with respect to w 2 can be calculated by

Since ( w , ψ w ) ∈ D 2 , one knows that | w 1 -1 | glyph[lessorequalslant] R and | w 2 | glyph[lessorequalslant] R . Thus, we have

where the first inequality holds because of R ∈ [0 , 1 / 2] . Then we have

where the second inequality holds based on R glyph[lessorequalslant] 1 / 2 . Meanwhile, one has

It is observed that 0 glyph[lessorequalslant] s w | w 2 | glyph[lessorequalslant] π 2 since s w ∈ [0 , π ] and | w 2 | glyph[lessorequalslant] R glyph[lessorequalslant] 1 2 . Then substituting Eqs. (36) and (37) into Eq. (35), we obtain

Thus, one knows from Eq. (35) that

where the first inequality holds from | a -b | glyph[lessorequalslant] max { a, b } for non-negative numbers a and b , and the second inequality holds based on | w 2 | glyph[lessorequalslant] R and η glyph[lessorequalslant] 3 R/ 2 . Thus, we have completed the proof.

The following two lemmas investigate the gradient with respect to ψ w in D 1 and D 2 , respectively.

Lemma24. Let ψ ′ w = ψ w -η ∇ ψ w L cc ( w , ψ w ) with ( w , ψ w ) ∈ D 1 . If R glyph[lessorequalslant] 1 / 2 , η glyph[lessorequalslant] π ( ψ u -ψ v ) / 3 , and η glyph[lessorequalslant] π ( ψ v -ψ l ) / 3 , then we have

Proof. For any ( w , ψ w ) ∈ D 1 , the gradient of L cc with respect to ψ w can be calculated by

where the gradient at ψ w = ψ v can be any subgradient. For any ( w , ψ w ) ∈ D 2 , we have ψ w ∈ [ ψ l , ψ u ] , which indicates 2 cos 2 ψ u glyph[lessorequalslant] 1 + cos(2 ψ w ) glyph[lessorequalslant] 2 . Meanwhile, all points in D 2 satisfies 1 -2 R 2 glyph[lessorequalslant] 1 -‖ w -v ‖ 2 glyph[lessorequalslant] 1 and (1 -R ) 2 glyph[lessorequalslant] ‖ w ‖ 2 glyph[lessorequalslant] (1 + R ) 2 + R 2 . Thus, the gradient of L cc with respect to ψ w can be bounded by

where the first and second inequalities holds based on R glyph[lessorequalslant] 1 / 2 . Then ψ ′ w satisfies

where the first inequality holds from discussing the relation between ψ w and ψ v , and the second inequality holds based on ψ w glyph[lessorequalslant] ψ u and η glyph[lessorequalslant] π ( ψ u -ψ v ) / 3 . Meanwhile, one has

where the first inequality holds from discussing the relation between ψ w and ψ v , and the second inequality holds based on ψ w glyph[greaterorequalslant] ψ l and η glyph[lessorequalslant] π ( ψ v -ψ l ) / 3 . Thus, we have completed the proof. Lemma25. Let ψ ′ w = ψ w -η ∇ ψ w L cc ( w , ψ w ) with ( w , ψ w ) ∈ D 2 . If R glyph[lessorequalslant] 1 / 2 and arcsin R +9 η glyph[lessorequalslant] ψ u -ψ v , then we have

Proof. For any ( w , ψ w ) ∈ D 1 , the gradient of L cc with respect to ψ w can be calculated by

It is observed that the above expression is the same as the gradient of L cr with respect to ψ in Eq. (16). The only difference comes from the domain of w , which is ‖ w -v ‖ glyph[lessorequalslant] R in Lemma 12 and ‖ w -v ‖ ∞ glyph[lessorequalslant] R here. Then according to ‖ x ‖ glyph[lessorequalslant] √ 2 ‖ x ‖ ∞ in R 2 , one knows from Lemma 12 that

where the first inequality holds according to | π/ 2 -ψ w | glyph[lessorequalslant] π/ 2 and | w 2 | glyph[lessorequalslant] 1 , and the third inequality holds based on R glyph[lessorequalslant] 1 / 2 . Then ψ ′ w satisfies

where the second inequality holds from the condition θ w , v glyph[greaterorequalslant] | ψ w -ψ v | in the definition of D 2 , and the third inequality holds according to

Meanwhile, it is observed that the gradient is always negative, which implies ψ ′ w glyph[greaterorequalslant] ψ w glyph[greaterorequalslant] ψ l . Thus, we have completed the proof.

C.2 Convergence Rate Lemmas

This section presents some sufficient conditions for convergence with an inversely proportional rate.

Lemma 26. Let f : K → R represent a function with a global minimum x ∗ , where K ⊂ R indicates the convex domain satisfying B ( x ∗ , r 1 ) ⊂ K ⊂ B ( x ∗ , r 2 ) . Suppose that there exist constants c 1 , c 3 , g l , g u such that c 1 glyph[lessorequalslant] r 1 /g u and for any x ∈ K , at least one of the following holds.

glyph[negationslash]

Then for any c 2 glyph[greaterorequalslant] max { 1 /c 3 , 2 r 2 /g l , 2 c 1 g u /g l } , the sequence { x t } ∞ t =1 generated by gradient descent x t +1 = x t -η t ∇ f ( x t ) with x 0 ∈ K and η t = min { c 1 , c 2 /t } satisfies

Proof. Firstly, we prove x t ∈ K . Suppose x t ∈ K for t = k . We prove x k +1 ∈ K by discussion.

If the first condition holds, then x k +1 is a convex combination of x k and x ∗ . Thus, x k +1 ∈ K .

If the second condition holds and sgn( x k +1 -x ∗ ) = sgn( x k -x ∗ ) , then x k +1 is a convex combination of x k and x ∗ . Thus, x k +1 ∈ K .

glyph[negationslash]

If the third condition holds and sgn( x k +1 -x ∗ ) = sgn( x k -x ∗ ) , then one knows from η t glyph[lessorequalslant] c 1 and |∇ f ( x ) | glyph[lessorequalslant] g u that | x k +1 -x ∗ | glyph[lessorequalslant] c 1 g u glyph[lessorequalslant] r 1 , where the second inequality holds based on c 1 glyph[lessorequalslant] r 1 /g u . Thus, B ( x ∗ , r 1 ) ⊂ K leads to x k +1 ∈ K .

Combining the cases above, x 0 ∈ K and mathematical induction completes the proof of x t ∈ K .

Secondly, we prove | x t -x ∗ | glyph[lessorequalslant] a/t . Let t 0 = c 2 /c 1 . According to c 2 glyph[greaterorequalslant] 2 c 1 g u /g l glyph[greaterorequalslant] 2 c 1 , one knows t 0 glyph[greaterorequalslant] 2 . For t < t 0 , it is observed that

where the first inequality holds based on K ⊂ B ( x ∗ , r 2 ) , the second inequality holds because of a = c 2 2 g l / (2 c 1 ) glyph[greaterorequalslant] r 2 t 0 . Thus, the conclusion holds for any t < t 0 . Suppose that | x k -x ∗ | glyph[lessorequalslant] a/k holds for k glyph[greaterorequalslant] t 0 -1 . We then prove | x k +1 -x ∗ | glyph[lessorequalslant] a/ ( k +1) by discussion.

If the first condition holds, then we have

where the first inequality holds based on the first condition and the induction hypothesis, and the second inequality holds from c 2 glyph[greaterorequalslant] 1 /c 3 . Thus, the conclusion holds for t = k +1 .

If the second condition holds and sgn( x k +1 -x ∗ ) = sgn( x k -x ∗ ) , then one knows

where the first inequality holds from the induction hypothesis and the second condition, and the second inequality holds because of

where the first equality holds based on c 2 glyph[greaterorequalslant] 1 /c 3 , the second equality holds from the choice of a and t 0 , and the first inequality holds from t 0 glyph[greaterorequalslant] 2 and k glyph[greaterorequalslant] t 0 -1 glyph[greaterorequalslant] t 0 / 2 . Thus, the conclusion holds for t = k +1 .

glyph[negationslash]

If the second condition holds and sgn( x k +1 -x ∗ ) = sgn( x k -x ∗ ) , then it is observed that

where the first inequality holds from the second condition, and the second inequality holds based on a = c 2 2 g l / (2 c 1 ) glyph[greaterorequalslant] c 2 g u . Thus, the conclusion holds for t = k +1 .

Combining the cases above, we have completed the proof.

Lemma 27. Let f : K → R represent a function with a global minimum x ∗ , where K ⊂ R indicates the convex domain satisfying B ( x ∗ , r 1 ) ⊂ K ⊂ B ( x ∗ , r 2 ) . Let { θ t } ∞ t =0 be a positive sequence bounded by θ t glyph[lessorequalslant] a/t . Suppose that there exist constants g l , g u such that for any x ∈ K , the following holds

If | x t -x ∗ | glyph[lessorequalslant] θ t , then |∇ f ( x t ) | glyph[lessorequalslant] g u .

Let c 1 > 0 , and c 2 glyph[greaterorequalslant] max { 2 r 2 /g l , 2 c 1 } . Suppose that the sequence { x t } ∞ t =1 generated by gradient descent x t +1 = x t -η t ∇ f ( x t ) with x 0 ∈ K and η t = min { c 1 , c 2 /t } satisfies x t ∈ K for any t ∈ N + . Then the following holds for any t ∈ N +

Proof. Let t 0 = 2 b/ ( c 2 g l ) glyph[greaterorequalslant] c 2 /c 1 glyph[greaterorequalslant] 2 . For any 0 < t < t 0 , it is observed that

Thus, the conclusion holds for 0 < t < t 0 . Suppose that | x k -x ∗ | glyph[lessorequalslant] b/k holds for k glyph[greaterorequalslant] t 0 -1 . We then prove | x k +1 -x ∗ | glyph[lessorequalslant] b/ ( k +1) by discussion.

If the first condition holds and sgn( x k +1 -x ∗ ) = sgn( x k -x ∗ ) , then we have

where the second inequality holds from the induction hypothesis, and the third inequality holds based on b = c 2 g l t 0 / 2 and t 0 / 2 glyph[lessorequalslant] t 0 -1 glyph[lessorequalslant] k . Thus, the conclusion holds for t = k +1 .

glyph[negationslash]

If the first condition holds and sgn( x k +1 -x ∗ ) = sgn( x k -x ∗ ) , then we have

which implies that the conclusion holds for t = k +1 .

If the second condition holds, then one knows

where the second inequality holds based on | x k +1 -x ∗ | glyph[lessorequalslant] θ k +1 glyph[lessorequalslant] a/ ( k + 1) , and the third inequality holds because of b glyph[greaterorequalslant] 2 a + c 2 g u . Thus, the conclusion holds for t = k +1 .

Combining the cases above, we have completed the proof.

Lemma 28. Let f : K → R represent a function with a global minimum x ∗ , where K ⊂ R indicates the convex domain satisfying K ⊂ B ( x ∗ , R ) . Let { x t } ∞ t =1 denote the sequence generated by gradient descent x t +1 = x t -η t ∇ f ( x t ) with x 0 ∈ K and η t = min { c 1 , c 2 /t } , satisfying x t ∈ K for t ∈ N + . Suppose that the gradient satisfies ∇ f ( x t ) = d ( x t -x ∗ ) + r t , where d l glyph[lessorequalslant] d glyph[lessorequalslant] d u and | r t | glyph[lessorequalslant] e/t . If c 1 glyph[lessorequalslant] 1 /d u and c 2 glyph[greaterorequalslant] 2 /d l , then we have

Proof. Let t 0 = c 2 /c 1 . We prove the conclusion by mathematical induction.

Base case. For 0 < t glyph[lessorequalslant] t 0 , it is observed that

Thus, the conclusion holds for 0 < t glyph[lessorequalslant] t 0 .

Induction. Suppose that | x k -x ∗ | glyph[lessorequalslant] c/k holds for k glyph[greaterorequalslant] t 0 -1 . Then we have

where the first inequality holds based on dη k glyph[lessorequalslant] c 1 d u glyph[lessorequalslant] 1 . Then the induction hypothesis leads to

where the first inequality holds according to c 2 d l glyph[greaterorequalslant] 2 , and the second inequality holds based on c glyph[greaterorequalslant] c 2 e . Thus, the conclusion holds for t = k +1 .

Therefore, mathematical induction completes the proof.

D Proof of Theorem 4

We begin the proof with two lemmas. For any non-zero vector a in R 2 and θ ∈ [0 , π ] , define S ( a , θ ) = { x ∈ R 2 | θ x ∈ [ θ a -θ, θ a + θ ] } as the sector region with central angle 2 θ that is symmetric with respect to a . Let N a ,θ represent the truncated standard Gaussian distribution on S ( a , θ ) , of which the probability density function is

The following lemma provides a lower bound for the expected squared inner product on S ( a , θ ) .

Lemma 29. Let d = 1 . For any w ∈ R 2 d , non-zero a ∈ R 2 d , and θ ∈ [0 , π/ 2] , we have

Proof. Let θ w indicate the phase of w , i.e., w = ‖ w ‖ (sin θ w + cos θ w i) . Then calculating the expectation in the polar coordinate system leads to

where the second equality holds based on integrating over r and φ separately, and the identity cos( θ a -θ w ) = cos θ a , w . The expectation in Eq. (38) can be further bounded by

where the first inequality holds according to θ ∈ [0 , π/ 2] , and the second inequality holds because of sin( x ) glyph[lessorequalslant] x -x 3 / 12 for all θ ∈ [0 , π/ 2] . Thus, we have completed the proof.

The following lemma provides a lower bound for expressing a complex-valued vector with four real-valued vectors under a symmetric constant.

Lemma 30. Let v k ∈ R d with k ∈ [4] and v ∈ R d . If v 1 + v 3 = v 2 + v 4 , then we have

Proof. According to the generalized mean inequality, one knows

where the second inequality holds because of the triangle inequality, and the first equality holds based on the condition v 1 + v 3 = v 2 + v 4 . Thus, we have completed the proof.

We are now ready to prove Theorem 4.

Proof of Theorem 4. We define N α , W = ∑ n i =1 α i τ ( w glyph[latticetop] i x ) for simplicity. From d = 1 , the weight vector w i is a 2 -dimensional real-valued vector. Let θ w i = arctan( w -1 i, 1 w i, 2 ) ∈ ( -ψ, 2 π -ψ ] denote the phase of w i . We assume θ v = 0 without loss of generality. Denote by Θ W the π/ 2 -symmetrical phase set induced from W and ψ , i.e.,

It is observed that there is an integer m glyph[lessorequalslant] n +2 such that | Θ W | = 4 m . We sort all phases in Θ W as

Let N β , U represent an arbitrary two-layer RVNN with weight phases from Θ W , i.e.,

It is observed that N β , U degenerates to N α , W with suitable parameters. Thus, the expected square loss L rc can be bounded as

where ∆ θ i = ( θ i -θ i -1 ) / 2 and a i = e ( θ i -∆ θ i )i with θ 0 = θ 4( n +1) . The indices can be divided into m groups as I i = { i +( k -1) m | k ∈ [4] } with i ∈ [ m ] . Denote by i ψ the index of ψ , i.e., θ i ψ = ψ . Then Eq. (39) becomes

where the first inequality holds since ∆ θ j remains the same in I i , the second inequality holds based on the activation regions of ReLU and zReLU, and the definition of v j as follows

Applying Lemma 29 to Eq. (40), we obtain

where the second inequality holds based on the definition of v i,k = v i +( k -1)( n +1) and ∆ θ j = ∆ θ i for any j ∈ I i . Based on Eq. (41), one has v i, 1 + v i, 3 = v i, 2 + v i, 4 . Then Lemma 30 implies

where the second inequality holds based on the generalized mean inequality, and the third one holds from max { 1 , i ψ -m +1 } -min { i ψ , m } glyph[lessorequalslant] m glyph[lessorequalslant] n +2 . Thus, we have completed the proof.

E Proof of Theorem 6

We begin with a lemma providing a lower bound for convergence.

Lemma 31. If there exists a constant c such that

then w ′ = w -η ∇ f ( w ) with η ∈ (0 , 1 / (2 c )) satisfies

Proof. From the updating rule, it is observed that

which completes the proof.

We then prove Theorem 6.

Proof of Theorem 6. Denote by R = ‖ w 0 -v ‖ . The convergence analysis consists of several stages.

Stage 1: the error of ψ decreases below a threshold fast. By the same arguments as those in the proof of Theorem 1, η ∈ (0 , 1 / (12 π )) indicates ( w t , ψ t ) ∈ D for any t ∈ N . Recalling the convergence of ψ in Eq. (7), we have ψ t glyph[greaterorequalslant] π/ 4 when t glyph[greaterorequalslant] glyph[ceilingleft] 16 η -1 (1 -R 2 ) -1 glyph[ceilingright] . From Eq. (4), one knows ∇ ψ L cr ( w t , ψ t ) glyph[greaterorequalslant] -6( ψ ∗ -ψ t ) . Then we have

Then we obtain from η ∈ (0 , 1 / 12) and Lemma 31 that

Thus, one has

Step 2: both errors of w and ψ decrease below small constants fast. Based on Eq. (8), we have

which, together with Eqs. (7) and (42), implies that

Step 3: w converges faster than ψ . For any t glyph[greaterorequalslant] T 2 , Lemmas 11 and 12 imply

where the second inequality holds based on Eq. (44). Then Lemma 31 indicates

which, together with Eq. (43), indicates

Step 4: ψ converges with an inversely proportional rate. For any t glyph[greaterorequalslant] T 3 , it is observed from Lemmas 11, 12, and Eq. (45) that

Let a t = 4 η ( ψ ∗ -ψ t ) . Then the updating rule implies a t +1 glyph[greaterorequalslant] a t (1 -a t ) . Choosing η ∈ (0 , 1 / (4 π )) guarantees a t ∈ [0 , 1 / 2] . Then Lemma 14 indicates

Step 5: the loss converges to 0 with an inversely proportional rate. Define non-negative quantities ∆ w = ‖ w -v ‖ and ∆ ψ = ψ ∗ -ψ . We provide a lower bound for L cr by discussion.

Suppose ( w , ψ ) ∈ D 1 . Then we have

where the first inequality holds based on sin(2 ψ )+2 ψ = sin(2∆ ψ )+2 ψ ∗ -2∆ ψ glyph[lessorequalslant] 2 ψ ∗ -∆ 3 ψ / 2 for any ψ ∈ [0 , π/ 2] , and the second inequality holds from ∆ ψ glyph[lessorequalslant] π/ 2 .

Suppose ( w , ψ ) ∈ D 2 . The expected loss can be rewritten as

where the first inequality holds from sin(2 ψ )+2 ψ glyph[lessorequalslant] 2 ψ ∗ -∆ 3 ψ / 2 and sin(2 ψ )+2 ψ +2 θ -2 ψ ∗ glyph[greaterorequalslant] 0 , the second inequality holds based on cos(2 ψ ) -1 glyph[greaterorequalslant] -2 and | w 2 | glyph[lessorequalslant] ∆ w .

Combining Eqs. (47) and (48), one knows that the following holds for any ( w 0 , ψ 0 ) ∈ D and t glyph[greaterorequalslant] T 3

where the second inequality holds from Eqs. (43) and (46). Thus, we have completed the proof.

F Simulation Experiments

Experimental settings. A training set of size 7,000 and a test set of size 3,000 are generated by a randomly initialized target neuron (can be a real-valued or a complex-valued neuron). After random initialization, a complex-valued neuron and a real-valued neuron are trained by gradient descent with

the empirical mean square loss and a learning rate of 0.1 for 100 epochs (or 300 epochs when the loss does not converge).

Experimental results. It should be noticed that a complex-valued neuron cannot always learn a target neuron. From the theoretical formulation, our convergence rate holds with a small constant probability. From the loss landscape, there exist constant pieces in the parameter space, i.e., the complex-valued neuron does not learn anything after initialization. Thus, we cannot expect a complex-valued neuron to learn a target neuron all the time. In the experiments, we train the complex-valued neuron with several random initializations and find that our theoretical conclusions occur in experiments. This phenomenon verifies our theories and also motivates a novel learning algorithm for CVNNs, as discussed in the conclusion part.

Peer reviews:
comments: SUMMARY:
The authors presents theoretical results on learning real-valued and complex-valued neurons with gradient descent. The key takeaways are that:
* complex-valued neuron learns real-values neurons and complex-valued neurons with convergence rate $O(t^{-3})$ and $O(t^{-1})$ respectively
* two-layer real-valued neural network cannot learn a single complex-valued neuron
* complex-valued neuron learns slower than real-valued neurons when learning real-value neurons

STRENGTHS:
* The paper is well-written and easy to understand even for non-experts (myself)
* This appears to be a significant advance in the theory of complex-valued neuron learning, with previous work largely focusing on real-valued neuron learning. However, as someone outside this field, I defer to other reviewers' for their assessment of significance.

WEAKNESSES:
* While the main results in this paper are theoretical proofs, the paper could be furthered strengthed with empirical experiments that validate the proofs.
* table 2 appears to containa typo? the result for theorem 2 should be $O(t^{-1})$?

QUESTIONS:
* Do the authors expect the convergence rates to be further improved with other learning rate schedules/learning algorithms (other than projected gradient descent)?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe goal is to explore the novel approach of complex valued neural networks using gradient descent. The paper investigates when and to what extent CVNNs outperform RVNNs using gradient descent for learning tasks. The researchers prove that a single complex-valued neuron can efficiently learn functions expressed by one real-valued neuron and those expressed by one complex-valued neuron, while a two-layer RVNN with finite width cannot learn a single complex-valued neuron. However, they also show that complex-valued neurons learn more slowly than real-valued neurons, with a lower bound of Ω(t−3) for learning functions expressed by one real-valued neuron using a complex-valued neuron. Theoretical comparisons between RVNNs and CVNNs provide insights into the success of CVNNs and the potential slow convergence of CVNNs learning.\n\nSTRENGTHS:\nThe authors appear to have thoroughly analysed a complex valued neural network and developed what appears to be rigorous theoretical results calculating the convergence rates of these complex valued neural networks. \n\nThe ideas appear to be sound\n\nWEAKNESSES:\n- While the notation appears to be self-consistent (in the main body - I did not read the appendix, which is very long and technical), the notation is very hard to parse through. If there were some tables to layout the important notation it could help lay the groundwork earlier on.\n\n- I am not familiar with these types of reformulations, but it is not clear to me why the authors want to try to use a two layer network to model a single neuron. The authors write \"Although learning one neuron is the simplest case of learning neural networks, it is sufficient to embody the learning capability and efficiency of neural networks.\", which I ask the authors to motivate more. Is it not obvious a two layer network with more parameters would be able to model a single neuron of just the weighted inputs?\n\n- I am not sure I fully understand this paper but it seems the authors only develop the proofs and theorem 1 and theorem 2 for 1d input such that they calculate the convergence rates for a two layer network to model a linear function wx where w is a vector length n and x is just a scaler number. If I understand this correctly, I am not sure how much we learn from this? Does this theorem apply for as real neuron where d is not one? \n\n-The ideas appear to be sound but they are not very palatable to a reader.\n\nQUESTIONS:\n- Motivation of application - I’m surprised the authors do not mention more applications to systems in the world of physics. Fluid dynamics, beam physics, particle physics, where phase information is ubiquitously relevant to quantum materials, dynamics, and \n\n-States that the CVNNs have shown to have some desirable properties - universal approximation - boundedness, stability → why is this desirable? Could the authors elaborate on this? \n\n-  It is not clear to me, in Theorem 1 and theorem 2 when d=1, does this mean that x is now no longer a linear summation of inputs but truly just an input and the weight parameter? I did not work through the proof, but does the theorem generalize to inputs with more than one dimension?\n\nLIMITATIONS:\nIt often feels like one cannot truly understand what they are discussing in the paper without referencing the very dense and long appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors contrast the learning and convergence properties of real-valued neural networks and complex-valued neural networks. Specifically, they study the problem of learning the function implemented by a single neuron using a 2-layer finite width network. Notably, they show that a complex valued neural network can learn functions expressed by a real-valued neuron as well as a complex-valued neuron. This result indicates the superior capacity of complex-valued neural networks as compared to their real-valued counterparts. Furthermore, they also study the convergence properties of learning a function expressed by a real-valued neuron with a complex-valued neuron and prove that its slower compared to learning with a real-valued neural network. Taken together, complex-valued neural networks have a better learning capacity at the cost of slower learning properties compared to real-valued neural networks. Overall, I think this paper makes a strong theoretical contribution to understanding the efficiency and capacity of complex-values neural networks in practice.\n\nSTRENGTHS:\n1. The paper has strong theoretical foundations and presents arguments through rigorous proofs. \n2. Despite the involved mathematical machinery, the authors present a simplified view of the underlying learning dynamics and present the existence of phases in the learning process. \n3. Despite the assumption of the MSE loss, I feel the authors present a strong theoretical framework to analyze learning in complex-valued neural networks and contrast them with real-valued neural networks.\n\nWEAKNESSES:\n1. The authors present strong theoretical results, and use specific assumptions to prove their results. However, it is unclear the extent to which these assumptions would hold in practice. \n2. In addition to their (very impressive) theoretical results, I feel the current version of the paper would benefit from some empirical validation. Having some toy experiments would also be sufficient to significantly improve the readability of the paper, while also increasing the reader pool. \n3. The manuscript is generally well-written, but there are certain typos or use of abbreviations/notations which make it confusing for the reader. E.g. it seems that there is a typo in Table 2, wherein the second row third column (complex-valued Neuron --> complex-valued neuron) should be $\\mathcal{O}(t^{-1})$. Also, abbreviations CVNN and RVNN or the notation $t$ is not introduced in the abstract.\n\nQUESTIONS:\n1. The caption of Fig. 2 is currently not very descriptive. Could you please update it such that the takeaway is clear to the reader without reading in detail the text?\n2. Is it possible to add a toy regression example where you can empirically demonstrate the different phases during the learning process?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper develops theoretical results for the convergence rates of complex and real valued neural networks on complex-valued problems. It develops analytical models for the training regimes of complex valued NNs.\n\nSTRENGTHS:\nThe results would be useful for those attempting to apply complex-valued networks to problem settings where they are not normally used. An intriguing result is that complex valued neurons have slower convergence rates for the same problems as real valued neurons. The theorems are thorough, and the diagrams provide excellent illustration to the proofs. They also provide a proof that real valued networks cannot learn complex valued functions, which is not surprising, but nice to have proved.\n\nWEAKNESSES:\nA fundamental flaw of the paper is that it does not have empirical verification of the theoretical results. The analysis makes many assumptions about different training regimes, but it is not obvious if complex valued neural networks actually behave like this. The complex-valued models in the theorems should be implemented and the convergence losses should be plotted alongside the theoretical predictions. Without supporting empirical evidence, the assumptions of the proof seem too strong. Fortunately, the problem settings should be simple to code up in any package, and the results would be easy to include in the author's presentation, given Figures 1 and 3.\n\nPage 4, line 156: “whereas a complex-valued neuron may only activate a small part as controlled by the parameter psi” This line makes it sound like the authors conclude that the primary benefit is the extra parameter in zReLU, not the complex arithmetic. The use of activation functions with trainable parameters is also a major distinction between the CVNN and the RVNN. Maybe all of the identified differences can come from a network with real weights, but an activation function with a learnable parameter? zReLU can be encoded in real-valued weights, if you interpret two real weights and use a learnable ReLU. Two variations are required to ablate this confounder:\n1. complex weights with a zReLU with a fixed psi (Would this be easier to study theoretically than the existing proofs? It would also be easy to demonstrate empirically.)\n2. real weights with a pseudo-zReLU with a learnable psi (Possibly hard to prove, but easy to empirically study.)\nI think considering case (1) is necessary to support the claim made by the paper that complex weights are the key differentiator.\n\nQUESTIONS:\n- In Section 4.3, could you clarify how the outputs of the RVNN are matched to the outputs of the complex valued target function?\n- In Section 5, does the trainable zReLU make the network effectively deeper? Does the convergence rate change if psi is not trainable?\n- Define the schema of $L_{rr}, L_{cr},$ etc., and remind the reader every time you use them. By the time I got to Lemma 4 and Lemma 5, I did not know which result was which.\n- $t4 was never defined.\n- Define rvnn and cvnn in the abstract\n- Add equation numbers.\n\nLIMITATIONS:\nA limitations section is missing from the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe theoretical paper compares the learnability of real-valued neurons and complex-valued neurons via gradient descent. In a nutshell, the paper proves that a complex-valued neuron has a lower convergence rate than its real-valued counterpart, but learn exponentially slower. The theoretical discoveries help explain the success of CVNNs and point out the potential slow convergence of CVNNs learning.\n\nSTRENGTHS:\n1. The paper makes the first attempt ever to comparing the learnability of real-valued and complex-valued neurons on a theoretical level.\n\n2. Even though the theoretical investigations are carried out on a single neuron, the findings still cast light on the success and failure of CVNNs.\n\n3. The discoveries are presented as Theorems and proved with rigor in the appendix, in which I did not find any technical errors.\n\n4. The paper is very well written. The implications of each Theorem are clearly explained and summarized in tables.\n\nWEAKNESSES:\n1.  The analysis is carried out on a simple type of complex neuron network, and it is hard to generalize the achieved findings to practical RVNNs and CVNNs.\n\n2. The discoveries could be more convincing by comparing the learning curves of RVNN and CVNN on toy examples, e.g. to show that a real neuron cannot well learn a complex neuron.\n\n3. The research questions should be reformulated. Currently they are quite ambiguous and do not directly reflect the research focus.\n\nQUESTIONS:\n1. It is argued in the paper that learning one neuron can embody the learning of neural networks. However, unlike RVNNs, multi-layered CVNNs may have intermediate layers with complex-valued input and outputs, a different setup from this study. Is it possible to employ a similar approach to this paper for studying these layers?\n\n2. The bias term is removed for technical reasons. Could you please elaborate on how the presence of bias term will influence the analysis, and whether similar findings can still be obtained?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-18"}