Real-Valued Backpropagation is Unsuitable for Complex-Valued Neural Networks
Authors: Zhi-Hao Tan
Yi Xie
Yuan Jiang
Zhi-Hua Zhou
Venue: NeurIPS 2022 Accept (2022)
Keywords: complex-valued neural network
complex backpropagation
neural tangent kernel

Abstract: Recently complex-valued neural networks have received increasing attention due to successful applications in various tasks and the potential advantages of better theoretical properties and richer representational capacity. However, the training dynamics of complex networks compared to real networks remains an open problem. In this paper, we investigate the dynamics of deep complex networks during real-valued backpropagation in the infinite-width limit via neural tangent kernel (NTK). We first extend the Tensor Program to the complex domain, to show that the dynamics of any basic complex network architecture is governed by its NTK under real-valued backpropagation. Then we propose a way to investigate the comparison of training dynamics between complex and real networks by studying their NTKs. As a result, we surprisingly prove that for most complex activation functions, the commonly used real-valued backpropagation reduces the training dynamics of complex networks to that of ordinary real networks as the widths tend to infinity, thus eliminating the characteristics of complex-valued neural networks. Finally, the experiments validate our theoretical findings numerically.

Real-Valued Backpropagation is Unsuitable for Complex-Valueee Neural Networks
Zhi-Hao Tan, Yi Xie, Yuan Jiang, Zhi-Hua Zhou National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China {tanzh, xiey, jiangy, zhouzh @lladda nnju edu.
Abstract
Recently complex-valued neural networks have received increasing attention due successful applications in various tasks and the potential advantages of better theoretical properties and richer representational capacity. However, the training dynamics of complex networks compared to real networks remains an open prob- lem. In this paper, we investigate the dynamics of deep complex networks during real-valued backpropagation
the sefinite-witth limit via neural tangent kernel (NTK). We first extend the Tensor Program to the complex domain, to show that the dynamics of any basic complex network architecture governed by its NTK under real-valued backpropagation. 'hen we propose way to investigate the comparison of training dynamics between complex and real networks by studying their NTKs. As result, we surprisingly prove that for most complex activation functions, the commonly used real-valued backpropagation reduces the training dynamics of complex networks to that of ordinary real networks the widths tend to infinity, thus eliminating the characteristics of complex-valuee neural networks. Finally, the experiments validate our theoretical findings numerically.
Introduction
Recently complex-valued neural networks have been successfully applied to various tasks, such as time-series prediction [Wisdom et al.] 2016]. computer vision (Trabelsi et al.] 2018), signal
processing Yao al.) (2020). Compared to real-valued neural networks, it is shown that complex networks have the potential to provide richer representational capacity [Arjovsky et al. 2016). faster
learning (Danihelka al. 2016), better motivation and generalization for signal-related tasks (Hirose)
and Yoshidal 2012 Tygert et al., 2016). Theoretically, there have been significant advances for complex networks regarding the universal approximation property [Voigtlaender 2020), critical
points NNitta 2002], local minima [Nitaa] 2013] and separation results [Zhang et al. 2022
However, training deep complex networks has been challenging because of several non-intuitive analytical properties of complex algebra. Firstly, in practice, we often deal with a real-valued cost function, which is non-analytic with respect to complex valued parameters. Secondly, Liouville's theorem asserts that every bounded and complex-differente diffferentiable function a constant. Thus almost all activation functions of ReLU [Scardapane et al. 2018). Moreover, Voigtlaender [2020) has proved that the universal approximation property holds only when using non-holomorphic activation functions.
analytic due to the preference for boundedness before the popularity
Due to these reasons, different backpropagation algorithms in the complex domain were independently proposed for non-holomorphic networks [Bassey et al.) [202]], mostly by optimizing the real and imaginary components separately. Recently, real-valued backpropagation [Nitta] 1997] widely
used due to the convenience of utilizing the real-valued deep learning library AArjossky et al. 2016
Trabelsi et al. 2018 Tan et al. 2020). optimized the complex network just like a real network by
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

computing partial derivatives of the cost with respect to the real and imaginary parts separately and achieved state-of-the-ar performance.
As a result, a natural and fundamental problem in complex-valued neural networks arises: Do complex networks have different inductive bias from real networks during training? Do complex networks trained by gradient descent tend to learn different hypotheses from real networks? However, to the best of the authors' knowledge, the training dynamics of backpropagation for complex networks compared to real networks remains open.
In this paper, we investigate the training dynamics of deep complex networks under real-valued backpropagation from neural tangent kernel (NTK) perspective [Jacot et al., 2018). which captures the optimization behavior of neural networks in the infinite-width limit. Then we provide way to investigate the comparison of training dynamics between complex and real networks via their NTKs. As result, we obtain informative results which may guide the algorithm selection for complex networks in practice if people want to take full advantage of complex networks.
Our contributions. Our main contributions can be summarized as follows:
First, we extend the Tensor Program Yang 2020 to the complex domain and show that for a complex- valued neural network of any basic architecture in the infinite-widtl limit, the training dynamics of real-valued backpropagation is determined by kernel gradient descent with a deterministic NTK at initialization.
Second, investigate the comparison of training dynamics between complex and real networks based on their NTKs. We surprisingly prove that the commonly used real-valued backpropagation reduces the training dynamics of complex-valued multi-layer perceptrons (MLPs) to that of ordinary real MLPs as the widths tend to infinity, thus eliminating the characteristics of complex-valus used complex activation functions, including all split activation functions such as CReLU, CSigmoid, Ctanh; and part of magnitude- based holomorphic activation functions.
neural networks. This result holds for most commonly
Finally, we study the results numerically, and the experiments verifies our findings. Specifi- cally, in several settings with different depths and various activation functions, the NTKs of complex networks converge to the NTKs of real networks the widths grow.
Organization. We start with some preliminaries and notations about complex networks and neural tangent kernels Section 2 In Section we investigate the NTK of complex-valued neural networks of any architecture during real-valued backpropagation in the infinite- width limit. Section 4 firstly presents the NTK of complex MLPs in any depth and then investigates the conditions that training dynamics of complex-valus empirically Section 5
MLPs reduce to that of ordinary real MLPs. We verify our results Finally we discuss the related works and conclude the paper. Due to the limited space, all proofs are placed in the appendices.
2 Preliminaries
2.1 Complex-Valie Valued Neural Networks
Without loss of generality, we focus on complex-valued neural networks with real-valued output fo(z) E [Rdo with parameter set 0 E CP, input = E Cd and = x ++ yi with æ, y E RR,, and trained by real-valued backpropagation algorithm (also called complex-
generalized complex BP, Appendix A for more details), which is conventional in the literature [Arjovsky et al., 2016 Wisdom ct al. |016 Trabelsi et al. 2018 Zhang and Zhou, 202T Wu et al.) [202] Zhang et al., [022]. Our analysis can be naturally applied to complex-valee valued output by decomposing the real and imaginary part of output into two functions, or applied real- real-valued input by treating its imaginary part as zero
For an L-hidden layer complex network, we denote the output of last hidden layer as h2 E C". Without loss of generality, consider that the output of complex network with linear readout layer is achieved via
f0(2) R L+ h
where W L+1 E Cuxddou Wisdom et al., 2016 Zhang et al. 2022]. Note that there are other two common forms of linear readout layer generate real-valued output, like fo(z) W R(h

---

R (hr.) 3 (hr.)
where the output weight WL+1 E R Ruxdous
Wu al. 2021 and fo(z) W L+1
where W WL+1 L+1 E 2nxdout Arjovsky et al.
2016]. They can treated as special cases of our settings.
For complex-valued neural network fz)),, we can always decompose all the complex operations into two-dimensional real-valued operations, denoted as f, y)) where Oz, 0y E IR? are the real and imaginary parts of all complex parameters respectively. However, it would be erroneous to assume that a complex network is equivalent to an ordinary real-valued neural network, because the operation of complex multiplication limits the degree of freedom [Hrose and Yoshida 2012).
2.2 Neural Tangent Kernel
For real-valued deep neural network (x) E Rdduu with parameter set 0, E I2 and input æ E R its Neural Tangent Kernel (NTK) under gradient descent is defined as
θ, (xx vV (a, So, (æ'))
which quantifies the functional gradient descent when taking an infinitely small gradient step on new observation. In case fe. corresponds an infinite width MLP, Pacot al. [2018) showed that G (x, æ') converges to a limiting kernel é, (æx x') at initialization and remains frozen during training, i.e.,
lim θ (x, (x,x²) = lim O (xx x²) = Ô, (a; x') 100000
V training time
n=+00
which could give an accurate description of training dynamics with kernel gradient descent trajectory. Thus infinitely wide neural network is governed by a linear model based on its first order Taylor expansion in the parameter space
[Lee et al. 2019]
Tensor Programs. After the original NTK was derived from multi-layer perceptrons, it is soon extended to a variety of network structures including convolution neural networks (CNTK) [Arora et al. 2. recurrent neural networks (RNTK) Yang) 2019 Alemohammad et al. [020]. I graph neural networks (GNTK) [Du et al. 2019] and so on. Importantly, Yang [2020). Yang and Littwin 2021 propose NETSOR program, basic form in Tensor Programs series, and prove that for real- valued neural network of any architecture that can be represented by NETSOR program language, its NTK converges a deterministic limit and stays frozen during training in the infinite-widee width limit.
Neural Tangent Kernel of complex networks. For a complex cemplex-vvlluee valued neural network fo(z), also denoted as 51 ([æ. y)), when it is trained by real-valued backpropagation, the empirical neural tangent kernel as follows due to that the real and imaginary parts are optimized separately
θ (z, x²) (Vofo(z), Vofo (z')) (Von (2), Jo (z')) + oVo, (z), Vo, fo, (z')).
We can always rewrite the NTK as O[[x, y) [x', y')). Note that at each layer of complex networks in both ferw-forward and backward procedure, complex matrix multiplication structure leads to numerous interactions and weight sharing between the real and imaginary parts, which makes the analysis of the complex NTK challenging.
3 Complex Tensor Program
In this section, we show that complex-vai -valued neural networks of any basic architecture also have NTK behavior in the infinite-width limit: the training dynamics of real-valued backpropagation is determined by kernel gradient descent with its NTK at initialization. Specifically, we extend the simplified NETSOR program Yang 2020 to the complex domain, and propose the basic Complex Tensor Program (CTP). Note that the complex networks are mostly non non-holomorphic, thus we do not require the complex tensor program to represent the backward propagation of the complex networks.
Definition (Complex Tensor Program) Given an initial set of random C" vectors and a set W of random C complex matrices, a sequence of C" vectors is called a complex tensor program if they are recursively generated through one of the following ways:
ConNonlin Given & Ck - C and z1 z1,.... t E C", generate ((2),.... ((2,,,..,,7
xt) E C.;
ComMatMul
Given W E C and z E C", generate Wz C
3

---

Obviously, the complex tensor program could represent the forward procedures of all basic complex network architectures, such as the generic reed-forward full-connected complex networks [Nitta] 2004], the complex-valued recurrent neural networks WWsdomm et al. 2016 and complex-valued
convolutional neural networks (Trabelsi et al. 2018 lan et al. 2020
3.1 Complex network setup
This subsection introduces the settings and assumptions of complex networks considered. Firstly we introduce the required assumption for activation functions of complex networks, which generalizes the assumption in Yang [2200] to the complex domain.
Assumption 2
We assume that the complex activation function o C* - C used in the complex i.e., 6(z) satisfies that [(())] S C|| +
networks and its derivative are polynomially-ouun for some C,p.c. e > and z E Ch; so its derivative.
It is worth mentioning that numerous activation functions have been proposed to deal with complex- valued representations and basically they satisfy the assumption. For example, the most commonly used complex sigmoid function CSigmoid [Nitta] 1997, 2004) and complex hyperbolic tangent function Ctanh [Nittal 2002. which apply sigmoid and hyperbolic tangent function respectively to the real part and imaginary part separately; Moreover, the recently proposed ReLUUbased complex activation functions like CReLU (Trabelsi et al. [018]] Tan al. 2020). zReLU [Guberman 2016
and modReLU [Arjovsky et al.) 2016] also satisfy the assumption.
Complex NTK parametrization. For the complex weights, we initialize each W W with W = A + Bi = 24A + vi Bi where Aug, Bas ~ A 1), which we refer to as complex NTK parametrization. Without loss of generality, all layers as σд and Øg respectively in the following paper. It naturally extends the real-valued NTK parametrization [Jacot et al. 2018 Lee et al.) [019]] to complex parameters. Note that this parametrization is non-vacuous because many previous works [Nitta 1997 2004 choose to initialize the real and imaginary parts separately.
set the variances of real and imaginary parts of
Setup. Consider a complex-vali -valued neural network fa(z) with complex NTK parametrization, its feed-forward procedure can be represented by a complex tensor program and complex activation functions all satisfy Assumption 2 Suppose that there is a multivariate Gaussian Nú defined on RaI such that the real and imaginary variables of the initial set of vectors V are sampled like (3(g). 4 E V) U (3)z)). 7 V} ~ My i.i.d. for each coordinate E [n]. For the output readout matrix WL+1. we also adopt complex NTK parametrization, and is sampled independently from all other parameters and is not used anywhere else in the interior of the network. Without loss of generality, the network is trained by SGD with batch-size I and learning rate 1.
3.2 NTK for any complex network
Theorem (Complex NTK at initialization) Consider a complex-valued neural network fe(z) with above setup, then as its widths go to infinity, its NTK ((2 z') initialization converges almost surely a deterministic limiting kernel ((2,z') over any finite set of inputs.
Corollary 4 (Complex NTK during training) Consider training a complex-valued neural network fo(z) with above setup. At training time denote the input sample as 2, and the loss function as : R. Suppose Le is continuous for all t. Then as widths approach infinity, for any Cd and training time fz(z) converges almost surely to a random variable fe(2) and
fz)
- (z, (Je (z+))
(2)
where è (z, 21) is the limiting NTK of the complex network at initialization.
The proofs Theorem and Corollary are given Appendix B. Theorem 3 and Corollary 4 show that for complex-valued neural network of any architecture trained by real-valued backpropagation, its NTK at initialization converges to a deterministic limiting kernel in the infinite-width limit, and the training dynamics is determined by kernel gradient descent with the NTK at initialization.

---

4 Comparison of training dynamics between complex and real networks
In this section, we focus on the important problem: when will complex networks have different inductive bias during training from real networks? The problem can not be solved unless the training dynamics of complex networks could be captured. Based the NTK theory obtained in Section 3 we could provide a way to compare the training dynamics of complex and real networks by comparing their NTKs. Specifically, we have derived the NTK formula of complex multi-layer perceptrons (MLPs) and investigated the conditions under which complex MLPs trained by real- valued backpropagation will reduce ordinary real MLPs in the mnfiite-widht limit.
4.1 Neural Tangent Kernel of complex multi-layer perceptrons
This subsection presents the NTK formula of the most generic complex network, i.e., the L-hidden layer complex MLPs.
The network performs the following computation at layer / E [1
h + rşi = o( (W) = o ((A, + ( (1)))((++++-)))) +r-))))
(3)
where W, A, + Byi is the complex weight matrix and h2 = 8: + ryi with h, E C is the output of 1-th layer. For the first layer, we set 80 and ro y where
yi and Cd Besides, the
output of the complex network with a linear read-out layer is achieved via fo(x) = Re [e[ L+1ht)
where WL+1 Coxdowe Suppose all activation functions & satisfy the Assumption Complex NTK parametrization is applied for all complex parameters W1 Cd WL+1 Wi E C for E [2, L].
xdaut and
We denote the real part of the 1-th hidden layer pre-activation as (z) and the imaginary part as (2)... In the feed-forward procedure, we denote the covariance kernel functions between the real and imaginary part of the pre-activations respectively as
E (z,x')
E a~N
[x)())
au (z') [
Ed,o (z, x²) = E az)()) (1 /n] g~
(4)
(z,z')
E 8~N
[B
B, (z') /n]
E/a (2,2")=
E p~N
[3) a (z') /n]
(5)
In the backward procedure, we denote the gradient vector of the real part of the 1-th hidden layer pre-activation as (z) == v (Vor(=)fo(z)) and that of the imaginary part as (z) =
Similarly, we denote the covariance kernel functions of the gradient vector between the real and imaginary part of the pre-activations respectively as
vn (x),(())
11 (z, 2') I''s (2,2')=
IE
[z)()) (z))//)) [[(())))
I1's (z,z')
E
16 (z) (z')/n] (6)
a~Ar
2~N
T. 8~N (3)(2) (=4)//m)
TT (z,2))
T. ana
())(2) 5% (z')/nn) (7)
Theorem For L-hhideen layer complex MLP, with all activation cociiioofnnciii functions satisfying Assumption in the limit all widths n. + co, the empirical NTK at initialization converges to the following limiting kernel
lim (z,2²) = ô (z,xz)= = θ (z, x2) 2 Idout n-000
(8)
where
θ (z,z²))=
Σ (u (z, DE' (z, x²) + 11ls (z, x') 2')2'' 12'2((1 (z,
(9)
+IT (z, Z')E% 0 2') + a (z, z') 2/)E% 2// (z, 2')) + E (2,
(10)
functioon where the covariance functions Σ/ Dj II',, II's are defined in Eq.
The result is proved in the Appendix C, where the detailed recursions of intermediate kernels for the NTK calculation are also presented.
Note that the NTK formula of a complex MLP looks very different from the NTK of a real MLP given by Jacot et al. 2018 due to the existence of interaction between real and imaginary parts, which is caused by joint weight sharing in complex matrix multiplication. However, we go deeper, does there exist situations that the NTK of complex MLP will reduce to that of real MLP?
5

---

4.2 Asymptotic equivalence of training dynamics
In this subsection, we provide our main results. Surprisingly, we show that, for commonly used complex activation functions, complex networks trained by real-valued backpropagation have the same inductive bias as real networks during training the infinite-width limit.
We first define asymptotic equivalence between neural networks, which represents a perspective to investigate when will complex network have different inductive bias from a real network during training. It also helps if we change the network structure or backpropagation algorithms.
Definition 6 Two neural networks trained by gradient descent are asymptotic equivalent if as all widths go infinity, their neural tangent kernels θ converge to the same deterministic limit Ô at initialization and have the same optimization trajectory during training.
The following theorem is our main result: propagation, under very common conditions, the complex MLPs are asymptotic equivalent with real MLPs, thus they have the same training dynamics.
if we train complex networks with real-valued back
Theorem 7 Consider complex MLP in Eq. B and ordinary real MLP with L hidden layers trained by real-valued backpropagation. Suppose a = а at initialization and the activation functions satisfy Assumption _ As the widths go to infinity, they asymptotic equivalent if the activation functions satisfy one of the following conditions
Condition 1
All split activation functions & satisfying Φ(a, 8) = = ( + OR(B)i, where oR
is a real-valued activation function;
Condition 2 A subset of holomorphic activation functions o satisfying ø2(a, 3) @1 (B, -a)
and (0.(0..)
0,
03
where the general complex activation function is denoted as oa, (1 + Φ2 B)i with inpul pre-activations o + Bi.
In the Appendix D the result is proved and we also give the sufficient and necessary conditions. For simplicity, here we only show the most informative conditions. The key idea of the proof is transforming asymptotic equivalence into four complex conditions and find the common solutions based on R.Ies.F.15 13 in Appendix F.
Discussion about the Condition 1. Note that most commonly used complex activation functions satisfy the Condition
ob))=)n()(())) + on (3()
like complex sigmoid function [Bevvento and Piazza 1992 Nitta 1997], complex hyperbolic tangent function (Hirose and Yoshida [002]]. etc. It is also worth mentioning that the recently proposed ReLU-based complex activation function CReLU also satisfies the Condition which has achieved the best performance in feed-forward complex networks in image processing tasks (Trabelsi Tan et al. 2020 among all ReLU-based complex activation functions.
al. 2018
Remark Because of Liouville's theorem, the only complex-valued functions that are bounded and analytic everywhere are constants. Thus in practice, must choose between boundedness and analyticity for a complex activation function. Before the popularity of ReLU, almost all activation functions in the real case were bounded. Consequently, previous works about complex networks always preferred non-andn -nalllyt functions to preserve boundedness. Most commonly they applied split activation functions separately to the real and imaginary parts, as investigated in Bassey et al. (2021) and Sndaddaaaaa et al. [2018). So the condition contains most complex networks in practice.
As result, the theorem demonstrates that for complex networks with all these common complex activations, they are trained by real-valued BP, then these complex networks reduce real networks as widths grow, despite the joint interaction weight sharing caused by complex matrix multiplication structure. Consequently, real-valued backpropagation totally eliminates the characteristics of complex networks at infinite width. This may guide the selection of training algorithm in practice if people want to take full advantage of complex networks, and encourage people to explore learning algorithms specially designed for complex networks.

---

L-1. CRAUU
1-1. modified mod 0.5
L-1. 2RoLU
complex,
n-100
complex, n-1000
complex, n=0000
2.0
complex, n=1000
real, ni
real, tim
real, n-
0.0
-2.5
-5.0
0.0
L=2, CReLU
L-2, modifled modReLU, 0.5
L= 2ReLU
complex,
20-
complex, n=100
complex, n-100
complex, n=1000
complex, n=1000
complex, 1=1000
real n=
real, n- oz:
real, n-
(2.
0-
801
1.0
0.5-
0.3
12
Figure 1: Convergence of complex NTKs to corresponding real NTKs at initialization. One input
z = - is fixed and the other input z' = cos a + sin ai varies with a E [0, 2T.. The black line is the limiting NTKs of real MLPs Ó,(z, z") while the light blue and blue ones are empirical NTKs of complex MLPs e (n) (zz x²) with widths = 100 and 1000. For each width, ((m) (z, 2²) is calculated 100 times randomly, corresponding to 100 lines in the figure.
Discussion about the Condition 2. Condition 2 is also non- vacuous since it is subset of Cauchy- ReL.U-tppe complex activation functions.
Riemann condition. includes all the magnitude- For example, we can easily obtain a modified modReL. (Arjovsky et al. (016) and a modified phase-based ReLU satisfying Condition as follows
if [ 2 P.
{2
if g(cos f2)) S %
0(2) to otherwise.
6(2) = {0
otherwise.
where g(cos 9.) can be any function of phase cos 0g. Note that, these activation functions are analytic almost everywhere, and according to previous theory, they enjoy better theoretical results like separation results (Zhang et al. [022] and local minima [Wu et al. (02T]. However, our results indicate that real-valued backpropagation eliminates these advantages of complex networks in the
wndhh limit, which further illustrates the inappropriateness of real-valued backpropagation.
5 Empirical study
In this section, we empirically verify the relationship between NTKs of the complex networks and real networks, and investigate the network widths required for the establishment of our results. We consider complex-valued MLPs with one or two hidden layers and we use CReLU, modified modReLU, CSigmoid, Ctanh and zReLU as the activation functions. Note that all these activation functions satisfy the conditions of our theorem except zReLU. Through the following experiments, we want to check whether the empirical complex NTKs (()) converge to the corresponding real NTKs é, with different activation functions as the widths n grow.
For real networks, the input is the concatenated vector of the real and imaginary parts of the complex- valued input. Corresponded to the complex-valus -valued fully-connected layer, we use the commonly used real-valued fully-connected layer without complex matrix multiplication structure. To implement the corresponding activation functions, complex-valle -aaueed activation functions are transformed to real- valued ones in the following way: we divide the pre-activation vector into two half, treat the first half as real parts and the second as imaginary parts, as the input of ó(a, 3). Then we concatenate the real and imaginary parts after activation. For split activation functions like CReLU, it can just correspond to real ReLU activation. In NTK initialization, the standard deviations are set as I for complex networks and sealed to /2 for real networks. All empirical NTKs of complex networks are calculated based on the Neural Tangents library NNova et al. 2019
7

---

-6 6.1+//6.11
16
6
2
2-
2
2
2-
2-
24
2
CRelU
CSigmold
24
zRaLU
modifeed modReLU 0'3
2-7
Ctanh
modified modReLL pm 0.3
2
2
26
21
29
2
2
2
2
2
2
2
2
2
Figure 2: Convergence of complex NTKs to corresponding real NTKs under various activation functions with a larger range of widths. The Y-axis is the difference between empirical complex NTKs and the corresponding real NTKs in terms of the relative Frobenius norm. At each point the relative Frobenius norm is calculated 20 times and the mean value shown. Left: three kinds of split activation functions satisfying our conditions. Right: other complex-valued activation functions where modified modReLU satisfies conditions and zReLU does not.
Verifying asymptotic equivalence initialization. The first experiment shows the distribution of empirical NTKs of complex MLPs (2, and analytic NTKs of real MLPs a synthetic dataset. We define =
=') with
i and z' cos sin ori for a E [0, 2m]. Then we can view @,m²( (z. z²) and (,,2, 2') as functions of a. For the complex 100, 1000 initialization and hidden layer number ( 1. In each case, we calculate ((m)) (z, =') 100 times with different random NTK initialization. We compare these empirical complex NTKs with corresponding real NTKs. In the case of CReLU, in the case zReLU and modified modReLU, it's 's hard to get the closed form solution, so we use the average of a large mount of wide empirical NTKs to approximate Ô,. The results are shown in Figure I In the figure observe that for CReLU and modified modReLU, which satisfy our conditions, their NTKs (() concentrate to the NTKs of real MLPs é, perfectly, and when n = 1000, the convergence is more concentrated and the complex NTKs almost equal to real NTKs; for zReLU which does not satisfy the conditions, there's ' gap between complex and real NTKs. Therefore, the results verify theorem perfectly and demonstrate our results are non-vacuous.
different z' initialization
networks, calculate empirical NTKs for hidden layer widths
calculate Ô, by the analytic form solution of NTK [Cho and Saul 2009];
Verifying asymptotic equivalence with more activation functions widths grow much larger. In the first experiment, although for zReLU there gap between complex and real networks, the ten- dency is still similar. For the second experiment, we do the similar experiment on the same synthetic dataset, to see what will happen when becomes much larger, so that can be more convincing to verify whether it converges. Besides, we consider more different activation functions which satisfy our conditions including CReLU, CSigmoid, Ctanh and modified modReLU with different hyper-
parameters p. We calculate relative Frobenius norm || | (X, X) -, (X, X) I (1,, ( X with widths n ranging from 2 2 which measures the difference between complex NTKs 6(m) and real NTKs é, Figure 2 shows the result. For all those split activation (unctions(CReeLu CSigmoid and Ctanh), the tendency of convergence remains unchanged
(x,
when the width goes to quite a large number 1 The two curves of modified modReLU act similarly with that of split activation functions;
However, the curve of zReLU does not converge at all at initialization.
Verifying asymptotic equivalence during training. The third experiment investigates the conver- gence of difference between complex NTKs 6(() and real NTKs è, during training as the widths go to infinity on MNIST [LeCun et al. 1998). We randomly choose subset of MNIST as training set D = -x (ID) 128), and treat the first half of features as real parts, the second half as imaginary parts. Then we calculate relative Frobenius norm between empirical NTKs of complex networks at time and real NTKs at initialization with widths n ranging from 25 to 210 at initialization (z=00 0) and during training (L. = 1000). Due to the memory limitations, we cannot try larger The learning rate y is 0.5 for 1 = and 0.2 for
The results are shown in Figure 3 We can see that in all these cases, the relative Frobenius norm decreases goes up. regardless of the training steps and

---

CReLU
modified modRelU, p 0.5
2
21
t-0,
L=1
1=/10/1F
t-1000, L-1
0.1:/1011
20
t-100,,
20
tw0, L-2
t=0,
2-
t-1000, L-2
2-1
t- 1-1000, L-2
2
2-2
2-1
2 2
27
200
2
2
2 2
2
2
2
Figure 3: Convergence of complex NTKs to corresponding real NTKs as widths grow during training. The Y-axis is the difference between empirical complex NTKs and the corresponding real NTKs in terms of the relative Frobenius norm. At each point the relative Frobenius norm is calculated 20 times and the mean value shown. The solid line indicates one hidden layer (T. = 1) while the
dashdot line indicates two hidden layers (I. = 2). The square marker indicates = 0 (at initialization) while the circle marker indicates 1000 (during training).
hidden layer numbers. Therefore, our theoretical results hold during training. More experiments verifying asymptotic equivalence can be found Appendix
Empirical results. Overall, in the case of complex activation functions satisfying our theorems, such as CReLU and modified modReLU, complex NTKs 6(")) converges to real NTKs é, quite well even the widths are about 1000; in the case of complex activation function that does satisfy our theorems like zReLU, the convergence from the complex NTK 6(") the real NTK é, does not occur. This validates our theory perfectly and demonstrates that our conditions are non-vacuous.
6 Related work
Long before the popularity of deep learning, there have been many investigations on complex- valued neural networks (Hirose 1992 [envenuto and Piazza 1992 Nitta 1997]. However, It is always challenging to train complex networks due to some non-intuitive analytical properties, of which the most notable reason that almost all cost functions real-valued and thus non- holomorphic. In order to perform backpropagation for complex networks, the conventional approach to overcome the limitation is use separate derivatives with respect the rell-imaginary parts of a non-analytic function (Nitta 2004], or split amplitude-phase parts [Hirose) 1992). Hirose and Yoshida 20122 has shown that split backpropagation for amplitude-phase parts could achieve better
generalization than real networks on signal processing tasks. Recently Wirtinger calculus [Wittnger
1927] has received increasing attention (Adai et al., 2011 Bassey
2021], which presents
an elegant formulation to derive complex-valuee gradient. Jacobian, and Hessian. The real-valued backpropagation [Nitta 1997] is most widely used, and becomes essential for training deep complex networks due to the convenience of utilizing the real-valued deep learning library [Arjovsky et al. 2016, Trabelsi et al. 2018 Tan et al. 2020) and achieves state-of-the performance. Theoretically, there have been important advances for complex networks regarding the universal approximation property VVogglaender] 2020], local minima [Nitta] 2013 Wu et al. [021]. separation results (Zhang ct al. |.22]. However, to our best knowledge, there is still no theoretical analysis of the training dynamics complex networks and the equivalence after training between complex and real networks.
7 Conclusion
In this paper, we propose way to compare the training dynamics between complex and real networks based on their neural tangent kernels (NTKs). Surprisingly, we find that the commonly used real- valued backpropagation reduces the training dynamics of complex-valun real MLPs as the widths tend to infinity, thus eliminating the characteristics of complex-valued neural networks. Empirical study verifies that our results are practical for commonly used complex activation functions. The results encourage the design of new training algorithms for complex networks in
MLPs to that of ordinary

---

the future. Besides, the proposed asymptotic equivalence of training dynamics between networks provides a new perspective for theoretical analysis of neural networks, which may offer possibility to divide various neural network architectures into equivalence classes.
Acknowledgment
This research was supported by NSFC (62176117, 61921006) and Collaborative Innovation Center of Novel Software Technology and Industrialization, and the Program B for Outstanding PhD candidate of Nanjing University. The authors would like to thank Jin-Hui Wu and Peng Zhao for helpful discussions. We are also grateful for the anonymous reviewers for their valuable comments.
References
Tülay Adali, Peter J Schreier, and Louis I. Scharf. Complex-valued signal processing: The proper way to deal with impropriety. IEEE Transactions on Signal Processing, 59(11)5100-55225 2011.
Sina Alemohammad, Zichao Wang, Randall Balestriero, and Richard Baraniuk. The recurrent neural tangent kernel. In International Conference on Learning Representations, 2020.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference Machine Learning, pages 1120-11288 2016.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances Neural Information Processing Systems, pages 8139-8148, 2019.
Joshua Bassey, Lijun Qian, and Xianfang Li. A survey of complex-valueee neural networks. arXiv preprint arXiv:2101. 122.4 2021.
Nevio Benvenuto and Francesco Piazza, On the complex backpropagation algorithm. IEEE Transac- fions Signal Processing, 40(4):967-969, 1992.
Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. In Advances in Neural Information Processing Systems, pages 342-350, 2009.
Ivo Danihelka, Greg Wayne, Benigno Uria, Nal Kalchbrenner, and Alex Graves. Associative long short-term memory. In International Conference Machine Learning, pages 1986-1994, 2016.
Simon S Du, Kangcheng Hou, Russ R Salakhutdinov, Barnabas Poczos, Ruosong Wang, and Keyulu Xu. Graph neural tangent kernel: Fusing graph neural networks with graph kernels. In Advances in Neural Information Processing Systems, pages 5724-5734, 2019.
Nitzan Guberman. On complex valued convolutional neural networks.
arXiv preprint
arXiv:1602.09046, 2016.
Akira Hirose. Continuous complex back-propagation learning. Electronics Letters, 28(20): 1855-1855, 1992.
Akira Hirose and Shotaro Yoshida. Generalization characteristics of complex-valued feedforward neural networks in relation to signal coherence. IEEE Transactions on Neural Networks and Learning Systems, 2314),
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, pages 8580-8589, 2018
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):227-2322 1998.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Babri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in Neural Information Processing Systems, pages 8572-8583, 2019.
10

---

Tohru Nitta. An extension of the back- propaaappottnon algorithm to complex numbers. Neural Networks, 10(8):1991-1415, 1997.
Tohru Nitta. On the critical points the complex-valuee neural network. In Proceedings of the 9th International Conference Neural Information Processing, pages 1099-1103 2002.
Tohru Nitta. Orthogonality of decision boundaries in complex-valued neural networks. Neural
Computation, 16(1):73-95 2004.
Tohru Nitta. Local minima in hierarchical structures of complex-valued neural networks. Neural Networks, 43:1-7, 2013.
Roman Novak, Lechao Xiao, Jiri Hron, Jachoon Lee, Alexander A Alemi, Jascha bohl-Dickstein, and Samuel S Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. In International Conference on Learning Representations, 2019.
Simone Scardapane, Steven Van Vaerenbergh, Amir Hussain, and Aurelio Uncini. Complex-values neural networks with nonparametric activation functions. IEEE Transactions Emerging Topics in Computational Intelligence, (2):144-150,, 2018.
Xiaofeng Tan, Ming Li, Peng Zhang, Yan Wu, and Wanying Song. Complex-valus valued 3-d convolutional neural network for polsar image classification. IEEE Geoscience and Remote Sensing Letters, 17 (6):1022-1026, 2020.
Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, Joao Felipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher J Pal. Deep complex networks. In International Conference on Learning Representations, 2018.
Mark Tygert, Joan Bruna, Soumith Chintala, Yann LeCun, Serkan Piantino, and Arthur Szlam. A mathematical motivation for complex-valued convolutional networks. Neural Computation, 28(5): 815-825, 2016.
Felix Voigtlaender. The universal approximation theorem for complex-valued neural networks. arXiv preprint arXiv:2002.0335// 2020.
Wilhelm Wirtinger. Zur formalen theorie der funktionen von mehr komplexen veränderlichen. Mathematische Annalen, 97(1):357-375 1927.
Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux, and Les Atlas. Full-capacity unitary recurrent neural networks. In Advances in Neural Information Processing Systems, pages 4880 4888, 2016.
Jin-Hui Wu, Shao-Qun Zhang, Yuan Jiang, and Zhi-Hua Zhou Towards theoretical under- standing of flexible transmitter networks via approximation and local minima. arXiv preprint arXiv:2111.06027, 2021.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint 19Xiv:1902.04777 2019.
Greg Yang. Tensor programs ii: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006.14555 2020.
Greg Yang and Etai Litwin. Tensor programs iib: Architectural universality of neural tangent kernel training dynamics. In International Conference on Machine Learning, pages 11722-11772, 2021.
Xin Yao, Xiaoran Shi, and Feng Zhou Human activities classification based complex-value convolutional neural network. IEEE Sensors Journal, 20(13::7169-7188) 2020.
Shao-Qun Zhang and Zhi-Hua Zhou. Flexible transmitter network. Neural Computation, 33(11): 2951-2970, 2021.
Chao-Qun Zhang, Wei Gao, and Zhi-Hua Zhou. Towards understanding theoretical ad advantages of complex-reaction networks, Neural Networks, 151:80-93, 2022.

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sectionn3je for assumptions.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section for assumptions.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [N/A] The numerical experiments only aim to verify the thcoretical results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] The numerical experiments only aim to verify the theoretical results.
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]] MNIST.
(b) Did you mention the license of the assets? [N/A] MNIST. GNU General Public License v3.0
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "oBacKl4sKfr", "reviewer": "Reviewer_E55z", "summary": "The paper derives the equation for the neural tangent kernel (NTK) for complex-valued neural networks trained via real-valued backpropagation (backpropagation on the real and imaginary components separately). Their main theoretical result shows that, under common choices of activation functions, the NTK in the infinite-width limit is the same as that of a real-valued neural network.", "questions": "- Please provide a proofreading because the paper has several typos.\n- A small introduction to questions of holomorphicity or complex backpropagation could be helpful to broaden the understanding of the results.", "limitations": "NA", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is an extremely interesting paper for the community working on complex-valued neural networks. The results are easy to follow and, by any point of view, extremely interesting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ML9JFFbBObE", "reviewer": "Reviewer_JGv9", "summary": "Complex-valued infinitely wide neural networks are considered in the paper. Through studying the kernel limit of complex-valued networks, the authors showed that real-valued backpropagation reduces complex NTK to a real NTK, under certain conditions on the activation function. Interestingly, many of practically used activation functions satisfy the condition, and therefore, using real-valued backpropagation may not be suitable to leverage all the potential of complex-valued networks.", "questions": "I am interested to know if complex-valued neural networks can beat the state-of-the-art real-valued neural networks in applications, such as computer vision and time series prediction.\n\nIs there any concrete motivation of studying the infinite width limit of complex-valued neural networks? As mentioned in the introduction, complex-valued neural networks have richer representation capacity and faster learning. Is overparameterization in complex-valued neural networks necessary for a good performance?\n\nIs it possible to suggest a proper backpropagation method based on the theoretical findings in the paper in order to further leverage the potential of complex-valued neural networks?", "limitations": "The major limitation comes from the NTK analysis. In practice, neural networks are often trained with a relatively large learning rate for a sufficiently large number of iterations. Therefore, the working regime of neural networks is actually not NTK. The theoretical findings undoubtedly enhance the understanding of complex-valued networks; they may have limited implications for practice. For example, in the infinite-width limit, real-valued backpropagation may not be appropriate. However, in practice, the theory does not apply due to aforementioned reasons. Whether using real-valued backpropagation is suitable remains open.\n\nI am fond of the theoretical findings in the paper. But the practical value of the paper is rather limited to my knowledge. Therefore, I have a neutral initial rating of the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The theoretical findings in the paper are interesting and intriguing. Complex-valued neural networks are relatively less studied, and whose theoretical understandings are lacking. This paper contributes towards this direction and the reported results read sound and correct. Moreover, there are empirical studies corroborate the theoretical findings. The organization of the paper is good and most of the results are easy to understand.\n\nI will discuss some weaknesses of the paper in the limitation section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EDw-LSBiS4E", "reviewer": "Reviewer_WKtx", "summary": "This paper extend the Tensor programs that was initially proposed for real valued neural networks to complex valued neural networks (CVNN), and as a result, providing the foundations to calculate the the neural tangent kernel (NTK) of any CVNN architecture. Later, they show that the NTK of any CVNN with a certain type of activation functions ( that contains majority of common used activation function ) is as expressive a real valued neural network by comparing their limiting NTKs, as a result CVNNs can not offer anything more that real valued neural networks. They also provide some examples of activation functions that differs CVNNs from their real valued neural networks.   ", "questions": "How would this results be affected if we replace the output layer instead of Real(WH), we use abs(WH)? can tensor programs still be applied to these kind of networks? how does it affect the results? ", "limitations": "It would be very useful to see the of effects of choosing different activation functions on performance of CVNNs and real neural works in real world datasets. How can zrelu be superior over other cost functions? What are the task that CVNNs are better than their exact real valued counterpart? A few carefully experiments on real valued neural networks can to answer these question can significantly boost the quality of this paper.  ", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Tensor programs is a series of papers that build the foundations for understanding wide neural network. It was initially proposed for real valued neural network to calculate the NTK of any architecture. This work extend it to CVNNs and thus provide a theoretical foundation to analysis CVNNs. The results of this theoretical foundation, other than calculating the NTK of any CVNN can be very valuable, one that is discussed in this paper on conditions on activation functions that distinguishes CVNNs from their real valued neural networks. The foundation that is built in this work also has the potential to be used to extend the theorems to analyze CVNNs outside of NTK regime where weights move from initialization like Tensor plus programs [1]. \n\n\n\n[1] Yang, Greg, and Edward J. Hu. \"Feature learning in infinite-width neural networks.\" arXiv preprint arXiv:2011.14522 (2020). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept