CosNet: A Generalized Spectral Kernel Network
Authors: Yanfang Xue
Pengfei Fang
Jinyue Tian
Shipeng Zhu
hui xue
Venue: NeurIPS 2023 poster (2023)
Keywords: Spectral kernel; complex-valued networks

Abstract: Complex-valued representation exists inherently in the time-sequential data that can be derived from the integration of harmonic waves. The non-stationary spectral kernel, realizing a complex-valued feature mapping, has shown its potential to analyze the time-varying statistical characteristics of the time-sequential data, as a result of the modeling frequency parameters. However, most existing spectral kernel-based methods eliminate the imaginary part, thereby limiting the representation power of the spectral kernel. To tackle this issue, we propose a generalized spectral kernel network, namely, \underline{Co}mplex-valued \underline{s}pectral kernel \underline{Net}work (CosNet), which includes spectral kernel mapping generalization (SKMG) module and complex-valued spectral kernel embedding (CSKE) module. Concretely, the SKMG module is devised to generalize the spectral kernel mapping in the real number domain to the complex number domain, recovering the inherent complex-valued representation for the real-valued data. Then a following CSKE module is further developed to combine the complex-valued spectral kernels and neural networks to effectively capture long-range or periodic relations of the data. Along with the CosNet, we study the effect of the complex-valued spectral kernel mapping via theoretically analyzing the bound of covering number and generalization error. Extensive experiments demonstrate that CosNet performs better than the mainstream kernel methods and complex-valued neural networks.

CosNet: A Generalized Spectral Kernel Network
Yanfang Xueli?, 1,2 Pengfei Fang Jinyue Tian Shipeng Zhu!:2 Hui Xue 1.2
School Computer Science and Engineering, Southeast University, Nanjing, 210096, China Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China (230218795, fangpengfei, 220222083, shipengzhu, hxue Oseu. edu.
Abstract
Complex-valee valued representation exists inherently the time-sequential data that can be derived from the integration of harmonic waves. The non-stationary spectral kernel, realizing complex-vail -valudd feature mapping, has shown its potential analyze the time-varying statistical characteristics of the time-sequential data, a result of the modeling frequency parameters. However, most existing spectral kernel-based methods eliminate the imaginary part, thereby limiting the representa- tion power of the spectral kernel. To tackle this issue, we propose a generalized spectral kernel network, namely, Complex cemplex-valued valued spectral kernel Network (Cos- Net), which includes spectral kernel mapping generalization (SKMG) module and complex-valle vaauud spectral kernel embedding (CSKE) module. Concretely, the SKMG module devised to generalize the spectral kernel mapping in the real number domain the complex number domain, recovering the inherent complex- valued representation for the real-valued data. Then a following CSKE module is further developed combine the complex-valued spectral kernels and neural networks to effectively capture long-range or periodic relations of the data. Along with the CosNet, we study the effect of the complex-valued spectral kernel map- ping via theoretically analyzing the bound of covering number and generalization error. Extensive experiments demonstrate that CosNet performs better than the mainstream kemee methods and complex-valued neural networks.
1 Introduction
Complex numbers represent the information the amplitude and phase simultancously. In contrast the amplitude, which can be revealed by the real number, the phase denote the time delay and advance, thereby encoding the temporal dependency of the data Hirose [2003]. This suggests that complex-valued models be employed in practical applications, especially for the time- sequential data where information is wave-related, such as signal analysis (aalysss/Hrrose Hirose et al. (2019): Yu er al. [2019] Zeng et al. 20022] speech processing Shafran al, (2018), and time series classification Yang
el al. 2020 2017 Wisdom et al. 2016))
In the learning community, the spectral kernel, which is constructed from the inverse Fourier transform. can naturally realize the complex-valuce mapping. That said, the spectral kernel can analyze the data in the frequency domain directly. As candidate of the spectral kernel family, the non-stationary spectral kernel is also proposed to conquer the local limitations of the classical kernels, such as stationarity and monotonicity Remes al. [017]] Tomppins et al. [2200]: Ton al. (0188: Li er al.) [2020]. Ideally, these advanced kernels can extract appropriate non-stationary time-varying characteristies of the data by modeling the frequency parameters, and hence infer the long-range
periodic relations of the input data.
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

However, most existing methods usually eliminate the imaginary part of the spectral kernel mapping arbitrarily for the convenience of calculation, For example, Raaimi and Recht (007]: Zhang et al. [2177a] ignore the imaginary part directly by replacing the integrand ejw(æ- with cos( (w )): Xue et al. eliminates the imaginary part with elaborate spectral density function definition for the non-stationary spectral kernel Xue et al. [001)]. Remarkably, existing research shows that complex numbers could lead to a rich representational capability for wave-related information processing Wisdom et al. (2016): Danihelka et al. (2016); Worrall et al. [2017]; Trouillon and Nickel [2017]. However, simply plugging the imginnrr part in the neural networks does not ensure that the model retains the property of the spectral kernel. Therefore, more efforts are required to develop a new framework that can involve the imaginary part in spectral kernel networks.
In this paper, we propose a new framework that generalizes the spectral kernel that endows with the complex-value representation, and we name it as a complex-valued spectral kernel network (CosNet). The proposed CosNet includes two modules: the spectral kernel mapping generalization (SKMG) module and the complex-value spectral kernel embedding (CSKE) module. Technically, we generalize the spectral kernel mapping in the real number domain to the complex number domain by defining the spectral density function in the SKMG module. We further embed the complex-valuee spectral kernel into neural networks to attain the proposed CosNet using the CSKE module. It is noted that a new initialization scheme is also proposed for the CSKE module that adopts the cosine and sine functions as the activation for the real and imaginary parts of the weight matrix. This initializing scheme retains the statistical characteristics of the non-stationary spectral kernel. It enables CosNet to take the relative distance of data into account by shifting between phases, such that capture the long- mang-range or periodic relations of data in the complex domain without increasing the number of parameters. Our contributions in this paper are shown as follows:
We propose a complex- valued spectral kernel network, i.e., CosNet, which takes both the real and imaginary parts of the spectral kernel mapping into account and thus improves the representational capability of the spectral kernel.
We propose an initialization scheme for the complex-valle valued weight matrix, which ensures that CosNet retains the property of non-stationary spectral kernels and takes the relative distance of data in the complex number domain without increasing the number of parameters.
We provide the lower generalization bound of CosNet than the real-valued non-stationary spectral kernel.
Thorough experiments demonstrate that our proposed method is totally superior to state-of- the-art kernel methods.
2
Related Work
Spectarl kernel networks Spectral approaches were developed to fully characterize general kernels with concise representation forms, such as sparse spectrum kernels Greaill et al.
(2010). sparse mixture kernels Wilson and Adams| [2013]. non-stationary spectral kernels Remes er
[2.17. and random Fourier features methods to deal with large-scale settings Li et al. [2019] Liu et al. 202T]. These methods commonly approximated the kernel function using an explicit spectral representation based on Bochner's 's theorem Bochner and others [1959] and Yaglom's 's theorem Yaglon
[187]]. Benefiting from the outstanding representation capability of neural networks with hierarchical nonlinear linking structures Bengio et al. (2006), researchers attempt to embed spectral representation (i.e., feature mapping of kernels) into the hierarchical architecture of neural networks to construct spectral kernel networks. ZZhang et al. [2017a] used the Random Fourier Feature to approach the stationary kernel mapping and embedded it into each layer of DNNs. Xue et al. [2019] proposed a deep spectral kernel network to embed the non-stationary spectral kernel into each layer of DNNs, which can approximate most of the kernels. Li et al. (2020) proposed an automated spectral kernel learning (ASKL) that incorporates the process of finding suitable non-stationary kernels and model training. However, for convenience of calculation, these models commonly use the real-valued representation, although spectral kernels lead to the complex-valus valued mapping.
Complex-valued neural networks lent efficiency compared to their real counterparts in biological Reichert and Serre [203) J speech enhancement (araaneeentTTTuuiii Tsuzuki el al. [2013); Choi el al. [2019), image Popal [[017] 7J: Wen el al. [2020], and
Complex-valued neural networks (CVNNs) have shown excel-
signal processing Kim and Guest 1990]: Wilmanski et al. [2016). In previous studies, researchers
2

---

commonly split the complex- valued input into pair of real-valued inputs and fed them into the real-valued neural networks with both real-valued weight matrix and activation function. This design cannot exploit the advantages of complex numbers completely, and the neural network convergence
strongly depends on proper initialization and the choice of learning rate Yang et [200]] Zhang ei al. [2009]. Subsequently, CVNNs with complex-valuee weight and activation functions are proposed in the complex number domain to deal with complex-valuee inputs Hirose [1992]; Dedmari et al.
2018); [Zhan et al. [2017b). Benefitting from the rich representation capability, researchers tend to extend CVNNs to other neural networks, such as complex-valee -valued convolutional neural networks Tra- belsi et al. (0188), , complex-valued residual neural networks Wang et al. [2018), and complex-valued recurrent neural networks Wolter and Yao 2018]; Arjovsky et al. [2016]. All these works have proved that the complex-valued models have richer representational capacity and perform better
real-world learning tasks by a set of experiments.
Complex-valued Spectral Kernel Networks
In this section, we first introduce concepts and notations of the non-stationary kernel and complex numbers. Then, we provide the overall architecture of our CosNet with two modules. Moreover, we explicitly provide the details of each module. In addition, we present detailed analysis of CosNet.
3.1 Preliminary
To better illustrate CosNet, we introduce the necessary preliminary knowledge and notation of non-stationary spectral kernels and complex numbers in this section.
Notations Formally, use R". C", mx and Cm to denote n-dimensional Euclidean spaces, n-dimensional complex number spaces, the space of m x n real-valued matrix and the space of m complex-valued matrix. Throughout the paper, the matrices, vectors and scalars are denoted by bold capital letters (e.g. X), bold lower-case letters (e.g. x)) and lower-case letters (e.g. x),, respectively. A complex number part u and an imaginary part z = u iv denotes the complex conjugate of 2. For any two complex numbers 21 Ui + i0i Z2 = U2 iV2 E C, z + z2 (u, + u2) + i(v) + v2), = = u2 U2 01 V1 + i(u11() 02 + 01 U2). To represent the complex-valuee layer with 2D features, we allocate the first D features to represent the real component and the remaining to represent the imaginary component.
CD is represented as
iv with real
Non-stationary spectral kernels are constructed from inverse Fourier transform in the frequency domain. Based on Yaglom's theorem Yaglom 1987]. a general kernel k(#,, æ²) is positive definite on RDD if and only if it admits the form:
Preliminary knowledge
k( = JRDXR
eilw "w-w" x') ulddw, dw')
(I)
where u(dw, dw') is the Lebesgue-Stieltjee measure associated with some positive semiidefinite spectral density function s(w, w²) with bounded variations. Therefore, general kernel can be defined as the following form:
k( x')
JRPXRD
ellw" a- x²) s()s(w w' ''wddo
(2)
where s(w, w²) can be understood a joint probability density function.
3.2 Overall architecture
To explore the capability of the imaginary part in the spectral kernel networks, we propose CosNet as a generalized framework. CosNet involves two modules: the SKMG module to achieve complex-valuce spectral kernel mapping and the CSKE module to embed the spectral kernel into neural networks. The overall architecture is shown in Figure I
Concretely, the SKMG module is denoted as (())) via generalizing the spectral kernel mapping in the real number domain to the complex number domain for the real-valued data æ. And the CSKE module is denoted as (h) via initializing cmmplex-valuee weight matrix with the cosine and sine function for the complex-valued spectral kernel mapping h.
3

---

Σ (igpA
co >,, sin(Agan0g
Σ,n coalAmyys En, salkroot
Σn, costAvnn May
Σ siniAw
Pra
Σn cootAeniitr Σn, antAensto
SKMG
CSKE
The structure of CosNet with two modules. The SKMG module is used to map the real- valued inputs to a complex-valuee representation. The CSKE module the complex-valus spectral kernel embedding with our initialization.
Figure I:
Based on the two modules, our CosNet with layers is defined as
CosNet(x)) (... v ( (æ))).
Moreover, the corresponding complex-valued spectral kernel is defined
KO (z. x²) =( -(1 (x)))) ('- (...(.... ( (x)))) CCosNet(()
where K(() (xx x²) denotes the l-layer complex-valued spectral kernel.
3.3 Complex-valued spectral kernel network (CosNet)
Spectral kernel mapping generalization module In this module, generalize the spectral kernel mapping in the real number domain to the complex number domain. Furthermore, the generalized mapping can be used in both stationary and non-stationary spectral kernels. Here elaborate on the detailed process.
According to Equation to produce positive semi-definite kernel, we need to include symmetries s(w, w/) s(w', w) and sufficient diagonal components s(w, w) and s(w', w'). Concretely, we replace the exponential component ei(w " "T-w"=") in Equation with Cwr (x, x'), which is defined as:
1 fl x www ollw Tx) x¹) + slw" a w-w x') + dll x TT)) (5)
Sw.wr (xc, x²)
Then, expand the exponential component to the complex-values valued representation with the cosine and sine function based on Euler's 's formula, and ww (æ, can rewritten as:
[[cos(ww T x - TT x') isin(w" x wTT x²)
S (x, x²)
TTTTT +cos(w") "x --ww wT TTw'
isin w wTx' x') +cos(w To -w wTx') + isin x w wTx') +cos(w'T, TT wT xx) isin (w x w [Ta)]
(6)
As result, the general spectral kernel Equation 2 can redefined as:
k(x, x') = /
JxxxR Sw (x, x' d')p(w, w' ddwdw
(7)
where p(w, w') ws( + s(w', w) + www)ww + s(w', [(w',w")) also can be considered as probability density function.

---

Subsequently, we approximate Equation 7 with Monte Carlo random sampling:
k(x,
/
Swrw (x, )p(w, w' w')dwdw' Ee Bor Swe (2r, a²))
JRD
M
cos(w,) x - w x') + isin(w; x - w/, x0)
4M
1=1
+ +cos(w; w/ x') isin ( a w/ x')
(8)
+cos(w, ( w x') isin w x')
+cos(w/ wtt x²) + isin(w, x w/ aa))
=(O(x), ((x))
where, (w,, w, w the frequency pairs, M is the sampling number. The generalized spectral kernel mapping in Equation
8 is defined as:
(cos(52 x)) + cos(S2' x)) + i(sin(S) x)) sin(S +Sin(((TT))) (T(Da))))
$ (x)
V
(9)
4M
and the frequency matrices Ω St denoted as:
Ω W1 W2.
wM.. D' WS. whrll
(10)
As a result, we obtain complex- valued spectral kernel mapping. The real part of the output is denoted as R( ((x))))))))))))))(((()))))))(( = cos x) + cos (Ss x). and the imaginary part denoted as ((b(x))
sin(S2 2² + sin( 0 mx).
Complex-valued spectral kernel embedding module In this module, we attempt to embed the complex-valued spectral kernel into each layer of neural networks construct CosNet. Spectral kernel, based on the general Fourier analysis, provides new explicit kernel mapping. These kernels not only approximate most kernels under specific conditions by some fundamental theorems Cox and Miller ]017]; Yaglom [987] but also provide an efficient way to combine neural networks with kernel methods to construct spectral networks. Most existing spectral kernel networks commonly embed the spectral kernel into neural networks by stacking the spectral kernel mapping in the hierarchical architecture of neural networks directly. However, the introduction of imaginary part enables that networks with simple stack of complex-valued mapping cannot be formulated as a spectral kernel (see the Supplementary Material for details).
To ensure the sub-network containing the first layer to arbitrary /-th layer (1 > 2) can be integrally seen as spectral kemnel, and following the form of complex-valuee parameters in CVNNs, we define the complex-vall valuue weight matrix of this module as:
W cos A) isin( A),
(11)
where real-valued matrix.
In this module, the convolution operates with the complex weight matrix W is defined as:
1
(h) = Wh
(coss)((Os(SE' x) + cos x))
4M
sin( (A)(sin( sin(TT(T:)))
(12)
VV 4M
sin (in(A)(cos((2" 2x) + cos 2:))
+ cos( (sin T x) + sin x))
The real and imaginary parts of the convolution operation are represented in the matrix notation:
R (R()
cos A)
sin (A) cos( A)
Toos( a:) + cSs x) sin x) sin sin (SY
(13)
3 (h))
sin A)
* 4M
5

---

To inherit the outstanding representation capability from neural networks, in this module, we construct the spectral kernel networks by stacking V/:
(o)) (.......W' V....V// (h)),,
(14)
where (2 S 1) denotes the l-layer complex-valued spectral kernel mapping and
=
1
(cos(S2 x)) cos((2" 2:) +i(sin((2) x) (e(TTTT)))
(15)
4M
3.4 Analysis of CosNet
CosNet, constructed by stacking the non-station complex-valucee spectral kernel mapping, not only re- tains the property of non-stationary spectral kernels, which can effectively reveal the input-dependence characteristics and long-range relations but also can learn hierarchy within Reproducing Kernel Hilbert Space, yielding cascade of non-linear features. Besides, CosNet takes the imaginary part of the complex-vale valuud spectral kernel mapping into account, leading richer representation capability.
Framework generality In spectral kernels view, CosNet will be reduced to a stationarity spectral kernel when w w' in Equation case (i.e. the imaginary part 3 equal to 0) of our complex-valuee mapping. In the data view, CosNet can analyze the real-valued data, where the complex-valuee representation can be found inherently. The first module of CosNet also can be considered as learning module, which transforms real-valued data into complex- -aaluee valued features by optimizing the learnable frequency matrices Ω and N'. CosNet also can analyze complex-valuee data with the framework that only includes the second module.
Besides, the real-valued spectral kernel mapping is the special
valued representation
Parameters In the complex-valued spectral kernel embedding module, we initialize the real and imaginary parts of weight matrices with the cosine and sine functions, respectively. Compared with CVNNs, which define the weight matrix as W = A + iB, the number of parameters used in CosNet decreases because of our periodic initialization strategy using only A. Compared -ta-taationary spectral kemels, the number of parameters is reduced since there is no need for sampling two different frequency matrices, $2 and S2'
We provide theoretical evidence of the generalization performance of CosNet, showing that CosNet has lower generalization error bound compared the real-valued spectral kernel networks of the same architecture. Concretely, we first bound the covering numbers of different layers in CosNet, followed by comparisons between covering numbers of real-valued spectral kernel networks and that of CosNet, which provide evidence of CosNet's 's improvements in generalization ability. Further, we derive the generalization bound of CosNet based on several theorems Bartlett el al. 2017]; Mohri et al. [2018).
Theoretical results
Theorem Denote the covering number of set S as Na(S, e). X E Rdd x is the input of samples and each sample is d dimmensiiieed. X E RRd is the input of layer I (1> 1) and A is the weight matrix of layer (1 2 2). The other notations remain the same as mentioned above. For different layers, their covering numbers satisfy that
1. In the first layer, Va(2'XX00 S d4 where &: 2 [wylli
max | 112 1.3
2. In layer (1> 1), Na < (2d' + 1)*, where k
W
X
Proof. The proof is relegated to the supplementary material of our paper due to space limitations.
Covering numbers also serves an indicator of models' representation ability, where the larger the covering number the greater the representation ability, but the more difficult it is to get the optimal solution. Note that when the weight matrices are the same, the bound of the covering number of each
layer of multilayer perception (MLP) is (2d' d' d'=1)², where k > | I And that of
real-valued spectral network is (4d' where 1: > || | which is twice large as that of MLP. can be observed that real-value spectral networks improve their representation ability at the cost of much larger covering number bounds and poorer generalization performance. However,

---

CosNet combines the advantages of both MLP and real-valued spectral networks. Compared to MLP, CosNet's representation ability is further improved by bringing complex-valued vaauued representations into the spectral kernel networks, while only the covering number bound of the first layer increases when constant terms are neglected, which has stronger characterization ability and makes it easier to find the optimal solution. Its superiority is even more clear when compared to real-valued spectral kernels such as DSKN, every layer of CosNet has a smaller complexity, which leads to significant difference when it comes to the complexity of the whole network.
Theorem 2.
} be sample data of size n. from distribution D. Given the weight matrices defined before (St) -22, A', A² ..... A.). and they satisfy that ||All| <
Let S {( y1), (æ2, y2), ..., (z yn)}
CG I| by || I at || 1/2 B, d' S W and T (Et= (a) ) /2/3/3/2 IT Ct- And the loss function C(CosNet(x)) y) I M. Then with the probability of at least 1 8. the proposed network CosNet satisfies:
[[[(ossNe((w).)]
8M
In(1/0)
L(CosNell(:)) v.) O
3/2
+ M
(16)
/wtp/////////p...pp In
W a X
+ [n(n)
Proof.
The proof is relegated to the supplementary material of our paper due to space limitations.
4 Experiments
In this section, we first introduce the implementation details containing comparison methods and evaluation datasets. Then we conduct systematical experiments to demonstrate the superiority of the proposed CosNet, especially on the time series classification task.
Datasets To systematically evaluate the performance of our CosNet, we conduct comparison experiments on several typical time-series datasets, including 12 sub-datasets with default training and testing data splitting from the UCR Archive [au et al. (2019) dataset for the classification task and 3 UCI Blake [1998] localization datasets for regression task. The overall statistics of the used datasets are shown in the Supplementary Material
Compared methods We compare the proposed CosNet with several mainstream kernel methods and CVNNs, as follows: SRFF Zhang er al.) [2017a]: Stacked Kernel Network, which stacks random Fourier features with stationary kernels; DSKN Due et al. [2019]: Deep Spectral Kernel Network; DCN Trabelsi et al. [0018]: Deep Complex Network. We compare two variants with different commonly used activation functions, including CReLU (DCN') and modReLU (DCN²); ASKL Li et al. 22200] Automated spectral kernel learning.
Implementation details All the experiments are implemented with Py Paszke et al. 2019 and conducted on a workstation with NVIDIA RTX 3090 GPU, AMD R7-5700X 3.40GHZ 8-core CPU, and 32 GB memory. Each method is trained by ADAM Kingma and Ba (2014) using cross- entropy loss for the classification task and L2 loss for the regression task. The learning rate equals 0.01, and the weight matrix is initialized from a normal distribution (00 0.01). Each model contains five layers, including the input layer, the output layer, and three hidden layers. As exemplified by the time series classification task, the input is a time series (i.e. vector) with a scalar at each time point. The output is the implied feature mapping (i.e. vector), which is used to conduct the classification task. Concretely, the operation in the first layer is defined as Φb FR + dd where d" denotes the dimension of the data. Via & in the first layer, the data result in complex-vall valued representations, which are fed into the CSKE module starting from the second layer. The operation of 1th layer
is defined Tp C Co +1 where d' denotes the number of hidden complex-valuee neuron. After the CSKE module, we obtain the implied complex-valee valued feathers. Moreover, these implied
complex-valued features are condensed into vector form by the operation C 2224" which concatenate real and imaginary parts, to conduct the classification task. Each experiment repeated twenty times with different random seeds. Note that, the width of networks in each dataset depends
7

---

on the length of the time series, respectively. Therefore, the detailed settings of different models are shown in the Supplementary Material.
4.1 Experimental results
Inherently complex-valued representation learning In practice, the observed data is always presented real numbers, while the complex-valued representation can be found inherently in information processing. In CosNet, we propose the CSKG module, which generalizes the spectral kernel mapping in the real number domain to the complex number domain. To show the capability of our method to recover the complex-vale valued representation, we conduct a simulation experiment and compare CosNet with two typical strategies to deal with complex- valued mapping, i.c., Fourier transform (FT) and eliminating the complex part (DSKN). In the experiment, a complex number sequence with 100 points {z, U, + 8t i= is randomly generated the ground truth, and the corresponding real number is given as 2; = |2,]. The first module of CosNet, i.e., CSKG, is used for recovering the given complex numbers {2, [=1 from the real numbers {ar; 1,000 The results shown in Figure 2 from which we can find that CosNet with the complex-valued non-stationary spectral kernel mapping, compared with FT and DSKN, can recover the inherently complex-van representation precisely. In contrast, the oscillations of the recovered sequences in both two domains by FT are intense while the DSKN fails to recover the phase information contained in the imaginary part.
DSKN
real ground brut rell Cred
Srang (wrand muth Imag $780
120
FT
CooNet
ground budi
iMLL
300
320
Figure 2: Comparison of complex-valuee representation learning. The left and right denote the learning of real and imaginary parts, respectively.
Time series classification To verify the effectiveness of our CosNet on the time-sequential data spectral kernel networks and CVNNs on the time series classification task. The results are shown in Table We can observe that our CosNet achieves state- of-the-art performance in all datasets. Specifically, CosNet outperforms other methods impressively and achieves 3% accuracy increment (83.06 + 85.46) on Wine and 2.7% (69.81 - 71.73) on FordB compared with the mainstream kernel methods and CVNNs. Furthermore, these results reveal that the performance of real-valued methods is limited to the data with the inherently complex-valued representation.
analysis, we compared the state-of-ther
Image classification and compression To further explore the representation capacity of our CosNet, we expand the application of CosNet convolutional networks (see the Supplementary Material for details) for image classification and compression tasks using Fashion-MNIST Xiao et al. (2017) and CIFAR10 Krizhevsky and Hinton (2099 datasets. For the classification task, accuracy applied to the metric of performance. For the compression task, we first extract the implicit features through various models, and then we conduct the clustering task based on these extracted features. In this task, Normalized Mutual Information (NMI) and Rand Index (RI) are used as the assessment metrics. All the results are reported in Table 2 We can find out that our CosNet outperforms the baseline methods on both classification task and compression task. Notadly, our CosNet achieves
8

---

Table 1: Classification accuracy (%) of each compared method on several time series datasets. The best results are highlighted in bold.
Dataset
SRFF
DSKN
DCN
DCN²
ASKL
CosNet
FordA
81.46
82.24
81.87
79.90
72.66
82.42
FordB
68.99
69.81
69.68
50.17
64.20
71.73
PhalangesSuutineSCooree
68.77
69.73
68.91
67.63
68.65
70.79
Wine
77.22
76.48
83.06
80.00
67.41
85.46
ECG200
73.40
77.80
89.80
89.85
87.53
90.10
ECG5000
91.98
91.14
93.11
93.50
92.75
93.70
Herring
57.73
56.64
65,23
58.13
59.52
65.39
Ham
51.52
48.81
71.10
67.76
68.52
71.29
Proxima/Phallaxxuuttinnggggg
79.49
79.51
81.80
81.59
80.00
82.71
Table Classification and compression results on image datasets. The best results are highlighted in bold.
Rashion-MNISS DCN2
CIFAR1O DCN
DCN
CosNet
DCN1
CosNet
Accuracy (%)
87.02
84.44
88.33
64.32
52.39
66.51
NMI (%)
86.04
81.31
90.86
57.14
41.69
66.18
RI (%)
97.15
95.97
98.31
87.78
84.07
89.97
1.5% accuracy improvment (87.02% - 88.33%), 5.6% NMI improvement (86.04% 90.86%)
1.19% RI improvement 95% + 98.13%) on FMNIST dataset, and 3.4% accuracy improvement (64.32% + 6..51. 15.82% NMI improvement (57.14% - 66.18%). 2.5% RI improvement (88.78% - 89 89.97% on CIFAR- 10 dataset. The results show that our CosNet has a greater representation capability than other complex-valued convolutional networks.
4.2 Ablation Study
Complex-valued representation capability Complex numbers, containing amplitude and phase information, lead rich representation capability. However, the re-introducing complex values brings extra parameters. To demonstrate that performance improvement comes from the powerful representation brought by complex values instead added parameters, we conduct an ablation study to evaluate the methods with different parameters. As shown in Table 3 we compare the proposed CosNet with three variants SRFF and DSKN, i.e., the variant the original paper (normal), the variant with more neurons per layer (wider), and the variant with more layers (deeper).
From Table 3] we can observe that our CosNet commonly performs better with fewer parameters. Without loss of generality, increasing parameters indeed can lead to the improvement of performance on certain datasets, but there still remains a gap compared to our CosNet. Therefore, we can conclude that using real-valued networks in some fields, where complex numbers occur either naturally or by design, still has limitations.
We propose to initialize the complex-valued weight matrix of the second module with the cosine and sine functions for the imaginary and real parts, respectively. This design ensures CosNet retains the property of non-stationary spectral kernels and takes the relative distance of data in the complex number domain without increasing the number of parameters. To explore the role of the designed initialization scheme, we compare the results of classification and regression tasks on CosNet with or without cosine and sine functions. The results reported Table[Jshov that our proposed initialization scheme performs better in all cases, which indicates that non-stationarity is necessary for analyzing the time-sequential data. Furthermore, the experimental results validate the effectiveness of our design in complex-value weight matrix on CosNet.
Initialization

---

Table 3: Classification accuracy (%) and parameters with wider and deeper cases. The best results are highlighted in bold.
Model
Setting
ECG200
ECG5000
Ham
Parameters
Accuracy
Parameters
Accuracy
Parameters
Accuracy
normal
22.25K
73,40
31.49K
91.98
251.12K
68.43
SRFF
wider
69.06K
83,90
84.98K
92.75
686.96K
70.52
deeper
42.35K
65.85
56.91K
92.55
459.22K
60.10
normal
44.42K
77.80
62.80K
91.14
502.1IK
69.76
DSKN
wider
137.99K
80.65
169.62 (69.62K
92.42
1260.001
71.76
deeper
137.99K
80.65
169.62K
92.42
1260.00K
71.76
CosNet
normal 19.65K
90.10
40.75K
93.70
375.14K
74.27
Table 4: Classification accuracy (%)and regression MSE on the benchmark datasets. (f) indicates the larger the better, while (4) indicates the smaller the better. The best results are highlighted bold.
Classification Accuracy (1)
Regression MSE (4)
Earthquakes
DistalPhalanxTW Strawberry
power
concreat
yacht
w/ cos, sin
71.76
63.60
97.22
0.8229
1.3606
3.6270
w/o cos, sin
69.93
63.02
96.80
0.8795
1.3731
3.7932
Conclusion
In this paper, propose complex-valued spectral kernel network (CosNet) with two core modules, ie.. SKMG module and CSKE module. Specifically, as the first module of CosNet, the SKMG module is employed to recover the inherent complex-valuce representation of the real-valued data. The CSKE module, designed by embedding the complex-valus -valucd spectral kernel mapping into neural networks with initialization scheme, is used effectively capture long-range periodic relations of data. Our proposed CosNet, benefiting from the non-stationary property of kernels, can effectively encode the dynamic input- deut-deeendent characteristics and long-range correlations. The complex- -valudd mapping can improve the representation capacity of models without increasing the number of param- eters. Furthermore, CosNet involves the transformation of the real-valued inputs in the optimization process to learn an expressive complex-valued representation. Moreover, some theoretical analyses of CosNet also presented. Detailed experiments reveal that our proposed approach indeed leads to significant performance improvements over state-of-the- relevant methods. Future work will focus on promoting the proposed CosNet more applications.
Limitation CosNet with the periodic function prone to local minima. However, our CosNet tends to perform well in the time-sequential data analysis since it can not only capture the long-range relation in an input-dependent manner but also take the imaginary part into account. In future work, will focus on promoting the proposed CosNet in the optimization method.
Acknowledgments
This work was supported by the National Natural Science Foundation China (Nos. 62076062 and 62306070) and the Social Development Science and Technology Project of Jiangsu Province (No. BE202281 I). Furthermore, the work was also supported by the Big Data Computing Center of Southeast University.
References
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In Proceeding of International Conference Machine Learning, volume 48, pages 1 1120-112 1128, 2016.
Peter I. Bartlett, Dylan Foster, and Matus J Telgarsky. Spectrally-normalizzdd margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
10

---

Yoshua Bengio, Pascal Lamblin, Dan Popovici. and Hugo Larochelle. Greedy layer-wise training of deep networks. Advances in neural information processing systems, 19, 2006.
Catherine Blake. Uci repository of machine learning databases. http:///ww. ics. uci. edu/ melaann/eRRppssiory, html, 1998.
Salomon Bochner et al. Lectures on Fourier integrals, volume 42. Princeton University Press, 1959.
Hyeong-Seok Choi, Jang-Hyun Kim, Jaesung Huh, Adrian Kim, Jung- Woo Ha, and Kyogu Lee. Phase-aware speech enhancement with deep complex u-net. In International Conference on Learning Representations, 2019.
David Roxbee Cox and Hilton David Miller. The theory of stochastic processes. Routledge, 2017.
Ivo Danihelka, Greg Wayne, Benigno Uria, Nal Kalchbrenner, and Alex Graves. Associative long short-term memory. In International Conference on Machine Learning, volume 48, pages 1986- 1994, 2016.
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chin Michael Yeh, Yan Zhu, Shaghayegh Gharghabi. Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series archive. IEEF/CAA Journal of Automatica Sinica, ((6):1293-130) 2019.
Muneer Ahmad Dedmari, Sailesh Conjeti, Santiago Estrada, Phillip Ehses, Tony Stöcker, and Martin Reuter. Complex fully convolutional neural networks for mr image reconstruction. In Machine Learning for Medical Image Reconstruction: First International Workshop, MLMIR 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 1, pages 30-38. Springer, 2018
Akira Hirose, Ryosho Nakane, and Gouhei Tanaka. Keynote speech: Information processing hardware, physical reservoir computing and complex-valued neural networks. In 2019 IEEE International Meeting for Future of Electron Devices, Kansai (IMFEDK), pages 19-24. IEEE, 2019.
Akira Hirose. Proposal of fully complex-valie neural networks. In (Proceedings 1992| IJCNN International Join! Conference Neural Networks, volume 4, pages 152-157. IEEE, 1992.
Akira Hirose. Complex-valued neural networks: theories and applications, volume 5. World Scientific, 2003.
M Soo Kim and Clark Guest. Modification of backpropagation networks for complex- valuud signal processing in frequency domain. In /990 IJCNN International Joint Conference on Neural Networks, pages 27-31. IEEE, 1990.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 2014.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases, 1(4), 2009.
Miguel
Joaquin Quinonero-Candela. Carl Edward Rasmussen, and Aníbal R Figueiras-Viide Sparse spectrum gaussian process regression. The Journal of Machine Learning Research, 1 11:1665-18811 2010.
Jian Li, Yong Liu, and Weiping Wang. Distributed learning with random features. arXiv preprint 2019.
Jian Li, Yong Liu, and Weiping Wang. Automated spectral kernel learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4618-4625, 2020.
Fanghui Liu, Xiaolin Huang, Yudong Chen, and Johan AK Suykens. Random features for kernel approximation: A survey algorithms, theory, and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):7128- 2021.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT
press, 2018

---

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style. tigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
C. A. Popa. Complex-valuee convolutional neural networks for real- image classification. In International Joint Conference on Neural Networks, 2017.
A. Rahimi and B. Recht. Random features for large scale kernel machines. In Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 3-6, 2007, 2007.
David P Reichert and Thomas Serre. Neuronal synchrony in complex-valued deep networks. arXiv preprint arXiv:1312.6115 2013.
Sami Remes, Markus Heinonen, and Samuel Kaski. Non-stationary spectral kernels. Advances in neural information processing systems, 30, 2017.
Izhak Shafran, Tom Bagby, and R. J. Skerry-Ryan. Complex evolution recurrent neural networks (cernns). In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing 2018.
(ICASSP), pages
Anthony Tompkins, Rafael Oliveira. and Fabio T Ramos. Sparse spectrum warped input measures for nonstationary kernel learning. Proceedings of Advances in Neural Information Processing Systems, pages 16153-16111 16164, 2020.
Jean-Francois Ton, Seth Flaxman, Dino Sejdinovic, and Samir Bhatt. Spatial mapping with gaussian processes and non-stationary fourier features. Spatial statistics, 28:59-78, 2018.
Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, Joao Felipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher J Pal. Deep complex networks. In International Conference on Learning Representations, 2018.
Théo Trouillon and Maximilian Nickel. Complex and holographic embeddings of knowledge graphs: comparison. arXiv preprint 177:::1707 2017.
Hirofumi Tsuzuki, Mauricio Kugler, Susumu Kuroyanagi, and Akira Iwata. An approach for sound source localization by complex-valued neural network. IEICE TRANSACTIONS on Information and Systems, 96( 96(10)22257-2266 2013.
Shanshan Wang, Huitao Cheng, Ziwen Ke, Leslie Ying, Xin Liu, Hairong Zheng, and Dong Liang. Complex-valued residual network learning for parallel mr imaging. In Proc. 26th Annu. Meeting of ISMRM. 2018.
Xab Wen, B Gma, Zab Feng, C HI, and Zab Lu. Polsar image classification via novel semi- supervised recurrent complex-valued convolution neural network. Neurocomputing, 388:255-268, 2020.
Michael Wilmanski, Chris Kreucher, and Alfred Hero. Complex input convolutional neural networks for wide angle sar atr. In 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pages 1037-1041, IEEE, 2016.
Andrew Wilson and Ryan Adams. Gaussian process kernels for pattern discovery and extrapolation. In International conference on machine learning, pages 1067-1075. PMLR, 2013.
Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux, and Les Atlas. Full-capacity unitary recurrent neural networks. Advances in neural information processing systems, 29, 2016.
Moritz Wolter and Angela Yao. Complex gated recurrent neural networks. Advances in neural information processing systems, 31, 2018.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5028-5037, 2017.
12

---

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07744 2017.
Hui Xue, Zheng-Fan Wu, and Wei-Xiang Sun. Deep spectral kernel learning. IJCAI, pages 4019-4025, 2019.
Akiva M Yaglom. Correlation Theory of Stationary and Related Random Functions, Volume Basic Results, volume 131. Springer, 1987.
Sheng-Sung Yang. Ch Ho, and Sammy Siu. Sensitivity analysis of the split-complex valued multilayer perceptron due the errors of the iid inputs and weights. IEEE transactions on neural networks, 18(5):1280-1299 2007.
Bin Yang, Wei Zhang, Li-Na Gong, and Huai-Zh Zhi Ma. Finance time series prediction using complex- valued flexible neural tree model. 2017 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), pages 54-58, 2017.
Mugiao Yang. Martin Q. Ma, Dongyu Li, Yao-Hung Hubert Tsai, and Ruslan Salakhutdinov. Complex transformer: A framework for modeling complex-valee valued sequence. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4232-4232 2020.
Lingjuan Yu, Yuehong Hu, Xiaochun Xie, Yun Lin, and Wen Hong. Complex-valued full convolu- tional neural network for target classification. IEEE Geoscience and Remote Sensing Letlers, 17(10):1522-175, 2019.
Zhiqiang Zeng. Jinping Sun, Zhu Han, and Wen Hong. Sar automatic target recognition method based on multi-stream complex- valued networks. IEEE Transactions on Geoscience and Remote Sensing, 60: 1-18, 2022.
Huisheng Zhang, Chao Zhang, and Wei Wu. Convergence of batch split-complex backpropagation algorithm for complex-valuee neural networks. Discrete Dynamics in Nature and Society, 2009, 2009.
Shuai Zhang, Jianxin Li, Pengtao Xie, Yingchun Zhang, Minglai Shao, Haoyi Zhou, and Mengyi Yan. Stacked kernel network. arXiv preprint arXiv:1711.09211, 2017.
Zhimian Zhang, Haipeng Wang, Feng Xu, and Ya-Qiu Jin. Complex-valuce convolutional neural net- work and its application in polarimetric sar image classification. IEEE Transactions on Geoscience and Remote Sensing, 55(12);1177-7188 2017.
13


Reviews:
[{"review_id": "24CQl34M0w", "reviewer": "Reviewer_gGnJ", "summary": "This paper mainly focuses on the issue that spectral kernel-based methods often eliminate the imaginary part when analyzing the characteristics of time-sequential data. This limits the representation capability of the spectral kernel. To address this issue, the authors propose a complex-valued spectral kernel network to take both the real and imaginary parts into account. The proposal mainly consists of two parts - the SKMG module recovers the complex-valued representation for the real-valued data and the CSKE module combines the complex-valued spectral kernels and neural networks. Theoretical and empirical results show the proposed method achieves state-of-the-art performance.", "questions": "See weakness.", "limitations": "Yes, the authors have addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is well-motivated. Involving the imaginary part is critical for preserving the amplitude and phase information for data and improving the representational capability of spectral kernel networks. \n2. The proposed approach is sound. The complex-valued representation for the real-valued data is recovered by the SKMG module and complex-valued spectral kernels are combined with neural networks via the CSKE module. The two parts are well integrated. \n3. Theoretical analysis and experimental evaluation are provided to show the state-of-the-art performance of the proposal.", "weaknesses": "1. In Section 3.3, the authors define the complex-valued weight matrix by Equation 11. But is unclear why this design ensures that the sub-network containing the first layer to arbitrary l-th layer is seen as a spectral kernel.\n2. The experimental results appear to be dependent on the choices of hyperparameters. Performance of CosNet with different learning rates, initializations and layer numbers should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: In Section 3.3, the authors define the complex-valued weight matrix by Equation 11. But is unclear why this design ensures that the sub-network containing the first layer to arbitrary l-th layer is seen as a spectral kernel.**\n\n**Response:**\nThanks for this valuable comment. For our CosNet, the first layer (*i.e.*} SKMG module) is constructed from the Yaglom’s theorem,  naturally resulting in a spectral kernel. We ensure the subnetwork from the first layer to the $l$-th layer is a spectral kernel by elaborating the complex-valued weight matrix and the feed-forward procedure which is explained in Equation (12). Upon expanding Equation (12),  it becomes apparent that the sub-network containing the first layer to arbitrary $l$-th layer is seen as a spectral kernel with the defined complex-valued weight. To enhance clarity, we present an illustrative example in $\\mathbb{C}^2$ to show more details in the below, and a more detailed explanation will be added to the revised manuscript.\n\n**Example:**\n\nLet \n$$\n\\pmb{z} = \\begin{bmatrix}\n    \\cos(u_{11}) + \\cos(u_{11}') \\\\\\\\\n    \\cos(u_{21}) + \\cos(u_{21}')\n\\end{bmatrix} + i \\begin{bmatrix}\n    \\sin(v_{11} + \\sin(v_{11}') \\\\\\\\\n    \\sin(v_{21}) + \\sin(v_{21}')\n\\end{bmatrix} \\in \\mathbb{C}^2\n$$\nbe the output of first layer. The complex-valued weight matrix is defined as $\\pmb{W}=\\cos(\\pmb{A}) + i\\sin(\\pmb{A})$, where $\\pmb{A}=[a_{11}, a_{12}] \\in \\mathbb{R}^{1 \\times 2}$ is a real-valued matrix. The complex-valued mapping is defined as:\n\n$$\n   \\Psi(\\pmb{z}) = \\pmb{W}*\\pmb{z} \n   =(\\cos(\\pmb{A})+i\\sin(\\pmb{A})) * (\\begin{bmatrix}\n     \\cos(u_{11}) + \\cos(u_{11}')  \\\\\\\\\n     \\cos(u_{21}) + \\cos(u_{21'})  \n\\end{bmatrix} + i\\begin{bmatrix}\n     \\sin(v_{11}) + \\sin(v_{11}')  \\\\\\\\\n     \\sin(v_{21}) + \\sin(v_{21}'))\n     \\end{bmatrix})\n$$\nWe rewrite the complex-valued mapping as the following matrix notation:\n\n$$\n\\Psi(\\pmb{z})\n=\\begin{bmatrix}\n     \\cos(\\pmb{A}) & -\\sin(\\pmb{A}) \\\\\\\\\n     \\sin(\\pmb{A}) & \\cos(\\pmb{A}) \n\\end{bmatrix} * \\begin{bmatrix}\n     \\Re(\\pmb{z}) \\\\\\\\\n     \\Im(\\pmb{z})\n\\end{bmatrix}\n=\\begin{bmatrix}\n     \\cos(a_{11}) & \\sin(a_{11}) \\\\\\\\\n     \\cos(a_{12}) & \\sin(a_{12}) \\\\\\\\\n     -\\sin(a_{11}) & \\cos(a_{11}) \\\\\\\\\n     -\\sin(a_{12}) & \\cos(a_{12})\n\\end{bmatrix}^\\top * \n\\begin{bmatrix}\n     \\cos(u_{11}) + \\cos(u_{11}')  \\\\\\\\\n     \\cos(u_{21}) + \\cos(u_{21}')  \\\\\\\\\n     \\sin(u_{11}) + \\sin(u_{11}') \\\\\\\\\n     \\sin(u_{21}) + sin(u_{21}')\n\\end{bmatrix}\\\\\n= \\begin{bmatrix}\n    \\Psi_{a_{11},u_{11},u_{11}'} + \\Psi_{a_{12},u_{21},u_{21}'} \\\\\\\\\n    \\Psi_{a_{11},v_{11},v_{11}'}' +\n    \\Psi_{a_{12},v_{21},v_{21}'}' \n\\end{bmatrix}\n$$\nwhere $\\Psi_{a_{11},u_{11},u_{11}'}=\\cos(a_{11}+u_{11})+\\cos(a_{11}+u_{11}')$, $\\Psi_{a_{12},u_{21},u_{21}'}=\\cos(a_{12}+u_{21})+\\cos(a_{12}+u_{21}')$, $\\Psi_{a_{11},v_{11},v_{11}'}'= \\sin(a_{11}+v_{11})+\\sin(a_{11}+v_{11}')$, and $\\Psi_{a_{12},v_{21},v_{21}'}'= \\sin(a_{12}+v_{21})+\\sin(a_{12}+v_{21}')$.\n\nWe can observe that $\\Psi_{a_{11},u_{11},u_{11}'}$, $\\Psi_{a_{12},u_{21},u_{21}'}$, $\\Psi_{a_{11},v_{11},v_{11}'}'$, and $\\Psi_{a_{12},v_{21},v_{21}'}'$ can be seen as two separate spectral kernel. Hence, the sub-network containing the first layer to arbitrary $l$-th layer also is a spectral kernel. \n\n\n\n\n**Q2: The experimental results appear to be dependent on the choices of hyperparameters. Performance of CosNet with different learning rates, initializations and layer numbers should be provided.**\n\n**Response:**\nThanks for this great comment. To ensure the reproducibility of our experimental findings, we unify the hyper-parameters, and the partial updated results (under the same learning rate (0.01), initialization (p = 0.01), and layer numbers (5)) are reported in the following Table. All the related reults will be reported in the revised manuscript.\n| Dataset                      | SRFF   | DSKN   | \\(DCN^1\\) | \\(DCN^2\\) | ASKL   | CosNet |\n|-----------------------------|--------|--------|----------|----------|--------|--------|\n| FordB                       | 68.99  | 69.81  | 69.68    | 50.17    | 64.20  | **71.73**  |\n| Wine                        | 77.22  | 76.48  | 83.06    | 80.00    | 67.41  | **85.46**  |\n| ECG200                      | 73.40  | 77.80  | 89.80    | 89.85    | 87.53  | **90.10**  |\n| ECG5000                     | 91.98  | 91.14  | 94.11    | 93.50     | 92.75  | **93.70**   |\n| Herring                     | 57.73  | 56.64  | 65.23    | 58.13    | 59.52  | **65.39**  |\n\nFurthermore, to evaluate the generalization of our CosNet, we explore the influence of varying hyper-parameters on the result based on ECG200 dataset. The result shows the superior performance and stability of our CosNet. Please see the results in the following table for more details, and we will include  these results  and more analysis in the revised manuscript. \n\n| lr     | init (p) | SRFF   | DSKN   | \\(DCN^1\\) | \\(DCN^2\\) | ASKL   | CosNet  |\n|--------|----------|--------|--------|----------|----------|--------|---------|\n| 0.1    | 1        | 64     | 60.85  | **90.30**| 83.15    | 61.00  | 86.05   |\n| 0.1    | 0.1      | 85.55  | 61.50  | **90.30**| 83.15    | 80.20  | 85.85   |\n| 0.1    | 0.01     | 78.25  | 62.80  | **90.30**| 83.15    | 74.10  | 85.05   |\n| 0.01   | 1        | 52.60  | 64.00  | 88.10    | 84.05    | 75.00  | **90.05**|\n| 0.01   | 0.1      | 85.40  | 67.95  | 88.10    | 84.05    | 89.75  | **91.30**|\n| 0.01   | 0.01     | 73.40  | 77.80  | 88.10    | 84.05    | 87.53  | **90.10**|\n| 0.001  | 1        | 50.85  | 62.75  | 80.35    | 79.25    | 72.55  | **89.25**|\n| 0.001  | 0.1      | 83.50  | 73.00  | 80.35    | 79.25    | **90.90**| 90.25   |\n| 0.001  | 0.01     | 64.00  | 84.40  | 80.35    | 79.25    | 88.90  | **90.45**|\n\n\n\n\n\n\n"}, {"review_id": "RmMTb13z6N", "reviewer": "Reviewer_YSFk", "summary": "The paper aims to extend the reach of kenel-based inference for time series by using a hilbert space over a complex field rather than working over the reals, by not discarding the phase component of the implied spectral representation of the kernels, which apparently is a common strategy. This method makes it feasible to define a broader class of kernels than the obvious (.e.g stationary)", "questions": "There are a lot of words about the usefulness of the complex-values covariance kernel which I think I can interpret but I do not actually know for sure, because the actual inference problem is not clearly set up. What is it? What is the baseline?\n\nI think my confusion starts in l122. Everything up to this point was fine, but now we have deployed the machinery of a complex-valued spectrum in  kernel definitions, without actually explaining what is different. there should be certain things that are better for a complex kernel from the Yaglom theorem. For one, a kernel so defined does needs to be stationary, as opposed to a Bochner-theorem style kernel, which I assume is  the main point, and very cool and indeed the authors mention this right in the abstract. So I like this! I've looked at Yaglom's theorem before and though tit would be great to make it tractable to use but I didn't see a way. Indeed this should be a very general kernel, and should generalise other kernel classes too (e.g. dot-product kernels). So the idea sounds promising.\nEven here though, I'm a little confused; OK, so we are using this kernel not as, say, a covariance kernel of a Gaussian process, but rather to directly define a similarity between data points for optimal interpolation, as far as I can tell, which is fine, but can you say more about that actual implied network structure? Do we keep around the training data so we can measure the kernel-similarity to other training points, or are we happy to use it as \"just another\" nonlinear NN layer. In which case, what does this layer do that an MLP does *not* do?\n\nin eq (10) we learn that the kernel is characterised by a finite list  $\\Omega$ of frequencies, right? Is it correct that this means that we are restricting our kernel spectral \"density\" $\\left(\\boldsymbol{\\omega}, \\boldsymbol{\\omega}^{\\prime}\\right)$ to be not a density as such but  but rather to be a collection of dirac deltas in the spectral space? I suspect we need to say so, in that case.\n\nl147/eq(8) the $M$ suddenly appeared and it looks important but is not explored. This is the number of MC samples we take to actually approximate the spectral kernel integral. So... when do we evaluate this integral? Is it inside the training loop? How do we choose $M$? is it robust against different choices of $M$? Should we not see $M$ pop up in evaluating the computational cost of this method.\n\nCan we put a simple function map notation description of *every operator and function*, e.g. $\\Phi_\\ell: \\mathbb{C}^{d_{\\ell}}\\to\\mathbb{C}^{d_{\\ell+1}}$?  Generally, for most functions in this paper I'm confused what the are mapping from and to. I have so many questions here that I cannot list them all. Here are some examples about function domains and ranges:\n\n1. Are we permitting the values of the kernel outputs in the intermediate layers to be complex? \n2. which values are permitted to be vectors and which scalars? I gather the inputs to the network are vectors (?) but the test examples seem to be all time series with a scalar index (?) In section 4,1a the experiments seem to be about estimating complex valued time series, and in section 4.1b it is classifying stuff based on the implied feature mapping. Is the model more general than this?\n\nIf the authors wish to remain very general, fine, but if so, could we have a running example that makes it clear? \n\nSince we also know that NNs exist which do not parameterise their output in terms of parameters of kernels, but directly in terms of _weights_, could we say anything about the relative representation power compared to that?\nThere is Theorem 1, which gives us a statistical learning theory result in terms of covering numbers, which is possibly supposed to help us. Perhaps the problem here is my own ignorance, but what is the equivalent results for the baseline that the authors hope to surpass? Can you quote an equivalent theorem for a baseline? Should I know one? Or is the intent here to show us how to trade off between allocating weights to _including more frequencies in a given kernel layer_ versus _adding more layers_? Can you spell out the implications of this theorem in terms of \"wide versus deep complex kernel networks\" and also \"whether complex kernel networks are better than MLPs\"?\n\nRelated: Why do we need to stack layers at all, rather than simply learning a big kernel? \n\n4.1: There is a neural network here. I support the authors not wasting space with excessive details about Adagrad learning rates etc, but I need just a little more information to know what is happening. What is the training procedure? If I am used to ERM for training an NN, do I need to worry about some alternative methods for these exotic kernel networks, or is it the same? If it is the same, why am I thinking about the kernels directly rather than just learning an MLP? remember, since the network has been non-specific throughout the paper, this is my example to learn what actual inputs and outputs this network can predict upon.", "limitations": "I don't know. I have a hard time deducing exactly the domain of applicability of the paper from the presentation here.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The paper seems to propose a novel way of characterising flexible covariance kernels such that they are PSD, and some neural networks that exploit this", "weaknesses": "This paper is hard to read. The main claims are difficult to extract and thus their correctness is hard to verify.\nI could be persuaded that I have underestimated the soundness of the results --- they could in fact be amazing --- but I have already blown my limited time budget trying to understand what is going on, because basic framing information is missing. \nI presume that some of this could be deduced by inspecting the references introduced in section 2, but I don't have time for that in the setting of urgent conf reviews. the paper needs to be self-contained, even if it we defer rigorous proof to the appendices.\n\nMy certainty rating reflects my relatively high confidence that this paper needs a rewrite. I do not have a high confidence that the idea itself is flawed, and would welcome clarifications that showed me what the idea actually is, because it might be great. I currently do no understand it, because the presentation is confusing enough that it will take more than the time I have to \"reverse engineer\" it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments! We will provide a point-to point response in the rebuttal.\n\n**1 Notations** In this paper, the matrices, vectors and scalars are denoted\nby bold capital letters (*e.g.* $\\pmb{X}$), bold lower-case letters (*e.g.* $\\pmb{x}$) and lower-case letters (*e.g.* $x$), respectively. In addition, for each equation or function, we will includ its domain and range in the revised manuscript. \n\n**2 Network architecture and experiment setting:**  In the inference procedure, as exemplified by time series classification task, the input is a time series (*i.e.* vector) with a scalar at each time point. The output is the implied feature (*i.e.* vector), which is used to conduct the classification task. Concretely, the operation in the first layer is defined as $\\Phi:\\mathbb{R}^{d^x}\\rightarrow \\mathbb{C}^{d^x}$, where $d^x$ denotes the dimension of the data. Via $\\Phi$ in the first layer, the data result in complex-valued representations, which are fed into the CSKE module starting from the second layer. The operation of $l^{th}$ layer is defined as $\\Psi^l: \\mathbb{C}^{d^l}\\rightarrow \\mathbb{C}^{d^{l+1}}$, where $d^l$ denotes the number of hidden complex-valued neuron. After CSKE module, we obtain the implied complex-valued feathers. Moreover, these implied complex-valued features are condensed into vector form by the operation $\\mathbb{C}^{d^L}\\rightarrow \\mathbb{R}^{2d^L}$, which concatenate real and imaginary parts, to conduct the classification task.\n\nIn the experiment, the learning rate, epoch and layer number are set as 0.01, 500 and 5, respectively. The batch\\_size is equal to number of samples and the width of networks in each dataset depends on the length of time series. The initialized weight matrices are sampled from the normal distribution $\\mathcal{N}(0, 0.01)$. The detalied information will be shown in the updated paper.\n\n**3 Kernel or nonlinear NN layer:**  Note that, in CosNet,  the kernel is defined using explicit kernel mapping rather than the covariance matrix. Specifically, Yaglom's theorem establishes a connection between a kernel and its spectral density. Basen on Monte Carlo random sampling, we can approximate the kernel with a explicit kernel mapping within eq(8). Our CosNet is constructed via stacking the explicit kernel mapping with multiple layers. Notably, in eq(8), we do not need to caculate the integral in the training process. Here, $M$ is the number of MC samples, also referred to as the count of frequencies, and it equal to the numbers of feature in this paper. Moreover, in order to explore the influence of $M$ on the result, we conduct the experiment based on our CosNet, and $M$ is set in the range $[\\frac{d^x}{4}, \\frac{d^x}{2}, d^x, 2d^x, 4d^x]$. The result shows that while the value of $M$ does exert some influence on the results, the effect is relatively minor. Please see TABLE I in the added pdf file for more detailed results, and  it will be shown in the revised paper.\n\n**4 Complex-valued kernel network versus MLPs:**  For the framework generality, our CosNet demonstrates the capability to analyze not only complex-valued data but also real-valued data that inherently include complex-valued information. In contrast, MLPs confined to analyzing solely real-valued data. Theoretically, our CosNet has greater representation ability compared to MLPs. Concretely, we bound the covering number of different layers in CosNet. Covering numbers also serves as an indicator of models’ representation ability, where the larger the covering number the greater the representation ability, but the more difficult it is to get the optimal solution. In Theorem 1, the covering number of each layer is bounded by $(2d^ld^{l-1})^k$ and $(4d^ld^{l-1})^k$ in MLPs and real-valued non-stationary spectral kernel networks, respectively. This provides insight into the comparison: 1) Compared with MLPs, our CosNet has greater representation ability with the bound of $(4d^ld^{l-1})^k$ covering numbers in the first layer; 2) Compared with real-valued spectral kernel networks, our CosNet makes it easier to find the optimal solution with the bound of $(2d^ld^{l-1})^k$ covering numbers from second layer. Therefore, our CosNet combines the advantages of kernel networks and MLPs, which has stronger characterization ability and is easier to find the optimal solution. We will show more analysis in the revised paper.\n\n**5 Why do we need to stack layers at all, rather than simply learning a big kernel?** It is important to note that traditional kernel methods are confined to learning a single layer of nonlinear features, potentially constraining their representational capacity. Inspired by neural networks which learn multi-layer hierarchical representations, deep kernel (stacked kernel) is developed to learn hierarchy within Reproducing Kernel Hilbert Space, yielding a cascade of nonlinear features. Therefore, stacked kernel combines the advantags of kernel and neural networks. For a more comprehensive understanding, we refer interested readers to the detailed explanations provided in the reference titled 'Stacked Kernel Network'. Moreover, we perform a comparative analysis using ECG200 dataset between stacked kernel networks and their corresponding big kernels with 1024 Monte Carlo samples. The result show that the stacked kernel performs better than a big kernel with single layer. Please see TABLE II in the added pdf file for more detailed results, and it will be shown in the revised manuscript.\n\n**6 in eq (10) we learn that the kernel is characterised by a finite list $\\pmb{\\Omega}$ \n of frequencies, right? Is it correct that this means that we are restricting our kernel spectral \"density\" $(\\pmb{\\omega}, \\pmb{\\omega}')$ \n to be not a density as such but but rather to be a collection of dirac deltas in the spectral space? I suspect we need to say so, in that case.**  Yes, you are right! We will statement this point in the revised manuscript.\n\n"}, {"review_id": "pEZH13Qf9x", "reviewer": "Reviewer_BAqh", "summary": "The paper proposes a new framework called Complex-valued spectral kernel network (CosNet) that generalizes the spectral kernel to include complex-valued representation. The proposed framework improves the representational capability of the spectral kernel and outperforms existing kernel methods and complex-valued neural networks. An initialization scheme for the complex-valued weight matrix is proposed, which ensures that CosNet retains the property of non-stationary spectral kernels and takes the relative distance of data in the complex number domain without increasing the number of parameters. The paper provides the lower generalization bound of CosNet than the real-valued non-stationary spectral kernel. The experiments demonstrate that CosNet performs better than the mainstream kernel methods and complex-valued neural networks in time-sequential data analysis.", "questions": "1. The paper presents experiments comparing the accuracy of CosNet to other methods with similar network size on the same dataset, highlighting the capacity gain from introducing the imaginary part. However, I believe that accuracy alone may not fully demonstrate the improvement of the model's representation capacity. Could the authors provide more evidence of CosNet's capacity gain from other aspects, such as compression and recovery of more complex datasets?\n2. The experiments in the paper adopt time-series datasets with low complexity. Could the authors provide more results that demonstrate the performance of CosNet on tasks such as image encoding and decoding or the signal processing in real-world scenarios, since these datasets may also be suitable cases for complex-valued spectral kernel modeling.\n3. In Figure 2, the authors only present the visualization results of two other methods for predicting and displaying the original time series curve. Could the authors provide additional visualizations of the performance of other comparable methods for comparison? This would provide a more comprehensive and fair comparison of the proposed method against existing ones.", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a new framework (CosNet) and provides a theoretical analysis of it.\n2. The experiments demonstrate that CosNet outperforms existing kernel methods in time-series data analysis, which shows the practical significance of the proposed framework.\n3. The writing of the paper is good.", "weaknesses": "1. The experiments presented in the paper lack sufficient evidence to convincingly demonstrate the properties of the model.\n2. The dataset used in the experiments is limited in diversity, which raises concerns about the generalizability of the findings.\n3. The initialization of CosNet's parameters varies across different datasets, and a unified initialization method is needed for consistency and reproducibility.\n\nTypo: In line 168, the imaginary part should also be multiplied by the weight $1/\\sqrt{\\frac{1}{4M}}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The initialization of CosNet's parameters varies across different datasets, and a unified initialization method is needed for consistency and reproducibility.**\n\n**Response:** To ensure the reproducibility of our experimental findings, we unify the hyper-parameters, and the partial updated results (under the same learning rate (0.01), initialization (p = 0.01), and layer numbers (5)) are reported in the following Table. The new results show that our CosNet performs better than the baseline methods, and all the related results will be reported in the revised main paper.\n| Dataset                        | SRFF   | DSKN   | \\(DCN^1\\) | \\(DCN^2\\) | ASKL   | CosNet  |\n|-------------------------------|--------|--------|----------|----------|--------|---------|\n| FordB                         | 68.99  | 69.81  | 69.68  | 50.17  | 64.20  | **71.73** |\n| Wine                          | 77.22  | 76.48  | 83.06  | 80.00  | 67.41  | **85.46** |\n| ECG200                        | 73.40  | 77.80  | 89.80  | 89.85  | 87.53  | **90.10** |\n| ECG5000                       | 91.98  | 91.14  | 94.11  | 93.50   | 92.75  | **93.70**  |\n| Herring                       | 57.73  | 56.64  | 65.23  | 58.13  | 59.52  | **65.39** |\n\n**Q2: The experiments in the paper adopt time-series datasets with low complexity. Could the authors provide more results that demonstrate the performance of CosNet on tasks such as image encoding and decoding or the signal processing in real-world scenarios, since these datasets may also be suitable cases for complex-valued spectral kernel modeling.**\n\n**Response:** On the one hand, we have included the automatic modulation (AM) classification task using the real-world signal dataset RML2016.10a [1], a classical complex-valued signal dataset. We conduct a comparative analysis between our proposed method and baseline approaches across varying signal-to-noise ratios (SNRs) ranges. On the other hand, we expand the application of CosNet to convolutional networks for image classification tasks on the FashionMNIST and CIFAR-10 datasets.  All the results show the effectiveness of our CosNet. Detailed results are provided in the following table for a comprehensive overview and will be incorporated into the revised manuscript.\n| Tasks               | Datasets     | \\(DCN^1\\) | \\(DCN^2\\) | CosNet   |\n|---------------------|--------------|----------|----------|----------|\n| Image classification| FashionMNIST | 87.02    | 84.44    | **88.33**|\n|                     | CIFAR-10     | 64.32    | 52.39    | **66.51**|\n\n| Tasks              | SNRs range | \\(DCN^1\\) | \\(DCN^2\\) | CosNet   |\n|--------------------|------------|----------|----------|----------|\n| AM classification  | 10-18      | 81.14  | 79.98 | **81.89**|\n|                    | 0-8  | 79.27 | 77.45 | **79.70**|\n\n[1] O'shea T J, West N. Radio machine learning dataset generation with gnu radio[C]//Proceedings of the GNU Radio Conference. 2016, 1(1).\n\n**Q3: The paper presents experiments comparing the accuracy of CosNet to other methods with similar network size on the same dataset, highlighting the capacity gain from introducing the imaginary part. However, I believe that accuracy alone may not fully demonstrate the improvement of the model's representation capacity. Could the authors provide more evidence of CosNet's capacity gain from other aspects, such as compression and recovery of more complex datasets?**\n\n**Response:**  To further explore the representation of our CosNet, we conduct more complex task utilizing the FashionMNIST and CIFAR-10 datasets. Concretely, we first extract the implicit features through various models, and then we conduct the clustering task based on these extracted features. In this task, Normalized Mutual Information (NMI) and Rand Index (RI) are used as the assessment metrics. Our CosNet achieves **5.6\\%** NMI improvement (86.04\\% $\\rightarrow $90.86\\%), **1.19\\%** RI improvement (97.15\\% $\\rightarrow$ 98.13\\%) on FMNIST dataset, and **15.82\\%** NMI improvement (57.14\\% $\\rightarrow$ 66.18\\%), **2.5\\%** RI improvement (87.78\\% $\\rightarrow$ 89.97\\%) on CIFARI-10 dataset.  The results show that our CosNet has a greater representation capbility than other complex-valued convolutional networks. Detailed results are provided in the following table. Further, we will include more compression and recovery tasks in the revised manuscript. \n| Dataset  | Metrics | \\(DCN^1\\) | \\(DCN^2\\) | CosNet   |\n|----------|---------|----------|----------|----------|\n| FMNIST   | NMI     | 86.04\\%   | 81.31\\%   | **90.86\\%**|\n|          | RI      | 97.15\\%   | 95.97\\%   | **98.31\\%**|\n| CIFAR-10 | NMI     | 57.14\\%   | 41.69\\%   | **66.18\\%**|\n|          | RI      | 87.78\\%   | 84.07\\%   | **89.97\\%**|\n\n**Q4: In Figure 2, the authors only present the visualization results of two other methods for predicting and displaying the original time series curve. Could the authors provide additional visualizations of the performance of other comparable methods for comparison? This would provide a more comprehensive and fair comparison of the proposed method against existing ones.**\n\n**Response:**  Figure 2 is designed to illustrate the effective capture of inherently complex-valued representations by the first layer of our CosNet, seamlessly feeding them into the subsequent complx-valued networks (*i.e.* the CSKE module). To validate this statement, we utilize DSKN and FT as the baseline methods, driven by two primary reasons. Firstly, Fourier transform is extensively utilized to map real-valued data into complex-valued representations, which serve as inputs for complex-valued networks. Second, DSKN encompasses most existing approaches, where the imaginary part is padded with zeros. While DSKN ignores the imaginary part that is learned from the real-valued data, the other methods directly consider the raw real-valued data and zeros as the real and imaginary parts. As a result, we select these two methods for comparison purposes.\n\n"}, {"review_id": "lJyUqigMKf", "reviewer": "Reviewer_uy3z", "summary": "The authors propose a complex valued neural architecture, composed of two modules: a Spectral kernel mapping generalization module and a Complex-valued spectral kernel embedding module. \nThey provide a generalization error bound for their model. In addition they propose a novel initialization scheme and provide experimental results on several datasets and learning tasks.", "questions": "No", "limitations": "No", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The proposed approach seem to be novel, and the experimental results are promising.", "weaknesses": "In my view, the main weakness of the manuscript is the presentation. While the approach itself seems sound, I find the presentation too poor for a conference like Neurips. Therefore I encourage the authors to re-write the manuscript and make sure it reads much better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: In my view, the main weakness of the manuscript is the presentation. While the approach itself seems sound, I find the presentation too poor for a conference like Neurips. Therefore I encourage the authors to re-write the manuscript and make sure it reads much better.**\n\n**Response:**\nWe sincerely appreciate your time and effort in reviewing our manuscript. In response to your comment regarding the presentation of our manuscript, we would like to bring to your attention that reviewers gGnJ and BAqh have both commended the contributions of our CosNet. They found our methodology and approach to be well-explained and insightful. This suggests that our efforts to enhance the clarity and presentation of our work have been positively received by these reviewers.\nNonetheless, we understand the importance of ensuring the highest level of clarity and coherence throughout the manuscript. We value your more constructive feedback and are dedicated to refining our manuscript based on your comments. We eagerly anticipate the opportunity to incorporate more of your insights into our revision process."}]

Author rebuttal:
 We include additional experimental result in the pdf.

Meta-review: This paper explores a key challenge in spectral kernel-based methods: the elimination of the imaginary part, which limits the kernel's representation when analyzing time-sequential data. To solve this problem, the authors propose a complex-valued spectral kernel network. The experiments demonstrate that CosNet outperforms existing methods. During the discussion, the authors provided a good rebuttal. They presented some preliminary results that addressed the reviewers' concerns. However, one reviewer still remains concerned about the presentation of this manuscript, while the other reviewers have not identified that issue. 
Most parts of this paper are well presented, and the authors should further improve this manuscript based on the rebuttal during the discussion period.
Overall, I recommend accepting this paper.
Decision: Accept (poster)