Title: CosNet: A Generalized Spectral Kernel Network
Venue: NeurIPS 2023
Accepted: yes
Keywords: Spectral kernel; complex-valued networks
Abstract: Complex-valued representation exists inherently in the time-sequential data that can be derived from the integration of harmonic waves. The non-stationary spectral kernel, realizing a complex-valued feature mapping, has shown its potential to analyze the time-varying statistical characteristics of the time-sequential data, as a result of the modeling frequency parameters. However, most existing spectral kernel-based methods eliminate the imaginary part, thereby limiting the representation power of the spectral kernel. To tackle this issue, we propose a generalized spectral kernel network, namely, \underline{Co}mplex-valued \underline{s}pectral kernel \underline{Net}work (CosNet), which includes spectral kernel mapping generalization (SKMG) module and complex-valued spectral kernel embedding (CSKE) module. Concretely, the SKMG module is devised to generalize the spectral kernel mapping in the real number domain to the complex number domain, recovering the inherent complex-valued representation for the real-valued data. Then a following CSKE module is further developed to combine the complex-valued spectral kernels and neural networks to effectively capture long-range or periodic relations of the data. Along with the CosNet, we study the effect of the complex-valued spectral kernel mapping via theoretically analyzing the bound of covering number and generalization error. Extensive experiments demonstrate that CosNet performs better than the mainstream kernel methods and complex-valued neural networks.

CosNet: A Generalized Spectral Kernel Network

Abstract

Complex-valued representation exists inherently in the time-sequential data that can be derived from the integration of harmonic waves. The non-stationary spectral kernel, realizing a complex-valued feature mapping, has shown its potential to analyze the time-varying statistical characteristics of the time-sequential data, as a result of the modeling frequency parameters. However, most existing spectral kernel-based methods eliminate the imaginary part, thereby limiting the representation power of the spectral kernel. To tackle this issue, we propose a generalized spectral kernel network, namely, Complex-valued spectral kernel Network (CosNet), which includes spectral kernel mapping generalization (SKMG) module and complex-valued spectral kernel embedding (CSKE) module. Concretely, the SKMG module is devised to generalize the spectral kernel mapping in the real number domain to the complex number domain, recovering the inherent complexvalued representation for the real-valued data. Then a following CSKE module is further developed to combine the complex-valued spectral kernels and neural networks to effectively capture long-range or periodic relations of the data. Along with the CosNet, we study the effect of the complex-valued spectral kernel mapping via theoretically analyzing the bound of covering number and generalization error. Extensive experiments demonstrate that CosNet performs better than the mainstream kernel methods and complex-valued neural networks.

1 Introduction

Complex numbers represent the information of the amplitude and phase simultaneously. In contrast to the amplitude, which can be revealed by the real number, the phase can denote the time delay and advance, thereby encoding the temporal dependency of the data Hirose [2003]. This suggests that complex-valued models can be employed in some practical applications, especially for the timesequential data where information is wave-related, such as signal analysis Hirose et al. [2019]; Yu et al. [2019]; Zeng et al. [2022], speech processing Shafran et al. [2018], and time series classification Yang et al. [2020, 2017]; Wisdom et al. [2016].

In the learning community, the spectral kernel, which is constructed from the inverse Fourier transform, can naturally realize the complex-valued mapping. That said, the spectral kernel can analyze the data in the frequency domain directly. As a candidate of the spectral kernel family, the non-stationary spectral kernel is also proposed to conquer the local limitations of the classical kernels, such as stationarity and monotonicity Remes et al. [2017]; Tompkins et al. [2020]; Ton et al. [2018]; Li et al. [2020]. Ideally, these advanced kernels can extract appropriate non-stationary time-varying characteristics of the data by modeling the frequency parameters, and hence infer the long-range or periodic relations of the input data.

However, most existing methods usually eliminate the imaginary part of the spectral kernel mapping arbitrarily for the convenience of calculation. For example, Rahimi and Recht [2007]; Zhang et al. [2017a] ignore the imaginary part directly by replacing the integrand e j ω ( x -x ′ ) with cos( ω ( x -x ′ )) ; Xue et al. eliminates the imaginary part with an elaborate spectral density function definition for the non-stationary spectral kernel Xue et al. [2019]. Remarkably, existing research shows that complex numbers could lead to a rich representational capability for wave-related information processing Wisdom et al. [2016]; Danihelka et al. [2016]; Worrall et al. [2017]; Trouillon and Nickel [2017]. However, simply plugging the imaginary part in the neural networks does not ensure that the model retains the property of the spectral kernel. Therefore, more efforts are required to develop a new framework that can involve the imaginary part in spectral kernel networks.

In this paper, we propose a new framework that generalizes the spectral kernel that endows with the complex-value representation, and we name it as a complex-valued spectral kernel network (CosNet). The proposed CosNet includes two modules: the spectral kernel mapping generalization (SKMG) module and the complex-valued spectral kernel embedding (CSKE) module. Technically, we generalize the spectral kernel mapping in the real number domain to the complex number domain by defining the spectral density function in the SKMG module. We further embed the complex-valued spectral kernel into neural networks to attain the proposed CosNet using the CSKE module. It is noted that a new initialization scheme is also proposed for the CSKE module that adopts the cosine and sine functions as the activation for the real and imaginary parts of the weight matrix. This initializing scheme retains the statistical characteristics of the non-stationary spectral kernel. It enables CosNet to take the relative distance of data into account by shifting between phases, such that capture the long-range or periodic relations of data in the complex domain without increasing the number of parameters. Our contributions in this paper are shown as follows:

We propose a complex-valued spectral kernel network, i.e., CosNet, which takes both the real and imaginary parts of the spectral kernel mapping into account and thus improves the representational capability of the spectral kernel.

We propose an initialization scheme for the complex-valued weight matrix, which ensures that CosNet retains the property of non-stationary spectral kernels and takes the relative distance of data in the complex number domain without increasing the number of parameters.

We provide the lower generalization bound of CosNet than the real-valued non-stationary spectral kernel.

Thorough experiments demonstrate that our proposed method is totally superior to state-ofthe-art kernel methods.

2 Related Work

Spectarl kernel networks Spectral approaches were developed to fully characterize general kernels with concise representation forms, such as sparse spectrum kernels Lázaro-Gredilla et al. [2010], sparse mixture kernels Wilson and Adams [2013], non-stationary spectral kernels Remes et al. [2017], and random Fourier features methods to deal with large-scale settings Li et al. [2019]; Liu et al. [2021]. These methods commonly approximated the kernel function using an explicit spectral representation based on Bochner's theorem Bochner and others [1959] and Yaglom's theorem Yaglom [1987]. Benefiting from the outstanding representation capability of neural networks with hierarchical nonlinear linking structures Bengio et al. [2006], researchers attempt to embed spectral representation (i.e., feature mapping of kernels) into the hierarchical architecture of neural networks to construct spectral kernel networks. Zhang et al. [2017a] used the Random Fourier Feature to approach the stationary kernel mapping and embedded it into each layer of DNNs. Xue et al. [2019] proposed a deep spectral kernel network to embed the non-stationary spectral kernel into each layer of DNNs, which can approximate most of the kernels. Li et al. [2020] proposed an automated spectral kernel learning (ASKL) that incorporates the process of finding suitable non-stationary kernels and model training. However, for convenience of calculation, these models commonly use the real-valued representation, although spectral kernels lead to the complex-valued mapping.

Complex-valued neural networks Complex-valued neural networks (CVNNs) have shown excellent efficiency compared to their real counterparts in biological Reichert and Serre [2013], speech enhancement Tsuzuki et al. [2013]; Choi et al. [2019], image Popa [2017]; Wen et al. [2020], and signal processing Kim and Guest [1990]; Wilmanski et al. [2016]. In previous studies, researchers

commonly split the complex-valued input into a pair of real-valued inputs and fed them into the real-valued neural networks with both real-valued weight matrix and activation function. This design cannot exploit the advantages of complex numbers completely, and the neural network convergence strongly depends on proper initialization and the choice of learning rate Yang et al. [2007]; Zhang et al. [2009]. Subsequently, CVNNs with complex-valued weight and activation functions are proposed in the complex number domain to deal with complex-valued inputs Hirose [1992]; Dedmari et al. [2018]; Zhang et al. [2017b]. Benefitting from the rich representation capability, researchers tend to extend CVNNs to other neural networks, such as complex-valued convolutional neural networks Trabelsi et al. [2018], complex-valued residual neural networks Wang et al. [2018], and complex-valued recurrent neural networks Wolter and Yao [2018]; Arjovsky et al. [2016]. All these works have proved that the complex-valued models have a richer representational capacity and perform better on real-world learning tasks by a set of experiments.

3 Complex-valued Spectral Kernel Networks

In this section, we first introduce concepts and notations of the non-stationary kernel and complex numbers. Then, we provide the overall architecture of our CosNet with two modules. Moreover, we explicitly provide the details of each module. In addition, we present a detailed analysis of CosNet.

3.1 Preliminary

To better illustrate CosNet, we introduce the necessary preliminary knowledge and notation of non-stationary spectral kernels and complex numbers in this section.

Notations Formally, we use R n , C n , R m × n and C m × n to denote n -dimensional Euclidean spaces, n -dimensional complex number spaces, the space of m × n real-valued matrix and the space of m × n complex-valued matrix. Throughout the paper, the matrices, vectors and scalars are denoted by bold capital letters ( e.g. X ), bold lower-case letters ( e.g. x ) and lower-case letters ( e.g. x ), respectively. A complex number z ∈ C D is represented as z = u + i v with a real part u and an imaginary part v . ¯ z = u -i v denotes the complex conjugate of z . For any two complex numbers z 1 = u 1 + i v 1 , z 2 = u 2 + i v 2 ∈ C , z 1 + z 2 = ( u 1 + u 2 ) + i ( v 1 + v 2 ) , z 1 z 2 = ( u 1 u 2 -v 1 v 2 ) + i ( u 1 v 2 + v 1 u 2 ) . To represent the complex-valued layer with 2 D features, we allocate the first D features to represent the real component and the remaining to represent the imaginary component.

Preliminary knowledge Non-stationary spectral kernels are constructed from inverse Fourier transform in the frequency domain. Based on Yaglom's theorem Yaglom [1987], a general kernel k ( x , x ′ ) is positive definite on R D if and only if it admits the form:

where µ ( d ω , d ω ′ ) is the Lebesgue-Stieltjes measure associated with some positive semi-definite spectral density function s ( ω , ω ′ ) with bounded variations. Therefore, a general kernel can be defined as the following form:

where s ( ω , ω ′ ) can be understood as a joint probability density function.

3.2 Overall architecture

To explore the capability of the imaginary part in the spectral kernel networks, we propose CosNet as a generalized framework. CosNet involves two modules: the SKMG module to achieve complex-valued spectral kernel mapping and the CSKE module to embed the spectral kernel into neural networks. The overall architecture is shown in Figure 1.

Concretely, the SKMG module is denoted as Φ( x ) via generalizing the spectral kernel mapping in the real number domain to the complex number domain for the real-valued data x . And the CSKE module is denoted as Ψ( h ) via initializing complex-valued weight matrix with the cosine and sine function for the complex-valued spectral kernel mapping h .

i=1

@1()xi

COS

+

×

4M

1

M

i=1

i=1

M

sin(A1(i)Φsi

YR1

Y

x1

D

cos

+

中31

cos(A1(i)Φsi +

sin(A1(i)ΦRi

y1

j=1

1

i=1

i=1

4M

x2

···

1

XD

D

i=1

WM(i)xi

COs

+

4M

PRM

i=1

cos(Ak(t)ΦRi

i=1

M

sin(Ak(t)Φsi

YRK

4K

D

WMXi

COS

cos(Ak(t)Φsi +

sin(Ak(t)ΦRi

YSk

i=1

4M

1

i=1

i=1

+

SKMG

CSKE

Based on the two modules, our CosNet with l layers is defined as :

Moreover, the corresponding complex-valued spectral kernel is defined as:

where K ( l ) ( x , x ′ ) denotes the l -layer complex-valued spectral kernel.

3.3 Complex-valued spectral kernel network (CosNet)

Spectral kernel mapping generalization module In this module, we generalize the spectral kernel mapping in the real number domain to the complex number domain. Furthermore, the generalized mapping can be used in both stationary and non-stationary spectral kernels. Here we elaborate on the detailed process.

According to Equation (2), to produce a positive semi-definite kernel, we need to include symmetries s ( ω , ω ′ ) = s ( ω ′ , ω ) and sufficient diagonal components s ( ω , ω ) and s ( ω ′ , ω ′ ) . Concretely, we replace the exponential component e i ( ω ⊤ x -ω ′⊤ x ′ ) in Equation (2) with ζ ω , ω ′ ( x , x ′ ) , which is defined as:

Then, we expand the exponential component to the complex-valued representation with the cosine and sine function based on Euler's formula, and ζ ω , ω ′ ( x , x ′ ) can be rewritten as:

As a result, the general spectral kernel in Equation (2) can be redefined as:

where p ( ω , ω ′ ) = 1 4 [ s ( ω , ω ′ ) + s ( ω ′ , ω ) + s ( ω , ω ) + s ( ω ′ , ω ′ )] also can be considered as a probability density function.

Subsequently, we approximate Equation (7) with Monte Carlo random sampling:

where, ( ω i , ω i ′ ) D i =1 is the frequency pairs, M is the sampling number.

The generalized spectral kernel mapping in Equation (8) is defined as:

and the frequency matrices Ω , Ω ′ are denoted as:

As a result, we obtain a complex-valued spectral kernel mapping. The real part of the output is denoted as ℜ (Φ( x )) = cos( Ω ⊤ x ) + cos( Ω ′⊤ x ) , and the imaginary part is denoted as ℑ (Φ( x )) = sin( Ω ⊤ x ) + sin( Ω ′⊤ x ) .

Complex-valued spectral kernel embedding module In this module, we attempt to embed the complex-valued spectral kernel into each layer of neural networks to construct CosNet. Spectral kernel, based on the general Fourier analysis, provides a new explicit kernel mapping. These kernels can not only approximate most kernels under specific conditions by some fundamental theorems Cox and Miller [2017]; Yaglom [1987] but also provide an efficient way to combine neural networks with kernel methods to construct spectral networks. Most existing spectral kernel networks commonly embed the spectral kernel into neural networks by stacking the spectral kernel mapping in the hierarchical architecture of neural networks directly. However, the introduction of an imaginary part enables that networks with a simple stack of complex-valued mapping cannot be formulated as a spectral kernel (see the Supplementary Material for details).

To ensure the sub-network containing the first layer to arbitrary l -th layer ( l ⩾ 2) can be integrally seen as a spectral kernel, and following the form of complex-valued parameters in CVNNs, we define the complex-valued weight matrix of this module as:

where A is a real-valued matrix.

In this module, the convolution operates with the complex weight matrix W is defined as:

The real and imaginary parts of the convolution operation are represented in the matrix notation:

To inherit the outstanding representation capability from neural networks, in this module, we construct the spectral kernel networks by stacking Ψ :

where Ψ l (2 ≤ l ) denotes the l -layer complex-valued spectral kernel mapping and

3.4 Analysis of CosNet

CosNet, constructed by stacking the non-station complex-valued spectral kernel mapping, not only retains the property of non-stationary spectral kernels, which can effectively reveal the input-dependence characteristics and long-range relations but also can learn hierarchy within Reproducing Kernel Hilbert Space, yielding a cascade of non-linear features. Besides, CosNet takes the imaginary part of the complex-valued spectral kernel mapping into account, leading to a richer representation capability.

Framework generality In spectral kernels view, CosNet will be reduced to a stationarity spectral kernel when ω = ω ′ in Equation (2). Besides, the real-valued spectral kernel mapping is the special case ( i.e., the imaginary part ℑ Φ( x ) equal to 0 ) of our complex-valued mapping. In the data view, CosNet can analyze the real-valued data, where the complex-valued representation can be found inherently. The first module of CosNet also can be considered as a complex-valued representation learning module, which transforms real-valued data into complex-valued features by optimizing the learnable frequency matrices Ω and Ω ′ . CosNet also can analyze complex-valued data with the framework that only includes the second module.

Parameters In the complex-valued spectral kernel embedding module, we initialize the real and imaginary parts of weight matrices with the cosine and sine functions, respectively. Compared with CVNNs, which define the weight matrix as W = A + i B , the number of parameters used in CosNet decreases because of our periodic initialization strategy using only A . Compared to non-stationary spectral kernels, the number of parameters is reduced since there is no need for sampling two different frequency matrices, Ω and Ω ′ .

Theoretical results We provide theoretical evidence of the generalization performance of CosNet, showing that CosNet has a lower generalization error bound compared to the real-valued spectral kernel networks of the same architecture. Concretely, we first bound the covering numbers of different layers in CosNet, followed by comparisons between covering numbers of real-valued spectral kernel networks and that of CosNet, which provide evidence of CosNet's improvements in generalization ability. Further, we derive the generalization bound of CosNet based on several theorems Bartlett et al. [2017]; Mohri et al. [2018].

Theorem 1. Denote the covering number of set S as N d ( S, ϵ ) . X ∈ R d x × n is the input of n samples and each sample is d x -dimensioned. X l ∈ R d l × n is the input of layer l ( l > 1) and A l is the weight matrix of layer l ( l ≥ 2 ). The other notations remain the same as mentioned above. For different layers, their covering numbers satisfy that

Proof. The proof is relegated to the supplementary material of our paper due to space limitations.

Covering numbers also serves as an indicator of models' representation ability, where the larger the covering number the greater the representation ability, but the more difficult it is to get the optimal solution. Note that when the weight matrices are the same, the bound of the covering number of each layer of a multilayer perception (MLP) is (2 d l d l -1 ) k , where k ≥ || W ij || 2 1 ϵ 2 || X l -1 || 2 1 . And that of real-valued spectral network is (4 d l d l -1 ) k , where k ≥ || W ij || 2 1 ϵ 2 || X l -1 || 2 1 , which is as twice large as that of MLP. It can be observed that real-value spectral networks improve their representation ability at the cost of much larger covering number bounds and poorer generalization performance. However,

CosNet combines the advantages of both MLP and real-valued spectral networks. Compared to MLP, CosNet's representation ability is further improved by bringing complex-valued representations into the spectral kernel networks, while only the covering number bound of the first layer increases when constant terms are neglected, which has stronger characterization ability and makes it easier to find the optimal solution. Its superiority is even more clear when compared to real-valued spectral kernels such as DSKN, every layer of CosNet has a smaller complexity, which leads to a significant difference when it comes to the complexity of the whole network.

Theorem 2. Let S = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , ..., ( x n , y n ) } be a sample data of size n from distribution D . Given the weight matrices defined before ( Ω 1 , Ω 2 , A 1 , A 2 , ..., A L ) , and they satisfy that || A l || ≤ c l , || W l || ≤ b l , || Ω l || ≤ a l , || X || 1 ≤ B , d l ≤ W and T = ( ∑ L l =1 ( b l c l ) 2 / 3 ) 3 / 2 ∏ L l =1 c l . And the loss function L ( CosNet ( x ) , y ) ≤ M . Then with the probability of at least 1 -δ , the proposed network CosNet satisfies:

Proof. The proof is relegated to the supplementary material of our paper due to space limitations.

4 Experiments

In this section, we first introduce the implementation details containing comparison methods and evaluation datasets. Then we conduct systematical experiments to demonstrate the superiority of the proposed CosNet, especially on the time series classification task.

Datasets To systematically evaluate the performance of our CosNet, we conduct comparison experiments on several typical time-series datasets, including 12 sub-datasets with default training and testing data splitting from the UCR Archive Dau et al. [2019] dataset for the classification task and 3 UCI Blake [1998] localization datasets for regression task. The overall statistics of the used datasets are shown in the Supplementary Material.

Compared methods We compare the proposed CosNet with several mainstream kernel methods and CVNNs, as follows: SRFF Zhang et al. [2017a]: Stacked Kernel Network, which stacks random Fourier features with stationary kernels; DSKN Xue et al. [2019]: Deep Spectral Kernel Network; DCN Trabelsi et al. [2018]: Deep Complex Network. We compare two variants with different commonly used activation functions, including C ReLU ( DCN 1 ) and modReLU ( DCN 2 ) ; ASKL Li et al. [2020]: Automated spectral kernel learning.

Implementation details All the experiments are implemented with PyTorch Paszke et al. [2019] and conducted on a workstation with NVIDIA RTX 3090 GPU, AMD R7-5700X 3.40GHz 8-core CPU, and 32 GB memory. Each method is trained by ADAM Kingma and Ba [2014] using crossentropy loss for the classification task and L2 loss for the regression task. The learning rate equals 0.01, and the weight matrix is initialized from a normal distribution N (0 , 0 . 01) . Each model contains five layers, including the input layer, the output layer, and three hidden layers. As exemplified by the time series classification task, the input is a time series ( i.e. vector) with a scalar at each time point. The output is the implied feature mapping ( i.e. vector), which is used to conduct the classification task. Concretely, the operation in the first layer is defined as Φ : R d x → C d x , where d x denotes the dimension of the data. Via Φ in the first layer, the data result in complex-valued representations, which are fed into the CSKE module starting from the second layer. The operation of l th layer is defined as Ψ l : C d l → C d l +1 , where d l denotes the number of hidden complex-valued neuron. After the CSKE module, we obtain the implied complex-valued feathers. Moreover, these implied complex-valued features are condensed into vector form by the operation C d L → R 2 d L , which concatenate real and imaginary parts, to conduct the classification task. Each experiment is repeated twenty times with different random seeds. Note that, the width of networks in each dataset depends

on the length of the time series, respectively. Therefore, the detailed settings of different models are shown in the Supplementary Material.

4.1 Experimental results

Inherently complex-valued representation learning In practice, the observed data is always presented as real numbers, while the complex-valued representation can be found inherently in information processing. In CosNet, we propose the CSKG module, which generalizes the spectral kernel mapping in the real number domain to the complex number domain. To show the capability of our method to recover the complex-valued representation, we conduct a simulation experiment and compare CosNet with two typical strategies to deal with complex-valued mapping, i.e., Fourier transform (FT) and eliminating the complex part (DSKN). In the experiment, a complex number sequence with 100 points { z i = u i + v i } 100 i =1 , is randomly generated as the ground truth, and the corresponding real number is given as x i = | z i | . The first module of CosNet, i.e., CSKG, is used for recovering the given complex numbers { z i } 100 i =1 from the real numbers { x i } 100 i =1 . The results are shown in Figure 2, from which we can find that CosNet with the complex-valued non-stationary spectral kernel mapping, compared with FT and DSKN, can recover the inherently complex-valued representation precisely. In contrast, the oscillations of the recovered sequences in both two domains by FT are intense while the DSKN fails to recover the phase information contained in the imaginary part.

DSKN

FT

CosNet

real_ground truth

imag_ground truth

2

real_pred

imag_pred

0

20

40

60

80

100

0

20

40

60

80

100

MMAN

0

real_groundtruth

imag_ground truth

-2

real_pred

2

imag_pred

0

20

40

60

80

100

20

40

60

80

100

real_ground truth

imag_ground truth

real_pred

2

imag_pred

0

20

40

60

80

100

0

20

40

60

80

100

Time series classification To verify the effectiveness of our CosNet on the time-sequential data analysis, we compared the state-of-the-art spectral kernel networks and CVNNs on the time series classification task. The results are shown in Table 1. We can observe that our CosNet achieves stateof-the-art performance in all datasets. Specifically, CosNet outperforms other methods impressively and achieves 3% accuracy increment (83.06 → 85.46) on Wine and 2 . 7% (69.81 → 71.73) on FordB compared with the mainstream kernel methods and CVNNs. Furthermore, these results reveal that the performance of real-valued methods is limited to the data with the inherently complex-valued representation.

Image classification and compression To further explore the representation capacity of our CosNet, we expand the application of CosNet to convolutional networks (see the Supplementary Material for details) for image classification and compression tasks using Fashion-MNIST Xiao et al. [2017] and CIFAR10 Krizhevsky and Hinton [2009] datasets. For the classification task, accuracy is applied to the metric of performance. For the compression task, we first extract the implicit features through various models, and then we conduct the clustering task based on these extracted features. In this task, Normalized Mutual Information (NMI) and Rand Index (RI) are used as the assessment metrics. All the results are reported in Table 2. We can find out that our CosNet outperforms the baseline methods on both classification task and compression task. Notadly, our CosNet achieves

1.5 % accuracy improvment (87.02 % → 88.33 % ), 5.6 % NMI improvement (86.04 % → 90.86 % ), 1.19 % RI improvement (97.15 % → 98.13 % ) on FMNIST dataset, and 3.4 % accuracy improvement (64.32 % → 66.51, 15.82 % NMI improvement (57.14 % → 66.18 % ), 2.5 % RI improvement (88.78 % → 89.97 % ) on CIFAR-10 dataset. The results show that our CosNet has a greater representation capability than other complex-valued convolutional networks.

4.2 Ablation Study

Complex-valued representation capability Complex numbers, containing amplitude and phase information, lead to a rich representation capability. However, the re-introducing of complex values brings extra parameters. To demonstrate that performance improvement comes from the powerful representation brought by complex values instead of added parameters, we conduct an ablation study to evaluate the methods with different parameters. As shown in Table 3, we compare the proposed CosNet with three variants of SRFF and DSKN, i.e., the variant in the original paper (normal), the variant with more neurons per layer (wider), and the variant with more layers (deeper).

From Table 3, we can observe that our CosNet commonly performs better with fewer parameters. Without loss of generality, increasing parameters indeed can lead to the improvement of performance on certain datasets, but there still remains a gap compared to our CosNet. Therefore, we can conclude that using real-valued networks in some fields, where complex numbers occur either naturally or by design, still has limitations.

Initialization We propose to initialize the complex-valued weight matrix of the second module with the cosine and sine functions for the imaginary and real parts, respectively. This design ensures CosNet retains the property of non-stationary spectral kernels and takes the relative distance of data in the complex number domain without increasing the number of parameters. To explore the role of the designed initialization scheme, we compare the results of classification and regression tasks on CosNet with or without cosine and sine functions. The results reported in Table 4 show that our proposed initialization scheme performs better in all cases, which indicates that non-stationarity is necessary for analyzing the time-sequential data. Furthermore, the experimental results validate the effectiveness of our design in a complex-valued weight matrix on CosNet.

5 Conclusion

In this paper, we propose a complex-valued spectral kernel network (CosNet) with two core modules, i.e., SKMG module and CSKE module. Specifically, as the first module of CosNet, the SKMG module is employed to recover the inherent complex-valued representation of the real-valued data. The CSKE module, designed by embedding the complex-valued spectral kernel mapping into neural networks with our initialization scheme, is used to effectively capture long-range or periodic relations of data. Our proposed CosNet, benefiting from the non-stationary property of kernels, can effectively encode the dynamic input-dependent characteristics and long-range correlations. The complex-valued mapping can improve the representation capacity of models without increasing the number of parameters. Furthermore, CosNet involves the transformation of the real-valued inputs in the optimization process to learn an expressive complex-valued representation. Moreover, some theoretical analyses of CosNet are also presented. Detailed experiments reveal that our proposed approach indeed leads to significant performance improvements over state-of-the-art relevant methods. Future work will focus on promoting the proposed CosNet in more applications.

Limitation CosNet with the periodic function is prone to local minima. However, our CosNet tends to perform well in the time-sequential data analysis since it can not only capture the long-range relation in an input-dependent manner but also take the imaginary part into account. In future work, we will focus on promoting the proposed CosNet in the optimization method.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Nos. 62076062 and 62306070) and the Social Development Science and Technology Project of Jiangsu Province (No. BE2022811). Furthermore, the work was also supported by the Big Data Computing Center of Southeast University.

References

Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In Proceeding of International Conference on Machine Learning , volume 48, pages 1120-1128, 2016.

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems , 30, 2017.

Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. Advances in neural information processing systems , 19, 2006.

Catherine Blake. Uci repository of machine learning databases. http://www. ics. uci. edu/˜ mlearn/MLRepository. html , 1998.

Salomon Bochner et al. Lectures on Fourier integrals , volume 42. Princeton University Press, 1959.

Hyeong-Seok Choi, Jang-Hyun Kim, Jaesung Huh, Adrian Kim, Jung-Woo Ha, and Kyogu Lee. Phase-aware speech enhancement with deep complex u-net. In International Conference on Learning Representations , 2019.

David Roxbee Cox and Hilton David Miller. The theory of stochastic processes . Routledge, 2017.

Ivo Danihelka, Greg Wayne, Benigno Uria, Nal Kalchbrenner, and Alex Graves. Associative long short-term memory. In International Conference on Machine Learning , volume 48, pages 19861994, 2016.

Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series archive. IEEE/CAA Journal of Automatica Sinica , 6(6):1293-1305, 2019.

Muneer Ahmad Dedmari, Sailesh Conjeti, Santiago Estrada, Phillip Ehses, Tony Stöcker, and Martin Reuter. Complex fully convolutional neural networks for mr image reconstruction. In Machine Learning for Medical Image Reconstruction: First International Workshop, MLMIR 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 1 , pages 30-38. Springer, 2018.

Akira Hirose, Ryosho Nakane, and Gouhei Tanaka. Keynote speech: Information processing hardware, physical reservoir computing and complex-valued neural networks. In 2019 IEEE International Meeting for Future of Electron Devices, Kansai (IMFEDK) , pages 19-24. IEEE, 2019.

Akira Hirose. Proposal of fully complex-valued neural networks. In [Proceedings 1992] IJCNN International Joint Conference on Neural Networks , volume 4, pages 152-157. IEEE, 1992.

Akira Hirose. Complex-valued neural networks: theories and applications , volume 5. World Scientific, 2003.

M Soo Kim and Clark C Guest. Modification of backpropagation networks for complex-valued signal processing in frequency domain. In 1990 IJCNN International Joint Conference on Neural Networks , pages 27-31. IEEE, 1990.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases , 1(4), 2009.

Miguel Lázaro-Gredilla, Joaquin Quinonero-Candela, Carl Edward Rasmussen, and Aníbal R Figueiras-Vidal. Sparse spectrum gaussian process regression. The Journal of Machine Learning Research , 11:1865-1881, 2010.

Jian Li, Yong Liu, and Weiping Wang. Distributed learning with random features. arXiv preprint arXiv:1906.03155 , 2019.

Jian Li, Yong Liu, and Weiping Wang. Automated spectral kernel learning. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 4618-4625, 2020.

Fanghui Liu, Xiaolin Huang, Yudong Chen, and Johan AK Suykens. Random features for kernel approximation: A survey on algorithms, theory, and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(10):7128-7148, 2021.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

A. Popa. Complex-valued convolutional neural networks for real-valued image classification. In International Joint Conference on Neural Networks , 2017.

Rahimi and B. Recht. Random features for large scale kernel machines. In Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 3-6, 2007 , 2007.

David P Reichert and Thomas Serre. Neuronal synchrony in complex-valued deep networks. arXiv preprint arXiv:1312.6115 , 2013.

Sami Remes, Markus Heinonen, and Samuel Kaski. Non-stationary spectral kernels. neural information processing systems , 30, 2017.

Advances in

Izhak Shafran, Tom Bagby, and R. J. Skerry-Ryan. Complex evolution recurrent neural networks (cernns). In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5854-5858, 2018.

Anthony Tompkins, Rafael Oliveira, and Fabio T Ramos. Sparse spectrum warped input measures for nonstationary kernel learning. Proceedings of Advances in Neural Information Processing Systems , pages 16153-16164, 2020.

Jean-Francois Ton, Seth Flaxman, Dino Sejdinovic, and Samir Bhatt. Spatial mapping with gaussian processes and non-stationary fourier features. Spatial statistics , 28:59-78, 2018.

Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, Joao Felipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher J Pal. Deep complex networks. In International Conference on Learning Representations , 2018.

Théo Trouillon and Maximilian Nickel. Complex and holographic embeddings of knowledge graphs: a comparison. arXiv preprint arXiv:1707.01475 , 2017.

Hirofumi Tsuzuki, Mauricio Kugler, Susumu Kuroyanagi, and Akira Iwata. An approach for sound source localization by complex-valued neural network. IEICE TRANSACTIONS on Information and Systems , 96(10):2257-2265, 2013.

Shanshan Wang, Huitao Cheng, Ziwen Ke, Leslie Ying, Xin Liu, Hairong Zheng, and Dong Liang. Complex-valued residual network learning for parallel mr imaging. In Proc. 26th Annu. Meeting of ISMRM , 2018.

Xab Wen, B Gma, Zab Feng, C Hl, and Zab Lu. Polsar image classification via a novel semisupervised recurrent complex-valued convolution neural network. Neurocomputing , 388:255-268, 2020.

Michael Wilmanski, Chris Kreucher, and Alfred Hero. Complex input convolutional neural networks for wide angle sar atr. In 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP) , pages 1037-1041. IEEE, 2016.

Andrew Wilson and Ryan Adams. Gaussian process kernels for pattern discovery and extrapolation. In International conference on machine learning , pages 1067-1075. PMLR, 2013.

Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux, and Les Atlas. Full-capacity unitary recurrent neural networks. Advances in neural information processing systems , 29, 2016.

Moritz Wolter and Angela Yao. Complex gated recurrent neural networks. Advances in neural information processing systems , 31, 2018.

Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5028-5037, 2017.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Hui Xue, Zheng-Fan Wu, and Wei-Xiang Sun. Deep spectral kernel learning. In IJCAI , pages 4019-4025, 2019.

Akiva M Yaglom. Correlation Theory of Stationary and Related Random Functions, Volume I: Basic Results , volume 131. Springer, 1987.

Sheng-Sung Yang, Chia-Lu Ho, and Sammy Siu. Sensitivity analysis of the split-complex valued multilayer perceptron due to the errors of the iid inputs and weights. IEEE transactions on neural networks , 18(5):1280-1293, 2007.

Bin Yang, Wei Zhang, Li-Na Gong, and Huai-Zhi Ma. Finance time series prediction using complexvalued flexible neural tree model. In 2017 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD) , pages 54-58, 2017.

Muqiao Yang, Martin Q. Ma, Dongyu Li, Yao-Hung Hubert Tsai, and Ruslan Salakhutdinov. Complex transformer: A framework for modeling complex-valued sequence. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4232-4236, 2020.

Lingjuan Yu, Yuehong Hu, Xiaochun Xie, Yun Lin, and Wen Hong. Complex-valued full convolutional neural network for sar target classification. IEEE Geoscience and Remote Sensing Letters , 17(10):1752-1756, 2019.

Zhiqiang Zeng, Jinping Sun, Zhu Han, and Wen Hong. Sar automatic target recognition method based on multi-stream complex-valued networks. IEEE Transactions on Geoscience and Remote Sensing , 60:1-18, 2022.

Huisheng Zhang, Chao Zhang, and Wei Wu. Convergence of batch split-complex backpropagation algorithm for complex-valued neural networks. Discrete Dynamics in Nature and Society , 2009, 2009.

Shuai Zhang, Jianxin Li, Pengtao Xie, Yingchun Zhang, Minglai Shao, Haoyi Zhou, and Mengyi Yan. Stacked kernel network. arXiv preprint arXiv:1711.09219 , 2017.

Zhimian Zhang, Haipeng Wang, Feng Xu, and Ya-Qiu Jin. Complex-valued convolutional neural network and its application in polarimetric sar image classification. IEEE Transactions on Geoscience and Remote Sensing , 55(12):7177-7188, 2017.

Peer reviews:
comments: SUMMARY:
The authors propose a complex valued neural architecture, composed of two modules: a Spectral kernel mapping generalization module and a Complex-valued spectral kernel embedding module. 
They provide a generalization error bound for their model. In addition they propose a novel initialization scheme and provide experimental results on several datasets and learning tasks.

STRENGTHS:
The proposed approach seem to be novel, and the experimental results are promising.

WEAKNESSES:
In my view, the main weakness of the manuscript is the presentation. While the approach itself seems sound, I find the presentation too poor for a conference like Neurips. Therefore I encourage the authors to re-write the manuscript and make sure it reads much better.

QUESTIONS:
No

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new framework called Complex-valued spectral kernel network (CosNet) that generalizes the spectral kernel to include complex-valued representation. The proposed framework improves the representational capability of the spectral kernel and outperforms existing kernel methods and complex-valued neural networks. An initialization scheme for the complex-valued weight matrix is proposed, which ensures that CosNet retains the property of non-stationary spectral kernels and takes the relative distance of data in the complex number domain without increasing the number of parameters. The paper provides the lower generalization bound of CosNet than the real-valued non-stationary spectral kernel. The experiments demonstrate that CosNet performs better than the mainstream kernel methods and complex-valued neural networks in time-sequential data analysis.\n\nSTRENGTHS:\n1. The paper proposes a new framework (CosNet) and provides a theoretical analysis of it.\n2. The experiments demonstrate that CosNet outperforms existing kernel methods in time-series data analysis, which shows the practical significance of the proposed framework.\n3. The writing of the paper is good.\n\nWEAKNESSES:\n1. The experiments presented in the paper lack sufficient evidence to convincingly demonstrate the properties of the model.\n2. The dataset used in the experiments is limited in diversity, which raises concerns about the generalizability of the findings.\n3. The initialization of CosNet's parameters varies across different datasets, and a unified initialization method is needed for consistency and reproducibility.\n\nTypo: In line 168, the imaginary part should also be multiplied by the weight $1/\\sqrt{\\frac{1}{4M}}$.\n\nQUESTIONS:\n1. The paper presents experiments comparing the accuracy of CosNet to other methods with similar network size on the same dataset, highlighting the capacity gain from introducing the imaginary part. However, I believe that accuracy alone may not fully demonstrate the improvement of the model's representation capacity. Could the authors provide more evidence of CosNet's capacity gain from other aspects, such as compression and recovery of more complex datasets?\n2. The experiments in the paper adopt time-series datasets with low complexity. Could the authors provide more results that demonstrate the performance of CosNet on tasks such as image encoding and decoding or the signal processing in real-world scenarios, since these datasets may also be suitable cases for complex-valued spectral kernel modeling.\n3. In Figure 2, the authors only present the visualization results of two other methods for predicting and displaying the original time series curve. Could the authors provide additional visualizations of the performance of other comparable methods for comparison? This would provide a more comprehensive and fair comparison of the proposed method against existing ones.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to extend the reach of kenel-based inference for time series by using a hilbert space over a complex field rather than working over the reals, by not discarding the phase component of the implied spectral representation of the kernels, which apparently is a common strategy. This method makes it feasible to define a broader class of kernels than the obvious (.e.g stationary)\n\nSTRENGTHS:\nThe paper seems to propose a novel way of characterising flexible covariance kernels such that they are PSD, and some neural networks that exploit this\n\nWEAKNESSES:\nThis paper is hard to read. The main claims are difficult to extract and thus their correctness is hard to verify.\nI could be persuaded that I have underestimated the soundness of the results --- they could in fact be amazing --- but I have already blown my limited time budget trying to understand what is going on, because basic framing information is missing. \nI presume that some of this could be deduced by inspecting the references introduced in section 2, but I don't have time for that in the setting of urgent conf reviews. the paper needs to be self-contained, even if it we defer rigorous proof to the appendices.\n\nMy certainty rating reflects my relatively high confidence that this paper needs a rewrite. I do not have a high confidence that the idea itself is flawed, and would welcome clarifications that showed me what the idea actually is, because it might be great. I currently do no understand it, because the presentation is confusing enough that it will take more than the time I have to \"reverse engineer\" it.\n\nQUESTIONS:\nThere are a lot of words about the usefulness of the complex-values covariance kernel which I think I can interpret but I do not actually know for sure, because the actual inference problem is not clearly set up. What is it? What is the baseline?\n\nI think my confusion starts in l122. Everything up to this point was fine, but now we have deployed the machinery of a complex-valued spectrum in  kernel definitions, without actually explaining what is different. there should be certain things that are better for a complex kernel from the Yaglom theorem. For one, a kernel so defined does needs to be stationary, as opposed to a Bochner-theorem style kernel, which I assume is  the main point, and very cool and indeed the authors mention this right in the abstract. So I like this! I've looked at Yaglom's theorem before and though tit would be great to make it tractable to use but I didn't see a way. Indeed this should be a very general kernel, and should generalise other kernel classes too (e.g. dot-product kernels). So the idea sounds promising.\nEven here though, I'm a little confused; OK, so we are using this kernel not as, say, a covariance kernel of a Gaussian process, but rather to directly define a similarity between data points for optimal interpolation, as far as I can tell, which is fine, but can you say more about that actual implied network structure? Do we keep around the training data so we can measure the kernel-similarity to other training points, or are we happy to use it as \"just another\" nonlinear NN layer. In which case, what does this layer do that an MLP does *not* do?\n\nin eq (10) we learn that the kernel is characterised by a finite list  $\\Omega$ of frequencies, right? Is it correct that this means that we are restricting our kernel spectral \"density\" $\\left(\\boldsymbol{\\omega}, \\boldsymbol{\\omega}^{\\prime}\\right)$ to be not a density as such but  but rather to be a collection of dirac deltas in the spectral space? I suspect we need to say so, in that case.\n\nl147/eq(8) the $M$ suddenly appeared and it looks important but is not explored. This is the number of MC samples we take to actually approximate the spectral kernel integral. So... when do we evaluate this integral? Is it inside the training loop? How do we choose $M$? is it robust against different choices of $M$? Should we not see $M$ pop up in evaluating the computational cost of this method.\n\nCan we put a simple function map notation description of *every operator and function*, e.g. $\\Phi_\\ell: \\mathbb{C}^{d_{\\ell}}\\to\\mathbb{C}^{d_{\\ell+1}}$?  Generally, for most functions in this paper I'm confused what the are mapping from and to. I have so many questions here that I cannot list them all. Here are some examples about function domains and ranges:\n\n1. Are we permitting the values of the kernel outputs in the intermediate layers to be complex? \n2. which values are permitted to be vectors and which scalars? I gather the inputs to the network are vectors (?) but the test examples seem to be all time series with a scalar index (?) In section 4,1a the experiments seem to be about estimating complex valued time series, and in section 4.1b it is classifying stuff based on the implied feature mapping. Is the model more general than this?\n\nIf the authors wish to remain very general, fine, but if so, could we have a running example that makes it clear? \n\nSince we also know that NNs exist which do not parameterise their output in terms of parameters of kernels, but directly in terms of _weights_, could we say anything about the relative representation power compared to that?\nThere is Theorem 1, which gives us a statistical learning theory result in terms of covering numbers, which is possibly supposed to help us. Perhaps the problem here is my own ignorance, but what is the equivalent results for the baseline that the authors hope to surpass? Can you quote an equivalent theorem for a baseline? Should I know one? Or is the intent here to show us how to trade off between allocating weights to _including more frequencies in a given kernel layer_ versus _adding more layers_? Can you spell out the implications of this theorem in terms of \"wide versus deep complex kernel networks\" and also \"whether complex kernel networks are better than MLPs\"?\n\nRelated: Why do we need to stack layers at all, rather than simply learning a big kernel? \n\n4.1: There is a neural network here. I support the authors not wasting space with excessive details about Adagrad learning rates etc, but I need just a little more information to know what is happening. What is the training procedure? If I am used to ERM for training an NN, do I need to worry about some alternative methods for these exotic kernel networks, or is it the same? If it is the same, why am I thinking about the kernels directly rather than just learning an MLP? remember, since the network has been non-specific throughout the paper, this is my example to learn what actual inputs and outputs this network can predict upon.\n\nLIMITATIONS:\nI don't know. I have a hard time deducing exactly the domain of applicability of the paper from the presentation here.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly focuses on the issue that spectral kernel-based methods often eliminate the imaginary part when analyzing the characteristics of time-sequential data. This limits the representation capability of the spectral kernel. To address this issue, the authors propose a complex-valued spectral kernel network to take both the real and imaginary parts into account. The proposal mainly consists of two parts - the SKMG module recovers the complex-valued representation for the real-valued data and the CSKE module combines the complex-valued spectral kernels and neural networks. Theoretical and empirical results show the proposed method achieves state-of-the-art performance.\n\nSTRENGTHS:\n1. The proposed method is well-motivated. Involving the imaginary part is critical for preserving the amplitude and phase information for data and improving the representational capability of spectral kernel networks. \n2. The proposed approach is sound. The complex-valued representation for the real-valued data is recovered by the SKMG module and complex-valued spectral kernels are combined with neural networks via the CSKE module. The two parts are well integrated. \n3. Theoretical analysis and experimental evaluation are provided to show the state-of-the-art performance of the proposal.\n\nWEAKNESSES:\n1. In Section 3.3, the authors define the complex-valued weight matrix by Equation 11. But is unclear why this design ensures that the sub-network containing the first layer to arbitrary l-th layer is seen as a spectral kernel.\n2. The experimental results appear to be dependent on the choices of hyperparameters. Performance of CosNet with different learning rates, initializations and layer numbers should be provided.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nYes, the authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}