SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yanfang Xue $^{1,2}$ , Pengfei Fang $^{1,2}$ , Jinyue Tian $^{1,2}$ , Shipeng Zhu $^{1,2}$ , Hui Xue $^{1,2*}$
$^{1}$ School of Computer Science and Engineering, Southeast University, Nanjing, 210096, China
$^{2}$ Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary
Applications (Southeast University), Ministry of Education, China
{230218795, fangpengfei, 220222083, shipengzhu, hxue}@seu.edu.cn
Complex-valued representation exists inherently in the time-sequential data that can be derived from the integration of harmonic waves. The non-stationary spectral kernel, realizing a complex-valued feature mapping, has shown its potential to analyze the time-varying statistical characteristics of the time-sequential data, as a result of the modeling frequency parameters. However, most existing spectral kernel-based methods eliminate the imaginary part, thereby limiting the representation power of the spectral kernel. To tackle this issue, we propose a generalized spectral kernel network, namely, Complex-valued spectral kernel Network (CosNet), which includes spectral kernel mapping generalization (SKMG) module and complex-valued spectral kernel embedding (CSKE) module. Concretely, the SKMG module is devised to generalize the spectral kernel mapping in the real number domain to the complex number domain, recovering the inherent complex-valued representation for the real-valued data. Then a following CSKE module is further developed to combine the complex-valued spectral kernels and neural networks to effectively capture long-range or periodic relations of the data. Along with the CosNet, we study the effect of the complex-valued spectral kernel mapping via theoretically analyzing the bound of covering number and generalization error. Extensive experiments demonstrate that CosNet performs better than the mainstream kernel methods and complex-valued neural networks.
Complex numbers represent the information of the amplitude and phase simultaneously. In contrast to the amplitude, which can be revealed by the real number, the phase can denote the time delay and advance, thereby encoding the temporal dependency of the data Hirose [2003]. This suggests that complex-valued models can be employed in some practical applications, especially for the time-sequential data where information is wave-related, such as signal analysis Hirose et al. [2019]; Yu et al. [2019]; Zeng et al. [2022], speech processing Shafran et al. [2018], and time series classification Yang et al. [2020, 2017]; Wisdom et al. [2016].
In the learning community, the spectral kernel, which is constructed from the inverse Fourier transform, can naturally realize the complex-valued mapping. That said, the spectral kernel can analyze the data in the frequency domain directly. As a candidate of the spectral kernel family, the non-stationary spectral kernel is also proposed to conquer the local limitations of the classical kernels, such as stationarity and monotonicity Remes et al. [2017]; Tompkins et al. [2020]; Ton et al. [2018]; Li et al. [2020]. Ideally, these advanced kernels can extract appropriate non-stationary time-varying characteristics of the data by modeling the frequency parameters, and hence infer the long-range or periodic relations of the input data.
However, most existing methods usually eliminate the imaginary part of the spectral kernel mapping arbitrarily for the convenience of calculation. For example, Rahimi and Recht [2007]; Zhang et al. [2017a] ignore the imaginary part directly by replacing the integrand  $e^{j\omega (x - x^{\prime})}$  with  $\cos (\omega (x - x^{\prime}))$ ; Xue et al. eliminates the imaginary part with an elaborate spectral density function definition for the non-stationary spectral kernel Xue et al. [2019]. Remarkably, existing research shows that complex numbers could lead to a rich representational capability for wave-related information processing Wisdom et al. [2016]; Danihelka et al. [2016]; Worrall et al. [2017]; Trouillon and Nickel [2017]. However, simply plugging the imaginary part in the neural networks does not ensure that the model retains the property of the spectral kernel. Therefore, more efforts are required to develop a new framework that can involve the imaginary part in spectral kernel networks.
In this paper, we propose a new framework that generalizes the spectral kernel that endows with the complex-value representation, and we name it as a complex-valued spectral kernel network (CosNet). The proposed CosNet includes two modules: the spectral kernel mapping generalization (SKMG) module and the complex-valued spectral kernel embedding (CSKE) module. Technically, we generalize the spectral kernel mapping in the real number domain to the complex number domain by defining the spectral density function in the SKMG module. We further embed the complex-valued spectral kernel into neural networks to attain the proposed CosNet using the CSKE module. It is noted that a new initialization scheme is also proposed for the CSKE module that adopts the cosine and sine functions as the activation for the real and imaginary parts of the weight matrix. This initializing scheme retains the statistical characteristics of the non-stationary spectral kernel. It enables CosNet to take the relative distance of data into account by shifting between phases, such that capture the long-range or periodic relations of data in the complex domain without increasing the number of parameters. Our contributions in this paper are shown as follows:
- We propose a complex-valued spectral kernel network, i.e., CosNet, which takes both the real and imaginary parts of the spectral kernel mapping into account and thus improves the representational capability of the spectral kernel.
- We propose an initialization scheme for the complex-valued weight matrix, which ensures that CosNet retains the property of non-stationary spectral kernels and takes the relative distance of data in the complex number domain without increasing the number of parameters.
- We provide the lower generalization bound of CosNet than the real-valued non-stationary spectral kernel.
- Thorough experiments demonstrate that our proposed method is totally superior to state-of-the-art kernel methods.
Spectral kernel networks Spectral approaches were developed to fully characterize general kernels with concise representation forms, such as sparse spectrum kernels Lázaro-Gredilla et al. [2010], sparse mixture kernels Wilson and Adams [2013], non-stationary spectral kernels Remes et al. [2017], and random Fourier features methods to deal with large-scale settings Li et al. [2019]; Liu et al. [2021]. These methods commonly approximated the kernel function using an explicit spectral representation based on Bochner's theorem Bochner and others [1959] and Yaglom's theorem Yaglom [1987]. Benefiting from the outstanding representation capability of neural networks with hierarchical nonlinear linking structures Bengio et al. [2006], researchers attempt to embed spectral representation (i.e., feature mapping of kernels) into the hierarchical architecture of neural networks to construct spectral kernel networks. Zhang et al. [2017a] used the Random Fourier Feature to approach the stationary kernel mapping and embedded it into each layer of DNNs. Xue et al. [2019] proposed a deep spectral kernel network to embed the non-stationary spectral kernel into each layer of DNNs, which can approximate most of the kernels. Li et al. [2020] proposed an automated spectral kernel learning (ASKL) that incorporates the process of finding suitable non-stationary kernels and model training. However, for convenience of calculation, these models commonly use the real-valued representation, although spectral kernels lead to the complex-valued mapping.
Complex-valued neural networks Complex-valued neural networks (CVNNs) have shown excellent efficiency compared to their real counterparts in biological Reichert and Serre [2013], speech enhancement Tsuzuki et al. [2013]; Choi et al. [2019], image Popa [2017]; Wen et al. [2020], and signal processing Kim and Guest [1990]; Wilmanski et al. [2016]. In previous studies, researchers
commonly split the complex-valued input into a pair of real-valued inputs and fed them into the real-valued neural networks with both real-valued weight matrix and activation function. This design cannot exploit the advantages of complex numbers completely, and the neural network convergence strongly depends on proper initialization and the choice of learning rate Yang et al. [2007]; Zhang et al. [2009]. Subsequently, CVNNs with complex-valued weight and activation functions are proposed in the complex number domain to deal with complex-valued inputs Hirose [1992]; Dedmari et al. [2018]; Zhang et al. [2017b]. Benefitting from the rich representation capability, researchers tend to extend CVNNs to other neural networks, such as complex-valued convolutional neural networks Trabelsi et al. [2018], complex-valued residual neural networks Wang et al. [2018], and complex-valued recurrent neural networks Wolter and Yao [2018]; Arjovsky et al. [2016]. All these works have proved that the complex-valued models have a richer representational capacity and perform better on real-world learning tasks by a set of experiments.
3 Complex-valued Spectral Kernel Networks
In this section, we first introduce concepts and notations of the non-stationary kernel and complex numbers. Then, we provide the overall architecture of our CosNet with two modules. Moreover, we explicitly provide the details of each module. In addition, we present a detailed analysis of CosNet.
To better illustrate CosNet, we introduce the necessary preliminary knowledge and notation of non-stationary spectral kernels and complex numbers in this section.
Notations Formally, we use  $\mathbb{R}^n$ ,  $\mathbb{C}^n$ ,  $\mathbb{R}^{m\times n}$  and  $\mathbb{C}^{m\times n}$  to denote  $n$ -dimensional Euclidean spaces,  $n$ -dimensional complex number spaces, the space of  $m\times n$  real-valued matrix and the space of  $m\times n$  complex-valued matrix. Throughout the paper, the matrices, vectors and scalars are denoted by bold capital letters (e.g.  $\mathbf{X}$ ), bold lower-case letters (e.g.  $\mathbf{x}$ ) and lower-case letters (e.g.  $\mathbf{x}$ ), respectively. A complex number  $\mathbf{z}\in \mathbb{C}^{D}$  is represented as  $\mathbf{z} = \mathbf{u} + i\mathbf{v}$  with a real part  $\mathbf{u}$  and an imaginary part  $\mathbf{v}$ .  $\bar{\mathbf{z}} = \mathbf{u} - i\mathbf{v}$  denotes the complex conjugate of  $\mathbf{z}$ . For any two complex numbers  $\mathbf{z}_1 = \mathbf{u}_1 + i\mathbf{v}_1$ ,  $\mathbf{z}_2 = \mathbf{u}_2 + i\mathbf{v}_2\in \mathbb{C}$ ,  $\mathbf{z}_1 + \mathbf{z}_2 = (\mathbf{u}_1 + u_2) + i(\mathbf{v}_1 + \mathbf{v}_2)$ ,  $\mathbf{z}_1\mathbf{z}_2 = (\mathbf{u}_1\mathbf{u}_2 - \mathbf{v}_1\mathbf{v}_2) + i(\mathbf{u}_1\mathbf{v}_2 + \mathbf{v}_1\mathbf{u}_2)$ . To represent the complex-valued layer with  $2D$  features, we allocate the first  $D$  features to represent the real component and the remaining to represent the imaginary component.
Preliminary knowledge Non-stationary spectral kernels are constructed from inverse Fourier transform in the frequency domain. Based on Yaglom's theorem Yaglom [1987], a general kernel  $k(\pmb{x},\pmb{x}^{\prime})$  is positive definite on  $\mathbb{R}^D$  if and only if it admits the form:
$$
k \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) = \int_ {\mathbb {R} ^ {D} \times \mathbb {R} ^ {D}} e ^ {i \left(\boldsymbol {\omega} ^ {\top} \boldsymbol {x} - \boldsymbol {\omega} ^ {\prime \top} \boldsymbol {x} ^ {\prime}\right)} \mu \left(d \boldsymbol {\omega}, d \boldsymbol {\omega} ^ {\prime}\right) \tag {1}
$$
where  $\mu (d\omega ,d\omega^{\prime})$  is the Lebesgue-Stieltjes measure associated with some positive semi-definite spectral density function  $s(\omega ,\omega^{\prime})$  with bounded variations. Therefore, a general kernel can be defined as the following form:
$$
k \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) = \int_ {\mathbb {R} ^ {D} \times \mathbb {R} ^ {D}} e ^ {i \left(\boldsymbol {\omega} ^ {\top} \boldsymbol {x} - \boldsymbol {\omega} ^ {\prime} ^ {\top} \boldsymbol {x} ^ {\prime}\right)} s \left(\boldsymbol {\omega}, \boldsymbol {\omega} ^ {\prime}\right) d \boldsymbol {\omega} d \boldsymbol {\omega} ^ {\prime} \tag {2}
$$
where  $s(\omega, \omega')$  can be understood as a joint probability density function.
3.2 Overall architecture
To explore the capability of the imaginary part in the spectral kernel networks, we propose CosNet as a generalized framework. CosNet involves two modules: the SKMG module to achieve complex-valued spectral kernel mapping and the CSKE module to embed the spectral kernel into neural networks. The overall architecture is shown in Figure 1.
Concretely, the SKMG module is denoted as  $\Phi(x)$  via generalizing the spectral kernel mapping in the real number domain to the complex number domain for the real-valued data  $x$ . And the CSKE module is denoted as  $\Psi(h)$  via initializing complex-valued weight matrix with the cosine and sine function for the complex-valued spectral kernel mapping  $h$ .
Figure 1: The structure of CosNet with two modules. The SKMG module is used to map the real-valued inputs to a complex-valued representation. The CSKE module is the complex-valued spectral kernel embedding with our initialization.
Based on the two modules, our CosNet with  $l$  layers is defined as:
$$
\operatorname {C o s N e t} (\boldsymbol {x}) = \Psi^ {l - 1} \left(\dots \Psi^ {1} \left(\Phi^ {1} (\boldsymbol {x})\right)\right). \tag {3}
$$
Moreover, the corresponding complex-valued spectral kernel is defined as:
$$
\mathbb {K} ^ {(l)} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \left\langle \Psi^ {l - 1} \left(\dots \Psi^ {1} \left(\Phi^ {1} (\boldsymbol {x})\right)\right), \Psi^ {l - 1} \left(\dots \Psi^ {1} \left(\Phi^ {1} \left(\boldsymbol {x} ^ {\prime}\right)\right)\right) \right\rangle = \left\langle C o s N e t (\boldsymbol {x}), C o s N e t \left(\boldsymbol {x} ^ {\prime}\right) \right\rangle \tag {4}
$$
where  $\mathbb{K}^{(l)}(\pmb{x},\pmb{x}^{\prime})$  denotes the  $l$ -layer complex-valued spectral kernel.
3.3 Complex-valued spectral kernel network (CosNet)
Spectral kernel mapping generalization module In this module, we generalize the spectral kernel mapping in the real number domain to the complex number domain. Furthermore, the generalized mapping can be used in both stationary and non-stationary spectral kernels. Here we elaborate on the detailed process.
According to Equation (2), to produce a positive semi-definite kernel, we need to include symmetries  $s(\omega, \omega') = s(\omega', \omega)$  and sufficient diagonal components  $s(\omega, \omega)$  and  $s(\omega', \omega')$ . Concretely, we replace the exponential component  $e^{i(\omega^\top x - \omega'^\top x')}$  in Equation (2) with  $\zeta_{\omega, \omega'}(x, x')$ , which is defined as:
$$
\zeta_ {\boldsymbol {\omega}, \boldsymbol {\omega} ^ {\prime}} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \frac {1}{4} \left[ e ^ {i \left(\boldsymbol {\omega} ^ {\top} \boldsymbol {x} - \boldsymbol {\omega} ^ {\prime \top} \boldsymbol {x} ^ {\prime}\right)} + e ^ {i \left(\boldsymbol {\omega} ^ {\prime \top} \boldsymbol {x} - \boldsymbol {\omega} ^ {\top} \boldsymbol {x} ^ {\prime}\right)} + e ^ {i \left(\boldsymbol {\omega} ^ {\top} \boldsymbol {x} - \boldsymbol {\omega} ^ {\top} \boldsymbol {x} ^ {\prime}\right)} + e ^ {i \left(\boldsymbol {\omega} ^ {\prime \top} \boldsymbol {x} - \boldsymbol {\omega} ^ {\prime \top} \boldsymbol {x} ^ {\prime}\right)} \right]. \tag {5}
$$
Then, we expand the exponential component to the complex-valued representation with the cosine and sine function based on Euler's formula, and  $\zeta_{\omega ,\omega '}(\pmb {x},\pmb{x}')$  can be rewritten as:
$$
\begin{array}{l} \zeta_ {\boldsymbol {\omega}, \boldsymbol {\omega} ^ {\prime}} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \frac {1}{4} \left[ \cos \left(\boldsymbol {\omega} ^ {\top} \boldsymbol {x} - \boldsymbol {\omega} ^ {\prime \top} \boldsymbol {x} ^ {\prime}\right) + i \sin \left(\boldsymbol {\omega} ^ {\top} \boldsymbol {x} - \boldsymbol {\omega} ^ {\prime \top} \boldsymbol {x} ^ {\prime}\right) \right. \\ + \cos \left(\omega^ {\prime \top} \boldsymbol {x} - \omega^ {\top} \boldsymbol {x} ^ {\prime}\right) + i \sin \left(\omega^ {\prime \top} \boldsymbol {x} - \omega^ {\top} \boldsymbol {x} ^ {\prime}\right) \tag {6} \\ + \cos (\boldsymbol {\omega} ^ {\top} \boldsymbol {x} - \boldsymbol {\omega} ^ {\top} \boldsymbol {x} ^ {\prime}) + i \sin (\boldsymbol {\omega} ^ {\top} \boldsymbol {x} - \boldsymbol {\omega} ^ {\top} \boldsymbol {x} ^ {\prime}) \\ \left. \right.\left. + \cos \left(\omega^ {\prime \top} \boldsymbol {x} - \omega^ {\prime \top} \boldsymbol {x} ^ {\prime}\right) + i \sin \left(\omega^ {\prime \top} \boldsymbol {x} - \omega^ {\prime \top} \boldsymbol {x} ^ {\prime}\right)\right]. \\ \end{array}
$$
As a result, the general spectral kernel in Equation (2) can be redefined as:
$$
k \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) = \int_ {\mathbb {R} ^ {D} \times \mathbb {R} ^ {D}} \zeta_ {\omega , \omega^ {\prime}} \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) p \left(\omega , \omega^ {\prime}\right) d \omega d \omega^ {\prime} \tag {7}
$$
where  $p(\omega, \omega') = \frac{1}{4} [s(\omega, \omega') + s(\omega', \omega) + s(\omega, \omega) + s(\omega', \omega')]$  also can be considered as a probability density function.
Subsequently, we approximate Equation (7) with Monte Carlo random sampling:
$$
\begin{array}{l} k (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \int_ {\mathbb {R} ^ {D} \times \mathbb {R} ^ {D}} \zeta_ {\boldsymbol {\omega}, \boldsymbol {\omega} ^ {\prime}} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) p (\boldsymbol {\omega}, \boldsymbol {\omega} ^ {\prime}) d \boldsymbol {\omega} d \boldsymbol {\omega} ^ {\prime} = \mathbb {E} _ {\boldsymbol {\omega}, \boldsymbol {\omega} ^ {\prime} \sim P} \left[ \zeta_ {\boldsymbol {\omega}, \boldsymbol {\omega} ^ {\prime}} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) \right] \\ \approx \frac {1}{4 M} \sum_ {i = 1} ^ {M} \left[ \cos (\pmb {\omega} _ {i} ^ {\top} \pmb {x} - \pmb {\omega} _ {i} ^ {\prime \top} \pmb {x} ^ {\prime}) + i \sin (\pmb {\omega} _ {i} ^ {\top} \pmb {x} - \pmb {\omega} _ {i} ^ {\prime \top} \pmb {x} ^ {\prime}) \right. \\ + \cos \left(\omega_ {i} ^ {\prime \top} \boldsymbol {x} - \omega_ {i} ^ {\top} \boldsymbol {x} ^ {\prime}\right) + i \sin \left(\omega_ {i} ^ {\prime \top} \boldsymbol {x} - \omega_ {i} ^ {\top} \boldsymbol {x} ^ {\prime}\right) \tag {8} \\ + \cos (\pmb {\omega} _ {i} ^ {\top} \pmb {x} - \pmb {\omega} _ {i} ^ {\top} \pmb {x} ^ {\prime}) + i \sin (\pmb {\omega} _ {i} ^ {\top} \pmb {x} - \pmb {\omega} _ {i} ^ {\top} \pmb {x} ^ {\prime}) \\ \left. \left. + \cos \left(\omega_ {i} ^ {\prime \top} \boldsymbol {x} - \omega_ {i} ^ {\prime \top} \boldsymbol {x} ^ {\prime}\right) + i \sin \left(\omega_ {i} ^ {\top} \boldsymbol {x} - \omega_ {i} ^ {\top} \boldsymbol {x} ^ {\prime}\right) \right] \right. \\ = \langle \Phi (\pmb {x}), \overline {{\Phi (\pmb {x} ^ {\prime})}} \rangle \\ \end{array}
$$
where,  $(\omega_{i},\omega_{i}^{\prime})_{i = 1}^{D}$  is the frequency pairs,  $M$  is the sampling number.
The generalized spectral kernel mapping in Equation (8) is defined as:
$$
\Phi (\boldsymbol {x}) = \sqrt {\frac {1}{4 M}} \left[ \left(\cos \left(\boldsymbol {\Omega} ^ {\top} \boldsymbol {x}\right) + \cos \left(\boldsymbol {\Omega} ^ {\prime \top} \boldsymbol {x}\right)\right) + i \left(\sin \left(\boldsymbol {\Omega} ^ {\top} \boldsymbol {x}\right) + \sin \left(\boldsymbol {\Omega} ^ {\prime \top} \boldsymbol {x}\right)\right) \right], \tag {9}
$$
and the frequency matrices  $\Omega, \Omega'$  are denoted as:
$$
\boldsymbol {\Omega} = \left[ \omega_ {1}, \omega_ {2}, \dots , \omega_ {M} \right], \boldsymbol {\Omega} ^ {\prime} = \left[ \omega_ {1} ^ {\prime}, \omega_ {2} ^ {\prime}, \dots , \omega_ {M} ^ {\prime} \right] \tag {10}
$$
As a result, we obtain a complex-valued spectral kernel mapping. The real part of the output is denoted as  $\Re (\Phi (\pmb {x})) = \cos (\pmb{\Omega}^{\top}\pmb {x}) + \cos (\pmb{\Omega}^{\prime \top}\pmb {x})$ , and the imaginary part is denoted as  $\Im (\Phi (\pmb {x})) = \sin (\pmb {\Omega}^{\top}\pmb {x}) + \sin (\pmb{\Omega}^{\prime \top}\pmb {x})$
Complex-valued spectral kernel embedding module In this module, we attempt to embed the complex-valued spectral kernel into each layer of neural networks to construct CosNet. Spectral kernel, based on the general Fourier analysis, provides a new explicit kernel mapping. These kernels can not only approximate most kernels under specific conditions by some fundamental theorems Cox and Miller [2017]; Yaglom [1987] but also provide an efficient way to combine neural networks with kernel methods to construct spectral networks. Most existing spectral kernel networks commonly embed the spectral kernel into neural networks by stacking the spectral kernel mapping in the hierarchical architecture of neural networks directly. However, the introduction of an imaginary part enables that networks with a simple stack of complex-valued mapping cannot be formulated as a spectral kernel (see the Supplementary Material for details).
To ensure the sub-network containing the first layer to arbitrary  $l$ -th layer ( $l \geqslant 2$ ) can be integrally seen as a spectral kernel, and following the form of complex-valued parameters in CVNNs, we define the complex-valued weight matrix of this module as:
$$
\boldsymbol {W} = \cos (\boldsymbol {A}) + i \sin (\boldsymbol {A}), \tag {11}
$$
where  $A$  is a real-valued matrix.
In this module, the convolution operates with the complex weight matrix  $\mathbf{W}$  is defined as:
$$
\begin{array}{l} \Psi (\pmb {h}) = \pmb {W} \pmb {h} = \sqrt {\frac {1}{4 M}} \Bigl [ \cos (\pmb {A}) (\cos (\pmb {\Omega} ^ {\top} \pmb {x}) + \cos (\pmb {\Omega} ^ {\prime \top} \pmb {x})) \\ \left. - \sin (\boldsymbol {A}) \left(\sin \left(\boldsymbol {\Omega} ^ {\top} \boldsymbol {x}\right) + \sin \left(\boldsymbol {\Omega} ^ {\prime \top} \boldsymbol {x}\right)\right) \right] \tag {12} \\ + i \sqrt {\frac {1}{4 M}} \left[ \sin (\boldsymbol {A}) \left(\cos \left(\boldsymbol {\Omega} ^ {\top} \boldsymbol {x}\right) + \cos \left(\boldsymbol {\Omega} ^ {\prime} ^ {\top} \boldsymbol {x}\right)\right) \right. \\ \left. \right.\left. + \cos (\boldsymbol {A}) \left(\sin \left(\boldsymbol {\Omega} ^ {\top} \boldsymbol {x}\right) + \sin \left(\boldsymbol {\Omega} ^ {\prime \top} \boldsymbol {x}\right)\right)\right]. \\ \end{array}
$$
The real and imaginary parts of the convolution operation are represented in the matrix notation:
$$
\left[ \begin{array}{l} \Re (\Psi (\boldsymbol {h})) \\ \Im (\Psi (\boldsymbol {h})) \end{array} \right] = \left[ \begin{array}{l l} \cos (\boldsymbol {A}) & - \sin (\boldsymbol {A}) \\ \sin (\boldsymbol {A}) & \cos (\boldsymbol {A}) \end{array} \right] * \sqrt {\frac {1}{4 M}} \left[ \begin{array}{l} \cos \left(\boldsymbol {\Omega} ^ {\top} \boldsymbol {x}\right) + \cos \left(\boldsymbol {\Omega} ^ {\prime \top} \boldsymbol {x}\right) \\ \sin \left(\boldsymbol {\Omega} ^ {\top} \boldsymbol {x}\right) + \sin \left(\boldsymbol {\Omega} ^ {\prime \top} \boldsymbol {x}\right) \end{array} \right]. \tag {13}
$$
To inherit the outstanding representation capability from neural networks, in this module, we construct the spectral kernel networks by stacking  $\Psi$ :
$$
\operatorname {C o s N e t} (\boldsymbol {x}) = \Psi^ {l - 1} (\dots \Psi^ {1} (\boldsymbol {h})), \tag {14}
$$
where  $\Psi^l (2\leq l)$  denotes the  $l$  -layer complex-valued spectral kernel mapping and
$$
\boldsymbol {h} = \sqrt {\frac {1}{4 M}} \left[ \left(\cos \left(\boldsymbol {\Omega} ^ {\top} \boldsymbol {x}\right) + \cos \left(\boldsymbol {\Omega} ^ {\prime \top} \boldsymbol {x}\right)\right) + i \left(\sin \left(\boldsymbol {\Omega} ^ {\top} \boldsymbol {x}\right) + \sin \left(\boldsymbol {\Omega} ^ {\prime \top} \boldsymbol {x}\right)\right) \right]. \tag {15}
$$
3.4 Analysis of CosNet