SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CosNet, constructed by stacking the non-station complex-valued spectral kernel mapping, not only retains the property of non-stationary spectral kernels, which can effectively reveal the input-dependence characteristics and long-range relations but also can learn hierarchy within Reproducing Kernel Hilbert Space, yielding a cascade of non-linear features. Besides, CosNet takes the imaginary part of the complex-valued spectral kernel mapping into account, leading to a richer representation capability.
**Framework generality** In spectral kernels view, CosNet will be reduced to a stationarity spectral kernel when  $\omega = \omega'$  in Equation (2). Besides, the real-valued spectral kernel mapping is the special case (i.e., the imaginary part  $\Im \Phi(\boldsymbol{x})$  equal to 0) of our complex-valued mapping. In the data view, CosNet can analyze the real-valued data, where the complex-valued representation can be found inherently. The first module of CosNet also can be considered as a complex-valued representation learning module, which transforms real-valued data into complex-valued features by optimizing the learnable frequency matrices  $\Omega$  and  $\Omega'$ . CosNet also can analyze complex-valued data with the framework that only includes the second module.
Parameters In the complex-valued spectral kernel embedding module, we initialize the real and imaginary parts of weight matrices with the cosine and sine functions, respectively. Compared with CVNNs, which define the weight matrix as  $\mathbf{W} = \mathbf{A} + i\mathbf{B}$ , the number of parameters used in CosNet decreases because of our periodic initialization strategy using only  $\mathbf{A}$ . Compared to non-stationary spectral kernels, the number of parameters is reduced since there is no need for sampling two different frequency matrices,  $\Omega$  and  $\Omega'$ .
Theoretical results We provide theoretical evidence of the generalization performance of CosNet, showing that CosNet has a lower generalization error bound compared to the real-valued spectral kernel networks of the same architecture. Concretely, we first bound the covering numbers of different layers in CosNet, followed by comparisons between covering numbers of real-valued spectral kernel networks and that of CosNet, which provide evidence of CosNet's improvements in generalization ability. Further, we derive the generalization bound of CosNet based on several theorems Bartlett et al. [2017]; Mohri et al. [2018].
Theorem 1. Denote the covering number of set  $S$  as  $N_{d}(S,\epsilon)$ .  $\mathbf{X} \in R^{d^x \times n}$  is the input of  $n$  samples and each sample is  $d^x$ -dimensioned.  $\mathbf{X}^l \in R^{d^l \times n}$  is the input of layer  $l$  ( $l > 1$ ) and  $\mathbf{A}^l$  is the weight matrix of layer  $l$  ( $l \geq 2$ ). The other notations remain the same as mentioned above. For different layers, their covering numbers satisfy that
1. In the first layer,  $N_{d}(\Omega^{1}\pmb {X},\epsilon)\leq (4d^{0}d^{x})^{k}$ , where  $k\geq \frac{||\boldsymbol{\omega}_{ij}||_1^2}{\epsilon^2}\max_{i,j}||\pmb{x}_{ij}||^2$
2. In layer  $l$  ( $l > 1$ ),  $N_{d}(\mathbf{A}^{l}\mathbf{X}^{l - 1},\epsilon)\leq (2d^{l}d^{l - 1} + 1)^{k}$ , where  $k\geq \frac{||\mathbf{W}_{ij}||_1^2}{\epsilon^2}\frac{\pi}{2} d^l ||\mathbf{X}^{l - 1}||_1^2$ .
Proof. The proof is relegated to the supplementary material of our paper due to space limitations.
Covering numbers also serves as an indicator of models' representation ability, where the larger the covering number the greater the representation ability, but the more difficult it is to get the optimal solution. Note that when the weight matrices are the same, the bound of the covering number of each layer of a multilayer perception (MLP) is  $(2d^{l}d^{l - 1})^{k}$ , where  $k\geq \frac{||W_{ij}||_1^2}{\epsilon^2} ||X^{l - 1}||_1^2$ . And that of real-valued spectral network is  $(4d^{l}d^{l - 1})^{k}$ , where  $k\geq \frac{||W_{ij}||_1^2}{\epsilon^2} ||X^{l - 1}||_1^2$ , which is as twice large as that of MLP. It can be observed that real-value spectral networks improve their representation ability at the cost of much larger covering number bounds and poorer generalization performance. However,
CosNet combines the advantages of both MLP and real-valued spectral networks. Compared to MLP, CosNet's representation ability is further improved by bringing complex-valued representations into the spectral kernel networks, while only the covering number bound of the first layer increases when constant terms are neglected, which has stronger characterization ability and makes it easier to find the optimal solution. Its superiority is even more clear when compared to real-valued spectral kernels such as DSKN, every layer of CosNet has a smaller complexity, which leads to a significant difference when it comes to the complexity of the whole network.
Theorem 2. Let  $S = \{(\pmb{x}_1, \pmb{y}_1), (\pmb{x}_2, \pmb{y}_2), \dots, (\pmb{x}_n, \pmb{y}_n)\}$  be a sample data of size  $n$  from distribution  $D$ . Given the weight matrices defined before  $(\Omega^1, \Omega^2, A^1, A^2, \dots, A^L)$ , and they satisfy that  $||A^l|| \leq c_l$ ,  $||W^l|| \leq b_l$ ,  $||\Omega^l|| \leq a_l$ ,  $||X||_1 \leq B$ ,  $d^l \leq W$  and  $T = (\sum_{l=1}^{L}(\frac{b_l}{c_l})^{2/3})^{3/2} \prod_{l=1}^{L}c_l$ . And the loss function  $\mathcal{L}(\operatorname{CosNet}(\pmb{x}), \pmb{y}) \leq M$ . Then with the probability of at least  $1 - \delta$ , the proposed network CosNet satisfies:
$$
\begin{array}{l} \underset {(\boldsymbol {x}, \boldsymbol {y}) \sim D} {E} [ (\mathcal {L} (\operatorname {C o s N e t} (\boldsymbol {x}), \boldsymbol {y}) ] \\ \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {L} \left(\operatorname {C o s N e t} \left(\boldsymbol {x} _ {i}\right), \boldsymbol {y} _ {i}\right) + \mathcal {O} \left(\frac {8 M}{n ^ {3 / 2}} + M \sqrt {\frac {\ln (1 / \delta)}{n}} \right. \tag {16} \\ + \ln (n) \frac {\sqrt {\ln (\boldsymbol {W}) \boldsymbol {W} | | \boldsymbol {X} ^ {0} | | ^ {2} T ^ {2} + \ln (\boldsymbol {W}) a _ {1} ^ {2} | | \boldsymbol {X} | | ^ {2}}}{n}). \\ \end{array}
$$
In this section, we first introduce the implementation details containing comparison methods and evaluation datasets. Then we conduct systematical experiments to demonstrate the superiority of the proposed CosNet, especially on the time series classification task.
Datasets To systematically evaluate the performance of our CosNet, we conduct comparison experiments on several typical time-series datasets, including 12 sub-datasets with default training and testing data splitting from the UCR Archive Dau et al. [2019] dataset for the classification task and 3 UCI Blake [1998] localization datasets for regression task. The overall statistics of the used datasets are shown in the Supplementary Material.
Compared methods We compare the proposed CosNet with several mainstream kernel methods and CVNNs, as follows: SRFF Zhang et al. [2017a]: Stacked Kernel Network, which stacks random Fourier features with stationary kernels; DSKN Xue et al. [2019]: Deep Spectral Kernel Network; DCN Trabelsi et al. [2018]: Deep Complex Network. We compare two variants with different commonly used activation functions, including CReLU  $(\mathbf{DCN}^1)$  and modReLU  $(\mathbf{DCN}^2)$ ; ASKL Li et al. [2020]: Automated spectral kernel learning.
Implementation details All the experiments are implemented with PyTorch Paszke et al. [2019] and conducted on a workstation with NVIDIA RTX 3090 GPU, AMD R7-5700X 3.40GHz 8-core CPU, and 32 GB memory. Each method is trained by ADAM Kingma and Ba [2014] using cross-entropy loss for the classification task and L2 loss for the regression task. The learning rate equals 0.01, and the weight matrix is initialized from a normal distribution  $\mathcal{N}(0,0.01)$ . Each model contains five layers, including the input layer, the output layer, and three hidden layers. As exemplified by the time series classification task, the input is a time series (i.e. vector) with a scalar at each time point. The output is the implied feature mapping (i.e. vector), which is used to conduct the classification task. Concretely, the operation in the first layer is defined as  $\Phi : \mathbb{R}^{d^x} \to \mathbb{C}^{d^x}$ , where  $d^x$  denotes the dimension of the data. Via  $\Phi$  in the first layer, the data result in complex-valued representations, which are fed into the CSKE module starting from the second layer. The operation of  $l^{th}$  layer is defined as  $\Psi^l : \mathbb{C}^{d^l} \to \mathbb{C}^{d^{l+1}}$ , where  $d^l$  denotes the number of hidden complex-valued neuron. After the CSKE module, we obtain the implied complex-valued feathers. Moreover, these implied complex-valued features are condensed into vector form by the operation  $\mathbb{C}^{d^L} \to \mathbb{R}^{2d^L}$ , which concatenate real and imaginary parts, to conduct the classification task. Each experiment is repeated twenty times with different random seeds. Note that, the width of networks in each dataset depends
on the length of the time series, respectively. Therefore, the detailed settings of different models are shown in the Supplementary Material.
4.1 Experimental results
Inherently complex-valued representation learning In practice, the observed data is always presented as real numbers, while the complex-valued representation can be found inherently in information processing. In CosNet, we propose the CSKG module, which generalizes the spectral kernel mapping in the real number domain to the complex number domain. To show the capability of our method to recover the complex-valued representation, we conduct a simulation experiment and compare CosNet with two typical strategies to deal with complex-valued mapping, i.e., Fourier transform (FT) and eliminating the complex part (DSKN). In the experiment, a complex number sequence with 100 points  $\{z_i = u_i + v_i\}_{i=1}^{100}$ , is randomly generated as the ground truth, and the corresponding real number is given as  $x_i = |z_i|$ . The first module of CosNet, i.e., CSKG, is used for recovering the given complex numbers  $\{z_i\}_{i=1}^{100}$  from the real numbers  $\{x_i\}_{i=1}^{100}$ . The results are shown in Figure 2, from which we can find that CosNet with the complex-valued non-stationary spectral kernel mapping, compared with FT and DSKN, can recover the inherently complex-valued representation precisely. In contrast, the oscillations of the recovered sequences in both two domains by FT are intense while the DSKN fails to recover the phase information contained in the imaginary part.
Figure 2: Comparison of complex-valued representation learning. The left and right denote the learning of real and imaginary parts, respectively.
Time series classification To verify the effectiveness of our CosNet on the time-sequential data analysis, we compared the state-of-the-art spectral kernel networks and CVNNs on the time series classification task. The results are shown in Table 1. We can observe that our CosNet achieves state-of-the-art performance in all datasets. Specifically, CosNet outperforms other methods impressively and achieves  $3\%$  accuracy increment  $(83.06 \rightarrow 85.46)$  on Wine and  $2.7\%$ $(69.81 \rightarrow 71.73)$  on FordB compared with the mainstream kernel methods and CVNNs. Furthermore, these results reveal that the performance of real-valued methods is limited to the data with the inherently complex-valued representation.
Image classification and compression To further explore the representation capacity of our CosNet, we expand the application of CosNet to convolutional networks (see the Supplementary Material for details) for image classification and compression tasks using Fashion-MNIST Xiao et al. [2017] and CIFAR10 Krizhevsky and Hinton [2009] datasets. For the classification task, accuracy is applied to the metric of performance. For the compression task, we first extract the implicit features through various models, and then we conduct the clustering task based on these extracted features. In this task, Normalized Mutual Information (NMI) and Rand Index (RI) are used as the assessment metrics. All the results are reported in Table 2. We can find out that our CosNet outperforms the baseline methods on both classification task and compression task. Notably, our CosNet achieves
Table 1: Classification accuracy (%) of each compared method on several time series datasets. The best results are highlighted in bold.
Dataset	SRFF	DSKN	DCN1	DCN2	ASKL	CosNet
FordA	81.46	82.24	81.87	79.90	72.66	82.42
FordB	68.99	69.81	69.68	50.17	64.20	71.73
PhalangesOutlinesCorrect	68.77	69.73	68.91	67.63	68.65	70.79
Wine	77.22	76.48	83.06	80.00	67.41	85.46
ECG200	73.40	77.80	89.80	89.85	87.53	90.10
ECG5000	91.98	91.14	93.11	93.50	92.75	93.70
Herring	57.73	56.64	65.23	58.13	59.52	65.39
Ham	51.52	48.81	71.10	67.76	68.52	71.29
ProximalPhalanxOutlineAgeGroup	79.49	79.51	81.80	81.59	80.00	82.71
Table 2: Classification and compression results on image datasets. The best results are highlighted in bold.
Fashion-MNIST	CIFAR10
DCN1	DCN2	CosNet	DCN1	DCN2	CosNet
Accuracy (%)	87.02	84.44	88.33	64.32	52.39	66.51
NMI (%)	86.04	81.31	90.86	57.14	41.69	66.18
RI (%)	97.15	95.97	98.31	87.78	84.07	89.97
1.5% accuracy improvement (87.02% → 88.33%), 5.6% NMI improvement (86.04% → 90.86%), 1.19% RI improvement (97.15% → 98.13%) on FMNIST dataset, and 3.4% accuracy improvement (64.32% → 66.51, 15.82% NMI improvement (57.14% → 66.18%), 2.5% RI improvement (88.78% → 89.97%) on CIFAR-10 dataset. The results show that our CosNet has a greater representation capability than other complex-valued convolutional networks.
Complex-valued representation capability Complex numbers, containing amplitude and phase information, lead to a rich representation capability. However, the re-introducing of complex values brings extra parameters. To demonstrate that performance improvement comes from the powerful representation brought by complex values instead of added parameters, we conduct an ablation study to evaluate the methods with different parameters. As shown in Table 3, we compare the proposed CosNet with three variants of SRFF and DSKN, i.e., the variant in the original paper (normal), the variant with more neurons per layer (wider), and the variant with more layers (deeper).
From Table 3, we can observe that our CosNet commonly performs better with fewer parameters. Without loss of generality, increasing parameters indeed can lead to the improvement of performance on certain datasets, but there still remains a gap compared to our CosNet. Therefore, we can conclude that using real-valued networks in some fields, where complex numbers occur either naturally or by design, still has limitations.
Initialization We propose to initialize the complex-valued weight matrix of the second module with the cosine and sine functions for the imaginary and real parts, respectively. This design ensures CosNet retains the property of non-stationary spectral kernels and takes the relative distance of data in the complex number domain without increasing the number of parameters. To explore the role of the designed initialization scheme, we compare the results of classification and regression tasks on CosNet with or without cosine and sine functions. The results reported in Table 4 show that our proposed initialization scheme performs better in all cases, which indicates that non-stationarity is necessary for analyzing the time-sequential data. Furthermore, the experimental results validate the effectiveness of our design in a complex-valued weight matrix on CosNet.
Table 3: Classification accuracy (%) and parameters with wider and deeper cases. The best results are highlighted in bold.
Model	Setting	ECG200	ECG5000	Ham
Parameters	Accuracy	Parameters	Accuracy	Parameters	Accuracy
SRFF	normal	22.25K	73.40	31.49K	91.98	251.12K	68.43
wider	69.06K	83.90	84.98K	92.75	686.96K	70.52
deeper	42.35K	65.85	56.91K	92.55	459.22K	60.10
DSKN	normal	44.42K	77.80	62.80K	91.14	502.11K	69.76
wider	137.99K	80.65	169.62K	92.42	1260.00K	71.76
deeper	137.99K	80.65	169.62K	92.42	1260.00K	71.76
CosNet	normal	19.65K	90.10	40.75K	93.70	375.14K	74.27
Table 4: Classification accuracy (%) and regression MSE on the benchmark datasets.  $(\uparrow)$  indicates the larger the better, while  $(\downarrow)$  indicates the smaller the better. The best results are highlighted in bold.
Classification Accuracy (↑)	Regression MSE (↓)
Earthquakes	DistalPhalanxTW	Strawberry	power	concreat	yacht
w/ cos, sin	71.76	63.60	97.22	0.8229	1.3606	3.6270
w/o cos, sin	69.93	63.02	96.80	0.8795	1.3731	3.7932
In this paper, we propose a complex-valued spectral kernel network (CosNet) with two core modules, i.e., SKMG module and CSKE module. Specifically, as the first module of CosNet, the SKMG module is employed to recover the inherent complex-valued representation of the real-valued data. The CSKE module, designed by embedding the complex-valued spectral kernel mapping into neural networks with our initialization scheme, is used to effectively capture long-range or periodic relations of data. Our proposed CosNet, benefiting from the non-stationary property of kernels, can effectively encode the dynamic input-dependent characteristics and long-range correlations. The complex-valued mapping can improve the representation capacity of models without increasing the number of parameters. Furthermore, CosNet involves the transformation of the real-valued inputs in the optimization process to learn an expressive complex-valued representation. Moreover, some theoretical analyses of CosNet are also presented. Detailed experiments reveal that our proposed approach indeed leads to significant performance improvements over state-of-the-art relevant methods. Future work will focus on promoting the proposed CosNet in more applications.
Limitation CosNet with the periodic function is prone to local minima. However, our CosNet tends to perform well in the time-sequential data analysis since it can not only capture the long-range relation in an input-dependent manner but also take the imaginary part into account. In future work, we will focus on promoting the proposed CosNet in the optimization method.
This work was supported by the National Natural Science Foundation of China (Nos. 62076062 and 62306070) and the Social Development Science and Technology Project of Jiangsu Province (No. BE2022811). Furthermore, the work was also supported by the Big Data Computing Center of Southeast University.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In Proceeding of International Conference on Machine Learning, volume 48, pages 1120-1128, 2016.
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
Hui Xue, Zheng-Fan Wu, and Wei-Xiang Sun. Deep spectral kernel learning. In *IJCAI*, pages 4019-4025, 2019.
Akiva M Yaglom. Correlation Theory of Stationary and Related Random Functions, Volume I: Basic Results, volume 131. Springer, 1987.
Sheng-Sung Yang, Chia-Lu Ho, and Sammy Siu. Sensitivity analysis of the split-complex valued multilayer perceptron due to the errors of the iid inputs and weights. IEEE transactions on neural networks, 18(5):1280-1293, 2007.
Bin Yang, Wei Zhang, Li-Na Gong, and Huai-Zhi Ma. Finance time series prediction using complex-valued flexible neural tree model. In 2017 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), pages 54-58, 2017.
Muqiao Yang, Martin Q. Ma, Dongyu Li, Yao-Hung Hubert Tsai, and Ruslan Salakhutdinov. Complex transformer: A framework for modeling complex-valued sequence. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4232-4236, 2020.
Lingjuan Yu, Yuehong Hu, Xiaochun Xie, Yun Lin, and Wen Hong. Complex-valued full convolutional neural network for sar target classification. IEEE Geoscience and Remote Sensing Letters, 17(10):1752-1756, 2019.
Zhiqiang Zeng, Jinping Sun, Zhu Han, and Wen Hong. Sar automatic target recognition method based on multi-stream complex-valued networks. IEEE Transactions on Geoscience and Remote Sensing, 60:1-18, 2022.
Huisheng Zhang, Chao Zhang, and Wei Wu. Convergence of batch split-complex backpropagation algorithm for complex-valued neural networks. Discrete Dynamics in Nature and Society, 2009, 2009.
Shuai Zhang, Jianxin Li, Pengtao Xie, Yingchun Zhang, Minglai Shao, Haoyi Zhou, and Mengyi Yan. Stacked kernel network. arXiv preprint arXiv:1711.09219, 2017.
Zhimian Zhang, Haipeng Wang, Feng Xu, and Ya-Qiu Jin. Complex-valued convolutional neural network and its application in polarimetric sar image classification. IEEE Transactions on Geoscience and Remote Sensing, 55(12):7177-7188, 2017.