COPU: Recognizing Time Series' Heterogeneity In Stacked Neural Network

Authors: Shaoqi Wang, Chunjie Yang, Siwei Lou, Feng Yan
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Neural Network, Time Series, Regression, Nonlinear Modeling, Natural Gradient

Abstract:
Neural networks (NNs) have been widely studied in complex fields due to their remarkable capacity for nonlinear modeling. 
However, in the realm of time series analysis, researches indicate that merely stacking NNs does not yield promising nonlinear modeling outputs and hinders model performance. Conventional NN architectures overemphasize homogeneous feature extraction, impeding the learning of diverse features and diminishing their nonlinear modeling capability. To address this gap, we propose the $\textbf{C}$ross-correlation Enhanced Approximated $\textbf{O}$rthogonal $\textbf{P}$rojection $\textbf{U}$nit (COPU) to quantify and augment the NN's nonlinear modeling capacity. COPU efficiently computes the local cross-correlation characteristics between features, amplifying heterogeneous components while compressing homogeneous ones. By reducing redundant information, COPU facilitates the learning of unique and independent features, thereby enhancing nonlinear modeling capability. Extensive experiments demonstrate that our method achieves superior performance across two real-world regression applications.

Decision: reject
Reviewer scores: 6, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1729167838010

thread_type:
official_review

<thread_object>
creator: reviewer_1q18

<content>
summary:
In this paper, the authors present an approach called Cross-correlation Enhanced Approximated Orthogonal Projection Unit (COPU) to quantify and augment a neural network’s (NN’s) nonlinear modeling capacity. The authors motivate their approach based on the observed inability of NNs to learn diverse features and effectively perform nonlinear modeling. The proposed method amplifies heterogeneous components and reduces homogeneous ones, enabling the learning of unique and independent features, thereby enhancing the NN's nonlinear modeling capability.

strengths:
- The authors motivate the proposed approach using realistic examples.

- They propose using rank ratio as a metric for measuring nonlinear modeling capability.

- Experiments conducted on real world datasets.

weaknesses:
- The presentation of the paper is quite confusing. There are several mistakes, such as Eq. (6) not being referenced anywhere, and in line 251, the notation "x \in \mathbb{R}^{d,b}" is unclear. Additionally, there are several sentences that are difficult to follow.
For example: 

- "This process enables the differentiation between redundant information and innovation, further suppressing homogeneous information among features while amplifying their differences"
what innovation means here?

- "while the augmentation modular focuses on diverse and informative feature extraction"
module?

- "We implement this approach by  employing dictionary lookup Ashish et al. (2017) to achieve feature mapping."
wrong reference type.

- Using scientific notation for MAPE (which is typically presented as percentage) is also confusing.	

- S4 acronym not introduced.

These are just a few examples, but there are numerous similar presentation issues throughout the paper, making it very difficult to follow. This is a significant shortcoming in itself, bringing the paper below ICLR  standards.

- Results on standardized and well-known datasets are expected to support the authors' claims and facilitate the reproduction of the presented results.

- In many cases, an MLP performs similarly to (or better than) the proposed approach, raising questions about the actual difficulty of the datasets used in the evaluation and whether the results truly support the authors' claims. The authors should also provide the results of a linear model to highlight the necessity of modeling non-linear components.

questions:
Overall, the paper is not yet ready for publication. Significant improvements are required in both the presentation and the experimental setup and evaluation in order to adequately support authors' claims. While the authors are welcome to respond to these shortcomings, I believe the issues are too fundamental to be addressed through a rebuttal alone and will likely require a major revision of the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729476535622

thread_type:
official_review

<thread_object>
creator: reviewer_vw2o

<content>
summary:
This paper think that conventional NN architectures overemphasize homogeneous feature extraction, impeding the learning of diverse features and diminishing their nonlinear modeling capability. To address this gap, they propose the Cross-correlation Enhanced Approximated Orthogonal Projection Unit (COPU) to quantify and augment the NN’s nonlinear modeling capacity.

strengths:
They introduced RR as a metric to quantify the proportion of linear dependencies transformed into independence by the network. Performing a comprehensive exploration of NN’s nonlinear modeling capability.  This paper proposes a new NN component termed CEP that has strong nonlinear representation performance in time series analysis and serves as the foundation for developing the COPU framework.

weaknesses:
1.  In page 1, the author said :  For these methods to be effectively applied in this field, it is
essential to recognize that time series has more ambiguous discriminative patterns than other forms
like images and text (Alec et al., 2021). Such ambiguity hinders the model’s ability to extract diverse
features from the input, obstructing its capacity for nonlinear modeling.

Then you give simple explanation, but the problem is you explanation is wrong. You classify original text and image data, while you do convoltuion on timeseries dat, and then you said "it is more challenging to discern the effect of two input sequences on the output
of a system". That is not resonable.


2. Some sentence is hard to understand like:
Page 7: This offers a novel perspective that elucidates the underlying efficacy of residual connections in NNs beyond the kernel (Duvenaud et al., 2014) and gradient (Kaiming et al., 2016) explanations.

3. In Figure 7, except for layer 7, it is hard for me to find difference CEP and other method, and I think with the iteration, the CEP curve will  close to other curve in the end.

4. You need to add explanation to different lines in Figure 8.

5. Last and most important Problem:
You propose a new method to solve the problem in time series prediction, but you did not conduct experiments on popular dataset and compare with some current SOTA models, like you refered in your paper : Are transformers effective for time series forecasting?

questions:
The same to Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622359365

thread_type:
official_review

<thread_object>
creator: reviewer_bobw

<content>
summary:
This paper proposes a new neural network component, called the cross-correlation Enhanced Perceptron (CEP), to solve the difficult problem of nonlinear modeling in time series analysis. CEP facilitates the learning of unique features by performing alignment and cross-correlation calculations on input features in a single step to distinguish and amplify different features while suppressing the influence of similar features. To further improve the model performance, the authors integrate CEP into the approximate orthogonal projection unit (AOPU) to form the COPU framework. COPU uses CEP to enhance nonlinear modeling capabilities and solves the limitations of AOPU in computational accuracy and expressiveness. The experimental results show that COPU is significantly superior to the existing methods in several real regression tasks. The contribution of this paper is to develop a CEP component specially used in time series analysis, which provides a new idea for feature modeling in a specific domain. At the same time, the authors also focus on an evaluation metric Rank Ratio (RR), because it can be interpreted as the proportion of linear dependencies that are translated into independence by neural networks.

strengths:
1. The author clearly demonstrates the problem that this paper aims to solve: that complex networks can effectively solve problems related to computer vision and natural language processing; But its ability to solve the problem of time series analysis is insufficient.
2. The author proposes a CEP scheme, which quantifies the similarity between features through Cross-correlation to ensure alignment without directly ignoring differences in time, which is quite innovative.
3. The author summarized relevant literature, explained the characteristics of time series analysis problems, and proposed that RR index has a good evaluation significance in measuring time series analysis problems, and this index can be relatively convincing in the subsequent research
4. The author cited sufficient literature to demonstrate the point of view, and designed ablation experiments to enhance feasibility.

weaknesses:
1. The advantages of indicator RR compared with other common indicators are not fully elaborated.
2. The relationship between CEP and COPU is not clearly stated, and it should be explained in detail how to achieve it.
3. The datasets used in the experimental design are insufficient, and more datasets are recommended to be added for verification.
4. In terms of experimental Settings, the author should provide setting parameters of different networks to compare the results more reasonably.

questions:
1. In the design process of CEP, why may the c value be negative? Why is it feasible to use sigmoid to force a negative number to be positive?
2. Does the CEP module have to be fixed with your network structure or can it be embedded in other models as well?
3. Will the paper provide code? I want to know about the implementation of CEP and COPU.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715917111

thread_type:
official_review

<thread_object>
creator: reviewer_mbum

<content>
summary:
This paper, titled COPU: Recognizing Time Series' Heterogeneity in Stacked Neural Network, addresses limitations in traditional neural network (NN) architectures when applied to time series data. The authors propose the COPU (Cross-correlation Enhanced Approximated Orthogonal Projection Unit), a new framework designed to enhance the NN's nonlinear modeling capabilities by emphasizing heterogeneous features. Traditional NN structures often focus on homogeneous feature extraction, which limits their performance on time series data where diverse feature extraction is essential. The COPU framework uses the Rank Ratio (RR) metric to measure the model's ability to capture unique information, enhancing heterogeneous components while compressing homogeneous ones. Experimental results demonstrate that COPU significantly outperforms existing NN structures in capturing nonlinear patterns, especially in time series datasets like SRU.

strengths:
Timely and Novel Research Direction: In time series forecasting research, most studies have traditionally focused on increasing model capacity. This paper takes a novel approach by addressing time series analysis from a fresh perspective that emphasizes data-specific structural improvements. This unique angle offers high novelty by shifting from merely increasing the capacity of NNs to a more data-driven design that aligns closely with the characteristics of time series data.

Innovative Methodology: COPU stands out for emphasizing heterogeneous over homogeneous elements in the data, which is well-suited to the nature of time series data. Unlike previous studies that focus on extracting homogeneous features, this paper proposes the Cross-correlation Enhanced Perceptron (CEP) to align features based on their correlations, suppressing redundant homogeneous information while amplifying unique features. This approach enhances the NN's ability to capture nonlinear aspects effectively, making it highly suitable for time series analysis.

Performance and Stability: COPU demonstrates superior performance compared to traditional NN structures, especially in terms of capturing diverse features without overfitting. This model excels in real-world scenarios by effectively learning complex and varied representations in time series data, showcasing both stability and robustness.

weaknesses:
Limited Applicability: COPU is specifically designed for time series data, which could restrict its application across different data types, such as image or text data. Although the focus on time series data is intentional, this may limit its adaptability to more general purposes.

Model Complexity: The architecture of COPU, particularly with CEP integration, is relatively complex, potentially requiring more computational resources than conventional NN models. This complexity could present challenges for implementation, especially for large-scale datasets or real-time applications.

questions:
Generalizability of the RR Metric: The paper shows that RR is effective for assessing nonlinear modeling capabilities in time series data, but it is unclear if RR can be reliably applied to other data types (e.g., continuous vs. discrete time series, multimodal data).

Balancing Homogeneous and Heterogeneous Elements: While the approach of suppressing homogeneous features and amplifying heterogeneous ones is innovative, the paper does not detail how the balance between these elements impacts performance. More insight into how this balance can be adjusted and its effect on model performance would be valuable.

Effect of Stack Depth on COPU’s Performance: Although increasing the depth of COPU stacks improves performance, the model exhibits diminishing returns at higher depths. It would be helpful to have guidelines on optimizing stack depth for this model.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732099286863

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Comment #1 on the first revision

comment:
Dear all Reviewers,

We are delighted to inform you that we have completed the revision of our manuscript. We would also like to thank all reviewers for the careful reading, helpful comments, and constructive suggestions. All these efforts have significantly improved the presentation of the manuscript. We paid specific attention to the following aspects:

- Experiment: Conducted experiments on additional datasets to strengthen the general applicability of COPU and RR.
- Presentation: Emphasized the research focus and contributions, refined certain expressions, and corrected some typos.

In the revised manuscript, **the content highlighted in red denotes additional statements, experimental results, or analytical conclusions, while the content highlighted in orange represents refinements to the original material.** The specific changes are as follows:

1. Added a new section in Appendix D to analyze additional experiments.
2. Adjusted the presentation order of datasets in Tables I and II to emphasize COPU's leading advantages on the more challenging dataset (Debutanizer).
3. Emphasized the differences between the COPU study and other research on line 88.
4. Provided additional explanation of the sigmoid function on line 204.
5. Adjusted the order of equations on line 253 and corrected typos.
6. Corrected two reference type errors.
7. Added more explanations to the manuscript.
8. Deleted some sentences and shrunk the figure size to make room for the adjustments.

For the reviewers’ convenience, we have listed the important content added in the revision below.



>**(Line 88)** The mainstream of conventional stacked network research involves empirically explore the feasibility of extracting features using base modules that exhibit outstanding performance under mean squared error or entropy loss in end-to-end tasks. However, it remains challenging to quantify and analyze whether these modules are qualified as feature extractors, and whether the features they extract are conducive to and consistent with the overall model. Furthermore, they struggle to monitor whether such extracted features are beneficial in optimization, much less guide directions for model improvement.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732163959968

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response #1 to three important issues

comment:
In this global response, we aim to provide explanations for the following three issues to offer a clear interpretation of our manuscript:

We sincerely hope that these explanations can assist the reviewers in better understanding and appreciating our manuscript.


- **Why did we choose to focus on studying RR instead of other metrics?**

- Investigating the RR is an unavoidable aspect of studying COPU. The loss function of COPU (Equation 9) is not always well-defined. When the matrix $\tilde{x}$ is not of full column rank, the matrix $\tilde{x}^T\tilde{x}$ becomes non-invertible. To compute Equation 9, numerical approximation is necessary; we utilize an inversion computation based on SVD. In this context, RR represents the proportion of non-zero singular values, which reflects the precision of the inversion. A higher RR leads to a more accurate computation of COPU’s loss, enabling a better restoration of the MSE. Conversely, a lower RR results in greater errors in COPU’s loss, tending towards a quadratic form. **This is the initial reason for studying RR; therefore, RR is not an arbitrarily chosen metric.** During our subsequent research on RR, we observe many thought-provoking phenomena, which motivates the study presented in this paper.

$\quad$

- **Why is the loss function formulated as shown in Equation 9, and why is the gradient calculated as depicted in Figure 4?**

- The objective of Equation 9 is to restore the MSE. Then, why do we not compute the MSE directly? Because COPU aims to restrict the backpropagation path, ensuring it passes through the dual parameter $m$. Why does COPU require that the gradient is calculated through the dual parameter $m$ as illustrated in Figure 4? Because the truncated gradient at the dual parameter is approximately the natural gradient of the trackable parameter. The trackable parameter is crucial since it is used in the forward computation. The degree of approximation to the natural gradient is also measured by RR. **Overall, these considerations are intended to enable the use of the natural gradient to update the trackable parameter.** The advantage of the natural gradient lies in its ability to recognize manifold structures; it is a gradient that considers the second-order derivatives of the parameter space, achieving more stable optimization and better performance compared to first-order gradient optimizers such as Adam **(see [1](https://openreview.net/forum?id=xqrlhsbcwN&referrer=%5Bthe%20profile%20of%20Shaoqi%20Wang%5D(%2Fprofile%3Fid%3D~Shaoqi_Wang3)),[2](https://openreview.net/forum?id=vuMD71R20q&referrer=%5Bthe%20profile%20of%20Wu%20Lin%5D(%2Fprofile%3Fid%3D~Wu_Lin2)),[3](https://openreview.net/forum?id=Y2wRKE0Qor&referrer=%5Bthe%20profile%20of%20Wu%20Lin%5D(%2Fprofile%3Fid%3D~Wu_Lin2)))**. It is important to note that COPU essentially employs a conditional natural gradient, as CEP is updated through standard optimizers.

$\quad$

- **What are the relationships among RR, CEP, and COPU?**

- RR is a general metric that can be applied to any model and modality. The variation of RR with batch size and sequence length reflects the intrinsic information density of the dataset (modality). The change of RR with the number of stacked layers reflects the network's nonlinear modeling capability. Specifically, in COPU, RR additionally measures the computational precision of the objective function, the degree to which the truncated gradient approximates the conditional natural gradient, and the extent to which the network output approximates the minimum variance estimation. CEP is an important stackable module within COPU, designed to address the issue that RR tends to be too small in time series data. A too-small RR has a devastating impact on COPU, as it causes the chaotic error information in the loss to overwhelm the effective information. CEP conducts an in-depth analysis of the cross-correlation characteristics of time series data, suppressing homogeneous information while accessing heterogeneous information to enhance RR. COPU is an integration of CEP, RR, dual parameters, and trackable parameters; it is a comprehensive time series regression model whose structure is illustrated in Figure 4.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732279218938

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Comment #1 on the second revision

comment:
Dear Reviewers,

We would like to report our latest revisions. In response to reviewer VW2o's request, we have added Section E in the appendix, where we provide a detailed analysis of the rationality of the COPU loss function (Equation 9). We have demonstrated that this loss function employs the orthogonal projection from $\tilde{x}$ to $\tilde{x}y$, which effectively restores the MSE, with computational accuracy measured by the RR metric. We also plan to include additional experimental results for SCINet, Pyraformer, TimesNet, and TimeMixer on the SRU, Debutanizer, ETTh2, and ETTm2 datasets. We will make every effort to complete these experiments within the next three days.

For your convenience, we have excerpted the key portions of Section E below. Reviewers may refer to these in conjunction with our **"Response #1 to Three Important Issues”**. We hope that these explanations can assist the reviewers in better understanding and appreciating our manuscript.

> In our previous discussion, we introduce the properties of the minimum variance estimator. Next, we will demonstrate that COPU is the augmented minimum variance estimator from $\tilde{x}$ to $\tilde{x}y$. Note in the derivation of COPU, the concepts of samples and features are interchanged. 
AOPU has already provided a detailed proof that the linear minimum variance estimator from $x$ to $y$ has the form $\text{E}[y]$+$\text{R}\_{yx}\text{R}\_{xx}^{-1}(x-\text{E}[x])$. 
Since the data are normalized, we assume for convenience that all variables have a mean of zero, where $\text{R}\_{yx}=\text{E}[y^Tx]$. Therefore, $\text{R}\_{yx}\text{R}\_{xx}^{-1}x$ simplifies to the linear minimum variance estimator from $x$ to $y$, which is also known the orthogonal projection.
>
> $\quad$
> 
> Consequently, the minimum variance estimator from $\tilde{x}$ to $\tilde{x}y$ is $\tilde{x}^T(\tilde{x}^T\tilde{x})^{-1}\tilde{x}^T\tilde{x}{y\_{copu}}$, noting that we have performed a transposition here. In this context, $R_{\tilde{x}\tilde{x}}^{-1}$ corresponds to $(\tilde{x}^T \tilde{x})^{-1}$, and $R\_{\tilde{x}y}$ corresponds to $\tilde{x}^T \tilde{x} y\_{\text{copu}}$. Here, $y\_{\text{copu}}$ represents the output of the forward process of COPU, i.e., $\tilde{x}^T \theta\_r$. Since $\tilde{x} R\_{\tilde{x}\tilde{x}} R_{\tilde{x}y}$ represents the minimum variance estimator from $\tilde{x}$ to $\tilde{x}y$, COPU uses $R\_{\tilde{x}\tilde{x}} R_{\tilde{x}y}$, i.e., $(\tilde{x}^T \tilde{x})^{-1} \tilde{x}^T m$, as the minimum variance estimator of $y$ to compute the loss in Equation 9. For this method to be effective, we must be able to restore $y$ from $\tilde{x}y$ given $\tilde{x}$; this requires that $\tilde{x}$ must be of full column rank, i.e., a larger RR.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733289765897

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary

comment:
Dear PCs, ACs, and all reviewers,

As the rebuttal-discussion period draws to a close, We would like to provide a summary of our work and the rebuttal-discussion process to offer clarity.

We proposed COPU and RR to advance our understanding of NNs in time series analysis. 
We proved that RR is a unique and indispensable core metric. 
An RR significantly less than 1 implies a high degree of redundant information in the time series field, suggesting that conventional feature extraction modules may be unsuitable. 
We introduced CEP to enhance RR. 
CEP increases RR by amplifying heterogeneous information while compressing homogeneous ones. 
We proposed COPU as a regression network model. 
We conducted extensive and detailed experiments to demonstrate COPU's advancements, specifically the improvements over AOPU in robustness and over SOTAs in performance.



During the rebuttal, we primarily:
1. Clarified certain misunderstandings.
2. Corrected typos, incorrect reference types, and errors in paragraph sequencing.
3. Added experiments on additional datasets and with additional SOTAs.
4. Provided theoretical proofs affirming the unique and indispensable nature of RR.

After the rebuttal, reviewers mbUm and bobw maintained their ratings (6$\rightarrow$6,6$\rightarrow$6). Reviewer 1q18 increased the rating (3$\rightarrow$5$\rightarrow$6). Reviewer VW2o decided to reject our work (5$\rightarrow$3).

We are not convinced by reviewer VW2o's decision. 
Reviewer VW2o did not provide sufficient reasoning and did not engage in effective discussion with us. 
Our work introduces unique innovations, uncovers significant issues within the domain, and offers valuable insights for all researchers in this field. 
We also met reviewer VW2o's requirements on additional experiments and proof.
However, reviewer VW2o did not mention these critical aspects at all in the rationale for rejection. 
PCs, ACs, and other reviewers can check our discussions with reviewer VW2o along with our global responses if interested.


We hope this summary will assist PCs, ACs, and all reviewers in making the final decision.

Sincerely,

All authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a data-specific structural improvements that is an uniquely novel approach. Authors propose a Cross-correlation Enhanced Perceptron (CEP) to achieve deep nonlinear modeling for time series data Furthermore, the random gaussian matrix with CEP
is introduced as the augmentation interface to develop a COPU, ensuring the accuracy of natural gradient calculations. The CEP, smart utilisation of RR and COPU are solid contributions.
Strengths:
1.Timely and Novel Research Direction, innovative, superior and stable performance .[Reviewer mBUm, bobw, VW2o]
2. Good motivation, good metric and good choice of dataset demonstration [Reviewer 1q18]
3. Novel introduction of utilizing RR as a metric to quantify the proportion of linear dependencies transformed into independence by the network, thereby reflecting the nonlinear capacity

Weaknesses:
1. While we agree with the authors claim that " When the number of layers equals 2, CEP’s performance is on par with other comparative methods; when the number of layers reaches 7, CEP surpasses all other NNs, both in stability and convergence accuracy", From Table 1, the performance of COPU seems quite similar to the baselines across the two data sets and various stacks - esp the improvement in performance in comparison to MLP is not very substantial

2. Evaluations done only on two data sets, and the recent foundation models for time-series forecasting has been ignored - although these models perform forecasting, they can also be used for regression based on their representations [PatchTST etc: Would like to bring this paper to the author's attention: https://arxiv.org/abs/2405.02358]

3. Authors make this claim: We wish to emphasize that CEP makes COPU much more robust to the change of RR while other NN structures do not. This can be attributed to CEP’s ability to maintain high RR during training

While the method is interesting, the definition of metrics are valuable contributions, the paper could benefit from more evaluations on more time-series data sets and more baselines. Furthermore, the impact of the method is also not very substantial (Although there are performance gains in regression problems)