Decoupling Variable and Temporal Dependencies: A Novel Approach for Multivariate Time Series Forecasting

Authors: zhou han, Yuntian Chen
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Time Series Forecasting, transformer, multivariate time series forecasting

Abstract:
In multivariate time series forecasting using the Transformer architecture, capturing temporal dependencies and modeling inter-variable relationships are crucial for improving performance. However, overemphasizing temporal dependencies can destabilize the model, increasing its sensitivity to noise, overfitting, and weakening its ability to capture inter-variable relationships. We propose a new approach called the Temporal-Variable Decoupling Network (TVDN) to address this challenge. This method decouples the modeling of variable dependencies from temporal dependencies and further separates temporal dependencies into historical and predictive sequence dependencies, allowing for a more effective capture of both. Specifically, the simultaneous learning of time-related and variable-related patterns can lead to harmful interference between the two. TVDN first extracts variable dependencies from historical data through a permutation-invariant model and then captures temporal dependencies using a permutation-equivariant model. By decoupling variable and temporal dependencies and historical and predictive sequence dependencies, this approach minimizes interference and allows for complementary extraction of both. Our method provides a concise and innovative approach to enhancing the utilization of temporal features. Experiments on multiple real-world datasets demonstrate that TVDN achieves state-of-the-art performance.

Decision: reject
Reviewer scores: 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1730622563090

thread_type:
official_review

<thread_object>
creator: reviewer_jesy

<content>
summary:
This paper introduces a dual-phase deep learning network architecture called the Temporal-Variable Decoupling Network(TVDN), which decouples the modeling of variable dependencies from temporal dependencies. First, during the variable dependence learning phase, the Cross-Variable Encoder (CVE), a permutation-invariant model, completely disregards the temporal dependence of the sequence and only extracts cross-features between variables, generating an initial prediction sequence. Once the CVE stabilizes, the second phase shifts to temporal dependency learning. The Cross-Temporal Encoder (CTE) divides time series dependence into two parts: Historical Sequence Temporal Dependency (HSTD), and Prediction Sequence Temporal Dependency (PSTD). The outputs of HSTD and PSTD are then fused to correct the initial prediction from the variable dependence learning phase. The approach reduces the risk that overemphasizing temporal dependencies can overfit the model, and enables broader parameter space exploration. The experiment in this paper shows that TVDN achieves state-of-the-art (SOTA) performance in electricity, traffic, weather, four ETT, and exchange fields by comparing it with some of the latest Time-Series Forecasting Transformer (TSFT) methods.

strengths:
1.	TVDN decouples the modeling of variable dependencies from temporal dependencies to reduce the interference between the two, enhance the utilization of temporal features, and improve forecasting accuracy.
2.	TVDN adopts a dual-phase architecture, starting with CVE to learn inter-variable dependencies and then shifting to CTE focused on temporal dependency learning. The two-phase approach effectively addresses the interference of variable and temporal learning, which avoids leading to a degradation of models’ performance.
3.	The proposed method demonstrates SOTA performance in real-world forecasting benchmarks by capturing complex dependencies across variables and time. This is achieved with minimal computational overhead, making the model efficient and scalable.

weaknesses:
1.	The experiment focuses on predicting accuracy and sensitivity to temporal dependencies but does not assess computational overhead and resource requirements. To provide a more comprehensive evaluation, it is recommended to include specific metrics for computational cost, such as training time, inference time, memory usage, and FLOPs. By reporting these metrics, the paper could offer a clearer picture of the model’s practical feasibility, particularly for deployment in large-scale, real-time, or resource-constrained environments.
2.	The experiment in Section 4.3 shows that the model’s performance significantly declines when the temporal order is shuffled, indicating a strong dependency on temporal features. The high dependency may decrease the model’s effectiveness on datasets that don’t have clear temporal patterns, limiting its application domain.

questions:
1.	Given the complexity of TVDN’s dual-phase network structure, could the authors provide a quantitative comparison of computational costs versus accuracy gains relative to baseline models to allow for a more objective assessment of potential trade-offs between accuracy and efficiency?
2.	In real-world applications, temporal sequences are often affected by various types of noise, which may impact model performance. Could the paper conduct experiments by introducing specific noise types, like Gaussian noise, to simulate random disturbances or missing values to simulate incomplete data, to observe TVDN's performance in these noisy environments, and to assess its robustness to noise?
3.	This paper has shown that TVDN performs excellently on various datasets. Is there any limitation to the model? In cases of data scarcity or extremely complex intervariable relationships, can the model maintain its performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730631348120

thread_type:
official_review

<thread_object>
creator: reviewer_tvns

<content>
summary:
In this paper, the authors argue that overemphasizing temporal dependencies can destabilize Transformer-based models, increasing their sensitivity to noise, overfitting, and weakening their ability to capture inter-variable relationships. Therefore, they propose a new Temporal-Variable Decoupling Networks (TVDN) which decouples the modeling of variable dependencies from temporal dependencies to address this challenge.

strengths:
1.	The proposed TVDN decouples the modeling of variable dependencies and temporal dependencies which useful in long-term time series forecasting.
2.	TVDN also separates the modeling of temporal dependencies into historical dependencies and predictive dependencies, with the latter having received relatively little attention in previous works.

weaknesses:
1.	In line 61, the authors said that “linear models and Cross-Variable Transformers do not extract accurate temporal dependencies because they essentially map historical series as unordered sets … ”, I disagree with the assertion that these models treat historical series as unordered sets. for example, the linear models predict future time steps by the equation $\hat{x}_{i+1}=\omega_{1}x_{i-L+1}+…+\omega_{L}x_{i}$, clearly treating historical series as ordered sets.
2.	In equation (5) and (6), the authors define $T^{0}=Z_{h}+ Z_{CVE}$; however, in figure 2, it seems that the input of PSTD is consists only $Z_{CVE}$.
3.	Also in figure 2, I cannot find the step (2) in the method section.
4.	In line 246, $Z_{proj}$ is mentioned without a prior definition, I believe it should be $Z_{h}$.
5.	In line 215, the notation $Z_{CVE} \in {R}^{O \times D}$ is used, yet I cannot find the definition of $D$.
6.	In line 315,“the permutation-invariant models overlook dynamic temporal features ”seems inappropriate. 
7.	In line 332, I disagree with the claim that “we identified that the bottleneck of the traditional Transformer model lies in the ineffective utilization of historical sequence information”, because the experimental results of PatchTST have shown that longer historical sequences will lead better performance. In addition, the reason why the transformer model performs poorly is that it over-captures channel dependencies, which can also be seen in the experimental results of PatchTST and DLinear. Furthermore, the poor performance of the Transformer model can be attributed to its tendency to over-capture channel dependencies, as evidenced by the experimental results of both PatchTST and DLinear.
8.	The experiments are unfair, since PatchTST and DLinear perform better with longer historical windows. Although the authors have conducted experiments with $L=144$, I suggest the authors also set $L=336$ and $L=512$, and compared the results with PatchTST and DLinear.
9.	In Figure 9, Transformer performs better than Transformer Decoder only which is insistent with  ”even with a significant reduction in historical information …”in Line 912.

questions:
please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730834889847

thread_type:
official_review

<thread_object>
creator: reviewer_hsgf

<content>
summary:
In this paper, the authors investigate the problem of multivariate time series forecasting. The authors claim that overemphasizing temporal dependencies can destabilize the model making the model sensitive to noise. And the simultaneous learning of time-related and variable-related patterns can lead to harmful interference. So the authors propose the temporal-variable decoupling network to model the temporal and variable relationships respectively. The authors evaluate the proposed method on several datasets and achieve good performance.

strengths:
N.A.

weaknesses:
1.	The relationship of inter-variables in the time sequence should have an instantaneous effect, but I am surprised that the author did not quote related work in this area [1,2].  
2.	Why do learning time-related and variable-related patterns simultaneously harm inference? Please provide some rigorous evidence.  
3.	The author said that Transformer puts too much emphasis on extracting time-dependent patterns and did not provide enough evidence that this statement was suspicious of the rationality of this assertion. From the perspective of the data generation process, future data generation has also received the effects of time delay and cross variables at the same time, which is the opposite of separating these two factors at the same time. This method highlights the intra-and inter- time series relationships, but many methods have studied this problem, such as [3].  
4.	Why does the proposed method need to model cross-variable relationships and then model cross-temporal relationships? What is the motivation behind it, what if it is in turn?  
5.	More recent comparison methods need to be considered, for example [4] [5].

[1] . Identification of hidden sources by estimating instantaneous causality in high-dimensional biomedical time series.  
[2] . Modeling nonstationary time series and inferring instantaneous dependency, feedback and causality: An application to human epileptic seizure event data.  
[3] Yu, Guoqi, et al. "Revitalizing multivariate time series forecasting: Learnable decomposition with inter-series dependencies and intra-series variations modeling." arXiv preprint arXiv:2402.12694 (2024).  
[4] Jia, Yuxin, et al. "WITRAN: Water-wave information transmission and recurrent acceleration network for long-range time series forecasting." Advances in Neural Information Processing Systems 36 (2024).  
[5] Xu, Zhijian, Ailing Zeng, and Qiang Xu. "FITS: Modeling time series with $10 k $ parameters." arXiv preprint arXiv:2307.03756 (2023).

questions:
Please refer to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732619853833

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall response

comment:
We sincerely thank all reviewers for their constructive comments and suggestions. We have improved the manuscript by incorporating their feedback, especially additional results and discussions on motivation, performance, and efficiency. We have uploaded a revised paper with new material highlighted and marked with reviewer names. 

As reviewers  noted, our core contribution is decoupling cross-variable and temporal dependency learning, and integrating historical and prediction sequences in our prediction paradigm. With less overhead, we achieved  SOTA accuracy on multiple datasets, demonstrated robustness under various noise levels and missing rates, and showed improved performance with longer historical windows.

Beyond these, we sincerely hope that the reviewers will consider the inspirational aspects of our work:

- **Complete Decoupling in Gradient Propagation**: Our decoupling is entirely in the gradient propagation dimension. This strategy may have **general applicability** in multivariate time series forecasting. Because as shown in Section 4.2, for all used datasets, models focusing on cross-variable learning show minimal or negative accuracy gains, while **switching to temporal learning leads to continuous improvements.** And combining both modes may result in slight gains or negative growth. This could inspire future work to decouple different patterns at the gradient level (eg., decomposing temporal patterns like seasonality and trends).
  
- **Integrating Historical and Prediction Sequences**: Dividing time series into historical and prediction sequences differs from methods focusing only on prediction sequences. We construct the prediction sequence $s_{\text{prediction}}$ in the first phase and form a complete sequence $z = [y_{\text{history}}, y_{\text{prediction}}]$. Using $z$ as input for another phase transforms the task from $x \rightarrow y_{\text{prediction}}$ to fine-tuning $z \rightarrow z'$. This offers two benefits:
  
  - **Expanded Horizon**: Directly predicting longer horizons achieves better accuracy than multi-step autoregressive decoding [1][2]. Including historical sequences may further expand the horizon.
    
  - **Leveraging Non-Stationary Information**: Normalizing input data removes non-stationary information, potentially limiting models before denormalization [3][4]. Our joint sequence $z = [y_h, y_p]$ allows the model to reconsider non-stationary information globally. Expanding the receptive field to include non-stationary information could be inspirational (e.g., concatenating $y_{\text{history}}$ and $y_{\text{prediction}}$ may capture local dependencies near their junction).
    
------
**Summary**

1. In response to reviewer **hsgf**'s comments on instantaneous variable relationships, motivation, effects of time delays and cross-variables, modeling order, and comparisons with new methods:

- **Instantaneous Variable Relationships (Appendix I)**
  
- **Motivation**: Expanded in Introduction.
  
- **Simultaneous Effects of Time Delays and Cross-Variables**: Discussed in Appendix I.
  
- **Modeling Order of Cross-Variables and Temporal Dependencies**: See Appendix H.
  
- **Comparison with New Methods**: Added in Appendix B.
  

2. To address reviewer **tVnS**'s comments, we corrected paper details, added experiments on extending historical sequence length, and discussed possible improvement sources in PatchTST and Dlinear. Main revisions:

- **Clarity and Rigor**: Improved in Sections 1, 3, and Appendix F.
  
- **Experiments on Historical Sequence Length**: Added in Appendix D.
  

3. For reviewer **Jesy**'s comments, we addressed "strong dependency on temporal features," added efficiency and robustness experiments:

- **Temporal Feature Dependency**: Discussed in Figure 6.
  
- **Efficiency Analysis**: Added in Section 4.4.
  
- **Robustness Analysis**: Added in Section 4.4.
  
---

**References**

[1] Garza A, Mergenthaler-Canseco M. TimeGPT-1. *arXiv preprint* arXiv:2310.03589, 2023.

[2] Zeng A, Chen M, Zhang L, et al. Are transformers effective for time series forecasting? In *Proceedings of the AAAI Conference on Artificial Intelligence*, 2023, 37(9): 11121-11128.

[3] Liu Y, Wu H, Wang J, et al. Non-stationary transformers: Exploring the stationarity in time series forecasting. *Advances in Neural Information Processing Systems*, 2022, 35: 9881-9893.

[4] Ma X, Li X, Fang L, et al. U-Mixer: An Unet-Mixer Architecture with Stationarity Correction for Time Series Forecasting. In *Proceedings of the AAAI Conference on Artificial Intelligence*, 2024, 38(13): 14255-14262.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a Temporal-Variable Decoupling Network (TVDN) to address the issue of overemphasizing temporal dependencies in multivariate time series forecasting, which the authors argue can destabilize models and lead to sensitivity to noise. The paper achieves a state-of-the-art performance on several datasets. However, as raised by the reviewer, the justification of claims and the experiences are somewhat limited.