TimeCapsule:  Solving the Jigsaw Puzzle of Long-Term Time Series Forecasting with Compressed Predictive Representations

Authors: Yihang Lu, Yangyang Xu, Xianwei Meng, Enhong Chen
Primary area: learning on time series and dynamical systems
Keywords: multivariate long-term time series forecasting; deep learning; infomation tensor modeling

Abstract:
Recent deep learning models for long-term time series forecasting (LTSF) often emphasize complex, handcrafted designs and traditional methodologies, while simpler architectures like linear models or MLPs have occasionally outperformed these intricate solutions. In this paper, we revisit and organize the core ideas behind several key techniques, such as redundancy reduction and multi-scale modeling, which are frequently employed in advanced LTSF models. Our goal is to streamline these ideas for more efficient deep learning utilization. To this end, we introduce TimeCapsule, a model built around the principle of high-dimensional information compression that unifies these key ideas in a generalized yet simplified framework. Specifically, we model time series as a 3D tensor, incorporating temporal, variate, and level dimensions, and leverage mode production to capture multi-mode dependencies while achieving dimensionality compression. We propose an internal forecast within the compressed representation domain, supported by the Joint-Embedding Predictive Architecture (JEPA) to monitor the learning of predictive representations. Extensive experiments on challenging benchmarks demonstrate the versatility of our method, showing that TimeCapsule can achieve performance comparable to state-of-the-art models. More importantly, the structure of our model yields intriguing empirical findings, prompting a rethinking of approaches in this area.

Decision: withdrawn
Reviewer scores: 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730207716836

thread_type:
official_review

<thread_object>
creator: reviewer_nedg

<content>
summary:
This paper integrates existing time series analysis methods including temporal, variate, multi-scale, etc. to deal with various multivariate time series scenarios. It proposes a corresponding preprocessing and prediction method framework. And applies the JEPA method to the highly compressed representation space and adds the corresponding loss.

strengths:
The research direction of the article is relatively innovative. Indeed, integrating various current methods is a good direction, and the methods used are logically reasonable. The article's expression and illustrations are overall very clear.

weaknesses:
1. I think there is a certain difference between the Target and context inputs of JEPA in this article and, for example, I-JEPA. In I-JEPA, the context and Target overlap. Here, I think it is worthy of further demonstration. Based on more sufficient experimental evidence, a more appropriate design should be given. I think the application of JEPA to TS problems here is not studied in-depth.
2. I think it is necessary to compare the effects and efficiencies of MLP and the Transformer Decoder. The lack of experiments means that it cannot be directly concluded as unreasonable.
3. For the order of TransBlocks and the comparison with the parallel method, there is a lack of experimental support.
4. The elaboration of Decomposition classification and level is not very clear. I think adding some visualizations or results after the processing of each TransBlock (T,V,L) and the analysis of how to correspond to the function will be more clear and intuitive.
5. I think the method proposed in the article has some limitations. For example, the selection of the lookback window is an unconventional 512. Of course, the article also analyzes different lengths. But it is obvious that the larger is more effective. I suggest clarifying this point and whether there are other limitations. I think limitations are also a contribution to the development of the field.
6. No code is provided for reproducibility.

questions:
1. Line 281-283 Utilizing JEPA, what specific inductive bias is introduced?
2. Why is TransBlocks in this order? Is there a difference in different processing orders for different types of datasets? What is the difference compared to parallel processing?
3. Why only select one subset, PEMS04 of PEMS? I think the advantages and limitations of a method should be supported by comprehensive experimental results, rather than conducting a large amount of analysis based on the performance on a single example dataset as in the Main Results part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730256081315

thread_type:
official_review

<thread_object>
creator: reviewer_5fak

<content>
summary:
This paper introduces **TimeCapsule**, a model aimed at simplifying and unifying key techniques in long-term time series forecasting (LTSF) while achieving comparable performance to complex, state-of-the-art models. TimeCapsule models time series as a 3D tensor, incorporating temporal, variate, and level dimensions, and leverages mode production to capture multi-mode dependencies while achieving dimensionality compression. Experiments on challenging benchmarks highlight TimeCapsule's effectiveness.

strengths:
1. This paper provides a new insight into treating time series as a 3D tensor.
2. The authors try hard to categorize the advanced LTSF models into four groups based on their core techniques.
3. The Figures are simple and easy to understand, effectively presenting the model's design concept intuitively.

weaknesses:
Generally, the technique of the manuscript is partially sound, and some major concerns are listed below:

1. The motivation of this paper is unclear, as noted in question 1. There is no evidence provided to demonstrate that combining effective components from related work will result in a better model. Even in the ablation study, the authors do not discuss this problem. Authors can analysis  why combining these components should lead to improved performance.
2. The authors review and categorize existing methods for long-term time series forecasting, stating that their classification is based on key techniques. However, the four categories are not on the same level. For instance, "Generalized Linear Dependency" would more accurately correspond to methods addressing nonlinear dependencies, representing a classification by learning framework. Yet, the authors unexpectedly group this with specific modeling techniques that focus on better leveraging time series characteristics within the same category.  Please provide a clearer justification for why you grouped these different types of approaches together, if you choose to keep the current categorization.
3. The authors should provide explanations for the function of symbols nearby, especially for less common symbols, such as $\times_2$ in Equation (2), even though they are defined in the appendix.

questions:
1. The classification criteria in Figure 1 are unclear. And what are the advantages of each category, and what are the benefits of combining these advantages? I believe this information is essential for the paper. Additionally, in the overlapping section on the right of the figure, does the “…” indicate a lack of relevant work, or that the authors couldn’t find any? If none exist, why is there no related work in this area?
2. In section 3.1, the encoding part includes a series of transformer-based blocks, and the decoder operates with a series of MLPs. Could this increase the model's complexity?
3. Can you further explain “to maintain the same volume of information while shortening the length of each dimension, thereby reducing the computational cost of self-attention.” in line 237?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730582085188

thread_type:
official_review

<thread_object>
creator: reviewer_cdzq

<content>
summary:
This paper proposes TimeCapsule, a new architecture for time-series forecasting by combining key model-blocks introduced in recent works. Additionally, this paper employs a self-supervised loss based on JEPA to enhance the forecasting performance. Overall, TimeCapsule consists of an encoder to learn representations that are then used for forecasting with an MLP-based decoder. The key model-components seem to be: multi-level modelling, multi-mode dependency, and compressed representation forecasting.

strengths:
* The paper presentation is very good considering that the model architecture being proposed in the paper is rather complex. Fig. 2 was also very helpful in understanding the various model components. 
* It is interesting to see the JEPA loss being applied on time-series. 
* The performance of the new architecture is evaluated on long-term time series forecasting datasets. 
* The paper presents some ablations and also analyses the compute-efficiency.

weaknesses:
* The main weakness, in my opinion, is that the paper only considers time-series forecasting to evaluate the model. To establish generality of the model architecture, it may be important to also consider other tasks such as classification, self-supervised learning, or irregular time-series forecasting/imputation. 
* The paper makes many contributions and it may be more valuable to carefully ablate each of these new components. Some of the design decisions/ alternatives could be clearly elaborated. For example, why apply EMA over input time-series Y in Eq. 15? Why introduce Gaussian noise at the start of the block (L. 251) --- what alternatives can we consider in practice? 
* The results suggest that some gains by JEPA loss are marginal. It'd have been nice if the authors demonstrated self-supervised pretraining based on JEPA helped various downstream tasks (e.g., forecasting/classification/etc). 

Overall, I feel that this architecture is interesting and can be valuable for future time-series applications. However, due to the introduction of many new architectural blocks (not necessarily a bad thing), the performance on other time-series applications is not clear. It would help to at least demonstrate self-supervised pretraining based on JEPA for downstream applications; this may be compared against self-supervised learning by Patch-TST (e.g., JEPA/masked-pretraining).

questions:
1. Based on your experiments, is TimeCapsule architecture key for the JEPA training? For example, if you tried to apply JEPA to PatchTST architecture, would we notice any improvements? 
2. In line 368, it says "Besides, our model outperforms PatchTST, particularly on larger datasets, without employing patching techniques." Why is it beneficial to eliminate patching? 
3. What kind of attention is used in the Tunnel layers?
4. What is coefficient considered for JEPA loss? 

Minor typo: Both MvTS and MTS seem to refer to multivariate time series and it may be better to use just one abbreviation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605342739

thread_type:
official_review

<thread_object>
creator: reviewer_mw4b

<content>
summary:
This manuscript introduces a framework that adds an extra level dimension to time series data and employs mode production to capture multi-mode dependencies. It utilizes transformer blocks to capture dependencies across temporal, level, and variate dimensions, respectively.

strengths:
This manuscript is well-written, logically organized, and easy to follow. It provides a comprehensive analysis of recent SOTA methods for MTS forecasting. The Mode Specific Multi-head Self-attention utilizes a vanilla attention mechanism to capture temporal dependencies. Although patching has been a widely used technique in recent years to reduce the length of input for attention mechanisms, this manuscript employs low-rank transforms as a replacement, achieving promising efficiency.
TimeCapsule applied the JEPA approach to time series forecasting and demonstrates its effectiveness. TimeCapsule has achieved superior performance in both accuracy and computational speed. Extensive ablation studies were conducted, which underscore the effectiveness of the core components of TimeCapsule.
In my opinion, this manuscript is novel and likely to generate interest among researchers in the field of time series analysis.

weaknesses:
Existing transformer-based methods partition time series sequences into patches of various sizes to capture multi-scale temporal dependencies. TimeCapsule introduces an additional level dimension to the 2-dimensional MTS data to capture multi-scale dependencies. However, the explanation of how multi-scale dependencies are captured is lacking. Specifically, definitions of how multi-scale levels and variate dependencies are defined are needed.

In Figure 2, the components below ReVin_Norm are depicted in a way that may be misleading. Consider using more intuitive shapes to represent tensors.

Table 2 presents an ablation study of the residual information connection. For ETTh2 and ETT m2 datasets, the accuracy achieved with original information is similar to that achieved with residual information connections. In addition, the gap between the two settings increases as the forecasting horizon extends. An analysis of these findings would be beneficial.

questions:
I have a few suggestions that may enhance the quality of the paper: 

(a) Please consider increasing the text size in some figures to improve readability. 

(b) Could you explain the rationale behind connecting the T-TransBlock, L-TransBlock, and V-TransBlock in series? Have you tested configuring these blocks in parallel, concatenating their outputs, and then using a linear layer to generate predictions? 

(c) Consider adding a description in Section 3.2 of how multi-scale dependencies for Temporal, Level, and Variate dimensions are captured.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732612117926

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions

comment:
We would like to express our gratitude to all the reviewers for their insightful feedback and constructive comments, which have significantly improved the quality and completeness of our paper.

In this work, we propose a novel framework for Long-term Time Series Forecasting (LTSF) by considering four key ideas. Unlike prior approaches, we design and integrate these elements in a streamlined deep-learning architecture, ensuring both effectiveness and efficiency. We have evaluated our model in a comprehensive and challenging benchmark, demonstrating its advantages in terms of both performance and computational efficiency. Additionally, we conducted experiments to validate the effectiveness of each model component, while exploring interesting problems in the field through the proposed framework. 

Overall, TimeCapsule is a novel, effective, and versatile model with a reasonable computational cost for LTSF. Through this framework, one can investigate many potential ingredients related to LTSF, as partially shown in this research.

We feel pleased that the reviewers have acknowledged the novelty, contributions, and presentations of our work:

- **Reviewer MW4b**: "Well-written, logically organized, and easy to follow"; "Novel and likely to generate interest among researchers in the field of time series analysis."
- **Reviewer CdzQ**: "The paper presentation is very good, considering the model architecture is complex"; "This architecture is interesting and can be valuable for future time-series applications"; "It is interesting to see JEPA loss applied to time series."
- **Reviewer 5FaK**: "Provides new insights into treating time series as a 3D tensor"; "The technique of the manuscript is partially sound."
- **Reviewer Nedg**: "The research direction is relatively innovative"; "Integrating various current methods is a good direction, and the methods are logically reasonable"; "The article’s expression and illustrations are overall very clear."

The reviewers also raised valuable and constructive concerns. In response, we have made the following revisions:

- **Typos** (All Reviewers): We carefully reviewed the manuscript for any typographical errors and resolved them.

- **Motivation and Model Design** (All Reviewers): We strengthened the explanation of our motivation in $\underline{\text{Section 1, 2}}$. Also, we clarified several minor design choices within the model in $\underline{\text{Section 3}}$, with further explanations and empirical results.

- **Interpretability and Further Exploration** (Reviewers MW4b, Nedg): We expanded the explanations on how multi-mode dependencies and multi-level properties are captured in  $\underline{\text{Section 3}}$. We also added a detailed visual exploration of this capability in  $\underline{\text{Appendix E}}$, which uncovered several interesting new insights. Additional analyses of these findings are included in  $\underline{\text{Section 4}}$ and  $\underline{\text{Appendix D}}$.

- **Generality** (Reviewer CdzQ): To broaden the scope of the model's applicability, we explored TimeCapsule’s potential for self-supervised learning with JEPA, as well as its performance in classification tasks, in $\underline{\text{Appendix G}}$.

All updates are highlighted in blue. We hope our response has fulfilled the reviewer's expectations and would be open to answering any further questions.
</content>
</thread_object>

</thread>


Metareview: