Time Series Representation Models for Multivariate Time Series Forecasting and Imputation

Authors: Robert Leppich, Michael Stenger, Daniel Grillmeyer, Vanessa Borst, Samuel Kounev
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: time series, attention, forecasting, imputation, multivariate

Abstract:
We introduce a multilayered representation learning architecture called Time Series Representation Model (TSRM) for multivariate time series forecasting and imputation. The architecture is structured around hierarchically ordered encoding layers, each dedicated to an independent representation learning task. Each encoding layer contains a representation layer designed to capture diverse temporal patterns and an aggregation layer responsible for combining the learned representations. The architecture is fundamentally based on a Transformer encoder-like configuration, with self-attention mechanisms at its core. The TSRM architecture outperforms state-of-the-art approaches on most of the seven established benchmark datasets considered in our empirical evaluation for both forecasting and imputation tasks while significantly reducing complexity in the form of learnable parameters. The source code is available at https://anonymous.4open.science/r/TSRM-D7BE.

Decision: reject
Reviewer scores: 5, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730072066430

thread_type:
official_review

<thread_object>
creator: reviewer_khpu

<content>
summary:
This paper proposes a multilayered representation learning architecture for multivariate time series forecasting and imputation, achieving superior performance on seven public datasets.

strengths:
The methodology presentation is clear, the code is provided, and the performance surpasses multiple models.

weaknesses:
1. **Unclear motivation:** The paper does not discuss which limitations of existing methods it aims to address, lacking a direct explanation in this regard.  
2. **Limited scope of tasks:** Although the paper proposes a representation learning approach, it is only applied to forecasting and imputation, which are tasks of the same nature. I hope to see applications in classification and anomaly detection as well.  
3. **Lack of discussion and discussion on most related works:** The paper compares only with end-to-end models, but it misses comparisons with the most relevant representation-based methods, such as Cost[1]. Additionally, recent foundation models, such as MOMENT[3] and Moirai[2], could also be leveraged as representations. Please discuss differences and make comparisions.

[1] Woo, Gerald, et al. "CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting." International Conference on Learning Representations.


[2] Woo, Gerald, et al. "Unified Training of Universal Time Series Forecasting Transformers." Forty-first International Conference on Machine Learning.


[3] Goswami, Mononito, et al. "MOMENT: A Family of Open Time-series Foundation Models." Forty-first International Conference on Machine Learning.

questions:
Please check limitations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730570687084

thread_type:
official_review

<thread_object>
creator: reviewer_hgyz

<content>
summary:
This paper introduces new architecture for forecasting/imputation of time-series. Overall, it introduces encoder blocks comprising of a Representation layer and Merge layer with self-attention in between. The representation and merge layers have no non-linearities and consist of convolution/transposed-convolution layers respectively. The models are evaluated on forecasting/imputation tasks.

strengths:
* Clear presentation and informative figure 1.
* Experiments on both forecasting/imputation. 
* TSRM are lighter than popular models such as TimesNet/Patch-TST in terms of parameter-counts.

weaknesses:
* The paper introduces a lot of hyperparameters. Based on Appendix A.4, the paper searches over 13 possible EncLayers $\times$ 5 possible attention-heads $\times$ 5 possible feature-embedding sizes $\times$ 2 possible attention-functions $\times$ (at least) 4 possible CNN configurations = 2600 configurations! 
* From the table having best hyperparameters, sparse-attention seems to be the best combination across all datasets. But, there seems to be significant variation across different datasets/horizons. Even for the same dataset, different horizon seems to have a different optimal configuration. 
* Although TSRM is lighter than other models and this may help accelerate the search, the computational cost of picking the right hyperparameters cannot be ignored. Also, I couldn't find any analysis of wallclock times (train/inference) in the paper. 
* Table 7 reports the Mean/Std across 5 runs of ETTh2 and weather. From my understanding, this mean/std is computed for running the model with optimal configuration for 5 times. In practice, the hyperparameters should be selected based on 5 runs and then, the corresponding test-performance should be reported. However, this may be very expensive given the hyperparameter space. 
* TimesNet seems to be uniformly better than TSRM on the imputation task. Also, PatchTST is not considered for the imputation task. 
* The model is trained on forecasting/imputation tasks only. Self-supervised learning may also be explored to establish the generality of this architecture. However, the hyperparameter-search may make this expensive. 
* Given the huge hyperparameter space of TSRM, it may be fairer to allow PatchTST to select its patch-size/input-length. Under optimal Patch-TST performance, does TSRM still perform favorably?

questions:
Please see weaknesses.
1) What is $\alpha$ in Eq 4?
2) How does TSRM generalize to different missing ratios at test time?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698560409

thread_type:
official_review

<thread_object>
creator: reviewer_mwng

<content>
summary:
The paper introduces a multilayered representation learning architecture called TSRM designed for multivariate time series forecasting and imputation. TSRM utilizes hierarchical encoding layers, each with a representation layer to capture diverse temporal patterns and an aggregation layer for combining learned representations. Based on a Transformer encoder-like configuration with self-attention mechanisms, TSRM outperforms SOTA approaches on seven benchmark datasets.

strengths:
1. The article emphasizes model interpretability which is crucial for understanding and trusting its predictions.
2.  The article includes links to the source code, enabling researchers to reproduce the results.
3. The article demonstrates the model's effectiveness and generalization capabilities by testing it on datasets from various domains.

weaknesses:
Generally, the technique of the manuscript is partially sound, and some major concerns are listed below:

1. The title of this paper could be clearer. It does not convey the specific techniques or strengths used, nor does it highlight the unique aspects of the work. The authors claim that this method can better handle different time scales, so perhaps emphasizing this could be beneficial. And this is just a suggestion.
2. The paper emphasizes that the targeted tasks are forecasting and imputation. However, the representation learning method in this paper does not specifically address forecasting and imputation, which may require additional clarification. Alternatively, future work could explore extending the method to other tasks.
3. There are too many subsections in the related work, leading to some redundant content. In particular, the "Few/zero-shot learning" subsection covers methods unrelated to this paper, so it could be appropriately condensed.

questions:
Generally, the technique of the manuscript is partially sound, and some major concerns are listed below:
How is the kernel size chosen—is it based on specific criteria, such as the period length, or selected empirically? What about the sensitivity of this parameter? The results using $K$ independent 1D CNN layers with varying kernel sizes have not been compared to those with a uniform kernel size, even in the ablation study, which seems more important than discussing $N$.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730766852819

thread_type:
official_review

<thread_object>
creator: reviewer_b9tq

<content>
summary:
The paper introduces a novel architecture called Time Series Representation Model (TSRM) for multivariate time series forecasting and imputation. The TSRM architecture is based on a Transformer encoder-like configuration with hierarchically ordered encoding layers.  The TSRM architecture integrates representation learning within a multilayered model, where each layer features a distinct representation learning module. Each representation learning layer independently captures representations at a different hierarchical level, restoring the original input dimensions to enable hierarchical stacking of layers independent of the input dimension. The proposed approach achiever similar or better performance than state-of-the-art approaches on forecasting and imputation tasks while significantly reducing complexity in the form of learnable parameters.

strengths:
- The paper focuses on an important problem of time series forecasting and imputation which can be useful in several practical use cases.
- Experiments show that the framework is able to outperform baselines on multiple datasets on the time series forecasting task.
- The paper is easy to follow.

weaknesses:
- One major concern is the motivation behind certain architectural choices, such as the introduction of merge layers and the inclusion of representation layers in each hierarchical level. The ablation studies in Figure 2 show that removing the merge layers results in only a minor drop in performance  (0.379 -> 0.382 and 0.161 -> 0.165) compared to other ablation experiments, which raises questions about their necessity. Additionally, it is unclear whether having multiple representation layers is more effective than using a single set of CNN layers at the beginning of the Transformer architecture, as is common in computer vision. Without clear evidence of performance improvements, the novelty of the proposed approach appears marginal.
- Furthermore, the claim that each representation learning layer captures representations at a different hierarchical level requires qualitative analysis to support it. 
- The paper's claim of achieving state-of-the-art performance on imputation tasks is not supported by the results in Table 2.
- How are masks handled in CNN layers (representation block)?
- Does the proposed approach need to be trained separately for different prediction horizons and different missing ratios or is it generalizable?

questions:
Listed above in the weaknesses section.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The manuscript introduces Time-series representational model that uses hierarchical encoding layer, each with a representational layer and aggregation layer, to capture diverse temporal patterns and combining the learned representations, respectively. 

Weaknesses:
1. Although the above claim has been made in the manuscript, authors responded to reviewers that there is an error in the manuscript, where the hierarchically arranged representation layers, but we are actually talking about the encoding layers. The representation layers are not designed for hierarchical learning, but to encode the input at different levels of abstraction, as explained in Section 3.1. This process is similar to that of TimesNet, where the abstraction levels are determined by Fast Fourier Transformation [1], but differs in that we use static abstraction levels. [See response to Reviewer B9TQ]. 
2. In another response to the Reviewer B9TQ, authors concede that their results outsmarts SOTA on some data sets, while they do not in others. This is a point of contention, although they have revised the abstract to reflect this. Multiple reviewers have highlighted the lack of considerable performance gains by TRSM, in comparison to SOTA [Reviewer B9TQ and Reviewer hGyz]. 
3. Too many hyperparameters and tuning them to optimize results will be cumbersome.
3. As indicated by the reviewers, The manuscript will also need a lot of edits to avoid redundancies in the manuscript.

All in all, this is a good paper with novel method. However, the manuscript needs a thorough overhaul in terms of descriptions of the method and evaluation/ablations. We hope the authors find the review useful enough to revise and resubmit the manuscript to another venue.