Towards Generalisable Time Series Understanding Across Domains

Authors: Özgün Turgut, Philip Müller, Martin J. Menten, Daniel Rueckert
Primary area: learning on time series and dynamical systems
Keywords: Time Series Analysis, Multi-Domain, Self-Supervised Learning

Abstract:
In natural language processing and computer vision, self-supervised pre-training on large datasets unlocks foundational model capabilities across domains and tasks. However, this potential has not yet been realised in time series analysis, where existing methods disregard the heterogeneous nature of time series characteristics. Time series are prevalent in many domains, including medicine, engineering, natural sciences, and finance, but their characteristics vary significantly in terms of variate count, inter-variate relationships, temporal dynamics, and sampling frequency. This inherent heterogeneity across domains prevents effective pre-training on large time series corpora. To address this issue, we introduce OTiS, an open model for general time series analysis, that has been specifically designed to handle multi-domain heterogeneity. We propose a novel pre-training paradigm including a tokeniser with learnable domain-specific signatures, a dual masking strategy to capture temporal causality, and a normalised cross-correlation loss to model long-range dependencies. Our model is pre-trained on a large corpus of 640,187 samples and 11 billion time points spanning 8 distinct domains, enabling it to analyse time series from any (unseen) domain. In comprehensive experiments across 15 diverse applications - including classification, regression, and forecasting - OTiS showcases its ability to accurately capture domain-specific data characteristics and demonstrates its competitiveness against state-of-the-art baselines. Our code and pre-trained weights are publicly available at \url{https://github.com/OTiS-official/OTiS}.

Decision: reject
Reviewer scores: 6, 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730044281664

thread_type:
official_review

<thread_object>
creator: reviewer_txlu

<content>
summary:
The paper presents OTiS, a deep model pre-trained on a large corpus (11B) for general time series analysis. In this paper, the authors highlight the challenge of heterogeneity when applying self-supervised pre-training on time series. An MAE-style pre-training method is adopted to obtain a general tokenizer for multivariate time series, and then different task heads are introduced to complete time series analysis tasks.  The model demonstrates strong performance across 15 diverse applications, including time series classification, regression, and forecasting.

strengths:
1. This paper researches an important question about generalizable time series understanding across diverse domains.
2. This work presents a large pre-training corpus, which can be greatly beneficial if the datasets are released.
3. The method exhibits promising results in handling multivariate time series analysis by leveraging variate and domain signatures.

weaknesses:
1. My major concern is about the novelty of the proposed method: The design of the encoder/decoder is very identical to MAE. Is there any adaptation for the time series modality? For example, considering the inherent reconstruction difficulties of time series and adjusting the mask ratio compared with the vision modality?
2. About the model design towards generalizable time series understanding: As the authors mention an important challenge of heterogeneity, I am slightly unconvinced that a shared unified patch embedding/projector can reflect different semantics among variates and domains, even if the patch is completely the same. Prior to this, Moirai adopted different patch sizes for different frequencies, will it further enhance OTiS?
3. This work adopts learnable embeddings as variate/domain signatures. I am convinced that the signatures can "distinguish" them, but how can they explicitly "capture inter-variate relationships"? This approach may also limit the generalization scope as the learned signatures do not apply to unseen variates/domains during inference.
4. About the experiments: Results of classification are not compared with supervised, trained deep models, for example, TimesNet and ModernTCN. For the regression rask, can you introduce some variate-centric models into this baseline, such as iTransformer? As for forecasting, the average improvement does not seem significant compared with PatchTST. Also, can you provide some explanations about Table 3 why OTiS has a significant improvement on some datasets (such as ETTh2) and a great degeneration on similar datasets like ETTh1?
5. A minor suggestion: the name "dual masking strategy" can be somewhat overstated to me, which generally refers to dual or antagonistic behavior (e.g., minimax). I would prefer to simplify the contribution as a "mixture" (of masking modeling and generative modeling in this paper), which is a common technique in fact. Also, I would like to know how the ratios (25% - 75% in this paper) of the two strategies are determined.
6. The pipeline of using the masked pre-trained models seems still somewhat tedious, i.e., lacking in generalization. Supervised training should be performed after large-scale pre-training. Can the author provide an overall promotion compared with training from random initialization, or try zero-shot generalization on downstream tasks?

questions:
1. Have you tried to pre-train separately according to different domains and then fine-tune it for domain-specific downstream tasks? As observed from Table 1, there are several discrepancies in different domains, such as the frequencies of Economics and EEG. Is it possible that separating datasets to pre-train domain-specific models works better?
2. The proposed method uses a fixed context for pre-training. Padding a large pre-training corpus, which generally contains univariate time series, into a fixed temporal/variate dimension. Will it cause a waste of computing resources?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730383237153

thread_type:
official_review

<thread_object>
creator: reviewer_yujh

<content>
summary:
This paper presents OTiS for multi-domain time series analysis, building on existing pre-training paradigms for time series. It allocates domain-specific variable embeddings to distinguish the heterogeneity of different variables across domains and enhances the model's ability to learn temporal causal relationships through a dual-masking strategy. Additionally, it introduces NCC loss to capture global patterns. Experimental results demonstrate that the proposed method achieves competitive performance in time series classification, regression, and forecasting tasks across multiple domains compared to SOTA methods. Visualization results further highlight the effectiveness and interpretability of the domain-specific variable embeddings.

strengths:
1. The paper is well-written, and the method is easy to understand. The authors clearly articulate how they consider the heterogeneity of different domain time series to achieve multi-domain time series forecasting.

2. This paper focuses on the problem of multi-domain time series analysis, which is crucial for building generalizable foundational models for time series.

3. The experimental section utilizes a large amount of data, and the model is open-source, contributing particular engineering value to the community.

weaknesses:
1. The paper mentions that one of the challenges of cross-domain time series models is the significant differences in temporal dynamics and sampling frequencies among different domains. However, the paper uses the same patch size for all domains when dividing patches, failing to accommodate the unique sampling rates of different domains. This oversight means the paper does not sufficiently consider the differences in sampling rates across domains. Additionally, using a shared patch projector to encode the temporal dynamics within each patch does not adequately address the differences in temporal dynamics between domains. While this approach may be common in previous works, it does not consider the temporal heterogeneity among domains.

2. The method of considering variable heterogeneity through learned variable embeddings is not uncommon. In spatiotemporal prediction, some methods [2][3] have already employed learnable embeddings to explicitly distinguish heterogeneous spatiotemporal patterns by learning time-specific and space-specific parameter spaces.

3. [1] proposed using textual descriptions to label different time series domains for cross-domain time series forecasting, utilizing a channel-independent strategy. In contrast, the domain-specific variable embeddings in this paper correspond to a channel-mixing strategy. I look forward to seeing a comparison between these two strategies in cross-domain time series.

4. The experimental section lacks details about the baselines. How were these methods selected? Were they pre-trained and fine-tuned? If so, what data was used for pre-training and fine-tuning?

5. How does the performance of the proposed method compare to conventional time series classification or forecasting methods trained on a single specific dataset?

[1] UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting, WWW, 2024

[2] Heterogeneity-Informed Meta-Parameter Learning for Spatiotemporal Time Series Forecasting, KDD, 2024

[3] Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting, NeurIPS, 2020

questions:
See the weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730447154090

thread_type:
official_review

<thread_object>
creator: reviewer_ujog

<content>
summary:
Authors spot the important fact that the variate structure is heterogeneous across domains and this structure may represent more complex relationships. Thus, they propose a time series pre-training pipeline called OTiS. The OTiS is composed of a specially designed tokenizer that can add domain-specific signature to the time series and a novel loss for pretraining.

strengths:
- The paper is well-written and easy to follow. 

- Authors spot the important fact that the variate structure is heterogeneous across domains and this structure may represent more complex relationships. 

- The visualization for variate embedding seems to be interesting and insightful. 

- A substantial portion of this research focuses on EEG signals, which presents a novel and promising approach. The authors introduce an innovative method to model a "specific set of systems" that, despite being observed differently—such as TDBrain and SEED with 19 channels versus LEMON with 62 channels—remain comparable.

weaknesses:
- As noted in the strengths, this work addresses the challenge of generalizing across datasets that contain time series of similar systems but are recorded differently, such as variations in sampling rates and physical values. However, the claims regarding cross-domain generalization may be overstated.

- From the perspective of generalized time series analysis, the primary contribution of variate-specific embedding may not be effective in other systems where the interrelationships between variates are not as straightforward as their spatial arrangement (e.g., the electrodes in EEG as depicted in Figure 3 of the manuscript). In different physical systems, two variates may exhibit complex computational relationships (e.g., voltage and current as described by Ohm's Law), complicating the direct modeling of variates as embeddings.

questions:
- How does the domain-specific tokenizer adapt to unseen domains with distinct variate structures?

- Additionally, how does the domain-specific tokenizer generalize across different systems within the same domain? For instance, while both electrical transformers and power generators belong to the "energy" domain, they exhibit differing properties and produce distinct time series readings. How does the sub-domain adaptation discussed in Section 3.1 address this scenario?

- A broader question, not specific to this paper: At what level of granularity should we define the domain?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730579003627

thread_type:
official_review

<thread_object>
creator: reviewer_pz7i

<content>
summary:
This paper proposes a time series model architecture and a pre-training objective. The key idea is that their architecture acknowledges that training and testing time series may have different sampling rates and variables. The authors propose a straightforward tokenization scheme to learn embeddings for different variables, which can get added onto regular patch and temporal embeddings, thereby conditioning the predictions on the measured variables. They then pre-train their model on a collection of existing datasets, and evaluate its performance by finetuning on new datasets for some forecasting, regression, and classification datasets. They find that finetuning their model on new datasets can outperform other recent methods.

strengths:
This paper has many strengths:
* The key idea to condition the model on different variables and domains is good. Indeed many related works effectively ignore this information.
* The paper is overall written quite well and arguments are presented clearly.
* The experiments investigate multiple axes, including ablation of their method and different dataset and model sizes, and visualizations of the embeddings.
* The public data and model weights will help the community build on this work.

weaknesses:
This paper has weaknesses to address:
* The major weakness of this paper is the extremely limited experiments section. There are many experiments, yet almost no explanation of how they're run or interpretation of the results. Most of the results are written like an advertisement, mostly just stating the method outperforms others. This leaves the reader unclear why the performance gains happen. Ultimately it's not clear when/why the findings would generalize. The result is that some claims appear to be quite overstated. For example, L423-L424 states *"embeddings of domains with shared high-level semantics cluster together, as depicted in Appendix E.1. For example, embeddings of mono and stereo audio group closely, as do those of banking and economics."* But this is cherry-picked---Temperature is way closer to Mono and Stereo Audio than Banking is to Economics.
* Similarly, many important experimental details are missing or relegated to the Appendix, and the Appendix also includes almost no explanations or interpretations. For example, the PCA experiments in Figures 3, 7, and 8 aren't explained.
* It's unclear how many variables actually overlap between training/testing, which seems to be a key element to make the model outperform others. Yet this isn't analyzed. Showing that others fail by ignoring other variables should be a key element of the experiments.

questions:
Please feel free to address any misunderstandings I've stated in the weaknesses. Answers to the following questions would help me better calibrate my score:
1. How long does it take to finetune on new tasks?
2. How does the finetuned model perform compared to task-specific models? Are these testing datasets really good cases that need pre-trained models?
3. How do you get the ground truth embeddings in Figure 3?
4. Is there any intuition around what information could be shared across such different domains to make pre-training on them useful?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730602944046

thread_type:
official_review

<thread_object>
creator: reviewer_aff6

<content>
summary:
This paper presents OTiS, a pre-trained foundation model on large-scale time series data to support multi-tasks across domains. Extensive experiments are conducted to demonstrate the powerful performance of the foundation model. This paper is prepared in high quality and can be accepted.

strengths:
1. This paper targets a very important research problem, the time series foundation model. Because the time series data has very high variance across different domains and different tasks. How to integrate them and train one foundation model remains challenging.
2. This paper has a very high quality of preparation. The writing, the organization, and the figure are prepared nicely and with enough details.
3. The results shown in Tab 1, 2, and 3 are competitive compared to baseline TS models.

weaknesses:
1. Add a subsection to show which category baselines will be compared. For example, traditional TS model, deep learning, TS foundation model, etc.
2. I expect a comparison with some SOTA TS foundation models. For example, https://arxiv.org/abs/2405.02358 . If this part is added, that would be great.
3. Currently, the author use fine-tune to adapt the pre-trained model to various downstream tasks. Can you also add one more subsection to test the prompting on this TS foundation model? That would be another great point.

questions:
See details in weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732894927696

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Manuscript Upload

comment:
We thank all reviewers for their constructive and insightful efforts in evaluating this work, we truly believe our work has improved as a result of your suggestions. We have uploaded revised files, with several modifications: 
1. more experiments, including new baselines and zero-shot evaluations;
2. revised experiments section and Appendix, to provide details on the baselines, experimental setup, and results;
3. more ablation studies, regarding the composition of the dual masking strategy and pre-training strategies;
4. more visualisations and examples in the Appendix, including domain signature analysis and latent space analysis.

The above points are marked in red (in both main paper and Appendix).
Besides these points, we have also revised figure captions, formulations, and other minor points in the manuscript. We have addressed the reviewers’ comments on a point-by-point basis below.

Thank you again for the constructive efforts in the comments and reviews,

Authors

P.S.: We are a bit late to the party, but hope for an active and lively discussion with the reviewers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733063368515

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Manuscript Feedback

comment:
Dear Reviewers,

We know you might have other plans on a Sunday, however, we would greatly value your feedback on our rebuttal.

In response to your insightful comments and questions, we have conducted additional experiments and provided detailed analyses in our rebuttal. Although these experiments required a significant part of the discussion period, we believe they have substantially improved the manuscript and we would welcome any further comments or discussion. If you believe that we have adequately addressed the points you raised, we would greatly appreciate an appropriate adjustment to your scores to reflect this. 

Thank you once again for your constructive comments, which have truly enhanced the quality of our study.

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733108648077

thread_type:
official_comment

<thread_object>
creator: area_chair_qyk2

<content>
comment:
Dear reviewers,

Could you please help to take a look at the author responses and let the authors know if your concerns have been addressed or not? Thank you very much!

Best regards,

AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1733183249289

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Manuscript Final Feedback

comment:
Dear Reviewers,

We know you have already put a lot of time and effort into evaluating multiple manuscripts, including our study. As the journey is almost over, we would like to take this final opportunity to ask whether you have enjoyed:

Reviewer ***Aff6***: The detailed summary of the baselines, the comparison with SOTA foundation models, or the additional zero-shot evaluations?

Reviewer ***pz7i***: The more careful explanations of the results, the clarification of the principal component analysis, or our extensive comparison with domain-specific models?

Reviwer ***UJog***: The additional experiments on generalisabilty across domains, the analysis of Weather-specific variate emebddings, or the open discussion on the definition of domains?

If so, we would greatly appreciate a final adjustment of your scores to reflect this. Of course, we are happy to discuss any further comments or concerns, just as we have been engaging with Reviewer ***yUJh*** and Reviewer ***tXLU***. 

Thanks once again for your constructive feedback and contributions, which have truly improved the quality of this study.

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes OTiS, a pre-trained foundation model for multi-domain time series analysis, designed to handle the heterogeneity of variables and temporal dynamics across domains. The key contributions include a domain-specific tokenizer, a dual-masking strategy, and a novel loss function (NCC). Experimental results are presented across multiple tasks, including classification, regression, and forecasting, and the authors provide visualizations to highlight the interpretability of the learned embeddings.

This paper is working on a very challenging task – cross-domain time series analysis. The novelty of this work is good. The manuscript is generally well-written, and the proposed methodology is easy to follow. After the rebuttal, the evaluation part is also enhanced. However, the major concern after the rebuttal is still the experiment part. For example, reviewers pointed out that the experiments lack the comparison with other key methods such as UniTS. Experimental results are presented without sufficient explanations or analysis, and important experimental details are missing. In addition, multiple reviewers have concerns about the model design, e.g., the shared patch size may not be able to capture domain-specific temporal dynamics. For these limitations, I am inclined to recommend rejecting this paper.