Towards Generalisable Time Series Understanding Across Domains

## Abstract

Abstract In natural language processing and computer vision, self-supervised pre-training on large datasets unlocks foundational model capabilities across domains and tasks.
However, this potential has not yet been realised in time series analysis, where existing methods disregard the heterogeneous nature of time series characteristics.
Time series are prevalent in many domains, including medicine, engineering, natural sciences, and finance, but their characteristics vary significantly in terms of variate count, inter-variate relationships, temporal dynamics, and sampling frequency.
This inherent heterogeneity across domains prevents effective pre-training on large time series corpora.
To address this issue, we introduce OTiS , an o pen model for general ti me s eries analysis, that has been specifically designed to handle multi-domain heterogeneity.
We propose a novel pre-training paradigm including a tokeniser with learnable domain-specific signatures, a dual masking strategy to capture
temporal causality,
and a normalised cross-correlation loss to model
long-range dependencies.
Our model is pre-trained on a large corpus of 640 , 187 640 187 640,187 640 , 187 samples and 11 11 11 11 billion time points spanning 8 8 8 8 distinct domains, enabling it to analyse time series from any (unseen) domain.
In comprehensive experiments across 15 15 15 15 diverse applications - including classification, regression, and forecasting - OTiS showcases its ability to accurately capture domain-specific data characteristics and demonstrates its competitiveness against state-of-the-art baselines.
Our code and pre-trained weights are publicly available at https://github.com/oetu/otis .

## 1 Introduction

In natural language processing (NLP) or computer vision (CV), generalisable language features, e.g. semantics and grammar (Radford et al., ; Touvron et al., ; Chowdhery et al., ), or visual features, e.g. edges and shapes (Geirhos et al., ; Dosovitskiy et al., ; Oquab et al., ), are learned from large-scale data.
Self-supervised pre-training paradigms are designed to account for the specific properties of language (Radford et al., ; Touvron et al., ; Chowdhery et al., ) or imaging (Zhou et al., ; Cherti et al., ; Oquab et al., ), unlocking foundational model capabilities that apply to a wide range of domains and downstream tasks.
This potential, however, remains largely unrealised in time series due to the lack of self-supervised pre-training paradigms that account for the heterogeneity of time series across domains.

Time series are widespread in everyday applications and play an important role in various domains, including medicine (Pirkis et al., ), engineering (Gasparin et al., ), natural sciences (Ravuri et al., ), and finance (Sezer et al., ).
They differ substantially with respect to the number of variates, inter-variate relationships, temporal dynamics, and sampling frequency (Fawaz et al., ; Ismail Fawaz et al., ; Ye & Dai, ; Wickstrøm et al., ).
For instance, standard $10$-$20$ system electroencephalography (EEG) recordings come with up to $256$ variates (Jurcak et al., ), while most audio recordings have only $1$ (mono) or $2$ (stereo) variates.
Weather data shows high periodicity, whereas financial data is exposed to long-term trends.
Both domains encompass low-frequency data recorded on an hourly ($278\,$mHz), daily ($12\,\mu$Hz), or even monthly ($386\,$nHz) basis, while audio data is sampled at high frequencies of $44.1\,$kHz or more.
Overall, this heterogeneity across domains renders the extraction of generalisable time series features difficult (Fawaz et al., ; Gupta et al., ; Iwana & Uchida, ; Ye & Dai, ).

While most existing self-supervised pre-training methods for time series are limited to single domains (Wu et al., ; ; Nie et al., ; Dong et al., ; Jiang et al., ), recent works propose simple techniques to incorporate time series from multiple domains (Yang et al., ; Das et al., ; Woo et al., ; Liu et al., ).
These works for instance crop all time series into segments of unified size (Jiang et al., ), resample them to a uniform frequency (Yang et al., ), or analyse each variate of a multi-variate time series independently (Liu et al., ).
While these naive techniques address differences in sampling frequency and variate count, they degrade the original time series and neglect the critical inter-variate relationships and temporal dynamics required for effective real-world analysis.
Consequently, there is a clear need for pre-training strategies that adequately handle heterogeneity in time series to unlock foundational model capabilities.

In this work, we propose a novel multi-domain pre-training paradigm that addresses the full spectrum of time series heterogeneity across domains.
Our approach facilitates the comprehensive extraction of generalisable features from diverse time series.
Pre-trained on a large corpus of publicly available data, our open model for general time series analysis (OTiS) can be fine-tuned on limited data of any (unseen) domain to perform a variety of downstream tasks, as showcased in Figure 1.

Figure: Figure 1: Overview of OTiS. Pre-trained on a large corpus of time series from diverse domains, OTiS enables general time series analysis. Its domain-specific tokeniser addresses time series heterogeneity across domains - including different numbers of variates, inter-variate relationships, temporal dynamics, and sampling frequencies - by learning unique domain signatures. After pre-training, the model can be fine-tuned on limited data from any domain, including previously unseen ones, to perform various tasks such as classification, regression, and forecasting.
Refer to caption: x1.png

Our key contributions can be summarised as follows:

- 1.
We present OTiS, an open model for general time series analysis, with our entire pipeline and pre-trained weights publicly available at [https://github.com/oetu/otis](https://github.com/oetu/otis).
- 2.
We propose a novel pre-training paradigm based on masked data modelling to address heterogeneity in multi-domain time series.
Our approach includes a novel tokeniser with learnable signatures to capture domain-specific data characteristics, a dual masking strategy to learn temporal causality, and a normalised cross-correlation loss to model long-range dependencies.
- 3.
We pre-train OTiS on a large corpus of publicly available time series from $8$ domains, spanning medicine, engineering, natural sciences, and finance.
With $640,187$ samples and $11$ billion time points, this corpus represents diverse time series characteristics, enabling generalisable feature extraction.
- 4.
We evaluate OTiS across $15$ downstream applications, including classification, regression, and forecasting.
Our comprehensive analysis demonstrates that OTiS accurately captures domain-specific data characteristics and is competitive with both specialised and general state-of-the-art (SOTA) models, achieving new SOTA performance in $10$ tasks.
Notably, none of the baselines is capable of performing all the tasks covered by OTiS.

## 2 Related Works

### 2.1 Self-Supervised Learning for Time Series

Time series vary significantly across domains, with differences in the number of variates, inter-variate relationships, temporal dynamics, and sampling frequencies.
Due to this inherent heterogeneity, most existing works focus on pre-training models within a single domain (Oreshkin et al., ; Tang et al., ; Wu et al., ; Zhou et al., ; Wu et al., ; Woo et al., ; Yue et al., ; Zhang et al., ; Li et al., ; Nie et al., ; Zeng et al., ; Dong et al., ).
To develop more generalisable time series models, recent methods have explored multi-domain pre-training by addressing certain aspects of the heterogeneity, such as differences in variate count and sampling frequency.
For instance, Liu et al. () treat each variate in multi-variate time series independently to standardise generative tasks like forecasting, while Goswami et al. () extend uni-variate analysis to discriminative tasks like classification.
Similarly, Jiang et al. () and Yang et al. () standardise time series by cropping them into segments of predefined size and resampling them to a uniform frequency, respectively, to enable general classification capabilities in medical domains.

While partially addressing time series heterogeneity, these methods limit model capabilities for general time series analysis.
Standardisation techniques like cropping or resampling may distort inter-variate relationships, temporal dynamics, and long-range dependencies.
Additionally, many of these approaches are tailored to specific applications, such as generative tasks (Das et al., ; Liu et al., ; Woo et al., ), or focused on particular domains like medicine (Jiang et al., ; Yang et al., ).
Moreover, recent foundational models (Das et al., ; Goswami et al., ; Liu et al., ) focus on uni-variate analysis, ignoring crucial inter-variate relationships essential for real-world applications, such as disease prediction (Schoffelen & Gross, ; Wu et al., ).
Our study aims to overcome these limitations by fully addressing heterogeneity of multi-domain time series, establishing a foundation for general time series analysis across domains and tasks.

### 2.2 Time Series Tokenisation

Transformers (Vaswani et al., ) have emerged as the preferred architecture for foundational models in NLP and CV due to their scalability (Kaplan et al., ; Gordon et al., ; Alabdulmohsin et al., ), enabling the training of models in the magnitude of $100$ billion parameters (Chowdhery et al., ; Touvron et al., ; Oquab et al., ; Ravi et al., ).
To utilise a Transformer for time series analysis, a tokeniser is required to map the time series into a compact latent space.
Current methods (Jin et al., ; Nie et al., ; Zhou et al., ; Das et al., ; Goswami et al., ; Jiang et al., ; Liu et al., ; Woo et al., ; Yang et al., ) follow established techniques from NLP and CV, dividing time series into patches of pre-defined size.
These patches are then flattened into a $1$D sequence, with positional embeddings used to retain positional information.
While uni-variate models (Nie et al., ; Das et al., ; Goswami et al., ; Liu et al., ) consider only temporal positions, multi-variate approaches (Woo et al., ; Yang et al., ; Jiang et al., ) account for both temporal and variate positions.
However, none of these methods address the unique characteristics of variates, mistakenly assuming that the relationships between variates are identical across domains.
Our work seeks to adapt the tokenisation process to preserve the domain-specific relationships between variates.

## 3 Methods

Figure: Figure 2: Architecture of OTiS. During pre-training, batches of time series from diverse domains are processed using a domain-specific tokeniser. This tokeniser splits a time series into fixed-size patches, which are then embedded using a patch projector shared across all variates and domains. A temporal embedding and a domain-specific variate embedding are added to each patch embedding. A dual masking strategy is employed to mask the resulting input tokens. The reconstruction of the multi-domain input tokens is guided using an auxiliary normalised cross-correlation (NCC) loss.
Refer to caption: x2.png

In this work, we present a novel multi-domain pre-training paradigm that enables generalisable feature extraction from large, heterogeneous time series corpora.
We introduce a domain-specific tokeniser with learnable signatures to address heterogeneity in multi-domain time series, as described in Section 3.1.
We tailor masked data modelling (MDM) for multi-domain time series to pre-train our open model for general time series analysis (OTiS) on a large, heterogeneous corpus, as detailed in Section 3.2.
In particular, we propose normalised cross-correlation as a loss term to capture global temporal dynamics in time series, as explained in Section 3.3.
Moreover, we introduce a dual masking strategy to capture bidirectional relationships and temporal causality, essential for general time series analysis, as described in Section 3.4.
After pre-training, we fine-tune OTiS on limited data to perform a variety of downstream tasks in any - including previously unseen - domain, as outlined in Section 3.5.
A graphical visualisation of our method is provided in Figure 2.

### 3.1 Domain-Specific Tokeniser

#### Overview.

Assume a time series sample $\displaystyle{\bm{X}}\in\mathbb{R}^{V_{S}\times T}$ from domain $S$, where $V_{S}$ denotes the number of variates specific to $S$ and $T$ denotes the number of time points.
We randomly crop or zero-pad $\displaystyle{\bm{X}}$ to a fixed context length of $\overline{T}$ time points.
We then split it into $T^{\prime}$ temporal patches of size $P$ along the time dimension, resulting in $V_{S}\cdot T^{\prime}$ patches $\displaystyle{\bm{x}}_{v,t}\in\mathbb{R}^{1\times P}$, where $v\in\{1,\dots,V_{S}\}$ and $t\in\{1,\dots,T^{\prime}\}$.

Next, we embed these patches using a shared patch projector across all variates and domains, resulting in patch embeddings $e^{\mathcal{P}}(\displaystyle{\bm{x}}_{v,t})=\displaystyle{\bm{e}}^{\mathcal{P
}}_{v,t}\in\mathbb{R}^{1\times D}$, where $D$ denotes the model dimension.
The patch projector consists of a $1$D convolutional layer followed by layer normalisation and GELU activation.

The permutation-equivariant nature of Transformers (Vaswani et al., ) requires the use of positional embeddings to accurately capture the inherent relationships in the input data.
Initially introduced for $1$D textual token sequences (Vaswani et al., ), positional embeddings simply introduce an ordering into the input sequence.
Modern implementations further extend their capabilities to encode more complex geometric information, such as $2$D spatial (Dosovitskiy et al., ) or graph (Kreuzer et al., ) structures.
For the analysis of any-variate time series, we distinguish between the temporal and variate structure.
The temporal structure is equivalent to a sequential $1$D structure, such that we use standard $1$D sinusoidal embeddings $e^{\mathcal{T}}(\displaystyle{\bm{x}}_{v,t})=\displaystyle{\bm{e}}^{\mathcal{T
}}_{t}\in\mathbb{R}^{1\times D}$.

The variate structure exhibits great heterogeneity across domains.
In domains with uni-variate and two-variate data, such as mono and stereo audio, the structure is either trivial or only requires a basic distinction between variates.
In other domains, however, the variate structure may represent more complex relationships, such as $3$D manifolds for electroencephalography (EEG) or electrocardiography (ECG) data, or be of non-spatial nature, such as for financial data.
Hence, we introduce learnable *domain-specific* variate embeddings to adequately address the heterogeneity across domains.
These embeddings, denoted as $e^{\mathcal{V}}_{\mathcal{S}}(\displaystyle{\bm{x}}_{v,t})=\displaystyle{\bm{e
}}^{\mathcal{V}}_{\mathcal{S},v}\in\mathbb{R}^{1\times D}$ for each variate $v$ in domain $S$, are designed to model the unique properties of a domain.
They capture the inter-variate relationships and temporal dynamics specific to domain $S$, forming what can be considered as the *signature* of the very domain.

Finally, the patch, temporal, and domain-specific variate embeddings are summed to form the input token $\displaystyle{\bm{e}}_{v,t}=\displaystyle{\bm{e}}^{\mathcal{P}}_{v,t}+
\displaystyle{\bm{e}}^{\mathcal{T}}_{t}+\displaystyle{\bm{e}}^{\mathcal{V}}_{
\mathcal{S},v}\in\mathbb{R}^{1\times D}$.
These input tokens collectively constitute the final input sequence $\displaystyle{\bm{E}}\in\mathbb{R}^{(V_{S}\cdot T^{\prime})\times D}$.
To support batches of any-variate time series from multiple domains, we pad the variate dimension to the maximum number of variates in a batch $\overline{V}=\max_{S}V_{S}$.
For samples where $V_{S}<\overline{V}$ or $T<\overline{T}$, attention masking is used to ensure that padded variate or temporal tokens are ignored.
The domain-specific tokeniser is trained end-to-end with the Transformer layers.

#### Definition of (Sub-)Domains.

The domain-specific tokeniser is designed to integrate different datasets within a domain.
Consider two EEG datasets, TDBrain (Van Dijk et al., ) and SEED (Zheng & Lu, ), which share $19$ identical variates but have different sampling frequencies of $500\,$Hz and $200\,$Hz, respectively.
In this case, a single EEG-specific tokeniser ($V_{\text{EEG}}=19$) is sufficient to accommodate both sampling frequencies, i.e. $\displaystyle{\bm{E}}^{\mathcal{V}}_{\text{EEG-TDBrain}}=\displaystyle{\bm{E}}
^{\mathcal{V}}_{\text{EEG-SEED}}=[\displaystyle{\bm{e}}^{\mathcal{V}}_{\text{
EEG},1},\dots,\displaystyle{\bm{e}}^{\mathcal{V}}_{\text{EEG},19}]^{\top}\in
\mathbb{R}^{19\times D}$, as demonstrated in our experiments in Section 4.
Note that while these positional embeddings are agnostic to variate ordering, we simplify processing by aligning the variate order across datasets within the same domain.
Consider another EEG dataset, LEMON (Babayan et al., ), which includes $62$ electrodes.
Of these, $15$ overlap with the electrodes in TDBrain (Van Dijk et al., ) and SEED (Zheng & Lu, ), while the remaining $47$ are unique to LEMON (Babayan et al., ).
In this scenario, the EEG-specific tokeniser can be extended by the $47$ new variates ($V_{\text{EEG}}=66$), such that $\displaystyle{\bm{E}}^{\mathcal{V}}_{\text{EEG-LEMON}}=[\displaystyle{\bm{e}}^
{\mathcal{V}}_{\text{EEG},1},\dots,\displaystyle{\bm{e}}^{\mathcal{V}}_{\text{
EEG},15},\displaystyle{\bm{e}}^{\mathcal{V}}_{\text{EEG},20},\dots,
\displaystyle{\bm{e}}^{\mathcal{V}}_{\text{EEG},66}]^{\top}\in\mathbb{R}^{62
\times D}$.
In this way, different datasets can be combined to approximate the underlying data distribution of a domain $S$, e.g. EEG, enabling the creation of large and diverse time series corpora.

#### Multi-Variate or Uni-Variate Analysis?

Consider the Electricity dataset (UCI, ), which contains electricity consumption data for $321$ households recorded from $2012$ to $2014$.
These $321$ observations are sampled from an underlying population and are assumed to be independent and identically distributed (*i.i.d.*).
In this scenario, we perform a uni-variate analysis ($V_{\text{Electricity}}=1$) of the data, initialising a single Electricity-specific variate embedding that models the hourly consumption of a household.
In contrast, the Weather dataset (Wetterstation, ) contains $21$ climatological indicators, such as air temperature, precipitation, and wind speed, which are not *i.i.d.* because they directly interact and correlate with one another.
Therefore, a multi-variate analysis ($V_{\text{Weather}}=21$) is conducted to account for the dependencies and interactions between the observations.

### 3.2 Pre-Training on Multi-Domain Time Series

We pre-train our model using masked data modelling (MDM) (He et al., ) to learn generalisable time series features across domains.
We mask a subset of input tokens and only encode the visible (i.e. non-masked) tokens using an encoder $f(\cdot)$.
Afterwards, we complement the encoded tokens with learnable mask tokens and feed them to a decoder $g(\cdot)$, reconstructing the original input tokens.

More precisely, we draw a binary mask $\displaystyle{\bm{m}}\in\{0,1\}^{V_{S}\cdot T^{\prime}}$, following the dual masking strategy proposed in Section 3.4, and apply it to the input sequence $\displaystyle{\bm{E}}\in\mathbb{R}^{(V_{S}\cdot T^{\prime})\times D}$.
Thus, we obtain a visible view $\displaystyle{\bm{E}}[\displaystyle{\bm{m}}]\in\mathbb{R}^{N_{1}\times D}$, where $N_{1}=\sum_{v=1}^{V_{S}}\sum_{t=1}^{T^{\prime}}m_{v,t}$ and $N_{0}=(V_{S}\cdot T^{\prime})-N_{1}$ denote the number of visible and masked tokens, respectively.
The visible view $\displaystyle{\bm{E}}[\displaystyle{\bm{m}}]$ is then fed to the encoder $f(\cdot)$ to compute the token features $\displaystyle{\bm{H}}\in\mathbb{R}^{N_{1}\times D}$:

$$ $\displaystyle{\bm{H}}=f(\displaystyle{\bm{E}}[\displaystyle{\bm{m}}]).$ (1) $$

To reconstruct the original input, these token features are fed to the decoder $g(\cdot)$ together with a special, learnable mask token $\displaystyle{\bm{e}}^{\mathcal{M}}\in\mathbb{R}^{1\times D}$, that is inserted at the masked positions where $m_{v,t}=0$:

$$ $\displaystyle{\bm{h}}^{\prime}_{v,t}=\begin{cases}\displaystyle{\bm{h}}_{v,t}& \text{if }m_{v,t}=1\\ \displaystyle{\bm{e}}^{\mathcal{M}}&\text{if }m_{v,t}=0\end{cases},$ (2) $$

such that $\displaystyle{\bm{H}}^{\prime}\in\mathbb{R}^{(V_{S}\cdot T^{\prime})\times D}$.
The decoder $g(\cdot)$ then predicts the reconstructed input $\displaystyle\widehat{{\bm{X}}}\in\mathbb{R}^{V_{S}\times(T^{\prime}\cdot P)}$:

$$ $\displaystyle\widehat{{\bm{X}}}=g(\displaystyle{\bm{H}}^{\prime}),$ (3) $$

where $(T^{\prime}\cdot P)=\overline{T}$, i.e. the context length specified in time points.
Eventually, the domain-specific tokeniser described in Section 3.1, the encoder $f(\cdot)$, and the decoder $g(\cdot)$ are optimised end-to-end using the mean squared error (MSE) loss on all reconstructed input tokens:

$$ $\mathcal{L}_{\text{MSE}}=\frac{1}{V_{S}\cdot T^{\prime}}\sum_{v=1}^{V_{S}}\sum _{t=1}^{T^{\prime}}\|\displaystyle{\bm{x}}_{v,t}-\displaystyle\widehat{{\bm{x} }}_{v,t}\|_{2}^{2}.$ (4) $$

### 3.3 Normalised Cross-Correlation Loss

MDM focuses on reconstructing masked parts of the data, emphasising *local* patterns through the MSE loss (4).
However, time series often exhibit long-range dependencies, where past values influence future outcomes over extended periods.
To accurately capture these *global* patterns, we introduce normalised cross-correlation (NCC) as a loss term in MDM for time series:

$$ $\mathcal{L}_{\text{NCC}}=\frac{1}{V_{S}\cdot\overline{T}}\sum^{V_{S}}_{v=1} \sum^{\overline{T}}_{t=1}\hskip 2.84526pt\frac{1}{\sigma_{\displaystyle{\bm{x} }_{v}}\sigma_{\displaystyle\widehat{{\bm{x}}}_{v}}}(x_{v,t}-\mu_{\displaystyle {\bm{x}}_{v}})(\widehat{x}_{v,t}-\mu_{\displaystyle\widehat{{\bm{x}}}_{v}})\in \left[-1,1\right],$ (5) $$

where $\mu$ and $\sigma$ denote the mean and standard deviation, respectively. Hence, to capture both local and global temporal dynamics, the total loss used to optimise OTiS is defined as

$$ $\mathcal{L}=\mathcal{L}_{\text{MSE}}+\lambda\cdot\left(1-\mathcal{L}_{\text{ NCC}}\right),$ (6) $$

where $\lambda$ is empirically set to $0.1$ during pre-training.

### 3.4 Dual Masking Strategy

We design the masking strategy to enhance foundational model capabilites in time series analysis.
Specifically, we randomly select between two masking schemes during pre-training, namely random masking and post-fix masking.
In $75\,$% of cases, we apply random masking, where each $m_{v,t}$ is independently sampled from a Bernoulli distribution with probability $p=1-\rho$, with $\rho$ denoting the masking ratio (i.e. $m_{v,t}\sim\text{Bernoulli}(1-\rho)$).
This encourages the model to learn complex inter-variate relationships across the entire time series.
In the remaining $25\,\%$ of cases, we employ post-fix masking, which masks the second half of the temporal dimension, leaving only the first half visible (i.e. $m_{v,t}=\mathbbm{1}_{[t\leq T^{\prime}/2]}$).
The prediction of future values solely based on past observations simulates real-world forecasting conditions, helping the model to capture temporal causality.
Overall, this dual masking strategy enables OTiS to learn both bidirectional relationships and temporal causality, which are essential for general time series analysis.

### 3.5 Fine-Tuning & Inference on (Unseen) Target Domains

#### Inclusion of Unseen Domains.

For a new domain $S$, a randomly initialised variate embedding $\displaystyle{\bm{E}}^{\mathcal{V}}_{\mathcal{S}}\in\mathbb{R}^{V_{S}\times D}$ is introduced.
The domain-specific tokeniser is then fine-tuned alongside the encoder $f(\cdot)$, and, if required, the decoder $g(\cdot)$, for the specific downstream task, as described in the following.

#### Classification & Regression.

We use the encoder $f(\cdot)$ and the unmasked input sequence $\displaystyle{\bm{E}}$ to compute all token features $\displaystyle{\bm{H}}=f(\displaystyle{\bm{E}})\in\mathbb{R}^{(V_{S}\cdot T^{
\prime})\times D}$. We average-pool these features into a global token $\displaystyle{\bm{h}}^{*}\in\mathbb{R}^{1\times D}$, which we feed through a linear layer to obtain the final model prediction. We optimise a cross-entropy and MSE loss for the classification and regression tasks, respectively.

#### Forecasting.

We apply post-fix masking to generate a binary mask $\displaystyle{\bm{m}}\in\{0,1\}^{V_{S}\cdot T^{\prime}}$ for the forecasting task.
The encoder $f(\cdot)$ is used to compute the visible token features $\displaystyle{\bm{H}}\in\mathbb{R}^{N_{1}\times D}$.
We then concatenate the sequence with learnable mask tokens to form $\displaystyle{\bm{H}}^{\prime}\in\mathbb{R}^{(V_{S}\cdot T^{\prime})\times D}$, which is passed through the decoder $g(\cdot)$ to produce the final output.
We optimise the MSE loss together with the NCC loss term over all reconstructed input tokens.

## 4 Experiments & Results

### 4.1 Model Variants and Implementation Details

We introduce OTiS in three different configurations, Base, Large, and Huge, with their specific architectures described in Appendix C.1, to explore scaling laws with respect to the model size.
We set the patch size and stride to $P=24$, respectively, to split the time series into $T^{\prime}=\frac{\overline{T}}{P}$ non-overlapping patches along the time dimension.
For pre-training, the context length specified in time points is set to $\overline{T}=1008$, resulting in $T^{\prime}=42$ sinusoidal temporal embeddings.
If longer context lengths are required during fine-tuning, these embeddings are linearly interpolated (i.e. $T^{\prime}_{\text{ft}}\geq 42$) to offer greater flexibility for downstream applications.
We tune the hyperparameters for pre-training and fine-tuning as described in Appendix C.
An overview of the computational costs is provided in Appendix D.

### 4.2 Large and Diverse Pre-Training Corpus

We aim to develop a general time series model that fully handles the heterogeneity in real-world data.
Specifically, our model is designed to handle time series with different variate counts $V_{S}$, inter-variate relationships, temporal dynamics, and sampling frequency, ensuring flexibility for downstream tasks.
To this end, we pre-train our model on a large and diverse corpus of publicly available data spanning $8$ domains, with a total of $640,187$ samples and $11$ billion time points, as summarised in Table 1.
A detailed description of the datasets included in our pre-training corpus can be found in Appendix A.
The time series corpus is split into $612,394$ training and $27,793$ validation samples for pre-training.

**Table 1: Overview of our large and diverse pre-training corpus. The corpus is built with unlabelled data from eight domains, encompassing medicine, engineering, natural sciences, and finance.**
| Domain | Name | Samples | Variates | Time points | Frequency | Disk size |
| --- | --- | --- | --- | --- | --- | --- |
| $S$ |  |  | $V_{S}$ |  |  |  |
| ECG | MIMIC-IV-ECG | $400,000$ | $12$ | $5,000$ | $500\,$Hz | $90\,$GB |
| Temperature | DWD | $203,340$ | $1$ | $720$ | (hourly) $278\,$mHz | $614\,$MB |
| Audio (stereo) | AudioSet-$20$K | $16,123$ | $2$ | $441,000$ | $44.1\,$kHz | $53\,$GB |
| Audio (mono) | AudioSet-$20$K | $3,491$ | $1$ | $441,000$ | $44.1\,$kHz | $6\,$GB |
| Electromechanics | FD-A | $13,640$ | $1$ | $5,120$ | $64\,$kHz | $161\,$MB |
| EEG | TDBrain | $2,692$ | $19$ | $60,000$ | $500\,$Hz | $12\,$GB |
| EEG | SEED | $675$ | $19$ | $37,000$ | $200\,$Hz | $2\,$GB |
| Banking | NN$5$ | $111$ | $1$ | $971$ | (daily) $12\,\mu$Hz | $370\,$KB |
| Economics | FRED-MD | $107$ | $1$ | $728$ | (monthly) $386\,$nHz | $330\,$KB |
| Economics | Exchange | $8$ | $1$ | $7,588$ | (daily) $12\,\mu$Hz | $240\,$KB |
|  |  | 640,187 |  | 11,052,756,981 |  | 164 GB |

### 4.3 Benchmarking Across Domains and Tasks

To assess the utility of OTiS in real-world settings, we conduct experiments on three key use cases in time series analysis: classification, regression, and forecasting.
For classification, we perform binary epilepsy detection using
EEG (Epilepsy ), multi-class fault detection in rolling bearings from
vibration signals (FD-B ), multi-class hand-gesture classification with
accelerometer signals (Gesture ), and multi-class muscular disease classification using
electromyographie (EMG ).
In regression, we predict five imaging-derived cardiac phenotypes from $12$-lead ECG (LVEDV, LVESV, LVSV, LVEF, LVM ).
For forecasting, we predict electricity transformer temperature (ETT ), weather (Weather ), and electricity consumption (Electricity ).
Further dataset details are provided in Appendix B.
We adhere to the established data splitting and evaluation procedures for the classification (Zhang et al., ), regression (Turgut et al., ), and forecasting (Zhou et al., ) tasks.
All experiments are reported as mean and standard deviation across five seeds set during fine-tuning.

**Table 2: Classification and regression performance on a total of $9$ benchmark tasks. OTiS is competitive with specialised baselines, setting new state-of-the-art on $6$ tasks and even outperforming the multimodal CM-AE and MMCL. This demonstrates the capability of OTiS to extract high-level semantics. Best score in bold, second best underlined. ^∙ indicates tasks in previously unseen domains.**
| Model | Epilepsy | FD-B | Gesture^∙ | EMG^∙ |
| --- | --- | --- | --- | --- |
| SimCLR | 90.71 | 49.17 | 48.04 | 61.46 |
| CoST | 88.40 | 47.06 | 68.33 | 53.65 |
| TS2Vec | 93.95 | 47.90 | 69.17 | 78.54 |
| TF-C | 94.95 | 69.38 | 76.42 | 81.71 |
| Ti-MAE | 89.71 | 60.88 | 71.88 | 69.99 |
| SimMTM | 95.49 | 69.40 | 80.00 | 97.56 |
| OTiS-Base | 94.25 | 99.24 | 63.61 | 97.56 |
| OTiS-Large | 94.03 | 98.62 | 62.50 | 98.37 |
| OTiS-Huge | 91.48 | 98.32 | 63.61 | 98.37 |

OTiS demonstrates strong classification capabilities, setting two new benchmarks, as shown in Table 2(a).
Additionally, OTiS excels in predicting imaging-derived cardiac phenotypes, even surpassing multimodal baselines that incorporate imaging data during pre-training, as summarised in Table 2(b).
Our model outperforms the baselines in $4$ out of $5$ regression tasks.
As shown in Table 3, OTiS also exhibits strong forecasting capabilities, outperforming current models in $4$ out of $6$ benchmarks.
Notably, all of the forecasting tasks are performed in previously unseen domains.
A visualisation of the forecast predictions can be found in Appendix F.
Overall, OTiS outperforms both specialised and general state-of-the-art baselines in $10$ out of $15$ diverse applications across $8$ domains, demonstrating its strong utility and generalisability across (unseen) domains and tasks.

**Table 3: Forecasting performance on $6$ benchmark tasks. OTiS is competitive with specialised and general baselines, setting new state-of-the-art on $4$ tasks and showcasing its ability to capture local time series features. A forecasting horizon of $96$ time points is predicted from the past $336$ (*$512$, ${}^{+}904$) time points. Mean squared error (MSE $\downarrow$) is reported. Best score in bold, second best underlined. ^∙ indicates tasks in previously unseen domains.**
| Model | ETTh1^∙ | ETTh2^∙ | ETTm1^∙ | ETTm2^∙ | Weather^∙ | Electricity^∙ |
| --- | --- | --- | --- | --- | --- | --- |
| N-BEATS | 0.399 | 0.327 | 0.318 | 0.197 | 0.152 | 0.131 |
| Autoformer | 0.435 | 0.332 | 0.510 | 0.205 | 0.249 | 0.196 |
| TimesNet | 0.384 | 0.340 | 0.338 | 0.187 | 0.172 | 0.168 |
| DLinear | 0.375 | 0.289 | 0.299 | 0.167 | 0.176 | 0.140 |
| PatchTST | 0.370 | 0.274 | 0.293 | 0.166 | 0.149 | 0.129 |
| Time-LLM | 0.408 | 0.286 | 0.384 | 0.181 | ^† | ^† |
| GPT4TS | 0.376 | 0.285 | 0.292 | 0.173 | 0.162 | 0.139 |
| MOMENT* | 0.387 | 0.288 | 0.293 | 0.170 | 0.154 | 0.136 |
| MOIRAI^+ | 0.375 | 0.277 | 0.335 | 0.189 | 0.167 | 0.152 |
| OTiS-Base | 0.424 | 0.212 | 0.337 | 0.161 | 0.139 | 0.128 |
| OTiS-Large | 0.446 | 0.205 | 0.362 | 0.173 | 0.142 | 0.127 |
| OTiS-Huge | 0.461 | 0.215 | 0.384 | 0.181 | 0.149 | 0.132 |
| ^† Experiments could not be conducted on a single NVIDIA RTX A$6000$-$48$GB GPU. |  |  |  |  |  |  |

### 4.4 Domain Signature Analysis

A key component of OTiS is its use of domain-specific variate embeddings.
While these embeddings are randomly initialised, we expect them to capture unique domain characteristics during training, eventually serving as the signature of their respective domain.
To validate this hypothesis, we analyse the domain-specific variate embeddings after pre-training using principal component analysis (PCA).

First, we find that OTiS unifies time series from diverse domains into a meaningful latent space, where embeddings of domains with shared high-level semantics cluster together, as depicted in Appendix E.1.
For example, embeddings of mono and stereo audio group closely, as do those of banking and economics.
Moreover, EEG-specific embeddings are clearly separated and ECG-specific embeddings form a tight cluster.

Second, OTiS preserves low-level semantics specific to each domain, such as the relationships between variates.
To explore this, we focus on the learned variate embeddings of EEG, the most complex domain in our corpus.
EEG variates correspond to actual electrodes, each associated with a $3$D position in space or a $2$D position on the scalp, which is ideal for studying inter-variate relationships.
Our analysis covers both (i) variate embeddings for $10$-$20$ system EEG recordings with $19$ electrodes learned during multi-domain pre-training, and (ii) variate embeddings for previously unseen EEG recordings with $32$ electrodes learned during fine-tuning of the pre-trained OTiS.
We find that the first three principal components explain (i) $74.7\,\%$
and (ii) $87.9\,\%$
of the variance, indicating that the $3$D PCA projections of the variate embeddings correspond closely to the actual electrode positions.
A visualisation of the learned $3$D electrode layout is provided in Appendix E.
Additionally, we linearly align these positions with the true EEG electrode layout on the $2$D scalp manifold, as depicted in
Figure 3.
The alignment yields $R^{2}$ values of (i) $0.81$ and (ii) $0.95$, confirming a strong correspondence between the learned variate embeddings and the actual electrode layout.

Figure: Figure 3: $2$D PCA projections of the learned EEG-specific variate embeddings, overlaid on the actual EEG electrode layout. (Left) Embeddings for $10$-$20$ system EEG recordings with $19$ electrodes learned during pre-training. (Right) Embeddings for previously unseen EEG recordings with $32$ electrodes learned during fine-tuning. The embeddings accurately reflect the spatial electrode layout, as confirmed by high correlations (R^2) between the PCA projections $\bullet$ and the true layout $\circ$.
Refer to caption: x3.png

### 4.5 Scaling Study

Figure: Figure 4: Performance of OTiS with different numbers of pre-training samples. Shaded regions indicate the standard deviation across $5$ seeds. Increasing dataset size generally improves downstream performance. Scaling model size requires even larger pre-training corpora to be effective.
Refer to caption: x5.png

We analyse the scaling behaviour of OTiS with respect to model and dataset size.
To this end, we subsample the pre-training data to $10\,\%$ and $1\,\%$ of its original size, ensuring that each subset is fully contained within the corresponding superset.
We evaluate the downstream performance of all OTiS variants across classification, regression, and forecasting tasks, as depicted in Figure 4.

The experiments demonstrate that downstream performance generally scales with dataset size, achieving the best results with the full pre-training dataset.
This trend, however, does not directly apply to model size, which is in line with the scaling behaviour observed in current time series foundational models (Woo et al., ; Goswami et al., ).
Given that performance generally improves across all models with increasing data size, we hypothesise that scaling the model size could prove beneficial with even larger pre-training corpora.

### 4.6 Ablation Study

Figure: Figure 5: Ablation study on key components of OTiS. Downstream performance is analysed across $5$ seeds. A leave-one-out approach is used to evaluate the influence of each component. The default setting, which includes all components, demonstrates superior model capabilities across tasks.
Refer to caption: x8.png

We perform an ablation study to analyse the impact of OTiS’ key components: the domain-specific tokeniser, dual masking strategy, and normalised cross-correlation (NCC) loss.
As shown in Figure 5, the best and most robust performance is achieved when all components are used during pre-training.

Replacing the domain-specific variate embeddings with domain-agnostic embeddings (i.e. learnable embeddings shared across all domains) consistently led to inferior performance across all tasks, demonstrating the importance of capturing domain-specific data characteristics during tokenisation.
Switching from dual masking to random masking resulted in performance degradation, although the impact was less notable for generative tasks than for discriminative tasks.
We hypothesise that the NCC loss already captures temporal causality, which is particularly crucial for generative tasks like forecasting.
Overall, removing the NCC loss caused performance declines across all downstream tasks, emphasising the role of long-range dependencies for general time series understanding.

## 5 Discussion & Conclusion

In this study, we explore the problem of effective pre-training on heterogeneous time series corpora.
Time series vary substantially across domains, e.g. with respect to inter-variate relationships and temporal dynamics, rendering generalisable feature extraction from multi-domain time series difficult.
To address this issue, we present OTiS, an open model for general time series analysis, specifically designed to handle multi-domain heterogeneity.
Our novel multi-domain pre-training paradigm, including a domain-specific tokeniser with learnable signatures, a dual masking strategy, and a normalised cross-correlation (NCC) loss, enables OTiS to extract generalisable time series features.

In extensive experiments, we demonstrate that OTiS generalises well across $15$ diverse downstream applications spanning $8$ distinct domains, achieving competitive performance with both specialised and general state-of-the-art (SOTA) models.
In a qualitative analysis, we further show that OTiS unifies time series from diverse domains in a meaningful latent space, while preserving low-level semantics of a domain including the inter-variate relationships.
Thereby, our work establishes a strong foundation for future advancements in interpretable and general time series analysis.

#### Limitations.

While OTiS outperforms SOTA models across $10$ tasks, our experiments in low-data regimes suggest that larger pre-training corpora could further enhance its performance.
Unlike in NLP and CV, where large datasets are curated from web-crawled data, foundational models in time series, including OTiS, still rely on manually curated datasets.
Future work could explore fully automatic pipelines, e.g. using embedding similarity, to filter and rebalance multi-domain time series from the web.
OTiS could further benefit from processing domain signatures during inference, potentially unlocking zero-shot capabilities, similarly to those seen in foundational models in NLP and CV.

## Appendix A Large Multi-Domain Pre-Training Corpus

In this section, we present an overview of our large and diverse pre-training corpus.
The corpus consists of publicly available data spanning eight domains, with a total of $640,187$ samples and $11$ billion time points.
In the following, we provide a detailed breakdown of the domains and the datasets they encompass.
Note that we apply channel-wise standard normalisation to the datasets unless otherwise specified.

#### ECG.

The MIMIC-IV-ECG dataset (Gow et al., ) contains diagnostic $10$-second, $12$-lead ECG recordings sampled at a frequency of $500$ Hz.
While the entire dataset comprises $800,035$ samples, we include only the first half of the recordings available in the database, preventing the ECG data from predominating in the pre-training corpus.
To remove the baseline drift from the ECG data, we use the asymmetric least square smoothing technique (Zhang et al., ).
Note that we apply standard normalisation separately to the Einthoven, Goldberger, and Wilson leads.

#### Temperature.

The Deutscher Wetterdienst (DWD) dataset (Wetterdienst, ) contains hourly air temperature measurements from $629$ weather stations across Germany.
Since the recording length varies significantly, ranging from $763$ to $1,148,290$ hours per station, we split the data into chunks of $720$ hours (approximately one month).

#### Audio.

The AudioSet dataset (Gemmeke et al., ) contains $10$-second YouTube clips for audio classification, featuring $527$ types of audio events that are weakly annotated for each clip.
The full training set includes a class-wise balanced subset (AudioSet-$20$K, $22,176$ clips) and an unbalanced (AudioSet-$2$M $2,042,985$ clips) set.
For our pre-training corpus, we use the balanced AudioSet-$20$K, which contains $3,491$ mono and $16,123$ stereo recordings, all sampled at $44,100\,$Hz.

#### Electromechanics.

The FD-A dataset (Lessmeier et al., ) collects vibration signals from rolling bearings in a mechanical system for fault detection purposes.
Each sample consists of $5,120$ timestamps, indicating one of three mechanical device states.
Note that the FD-B dataset is similar to FD-A but includes rolling bearings tested under different working conditions, such as varying rotational speeds.

#### EEG.

The TDBrain dataset (Van Dijk et al., ) includes raw resting-state EEG data from $1,274$ psychiatric patients aged $5$ to $89$, collected between $2001$ and $2021$.
The dataset covers a range of conditions, including Major Depressive Disorder ($426$ patients), Attention Deficit Hyperactivity Disorder ($271$ patients), Subjective Memory Complaints ($119$ patients), and Obsessive-Compulsive Disorder ($75$ patients).
The data was recorded at $500\,$Hz using $26$ channel EEG-recordings, based on the $10$-$10$ electrode international system.

The SEED dataset (Zheng & Lu, ) contains EEG data recorded under three emotional states: positive, neutral, and negative.
It comprises EEG data from $15$ subjects, with each subject participating in experiments twice, several days apart.
The data is sampled at $200\,$Hz and recorded using $62$ channel EEG-recordings, based on the $10$-$20$ electrode international system.

For simplicity, we only consider the $19$ channels common to both datasets, i.e. the channels that correspond to the $10$-$20$ electrode international system.

#### Banking.

The NN$5$ competition dataset (Taieb et al., ) consists of daily cash withdrawals observed at $111$ randomly selected automated teller machines across various locations in England.

#### Economics.

The FRED-MD dataset (McCracken & Ng, ) contains $107$ monthly time series showing a set of macro-economic indicators from the Federal Reserve Bank of St Louis.
The data was extracted from the FRED-MD database.

The Exchange dataset (Lai et al., ) records the daily exchange rates of eight different nations, including Australia, Great Britain, Canada, Switzerland, China, Japan, New Zealand, and Singapore, ranging from $1990$ to $2016$.

## Appendix B Benchmark Dataset Details

We provide an overview of the datasets used to benchmark our model on $15$ distinct downstream applications in Table 4. Our benchmarking experiments include classifcation, regression, and forecasting tasks.

**Table 4: Experimental setup with all conducted downstream applications, evaluation metrics, and datasets used for benchmarking.**
| Task | Metric | Dataset |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Domain | Name | Samples | Variates | Time points | Frequency |  |  |
|  |  | $S$ |  |  | $V_{S}$ |  |  |
| Classification | ACC | EEG | Epilepsy | $11,500$ | $1$ | $178$ | $174\,$Hz |
| Electromechanics | FD-B | $13,640$ | $1$ | $5,120$ | $64\,$kHz |  |  |
| Acceleration | Gesture | $560$ | $3$ | $206$ | $100\,$Hz |  |  |
| EMG | EMG | $204$ | $1$ | $1,500$ | $4\,$kHz |  |  |
| Regression | $R^{2}$ | ECG | UK BioBank | $18,926$ | $12$ | $5,000$ | $500\,$Hz |
| Forecasting | MSE | Energy | ETTh1 | $1$ | $7$ | $17,420$ | (hourly) $278\,$mHz |
| ETTh2 | $1$ | $7$ | $17,420$ | (hourly) $278\,$mHz |  |  |  |
| ETTm1 | $1$ | $7$ | $69,680$ | (minutely) $1.1\,$mHz |  |  |  |
| ETTm2 | $1$ | $7$ | $69,680$ | (minutely) $1.1\,$mHz |  |  |  |
| Weather | Weather | $1$ | $21$ | $52,696$ | (minutely) $2.8\,$mHz |  |  |
| Electricity | Electricity | $321$ | $1$ | $26,304$ | (hourly) $278\,$mHz |  |  |

## Appendix C Experiment Details

### C.1 Model Variants

To explore the scaling laws with respect to the model size, we provide OTiS in three variants, as summerised in Table 5.

**Table 5: Details of model variants.**
| Model | Layers | Hidden size D | MLP size | Heads | $d_{kv}$ | Parameters |
| --- | --- | --- | --- | --- | --- | --- |
| OTiS-Base | $12$ | $192$ | $768$ | $3$ | $64$ | $8\,$M |
| OTiS-Large | $18$ | $384$ | $1536$ | $6$ | $64$ | $44\,$M |
| OTiS-Huge | $24$ | $576$ | $2304$ | $8$ | $72$ | $131\,$M |

### C.2 Pre-Training & Fine-Tuning Parameters

We provide the hyperparameters used to pre-train all variants of OTiS in Table 6. The hyperparameters used to fine-tune our models for the classification, regression, and forecasting tasks are provided in Table 7, 8, and 9, respectively.

**Table 6: Hyperparameters used for pre-training. Pre-training is performed on $4$ NVIDIA A$100$-$80$GB GPUs. A cosine learning rate scheduler is applied with a $10\,\%$ warmup. All OTiS configurations use a shallow decoder with $2\,$M parameters, consisting of $4$ layers with a hidden size of $160$, an MLP with size $640$, and $5$ heads.**
| Model | Epochs | Batch size | Base LR | LR decay | NCC $\lambda$ | Mask ratio $\rho$ | Weight decay |
| --- | --- | --- | --- | --- | --- | --- | --- |
| OTiS-Base | $200$ | $5120$ | $3$e-$5$ | cosine | $0.1$ | $0.75$ | $0.10$ |
| OTiS-Large | $200$ | $3328$ | $1$e-$5$ | cosine | $0.1$ | $0.75$ | $0.15$ |
| OTiS-Huge | $200$ | $2880$ | $3$e-$6$ | cosine | $0.1$ | $0.75$ | $0.05$ |

**Table 7: Hyperparameters used for fine-tuning the classification tasks on a single NVIDIA RTX A$6000$-$48$GB GPU. A cosine learning rate scheduler is applied with a $10\,\%$ warmup.**
| Dataset | Model | Epochs | Batch size | Base LR | Drop path | Layer<br>decay | Weight<br>decay | Label<br>smoothing |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Epilepsy | OTiS-Base | $75$ | $32$ | $1$e-$3$ | $0.2$ | $0.75$ | $0.2$ | $0.1$ |
| OTiS-Large | $75$ | $32$ | $3$e-$3$ | $0.2$ | $0.50$ | $0.1$ | $0.1$ |  |
| OTiS-Huge | $75$ | $32$ | $3$e-$3$ | $0.0$ | $0.75$ | $0.2$ | $0.2$ |  |
| FD-B | OTiS-Base | $75$ | $32$ | $3$e-$4$ | $0.0$ | $0.75$ | $0.1$ | $0.1$ |
| OTiS-Large | $75$ | $32$ | $1$e-$3$ | $0.1$ | $0.75$ | $0.1$ | $0.2$ |  |
| OTiS-Huge | $75$ | $32$ | $3$e-$4$ | $0.1$ | $0.75$ | $0.2$ | $0.1$ |  |
| Gesture | OTiS-Base | $75$ | $32$ | $1$e-$2$ | $0.1$ | $0.25$ | $0.2$ | $0.1$ |
| OTiS-Large | $75$ | $32$ | $3$e-$3$ | $0.2$ | $0.75$ | $0.1$ | $0.0$ |  |
| OTiS-Huge | $75$ | $32$ | $1$e-$2$ | $0.0$ | $0.75$ | $0.1$ | $0.1$ |  |
| EMG | OTiS-Base | $75$ | $32$ | $1$e-$3$ | $0.2$ | $0.75$ | $0.1$ | $0.2$ |
| OTiS-Large | $75$ | $32$ | $3$e-$3$ | $0.1$ | $0.75$ | $0.2$ | $0.1$ |  |
| OTiS-Huge | $75$ | $32$ | $3$e-$3$ | $0.1$ | $0.75$ | $0.2$ | $0.2$ |  |

**Table 8: Hyperparameters used for fine-tuning the regression tasks on a single NVIDIA RTX A$6000$-$48$GB GPU. A cosine learning rate scheduler is applied with a $10\,\%$ warmup.**
| Dataset | Model | Epochs | Batch size | Base LR | Drop path | Layer<br>decay | Weight<br>decay |
| --- | --- | --- | --- | --- | --- | --- | --- |
| UK BioBank | OTiS-Base | $50$ | $192$ | $3$e-$4$ | $0.2$ | $0.75$ | $0.1$ |
| OTiS-Large | $50$ | $160$ | $1$e-$4$ | $0.2$ | $0.75$ | $0.1$ |  |
| OTiS-Huge | $50$ | $200$ | $1$e-$4$ | $0.2$ | $0.75$ | $0.1$ |  |

**Table 9: Hyperparameters used for fine-tuning the forecasting tasks. A cosine learning rate scheduler is applied with a $10\,\%$ warmup.**
| Dataset | Model | Epochs | Batch size | Base LR | NCC $\lambda$ | Weight decay |
| --- | --- | --- | --- | --- | --- | --- |
| ETTh1 | OTiS-Base | $1000$ | $1$ | $1$e-0 0 | $0.1$ | $0.15$ |
| OTiS-Large | $1000$ | $1$ | $1$e-$1$ | $0.2$ | $0.15$ |  |
| OTiS-Huge | $1000$ | $1$ | $3$e-$1$ | $0.1$ | $0.15$ |  |
| ETTh2 | OTiS-Base | $1000$ | $1$ | $1$e-0 0 | $0.2$ | $0.25$ |
| OTiS-Large | $1000$ | $1$ | $1$e-$1$ | $0.1$ | $0.25$ |  |
| OTiS-Huge | $1000$ | $1$ | $3$e-$1$ | $0.0$ | $0.25$ |  |
| ETTm1 | OTiS-Base | $1000$ | $1$ | $3$e-$1$ | $0.2$ | $0.25$ |
| OTiS-Large | $1000$ | $1$ | $3$e-$1$ | $0.2$ | $0.25$ |  |
| OTiS-Huge | $1000$ | $1$ | $1$e-$1$ | $0.1$ | $0.15$ |  |
| ETTm2 | OTiS-Base | $1000$ | $1$ | $3$e-$1$ | $0.1$ | $0.25$ |
| OTiS-Large | $1000$ | $1$ | $3$e-$1$ | $0.2$ | $0.25$ |  |
| OTiS-Huge | $1000$ | $1$ | $1$e-$1$ | $0.2$ | $0.25$ |  |
| Weather | OTiS-Base | $1000$ | $1$ | $3$e-$1$ | $0.2$ | $0.15$ |
| OTiS-Large | $1000$ | $1$ | $3$e-$1$ | $0.2$ | $0.15$ |  |
| OTiS-Huge | $1000$ | $1$ | $1$e-$1$ | $0.2$ | $0.05$ |  |
| Electricity | OTiS-Base | $250$ | $32$ | $3$e-$2$ | $0.0$ | $0.25$ |
| OTiS-Large | $250$ | $32$ | $3$e-$2$ | $0.0$ | $0.15$ |  |
| OTiS-Huge | $250$ | $32$ | $3$e-$2$ | $0.2$ | $0.15$ |  |

## Appendix D Computation Costs

We provide an overview of the computational resources used to train OTiS in Table 10.

**Table 10: Computational resources used to pre-train OTiS. Note that fine-tuning and inference of all OTiS variants on downstream applications were performed using a single NVIDIA RTX A$6000$-$48$GB and $32$ CPUs.**
| Model | Parameters | Power<br>consumption | CPU<br>count | GPU |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Count | Hours | Type |  |  |  |  |
| OTiS-Base | $8$ M | $700\,$W* | $128$ | $4$ | $115$^† | NVIDIA A$100$-$80$GB |
| OTiS-Large | $44$ M | $800\,$W* | $128$ | $4$ | $154$^† | NVIDIA A$100$-$80$GB |
| OTiS-Huge | $131$ M | $960\,$W* | $128$ | $4$ | $219$^† | NVIDIA A$100$-$80$GB |
| * Total power consumption across all GPUs. |  |  |  |  |  |  |
| ^† Total hours across all GPUs. |  |  |  |  |  |  |

## Appendix E Domain Signature Analysis

To analyse the domain signatures, we reduce the dimensionality of the domain-specific variate embeddings by employing a principal component analysis (PCA).
Our analysis shows that OTiS unifies time series from diverse domains into a meaningful latent space, while accurately capturing the inter-variate relationships within a domain.

### E.1 Inter-Domain Analysis

A visualisation of all domain-specific variate embeddings learned during pre-training is provided in Figure 6.
We find that OTiS learns a meaningful latent space, where embeddings of domains with shared high-level semantics cluster closely together.

Figure: (a) $3$D projection.
Refer to caption: x11.png

### E.2 Intra-Domain Analysis

A visualisation of the domain-specific variate embeddings for EEG and ECG are given in Figure 7 and Figure 8, respectively.
We find that OTiS preserves low-level semantics specific to each domain, accurately capturing the relationships between variates.

Figure: (a) True electrode layout.
Refer to caption: extracted/5914143/Figures/EEG_3D_grid_gt.png

Figure: (a) Side view.
Refer to caption: x13.png

## Appendix F Forecast Visualisation

We visualise the performance of our model on $6$ forecasting benchmarks in Figure 9.

Figure: Figure 9: Visualisation of OTiS-Base forecast predictions on $6$ benchmark datasets. A forecasting horizon of $96$ time points is predicted from the past $336$ time points. Ground truth in blue, prediction in orange. Areas highlighted in grey are not visible to the model.
Refer to caption: x16.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: See details in weakness.
soundness: 3
strengths: 1. This paper targets a very important research problem, the time series foundation model. Because the time series data has very high variance across different domains and different tasks. How to integrate them and train one foundation model remains challenging.
2. This paper has a very high quality of preparation. The writing, the organization, and the figure are prepared nicely and with enough details.
3. The results shown in Tab 1, 2, and 3 are competitive compared to baseline TS models.
summary: This paper presents OTiS, a pre-trained foundation model on large-scale time series data to support multi-tasks across domains. Extensive experiments are conducted to demonstrate the powerful performance of the foundation model. This paper is prepared in high quality and can be accepted.
weaknesses: 1. Add a subsection to show which category baselines will be compared. For example, traditional TS model, deep learning, TS foundation model, etc.
2. I expect a comparison with some SOTA TS foundation models. For example, https://arxiv.org/abs/2405.02358 . If this part is added, that would be great.
3. Currently, the author use fine-tune to adapt the pre-trained model to various downstream tasks. Can you also add one more subsection to test the prompting on this TS foundation model? That would be another great point.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: Please feel free to address any misunderstandings I've stated in the weaknesses. Answers to the following questions would help me better calibrate my score:
1. How long does it take to finetune on new tasks?
2. How does the finetuned model perform compared to task-specific models? Are these testing datasets really good cases that need pre-trained models?
3. How do you get the ground truth embeddings in Figure 3?
4. Is there any intuition around what information could be shared across such different domains to make pre-training on them useful?
soundness: 2
strengths: This paper has many strengths:
* The key idea to condition the model on different variables and domains is good. Indeed many related works effectively ignore this information.
* The paper is overall written quite well and arguments are presented clearly.
* The experiments investigate multiple axes, including ablation of their method and different dataset and model sizes, and visualizations of the embeddings.
* The public data and model weights will help the community build on this work.
summary: This paper proposes a time series model architecture and a pre-training objective. The key idea is that their architecture acknowledges that training and testing time series may have different sampling rates and variables. The authors propose a straightforward tokenization scheme to learn embeddings for different variables, which can get added onto regular patch and temporal embeddings, thereby conditioning the predictions on the measured variables. They then pre-train their model on a collection of existing datasets, and evaluate its performance by finetuning on new datasets for some forecasting, regression, and classification datasets. They find that finetuning their model on new datasets can outperform other recent methods.
weaknesses: This paper has weaknesses to address:
* The major weakness of this paper is the extremely limited experiments section. There are many experiments, yet almost no explanation of how they're run or interpretation of the results. Most of the results are written like an advertisement, mostly just stating the method outperforms others. This leaves the reader unclear why the performance gains happen. Ultimately it's not clear when/why the findings would generalize. The result is that some claims appear to be quite overstated. For example, L423-L424 states *"embeddings of domains with shared high-level semantics cluster together, as depicted in Appendix E.1. For example, embeddings of mono and stereo audio group closely, as do those of banking and economics."* But this is cherry-picked---Temperature is way closer to Mono and Stereo Audio than Banking is to Economics.
* Similarly, many important experimental details are missing or relegated to the Appendix, and the Appendix also includes almost no explanations or interpretations. For example, the PCA experiments in Figures 3, 7, and 8 aren't explained.
* It's unclear how many variables actually overlap between training/testing, which seems to be a key element to make the model outperform others. Yet this isn't analyzed. Showing that others fail by ignoring other variables should be a key element of the experiments.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - How does the domain-specific tokenizer adapt to unseen domains with distinct variate structures?

- Additionally, how does the domain-specific tokenizer generalize across different systems within the same domain? For instance, while both electrical transformers and power generators belong to the "energy" domain, they exhibit differing properties and produce distinct time series readings. How does the sub-domain adaptation discussed in Section 3.1 address this scenario?

- A broader question, not specific to this paper: At what level of granularity should we define the domain?
soundness: 2
strengths: - The paper is well-written and easy to follow. 

- Authors spot the important fact that the variate structure is heterogeneous across domains and this structure may represent more complex relationships. 

- The visualization for variate embedding seems to be interesting and insightful. 

- A substantial portion of this research focuses on EEG signals, which presents a novel and promising approach. The authors introduce an innovative method to model a "specific set of systems" that, despite being observed differently—such as TDBrain and SEED with 19 channels versus LEMON with 62 channels—remain comparable.
summary: Authors spot the important fact that the variate structure is heterogeneous across domains and this structure may represent more complex relationships. Thus, they propose a time series pre-training pipeline called OTiS. The OTiS is composed of a specially designed tokenizer that can add domain-specific signature to the time series and a novel loss for pretraining.
weaknesses: - As noted in the strengths, this work addresses the challenge of generalizing across datasets that contain time series of similar systems but are recorded differently, such as variations in sampling rates and physical values. However, the claims regarding cross-domain generalization may be overstated.

- From the perspective of generalized time series analysis, the primary contribution of variate-specific embedding may not be effective in other systems where the interrelationships between variates are not as straightforward as their spatial arrangement (e.g., the electrodes in EEG as depicted in Figure 3 of the manuscript). In different physical systems, two variates may exhibit complex computational relationships (e.g., voltage and current as described by Ohm's Law), complicating the direct modeling of variates as embeddings.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See the weaknesses
soundness: 2
strengths: 1. The paper is well-written, and the method is easy to understand. The authors clearly articulate how they consider the heterogeneity of different domain time series to achieve multi-domain time series forecasting.

2. This paper focuses on the problem of multi-domain time series analysis, which is crucial for building generalizable foundational models for time series.

3. The experimental section utilizes a large amount of data, and the model is open-source, contributing particular engineering value to the community.
summary: This paper presents OTiS for multi-domain time series analysis, building on existing pre-training paradigms for time series. It allocates domain-specific variable embeddings to distinguish the heterogeneity of different variables across domains and enhances the model's ability to learn temporal causal relationships through a dual-masking strategy. Additionally, it introduces NCC loss to capture global patterns. Experimental results demonstrate that the proposed method achieves competitive performance in time series classification, regression, and forecasting tasks across multiple domains compared to SOTA methods. Visualization results further highlight the effectiveness and interpretability of the domain-specific variable embeddings.
weaknesses: 1. The paper mentions that one of the challenges of cross-domain time series models is the significant differences in temporal dynamics and sampling frequencies among different domains. However, the paper uses the same patch size for all domains when dividing patches, failing to accommodate the unique sampling rates of different domains. This oversight means the paper does not sufficiently consider the differences in sampling rates across domains. Additionally, using a shared patch projector to encode the temporal dynamics within each patch does not adequately address the differences in temporal dynamics between domains. While this approach may be common in previous works, it does not consider the temporal heterogeneity among domains.

2. The method of considering variable heterogeneity through learned variable embeddings is not uncommon. In spatiotemporal prediction, some methods [2][3] have already employed learnable embeddings to explicitly distinguish heterogeneous spatiotemporal patterns by learning time-specific and space-specific parameter spaces.

3. [1] proposed using textual descriptions to label different time series domains for cross-domain time series forecasting, utilizing a channel-independent strategy. In contrast, the domain-specific variable embeddings in this paper correspond to a channel-mixing strategy. I look forward to seeing a comparison between these two strategies in cross-domain time series.

4. The experimental section lacks details about the baselines. How were these methods selected? Were they pre-trained and fine-tuned? If so, what data was used for pre-training and fine-tuning?

5. How does the performance of the proposed method compare to conventional time series classification or forecasting methods trained on a single specific dataset?

[1] UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting, WWW, 2024

[2] Heterogeneity-Informed Meta-Parameter Learning for Spatiotemporal Time Series Forecasting, KDD, 2024

[3] Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting, NeurIPS, 2020

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Have you tried to pre-train separately according to different domains and then fine-tune it for domain-specific downstream tasks? As observed from Table 1, there are several discrepancies in different domains, such as the frequencies of Economics and EEG. Is it possible that separating datasets to pre-train domain-specific models works better?
2. The proposed method uses a fixed context for pre-training. Padding a large pre-training corpus, which generally contains univariate time series, into a fixed temporal/variate dimension. Will it cause a waste of computing resources?
soundness: 3
strengths: 1. This paper researches an important question about generalizable time series understanding across diverse domains.
2. This work presents a large pre-training corpus, which can be greatly beneficial if the datasets are released.
3. The method exhibits promising results in handling multivariate time series analysis by leveraging variate and domain signatures.
summary: The paper presents OTiS, a deep model pre-trained on a large corpus (11B) for general time series analysis. In this paper, the authors highlight the challenge of heterogeneity when applying self-supervised pre-training on time series. An MAE-style pre-training method is adopted to obtain a general tokenizer for multivariate time series, and then different task heads are introduced to complete time series analysis tasks.  The model demonstrates strong performance across 15 diverse applications, including time series classification, regression, and forecasting.
weaknesses: 1. My major concern is about the novelty of the proposed method: The design of the encoder/decoder is very identical to MAE. Is there any adaptation for the time series modality? For example, considering the inherent reconstruction difficulties of time series and adjusting the mask ratio compared with the vision modality?
2. About the model design towards generalizable time series understanding: As the authors mention an important challenge of heterogeneity, I am slightly unconvinced that a shared unified patch embedding/projector can reflect different semantics among variates and domains, even if the patch is completely the same. Prior to this, Moirai adopted different patch sizes for different frequencies, will it further enhance OTiS?
3. This work adopts learnable embeddings as variate/domain signatures. I am convinced that the signatures can "distinguish" them, but how can they explicitly "capture inter-variate relationships"? This approach may also limit the generalization scope as the learned signatures do not apply to unseen variates/domains during inference.
4. About the experiments: Results of classification are not compared with supervised, trained deep models, for example, TimesNet and ModernTCN. For the regression rask, can you introduce some variate-centric models into this baseline, such as iTransformer? As for forecasting, the average improvement does not seem significant compared with PatchTST. Also, can you provide some explanations about Table 3 why OTiS has a significant improvement on some datasets (such as ETTh2) and a great degeneration on similar datasets like ETTh1?
5. A minor suggestion: the name "dual masking strategy" can be somewhat overstated to me, which generally refers to dual or antagonistic behavior (e.g., minimax). I would prefer to simplify the contribution as a "mixture" (of masking modeling and generative modeling in this paper), which is a common technique in fact. Also, I would like to know how the ratios (25% - 75% in this paper) of the two strategies are determined.
6. The pipeline of using the masked pre-trained models seems still somewhat tedious, i.e., lacking in generalization. Supervised training should be performed after large-scale pre-training. Can the author provide an overall promotion compared with training from random initialization, or try zero-shot generalization on downstream tasks?

Meta-review: This paper proposes OTiS, a pre-trained foundation model for multi-domain time series analysis, designed to handle the heterogeneity of variables and temporal dynamics across domains. The key contributions include a domain-specific tokenizer, a dual-masking strategy, and a novel loss function (NCC). Experimental results are presented across multiple tasks, including classification, regression, and forecasting, and the authors provide visualizations to highlight the interpretability of the learned embeddings.

This paper is working on a very challenging task – cross-domain time series analysis. The novelty of this work is good. The manuscript is generally well-written, and the proposed methodology is easy to follow. After the rebuttal, the evaluation part is also enhanced. However, the major concern after the rebuttal is still the experiment part. For example, reviewers pointed out that the experiments lack the comparison with other key methods such as UniTS. Experimental results are presented without sufficient explanations or analysis, and important experimental details are missing. In addition, multiple reviewers have concerns about the model design, e.g., the shared patch size may not be able to capture domain-specific temporal dynamics. For these limitations, I am inclined to recommend rejecting this paper.

Decision: Reject