Title: SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling
Venue: NeurIPS 2023
Accepted: yes
Keywords: Time-series analysis

pre-training

masked time-series modeling
Abstract: Time series analysis is widely used in extensive areas. Recently, to reduce labeling expenses and benefit various tasks, self-supervised pre-training has attracted immense interest. One mainstream paradigm is masked modeling, which successfully pre-trains deep models by learning to reconstruct the masked content based on the unmasked part. However, since the semantic information of time series is mainly contained in temporal variations, the standard way of randomly masking a portion of time points will seriously ruin vital temporal variations of time series, making the reconstruction task too difficult to guide representation learning. We thus present SimMTM, a Simple pre-training framework for Masked Time-series Modeling. By relating masked modeling to manifold learning, SimMTM proposes to recover masked time points by the weighted aggregation of multiple neighbors outside the manifold, which eases the reconstruction task by assembling ruined but complementary temporal variations from multiple masked series. SimMTM further learns to uncover the local structure of the manifold, which is helpful for masked modeling. Experimentally, SimMTM achieves state-of-the-art fine-tuning performance compared to the most advanced time series pre-training methods in two canonical time series analysis tasks: forecasting and classification, covering both in- and cross-domain settings.

SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling

Abstract

Time series analysis is widely used in extensive areas. Recently, to reduce labeling expenses and benefit various tasks, self-supervised pre-training has attracted immense interest. One mainstream paradigm is masked modeling, which successfully pre-trains deep models by learning to reconstruct the masked content based on the unmasked part. However, since the semantic information of time series is mainly contained in temporal variations, the standard way of randomly masking a portion of time points will seriously ruin vital temporal variations of time series, making the reconstruction task too difficult to guide representation learning. We thus present SimMTM, a Simple pre-training framework for Masked Time-series Modeling. By relating masked modeling to manifold learning, SimMTM proposes to recover masked time points by the weighted aggregation of multiple neighbors outside the manifold, which eases the reconstruction task by assembling ruined but complementary temporal variations from multiple masked series. SimMTM further learns to uncover the local structure of the manifold, which is helpful for masked modeling. Experimentally, SimMTM achieves state-of-the-art fine-tuning performance compared to the most advanced time series pre-training methods in two canonical time series analysis tasks: forecasting and classification, covering both in- and cross-domain settings. Code is available at https://github.com/thuml/SimMTM.

1 Introduction

Time series analysis has attached immense importance in extensive real-world applications, such as financial analysis, energy planning and etc [47, 51]. Vast amounts of time series are incrementally collected from IoT and wearable devices. However, the semantic information of time series is mainly buried in human-indiscernible temporal variations, making it difficult to annotate. Recently, self-supervised pre-training has been widely explored [23, 15], which benefits deep models from pretext knowledge learned over large-scale unlabeled data and further promotes the performance of various downstream tasks. Mainly, as a well-recognized pre-training paradigm, masked modeling has achieved great success in many areas, such as masked language modeling (MLM) [7, 31, 32, 3, 10] and masked image modeling (MIM) [11, 50, 20]. This paper extends pre-training methods to time series, especially masked time-series modeling (MTM).

The canonical technique of masked modeling is to optimize the model by learning to reconstruct the masked content based on the unmasked part [7]. However, unlike images and natural languages whose patches or words contain much even redundant semantic information, we observe that the valuable semantic information of time series is mainly included in the temporal variations, such as the trend, periodicity, and peak valley, which can correspond to unique weather processes, abnormal faults, etc. in the real world. Therefore, directly masking a portion of time points will seriously ruin

Manifold

Manifold

Multiple

ŏ

0

20

40

60

80

-0.8

-1.0

-1.2

-1.4

-1.6

-1.8

0

20

40

60

80

-0.8

-1.0

-1.2

-1.4

-1.6

-1.8

Original Time Series

Reconstructed Time Series

Original Time Series

Reconstructed Time Series

Time

Time

Value

Value

(a) Canonical Masked Modeling

(b) SimMTM

Direct Reconstruction Neighborhood Aggregation the temporal variations of the original time series, which makes the reconstruction task too difficult to guide representation learning of time series. (Figure 1).

Value

-0.8

-1.0

-1.2

-1.4

-1.6

-1.8

Manifold Original Series Masked Series Manifold Original Series Multiple Masked Series ŏ 0 20 40 60 80 -0.8 -1.0 -1.2 -1.4 -1.6 -1.8 Original Time Series Reconstructed Time Series Time Value 0 20 40 60 80 Original Time Series Reconstructed Time Series Time According to the analysis in stacked denoising autoencoders [40], as shown in Figure 1, we can view the randomly masked series as the 'neighbor' of the original time series outside the manifold and the reconstruction process is to project the masked series back to the manifold of original series. However, as we analyzed above, direct reconstruction may fail since the essential temporal variations are ruined by random masking. Inspired by the manifold perspective, we go beyond the straightforward reconstruction convention of masked modeling and propose a natural idea of reconstructing the original data from its multiple 'neighbors', i.e. multiple masked series. Although the temporal variations of the original time series have been partially dropped in each randomly masked series, the multiple randomly masked series will complement each other, making the reconstruction process much more accessible than directly reconstructing the original series from a single masked series. This process will also pre-train the model to uncover the local structure of the time series manifold implicitly, thereby benefiting masked modeling and representation learning [35, 41].

(a) Canonical Masked Modeling (b) SimMTM Based on the above insights, we propose the SimMTM as a simple but effective pre-training framework for time series in this paper. Instead of directly reconstructing the masked time points from unmasked parts, SimMTM recovers the original time series from multiple randomly masked time series. Technically, SimMTM presents a neighborhood aggregation design for reconstruction, which is to aggregate the point-wise representations of time series based on similarities learned in the series-wise representation space. In addition to the reconstruction loss, a constraint loss is presented to guide the series-wise representation learning based on the neighborhood assumption of the time series manifold. Empowering by above designs, SimMTM achieves consistent state-of-the-art in various time series analysis tasks when fine-tuning the pre-trained model into downstream tasks, covering both the low-level forecasting and high-level classification tasks, even if there is a clear domain shift between pre-training and fine-tuning datasets. Overall, our contributions are summarized as follows:

Inspired by the manifold perspective of masking, we propose a new task for masked timeseries modeling, which is to reconstruct the original time series on the manifold based on multiple masked series outside the manifold.

Technically, we present SimMTM as a simple but effective pre-training framework, which aggregates point-wise representations for reconstruction based on the similarities learned in series-wise representation space.

SimMTM consistently achieves the state-of-the-art fine-tuning performance in typical time series analysis tasks, including low-level forecasting and high-level classification, covering both in- and cross-domain settings.

2 Related Work

2.1 Self-supervised Pre-training

Self-supervised pre-training is an important research topic for learning generalizable and shared knowledge from large-scale data and further benefiting downstream tasks [15]. Originally, this topic has been widely explored in computer vision and natural language processing. Elaborative manuallydesigned self-supervised tasks are presented, which can be roughly categorized into contrastive learning [12, 5, 4] and masked modeling [7, 11]. Recently, following the well-established contrastive learning and masked modeling paradigms, some self-supervised pre-training methods for time series have been proposed [9, 27, 34, 33, 36, 54, 52, 6].

Contrastive learning. The critical insight of contrastive learning is to optimize the representation space based on the manually designed positive and negative pairs. where representations of positive pairs are optimized to be close to each other. In contrast, negative ones tend to be far apart [48, 14]. The canonical design presented in SimCLR [37] views different augmentations of the same sample as positive pairs and augmentations among different samples as negative pairs.

Recently, in time series pre-training, many designs of positive and negative pairs have been proposed by utilizing the invariant properties of time series. Concretely, to make the representation learning seamlessly related to temporal variations, TimCLR [53] adopts the DTW [25] to generate phase-shift and amplitude-change augmentations, which is more suitable for time series context. TS2Vec [55] splits multiple time series into several patches and further defines the contrastive loss in both instancewise and patch-wise aspects. TS-TCC [8] presents a new temporal contrastive learning task as making the augmentations predict each other's future. Mixing-up [45] exploits a data augmentation scheme in which new samples are generated by mixing two data samples. LaST [42] aims to disentangle the seasonal-trend representations in the latent space based on variational inference. Afterward, CoST [46] employs contrastive losses in both time and frequency domain to learn discriminative seasonal and trend representations. Besides, TF-C [57] proposes a novel time-frequency consistency architecture and optimizes time-based and frequency-based representations of the same example to be close to each other. Note that contrastive learning mainly focuses on the high-level information [49], and the series-wise or patch-wise representations inherently mismatch the low-level tasks, such as time series forecasting. Thus, in this paper, we focus on the masked modeling paradigm.

Masked modeling. The masked modeling paradigm optimizes the model by learning to reconstruct the masked content from the unmasked part. This paradigm has been widely explored in computer vision and natural language processing, which is to predict the masked words of a sentence [7] and masked patches of an image [2, 11, 50] respectively. As for the time series analysis, TST [56] follows the canonical masked modeling paradigm and learns to predict removed time points based on the remaining time points. Next, PatchTST [26] proposes to predict masked subseries-level patches to capture the local semantic information and reduce memory usage. Ti-MAE [21] uses mask modeling as an auxiliary task to boost the forecasting and classification performances of advanced Transformer-based methods. However, as we stated before, directly masking time series will ruin the essential temporal variations, making the reconstruction too difficult to guide the representation learning. Unlike the direct reconstruction in previous works, SimMTM presents a new masked modeling task, which is reconstructing the original time series from multiple randomly masked series.

2.2 Understanding Masked Modeling

Masked modeling has been explored in stacked denoising autoencoders [40], where masking is viewed as adding noise to the original data and the masked modeling is to project masked data from the neighborhood back to the original manifold, namely denoising. It has recently been widely used in pre-training, which can learn valuable low-level information from data unsupervisedly [49]. Inspired by the manifold perspective, we go beyond the classical denoising process and project the masked data back to the manifold by aggregating multiple masked time series within the neighborhood.

3 SimMTM

As aforementioned, to tackle the problem that randomly masking time series will ruin the essential temporal variation information, SimMTM proposes to reconstruct the original time series from multiple masked time series. To implement this, SimMTM first learns similarities among multiple time series in the series-wise representation space and then aggregates the point-wise representations of these time series based on pre-learned series-wise similarities. Next, we will detail the techniques in both model architecture and pre-training protocol aspects.

3.1 Overall Architecture

The reconstruction process of SimMTM involves the following four modules: masking, representation learning, series-wise similarity learning and point-wise reconstruction.

has obtained

breakthrough

144

ETTM2

145

176

146

177

147

178

ao et al., 2020; Radford et al.,

179

148

CV) (He et al., 2022; Li et al.,

S

, we can derive the series-wise contrastive loss as follows:

1.153

0.857

parameter.

x

,

y

) =

xy

τ

T

(

and where

3.1. Temporal Masking

Relation between original series and its masked se-

132

3.2. Series-wise Contrastive

0.416

0.388

1.125

0.771

134

1.755

0.947

ries: Following the well-acknowledged mask modeling

133

•

At the crossroads of contrastive learning and mask modeling, is the temperature hyper-

1.420

0.912

N

1.466

0.957

0.306

0.347

M

L

L

paradigm, the temporal masking can also be seen as after temporal masking, we feed it into

Given the input

136

, the model can

By optimizing with contrastive

representations. After a channel projector, we can obtain the d data in deep learning.

149

alleviate the problem of over-

R

180

we find that temporal masking of time series will bring two

X

135

types of self-supervised tasks, which corresponds to series- reconstructed original time series

L

181

150

different but related tasks. Al-

151

learn generalizable and shared

ˆ

x

j

i

contrastive

=

s

(

i

e

Implementation

=1

log

j

=1

∑

,

e

s

}

N

s

)

/τ

s

i

extract valuable series-wise properties to representations.

missing values

, where the task is to reconstruct the the encoder and obtain the deep representations:

137

masked parts based on the reserved time points, namely

i

138

wise and point-wise properties of time series respectively.

)

C

N

Technically, given a batch of time series samples

N

z

z

j M

= Enocder(

×

= Projector(ˆ

z

the point-wise reconstruction task.

j

.

139

formalized as follows:

152

182

153

183

e self-supervised pre-training

184

le et al., 2021; Shi et al., 2021;

185

155

et al., 2022). Unlike CV and

154

ral visual elements and latent ociations through pre-training,

156

ablished such pre-training prin-

157

186

140

,

ˆ

x

Table 3 is a summary of experiment

e

∑

∈S\{

′

s

′

,

)

/τ

s

i

)

/τ

-

i

∑

3.3. Point-wise Reconstruction

M

Thus, the point-wise reconstruction loss can be

i

=1

,

x

i

i

N

=

i

=1

)

x

i

=1

L

C

L

C

{

} ∪ {

}

Z

i

benchmarks. More details about datasets, experiment im-

j

s

,

s

i

)

-

i

log

)

s

,

s

′

pendix

j

∈

i

s

data,

′

i

(4)

Instead of directly combining these two types of tasks, we where

R

×

contains time points and

⋃

(

141

observed

{

}

X

variates, we can easily generate a set of series for each sam-

)

present a special mask modeling task to unify the series-

142

ple

i

∈

where

N

x

z

i

,

which can be formalized as follows:

144

143

the series-wise representations:

where

145

r

[0

Giving consideration to both series-wise and point-wise

158

187

properties, The overall optimization process of SimMTM

159

hifts, and temporal dynamics

188

nals due to insufficient labeled

189

160

146

Given the input

∈

N

rameter for the number of masked time series.

148

represents the

S

i

=1

can be represented as follows:

190

161

149

or time series

162

192

⋃

where

150

163

s

,

tions for original time series

i

151

clarity, we present the input series in a matrix as follows:

x

where

191

positive close neighbors and

Θ

j

i

Since the point-wise reconstruction is based on the series- neighbor

164

194

∑

∈S\{

}

s

(

j

i

Instead of directly generating missing values for masked

=1

e

plementation and model configuration can be found in Ap-

s

(

j

k

,

s

)

i

)

/τ

=1

j

we attempt to reconstruct the original time series

N

e

i

i

/τ

)

log by weighted aggregating the point-wise represen-

/negationslash

x

by randomly masking along the temporal dimension,

{

x

j

i

}

j M

=1

= Temporal

-

Mask

r

(

x

i

)

,

(2)

,

1]

denotes the masked ratio.

M

is a hyperpa-

x

j

i

∈

R

L

×

C

j

-th masked time series of

x

i

, where the val-

ues of masked time points are replaced by zeros. By doing

this, we can obtain a batch of augmented time series. For

X

=

N

{

x

i

} ∪ {

x

j

i

}

j M

=1

.

(3)

Encoder

wise and point-wise properties of time series and make them

collaborate with each other.

3.2. Series-wise Contrastive

X

after temporal masking, we feed it into

the encoder and obtain the deep representations:

Z

=

N

⋃

i

=1

(

{

z

i

} ∪ {

z

j

i

}

j M

=1

)

= Enocder(

X

)

(4)

z

i

,

z

j

i

∈

R

L

×

d

model

. Further, as shown in Figure 1, to

1

k

M,k

=

j

∑

s

′

∈S\{

s

j

i

}

e

s

i

,

s

′

)

/τ

(6)

where

x

,

y

) =

xy

T

and

τ

is the temperature hyper-

parameter.

By optimizing with

L

contrastive

, the model can

extract valuable series-wise properties to representations.

3.3. Point-wise Reconstruction

Instead of directly generating missing values for masked

data,

we attempt to reconstruct the original time series

{

x

i

}

N

i

=1

by weighted aggregating the point-wise represen-

tations

Z

of other series, which is based on the similarity

matrix calculated from series-wise representations

.

Projector

z

j

i

∈

R

L

×

d

model

. Further, as shown in Figure 1, to

obtain series-wise representations, we employ a temporal

pooler to summarize the temporal information and obtain

(

{

s

i

} ∪ {

s

j

i

}

j M

=1

)

= Temporal

-

Pool(

Z

)

,

(5)

s

j

i

∈

R

1

×

d

models

denotes series-wise representa-

x

i

and its mask-augmentation

In specific,

S

contains

(

M

+ 1)

×

N

tations

Z

of other series, which is based on the similarity

matrix calculated from series-wise representations

S

.

Since the similarity matrix of

S

has already been calcu-

lated in Eq. (6). As shown in Figure 1, we can present the

reconstruction of

i

-th original time series as follows:

ˆ

z

i

=

∑

s

′

∈S\{

s

i

}

e

s

i

,

s

′

)

/τ

∑

s

′′

∈S\{

s

i

}

(

e

s

i

,

s

′′

)

/τ

)

z

′

,

(7)

where

z

′

denote the corresponding point-wise representation

of

s

′

and

ˆ

z

i

∈

R

L

×

d

model

are the reconstructed point-wise

Series-wise Similarity

L

reconstruction

=

∑

i

=1

‖

x

i

-

ˆ

x

i

‖

2

2

.

(8)

min

Θ

L

contrastive

+

L

reconstruction

,

(9)

denotes the set of all parameters of the backbone.

Baselines

To verify the generality and effectiveness of

SimMTM, we compared three advanced and prevalent foun-

dation models for time series on four ETT datasets,

in-

cluding NSTransformer (Liu et al., 2022), AutoFormer

(Wu et al., 2021), and vanilla Transformer (Vaswani et al.,

2017); Besides, we also compared five state-of-the-art self-

supervised time series pre-training methods, including the

contrastive learning methods:

TF-C (Zhang et al., 2022),

TS-TCC (Eldele et al., 2021), Mixing-up (Wickstrøm et al.,

2022), TS2Vec (Yue et al., 2022), and masked modeling

method: TST (Zerveas et al., 2021)

Decoder

Point-wise Aggregation

Self-supervised

Pre-training

Far away from each other

Close to each other

Representations of Original Series

Representations of Masked Series

Time

Value

ŏ

Encoder

Projector

…

Point-wise Representations

…

Series-wise Representations

…

…

…

…

…

Series-wise

Similarity

Point-wise

Aggregation

Decoder

Reconstructed

Original Series

Original &

Masked

Series

+

…

…

…

+

respectively.

153

152

154

obtain series-wise representations, we employ a temporal

193

tations of other instances, each

⋃

(

wise similarity matrix, the series-wise contrastive learning

195

will also benefit the reconstruction process. And the opti-

196

197

199

198

200

202

201

203

205

204

206

208

207

209

211

210

212

214

213

215

217

216

218

219

i

=1

From Eq. (3), we can naturally observe the following two types of self-supervised tasks: 156 157 158 159 160 161 162 163 164 the series-wise representations: S = N ⋃ i =1 ( { s i } ∪ { s j i } j M =1 ) = Temporal -Pool( Z ) , (5) where s i , s j i ∈ R 1 × d models denotes series-wise representations for original time series x i and its mask-augmentation x j i respectively. In specific, S contains ( M + 1) × N Since the similarity matrix of S has already been calculated in Eq. (6). As shown in Figure 1, we can present the reconstruction of i -th original time series as follows: ˆ z i = ∑ s ′ ∈S\{ s i } ′ ∑ s ′′ ∈S\{ s i } ( e Sim( s i , s ′′ ) /τ ) where z ′ denote the corresponding point-wise representation of s ′ and ˆ z i ∈ R L × d model are the reconstructed point-wise mization of point-wise representations with reconstruction loss will also bring better series-wise representations for the contrastive loss. Thus, both parts will collaborate with each other for better representation learning of time series. 4. Experiments To evaluate the proposed SimMTM, we extensively experi4.1. Main results As a self-supervised time series pre-training method, our proposed SimMTM consistently achieves state-of-the-art performance on nine real-world time series benchmarks of three main-stream time series analysis tasks, including multivariate time series forecasting, classification, and imputation.(Figure x-axis: classification; y-axis: forecasting; z-axis: imputation) Figure 2: Architecture of SimMTM, which reconstructs the original time series by adaptive aggregating multiple masked time series based on series-wise similarities learned contrastively from data. Masking. Given { x i } N i =1 as a mini-batch of N time series samples, where x i ∈ R L × C contains L time points and C observed variates, we can easily generate a set of masked series for each sample x i by randomly masking a portion of time points along the temporal dimension, formalizing by:

stream time series analysis tasks: multivariate time series ment on nine real-world benchmarks, covering three main-

forecasting, classification, and imputation. The nine bench-

(

e

,

S

z

′

,

(7)

setups, in-domain and cross-domain, in every type of time series analysis task. In In-Domain evaluation, we pre-train and fine-tune using the same or same domain dataset. But in Cross-Domain evaluation, we pre-train a model on one pre-training baselines as a masked pre-training method by randomly masking single time steps of the time series and reconstructing the missing content, indicating that masked time series modeling by temporal-wise reconstrucRepresentation learning. After the encoder and projector layer, we can obtain the point-wise representations Z and series-wise representations S , which can be formalized as:

marks cover different numbers of channels, varying series lengths, and application scenarios, including electricity systems, neurological healthcare, human activity recognition, mechanical fault detection, and physical status monitoring. Then we do self-supervised pre-training to evaluate whether SimMTM can capture effective representations with two In-Domain All results of the in-domain evaluation for the forecasting task in Table 1. We pre-train the model on one pre-training dataset and finetune it to the same target dataset to compare the effect of different time series self-supervised pre-training methods. As shown in Table1, TST outperforms all the contrastive where r ∈ [0 , 1] denotes the masked portion. M is a hyperparameter for the number of masked time series. x j i ∈ R L × C represents the j -th masked time series of x i , where the values of masked time points are replaced by zeros. With above process, we can obtain a batch of augmented time series. For clarity, we present all the ( N ( M +1)) input series in a set as X = ⋃ N i =1 ( { x i } ∪ { x j i } j M =1 ) .

where z i , z j i ∈ R L × d model and s i , s j i ∈ R 1 × d model . Encoder( · ) denotes the model encoder, which can project the input data into deep representations and will be transferred to downstream tasks during the fine-tuning process. In this paper, we implement the encoder as a well-acknowledged Transformer [39] and ResNet [13]. As for the Projector( · ) , we employ a simple MLP layer along the temporal dimension to obtain series-wise representations. More details can be found in Section 4. Technically, the encoder is applied to input series separately, namely ⋃ N i =1 (Encoder( x i ) ∪ { Encoder( x j i ) } j M =1 ) , and so does for projector. Here we adopt the set-style formalization for conciseness.

Series-wise similarity learning. Note that directly averaging multiple masked time series will result in the over-smoothing problem [40], impeding the representation learning. Thus, to precisely reconstruct the original time series, we attempt to utilize the similarities among series-wise representations S for weighted aggregation, namely exploiting the local structure of the time series manifold. For simplification, we formalize the calculation of series-wise similarities as follows:

where R is the matrix of pair-wise similarities for ( N ( M +1)) input samples in series-wise representation space, which are measured by the cosine distance. R u , v is the calculated similarity between series-wise representations u , v ∈ S .

Point-wise aggregation. As shown in Figure 2, based on the learned series-wise similarities, the aggregation process for the i -th original time series is:

where z ′ represents the corresponding point-wise representation of s ′ , i.e. z ′ = Projector( s ′ ) . ̂ z i ∈ R L × d model is the reconstructed point-wise representation. τ denotes the temperature hyperparameter

155

where pooler to summarize the temporal information and obtain

)

∑

∑

+

=1

∑

)

=

147

{

}

i

i

i

)

s

i

,

s

)

j

/τ

0.302

0.342

of softmax normalization for series-wise similarities. It is notable that as described in Eq. (4), for each time series x i , the reconstruction is not only based its own masked series { x j i } j M =1 . We also introduce other series representations S\{ s i } into aggregation, which requires the model to suppress the interference of less-related noise series and precisely learn similar representations for both the masked and the original series, namely guiding the model to learn the manifold structure better. After the decoder, we can obtain the reconstructed original time series, namely

{ ̂ x i } N i =1 = Decoder( { ̂ z i } N i =1 ) , (5) where ̂ x i ∈ R L × C is the reconstruction to x i . Decoder( · ) is instantiated as a simple MLP layer along the channel dimension following [50].

3.2 Self-supervised Pre-training

Following the masked modeling paradigm, SimMTM is supervised by a reconstruction loss:

Note that the reconstruction process is directly based on the series-wise similarities, while it is hard to guarantee the model captures the precise similarities without explicit constraints in the series-wise representation space. Thus, to avoid trivial aggregation, we also utilize the neighborhood assumption of the time series manifold to calibrate the structure of series-wise representation space S . For clarity, we formalize the neighborhood assumption by defining positive and negative pairs as follows:

̸

where s + i and s -i mean the elements that are assumed as close to and far away from s i respectively. Eq. (7) indicates that the original time series and its masked series will present close representations and be far away from the representations from other series in S . For each series-wise representation s ∈ S , we define the set of its assumed close series as S + ⊂ S . Note that to avoid the dominating representation, we assume that s / ∈ S + . With the above formalization, we can define manifold constraint to series-wise representation space as:

which can optimize the learned series-wise representations to satisfy the neighborhood assumption in Eq. (7) better. Finally, the overall optimization process of SimMTM can be represented as follows:

where Θ denotes the set of all parameters of the deep architecture. To trade off the two parts in Eq. (9), we adopt the tuning strategy presented by [17], which can adjust the hyperparameters λ adaptively according to the homoscedastic uncertainty of each loss.

4 Experiments

To fully evaluate SimMTM, we conduct experiments on two typical time series analysis tasks: forecasting and classification, covering low-level and high-level representation learning. Further, we present the fine-tuning performance for each task under in- and cross-domain settings.

Benchmarks. We summarize the experiment benchmarks in Table 1, comprising twelve real-world datasets that cover two mainstream time series analysis tasks: time series forecasting and classification. Concretely, we have followed the standard experimental setups in Autoformer [47] for the forecasting tasks and the experiment settings proposed by TF-C [57] for classification.

Implementations. We compare SimMTM with six competitive state-of-the-art baseline methods, including the contrastive learning methods: TF-C [57], CoST [46], TS2Vec [55], LaST [42], the masked modeling method: TiMAE [21], TST [56]. TF-C [57] and TiMAE [21] are the previous best pre-training methods. We experiment on both in- and crossdomain settings. For the in-domain setting, we pre-train and fine-tune the model using the same dataset. As for the cross-domain setting, we pretrain the model on a certain dataset and fine-tune the encoder to different datasets. More details are provided in Appendix A.

Unified encoder. To make a fair comparison, we attempt to unify encoder for these pre-

training methods. Concretely, we adopt vanilla Transformer [39] with the channel independence [26] as the unified encoder for forecasting. The channel-independent design allows models to accomplish cross-domain transfer between datasets with different variate numbers. As for the classification, we use 1D-ResNet [13] as the shared encoder following [57]. Notably, for LaST [42] and TFC [57], since their designs are closely related to model structures, we directly report results from their papers or reproduce with official codes. For other baselines and SimMTM, the results in the main text are from the unified encoder. The results from their official papers are also compared in Appendix A.2. For all baselines, the results with a unified encoder generally surpass results reported by themselves.

4.1 Main results

We summarize the results in forecasting and classification of in- and cross-domain settings in Figure 3. In all these settings, SimMTM outperforms other baselines significantly. It is also notable that although the masking-based method Ti-MAE [21] achieves good performance in the forecasting task (x-axis of Figure 3), it fails in the classification task (y-axis). Besides, contrastive-based methods fail in low-level forecasting tasks. These results indicate that previous methods cannot simultaneously cover high-level and low-level tasks, highlighting the advantages of SimMTM in task generality.

Cross-Domain

In-Domain

4.2 Forecasting

In-domain. As shown in Table 2, empowering by SimMTM pre-training, the model performance is promoted significantly (SimMTM vs. Random init.). Besides, SimMTM consistently outperforms other pre-training methods. On the average of all benchmarks, SimMTM achieves 8.3% MSE reduction and 4.3% MAE reduction compared to the advanced masked modeling baseline Ti-MAE [21],

14.7% MSE reduction and 12.0% MAE reduction compared to the contrastive baseline CoST [46]. It is also notable that both Ti-MAE [21] and TST [56] outperform all the contrastive-based baselines. This indicates that masked modeling based on point-wise reconstruction suits the forecasting task better than the series-wise contrastive pre-training.

Cross-domain. As demonstrated in Table 3, we present multiple scenarios to verify the effectiveness under the cross-domain setting, where SimMTM consistently outperforms other baselines. Note that the channel-independent encoder enables the comparing baselines to be capable of transferring pre-trained models between datasets with different variate numbers: Weather → {ETTh1, ETTm1}. But for LaST and TF-C with model-specific designs, they cannot be applied to these scenarios. While negative migration has been observed in some cross-domain scenarios, such as Weather → ETTh1 and ETTh2 → ETTm1, SimMTM is still significantly overall superior to other baselines.

In-Domain - Epilepsy to Epilepsy

Cross-Domain - SleepEEG to EMG

In-Domain - ETTh1 to ETTh1

Cross-Domain - ETTh2 to ETTh1

Classification (Accuracy)

Random

W/o Lrec. W/o Lcon.

SimMTM

Forecasting (MSE)

Classification (Accuracy)

Random

W/o Lrec. W/o Lcon.

SimMTM

Forecasting (MSE)

Random

W/o Lcon.

SimMTM

Random

W/o Lcon.

SimMTM

0.44

0.44

100

100

97.56

0.43

0.431

0.43

0.431

95

93.80

94.75

95

0.42

0.414

0.417

0.42

0.420

0.423

0.415

90

89.83

90.99

90

90.24

85.37

0.41

0.409

0.41

85

85

80

80

77.80

0.40

0.40

75

75

4.3 Classification

In-domain. We investigate the in-domain pre-training effect on the classification tasks in Table 4. Note that different from forecasting, the classification task requires the model to learn high-level time series representations. From Table 4, we can find that the contrastive pre-training baselines achieve competitive performances. In contrast, the masking-based model Ti-MAE [21] and TST [56] perform poorly, and TST even exhibits a negative transfer phenomenon compared to the random initialization, indicating that contrastive learning is generally more suitable for classification tasks. Surprisingly, although SimMTM follows the masked modeling paradigm, with our specially-designed reconstruction task, it can still achieve the best performance in the classification task. This is benefited from the neighborhood aggregation from multiple masked series, which enables the model to exploit the local structure of the time series manifold.

Cross-domain. We experiment with four cross-domain transfer scenarios in Table 4: SleepEEG → {Epilepsy, FD-B, Gesture, EMG}, where the target datasets are distinct from the pre-training dataset. Due to the large gap between pre-training and fine-tuning datasets, the baselines perform poorly in most cases, while SimMTM still surpasses other baselines and the random initialization significantly. Especially for SleepEEG → EMG, SimMTM remarkably surpasses previous state-of-the-art TF-C ( Accuracy : 97.56% vs. 81.74%). These results demonstrate that SimMTM can precisely capture valuable knowledge from pre-training datasets and uniformly benefit extensive downstream datasets.

4.4 Model Analysis

Ablations. As shown in Figure 4, we provide ablations to two parts of the training loss in SimMTM. It is observed that both L reconstruction and L constraint are essential to the final performance. Especially, for the SleepEEG → EMG experiment, SimMTM surpasses the random initialization remarkably, where reconstruction and constraint losses provide 9.7% and 16.0% absolute improvement respectively. Besides, we can also find that in comparison to L reconstruction , L constraint provides more contributions to the final results. This comes from our design that the constraint loss uncovers a proper time series

manifold helpful for reconstruction from multiple masked series, without which the neighborhood aggregation will degenerate to the trivial average.

Representation analysis. To illustrate the advantages of SimMTM intuitively, we provide a representation analysis in Table 5, where we can find the following observations. Firstly, we can find that the CKA value of SimMTM in the classification task is clearly smaller than the values in the forecasting task, where the former is a high-level task and the latter requires low-level representations. These results demonstrate that SimMTM can learn adaptive representations for different tasks, which can be benefited from our design in the pre-training loss. Concretely, the temporal variations of classification pre-training datasets are much more diverse than the forecasting datasets. Thus, the L constraint will be easier for optimization in classification, deriving a smaller CKA value. Secondly, from | ∆ CKA | , it is observed that the models pre-trained from SimMTM present a smaller gap with respect to the final fine-tuned model in representation learning properties, which is why SimMTM can consistently improve downstream tasks.

Model generality. From Table 6, we can find that as a general time series pre-training framework, SimMTM can consistently improve the forecasting performance of diverse advanced base models, even for the state-of-the-art time series forecasting model PatchTST [26]. This generality also

0.425

0.417

0.427

0.430

0.429

0.416

0.409

0.419

0.440

0.428

0.422

0.422

1

2

3

4

25%

50%

75%

Masked Ratio

(r)

Numbers of Masked Series ( M )

indicates that we can further improve the model's performance by employing advanced base models as encoders. It is also notable that different from the negative transfer phenomenon caused by the canonical sub-series masked modeling used in the PatchTST paper [26], the consistency improvement of SimMTM further verifies the effectiveness of our design.

Fine-tuning to limited data scenarios. One essential application of pre-training models is to provide prior knowledge for downstream tasks, especially for limited data scenarios, which is critical to the fast-adaption of deep models. Thus, to verify the effectiveness of SimMTM and other pretraining methods in data-limited scenarios, we pre-train a model on ETTh2 and fine-tune it to ETTh1 with different choices for the remaining proportions of training data. All results are presented in Figure 5. We can find that SimMTM achieves significant performance gains in different data proportions compared to other time series pre-training methods. Specifically, for the 10% data finetuning setting, SimMTM significantly outperforms the advanced masking-based method Ti-MAE [21] ( MSE : 0.591 vs. 0.660). Compared with the contrastive-based method TF-C [57], SimMTM also achieves 26.0% MSE reduction. These results further verify that SimMTM can effectively capture valuable knowledge from datasets and boost the final performance, even in limited data scenarios.

Masking strategy. Note that the difficulty of reconstructing the original time series increases along with the increase of the masked ratio, but decreases when the number of neighbor masked series increases. We explore the potential relationship between the masked ratio and the number of masked series used for reconstruction, namely r and M in Eq. (1) respectively. The experimental results in Figure 5 show that we need to set M ∝ r to obtain better results. Experimentally, we choose the masking ratio as 50% and adopt three masked series for reconstruction throughout this paper. In addtion, we can find that only using one masked series ( M = 1 ) for pre-training generally performs worse than the settings with larger M , where the latter enables the model to discover the relations between input series and its neighbors. See Figure 1 for an intuitive understanding. These results further highlight the advantage of our design in manifold learning.

5 Conclusion

This paper presents SimMTM, a simple pre-training framework for masked time-series modeling. Going beyond the previous convention in reconstructing the original time series from unmasked time points, SimMTM proposes a new masked modeling task as reconstructing the original series from its multiple neighbor masked series. Concretely, SimMTM aggregates the point-wise representations based on the series-wise similarities, which are carefully constrained by the neighborhood assumption on the time series manifold. Experimentally, SimMTM can furthest bridge the gap between pretrained and fine-tuned models, thereby achieving consistent state-of-the-art in distinct forecasting and classification tasks compared to the most advanced time series pre-training methods, covering both in-domain and cross-domain settings. In the future, we will further extend SimMTM to large-scale and diverse pre-training datasets in pursuing the foundation model for time series analysis.

Acknowledgments

This work was supported by the National Key Research and Development Plan (2021YFB1715200), National Natural Science Foundation of China (62022050 and 62021002), and Beijing Nova Program (Z201100006820041).

References

Ralph G Andrzejak, Klaus Lehnertz, Florian Mormann, Christoph Rieke, Peter David, and Christian E Elger. Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state. Physical Review E , 2001.

Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In ICML , 2022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS , 2020.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS , 2020.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML , 2020.

Mingyue Cheng, Qi Liu, Zhiding Liu, Hao Zhang, Rujiao Zhang, and Enhong Chen. Timemae: Selfsupervised representations of time series with decoupled masked autoencoders. T-KDE , 2023.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL , 2018.

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In IJCAI , 2021.

Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In NeurIPS , 2019.

Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In IJCNLP , 2020.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR , 2022.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , 2020.

Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies , 2020.

Junguang Jiang, Yang Shu, Jianmin Wang, and Mingsheng Long. Transferability in deep learning: A survey. arXiv preprint arXiv:2201.05867 , 2022.

Bob Kemp, Aeilko H Zwinderman, Bert Tuk, Hilbert AC Kamphuisen, and Josefien JL Oberye. Analysis of a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the eeg. TBME , 2000.

Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR , 2018.

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural network representations revisited. In ICML , 2019.

Christian Lessmeier, James Kuria Kimotho, Detmar Zimmer, and Walter Sextro. Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In PHM , 2016.

Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In CVPR , 2023.

Zhe Li, Zhongwen Rao, Lujia Pan, Pengyun Wang, and Zenglin Xu. Ti-mae: Self-supervised masked time series autoencoders. arXiv preprint arXiv:2301.08871 , 2023.

Jiayang Liu, Lin Zhong, Jehan Wickramasuriya, and Venu Vasudevan. uwave: Accelerometer-based personalized gesture recognition and its applications. In PMC , 2009.

Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative or contrastive. TKDE , 2021.

Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. In NeurIPS , 2022.

Abdullah Al Mueen and Eamonn J. Keogh. Extracting optimal performance from dynamic time warping. KDD , 2016.

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In ICLR , 2023.

Manuel T Nonnenmacher, Lukas Oldenburg, Ingo Steinwart, and David Reeb. Utilizing expert features for contrastive learning of time-series representations. In ICML , 2022.

Adam Paszke, S. Gross, Francisco Massa, A. Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Z. Lin, N. Gimelshein, L. Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS , 2019.

PeMS. Traffic Dataset. http://pems.dot.ca.gov/ .

PhysioToolkit PhysioBank. Physionet: components of a new research resource for complex physiologic signals. Circulation , 2000.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR , 2020.

Quentin Rebjock, Baris Kurt, Tim Januschowski, and Laurent Callot. Online false discovery rate control for anomaly detection in time series. In NeurIPS , 2021.

Pritam Sarkar and Ali Etemad. Self-supervised learning for ecg-based emotion recognition. In ICASSP , 2020.

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR , 2015.

Fan-Keng Sun, Chris Lang, and Duane Boning. Adjusting for autocorrelated errors in neural networks for time series. In NeurIPS , 2021.

Chi Ian Tang, Ignacio Perez-Pozuelo, Dimitris Spathis, and Cecilia Mascolo. Exploring contrastive learning in human activity recognition for healthcare. In NeurIPS , 2020.

UCI. UCI Electricity Load Time Series Dataset. https://archive.ics.uci.edu/ml/datasets/ ElectricityLoadDiagrams20112014 .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Pascal Vincent, H. Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. JMLR , 2010.

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML , 2020.

Zhiyuan Wang, Xovee Xu, Weifeng Zhang, Goce Trajcevski, Ting Zhong, and Fan Zhou. Learning latent seasonal-trend representations for time series forecasting. In NeurIPS , 2022.

Wetterstation. Weather Dataset. https://www.bgc-jena.mpg.de/wetter/ .

Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask 15% in masked language modeling? In EACL , 2023.

Kristoffer Wickstrøm, Michael Kampffmeyer, Karl Øyvind Mikalsen, and Robert Jenssen. Mixing up contrastive learning: Self-supervised representation learning for time series. PRL , 2022.

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. In ICLR , 2022.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In NeurIPS , 2021.

Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via nonparametric instance discrimination. In CVPR , 2018.

Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. In CVPR , 2023.

Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR , 2022.

Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. In ICLR , 2021.

Ling Yang and Shenda Hong. Unsupervised time-series representation learning with iterative bilinear temporal-spectral fusion. In ICML , 2022.

Xinyu Yang, Zhenguo Zhang, and Rongyi Cui. Timeclr: A self-supervised contrastive learning framework for univariate time series representation. KBS , 2022.

Hugo Yèche, Gideon Dresdner, Francesco Locatello, Matthias Hüser, and Gunnar Rätsch. Neighborhood contrastive learning applied to online patient monitoring. In ICML , 2021.

Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. TS2Vec: Towards Universal Representation of Time Series. In AAAI , 2022.

George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In SIGKDD , 2021.

Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. In NeurIPS , 2022.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI , 2021.

A Implementation Details

All the experiments are repeated five times, implemented in PyTorch [28] and conducted on NVIDIA A100 SXM4 40GB GPU. We implement the baselines based on their official implementation and follow the configuration from their original papers. For the metrics, we adopt the mean square error (MSE) and mean absolute error (MAE) for the time series forecasting. As for the classification, accuracy, precision, recall, F1 score, and their average value are recorded.

A.1 Dataset Description

We conduct experiments to evaluate the effect of our method under in-domain and cross-domain settings on twelve real-world datasets for two typical time series analysis tasks: forecasting and classification, covering diverse application scenarios (electricity system, neurological healthcare, human activity recognition, mechanical fault detection, and physical status monitoring), different types of signals (ECG, EMG, acceleration, vibration, power load, weather, and transoirtation), multivariate channel dimensions (from 1 to 862), varying times series lengths (from 96 to 5120) and large span sampling ratio (from 100 Hz to 4000 Hz). The detailed descriptions of these datasets are summarized in Table 7.

(1) ETT (4 subsets) [58] contains the time series of oil temperature and power load collected by electricity transformers from July 2016 to July 2018. ETT is a group of four subsets with different recorded frequencies: ETTh1/ETTh2 are recorded every hour, and ETTm1/ETTm2 are recorded every 15 minutes.

(2) WEATHER [43] includes meteorological time series with 21 weather indicators collected every 10 minutes from the Weather Station of the Max Planck Biogeochemistry Institute in 2020.

(3) ELECTRICITY [38] records the hourly electricity consumption of 321 clients from 2012 to 2014. Values are in kW of each 15 min. All time labels report to Portuguese hour. However, all days present 96 measures (24 × 4). For every year in March, time change day (which has only 23 hours), values between 1:00 am and 2:00 am are zero for all points. For every year in October, time change day (which has 25 hours), the values between 1:00 am and 2:00 am aggregated consumption of two hours.

(4) TRAFFIC [29] encompasses the hourly measures of road occupancy rates obtained from 862 sensors situated in the San Francisco Bay area freeways. These measurements were carried out between January 2015 and December 2016.

(5) SLEEPEEG [16] contains 153 whole-night sleeping electroencephalography (EEG) recordings from 82 healthy subjects. We follow the same data preprocessing approach as [57] and get 371,055 univariate brainwaves. Each brainwave is sampled at a frequency of 100 Hz and associated with one of five sleeping stages: Wake, Non-rapid eye movement (3 sub-states), and Rapid Eye Movement.

(6) EPILEPSY [1] monitors the brain activities of 500 subjects with a single-channel EEG sensor. Every subject is recorded for 23.6 seconds of brain activities. The dataset is sampled at 178 Hz and contains 11,500 samples in total. We follow the procedure described by [57]. The first four classes (eyes open, eyes closed, EEG measured in the healthy brain region, and EEG measured in the tumor region) of the original five categories of each sample are classified as positive, and the remaining classes are used as negative.

(7) FD-B [19] is generated by electromechanical drive systems. It monitors the condition of rolling bearings and detects their failures based on the monitoring conditions, which include speed, load torque, and radial force. Concretely, FD-B has 13,640 samples in total. Each recording is sampled at 64k Hz with 3-class labels: undamaged, inner damaged, and outer damaged.

(8) GESTURE [22] are collected from 8 hand gestures based on the paths of hand movement recorded by an accelerometer. The eight gestures are hand swiping left, right, up, and down, hand waving in a counterclockwise or clockwise circle, hand waving in a square, and waving a right arrow, respectively. This dataset contains 440 examples of balanced classification labels that can be used, and each sample includes eight different kinds of gesture categories.

(9) EMG [30] is sampled with 4K Hz and consists of 163 single-channel EMG recordings from the anterior tibialis muscle of three healthy volunteers suffering from neuropathy and myopathy. Each patient is a classification category, so each sample is associated with one of three classes.

A.2 Baselines Implementation

We compare SimMTM against six state-of-the-art baselines. To make a fair and comprehensive comparison, we tried two baseline implementation approaches for forecasting and classification tasks: the unified encoder and reproduced with their official implementation encoder. Notably, LaST [42] and TF-C [57] are closely related to model structures. We directly report results from their papers or reproduce codes with official implementation.

(1) Unified encoder . We attempt to unify the encoder for these pre-training methods. Specifically, we adopt the vanilla Transformer [39] with channel independent [26] for forecasting to accomplish cross-domain transfer between datasets with different variate numbers. As for the classification, we use 1D-ResNet [13] as the

encoder following [57]. Besides, we do a comprehensive hyperparameter search for all baselines. For the Transformer encoder, we vary the number of Transformer layers in { 1 , 2 , 3 , 4 } , select the model dimension from { 16 , 32 , 64 , 128 , 256 } , and the attention head from { 4 , 8 , 16 , 32 } . For the 1D-ResNet, we search the number of 1D-ResNet layers from { 1 , 2 , 3 , 4 } , the kernel size from { 3 , 5 , 8 } respectively. Additionally, for the masked modeling methods TST [56], Ti-MAE [21], we also searched the masked ratio r = { 0 . 125 , 0 . 25 , 0 . 5 , 0 . 75 } for better performance.

(2) Official implementation . We also implement the baselines following the corresponding official codes, including encoder, hyperparameters, etc. The comparisons are included in Section E of this supplementary material. We directly report the results from their original papers for the same set. For mismatched settings, the results are from our implementation.

Finally, for baselines Ti-MAE [21], TST [56], CoST [46], and TS2Vec [55], we report the results based on the unified encoder in the main text. But for baselines LaST [42] and TF-C [57], we report the results of the official code implementation or their original paper, which are limited by their model structures. As a result, the performances of all baselines with unified encoder (that we reported in the main text) generally surpass their official implementation and results reported in their own paper. Table 8 shows more details. Full experimental results are in Section E.

A.3 Pre-training and Fine-tuning Configuration

We built two types of pre-training and fine-tuning scenarios, in-domain and cross-domain, based on the benchmarks of forecasting and classification tasks to compare the effectiveness of our method and other time series pre-training methods.

We pre-train a model on one subset for forecasting tasks and fine-tune it to the same dataset to build seven in-domain transfer evaluation scenarios. In cross-domain evaluation, we pre-train a model on one specific dataset and use other datasets for fine-tuning. Based on the above settings, we constructed fifteen in- and cross-domain pre-training and fine-tuning experiments, covering the same dataset with the same sampled frequency, different datasets with the same sampled frequency, and different datasets with different sampled frequencies.

We use the same dataset, Epilepsy, to construct the in-domain setting for classification tasks. For the crossdomain setting, we pre-train a model for classification tasks on a univariate time series dataset SleepEEG with the most complex temporal dynamics and the most samples. And then fine-tune the model separately on Epilepsy, FD-B, Gesture, and EMG. Furthermore, we constructed four cross-domain evaluation scenarios by pre-training from SleepEEG and fine-tuning to Epilepsy, FD-B, Gesture, and EMG because of fewer commonalities and the enormous gap among these datasets. Table 9 shows detailed pre-training and fine-tuning settings.

A.4 Model and Training Configuration

Following the previous convention, we choose the encoder part of Transformer [39] with channel independent as the feature extractor for forecasting tasks. For the classification tasks, we adopt 1D-ResNet [13] as the encoder following [57]. In the pre-training stages, we pre-train the model with different learning rates and batch sizes according to the pre-train datasets. Then we fine-tune it to downstream forecasting and classification tasks supervised by L2 and Cross-Entropy losses, respectively. The configuration details are in Table 10.

B Hyperparameter Sensitivity

We verify the hyperparameter sensitivity of SimMTM on ETTh1 in Table 11, including masked ratio ( r ), the number of masked series ( M ), temperature ( τ ), masked function ( Mask ), encoder depth ( e layers ), and the hidden dimension ( d model ). Lower MSE and MAE represent better performance.

As shown in Table 11 (a) and 11 (b), we can observe the effect of the method is closely related to the trade-off of the masked ratio and the number of masked series. Hence, a reasonable balance between the two kinds of parameters is critical. For the temperature hyperparameter of softmax normalization ( τ ), we use an appropriately small τ that leads to higher differences and diversity of masked sequences. For the masked methods, we chose two masked methods for verification: masking following random distribution and masking following geometric distribution [56]. The results show that the method based on geometric masking is better than random masking modeling. Besides, we can find that 2 encoder layers are enough for reconstruction tasks. Note our method SimMTM consistently performs better than training from scratch under various hyperparameter changes.

C Ablations on Aggregation Setting

SimMTM proposes to recover masked time points by the weighted aggregation of multiple neighbors outside the manifold. We explored two types of aggregation settings.

(1) Positive Samples Aggregation (PSA) : only aggregate multiple positive neighbors (the masked series of the same sample) to reconstruct masked time points.

(2) Positive and Negative Samples Aggregation (PNSA) : aggregate both positive and negative neighbors (the masked series of all samples) to reconstruct masked time points.

As shown in Table 12, although PSA made good progress compared to training from scratch (Random Init.), PNSA is consistently better than SimMTM PSA in all ablation settings. In masked time-series modeling, masking can be viewed as adding noise to the original data, and masked modeling is to project masked data from the neighborhood back to the original manifold. We use positive and negative masked time series as the reconstruction candidates to drive the model to select the positive samples adaptively, which can make the model learn the structure of the manifold better. Therefore, as stated in the Method Section of the main text, we choose positive and negative sample aggregation (PNSA) as the standard aggregation setting of SimMTM.

D Comparison of Masked Modeling

To investigate the reconstruction process of different masked modeling methods, we plot both original and reconstructed time series from TST and SimMTM in Figure 6, where TST [56] follows the canonical masked modeling paradigm and learns to predict removed time points based on the remaining time points. In Figure 6, we can find that direct reconstruction is too difficult in time series, even for the 12.5% masking ratio. As for the 75% masking ratio, TST degenerates more seriously. Because of this poor reconstruction effect, direct reconstruction is difficult to provide reliable guidance to model pre-training. In contrast, our proposed SimMTM can precisely reconstruct the original time series, benefiting the representation learning. These results also support our design in neighborhood reconstruction.

TST    Masked 12.5%

TST    Masked 25%

TST    Masked 50%

TST    Masked 75%

SimMTM (Ours)    Masked 12.5%

SimMTM (Ours)    Masked 25%

SimMTM (Ours)    Masked 50%

SimMTM (Ours)    Masked 75%

Original Time Series

Reconstructed Time Series

Original Time Series

Reconstructed Time Series

Original Time Series

Reconstructed Time Series

Original Time Series

Reconstructed Time Series

Original Time Series

Reconstructed Time Series

Original Time Series

Reconstructed Time Series

Original Time Series

Reconstructed Time Series

Original Time Series

Reconstructed Time Series

E Full Results

Due to the limited length of the text, we summarize all the experiments in the main text into two parts: the main experiment and the analytical experiment. We categorize and index them in Tabel 13, 14.

F Limitations

SimMTM is inspired by the manifold perspective of masked modeling. Although we have provided relatively comprehensive results to verify the model's effectiveness, the model performance still needs theoretical guarantees. In fact, the most high-impact works in the self-supervised pre-training community are without theoretical analysis, such as BERT [7], GPT-3 [10], MAE [11] and SimMIM [50]. Thus, we would like to leave this problem as a future work.

The masking ratio of masked modeling methods is an essential hyper-parameter. Although we have provided a chosen principle to masking ratio r and the number of masked time series M as M ∝ r in the main text, we still need to tune these two hyperparameters for different datasets to achieve the best performance. Notably, previous methods also chose the masking ratio solely based on the empirical results [7, 11]. Thus, despite there exist limitations of SimMTM in choosing hyperparameters, the principle of M ∝ r can somewhat ease this problem. And the chosen strategy of the masking ratio can also be a potential topic in masked modeling [44].

G Social Impacts

This paper presents SimMTM as a new masked modeling method for time series. SimMTM achieves state-ofthe-art in two mainstream time series analysis tasks, which can be a good supplement for the self-supervised pre-training community. We will also publish the codebase of time-series pre-training to facilitate future research.

This paper only focuses on the algorithm design. Using all the codes and datasets strictly follows the corresponding licenses (Appendix A.1). There is no potential ethical risk or negative social impact.

Peer reviews:
comments: SUMMARY:
This paper presents SimMTM, a simple pre-training framework for masked time-series modeling. The core idea is to train the model to reconstruct the time series by aggregating information from multiple masked series rather than one single masked series. While the temporal pattern is destroyed in a single masked series, multiple masked series contain complementary information, making the reconstruction process more accessible. The aggregation of point-wise representations is weighted by their series-wise similarities. The proposed loss adds the reconstruction loss with a contrastive loss on the series representations. The author provides extensive empirical studies on both time series classification and time series forecasting tasks to evaluate the learned representations.

STRENGTHS:
The proposed method is sound and well-motivated. Reconstructing the original time series from multiple randomly masked series is novel, in my opinion. The write-up is easy to follow. The overview figure is also illustrative and helps the understanding. The empirical evaluation is comprehensive and convincing.

WEAKNESSES:
1. In Eq 4, the sum over the variable $s'$ is confusing without carefully reading the text below it. In my opinion, it would be more clear to have the sum over $z'$ and have $s' = Projector(z')$ on the right.

2. Eq 7 is not precise. I understand that the authors mean the intra-similarity of samples should be maximized. However,  the sets before and after $~$ are the same, and therefore they are already "close". Please considering revise eq 7.

QUESTIONS:
The paper says, "For each time series, the reconstruction is not only based on its own masked series." I wonder how much does the method benefit from aggregating representations from other time series?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
Pre-training models have been using self-supervised learning in various fields (NLP, Vision). However, masked modeling, a representative method of self-supervised learning, was difficult to apply to time-series tasks. Randomly masked data is difficult to show good performance because semantic information of time series including temporal changes is broken. Thus, SimMTM recovers the masked point in time through weighted aggregation of multiple neighbors outside the manifold. This not only restores simple, but also reconstructs complementarily, leading to good performance. SimMTM performs forecasting and classification (in-domain, cross-domain) on various datasets and shows good performance.

STRENGTHS:
1. They explained the problem of masked modeling in time-series data well from the time-series point of view.
2. Also, to apply masked modeling to time-series data, the characteristics of time-series were well utilized. In other words, masked modeling was not simply applied to time-series, but the characteristics of time-series data were well understood and applied accordingly.
3. Experiments were conducted on various tasks of the pre-training model, and remarkable performance improvements were achieved.

WEAKNESSES:
It seems that there is a part that does not match the notation in the formula and Figure 2.

QUESTIONS:
Regarding weakness, I don't know what $z'$ means in Eq.4 and line 154. No explanation of $s'$ in "$z'$ represents the corresponding point-wise representation of $s'$". Does $s'$ mean series-wise representations $S$?

LIMITATIONS:
In fact, there is no limit to this work
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose SimMTM, a novel representation learning method for time-series data based on random masking. They experimentally verify the effectiveness of the proposed method from both in-domain and cross-domain perspectives. Comprehensive comparative experiments with existing methods are conducted, demonstrating the effectiveness of the proposed approach.\n\nSTRENGTHS:\nThe authors have conducted extensive comparative experiments with existing methods and quantitatively demonstrated the effectiveness of the proposed method. In addition to in-domain generalization, they also focus on cross-domain generalization. This verification is indispensable for validating the effectiveness of deep representation learning assuming large-scale open data.\n\nWEAKNESSES:\n**Presentation quality**\n\nThe current presentation style of this paper is confusing. Particularly, there is a lack of consistency in the representation of formulas and symbols, making it very difficult to grasp the overall logic of the paper. Some specific examples include:\n\n1. In Equation 3, the symbol $\\times$ is used to denote both the Cartesian product and scalar multiplication when defining the domain of $R$.\n2. The transposition notation in Equation 3 is counterintuitive as $\\mathbf{s}$ is defined as a row vector.\n3. Equation 7 is overly verbose and exacerbates the difficulty in reading, as it does not contribute to the subsequent. It would be sufficient to simply declare that only the situation where $i = k$ is treated as positive pairs.\n4. While $\\mathbf{s}$ is defined as a single vector, $\\mathbf{s}^+$ is defined as a set of vectors.\n\nSome other points that I could not make from the paper are discussed below in the Questions section.\n\n**Comparison with prior literature**\n\nThe comparison with prior literature is insufficient. For example, references [1, 2] can be cited as representative examples of representation learning methods for deep neural networks under time-series data. In particular, a model that generalized the method in [2] as a generalized contrastive learning method not limited to the time-series domain was proposed later. Both [2] and [3] have superior theoretical guarantees for identifiability from the perspective of nonlinear ICA. The authors should clearly indicate the proposed model's advantages over these approaches.\n\n**References**\n\n1. Hyvarinen, A., & Morioka, H. (2016). Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. *Advances in neural information processing systems*, *29*.\n2. Oord, A. V. D., Li, Y., & Vinyals, O. (2018). Representation learning with contrastive predictive coding. *arXiv preprint arXiv:1807.03748*.\n3. Hyvarinen, A., Sasaki, H., & Turner, R. (2019, April). Nonlinear ICA using auxiliary variables and generalized contrastive learning. In *The 22nd International Conference on Artificial Intelligence and Statistics* (pp. 859-868). PMLR.\n\nQUESTIONS:\n- What is the definition of $\\operatorname{Encoder}(\\mathcal{X})$? Typically, a function $f \\colon X \\rightarrow Y$ can use all the information in $X$ to calculate $Y$, but it seems that the Encoder function defined here applies the same function in parallel to some dimension. However, it is not clear from the text which dimension this parallel processing applies to.\nFor example, there would be at least two possibilities: $\\operatorname{Encoder}(\\mathcal{X})$ is kind of a “syntax suger” for either $\\bigcup\\_i \\operatorname{Encoder}( \\\\{x\\_i\\\\} \\cup \\\\{\\bar{x}\\_i^j \\\\}\\_{j} )$ or $\\bigcup\\_i ( \\\\{ \\operatorname{Encoder}(x\\_i) \\\\} \\cup \\\\{ \\operatorname{Encoder}(\\bar{x}\\_i^j) \\\\}\\_{j} )$. The same can be said about the definitions of the Projector and Decoder.\n- Although I understood the training pipeline of the proposed method, I had questions regarding its behavior at inference time. Does the proposed method compute internal representations from the true sequence and $M$ randomly masked sequences at inference time, similar to training time? Or does it only use the true sequence without random masking at inference time? Additionally, how is training conducted during the fine-tuning stage in the pipeline?\n- The authors mention the analysis of a quantity called the CKA value in line 262 of Section 4.4, but it is unclear what this quantity actually is. While there is an explanation of Pre-training/Fine-tuning CKA in the caption of Table 5, if the CKA value mentioned in the main text refers to these, its meaning should be clearly explained in the text as well. Moreover, I did not understand why a small $| \\Delta_{\\mathrm{CKA}} |$ leads to “acquiring adaptive representations for different tasks” (l.264). In principle, it should be possible to drastically change the internal representation by fine-tuning while maintaining the degree of change in the internal representation along the evolution of the layers. Suppose $x_\\mathrm{first}$ and $x_\\mathrm{last}$ be the representation in the first and the last layer at the pre-training phase, respectively. We can consider one idealistic situation that the representations after fine-tuning are given as $y_{\\mathrm{first}} = x_{\\mathrm{first}} + \\delta_{\\mathrm{task}}$ and $y_{\\mathrm{last}} = x_{\\mathrm{last}} + \\delta_{\\mathrm{task}}$ with a task-dependent fixed vector $\\delta$. In such a situation, the values of Pre-training/Fine-tuning CKA would be nearly the same, and the difference in CKA values would go to zero by construction. However, the internal representations drastically change depending on the task choice. In this context, I believe there is a logical gap in the authors' argument.\n- In masked modeling-based representation learning, the proposed method uses the true sequence without masking, along with the input. This setup is generally considered to make it more challenging to avoid trivial local solutions that merely output the input as is. Can it be said that such a phenomenon does not occur? Moreover, if a trivial local solution can be avoided, what factors of the proposed method enable this?\n- As a naive control condition for the idea of using multiple different random masks, it is possible to consider the variant of using only one random mask that reduces the masked portion $r$, or takes the position of the mask to have a specific rules that is not completely iid. Can the proposed method be claimed to be effective against such variants? In particular, the model structure of using multiple random masks is expected to have at least some overhead in terms of training/inference computation time compared to such simple control conditions.\n\nLIMITATIONS:\nThe limitations of this paper are adequately discussed in the appendix. Furthermore, the effectiveness of the proposed method can be confirmed from various perspectives through exhaustive comparative experiments. However, as mentioned earlier, there are major concerns primarily about the presentation of the paper. These are points that affect the reproducibility of the results and the authors' claims, so it would be necessary to resolve all problems of unclear logical progression for this paper to be accepted.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a simple pre-training framework for masked time-series modeling. Instead of reconstructing origin data directly, which is unsuitable for time series, this paper recovers masked time points by the weighted aggregation of multiple neighbors outside the manifold. The reviewer appreciates the novelty method proposed, but is still concerned about some results.

STRENGTHS:
1) The method is of novelty. The multi-level aggregation and representation method is appealing.
2) Experimentally, both high-/low-level tasks benefit from this method.

WEAKNESSES:
1. The experimental results need more justification (see Questions). 
2. The relationship with manifold learning needs more description. What is the progress in this aspect?
3. The masked parts in Figure 1 and Figure 1 in Supplement material should be annotated and more results should be listed. A well-trained network usually predicts low-frequency results. Why the result of TST in the right-upper part in Figure 1 in Supplement material contains too many high-frequency parts?

QUESTIONS:
In terms of experimental results, why Random init. outperforms TF-C in Table 2? Does SimMTM use a stronger baseline? Is it an unfair comparison? Maybe TF-C or other methods will outperform SimMTM with the same baseline. It seems that the model learns directly aggregate information from different masked series, which learns more temporal information than other semantic information.

LIMITATIONS:
The authors should discuss more implications about using large-scale pre-training in forecasting in terms of fairness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07