SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiaxiang Dong\* Haixu Wu\* Haoran Zhang, Li Zhang, Jianmin Wang, Mingsheng Long
School of Software, BNRist, Tsinghua University, China {djx20,z-hr20}@mails.tsinghua.edu.cn, wuhaiu98@gmail.com {lizhang,jimwang,mingsheng}@tsinghua.edu.cn
Time series analysis is widely used in extensive areas. Recently, to reduce labeling expenses and benefit various tasks, self-supervised pre-training has attracted immense interest. One mainstream paradigm is masked modeling, which successfully pre-trains deep models by learning to reconstruct the masked content based on the unmasked part. However, since the semantic information of time series is mainly contained in temporal variations, the standard way of randomly masking a portion of time points will seriously ruin vital temporal variations of time series, making the reconstruction task too difficult to guide representation learning. We thus present SimMTM, a Simple pre-training framework for Masked Time-series Modeling. By relating masked modeling to manifold learning, SimMTM proposes to recover masked time points by the weighted aggregation of multiple neighbors outside the manifold, which eases the reconstruction task by assembling ruined but complementary temporal variations from multiple masked series. SimMTM further learns to uncover the local structure of the manifold, which is helpful for masked modeling. Experimentally, SimMTM achieves state-of-the-art fine-tuning performance compared to the most advanced time series pre-training methods in two canonical time series analysis tasks: forecasting and classification, covering both in- and cross-domain settings. Code is available at https://github.com/thuml/SimMTM.
Time series analysis has attached immense importance in extensive real-world applications, such as financial analysis, energy planning and etc [47, 51]. Vast amounts of time series are incrementally collected from IoT and wearable devices. However, the semantic information of time series is mainly buried in human-indiscernible temporal variations, making it difficult to annotate. Recently, self-supervised pre-training has been widely explored [23, 15], which benefits deep models from pretext knowledge learned over large-scale unlabeled data and further promotes the performance of various downstream tasks. Mainly, as a well-recognized pre-training paradigm, masked modeling has achieved great success in many areas, such as masked language modeling (MLM) [7, 31, 32, 3, 10] and masked image modeling (MIM) [11, 50, 20]. This paper extends pre-training methods to time series, especially masked time-series modeling (MTM).
The canonical technique of masked modeling is to optimize the model by learning to reconstruct the masked content based on the unmasked part [7]. However, unlike images and natural languages whose patches or words contain much even redundant semantic information, we observe that the valuable semantic information of time series is mainly included in the temporal variations, such as the trend, periodicity, and peak valley, which can correspond to unique weather processes, abnormal faults, etc. in the real world. Therefore, directly masking a portion of time points will seriously ruin
Figure 1: Comparison between (a) canonical masked modeling and (b) SimMTM in both manifold perspective and reconstruction performance. The showcase is to recover  $50\%$  masked time series.
the temporal variations of the original time series, which makes the reconstruction task too difficult to guide representation learning of time series. (Figure 1).
According to the analysis in stacked denoising autoencoders [40], as shown in Figure 1, we can view the randomly masked series as the "neighbor" of the original time series outside the manifold and the reconstruction process is to project the masked series back to the manifold of original series. However, as we analyzed above, direct reconstruction may fail since the essential temporal variations are ruined by random masking. Inspired by the manifold perspective, we go beyond the straightforward reconstruction convention of masked modeling and propose a natural idea of reconstructing the original data from its multiple "neighbors", i.e. multiple masked series. Although the temporal variations of the original time series have been partially dropped in each randomly masked series, the multiple randomly masked series will complement each other, making the reconstruction process much more accessible than directly reconstructing the original series from a single masked series. This process will also pre-train the model to uncover the local structure of the time series manifold implicitly, thereby benefiting masked modeling and representation learning [35, 41].
Based on the above insights, we propose the SimMTM as a simple but effective pre-training framework for time series in this paper. Instead of directly reconstructing the masked time points from unmasked parts, SimMTM recovers the original time series from multiple randomly masked time series. Technically, SimMTM presents a neighborhood aggregation design for reconstruction, which is to aggregate the point-wise representations of time series based on similarities learned in the series-wise representation space. In addition to the reconstruction loss, a constraint loss is presented to guide the series-wise representation learning based on the neighborhood assumption of the time series manifold. Empowering by above designs, SimMTM achieves consistent state-of-the-art in various time series analysis tasks when fine-tuning the pre-trained model into downstream tasks, covering both the low-level forecasting and high-level classification tasks, even if there is a clear domain shift between pre-training and fine-tuning datasets. Overall, our contributions are summarized as follows:
- Inspired by the manifold perspective of masking, we propose a new task for masked time-series modeling, which is to reconstruct the original time series on the manifold based on multiple masked series outside the manifold.
- Technically, we present SimMTM as a simple but effective pre-training framework, which aggregates point-wise representations for reconstruction based on the similarities learned in series-wise representation space.
- SimMTM consistently achieves the state-of-the-art fine-tuning performance in typical time series analysis tasks, including low-level forecasting and high-level classification, covering both in- and cross-domain settings.
2.1 Self-supervised Pre-training
Self-supervised pre-training is an important research topic for learning generalizable and shared knowledge from large-scale data and further benefiting downstream tasks [15]. Originally, this topic has been widely explored in computer vision and natural language processing. Elaborative manually-designed self-supervised tasks are presented, which can be roughly categorized into contrastive learning [12, 5, 4] and masked modeling [7, 11]. Recently, following the well-established contrastive learning and masked modeling paradigms, some self-supervised pre-training methods for time series have been proposed [9, 27, 34, 33, 36, 54, 52, 6].
Contrastive learning. The critical insight of contrastive learning is to optimize the representation space based on the manually designed positive and negative pairs. where representations of positive pairs are optimized to be close to each other. In contrast, negative ones tend to be far apart [48, 14]. The canonical design presented in SimCLR [37] views different augmentations of the same sample as positive pairs and augmentations among different samples as negative pairs.
Recently, in time series pre-training, many designs of positive and negative pairs have been proposed by utilizing the invariant properties of time series. Concretely, to make the representation learning seamlessly related to temporal variations, TimCLR [53] adopts the DTW [25] to generate phase-shift and amplitude-change augmentations, which is more suitable for time series context. TS2Vec [55] splits multiple time series into several patches and further defines the contrastive loss in both instance-wise and patch-wise aspects. TS-TCC [8] presents a new temporal contrastive learning task as making the augmentations predict each other's future. Mixing-up [45] exploits a data augmentation scheme in which new samples are generated by mixing two data samples. LaST [42] aims to disentangle the seasonal-trend representations in the latent space based on variational inference. Afterward, CoST [46] employs contrastive losses in both time and frequency domain to learn discriminative seasonal and trend representations. Besides, TF-C [57] proposes a novel time-frequency consistency architecture and optimizes time-based and frequency-based representations of the same example to be close to each other. Note that contrastive learning mainly focuses on the high-level information [49], and the series-wise or patch-wise representations inherently mismatch the low-level tasks, such as time series forecasting. Thus, in this paper, we focus on the masked modeling paradigm.
Masked modeling. The masked modeling paradigm optimizes the model by learning to reconstruct the masked content from the unmasked part. This paradigm has been widely explored in computer vision and natural language processing, which is to predict the masked words of a sentence [7] and masked patches of an image [2, 11, 50] respectively. As for the time series analysis, TST [56] follows the canonical masked modeling paradigm and learns to predict removed time points based on the remaining time points. Next, PatchTST [26] proposes to predict masked subseries-level patches to capture the local semantic information and reduce memory usage. Ti-MAE [21] uses mask modeling as an auxiliary task to boost the forecasting and classification performances of advanced Transformer-based methods. However, as we stated before, directly masking time series will ruin the essential temporal variations, making the reconstruction too difficult to guide the representation learning. Unlike the direct reconstruction in previous works, SimMTM presents a new masked modeling task, which is reconstructing the original time series from multiple randomly masked series.
2.2 Understanding Masked Modeling
Masked modeling has been explored in stacked denoising autoencoders [40], where masking is viewed as adding noise to the original data and the masked modeling is to project masked data from the neighborhood back to the original manifold, namely denoising. It has recently been widely used in pre-training, which can learn valuable low-level information from data unsupervisedly [49]. Inspired by the manifold perspective, we go beyond the classical denoising process and project the masked data back to the manifold by aggregating multiple masked time series within the neighborhood.
3 SimMTM
As aforementioned, to tackle the problem that randomly masking time series will ruin the essential temporal variation information, SimMTM proposes to reconstruct the original time series from multiple masked time series. To implement this, SimMTM first learns similarities among multiple time series in the series-wise representation space and then aggregates the point-wise representations of these time series based on pre-learned series-wise similarities. Next, we will detail the techniques in both model architecture and pre-training protocol aspects.
The reconstruction process of SimMTM involves the following four modules: masking, representation learning, series-wise similarity learning and point-wise reconstruction.
Figure 2: Architecture of SimMTM, which reconstructs the original time series by adaptive aggregating multiple masked time series based on series-wise similarities learned contrastively from data.
Masking. Given  $\{\mathbf{x}_i\}_{i=1}^N$  as a mini-batch of  $N$  time series samples, where  $\mathbf{x}_i \in \mathbb{R}^{L \times C}$  contains  $L$  time points and  $C$  observed variates, we can easily generate a set of masked series for each sample  $\mathbf{x}_i$  by randomly masking a portion of time points along the temporal dimension, formalizing by:
$$
\{\overline {{\mathbf {x}}} _ {i} ^ {j} \} _ {j = 1} ^ {M} = \operatorname {M a s k} _ {r} (\mathbf {x} _ {i}), \tag {1}
$$
where  $r \in [0,1]$  denotes the masked portion.  $M$  is a hyperparameter for the number of masked time series.  $\overline{\mathbf{x}}_i^j \in \mathbb{R}^{L \times C}$  represents the  $j$ -th masked time series of  $\mathbf{x}_i$ , where the values of masked time points are replaced by zeros. With above process, we can obtain a batch of augmented time series. For clarity, we present all the  $(N(M + 1))$  input series in a set as  $\mathcal{X} = \bigcup_{i=1}^{N} \left( \{\mathbf{x}_i\} \cup \{\overline{\mathbf{x}}_i^j\}_{j=1}^M \right)$ .
Representation learning. After the encoder and projector layer, we can obtain the point-wise representations  $\mathcal{Z}$  and series-wise representations  $S$ , which can be formalized as:
$$
\mathcal {Z} = \bigcup_ {i = 1} ^ {N} \left(\left\{\mathbf {z} _ {i} \right\} \cup \left\{\overline {{\mathbf {z}}} _ {i} ^ {j} \right\} _ {j = 1} ^ {M}\right) = \operatorname {E n o c d e r} (\mathcal {X}), \quad \mathcal {S} = \bigcup_ {i = 1} ^ {N} \left(\left\{\mathbf {s} _ {i} \right\} \cup \left\{\overline {{\mathbf {s}}} _ {i} ^ {j} \right\} _ {j = 1} ^ {M}\right) = \operatorname {P r o j e c t o r} (\mathcal {Z}), \tag {2}
$$
where  $\mathbf{z}_i, \overline{\mathbf{z}}_i^j \in \mathbb{R}^{L \times d_{\mathrm{model}}}$  and  $\mathbf{s}_i, \overline{\mathbf{s}}_i^j \in \mathbb{R}^{1 \times d_{\mathrm{model}}}$ . Encoder( $\cdot$ ) denotes the model encoder, which can project the input data into deep representations and will be transferred to downstream tasks during the fine-tuning process. In this paper, we implement the encoder as a well-acknowledged Transformer [39] and ResNet [13]. As for the Projector( $\cdot$ ), we employ a simple MLP layer along the temporal dimension to obtain series-wise representations. More details can be found in Section 4. Technically, the encoder is applied to input series separately, namely  $\bigcup_{i=1}^{N} (\text{Encoder}(\mathbf{x}_i) \cup \{\text{Encoder}(\overline{\mathbf{x}}_i^j)\}_{j=1}^{M})$ , and so does for projector. Here we adopt the set-style formalization for conciseness.
Series-wise similarity learning. Note that directly averaging multiple masked time series will result in the over-smoothing problem [40], impeding the representation learning. Thus, to precisely reconstruct the original time series, we attempt to utilize the similarities among series-wise representations  $S$  for weighted aggregation, namely exploiting the local structure of the time series manifold. For simplification, we formalize the calculation of series-wise similarities as follows:
$$
\mathbf {R} = \operatorname {S i m} (\mathcal {S}) \in \mathbb {R} ^ {D \times D}, D = N (M + 1), \quad \mathbf {R} _ {\mathbf {u}, \mathbf {v}} = \frac {\mathbf {u v} ^ {\top}}{\| \mathbf {u} \| \| \mathbf {v} \|}, \mathbf {u}, \mathbf {v} \in \mathcal {S}, \tag {3}
$$
where  $\mathbf{R}$  is the matrix of pair-wise similarities for  $(N(M + 1))$  input samples in series-wise representation space, which are measured by the cosine distance.  $\mathbf{R}_{\mathbf{u},\mathbf{v}}$  is the calculated similarity between series-wise representations  $\mathbf{u},\mathbf{v}\in S$ .
Point-wise aggregation. As shown in Figure 2, based on the learned series-wise similarities, the aggregation process for the  $i$ -th original time series is:
$$
\widehat {\mathbf {z}} _ {i} = \sum_ {\mathbf {s} ^ {\prime} \in \mathcal {S} \backslash \{\mathbf {s} _ {i} \}} \frac {\exp \left(\mathbf {R} _ {\mathbf {s} _ {i} , \mathbf {s} ^ {\prime}} / \tau\right)}{\sum_ {\mathbf {s} ^ {\prime \prime} \in \mathcal {S} \backslash \{\mathbf {s} _ {i} \}} \exp \left(\mathbf {R} _ {\mathbf {s} _ {i} , \mathbf {s} ^ {\prime \prime}} / \tau\right)} \mathbf {z} ^ {\prime}, \tag {4}
$$
where  $\mathbf{z}'$  represents the corresponding point-wise representation of  $\mathbf{s}'$ , i.e.  $\mathbf{z}' = \mathrm{Projector}(\mathbf{s}')$ .  $\widehat{\mathbf{z}}_i \in \mathbb{R}^{L \times d_{\mathrm{model}}}$  is the reconstructed point-wise representation.  $\tau$  denotes the temperature hyperparameter
of softmax normalization for series-wise similarities. It is notable that as described in Eq. (4), for each time series  $\mathbf{x}_i$ , the reconstruction is not only based its own masked series  $\{\overline{\mathbf{x}}_i^j\}_{j=1}^M$ . We also introduce other series representations  $\mathcal{S} \backslash \{\mathbf{s}_i\}$  into aggregation, which requires the model to suppress the interference of less-related noise series and precisely learn similar representations for both the masked and the original series, namely guiding the model to learn the manifold structure better. After the decoder, we can obtain the reconstructed original time series, namely
$$
\left\{\widehat {\mathbf {x}} _ {i} \right\} _ {i = 1} ^ {N} = \operatorname {D e c o d e r} \left(\left\{\widehat {\mathbf {z}} _ {i} \right\} _ {i = 1} ^ {N}\right), \tag {5}
$$
where  $\widehat{\mathbf{x}}_i\in \mathbb{R}^{L\times C}$  is the reconstruction to  $\mathbf{x}_i$ .  $\mathrm{Decoder}(\cdot)$  is instantiated as a simple MLP layer along the channel dimension following [50].
3.2 Self-supervised Pre-training
Following the masked modeling paradigm, SimMTM is supervised by a reconstruction loss:
$$
\mathcal {L} _ {\text {r e c o n s t r u c t i o n}} = \sum_ {i = 1} ^ {N} \| \mathbf {x} _ {i} - \widehat {\mathbf {x}} _ {i} \| _ {2} ^ {2}. \tag {6}
$$
Note that the reconstruction process is directly based on the series-wise similarities, while it is hard to guarantee the model captures the precise similarities without explicit constraints in the series-wise representation space. Thus, to avoid trivial aggregation, we also utilize the neighborhood assumption of the time series manifold to calibrate the structure of series-wise representation space  $S$ . For clarity, we formalize the neighborhood assumption by defining positive and negative pairs as follows:
Positive pairs:  $(\mathbf{s}_i,\mathbf{s}_i^+),\mathbf{s}_i^+\in \{\overline{\mathbf{s}}_i^j\}_{j = 1}^M$
Negative pairs:  $(\mathbf{s}_i,\mathbf{s}_i^{-}),\mathbf{s}_i^{-}\in \{\mathbf{s}_k\} \cup \{\overline{\mathbf{s}}_k^j\}_{j = 1}^M,i\neq k$  (7)
where  $\mathbf{s}_i^+$  and  $\mathbf{s}_i^-$  mean the elements that are assumed as close to and far away from  $\mathbf{s}_i$  respectively. Eq. (7) indicates that the original time series and its masked series will present close representations and be far away from the representations from other series in  $S$ . For each series-wise representation  $\mathbf{s} \in S$ , we define the set of its assumed close series as  $S^+ \subset S$ . Note that to avoid the dominating representation, we assume that  $\mathbf{s} \notin S^+$ . With the above formalization, we can define manifold constraint to series-wise representation space as:
$$
\mathcal {L} _ {\text {c o n s t r a i n t}} = - \sum_ {\mathbf {s} \in \mathcal {S}} \left(\sum_ {\mathbf {s} ^ {\prime} \in \mathcal {S} ^ {+}} \log \frac {\exp \left(\mathbf {R} _ {\mathbf {s} , \mathbf {s} ^ {\prime}} / \tau\right)}{\sum_ {\mathbf {s} ^ {\prime \prime} \in \mathcal {S} \backslash \{\mathbf {s} \}} \exp \left(\mathbf {R} _ {\mathbf {s} , \mathbf {s} ^ {\prime \prime}} / \tau\right)}\right), \tag {8}
$$
which can optimize the learned series-wise representations to satisfy the neighborhood assumption in Eq. (7) better. Finally, the overall optimization process of SimMTM can be represented as follows:
$$
\min  _ {\Theta} \mathcal {L} _ {\text {r e c o n s t r u c t i o n}} + \lambda \mathcal {L} _ {\text {c o n s t r a i n t}}, \tag {9}
$$
where  $\Theta$  denotes the set of all parameters of the deep architecture. To trade off the two parts in Eq. (9), we adopt the tuning strategy presented by [17], which can adjust the hyperparameters  $\lambda$  adaptively according to the homoscedastic uncertainty of each loss.
To fully evaluate SimMTM, we conduct experiments on two typical time series analysis tasks: forecasting and classification, covering low-level and high-level representation learning. Further, we present the fine-tuning performance for each task under in- and cross-domain settings.
Benchmarks. We summarize the experiment benchmarks in Table 1, comprising twelve real-world datasets that cover two mainstream time series analysis tasks: time series forecasting and classification. Concretely, we have followed the standard experimental setups in Autoformer [47] for the forecasting tasks and the experiment settings proposed by TF-C [57] for classification.
Implementations. We compare SimMTM with six competitive state-of-the-art baseline methods, including the contrastive learning methods: TF-C [57], CoST [46], TS2Vec [55], LaST [42], the masked modeling method: TiMAE [21], TST [56]. TF-C [57] and TiMAE [21] are the previous best pre-training methods. We experiment on both in- and cross-domain settings. For the in-domain setting, we pre-train and fine-tune the model using the same dataset. As for the cross-domain setting, we pretrain the model on a certain dataset and fine-tune the encoder to different datasets. More details are provided in Appendix A.
Unified encoder. To make a fair comparison, we attempt to unify encoder for these pre
training methods. Concretely, we adopt vanilla Transformer [39] with the channel independence [26] as the unified encoder for forecasting. The channel-independent design allows models to accomplish cross-domain transfer between datasets with different variate numbers. As for the classification, we use 1D-ResNet [13] as the shared encoder following [57]. Notably, for LaST [42] and TFC [57], since their designs are closely related to model structures, we directly report results from their papers or reproduce with official codes. For other baselines and SimMTM, the results in the main text are from the unified encoder. The results from their official papers are also compared in Appendix A.2. For all baselines, the results with a unified encoder generally surpass results reported by themselves.
Table 1: Summary of experiment benchmarks.
Tasks	Datasets	Semantic
Forecasting	ETTh1,ETTh2	Electricity
ETTm1,ETTm2	Electricity
Weather	Weather
Electricity	Electricity
Traffic	Transportation
Classification	SleepEEG	EEG
Epilepsy	EEG
FD-B	Faulty Detection
Gesture	Hand Movement
EMG	Muscle Responses
We summarize the results in forecasting and classification of in- and cross-domain settings in Figure 3. In all these settings, SimMTM outperforms other baselines significantly. It is also notable that although the masking-based method Ti-MAE [21] achieves good performance in the forecasting task (x-axis of Figure 3), it fails in the classification task (y-axis). Besides, contrastive-based methods fail in low-level forecasting tasks. These results indicate that previous methods cannot simultaneously cover high-level and low-level tasks, highlighting the advantages of SimMTM in task generality.
Figure 3: Performance comparison of time series pre-training methods in forecasting  $(MSE\downarrow)$  and classification  $(Acc\uparrow)$  tasks, including both in-domain (left) and cross-domain (right) settings.
4.2 Forecasting
In-domain. As shown in Table 2, empowering by SimMTM pre-training, the model performance is promoted significantly (SimMTM vs. Random init.). Besides, SimMTM consistently outperforms other pre-training methods. On the average of all benchmarks, SimMTM achieves  $8.3\%$  MSE reduction and  $4.3\%$  MAE reduction compared to the advanced masked modeling baseline Ti-MAE [21],