A Transformer-based Framework for Multivariate Time Series Representation Learning
Authors: George Zerveas
Srideepika Jayaraman
Dhaval Patel
Anuradha Bhamidipaty
Carsten Eickhoff
Venue:  (2021)
Keywords: transformer
multivariate time series
unsupervised representation learning
deep learning

Abstract: In this work we propose for the first time a transformer-based framework for unsupervised representation learning of multivariate time series. Pre-trained models can be potentially used for downstream tasks such as regression and classification, forecasting and missing value imputation. We evaluate our models on several benchmark datasets for multivariate time series regression and classification and show that they exceed current state-of-the-art performance, even when the number of training samples is very limited, while at the same time offering computational efficiency. We show that unsupervised pre-training of our transformer models offers a substantial performance benefit over fully supervised learning, even without leveraging additional unlabeled data, i.e., by reusing the same data samples through the unsupervised objective.

Published as a conference paper at ICLR 2021
A TRANSFORMER-BASER FRAMEWORK FOR MULTI- VARIATE TIME SERIES REPRESENTATION LEARNING
Anonymous Authors Paper under double-blind review
ABSTRACT
In this work propose for the first time transformer-based framework for supervised representation learning of multivariate time series. Pre-trained models can be potentially used for downstream tasks such regression and classifica- tion, forecasting and missing value imputation. We evaluate our models on several benchmark datasets for multivariate time series regression and classification and show that they exceed current tfte--of-the-at performance, even when the number of training samples very limited, while at the same time offering computational efficiency. We show that unsupervised pre-training of our transformer models offers a substantial performance benefit over fully supervised learning, even with- out leveraging additional unlabeled data, i.e., by reusing the same data samples through the unsupervised objective.
INTRODUCTION
Multivariate time series (MTS) an important type of data that is ubiquitous in wide variety of domains, including science, medicine, finance, engineering and industrial applications. Despite the recent abundance of MTS data in the much touted era of "Big Data", the availability of labeled data in particular is far more limited: extensive data labeling is often prohibitively expensive or impractical, as it may require much time and effort, special infrastructure or domain expertise. For this reason, in all aforementioned domains there is great interest in methods which can offer high accuracy by using only limited amount of labeled data or by leveraging the existing plethora of unlabeled data.
There is a large variety of modeling approaches for univariate and multivariate time series, with deep learning models recently challenging or replacing the state of the art in tasks such as forecasting, regression and classification (De Brouwer et al. 2019: Tan et al. 2020a Fawaz et al. 2019b). How- ever, unlike in domains such Computer Vision or Natural Language Processing (NLP), the dom- inance of deep learning for time series is far from established: in fact, non-deep learning methods such TS- CHSCH (Shifaz al., 2020). HIVE-COTE (Lines et al.) 2018), and ROCKET (Dempster jet al., (020) currently hold the record on time series regression and classification dataset bench- marks (Tan et al.) 2020a Bagnall et al. (017), matching or even outperforming sophisticated deep architectures such as Inception Time (Fawaz et al. (019a) and ResNet (Fawaz et al. 2019b).
In this work, we investigate, for the first time, the use of transformer encoder for unsupervised representation learning of multivariate time series, as well as for the tasks of time series regression and classification. Transformers are important, recently developed class of deep learning models, which were first proposed for the task of natural language translation (Vaswani et al. (017) but have since come monopolize the state-of-the-un performance across virtually all NLP tasks (Raffel) et al. 2019). A key factor for the widespread success of transformers in NLP is their aptitude for learning how to represent natural language through unsupervised pre-training (Brown et al., 2020) Raffel al. 2019 Devlin et al.) 2018). Besides NLP, transformers have also set the state of the art in several domains of sequence generation, such as polyphonic music composition (Huang et al.) 2018).
Transformer models are based on a multi-headed attention mechanism that offers several key ad- vantages and renders them particularly suitable for time series data (see Appendix section A.4 for details). Inspired by the impressive results attained through unsupervised pre-training of transformer models in NLP. main contribution, in the present work develop generally applicable

---

Published as a conference paper at ICLR 2021
methodology (framework) that can leverage unlabeled data by first training transformer model to extract dense vector representations of multivariate time series through an input denoising (au- toregressive) objective. The pre-trained model can be subsequently applied to several downstream tasks, such as regression, classification, imputation, and forecasting. Here, we apply our framework for the tasks of multivariate time series regression and classification on several public datasets and demonstrate that transformer models can convincingly outperform all current state-of-the-an model- ing approaches, even when only having access to very limited amount of training data samples (on the order of hundreds of samples), an unprecedented success for deep learning models. Importantly, despite common preconceptions about transformers from the domain of NLP, where top performing models have billions of parameters and require days to weeks of pre-training on many parallel GPUs or TPUs, we also demonstrate that our models, using at most hundreds of thousands of parameters, can be trained even on CPUs, while training them on GPUs allows them to be trained as fast as even the fastest and most accurate non-deep learning based approaches.
RELATED WORK
Regression and classification of time series: Currently, non-deep learning methods such as TS- CHIEF (Shifaz et al. 2020)) HIVE-COTE (Lines et al. 2018), and ROCKET (Dempster et al.) 2020) constitute the state of the art for time series regression and classification based on evaluations on public benchmarks (Tan et al. 2020a Bagnall et al. 2017). followed by CNN-based deep archi- tectures such Inception Time Fawaz et al. 2019a) and ResNet (Fawaz et al. 2019b). ROCKET, which on average is the best ranking method, is a fast method that involves trainig a linear classi- fier top of features extracted by a flat collection of numerous and various random convolutional kernels. HIVE-COTE and TS-CHIEF (itself inspired by Proximity Forest (Lucas et al.] 2019 ). very sophisticated methods which incorporate expert insights on time series data and consist of large, heterogeneous ensembles of classifiers utilizing shapelet transformations, elastic similarity measures, spectral features, random interval and dictionary- ods are highly complex, involve significant computational cost, cannot benefit from GPU hardware and scale poorly to datasets with many samples and long time series; moreover, they have been developed for and only been evaluated on univariate time series.
techniques: however, these meth-
Unsupervised learning for multivariate time series: Recent work on unsupervised learning for multivariate time series has predominantly employed autoencoders, trained with an input recon- struction objective and implemented either as Multi-Layer Perceptrons (MLP) or RNN (most com- monly, LSTM) networks. As interesting variations of the former, Kopf et al. (2019) and Fortuin ct al. 2019) additionally incorporated Variational Autoencoding into this approach, but focused on clustering and the visualization of shifting sample topology with time. As an example of the lat- ter, Malhotra et al. (2017) presented a multi-layered RNN peuuencc-to-sequeee
autoencoder, while Lyu al. (2018) 8) developed multi-layered LSTM with an attention mechanism and evaluated both an input reconstruction (autoencoding) as well as a forecasting loss for unsupervised representation learning of Electronic Healthcare Record multivariate time series.
As a novel take on autoencoding, and with the goal of dealing with missing data, Bianchi et al. (2019) employ stacked bidirectional RNN encoder and stacked RNN decoder to reconstruct the imput, and at the same time use a user-provided kernel matrix as prior information to condition internal representations and encourage learning similarity preerttii representations of the input. They evaluate the method on the tasks of missing value imputation and classification of time series under increasing "missingness" of values.
A distinct approach is followed by Zhang et al. (2019), who use a composite convolutional - LLTM network with attention and a loss which aims reconstructing correlation matrices between the variables of the multivariate time series input. They use and evaluate their method only for the task of anomaly detection.
Finally, Jansen et al. (2018) rely triplet loss and the idea of temporal proximity (the loss rewards similarity of representations between proximal segments and penalizes similarity between distal segments of the time series) for unsupervised representation learning of non-speech audio data. This idea is explored further by Franceschi et al.] (2019), who combine the triplet loss with a deep causal dilated CNN, in order make the method effective for very long time series.
2

---

Published as a conference paper at ICLR 2021
Transformer models for time series: Recently, full encoder-decoder transformer architecture was employed for univariate time series forecasting: Li et al. (2019) showed superior performance compared to the classical statistical method ARIMA, the recent matrix factorization method TRMF, an RNN-based autoregressive model (DeepAR) and an RNN-based state space model (DeepState) on 4 public forecasting datasets, while Ww al. (2020) used transformer to forecast influenza prevalence and similarly showed performance benefits compared to ARIMA, an LSTM and a GRU Seq2Seq model with attention, and [im al. (2020) used a transformer for multi-horizon uni- variate forecasting, supporting interpretation of temporal dynamics. Finally, Ma et al. (2019) use an encoder-decoder architecture with variant of self-attention for imputation missing values in mul- tivariate, geo-tagged time series and outperform classic as well as the state-of-the-as RNN-based imputation methods on public and competition datasets for imputation.
By contrast, our work aspires to generalize the use of transformers from solutions to specific gen- erative tasks (which require the full encoder-decoder architecture) to framework which allows for unsupervised pre-training and with minor modifications can be readily used for wide variety of downstream tasks; this is analogous to the way BERT (Devlin et al. 2018)) converted a translation model into generic framework based unsupervised learning, an approach which has become de facto standard and established the dominance of transformers in NLP.
3 METHODOLOGY
3.1 BASE MODEL
At the core of method lies transformer encoder, described in the original transformer work by Vaswani et al. (2017); however, we do not use the decoder part of the architecture. schematic diagram of the generic part of our model, common across all considered tasks, shown in Figure We refer the reader to the original work for a detailed description of the transformer model, and here present the proposed changes that make it compatible with multivariate time series data, instead of sequences of discrete word indices.
In particular, each training sample X _=xm which is a multivariate time series of length w and m different variables, constitutes a sequence of w feature vectors E R" X E R= [x1 X2 .... x... The original feature vectors x; are first normalized (for each dimension, we sub- tract the mean and divide by the variance across the training set samples) and then linearly projected onto a d-dimensional vector space, where the dimension of the transformer model sequence element representations (lypically called model dimension):
U = Wpx, + +bp
(1)
where Wp E Rdxm bp E Rd are learnable parameters and u, RR == 0, ..... w are the model input vectors]] These will become the queries, keys and values of the self-attention layer, after adding the positional encodings and multiplying by the corresponding matrices.
We note that the above formulation also covers the univariate time series case, i.e., m 1, although we only evaluate our approach on multivariate time series in the scope of this work. We additionally note that the input vectors u, need not necessarily be obtained from the (transformed) feature vectors at a time step t: because the computational complexity of the model scales as (ww2) and the number of parameter as (ww) with the input sequence length w, to obtain ut in case the granularity (temporal resolution) of the data is very fine, one may instead use ID-convolutional layer with input and output channels and kernels K; of size (ki m)), where & is the width in number time steps and i the output channel:
U4 = u(t, =
+ -(t, xlt 3 h
1=1,..,,d
(2)
'Although equation I shows the operation for a single time step for clarity, all input vectors are embedded concurrently by a single matrix-matrix multiplication "Specifically, the learnable positional encoding, batch normalization and output layers
3

---

Published as a conference paper at ICLR 2021
X1
82
2,6 R model estimate
R" representation at
Z1
Z2
Zuy
B
Zw
Transformer Encoder
Transformer Encoder
Input encoding
Input encoding
Positional Encoding
Xn
Rdd
X1
X2 X,E IRT
Xw
X X2 2,E RM masked Input
X
Figure 1: Left: Generic model architecture, common all tasks. The feature vector x at each time step / is linearly projected to vector of the same dimensionality d as the internal representation vectors of the model and is fed to the first self-attention layer form the keys, queries and values after adding positional encoding. Right: Training setup of the unsupervised pre-training task. each variable sequence in the input independently. such that across each
We mask a proportion variable, time segments of mean length im masked, each followed by unmasked segment of mean length L, I- Using linear layer on top of the final vector representations zt, at each time step the model tries to predict the full, uncorrupted input vectors xe; however, only the predictions on the masked values are considered in the Mean Squared Error loss.
In this way, one may control the temporal resolution by using stride or dilation factor greater than Moreover, although in the present work we only used equation [ one may use equation 2 an input to compute the keys and queries and equation Io compute the values of the self-attention layer. This is particularly useful in the case of univariate time series, where self-attention would otherwise match (consider relevant/compatibe)) all time steps which share similar values for the independent variable, as noted by et (2019).
Finally, since the transformer is feed-forwer forward architecture that is insensitive the ordering of in- put, in order to make aware of the sequential nature of the time series, we add positional encodings Wpos Raxd the input vectors U IR" xx
[u,,-.. .... uw]: U U W pos-
Instead of deterministic, sinusoidal encodings, which were originally proposed by Vaswani et al. (2017).
fully learnable positional encodings, we observed that they perform better for all datasets presented in this work. Based on the performance of our models, we also observe that the positional encodings generally appear
significantly interfere with the numerical information of the time series, similar to the case of word embeddings; we hypothesize that this is because they are learned so to occupy a different, approximately orthogonal, subspace to the one which the projected time series samples reside. This approximate orthogonality condition is much easier to satisfy in high dimensional spaces.
An important consideration regarding time series data is that individual samples may display con- siderable variation length. This issue effectively dealt with our framework: after setting maximum sequence length for the entire dataset, shorter samples are padded with arbitrary values, and we generate padding mask which adds a large negative value to the attention scores for the padded positions, before computing the self-attention distribution with the softmax function. This forces the model to completely ignore padded positions, while allowing the parallel processing of samples in large minibatches.
4

---

Published as a conference paper at ICLR 2021
Transformers in NLP use layer normalization after computing self-attention and after the feed- forward part of each encoder block, leading to significant performance gains over batch normaliza- tion, originally proposed by Vaswani al. (017). However, here instead use batch normaliza- tion, because it can mitigate the effect of outlier values in time series, an issue that does not arise in NLP word embeddings. Additionally, the inferior performance of batch normalization in NLP has been mainly attributed to extreme variation in sample length (i.c., sentences in most tasks) (Shen) et al. 2020), while in the datasets we examine this variation is much smaller. In Table [ of the Appendix we show that batch normalization can indeed offer very significant performance benefit over layer normalization, while the extent can vary depending on dataset characteristics.
3.2 REGRESSION AND CLASSIFICATION
The base model architecture presented in Section 3.1 and depicted in Figure T can be used for the purposes of regression and classification with the following modification: the final representation vectors 2, E R corresponding to all time steps are concatenated into a single vector Z E R Zw], which serves the input to linear output layer with parameters W. E R=x(d=)), bo E R", where n is the number of scalars to be estimated for the regression problem (typically 1), or the number of classes for the classification problem:
8 W. + bo
(3)
In the case of regression, the loss for single data sample will simply be the squared error L || - -yll,, where y E Rd are the ground truth values. We clarify that regression in the context of this work means predicting a numeric value for a given sequence (time series sample). This numeric value is of different nature than the numerical data appearing in the time series: for example, given a sequence of simultaneous temperature and humidity measurements of rooms in a house, as well as weather and climate data such temperature, pressure, humidity, wind speed, visibility and dewpoint, we wish to predict the total energy consumption in kWh of a house for that day. The parameter n corresponds to the number of scalars (or the dimensionality of a vector) be estimated.
In the case of classification, the predictions y will additionally be passed through a softmax function to obtain distribution over classes, and its cross-entropy with the categorical ground truth labels will be the sample loss.
Finally, when fine-tuning the pre-trained models, allow training of all weights; instead, freezing all layers except for the output layer would be equivalent to using static, pre-extracted time-series representations of the time series. In Table 12] in the Appendix we show the trade-off in terms of speed and performance when using fully trainable model versus static representations.
3.3 UNSUPERVISED -RAINIGIN
As a task for the unsupervised pre-training of our model we consider the autoregressive task of de- noising the input: specifically, we set part of the input to 0 and ask the model to predict the masked values. The corresponding setup is depicted in the right part of Figure [] A binary noise mask M E _ is created independently for each training sample, and the input is masked by elemen- twise multiplication: X M X. On average, a proportion of each mask column of length (corresponding to a single variable in the multivariate time series) is set to 0 by alternating between segments of Os and Is. We choose the state transition probabilities such that each masked segment (sequence of Os) has a length that follows a geometric distribution with mean in and is succeeded by an unmasked segment (sequence of 1s) of mean length 4, lm We chose 4m for all pre- sented experiments. The reason why we wish to control the length of the masked sequence, instead of simply using a Bernoulli distribution with parameter to set all mask elements independently at random, is that very short masked sequences (e.g.. of 1 masked element) in the input can often be trivially predicted with good approximation by replicating the immediately preceding or succeeding values or by the average thereof. In order to obtain enough long masked sequences with relatively high likelihood, a very high masking proportion r would be required, which would render the over- all task detrimentally challenging. Following the process above, each time step average m variables will be masked. We empirically chose r = 0.15 for all presented experiments. This input masking process is different from the "cloze type" masking used by NLP models such BERT,
5

---

Published as a conference paper at ICLR 2021
where a special token and thus word embedding vector replaces the original word embedding, i.e., the entire feature vector at affected time steps. We chose this masking pattern because it encour- ages the model to learn to attend both preceding and succeeding segments in individual variables, as well as to existing contemporary values of the other variables in the time series, and thereby to learn to model deteeddependencsss between variables. In Table TO the Appendix we show that this masking scheme is more effective than other possibilities for denoising the input.
Using a linear layer with parameters W. Rmxdd bo R on top of the final vector repre- sentations Zt. E IE,, for each time step the model concurrently outputs its estimate & of the full, uncorrupted input vectors x however, only the predictions on the masked values (with indices in the set M = {(t,t) my, = 0}, where ma, are the elements of the mask M), are considered in the Mean Squared Error loss for each data sample:
8t = Wo7t bo
(4)
MSE
ΣΣO ($(t,)) - xxt,
(5)
M
(LAEMM
This objective differs from the one used by denoising autoencoders, where the loss considers re- construction of the entire input, under (typically Gaussian) noise corruption. Also, we note that the approach described above differs from simple dropout on the input embeddings, both with respect to the statistical distributions of masked values, well as the fact that here the masks also determine the loss function. In fact, we additionally use dropout of 10% when training all of our supervised and unsupervised models.
EXPERIMENTS & RESULTS
In the experiments reported below we use the predefined training test set splits of the benchmark datasets and train all models long enough to ensure convergence. We do this to account for the fact that training the transformer models in a fully supervised way typically requires more epochs than fine-tuning the ones which have already been pre-trained using the unsupervised methodology of Section 3.3 Because the benchmark datasets are very heterogeneous in terms of number of samples, dimensionality and length of the time series, as well as the nature of the data itself, observed that we can obtain better performance by cursory tuning of hyperparameters (such as the number of encoder blocks, the representation dimension, number of attention heads or dimension of the feed- forward part of the encoder blocks) separately for each dataset. To select hyperparameters, for each dataset we randomly split the training set in two parts, -0%, and used the 20% as a validation set for hyperparameter tuning. After fixing the hyperparameters. the entire training set was used to train the model again, which was finally evaluated on the test set. A set of hyperparameters which has consistently good performance on all datasets is shown in Table [4 in the Appendix, alongside the hyperparameters that we have found to yield the best performance for each dataset (Tables
16
18
4.1 REGRESSION
We select a diverse range of datasets from the Monash University, UEA, UCR Time Series Regres- sion Archive TTnn et al. (2020a) in a way so as to ensure diversity with respect to the dimensionality and length of time series samples, well as the number of samples (see Appendix Table for dataset characteristics). Table I shows the Root Mean Squared Error achieved by of our models, named TST for "Time Series Transformer", including a variant trained only through supervision, and one first pre-trained on the same training set in an unsupervised way. We compare them with the currently best performing models as reported in the archive. Our transformer models rank first on all but two of the examined datasets, for which they rank second. They thus achieve an average rank of 1.33, setting them clearly apart from all other models; the overall second best model, XG- Boost, has average rank of 3.5, ROCKET (which outperformed ours ranks in 5.67th place and Inception (which outperformed ours on the second dataset) also has an average rank of 5.67. On average, our models attain 30% lower RMSE than the mean RMSE among
dataset) on average

---

Published as a conference paper at ICLR 2021
all models, and approx. 16% lower RMSE than the overall second best model (XGBoost), with ab- solute improvements varying among datasets from approx. 4% to 36%. We note that all other deep learning methods achieve performance close to the middle of the ranking lower. In Table [ we report the "average relative difference from mean" metric ry for each model j. the over N datasets:
R(i,)) Rr R
1 M
M
r
R.
R(i
where j) is the RMSE of model j on dataset i and M is the number of models.
Importantly, also observe that the pre-trained transformer models outperform the fully super- vised ones in 3 out of datasets. This is interesting, because no additional samples are used for pre-training: the benefit appears to originate from reusing the same training samples for learning through an unsupervised objective. To further elucidate this observation, we investigate the follow- ing questions:
Q1: Given a partially labeled dataset of a certain size, how will additional labels affect perfor- mance? This pertains to one of the most important decisions that data owners face, namely, to what extent will further annotation help. To clearly demonstrate this effect, we choose the largest dataset we have considered from the regression archive (12.5k samples). in order to avoid the variance in- troduced by small set sizes. The left panel of Figure 2 (where each marker is an experiment) shows how performance on the entire test set varies with an increasing proportion of labeled training set data used for supervised learning. As expected, with an increasing proportion of available labels performance improves both for a fully supervised model, as well as the same model that has been first pre-trained on the entire training set through the unsupervised objective and then fine-tuned. In- terestingly, not only does the rre-trained model outperform the fully supervised one, but the benefit persists throughout the entire range of label availability, even when the models are allowed to all labels; this is consistent with our previous observation on Table TableeTregee regarding the advantage of reusing samples.
Q2: Given a labeled dataset, how will additional unlabeled samples affect performance? In other words, to what extent does unsupervised learning make it worth collecting more data, even if no additional annotations are available? This question differs from the above, as we now only scale the availability of data samples for unsupervised pre-training, while the number of labeled samples is fixed. The right panel of Figure 2 (where each marker is an experiment) shows that, for a given number of labels (shown as percentage of the totally available labels), the more data samples are used for unsupervised learning, the lower the error achieved (note that the horizontal axis value corresponds to fully supervised training only, while all other values to unsupervised pre-training followed by supervised fine-tuning) This trend is more linear in the case of supervised learning on 20% of the labels (approx. 2500). Likely due to a small sample (here, meaning set) effect, in the case of having only 10% of the labels (approx. 1250) for supervised learning, the error first decreases rapidly as we use more samples for unsupervised pre-training, and then momentarily increases, before it decreases again (for clarity, the same graphs shown separately in Figure in the Appendix). Consistent with our observations above, it is interesting to again note that, for given number of labeled samples, even reusing a subset of the same samples for unsupervised pre- training improves performance: for the 1250 labels (blue diamonds of the right panel of Figure 2or left panel of Figure the Appendix) this can be observed in the horizontal axis range [0, 0.1), and for the 2500 labels (blue diamonds of the right panel of Figure 2 or right panel of Figure 3 in the Appendix) in the horizontal axis range (0, 0.2].
4.2 CLASSIFICATION
We select set of 11 multivariate datasets from the UEA Time Series Classification Archive (Bag- nall et al. 2018) with diverse characteristics in terms of the number, dimensionality and length of time series samples, as well as the number of classes (see Appendix Table As this archive is new, there have not been many reported model evaluations; we follow Franceschi et al. (2019) and use as a baseline the best performing method studied by the creators of the archive, DTWp WD (dimension- Dependent DTW). together with the method proposed by Franceschi et al. (019) themselves (a dilation-CNN leveraging unsupervised and supervised learning.. Additionally, we use the publicly available implementations Tan et al. (2020b) of ROCKET, which is currently the top performing
7

---

Published as a conference paper at ICLR 2021
68
66
64
RMSE
RMSE
60
64
58
62
Unsup pretrained
la
54
Supervised
60
0.2 of labels
52
0.2
0.4
0.8
0
0.2
0.4 0.6
0.8
1
Availability of labels
Use of data for unsupervised pre-training
Figure 2: Dataset: BeijinPM22SQulity. Left: Root Mean Squared Error of fully supervised transformer (orange circles) and the same model pre-trained (blue diamonds) the training set through the unsupervised objective and then fine-tuned on available labels, versus the proportion of labeled data in the training set. Right: Root Mean Squared Error of a given model the number of samples (here, shown as a proportion of the total number of samples in the training set) used for unsupervised pre-training. For supervised learning. two levels of label availability are depicted: 10% (purple circles) and 20% (green squares) of all training data labels. Note that horizontal axis value of 0 means fully supervised learning only, while all other values correspond to unsupervised pre-training followed by supervised fine-tuning.
function of
Root MSE
Ours
Dutuset
SVR
Random "Forest
XGBoost
I-NN-ED
-ED
DTWD
DTWD
Rocket
FCN
ResNet
Inception
(sup. only)
(pretrained)
Appliincesénerg
3,457
3.455
3.489
5.231
4,227
6,036
4,019
2.299
2.865
3,065
4,435
2.228
2.375
BenaeneConcentt.
0.855
4.983
4,061
1.584
0.517
BeijingPM10
110.574
94.072
93.138
139.229
115.669
139.134
115.502
120.057
94.348
95.489
96.749
91.344
86.866
BeijingMME
75.734
63.301
59.495
88. 195
74.156
88. 88.256
72.717
62.769
59.726
64.462
62.227
61.357
53.492
LiveFuelMoisture
43.021
11.657
41.295
58.238
46.331
57.111
46,290
41,829
47.877
51,632
51,539
42.607
43.138
TEEEPPG
31,487
33.208
37.140
33.572
36.515
33.150
25.042
Avg Rel. diff, from
0.087
0.172
00197
0,377
0.132
0.355
0.121
40,018
0.021
0.005
-0.008
-0,301
-0,303
Aug Rank
4.
10333
11.167
3.007
6.167
6.333
5,606
Table 1: Performance on multivariate regression datasets, in terms of Root Mean Squared Error. Bold indicates best values, underlining indicates second best.
model for univariate time series and one of the best in our regression evaluation, and XGBoost, which one of the most commonly used models for univariate and multivariate time series, and also the best baseline model in our regression evaluation (Section 4.1p. Finally, did not find any reported evaluations of RNN-based based models on any of the UCR/UEA archives, possibly because of common perception for long training and inference times, as well as difficulty in training (Fawaz) et al. 2019b); therefore, we implemented a stacked LSTT model and also include it in the compar- ison. The performance of the basclines alongside our own models are shown in Table in terms of accuracy, to allow comparison with reported values.
It be seen that models performed best out of the 11 datasets, achieving an average rank of 1.7, followed by ROCKET, which performed best on 3 datasets and on average ranked 2.3th. The dilation-CNN (Franceschi et al., 2019) and XGBoost, which performed best on the remaining dataset, tied and on average ranked 3.7th and 3.8th respectively. Interestingly, we observe that all datasets on which ROCKET outperformed our model were very low dimensional (specifically, 3-dimensional). Although our models still achieved the second best performance for UWaveG- estureLibrary, in general we believe that this indicates a relative weakness of our current models when dealing with very low dimensional time series. As discussed in Section 3.1 this may be due to the problems introduced by a low-dimensional representation space to the attention mecha- nism, well as the added positional embeddings; mitigate this issue, in future work intend to use a cocovvolutional layer to extract more meaningful representations of low-dimensional in- put features (see Section 3.1). Conversely, our models performed particularly well on very high-
8

---

Published as a conference paper at ICLR 2021
Ours
Dataset
TST (pretrained)
TST (sup. only)
Rocket
XGBoost
LSTM
Frans. et al
DTW D
EthanolConcentratiion
0.326
0.337
0,452
0.437
0.323
0.289
0.323
FaceDetection
0.689
0.681
0.647
0.633
0.577
0.528
0.529
Handwriting
0.359
0.305
0,588
0.158
0.152
0.533
0.286
Heartbeat
0.776
0.776
0.756
0,732
0.722
0.756
0.717
Japanese Vowels
0,997
0.994
0,962
0.865
0.797
0.989
0.949
Bes
0.687
0.684
0,369
0.176
0.16
PEMS SF
0.896
0.919
0.751
0,983
0.399
0,688
0.711
SelfRegulationSCPP
0.922
0.925
0,908
0,846
0.689
0.846
0.775
SelfRegulation
0.604
0.589
0.533
0.489
0.466
0.556
0.539
SpokcnArrbiiDiigtss
0.998
0.993
0.712
0.696
0.319
0.956
0.963
UWaveGesturel.. iibrary
0.913
0.903
0.944
0.759
0.412
0.884
0.903
Avg Accuracy
0.748
0.742
0.725
0.659
0.486
0.703
0.669
(excl. InsectWingBeat)
Avg Rank
1.7
2.3
3.8
5,4
3.7
4.1
Table 2: Accuracy on multivariate classification datasets. Bold indicates best and underlining second best values. dash indicates that the corresponding method failed run on this dataset.
dimensional datasets (FaceDetection, HeartBeat, InneetWingBeat, PEMS-SF), and/or datasets with relatively more training samples. As a characteristic example, on InsectWingBeat (which is by far the largest dataset with 30k samples and contains time series of 200 dimensions and highly irregular length) our model reached an accuracy of 0.689, while all other methods performed very poorly the second best was XGBoost with accuracy of 0.369, However, we note that our model performed exceptionally well also on datasets with only a couple of hundred samples, which in fact constitute out of the examined datasets.
Finally, we observe that the pre-trained transformer models performed better than the fully super- vised ones in 8 out of 11 datasets, sometimes by substantial Baargnn.Again, no additional samples were available for unsupervised pre-training, so the benefit appears to originate from reusing the same samples.
CONCLUSION
In this work propose novel framework for multivariate time series representation learning based on the transformer encoder architecture. The framework includes an unsupervised pre-training scheme, which show that offer substantial performance benefits over fully supervised learn- ing, even without leveraging additional unlabeled data, i.c., by reusing the same data samples. By evaluating our framework on several public multivariate time series datasets from various domains and with diverse characteristics, we demonstrate that it is currently the best performing method for regression and classification, even for datasets where only a few hundred training samples are available.
REFERENCES
A. Bagnall, J. Lines, A. Bostrom, J. Large, and E. Keogh. The great time series classification bake off: review and experimental evaluation of recent algorithmic advances. Data Mining and Knowledge Discovery, 31:606-660 66066-6606660 2017.
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The UEA multivariate time series classification archive, 2018. arXiv: 081188110007 [cs, stat], October 2018. arXiv: 1811.00075.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The Long- Transformer. xrxiv:2004.05 [cs], April 2020. URL http:://xxxvvvvo wttp:/o///bbb//////// 05500000000 arXiv: 2004.05150.

---

Published as a conference paper at ICLR 2021
Filippo Maria Bianchi, Lorenzo Livi, Karl Øyvind Mikalsen, Michael Kampffineyer, and Robert Jenssen. Learning representations of multivariate time series with missing data. Pattern Recognition, 96:106973, December 2019. ISSN 0031-3203. doi: 2000:///.atcog.22111099999
Brown, B. Mann, Nick Ryder, Melanie Subbiah, Kaplan, P. Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- G. Krüger, Tom Henighan, R. Child, Aditya Ramesh, D. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, E. Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, J. Clark, Christopher Berner, Sam McCandlish, A. Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020.
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime Carbonell, Quoe V. Le, and Ruslan Salakhutdinov. Transformer-XL XL: Attentive Language Models Beyond a Fixed- Context. arXiv:1900 1900.9901.000 .02860 [cs, stat], June 2019. URLhets /tttp:://arxi...o org/
www.oo/ggg//aa/eeeeee.eeeeeeeeee abs/ 1901. 02860 arXiv: 1901.02860.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE-Bayes Continuous Modeling of Sporadically-Ohsee Observed Time Series. In Wallach, Larochelle, A. Beygelzimer, F. d' texquotessiggl Alché-Buc Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 7379-7390. Curran Associates, Inc., 2019.
Angus Dempster, Franccois Petitjean, and Geoffrey I. Webb, ROCKET: exceptionally fast and accu- rate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 2020. doi: 10. 101100000101888 020-0000000022
Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR, 110.08800,,485, 2018. eprint: 1810.04805.
H. Fawaz, B. Lucas, G. Forestier, Charlotte Pelletier, D. Schmidt, Jonathan Weber, Geoffrey Webb, L. Idoumghar, Pierre-Alain Muller, and Franccois Petitjean. Inception Time: Finding AlexNet for Time Series Classification. ArXiv, 2019a. doi: 10. 10100//00001-000000001011
Hassan Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. Deep learning for time series classification: a review. Data Mining and Knowledge Discovery, 33 (4):917-963, July 2019b. ISSN 1573-756X. doi: 10. 1...1007/s0018-01111 019-00619-11
Vincent Fortuin, M. Hüser, Francesco Locatello, Heiko Strathmann, and G. Rätsch. SOM-VAE. Interpretable Discrete Representation Learning Time Series. ICLR, 2019.
Jean- Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised Scalable Represen- tation Learning for Multivariate Time Series. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' texquuotesiige Alché-Buu E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems 32, pp. 4650-4661. Curran Associates, Inc., 2019.
Sepp Hochreiter. The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions. Int. J. Uncertain. Fuzziness Knowl.-Based Syst., 6(2): 16, April 1998. ISSN 0218-4885. doi: 10. World Scientific Publishing Co., Inc
Place: River Edge, NJ, USA Publisher:
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Mathew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. In International Conference Learning Representations, 2018
A. Jansen, M. Plakal, Ratheet Pandya, D. Ellis, Shawn Hershey, Jiayang Liu, R. C. Moore, and R. A. Saurous. Unsupervised Learning of Semantic Audio Representations. 2018 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), 2018. doi: 10. 109/SSP/C.S 2018,8461684.
A. Kopf, Vincent Fortuin, Vignesh Ram Somnath, and M. Claassen. Mixture-of -fxtert- Variational Autoencoder for clustering and generating from similarity-bart based representations. ICLR 2019, 2019.
10

---

Published as a conference paper at ICLR 2021
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In Advances in Neural Information Processing Systems, pp. 5243-5253, 2019.
Bryan Lim, Sercan O. Arik, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. 2020.
J. Lines, Sarah Taylor, and Anthony J. Bagnall. Time Series Classification with HIVB-COTE. ACM Trans. Knowl. Discov. Data, 2018. doi: 10.1145/3182382.
Benjamin Lucas, Ahmed Shifaz, Charlotte Pelletier, Lachlan O'Neill Nayyar Zaidi, Bart Goethals, Francois Petitjean, and Geoffrey Webb. Proximity Forest: An effective and scalable distance- based classifier for time series. Data Mining and Knowledge Discovery, 33(3):607-633 May 2019. ISSN 1384-5810, 1573-756X. doi: 10.10077/10618-00990000
arXiv: 1808. 10594.
Xinrui Lyu, Matthias Hueser, Stephanie L. Hyland, George Zerveas, and Gunnar Raetsch. Improving Clinical Predictions through Unsupervised Time Series Representation Learning, Proceedings of the NeurlPS 2018 Workshop Machine Learning for Health, 2018. -eprint: 1812.00490.
Ma, Zheng Shou, Alireza Zareian, Hassan Mansour, A. Vetro, and S. Chang. Cdsa: Cross-dimensions. self-attention for multivariate, geo-tagged time series imputation. ArXiv, 0tss/995500994,, 2019.
Malhotra, T. Vishnu, L. Vig, Puncet Agarwal, and G. Shroff. TimeNet: Pre-trained deep recurrent neural network for time series classification. ESANN, 2017.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on International Conference on Machine Learning Volume 28, ICML' pp. I11-1310-111-111, Atlanta, GA, USA, June 2013. JMLR.org.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to- Text Transformer. ArXiv, abs//991..1063. 2019.
Sheng Shen, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. PowerNorm: Rethinking Batch Normalization in Transformers. arXiv:2003.07845 [cs], June 2020. arXiv: 2003.07845.
Ahmed Shifaz, Charlotte Pelletier, F. Petitjean, and Geoffrey I. Webb. TS-CHIEF: scalable and accurate forest algorithm for time series classification. Data Mining and Knowledge Discovery, 2020. doi: 10.1// 00018000-00079-888
C. Tan, C. Bergmeir, François Petitjean, and Geoffrey I. Webb. Monash University, UEA, UCR Time Series Regression Archive. ArXiv, 2020a.
Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey Webb. Time series regres- sion. arXiv preprint arXiv:2006. 12672, 2020b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5998 $998-6008. Curran Associates, Inc., 2017.
Neo Wu, Bradley Green, Xue Ben, and Shawn O' O'Banoo.. Deep transformer models for time series forecasting: The influenza prevalence case, 2020.
Chuxu Zhang, Dongjin Song, Yuncong Chen, Xinyang Feng, C. Lumezanu, Wei Cheng, Jingchao Ni, B. Zong, H. Chen, and Nitesh V. Chawla. A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data. In AAAI, 2019. doi: 10. 1009/aaa.. v33101 3301 1409.
11

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 ADDITIONAL. POINTS & FUTURE WORK
Execution time for training: While precise comparison in terms of training time is well out of scope for the present work, Section A.3 of the Appendix we demonstrate that our transformer- based method is economical in terms of its use of computational resources. However, alternative self-attention schemes, such as sparse attention patterns (L.i et al.) 2019), recurrence (Dai et al.) 2019) compressed (global-local) attention (Beltagy al., 2020), can help drastically reduce the (w complexity of the self-attention layers with respect to the time series length w, which the main performance bottleneck.
The model and training process described in Section 3.3 is exactly the setup required to perform imputation of missing values, without any modifications, and observed that it was possible to achieve very good results following this method; as a rough indication, our models could reach Root Mean Square Errors very close to when asked perform the input denoising (autoregressive) task on the test set, after being subjected to unsupervised pre-training on the training set. We also show example results of imputation one of the datasets presented in this work in 5 However, we defer a systematic quantitative comparison with the state of the art to future work. Furthermore, we note that one may simply use different patterns of masking achieve different objectives, while the rest of the model and setup remain the same. For example, using mask which conceals the last part of variables simultaneously, one may perform forecasting (see Figure 4 in Appendix), while for longer time series one may additionally perform this process within sliding window. Again, we defer systematic investigation to future work.
Imputation and forecasting:
Extracted representations: The representations extracted by the transformer models can be used directly for evaluating similarity between time series, clustering, visualization and any other use cases where time series representations are used in practice. A valuable benefit offered by transformers is that representations can be independently addressed for each time step; this means that, for example, greater weight can be placed at the beginning, middle end of the time series, which allows to selectively compare time series, visualize temporal evolution of samples etc.
Dataset AppliancesEnergy
Train Size
Test Size
Length
Dimension
Missing Values No
96
42
144
24
BenzeneConcemtratioo
3433
5445
240
Yes
BeijingPMI0Quil
12432
5100
24
Yes
BeijingPM25Quality
12432
5100
24
Yes
LiveFluelMoistreCConen
3493
1510
365
No
JEEEPPG
1768
1328
1000
No
Table 3: Multivariate Regression Datasets
Dataset
TrainSize
TestSize
NumDimensiios
LenitsLength
NumClasses
SchaanoConcennraatin
261
263
3
I751
4
FaceDetection
5890
3524
144
62
2
Handwriting
150
850
3
152
26
Heartbeat
204
205
61
405
2
InsectWingbeat
30000
20000
200
30
10
Japanese Vowels
270
370
29
9
PEMS SF
267
173
963
144
7
SelfRegulationSCCPI
268
293
896
2
SelfRegulationSCP2
200
180
7
1152
2
SpokenArabiiDiigits
6599
2199
10
UWaveGestureLLbbrayy
120
320
3
315
8
Table 4: Multivariate Classification Datasets
12

---

Published as a conference paper at ICLR 2021
Standard deviation
Dataset
Supervised TST
Pre-trained TST
AppliancesEnergy
0.240
0.163
BenzzeneConceettration
0.031
0.092
I0quaityMiQQuality
0.689
0.813
BeijinPPM2SQulitty
0.189
0.253
LiveFuclMoisurrCContnn 0.735
0.013
TEEEPPG
1.079
1.607
Table 5: Standard deviation of the Root Mean Square Error displayed by the Time Series Trans- former models on multivariate regression datasets
A.2 CRITERIA FOR DATASET SELECTION
We select diverge range datasets from the Monash University, UEA, UCR Time Series Regres- sion and Classification Archives, in way and length of time series samples, as well as the number samples. Additionally, we have tried to include both "easy" and "difficult" datasets (where the baselines perform very well less well). In the following we provide a more detailed rationale for each of the selected multivariate datasets.
ensure diversity with respect to the dimensionality
EthanolConcentrationn very low dimensional, very few samples, moderate length, large number of classes, challenging
FaceDetection:
very high dimensional, many samples, very short length, minimum number of
classes
Handwriting: very low dimensional, very few samples, moderate length, large number of classes
Heartbeat: high dimensional, very few samples, moderate length, minimum number of classes
Japanese Vowels: very heterogeneous sample length, moderate num. dimensions, very few samples, very short length, moderate number of classes, all baselines perform well
InsgetWingBeat: very high dimensional, many samples, very short length, moderate number of classes, very challenging
PEMS-SF: extremely high dimensional, very few samples, moderate length, moderate number of classes
SelfRegulationSCPII Few dimensions, very few samples, long length, minimum number of classes, baselines perform well
SelfRegalationSCPP: similar to SeliRegulationSCPII, but challenging
SpokenArabieDigits: Moderate number of dimensions, many samples, very heterogeneous length, moderate number of classes, most baselines perform well
UWaveGestureLibbaayy: very low dimensional, very few samples, moderate length, moderate num- ber of classes, baselines perform well
A.3 EXECUTION TIME
We recorded the times required for training our fully supervised models until convergence on GPU, as well as for the currently fastest and top performing (in terms of classification accuracy and regression error) baseline methods, ROCKET and XGBoost on a CPU. These have been shown be orders of magnitude faster than methods such as TS-CHIEF, Proximity Forest, Elastic Ensembles, DTW and HIVE-COTE, but also deep learning based methods Dempster et al. (2020). Although XGBoost and ROCKET are incomparably faster than the transformer
CPU, can be seen in Table 7 in the Appendix, exploiting commercial GPUs and the parallel processing capabilities of a transformer typically enables as fast (and sometimes faster) training times these (currently fastest available) methods. In practice, despite allowing for many hundreds of epochs, using a GPU we never trained our models longer than hours any of the examined datasets.
13

---

Published as a conference paper at ICLR 2021
Standard deviation
Dataset
Supervised TST
Pre-trained TST
EdhanolConcentraation
0.024
0.002
FaceDetection
0.007
0.006
Handwriting
0.020
0.006
Heartbeat
0.018
0.018
Insect InseetWingbeat
0.003
0.026
Japanese Vowels
0.000
0.0016
PEMS-SP
0.017
0.003
SelfRegulationSCPPI
0.005
0.006
SelfRegulationSCP2
0.020
0.003
SpokenArabicDigiit
0,0003
0.001
UWaveGestureLibrary 0.005
0.003
Table 6: Standard deviation of accuracy displayed by the Time Series Transformer models on mul-
tivariate classification datasets
Dataset
Rocket
XGBoost
TST (GPU)
Concettaoniiniiiionnnnn
41.937
3.760
34.72
FaceDetection
279.033
57.832
67.8
Handwriting
6.705
1.836
134.4
Heartbeat
35.825
3.013
2.57
InsectWingBeat
64.883
4565
Japanese Vowels
5.032
0.527
4.71
PEMS-SF
369.198
150.879
341
SelfRegulationSCPI
30.578
0.967
3.46
SelfRegulationSCPP2
28.286
1.213
97.3
SiokenArrabieDiggtss 65.143
3.129
73.2
UWaveGestureLibrary 3,078
0.636
2.90
Table 7: Total training time (time until maximum accuracy is recorded) in seconds: for the fastest currently available methods (Rocket, XGBoost) on the same CPU, as well as for our fully supervised transformer models GPU. On CPU, training for our model is typically at least order of magnitude slower.
As regards deep learning models, LSTMs are well known be slow, as they require O(w) sequential operations (where w is the length of the time series) for each sample, with the complexity per layer scaling as O(N d²), where the internal representation dimension (hidden state size). We refer the reader to the original transformer paper (Vaswani et al. (017)) for detailed discussion about how tranformers compare to Convolutional Neural Networks in terms of computational efficiency.
Ranks
Dulasel Name
TST
SVR
Random Forest
XGBoost
I-NN
S-NN
-ED
I-NN- DTWD
DTWD
5-NN-
Rocket
FCN
ResNet
Inception
Rel diff from
2nd
Appliancestinery
11
9
12
8
2
4
10
-0.661
DenaeneeCuncetzattion
.0.255
BejjigPMM0QQulity
7
12
9
11
10
5
6
-0.067
PM250ulyM
12
4
-0.101
Livetue/MoisurCConeee
2
3
12
7
11
8
10
9
-0.038
JEEEPPG
12
Table Relative ranks of all methods on Regression datasets. Last column indicates relative diff of method from the 2nd best, whenever the rank is 1.
14

---

Published as a conference paper at ICLR 2021
Problem
TST
Rocket
XGBoost
LSTM
Franseschi al
DTW D
Rel. Diff from 2nd best
EthanolConcentrratee
2
3
DeteCitttt
3
4
6
-0.041
Handwriting
5
6
Hcartbeat
2
4
5
-0.02
Japanese Vowels
5
6
-0.008
PEMS-SF
2
I
6
5
SellRegulationSCPI
3
6
-0.017
SellRegulationSCP2
4
5
6
2
-0.055
SpocenArraicDDiiiss
5
-0.035
UWaaveeestureLibbaay
2
1
5
6
4
Table 9: Relative ranks of methods on the Classification datasets. The last column is the relative diff' of our method from the 2nd best, whenever we rank Ist.
Dataset
Task (Metric)
Sep., Bern.
Sync., Bern.
Sep., Stateful
Sync., Stateful
Heartheat
Classif. (Accuracy)
0.761
0.756
0.776
0.751
InngeeWingbeat
Classif. (Accuracy)
0.641
0.632
0.687
0.689
SpokenAAraiiDiigisss
Classif. (Accuracy)
0.994
0.994
0.998
0.996
PEMS-SF
Classif. (Accuracy)
0.873
0.879
0.896
0.879
BenzzenCConccenrattio
Regress. (RMSE)
0.681
0.493
0.494
0.684
BejjingPM25Quality LiveFucelMostuueCCnneett
Regress, (RMSE)
57.241
59.529
53.492
59.632
Regress. (RMSE) 44.398
43.519
43.138
43.420
Table 10: Comparison of four different input value masking schemes evaluated for unsupervised learning on 4 classification and 3 regression datasets. Two of the variants involve separately gen- erating the mask for each variable, and two involve single distribution over "time steps", applied synchronously to all variables. Also, two of the variants involve sampling each "time step" inde- pendently based involve using Markov chain with two states, "masked" or "unmasked", with different transition probabilities pm 1 and Pa, Pm - such that the masked sequences follow a geometric dis- tribution with a mean length of lm. = 3 and each variable masked on average by 15%. We observe that our proposed scheme, separately masking each variable through stateful generation, performs consistently well and shows the overall best performance across examined datasets.
Bernoulli distribution with parameter p
15%, while the remaining two
Im
Dataset
Task (Metric)
Layer.Nrm
BatchNorm
Heartbeat
Classif. (Accuracy)
0.741
0.776
InsectWingbeat
Classif. (Accuracy)
0.658
0.684
SrakenArabicDigitss
Classif. (Accuracy)
0.993
0.993
PEMS-SF
Classif. (Accuracy)
0.832
0.919
BenzznneConcentration
Regress. (RMSE)
2.053
0.516
BeijingPM25Qulitty LiveFuelMoostureContenn
Regress, (RMSE)
61.082
60.357
Regress. (RMSE)
42.993
42.607
Table 11: Performance comparison between using layer normalization and batch normalization in our supervised transformer model. The batch size 128.
15

---

Published as a conference paper at ICLR 2021
70
68
3
64
62
62
RMSEE
RMSE
60
60
58
58
56
56
54
52
unsupervised pre-training
Use data for unsupervised pre-training
Figure 3: Root Mean Squared Error of given model as a function of the number of samples (here, shown as a proportion of the total number samples in the training set) used for unsupervised pre- training. Two levels of label availability (used for supervised learning) are depicted: 10% (left panel) and 20% (right panel) of all training data labels. Note that a horizontal axis value means fully supervised learning only, while all other values correspond unsupervised pre-training followed by supervised fine-tuning
Static Epoch time (s)
Fine-tuned
Dataset
Task (Metric)
Metric
Metric
Epoch time (s)
Heartheat
Classif. (Accuracy)
0.756
0.082
0.776
0.14
Insect Winghbat
Classif. (Accuracy)
0.236
4.52
0.687
6.21
SpokenArabicDigiis
Classif. (Accuracy)
0.996
1.29
0.998
2.00
PEMS-SF
Classif. (Accuracy)
0.844
0.208
0.896
0.281
BenzznneConcentration
Regress. (RMSE)
4.684
0.697
0.494
1.101
BeijingMM25Quality Regress. (RMSE)
65.608
1.91
53,492
2.68
LiveFuelMoisureeCnneen
Regress. (RMSE)
48.724
1.696
43.138
3.57
Performance comparison between allowing all layers of pre-trained transformer to be fine-tuned, versus using static ("extracted")) representations of the time series as input to the output layer (which is equivalent freezing all model layers except for the output layer). The per-epoch training time on GPU is also shown.
Table 12:
forecasting
synchronous noise
Xm
X
X,
t2
t2
t
E
t2
tw
X1
Xw
X1
X
Figure 4: Masking schemes within our transformer encoder framework: for implementation of fore- casting objective (left), for an alternative unsupervised learning objective involving a single noise distribution over time steps, applied synchronously to all variables (right).
16

---

Published as a conference paper at ICLR 2021
True, input
True, masked Prediction
Concentration [mg/m3) *]
-1
50
100
150
200
250
time (hours)
Dimensions
200
150
220
220
120
120
220
200
50
220
120
200
250
50
120
120
200
200
100
200
Samples
320
15o
350
55
150
220
220
220
230
220
200
50
150
230
220
250
250
h/4
1.0
250
120
150
200
250
300
2000
250
50
330
2000
250
100
350
Figure 5: Top: Imputation of missing values in the of BenzzneConneentration dataset. The continuous blue line is the ground truth signal, the light blue circles indicate the values hidden from the model and the orange dots its prediction. We observe that imputed values approximate true values very well, even in cases of rapid transitions and in cases where many contiguous values are missing. Bottom: Same, shown for different dimensions of the time series (here, these are concentrations of different substances) as columns and 4 different, randomly selected samples as rows.
17

---

Published as a conference paper at ICLR 2021
A.4 ADVANTAGES TRANSFORMERS
Transformer models are based on multi-headed attention mechanism that offers several key advan- tages and renders them particularly suitable for time series data:
They can concurrently take into account long contexts of input sequence elements and learn to represent each sequence element by selectively attending to those input sequence elements which the model considers most relevant. They do so without position-dependent prior bias; this is to be contrasted with RNN- ANN-based models: a) even bi-directional RNNs treat elements in the middle of the input sequence differently from elements close to the two endpoints, and b) despite careful design, even LSTM (Long Short Term Memory) and GRU (Gated Recurrent Unit) networks practically only retain information from a limited number of time steps stored inside their hidden state (vanishing gradient problem (Hochreiter 1998 Pascanu et al. ).13), and thus the context used for representing each sequence element is inevitably local.
Multiple attention heads can consider different representation subspaces, i.e., multiple as- pects of relevance between input elements. For example, in the context of a signal with two frequency components, 1/Ti and 1/T2 one attention head can attend to neighboring time points, while another one may attend to points spaced period Tj before the currently examined time point, third period T2 before, etc. This is be contrasted with at- tention mechanisms in RNN models, which learn a single global aspect/mode of relevance between sequence elements.
After each stage of contextual representation (i.e., transformer encoder layer), attention is redistributed over the sequence elements, taking into account progressively more abstract representations of the input elements information flows from the input towards the out- put. By contrast, RNN models with attention use a single distribution of attention weights extract representation of the input, and most typically attend over single layer of representation (hidden states).
A.5 HYPERPARAMETERS
Parameter
Value
activation
gelu
dropout learning rate
0.
0.001
pos. encoding
Icarnable
Table 13: Common (fixed) hyperparameters used for all transformer models.
18

---

Published as a conference paper at ICLR 2021
Parameter
Value
dim. model dim. feedforward
128
256
num. heads
16
encoder blocks
3
batch size
128
Table 14: Hyperparameter configuration that performs reasonably well for all transformer models.
Dataset
num. blocks
num. heads
dim. model
dim. feedforward
AppliancesEnergy
3
128
512
BenzeneConnennraation
3
8
128
256
BeijingPMiQQuality
3
64
256
BeijigPPM25Qulitt
3
64 (128)
256
LiveFuelMooitureCooneen
3
64
256
IEEEPPG
3
512
512
Table 15: Supervised TST model hyperparameters for the multivariate regression datasets
Dataset
blocks
heads
dim. model
dim. feedforward
AppliancesEnergy
3
16
128
512
Benzene@ooncentration
128
256
PejjingMMOQQualiyy
3
8
64
256
BejjngPM22Quality
3
128
256
LiveFuelMoisuurCooneent
3
8
64
256
IEEEPPG
4
512
512
Table 16: Unsupervised TST model hyperparameters for the multivariate regression datasets
Dataset EthanolConnenttration
num. blocks
num.
heads
dim. model
dim. feedforward
1
64
256
FaceDetection
3
8
128
256
Handwriting
1
128
256
Heartbeat
1
8
64
256
Japanese Vowels
3
128
256
PEMS-SF
I
8
128
512
SellRegulationSCPI
3
128
256
SelfRegulationSCCP2
3
8
128
256
SrokenAArabieDigits
3
64
256
UWaveGestureLibraayy
3
16
256
256
Table 17: Supervised TST model hyperparameters for the multivariate classification datasets
Dataset
blocks
num. heads
dim. model
dim. feedforward
EthanolConneenrration
8
64
256
FaceDetection
3
128
256
Handwriting
3
16
64
256
Heartbeat
I
64
256
Japanese Vowels
3
8
128
256
PEMS-SF
1
256
512
SelfRegulationSCPP1
3
16
256
512
SelfRegulationSCCP2
3
8
256
512
SpokenAraabcDigits
3
64
256
UWWaveeestureLibaar
3
16
256
512
Table 18: Unsupervised TST model hyperparameters for the multivariate classification datasets
19


Reviews:
[{"review_id": "kXkFZ6qDNcL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n# Summary:\n\nThe paper proposes an unsupervised learning framework, similar to BERT idea but for multivariate time series.\n### Architecture: \n- Inputs are projected into d-dimensional vector space where d is the dimension of the transformer model sequence element representations.\n- A learned positional encoding is added to the input which is passed to the transformer encoder.\n-The output of the encoder is then passed to another projection layer that depends on the application (i.e classification or regression)\n-The transformer architecture is similar to Vaswani et al. (2017), however, they replaced layer norm with batch norm.\n\n###  Unsupervised Learning Training:\nFor unsupervised learning, the paper proposed taking the input data masking it, and predicting an output sequence, the loss is then calculated as the mean square error between the actual input and the predicted input.\n\n###  Evaluation and Results:\n- The paper considered two tasks both regression and classification.  For regression, the proposed method was evaluated on datasets from the Monash University, UEA, UCR Time Series Regression Archive Tan et al. (2020a); for classification, the paper used the UEA Time Series Classification Archive (Bagnall et al., 2018).\n- For each dataset, they showed the results of using the proposed transformer architecture in a supervised manner and then using the same dataset for pretraining the transformer by the proposed masking approach in an unsupervised manner followed by fine-tuning using labels. Both methods were compared to other state of the art methods for the same task.\n- For multivariate regression, the pretraining followed by supervised training showed improvements in 3 out of 6 datasets. For classification 7 out of 11 datasets showed improvements.\n\n# Strength:\n- The paper focues  an important yet a relatively unexplored area.\n- The paper is clear, well written and well motivated.\n- The paper benchmarked accorss multiple datasets on both classification and regression tasks.\n\n# Weakness:\n- My main concern is the lack of novelty the paper is basically suggesting to use a  transformer encoder and add a dense layer before and after, and if we use unsupervised training of the transformer with the same dataset we ***may*** achieve better results.\n- The main success of BERT is the ability to transfer and improve the performance on unretaleted task however here they did not include any experiments showing that if we train on model we can use to transfer on another model. I believe critical experiments that are missing is unsupervised training say with dataset 1 and the fine tuning on dataset 2 and showing improvements on multiple task (However, even if this experiments are provided I still believe the paper lacks novelty, maybe invistaginting what properties are being transferred in multivariate time series and showing difference between transferring from unvariate to multivariate and vice versa will help).\n- The paper replaced layer norm with batch norm and only stated \"here we instead use batch normalization, because it can mitigate the effect of outlier values in time series, an issue that does not arise in NLP word embedding\" they did not show the effect of using batch norm on the accuracy or gave any insights on why it *mitigate the effect of outlier values in time series*.\n- The code implementing the paper is not provided.\n- The effect of changing values variable r in masking is not investigated.\n\n\n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "hyo7KA8V_jT", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors targeted an important data format, multivariate time series, and extended the usage of the transformer to this format. The effort is appreciated as multivariate time series data is an important problem while the researches there is limited comparing to other sequences problems, e.g. language. Due to the simple idea of extension and similar works before [1], I expected a higher quality of experiments and presentation of this paper. However, experiments and writing need significant improvement for the next submission. \n\nMajor Concerns: \n\n1. The experiment's presentation is confusing and I just picked a few examples from Table 1: \na. There is no standard deviation of the result. \nb. Some methods are consistently worse than others, like 1-NN-DTWD comparing to 5-NN-DTWD. I am not sure why they are needed in the table. \nc. Averaging the RMSE does not deliver much information, especially when there is no normalization on each dataset. For example, the BeijingPM10 will dominate the averaged result. \n\n2. The structure of writing only has a lot of problems. For example, usually, the fully supervised should be presented before semi-supervised while the paper did in a reverse way. Another problem is the classification result, which seems an important part but the table is shown in the Appendix. \n\n3. The selection of datasets needs more justifications. \n\nReference: \n1. https://arxiv.org/pdf/2001.08317.pdf\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "hQ39VouyhS3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to develop a transformer-based pre-trained model for multivariate time series representation learning. Specifically, the transformer’s encoder is only used and a time-series imputation task is constructed as their unsupervised learning objective. This is a bit similar to the BERT model in NLP. But authors added a mask for each variable of the time series. After pretraining with this imputation loss, the transformer can be used for downstream tasks, such as regression and classification. As the authors mentioned on page 6, this is achieved by further fine-tuning all weights of the pre-trained transformer. \n\nIt is natural to use the transformer model from NLP for time series modeling since both sentences and time series are sequential data. In this work, the authors’ contributions or changes should include two-points: 1) constructing that imputation task for multivariate time series data. 2) using a learned positional encoding (page 4). I think these two things seem a bit interesting. \nMy concerns mainly include:\n\n1. Actually, there are existing works that have tried to use the transformer for time series. But you didn’t compare them in your experiments. At least, I think you should clarify what’s your advantages comparing to these existing works:\n- 2019 NeurIPS Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting\n- 2020 CVPR Sketch-BERT: Learning Sketch Bidirectional Encoder Representation From Transformers by self-supervised Learning of Sketch Gestalt \n\n3. As a core topic in this work, I encourage authors to clarify the definition of the time series representation. What’s a good representation in time series? There is a related work for time series representation learning: \n- 2019 NeurIPS Unsupervised Scalable Representation Learning for Multivariate Time Series \nI notice that they also train a dilated CNN model to get a single feature for a segment of time series. But in your case, you directly concatenate states from all-time steps to form your final vector. I just wonder if your strategy is reasonable. Because when we deal with long time series, concatenating them will create another long time-series/hidden state sequence. This may not be representation learning and it cannot handle the segment-level tasks such as classification. Thus, I encourage authors to give more insights into the questions what’s a good representation of the time series. \n\n3. Besides, in the above NeurIPS representation learning paper, I find they didn’t fine-tune their model’s parameters. They just added SVM to test the performance of learned representations. But in this work, the authors fine-tuning their model. Thus, I wonder what’s the performance of your model without additional fine-tuning. What's your parameter settings for the fine-tuning procedure. \n\n4. Another question I am concerned about is your learnable position encoding. It is with a shape of w-by-d where w should be your window length and d is your input dimension. Since time series is dynamic data and its length would be much longer, this design maybe not good as it largely increases the number of parameters.  \n\n5. For experimental results, authors mainly consider their model’s performance on regression and classification tasks. But I would like to see more analysis of their learned presentations. Adding more visualization analysis would be helpful to demonstrate your framework’s effectiveness. \n\n6. Experimental settings are unclear. For example, since your datasets only contain train/test sets. How to pick your hyperparameters? Is it based on that performance on that test set? What kind of regression task is used in your regression experiments? Is it a one-step-ahead prediction task? \n\n7. In Table 5, you claim your model is faster than the ROCKET model. But the running time reported for your model is per-epoch training time, not the total training time. I think this seems a bit unreasonable. Please check it. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "P7KF2SGF-UZ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper uses transformer to improve mutlivariate time series classification and regression using a BERT inspired self-supervised loss. The authors show improvement over multiple standard datasets. The use of self-supervision improves performance in lower (labeled) data regime.\n\n\n#### Strong points\n\n- The method is well explained and take good inspiration of BERT pretraining.\n- The data-available experiment shows that training with self-supervision helps for classification.\n- This work is well positioned in the literature of the field.\n\n#### Weak points\n\n- The timeserie representation is obtained by concatenation of the tokens' representations. This means that the representations are not of fixed size (or contain padding representations).\n- The introduced masking loss could suffer from discrepancy between training and inference where all the covariates at a given timesteps need to be predicted. Moreover, the model could rely on very correlated covariates (which frequently happens) to recover the masked variables limiting the effect of self-supervised training. I would like to see if masking all variables at some times step helps/hurts the pretraining.\n- The paper does not consider if finetuning is needed or if the network could be frozen and only a small MLP could be learned on top of the \"pre-extracted times-series representations\".\n- The hyperparameters of the network are chosen per dataset but are not reported. It is not clear if the tuning is done with cross validation or looking at the test error. The authors should report the performance of the transformer with common hyperparameters specified in the Appendix. This would be more fair to compare with Franceschi et al., who do not tune the hyperparameters of their encoder.\n- The claim in the abstract that the method offers \"computational efficiency\" is not backed with evidence. Section A.4 \"Execution Time\" does not report execution time of the method compared to the baselines.\n\n\n#### Decision\n\nI tend to reject this paper. The idea is interesting but some design decisions of the method (representation pooling, self-supervised loss, need for finetuning) should be better justified. Secondly the evaluation of the´ method should be more precise (hyperparameters tuning, speed).\n\n#### Questions/Remarks\n\n- In section 3.1 \"because the computational complexity and the number of parameters of the model\nscale as $O(w^2)$ with the input sequence length $w$\". The number of parameters of the transformer architecture does not scale with the input length. Only the memory footprint and the computation scales quadratically.\n\n- \"we note that similar to the case of word embeddings, the positional encodings generally appear not to interfere with the numerical information of the time series\". Do you have evidence for this?\n\n- Have you tried PowerNorm by Shen et al. (2020) that you cite in 3.1?\n\n- You could mention Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting (Lim et al., 2019) arXiv:1912.09363 for another use of transformers for univariate (quantile) forecasting.\n\n- Add the dataset you are experimenting with in the caption of Figure 2.\n\n#### Additional feedback\n\n- Figure 2: chose different colors for left and right figures.\n- Table 1: consider using booktabs for table format and transpose dataset and models to make it fit in the margin.\n- Section 3.2: The sentence starting with \"The reason ...\" that spans 6 lines could be split and reworded.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors extends the transformer to multivariate time series. The proposed extension is simple, and lacks novelty. Some design decisions of the proposed method should be better justified. Similar works that also use the transformer for timeseries are not compared.

Experimental results are not convincing. The settings are unclear, and the selection of datasets needs more justifications. Some important experiments are missing.

Finally, writing can also be improved.
Decision: Reject