A Transformer-based Framework for Multivariate Time Series Representation Learning

1 INTRODUCTION . Multivariate time series ( MTS ) are an important type of data that is ubiquitous in a wide variety of domains , including science , medicine , finance , engineering and industrial applications . Despite the recent abundance of MTS data in the much touted era of “ Big Data ” , the availability of labeled data in particular is far more limited : extensive data labeling is often prohibitively expensive or impractical , as it may require much time and effort , special infrastructure or domain expertise . For this reason , in all aforementioned domains there is great interest in methods which can offer high accuracy by using only a limited amount of labeled data or by leveraging the existing plethora of unlabeled data . There is a large variety of modeling approaches for univariate and multivariate time series , with deep learning models recently challenging or replacing the state of the art in tasks such as forecasting , regression and classification ( De Brouwer et al. , 2019 ; Tan et al. , 2020a ; Fawaz et al. , 2019b ) . However , unlike in domains such as Computer Vision or Natural Language Processing ( NLP ) , the dominance of deep learning for time series is far from established : in fact , non-deep learning methods such as TS-CHIEF ( Shifaz et al. , 2020 ) , HIVE-COTE ( Lines et al. , 2018 ) , and ROCKET ( Dempster et al. , 2020 ) currently hold the record on time series regression and classification dataset benchmarks ( Tan et al. , 2020a ; Bagnall et al. , 2017 ) , matching or even outperforming sophisticated deep architectures such as InceptionTime ( Fawaz et al. , 2019a ) and ResNet ( Fawaz et al. , 2019b ) . In this work , we investigate , for the first time , the use of a transformer encoder for unsupervised representation learning of multivariate time series , as well as for the tasks of time series regression and classification . Transformers are an important , recently developed class of deep learning models , which were first proposed for the task of natural language translation ( Vaswani et al. , 2017 ) but have since come to monopolize the state-of-the-art performance across virtually all NLP tasks ( Raffel et al. , 2019 ) . A key factor for the widespread success of transformers in NLP is their aptitude for learning how to represent natural language through unsupervised pre-training ( Brown et al. , 2020 ; Raffel et al. , 2019 ; Devlin et al. , 2018 ) . Besides NLP , transformers have also set the state of the art in several domains of sequence generation , such as polyphonic music composition ( Huang et al. , 2018 ) . Transformer models are based on a multi-headed attention mechanism that offers several key advantages and renders them particularly suitable for time series data ( see Appendix section A.4 for details ) . Inspired by the impressive results attained through unsupervised pre-training of transformer models in NLP , as our main contribution , in the present work we develop a generally applicable methodology ( framework ) that can leverage unlabeled data by first training a transformer model to extract dense vector representations of multivariate time series through an input denoising ( autoregressive ) objective . The pre-trained model can be subsequently applied to several downstream tasks , such as regression , classification , imputation , and forecasting . Here , we apply our framework for the tasks of multivariate time series regression and classification on several public datasets and demonstrate that transformer models can convincingly outperform all current state-of-the-art modeling approaches , even when only having access to a very limited amount of training data samples ( on the order of hundreds of samples ) , an unprecedented success for deep learning models . Importantly , despite common preconceptions about transformers from the domain of NLP , where top performing models have billions of parameters and require days to weeks of pre-training on many parallel GPUs or TPUs , we also demonstrate that our models , using at most hundreds of thousands of parameters , can be trained even on CPUs , while training them on GPUs allows them to be trained as fast as even the fastest and most accurate non-deep learning based approaches . 2 RELATED WORK . Regression and classification of time series : Currently , non-deep learning methods such as TSCHIEF ( Shifaz et al. , 2020 ) , HIVE-COTE ( Lines et al. , 2018 ) , and ROCKET ( Dempster et al. , 2020 ) constitute the state of the art for time series regression and classification based on evaluations on public benchmarks ( Tan et al. , 2020a ; Bagnall et al. , 2017 ) , followed by CNN-based deep architectures such as InceptionTime ( Fawaz et al. , 2019a ) and ResNet ( Fawaz et al. , 2019b ) . ROCKET , which on average is the best ranking method , is a fast method that involves training a linear classifier on top of features extracted by a flat collection of numerous and various random convolutional kernels . HIVE-COTE and TS-CHIEF ( itself inspired by Proximity Forest ( Lucas et al. , 2019 ) ) , are very sophisticated methods which incorporate expert insights on time series data and consist of large , heterogeneous ensembles of classifiers utilizing shapelet transformations , elastic similarity measures , spectral features , random interval and dictionary-based techniques ; however , these methods are highly complex , involve significant computational cost , can not benefit from GPU hardware and scale poorly to datasets with many samples and long time series ; moreover , they have been developed for and only been evaluated on univariate time series . Unsupervised learning for multivariate time series : Recent work on unsupervised learning for multivariate time series has predominantly employed autoencoders , trained with an input reconstruction objective and implemented either as Multi-Layer Perceptrons ( MLP ) or RNN ( most commonly , LSTM ) networks . As interesting variations of the former , Kopf et al . ( 2019 ) and Fortuin et al . ( 2019 ) additionally incorporated Variational Autoencoding into this approach , but focused on clustering and the visualization of shifting sample topology with time . As an example of the latter , Malhotra et al . ( 2017 ) presented a multi-layered RNN sequence-to-sequence autoencoder , while Lyu et al . ( 2018 ) developed a multi-layered LSTM with an attention mechanism and evaluated both an input reconstruction ( autoencoding ) as well as a forecasting loss for unsupervised representation learning of Electronic Healthcare Record multivariate time series . As a novel take on autoencoding , and with the goal of dealing with missing data , Bianchi et al . ( 2019 ) employ a stacked bidirectional RNN encoder and stacked RNN decoder to reconstruct the input , and at the same time use a user-provided kernel matrix as prior information to condition internal representations and encourage learning similarity-preserving representations of the input . They evaluate the method on the tasks of missing value imputation and classification of time series under increasing “ missingness ” of values . A distinct approach is followed by Zhang et al . ( 2019 ) , who use a composite convolutional - LSTM network with attention and a loss which aims at reconstructing correlation matrices between the variables of the multivariate time series input . They use and evaluate their method only for the task of anomaly detection . Finally , Jansen et al . ( 2018 ) rely on a triplet loss and the idea of temporal proximity ( the loss rewards similarity of representations between proximal segments and penalizes similarity between distal segments of the time series ) for unsupervised representation learning of non-speech audio data . This idea is explored further by Franceschi et al . ( 2019 ) , who combine the triplet loss with a deep causal dilated CNN , in order to make the method effective for very long time series . Transformer models for time series : Recently , a full encoder-decoder transformer architecture was employed for univariate time series forecasting : Li et al . ( 2019 ) showed superior performance compared to the classical statistical method ARIMA , the recent matrix factorization method TRMF , an RNN-based autoregressive model ( DeepAR ) and an RNN-based state space model ( DeepState ) on 4 public forecasting datasets , while Wu et al . ( 2020 ) used a transformer to forecast influenza prevalence and similarly showed performance benefits compared to ARIMA , an LSTM and a GRU Seq2Seq model with attention , and Lim et al . ( 2020 ) used a transformer for multi-horizon univariate forecasting , supporting interpretation of temporal dynamics . Finally , Ma et al . ( 2019 ) use an encoder-decoder architecture with a variant of self-attention for imputation of missing values in multivariate , geo-tagged time series and outperform classic as well as the state-of-the-art , RNN-based imputation methods on 3 public and 2 competition datasets for imputation . By contrast , our work aspires to generalize the use of transformers from solutions to specific generative tasks ( which require the full encoder-decoder architecture ) to a framework which allows for unsupervised pre-training and with minor modifications can be readily used for a wide variety of downstream tasks ; this is analogous to the way BERT ( Devlin et al. , 2018 ) converted a translation model into a generic framework based on unsupervised learning , an approach which has become a de facto standard and established the dominance of transformers in NLP . 3 METHODOLOGY . 3.1 BASE MODEL . At the core of our method lies a transformer encoder , as described in the original transformer work by Vaswani et al . ( 2017 ) ; however , we do not use the decoder part of the architecture . A schematic diagram of the generic part of our model , common across all considered tasks , is shown in Figure 1 . We refer the reader to the original work for a detailed description of the transformer model , and here present the proposed changes that make it compatible with multivariate time series data , instead of sequences of discrete word indices . In particular , each training sample X ∈ Rw×m , which is a multivariate time series of length w and m different variables , constitutes a sequence of w feature vectors xt ∈ Rm : X ∈ Rw×m = [ x1 , x2 , . . . , xw ] . The original feature vectors xt are first normalized ( for each dimension , we subtract the mean and divide by the variance across the training set samples ) and then linearly projected onto a d-dimensional vector space , where d is the dimension of the transformer model sequence element representations ( typically called model dimension ) : ut = Wpxt + bp ( 1 ) where Wp ∈ Rd×m , bp ∈ Rd are learnable parameters and ut ∈ Rd , t = 0 , . . . , w are the model input vectors1 . These will become the queries , keys and values of the self-attention layer , after adding the positional encodings and multiplying by the corresponding matrices . We note that the above formulation also covers the univariate time series case , i.e. , m = 1 , although we only evaluate our approach on multivariate time series in the scope of this work . We additionally note that the input vectors ut need not necessarily be obtained from the ( transformed ) feature vectors at a time step t : because the computational complexity of the model scales as O ( w2 ) and the number of parameters2 as O ( w ) with the input sequence length w , to obtain ut in case the granularity ( temporal resolution ) of the data is very fine , one may instead use a 1D-convolutional layer with 1 input and d output channels and kernels Ki of size ( k , m ) , where k is the width in number of time steps and i the output channel : ut i = u ( t , i ) = ∑ j ∑ h x ( t + j , h ) Ki ( j , h ) , i = 1 , . . . , d ( 2 ) 1Although equation 1 shows the operation for a single time step for clarity , all input vectors are embedded concurrently by a single matrix-matrix multiplication 2Specifically , the learnable positional encoding , batch normalization and output layers In this way , one may control the temporal resolution by using a stride or dilation factor greater than 1 . Moreover , although in the present work we only used equation 1 , one may use equation 2 as an input to compute the keys and queries and equation 1 to compute the values of the self-attention layer . This is particularly useful in the case of univariate time series , where self-attention would otherwise match ( consider relevant/compatible ) all time steps which share similar values for the independent variable , as noted by Li et al . ( 2019 ) . Finally , since the transformer is a feed-forward architecture that is insensitive to the ordering of input , in order to make it aware of the sequential nature of the time series , we add positional encodings Wpos ∈ Rw×d to the input vectors U ∈ Rw×d = [ u1 , . . . , uw ] : U ′ = U + Wpos . Instead of deterministic , sinusoidal encodings , which were originally proposed by Vaswani et al . ( 2017 ) , we use fully learnable positional encodings , as we observed that they perform better for all datasets presented in this work . Based on the performance of our models , we also observe that the positional encodings generally appear not to significantly interfere with the numerical information of the time series , similar to the case of word embeddings ; we hypothesize that this is because they are learned so as to occupy a different , approximately orthogonal , subspace to the one in which the projected time series samples reside . This approximate orthogonality condition is much easier to satisfy in high dimensional spaces . An important consideration regarding time series data is that individual samples may display considerable variation in length . This issue is effectively dealt with in our framework : after setting a maximum sequence length w for the entire dataset , shorter samples are padded with arbitrary values , and we generate a padding mask which adds a large negative value to the attention scores for the padded positions , before computing the self-attention distribution with the softmax function . This forces the model to completely ignore padded positions , while allowing the parallel processing of samples in large minibatches . Transformers in NLP use layer normalization after computing self-attention and after the feedforward part of each encoder block , leading to significant performance gains over batch normalization , as originally proposed by Vaswani et al . ( 2017 ) . However , here we instead use batch normalization , because it can mitigate the effect of outlier values in time series , an issue that does not arise in NLP word embeddings . Additionally , the inferior performance of batch normalization in NLP has been mainly attributed to extreme variation in sample length ( i.e. , sentences in most tasks ) ( Shen et al. , 2020 ) , while in the datasets we examine this variation is much smaller . In Table 11 of the Appendix we show that batch normalization can indeed offer a very significant performance benefit over layer normalization , while the extent can vary depending on dataset characteristics .

Summary: This paper aims to develop a transformer-based pre-trained model for multivariate time series representation learning. Specifically, the transformer’s encoder is only used and a time-series imputation task is constructed as their unsupervised learning objective. This is a bit similar to the BERT model in NLP. But authors added a mask for each variable of the time series. After pretraining with this imputation loss, the transformer can be used for downstream tasks, such as regression and classification. As the authors mentioned on page 6, this is achieved by further fine-tuning all weights of the pre-trained transformer.