Multi-Time Attention Networks for Irregularly Sampled Time Series

1 INTRODUCTION . Irregularly sampled time series occur in application domains including healthcare , climate science , ecology , astronomy , biology and others . It is well understood that irregular sampling poses a significant challenge to machine learning models , which typically assume fully-observed , fixed-size feature representations ( Marlin et al. , 2012 ; Yadav et al. , 2018 ) . While recurrent neural networks ( RNNs ) have been widely used to model such data because of their ability to handle variable length sequences , basic RNNs assume regular spacing between observation times as well as alignment of the time points where observations occur for different variables ( i.e. , fully-observed vectors ) . In practice , both of these assumptions can fail to hold for real-world sparse and irregularly observed time series . To respond to these challenges , there has been significant progress over the last decade on building and adapting machine learning models that can better capture the structure of irregularly sampled multivariate time series ( Li & Marlin , 2015 ; 2016 ; Lipton et al. , 2016 ; Futoma et al. , 2017 ; Che et al. , 2018 ; Shukla & Marlin , 2019 ; Rubanova et al. , 2019 ) . In this work , we introduce a new model for multivariate , sparse and irregularly sampled time series that we refer to as Multi-Time Attention networks or mTANs . mTANs are fundamentally continuous-time , interpolation-based models . Their primary innovations are the inclusion of a learned continuous-time embedding mechanism coupled with a time attention mechanism that replaces the use of a fixed similarity kernel when forming representation from continuous time inputs . This gives mTANs more representational flexibility than previous interpolation-based models ( Shukla & Marlin , 2019 ) . Our approach re-represents an irregularly sampled time series at a fixed set of reference points . The proposed time attention mechanism uses reference time points as queries and the observed time points as keys . We propose an encoder-decoder framework for end-to-end learning using an mTAN module to interface with given multivariate , sparse and irregularly sampled time series inputs . The encoder takes the irregularly sampled time series as input and produces a fixed-length latent representation over a set of reference points , while the decoder uses the latent representations to produce reconstructions conditioned on the set of observed time points . Learning uses established methods for variational autoencoders ( Rezende et al. , 2014 ; Kingma & Welling , 2014 ) . 1Implementation available at : https : //github.com/reml-lab/mTAN The main contributions of the mTAN model framework are : ( 1 ) It provides a flexible approach to modeling multivariate , sparse and irregularly sampled time series data ( including irregularly sampled time series of partially observed vectors ) by leveraging a time attention mechanism to learn temporal similarity from data instead of using fixed kernels . ( 2 ) It uses a temporally distributed latent representation to better capture local structure in time series data . ( 3 ) It provides interpolation and classification performance that is as good as current state-of-the-art methods or better , while providing significantly reduced training times . 2 RELATED WORK . An irregularly sampled time series is a time series with irregular time intervals between observations . In the multivariate setting , there can also be a lack of alignment across different variables within the same multivariate time series . Finally , when gaps between observation times are large , the time series is also considered to be sparse . Such data occur in electronic health records ( Marlin et al. , 2012 ; Yadav et al. , 2018 ) , climate science ( Schulz & Stattegger , 1997 ) , ecology ( Clark & Bjørnstad , 2004 ) , biology ( Ruf , 1999 ) , and astronomy ( Scargle , 1982 ) . It is well understood that such data cause significant issues for standard supervised machine learning models that typically assume fully observed , fixed-size feature representations ( Marlin et al. , 2012 ) . A basic approach to dealing with irregular sampling is fixed temporal discretization . For example , Marlin et al . ( 2012 ) and Lipton et al . ( 2016 ) discretize continuous-time observations into hour-long bins . This has the advantage of simplicity , but requires ad-hoc handling of bins with more than one observation and results in missing data when bins are empty . The alternative to temporal discretization is to construct models with the ability to directly use an irregularly sampled time series as input . Che et al . ( 2018 ) present several methods based on gated recurrent unit networks ( GRUs , Chung et al . ( 2014 ) ) , including an approach that takes as input a sequence consisting of observed values , missing data indicators , and time intervals since the last observation . Pham et al . ( 2017 ) proposed to capture time irregularity by modifying the forget gate of an LSTM ( Hochreiter & Schmidhuber , 1997 ) , while Neil et al . ( 2016 ) introduced a new time gate that regulates access to the hidden and cell state of the LSTM . While these approaches allow the network to handle event-based sequences with irregularly spaced vector-valued observations , they do not support learning directly from vectors that are partially observed , which commonly occurs in the multivariate setting because of lack of alignment of observation times across different variables . Another line of work has looked at using observations from the future as well as from the past for interpolation . Yoon et al . ( 2019 ) and Yoon et al . ( 2018 ) presented an approach based on the multi-directional RNN ( M-RNN ) that can leverage observations from the relative past and future of a given time point . Shukla & Marlin ( 2019 ) proposed the interpolation-prediction network framework , consisting of several semi-parametric RBF interpolation layers that interpolate multivariate , sparse , and irregularly sampled input time series against a set of reference time points while taking into account all observed data in a time series . Horn et al . ( 2020 ) proposed a set function-based approach for classifying time-series with irregularly sampled and unaligned observation . Chen et al . ( 2018 ) proposed a variational auto-encoder model ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) for continuous time data based on the use of a neural network decoder combined with a latent ordinary differential equation ( ODE ) model . They model time series data via a latent continuous-time function that is defined via a neural network representation of its gradient field . Building on this , Rubanova et al . ( 2019 ) proposed a latent ODE model that uses an ODE-RNN model as the encoder . ODE-RNNs use neural ODEs to model the hidden state dynamics and an RNN to update the hidden state in the presence of a new observation . De Brouwer et al . ( 2019 ) proposed GRU-ODE-Bayes , a continuous-time version of the Gated Recurrent Unit ( Chung et al. , 2014 ) . Instead of the encoder-decoder architecture where the ODE is decoupled from the input processing , GRU-ODE-Bayes provides a tighter integration by interleaving the ODE and the input processing steps . Several recent approaches have also used attention mechanisms to model irregularly sampled time series ( Song et al. , 2018 ; Tan et al. , 2020 ; Zhang et al. , 2019 ) as well as medical concepts ( Peng et al. , 2019 ; Cai et al. , 2018 ) . Most of these approaches are similar to Vaswani et al . ( 2017 ) where they replace the positional encoding with an encoding of time and model sequences using self-attention . However , instead of adding the time encoding to the input representation as in Vaswani et al . ( 2017 ) , they concatenate it with the input representation . These methods use a fixed time encoding similar to the positional encoding of Vaswani et al . ( 2017 ) . Xu et al . ( 2019 ) learn a functional time representation and concatenate it with the input event embedding to model time-event interactions . Like Xu et al . ( 2019 ) and Kazemi et al . ( 2019 ) , our proposed method learns a time representation . However , instead of concatenating it with the input embedding , our model learns to attend to observations at different time points by computing a similarity weighting using only the time embedding . Our proposed model uses the time embedding as both the queries and keys in the attention formulation . It learns an interpolation over the query time points by attending to the observed values at key time points . Our proposed method is thus similar to kernel-based interpolation , but learning the time attention based similarity kernel gives our model more flexibility compared to methods like that of Shukla & Marlin ( 2019 ) that use similarity kernels with fixed functional forms . Another important difference relative to many of these previous methods is that our proposed approach attends only to the observed data dimensions at each time point and hence does not require a separate imputation step to handle vector valued observations with an arbitrary collection of dimensions missing at any given time point . 3 THE MULTI-TIME ATTENTION MODULE . In this section , we present the proposed Multi-Time Attention Module ( mTAN ) . The role of this module is to re-represent a sparse and irregularly sampled time series in a fixed-dimensional space . This module uses multiple continuous-time embeddings and attention-based interpolation . We begin by presenting notation followed by the time embedding and attention components . Notation : In the case of a supervised learning task , we let D = { ( sn , yn ) |n = 1 , ... , N } represent a data set containing N data cases . An individual data case consists of a single target value yn ( discrete for classification ) , as well as a D-dimensional , sparse and irregularly sampled multivariate time series sn . Different dimensions d of the multivariate time series can have observations at different times , as well as different total numbers of observations Ldn . Thus , we represent time series d for data case n as a tuple sdn = ( tdn , xdn ) where tdn = [ t1dn , ... , tLdndn ] is the list of time points at which observations are defined and xdn = [ x1dn , ... , xLdndn ] is the corresponding list of observed values . In the case of an unsupervised task such as interpolation , each data case consists of a multivariate time series sn only . We drop the data case index n for brevity when the context is clear . Time Embedding : Time attention module is based on embedding continuous time points into a vector space . We generalize the notion of a positional encoding used in transformer-based models to continuous time . Time attention networks simultaneously leverage H embedding functions φh ( t ) , each outputting a representation of size dr. Dimension i of embedding h is defined as follows : φh ( t ) [ i ] = { ω0h · t+ α0h , if i = 0 sin ( ωih · t+ αih ) , if 0 < i < dr ( 1 ) where the ωih ’ s and αih ’ s are learnable parameters . The periodic terms can capture periodicity in time series data . In this case , ωih and αih represent the frequency and phase of the sine function . The linear term , on the other hand , can capture non-periodic patterns dependent on the progression of time . For a given difference ∆ , φh ( t+ ∆ ) can be represented as a linear function of φh ( t ) . Learning the periodic time embedding functions is equivalent to using a one-layer fully connected network with a sine function non-linearity to map the time values into a higher dimensional space . By contrast , the positional encoding used in transformer models is defined only for discrete positions . We note that our time embedding functions subsume positional encodings when evaluated at discrete positions . Multi-Time Attention : The time embedding component described above takes a continuous time point and embeds it into H different dr-dimensional spaces . In this section , we describe how we leverage time embeddings to produce a continuous-time embedding module for sparse and irregularly sampled time series . This multi-time attention embedding module mTAN ( t , s ) takes as input a query time point t and a set of keys and values in the form of a D-dimensional multivariate sparse and irregularly sampled time series s ( as defined in the notation section above ) , and returns a J- dimensional embedding at time t. This process leverages a continuous-time attention mechanism applied to the H time embeddings . The complete computation is described below . mTAN ( t , s ) [ j ] = H∑ h=1 D∑ d=1 x̂hd ( t , s ) · Uhdj ( 2 ) x̂hd ( t , s ) = Ld∑ i=1 κh ( t , tid ) xid ( 3 ) κh ( t , tid ) = exp ( φh ( t ) wv Tφh ( tid ) T / √ dk ) ∑Ld i′=1 exp ( φh ( t ) wvTφh ( ti′d ) T / √ dk ) ( 4 ) As shown in Equation 2 , dimension j of the mTAN embedding mTAN ( t , s ) [ j ] is given by a linear combination of intermediate univariate continuous-time functions x̂hd ( t , s ) . There is one such function defined for each input data dimension d and each time embedding h. The parameters Uhdj are learnable linear combination weights . As shown in Equation 3 , the structure of the intermediate continuous-time function x̂hd ( t , s ) is essentially a kernel smoother applied to the dth dimension of the time series . However , the interpolation weights κh ( t , tid ) are defined based on a time attention mechanism that leverages time embeddings , as shown in Equation 4 . As we can see , the same time embedding function φh ( t ) is applied for all data dimensions . The form of the attention mechanism is a softmax function over the observed time points tid for dimension d. The activation within the softmax is a scaled inner product between the time embedding φh ( t ) of the query time point t and the time embedding φh ( tid ) of the observed time point , the key . The parameters w and v are each dr × dk matrices where dk ≤ dr. We use a scaling factor 1√ dk to normalize the dot product to counteract the growth in the dot product magnitude with increase in the dimension dk . Learning the time embeddings provides our model with flexibility to learn complex temporal kernel functions κh ( t , t′ ) . The use of multiple simultaneous time embeddings φh ( t ) and a final linear combination across time embedding dimensions and data dimensions means that the final output representation function mTAN ( t , s ) is extremely flexible . Different input dimensions can leverage different time embeddings via learned sparsity patterns in the parameter tensor U . Information from different data dimensions can also be mixed together to create compact reduced dimensional representations . We note that all of the required computations can be parallelized using masking variables to deal with unobserved dimensions , allowing for efficient implementation on a GPU . Discretization : Since the mTAN module defines a continuous function of t given s , it can not be directly incorporated into neural network architectures that expect inputs in the form of fixeddimensional vectors or discrete sequences . However , the mTAN module can easily be adapted to produce such an output representation by materializing its output at a set of reference time points r = [ r1 , ... , rK ] . In some cases , we may have a fixed set of such points . In other cases , the set of reference time points may need to depend on s itself . In particular , we define the auxiliary function ρ ( s ) to return the set of time points at which there is an observation on any dimension of s. Given a collection of reference time points r , we define the discretized mTAN module mTAND ( r , s ) as mTAND ( r , s ) [ i ] = mTAN ( ri , s ) . This module takes as input the set of reference time points r and the time series s and outputs a sequence of mTAN embeddings of length |r| , each of dimension J . The architecture of the mTAND module is shown in Figure 1 . The mTAND module can be used to interface sparse and irregularly sampled multivariate time series data with any deep neural network layer type including fully-connected , recurrent , and convolutional layers . In the next section , we describe the construction of a temporal encoder-decoder architecture leveraging the mTAND module , which can be applied to both classification and interpolation tasks .

Summary: This paper proposes a novel approach to learn an embedding of continuous time values and use an attention mechanism to produce a fixed-length representation of a time series containing a variable number of observations. In particular, it proposes an mTAN network to leverage the mTAN module in an encoder-decoder framework for both unsupervised and supervised Learning.  The main contribution of this paper is the introduction of Multi-Time Attention Networks to learns a time representation and learns to attend to observations at different time points by computing a similarity weighting by the learning time embedding.  Empirical studies are performed to show the superiority of the proposed model mTANs over several baseline approaches on the tasks unsupervised and supervised learning.