Multi-Time Attention Networks for Irregularly Sampled Time Series
Authors: Satya Narayan Shukla
Benjamin Marlin
Venue: ICLR 2021 Poster (2021)
Keywords: irregular sampling
multivariate time series
attention
missing data

Abstract: Irregular sampling occurs in many time series modeling applications where it presents a significant challenge to standard deep learning models. This work is motivated by the analysis of physiological time series data in electronic health records, which are sparse, irregularly sampled, and multivariate. In this paper, we propose a new deep learning framework for this setting that we call Multi-Time Attention Networks. Multi-Time Attention Networks learn an embedding of continuous time values and use an attention mechanism to produce a fixed-length representation of a time series containing a variable number of observations. We investigate the performance of this framework on interpolation and classification tasks using multiple datasets. Our results show that the proposed approach performs as well or better than a range of baseline and recently proposed models while offering significantly faster training times than current state-of-the-art methods.

Published as a conference paper at ICLR 2021
MULTI-TIME ATTENTION NETWORKS FOR IRREGULARLY SAMPLED TIME SERIES
Satya Narayan Shukla & Benjamin M. Marlin College of Information and Computer Sciences University of Massachusetts Amherst Amherst, MA 01003, USA
(snshukla, marlin &cs. umass edu
ABSTRACT
Irregular sampling occurs in many time series modeling applications where it presents significant challenge standard deep learning models. This work is motivated by the analysis of physiological time series data in electronic health records, which are sparse, irregularly sampled, and multivariate. In this paper, we propose a new deep learning framework for this setting that we call Multi- Time Attention Networks. Multi-Time Attention Networks learn an embedding of continuous time values and use attention mechanism to produce fixed-length representation of time series containing a variable number of observations. We investigate the performance of this framework tasks using multiple datasets. Our results show that the proposed approach performs well or better than range of baseline and recently proposed models while offering significantly faster training times than current state-of-the-ar methods
interpolation and classification
INTRODUCTION
Irregularly sampled time series occur in application domains including healthcare, climate science, ecology, astronomy, biology and others. It is well understood that irregular sampling poses a significant challenge to machine learning models, which typically assume fully-observed, fixed-size feature representations (Marlin et al., 2012 Yadav et al. 2018). While recurrent neural networks (RNNs) have been widely used model such data because of their ability to handle variable length sequences, basic RNNs assume regular spacing between observation times as well as alignment of the time points where observations occur for different variables (i.e., fully-observed vectors). In practice, both of these assumptions can fail to hold for real-world sparse and irregularly observed time series. To respond to these challenges, there has been significant progress over the last decade on building and adapting machine learning models that can better capture the structure of irregularly sampled
multivariate time series (Li & Marlin| 2015; 2016 Lipton et al." 2016 Futoma et al. 2017 Che et al.
2018 Shukla & Marlin 2019 Rubanova et al.) 2019).
In this work, we introduce new model for multivariate, sparse and irregularly sampled time series that refer to Multi-Time Attention networks m'ANNs. m'TANs fundamentally continuous-time interpolation- based models. Their primary innovations are the inclusion a learned continuous-time embedding mechanism coupled with time attention mechanism that replaces the of fixed similarity kernel when forming representation from continuous time inputs. This gives mTANs more representational flexibility than previous interpolation-based models (Shukla & Marlin 2019).
Our approach re-represents an irregularly sampled time series at a fixed set of reference points. The proposed time attention mechanism uses reference time points as queries and the observed time points as keys. We propose an encoder-decoder framework for end-to-end learning using an mTAN module interface with given multivariate, sparse and irregularly sampled time series inputs. The encoder takes the irregularly sampled time series input and produces a fixed-length latent representation over set of reference points, while the decoder uses the latent representations to produce reconstructions conditioned the set of observed time points. Learning uses established methods for variational autoencoders (Rezende et al. 2014 Kingma & Welling 2014).
"Imllementation availahle

---

Published as a conference paper at ICLR 2021
The main contributions of the mTAN model framework are: (1) It provides flexible approach to modeling multivariate, sparse and irregularly sampled time series data (including irregularly sampled time series of partially observed vectors) by leveraging time attention mechanism to learn temporal similarity from data instead of using fixed kernels. (2) It uses a temporally distributed latent representation better capture local structure in time series data. (3) It provides interpolation and classification performance that is as good as current state-of-the-an methods or better, while providing significantly reduced training times.
2 RELATED WORK
An irregularly sampled time series is a time series with irregular time intervals between observations. In the multivariate setting, there can also be a lack of alignment across different variables within the same multivariate time series. Finally, when gaps between observation times are large, the time series is also considered to sparse. Such data occur in electronic health records (Marlin et al. 2012: Yadav et al., 2018), climate science (Schulz & Stattegger 1997). ecology (Clark & Bjørnstad 2004). biology (Rui 1999), and astronomy (Scargle) 1982). is well understood that such data cause significant issues for standard supervised machine learning models that typically assume fully observed, fixed-size feature representations (Marlin et al. 2012).
A basic approach to dealing with irregular sampling is fixed temporal discretization. For example, Marlin et al. (2012) and Lipton et al. (2166) discretize continuous-time observations into hour-long bins. This has the advantage of simplicity, but requires ad-hoc handling of bins with more than one observation and results in missing data when bins are empty.
The alternative to temporal discretization is to construct models with the ability to directly use an irregularly sampled time series as input. [Che et al. (2018) present several methods based on gated recurrent unit networks (GRUs, Chung et al. (2014), including approach that takes as input sequence consisting of observed values, missing data indicators, and time intervals since the last PPham observation. Pham et al. 2017) proposed to capture time irregularity by modifying the forget gate of
an LSTM (Hochreiter & Schmidhuber 1997). while Neil et al. (2016) introduced new time gate that regulates access to the hidden and cell state of the LSTM. While these approaches allow the network to handle event- sequences with irregularly spaced vector-valued observations, they do not support learning directly from vectors that are partially observed, which commonly occurs in the multivariate setting because of lack of alignment of observation times across different variables.
Another line of work has looked at using observations from the future as well as from the past for interpolation. Yoon et al.) (2019) and Yoon et al. (2018) presented an approach based on the multi-directional RNN (M-RNN) that can leverage observations from the relative past and future of a given time point. Shukkl & Marlin 2019) proposed the cerrppolatio--redditti network framework. consisting of several semi- parametric RBF interpolation layers that interpolate multivariate, sparse, and irregularly sampled input time series against set of reference time points while taking into account all observed data in time series. |Horn et al. (2020) proposed a set function-based approach for classifying time-series with irregularly sampled and unaligned observation.
Chen et al. (2018) proposed a variational auto-encoder model (Kingma & Welling 2014 Rezende al. 2014) for continuous time data based on the use of a neural network decoder combined with a latent ordinary differential equation (ODE) model. They model time series data via latent continuous-time function that is defined via neural network representation of its gradient field. Building on this, Rubanova et al. (2019) proposed latent ODE model that uses an ODE-RNN model as the encoder. RDE- use neural ODEs to model the hidden state dynamics and an RNN to update the hidden state in the presence of a new observation. De Brouwer et al. (2019) proposed GRU-ODE- 2014). Instead of the encoder-decoder architecture where the ODE is decoupled from the input processing, GRU-ODE-Bayee provides a tighter integration by interleaving the ODE and the input processing steps.
a continuous-time version of the Gated Recurrent Unit (Chung et al.)
Several recent approaches have also used attention mechanisms to model irregularly sampled time series (Song et al. 2018 Tan et al. 2020 Zhang al. 2019) well as medical concepts (Peng et al., 2019) Cai et al. 2018). Most of these approaches are similar to Vaswani et al. (2017) where they replace the positional encoding with encoding of time and model sequences using self-attention.
2

---

Published as a conference paper at ICLR 2021
However, instead of adding the time encoding to the imput representation in Vaswani et al. (017)) they concatenate it with the imput representation. These methods use a fixed time encoding similar to the positional encoding of Vaswani et al. (017)) Xu et al. (2019) learn functional time representation and concatenate it with the input event embedding to model time-event interactions.
Like Xu al. (2019) and Kazemi et al. 2201)), our proposed method learns time representation. However, instead of concatenating it with the input embedding, our model learns to attend to observations different time points by computing similarity weighting using only the time embedding. Our proposed model uses the time embedding as both the queries and keys in the attention formulation.
learns an interpolation over the query time points by attending to the observed values at key time points. Our proposed method is thus similar to kernel- based interpolation, but learning the time attention based similarity kernel gives our model more flexibility compared to methods like that of Shukla & Marlin (2019) that use similarity kernels with fixed functional forms. Another important difference relative to many of these previous methods that our proposed approach attends only to the observed data dimensions at each time point and hence does not require separate imputation step to handle vector valued observations with an arbitrary collection of dimensions missing any given time point.
3 THE MULTI-TIME ATTENTION MODULE
In this section, we present the proposed Multi-Time Attention Module (mTAN). The role of this module is to re-represent sparse and irregularly sampled time series in a exed-dimensional space. This module uses multiple continuous-ting time embeddings and attention- interpolation. We begin by presenting notation followed by the time embedding and attention components.
Notation:
In the case of a supervised learning task, we let D {SA,, Un Jn)(n 1,..,, N) represent data set containing N data cases. An individual data case consists of single target value yn (discrete for classification), well as D-dimensional, sparse and irregularly sampled multivariate time series Sn. Different dimensions d of the multivariate time series can have observations at different times, well as different total numbers of observations Lan- Thus, represent time series d for data case n as a tuple San (tan. xan) where tun = [iid .... fLa. du is the list of time points at which observations are defined and Xdn [x idn: ..., I Lu. dn. is the corresponding list of observed values. In the case of an unsupervised task such as interpolation, each data case consists of a multivariate time series S,, only. We drop the data case index for brevity when the context is clear.
Time Embedding:
Time attention module is based on embedding continuous time points into a vector space. We generalize the notion of a positional encoding used in transformer-based models to continuous time. Time attention networks simultancously leverage / embedding functions on (f), each outputting a representation of size d,. Dimension of embedding h is defined as follows:
won t + aoh sin(was t o..)
if i= if 0 O<i<d,
du [+]
(1)
where the Win's 's and ass 's are learnable parameters. The periodic terms can capture periodicity in time series data. In this case, Win. and aih represent the frequency and phase of the sine function. The linear term, on the other hand, can capture non-periodic patterns dependent on the progression of time. For a given difference A, 0% A) can be represented as a linear function of (t).
Learning the periodic time embedding functions is equivalent to using one-layer fully connected network with sine function non- lineariyy to map the time values into higher dimensional space. By contrast, the positional encoding used in transformer models is defined only for discrete positions. We note that our time embedding functions subsume positional encodings when evaluated discrete positions.
Multi-Time Attention:
The time embedding component described above takes a continuous time point and embeds it into H different d,-dimensional spaces. In this section, we describe how we leverage time embeddings to produce a continuous-tin time embedding module for sparse and irregularly sampled time series. This multi-time attention embedding module mTAN(t, s) takes input query time point and a set of keys and values in the form of a D-dimensional multivariate sparse and irregularly sampled time series (as defined in the notation section above), and returns a J-
3

---

Published as a conference paper at ICLR 2021
bg
[B...... .bg]
TAND outbut at ghrnn ael relerence poiits
Linen
iTTAND
P..... relorence
ATT
7x
ATT
ATT
ATT
Ray Ri)
Inagularly samp ad mutt vanate time series
Figure I: Architecture of the mTANND module. It takes irregularly sampled time points and corre- sponding values keys and values and produces fixed dimensional representation at the query time points. The attention blocks (ATT) perform scaled dot product attention over the observed values using the time embedding of the query and key time points. Equation 3 and deffine this operation. Note that the output at all query points can be computed in parallel.
dimensional embedding at time t. This process leverages continuous-tima attention mechanism applied to the H time embeddings. The complete computation is described below.
mTAN(t. st]]] ΣΣ tna s) Unds
La
(n)(l,8) = Σ
kn,l, (L, lid) Wid
(3)
1=1
exp (dha (l) (naaaa) /dax)
(n,tttaa)
(4)
Cx=1 exp (o (tv /vd
Li
As shown in Equation 2 dimension j of the mTAAN embedding (t, is given by linear combination of intermediate univariate continuous-time functions (t, s). There is one such function defined for each input data dimension d and each time embedding h. The parameters Unai learnable linear combination weights.
As shown in Equation 3 the structure of the intermediate continuous-time function (1, s) is essen- tially kernel smoother applied the tt dimension of the time series. However, the interpolation weights Ky. (t, tua) are defined based on a time attention mechanism that leverages time embeddings, as shown Equation 4 As we can see, the same time embedding function Φn (t) is applied for all data dimensions. The form of the attention mechanism is a softmax function over the observed time points tu for dimension d. The activation within the softmax time embedding on (1) of the query time point i and the time embedding (tia)) of the observed time
scaled inner product between the
point, the key. The parameters w and v are each d, x dg matrices where dy S d,.. We use a scaling factor Vdz to normalize the dot product to counteract the growth in the dot product magnitude with increase in the dimension dę.
Learning the time embeddings provides our model with flexibility to learn complex temporal kernel functions K₁ (t, t'). The use of multiple simultaneous time embeddings ou (t) and final linear combination across time embedding dimensions and data dimensions means that the final output representation function mTAA((,,s) is extremely flexible. Different input dimensions can leverage different time embeddings via learned sparsity patterns in the parameter tensor U. Information from different data dimensions can also mixed together to create compact reduced dimensional representations. We note that all of the required computations can be parallelized using masking variables to deal with unobserved dimensions, allowing for efficient implementation on a GPU.
Since the mTAN module defines continuous function of t given s, be directly incorporated into neural network architectures that expect inputs in the form of fixed- dimensional vectors or discrete sequences. However, the m TANN module can easily be adapted to
Discretization:
not
4

---

Published as a conference paper at ICLR 2021
Classiffer
latent staree
varitionall distribulion
(x)))))))) o (2ll))))))
0. fra
decoder
Bidirectional RNN
Linear
decoder
encoder RNN
Bidirectional RNN
mTAAND
Iimo points
encoder mTAADD
Linear
reforence points
(2)x 1ps
f]ix tajo
Iregularly sempled lime series
Irreguarly sempled
time series
(a) Generative Model (Decoder)
(b) Inference Network (Encoder)
Figure 2: Architecture of the proposed encoder-decoder framework mTAND-Full, The classifier is required only for performing classification tasks. The m'TNND module is shown in Figure
produce such an output representation by materializing its output at a set of reference time points
[ri...,,"'" rx). In some cases, we may have fixed set of such points. In other cases, the set of reference time points may need to depend on s itself. In particular, we define the auxiliary function p(s) to return the set of time points which there an observation on any dimension of s.
Given collection of reference time points we define the discretized mTAN module mTAND(r, s) mTAND(r, [)] mTAA((r,,s) s). This module takes as input the set of reference time points and the time series s and outputs a sequence of mTAN embeddings of length |r), each of dimension J. The architecture of the mTAND module is shown in Figure [ The mTANN module can be used to interface sparse and irregularly sampled multivariate time series data with any deep neural network layer type including fully-connected, recurrent, and convolutional layers. In the next section, we describe the construction of a temporal encoder-decoder architecture leveraging the mTAND module, which can be applied to both classification and interpolation tasks.
4 ENCODER- -DCCDDE FRAMEWORK
As described the last section, we leverage the discretized mTAN module in an encoder-decoder framework the primary model in this paper, which we refer to mTAN network. We develop the encoder-decoder framework within the variational autoencoder (VAE) framework this section. The architecture for the model framework is shown in Figure
Model Architecture: As we are modeling time series data, we begin by defining a sequence of latent states z, Each of these latent states are IID-distributed according to standard multivariate normal distribution plz.). We define the set of latent states z = [2),- .... z at K reference time points.
We define three-stage decoder. First, the latent states processed through RNN decoder module to induce temporal dependencies resulting in a first set of deterministic latent variables RNN [hdee [ .... dee N. Second, the output of the RNN decoder stage and the K time dec RRN are provided to the mTAND module along with a set Tf query time points t. The mTAND module outputs a sequence of embeddings h = [h A... hfre AND length |t]. Third, the mTAN embeddings are independently decoded using fully connected decoder fdee() and the result is used to parameterize an output distribution. In this work, we use a diagonal covariance Gaussian distribution with mean given by the final decoded representation and a fixed variance
points
o². The final generated time series is given by $ = (t, x) with all data dimensions observed. The full generative process is shown below. We let po px|2,, t.) define the probability distribution
5

---

Published as a conference paper at ICLR 2021
the values of the time series given the time points t and the latent variables z. A represents the parameters of all components of the decoder.
Z₁: ~ p(z+)
(5)
h dec
RNN RNNNoo (z)
(6)
TAN mTANNdDe (t, RNN)
(7)
Lid ~ (a f fdee (h o²
(8)
For an encoder, we simply invert the structure of the generative process. We begin by mapping the input time series s through the mTAND module along with a collection of K reference time points We apply an RNN encoder to the mTANND model that outputs h to encode longer-range temporal structure. Finally,
construct distribution over latent variables at each reference time point using a diagonal Gaussian distribution with mean and variance output by fully connected layers applied to the RNN outputs RNN- N- The complete encoder architecture is described below. We define q, (z|r, s) to be the distribution over the latent variables induced by the input time series s and the reference time points r. y represents all of the parameters in all of the encoder components.
TAN mTAAN (r,s)
(9)
HRNA A RNN TMC (hTAN)
(10)
Zi. ~ 97 (2k Pk, R) μ4 = fa (h Enc gRNN)) of = cexp(fance (aefaanChhhhhaa one (gNRN))
(11)
Unsupervised Learning: To learn the parameters of our encoder-decoder model given data set of sparse and irregularly sampled time series, we follow a slightly modified VAE training approach and maximize normalized variational lower bound on the log marginal likelihood based on the evidence lower bound ELBO, The learning objective is defined below where pe (a (z, tn) and q,y (z/rr, s.) are defined in the previous section.
N
LNVAE(0,7) =
Σ n=1
( (Bw..terr...))ooooo 2, t.)) - DKL(a (2/r, (2(1, 8.) sw)lp(()))
(12)
Ed Lidn
K
(2)1,
DKL. (2,1,,)))
(13)
1m1
Junn
logpo(Xnlzz bn)
ΣΣ
log pe (J (z, Lidn)
(14)
Since irregularly sampled time series can have different numbers observations across different dimensions as well as across different data cases, it can be helpful to normalize the terms in the standard ELBO objective to avoid the model focusing more on sequences that are longer at the expense of sequences that shorter. The objective above normalizes the contribution of each data case by the total number of observations it contains. The fact that all data dimensions are not observed all time points is accounted for in Equation 14 In practice, distribution (z) (2)r,8,) 8,) to compute the learning objective.
k samples from the variational
We can also augment the encoder-decodes model with a supervised learning component that leverages the latent states as feature extractor. We define this component to be of the form pa (8 (z) where ō are the model parameters. This leads an augmented learning objective as shown in Equation 15 where the X term trades off the supervised and unsupervised terms.
Supervised Learning:
Suupervised (0, 7, 5) = LNVAE(O, 7) XE (xlr.s..) log (g..lz))))))
(15)
In this work, we focus on classification as an illustrative supervised learning problem. For the classification model ps Ps( Iz), we use a GRU followed by 2-layer fully connected network. We use a small number of samples to approximate the required intractable expectations during both learning and prediction. Predictions are computed by marginalizing over the latent variable shown below.
y" arg max E (e)r,)) [log ps (s/lz)
(16)
vey

---

Published as a conference paper at ICLR 2021
5 EXPERIMENTS
In this section, we present interpolation and classification experiments using a range of models and three wealdworld data sets (Physionet Challenge 2012, MIMIC-III, and Human Activity dataset). Additional illustrative results on synthetic data can be found in Appendix A.2
Datasets: The PhysioNet Challenge 2012 dataset (Silva et al., 2012) consists of multivariate time series data with 37 variables extracted from intensive care unit (ICU) records. Each record contains sparse and irregularly spaced measurements from the first 48 hours after admission ICU. We follow the procedures of |Rubanova er al. (2019) and round the observation times to the nearest minute. This leads to 2880 possible measurement times per time series. The data includes 4000 labeled instances and 4000 unlabeled instances. We use all 8000 instances for interpolation experiments and the 4000 labeled instances for classification experiments. We focus on predicting in-hospital mortality. 13.8% of examples are in the positive class.
The MIMIC-III data set (Johnson et al. 2016) is multivariate time series dataset consisting of sparse and irregularly sampled physiological signals collected at Beth Israel Deaconess Medical Center from 2001 to 2012. Following the procedures (f & Marlin (2019), we extract 53, 211 records each containing 12 physiological variables. We focus on predicting in-hospital mortality using the first 48 hours of data. 8.1% of the instances have positive labels.
The human activity dataset consists of 3D positions of the waist, chest and ankles collected from five individuals performing various activities including walking, sitting, lying, standing, etc. We follow the data preprocessing steps of Rubanova et al. (2019) and construct dataset of 6, 554 sequences with 12 channels and 50 time points. We focus on classifying each time point in the sequence into one of eleven types of activities.
Experimental Protocols: We conduct interpolation experiments using the 8000 data cases in the PhysioNet data set. We randomly divide the data set into training set containing 80% of the instances, and a test set containing the remaining 20% of instances. We use 20% of the training data for validation. In the interpolation task, we condition on subset of available points and predict values for rest of the time points. We perform interpolation experiments with a varying percentage of observed points ranging from 50% to 90% of the available points. At test time, the values of observed points are conditioned on and each model is used to infer the values at rest of the available time points in the test instance. We repeat each experiment five times using different random seeds to initialize the model parameters. We assess performance using mean squared error (MSE).
We use the labeled data in all three data sets to conduct classification experiments. The PhysioNet and MIMIC III problems are whole time series classification problems. Note that for the human activity dataset, we classify each time point in the time series. We treat this as a smoothing problem and condition on all available observations when producing the classification at each time-point (similar to labeling in a CRF). We use bidirectional RNNs as the RNN-based baselines for the human activity dataset. We randomly divide each data set into a training set containing 80% of the time series, and a test set containing the remaining 20% of instances. We use 20% of the training set for validation. We repeat each experiment five times using different random seeds to initialize the model parameters. Due to class imbalance in the Physionet and MIMIC-III data sets, we assess classification performance using area under the ROC curve (the AUC score). For the Human Activity dataset, we evaluate models using accuracy.
For both interpolation and prediction, we select hyper-parameters on the held-out validation set using grid search, and then apply the best trained model to the test set. The hyper-parament parameter ranges searched for each model/dataset/task are fully described in Appendix
The model we focus on is the encoder-decoder architecture based on the discretized multi- time attention module (mTTADDFFul). In the classification experiments, the hidden state at the last observed point is passed to a two-layer binary classification module for all models. For each data set, the structure of this classifier is the same for all models. For the proposed model, the sequence of latent states is first passed through a GRU and then the final hidden state is passed through the same classification module. For the classification task only, we consider an ablation of the full model that uses the proposed mTAND encoder, which consists of our mTAND module followed by GRU to extract a final hidden state, which is then passed to the classification module (m mmTAN--Eee We compare several deep learning models that expand recurrent networks accommodate irregular
Models:
7

---

Published as a conference paper at ICLR 2021
Table Interpolation performance versus percent observed time points on PhysioNet
Model
Mean Squared Error (x10
RNN-VAE
13.418
12.39-1 + 0.001
11.887 + 0.005
11.133
0.007
11.470 0.006
L-ODE-RNN
8.132 + 0.020
8.140 + 0.018
8.171 +: 0.030
8.143 :: 0.025
8.402 + 0.022
L.ODE-ODE
6.816
6.798
6.850
mTAND-Full
4.139 = 0.029
4.018 = 0.048
4.157
0.053
4.410
0.149
4.798 + 0.036
Observed %
50%
60%
70%
80%
90%
sampling. We also compare to several encoder-decoder approaches. The full list of model variants is briefly described below. We use Gated Recurrent Unit (GRU) (Chung et al., (0144) module as the recurrent network throughout. Architecture details can be found in Appendix A.3
RNN-Impute: Missing observations replaced with weighted average of last observed measurement within that time series and global mean of the variable across training examples (Che et al. 201 8).
RNN-A,: Input is concatenated with masking variable and time interval A, indicating how long the particular variable is missing.
RNN-Decay: RNN with exponential decay hidden states (Mozer al. 2017 Che et al. 2018
GRU-D: combining hidden state decay with input decay (Che al. 2018).
Phased-LSTM: Captures time irregularity by time gate that regulates access the hidden and cell state of the LSTM (Neil et al. 2016) with forward filling to handle partially observed vectors.
IP-Nets: Interpolation prediction networks, which use several semi-parametric RBF interpolation layers, followed by GRU (Shukla & Marlin 2019).
SeFT: Uses function based approach where all the observations modeled individually before pooling them together using an attention based approach (Horn et al.) 2020).
RNN-VAE: A VAE-based model where the encoder and decoder are standard RNN models.
Uses neural ODEs to model hidden state dynamics and an RNN to update the hidden state in presence of new observation (Rubanova al. 2019).
ODE-RNN:
L--OD--RN:: Latent ODE where the encoder is RNN and decoder is neural ODE (Chen ct al. 2018).
L-ODE-ODE: Latent ODE where the encoder is an ODE-RNN and decoder is a neural ODE (Rubanova al. 2019).
Physionet Experiments: Table 1 compares the performance of all methods the interpolation task where we observe 50% 90% of the values in the test instances. As we can see, the proposed method (mTAND-Full) consistently and substantially outperforms all of the previous approaches across all of the settings of observed time points. We note that in this experiment, different columns correspond to different setting (for example, in the case of 70%, condition on 70% of data and predict the rest of the data; i.e., 30%) and, hence the results across columns are not comparable.
Table compares predictive performance on the PhysioNet mortality prediction task. The full Multi- Time Attention network model (mTAND-Full) and the classifier based only on the Multi-Time Attention network encoder (mTTAN--EL achieve significantly improved performance relative the current state-of-thh-arr methods (ODE-RNN and --DE--DE)) and other baseline methods.
We also report the time per epoch in minutes for all the methods. We note that the ODE-based models require substantially more run time than other methods due to the required use of an ODE solver (Chen et al., 2018: Rubanova al., 2019). These methods also require taking the union of all observation time points in batch, which further slows down the training process. As we proposed full Multi- Time Attention network (mTAAN-FFlll over 85 times faster than ODE-RNN and over 100 times faster than L-ODE-ODE, the best perforrirg ODE-based models.
the
MIMIC-III Experiments: Table 2 compares the predictive performance the models on the mortality prediction task MIMIC- III. The Multi-Time Attention network- based encoder-decode framework (mTAND-Full) achieves better performance than the recent IP-Net and SeFT model as well as all of the RNN baseline models. While ODE-RNN and L-ODE-ODE both have slightly better
8

---

Published as a conference paper at ICLR 2021
Table 2: Classification Performance on PhysioNet, MIMIC-III and Human Activity dataset
AUC Score
Accuracy
Model
time per epoch
PhysioNet
MIMIC-III
Human Activity
RNN-Impute
0.764 1 0.016
0.8249 1 0.0010
0.859 0.004
0.5
RNN-A.
0.787 1 0.014
0.8364 1: 0.0011
0.857 +: 0.002
0.5
RNN-Decay
0,807 = 0.003
0.8392 + 0.0012
0.860 + 0.005
0.7
RNN GRU-D
0.818 = 0.008
0.8270 0.0010
0.862 0.005
0.7
Phased-LSTM
0.836 = 0.003
0.8429 + 0.0035
0.855 + 0.005
0.3
IP-Nets
0.819 + 0.006
0.8390 + (10011
0.869 + 0.007
1.3
ScFT
0.795 + 0.015
0.8485 + 0.0022
0.815 + 0.002
0.5
RNN-VAE
0.515 + 0.040
0.5175 + 0.0312
0.343 + 0.040
2.0
ODE-RNN
0.833 = 0.009
0.8561 0.0051
0.885 0.008
16.5
L-ODE-RNN
0.781 + 0.018
1.7034
0.838 0.004
6.7
L-ODE-ODE
0.829 + 0.004
0.8559
0,870 + 0.028
22.0
n'TAND-Enc
0.854 1 0.001
0.8419 1 0.0017
0.907 0.002
0.1
mTTAND-Full
0.858 1 0.004
0.8544 0.0024
0.910 1 0.002
0.2
mean AUC than mTANDD-uul, the differences are not statistically significant. Further, as shown on the PhysioNet classification problem, mTAANDFull is more than order of magnitude faster than the based methods.
Human Activity Experiments: Table shows that the mTANDD-baeed classifiers achieve signifi- cantly better performance than the baseline models on this prediction task, followed by ODE-based models and IP-Nets.
we demonstrate the effectiveness of learning temporally distributed latent representations with mTANs on a synthetic dataset. We show that mTANs are able to capture local structure in the time series better than latent ODE-based methods that encode to single time point. This property of mTANs helps improve the interpolation performance in terms of mean squared error.
Additional Experiments: In Appendix A.2
We also perform ablation experiments show the performance gain achieved by learning similarity kernels and time embeddings in Appendix A. In particular, we show that learning the time embedding improves classification performance compared using fixed positional encodings. We also demonstrate the effectiveness of learning the similarity kemnel by comparing uses fixed RBF kernels. Appendix A. shows that learning the similarity kernel using the mTAND module performs as well or better than using a fixed RBF kernel.
approach that
DISCUSSION AND CONCLUSIONS
In this paper, we have presented the Multi-Time Attention (mTAN) module for learning from sparse and irregularly sampled data along with VAE-based encoder-decoder model leveraging this module. Our results show that the resulting model performs as well or better than range of baseline and ththeotf models on both the interpolation and classification tasks, while offering training times that are one to two orders of magnitude faster than previous state of the art methods. While in this work we have focused on VAE- meoder-decoder architecture, the proposed mTAN module can be used to provide an interface between sparse and irregularly sampled time series and many different types of deep neural network architectures including GAN-hased models. Composing the mTAN module with convolutional networks instead of recurrent architectures may also provide further computational enhancements due to improved parallelism.
ACKNOWLEDGEMENTS
Research reported in this paper was partially supported by the National Institutes of Health under award numbers 5U01CA229445 and 1P41EB028242.

---

Published as a conference paper at ICLR 2021
REFERENCES
Xiangrui Cai, Jinyang Gao, Kee Yuan Ngiam, Beng Chin Ooi, Ying Zhang, and Xiaojie Yuan. Medical concept embedding with time-aware attention. In IJCAI, 2018.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 8(1):6085, 2018.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. In Advances in Neural Information Processing Systems 31, pp. 6571-6583. 2018.
Junyoung Chung, Çağlar Gülgehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv e-prints, abs/ 10123555,,55 2014. URL
wwtp/a//g//a...eee//ee 3555 Presented at the Deep Learning workshop at
NIPS2014.
J.S. Clark and O.N. Bjørnstad. Population time series: process variability, observation errors, missing values, lags, and hidden states. Ecology, 85(11):3140-3155 2004.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. cru-ode-bayes: Continuous modeling of sporadically-observe time series. In Advances in Neural Information Processing Systems 32, pp- 7379-7390. Curran Associates, Inc., 2019.
Joseph Futoma, Sanjay Hariharan, and Katherine A. Heller. Learning detect sepsis with multitask gaussian process RNN classifier. Proceedings of the 34th International Conference on Machine Learning, pp. 1 177--1182 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for time series. In Proceedings of the 25th International Conference Machine Learning. 2020.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific Data, 30035, 2016.
S. Kazemi, R. Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, S. Wu, C. Smyth, P. Poupart, and Marcus A. Brubaker. Time2vee: Learning vector representation of time. ArXiv,
2019.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, 2014.
Steven Cheng-Xine Xian Li and Benjamin M Marlin. A scalable end-to-end gaussian process adapter for irregularly sampled time series classification. In Advances In Neural Information Processing Systems, pp. 1804-1812, 2016.
Steven Cheng-Xina Xian and Benjmain M. Marlin. Classification of sparse and irregularly sampled time series with mixtures of expected Gaussian kernels and random features. In 31 Conference on Uncertainty in Artificial Intelligence, 2015.
Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with mns:: Improved classification of clinical time series. Machine Learning for Healthcare Conference, pp. 253-270, 2016.
Benjamin M. Marlin, David C. Kale, Robinder G. Khemani, and Randall C. Wetzel. Unsupervised pattern discovery in electronic health care data using probabilistic clustering models. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium. pp. 389-398, 2012.
Michael C. Mozer, Denis Kazakov, and Robert V. Lindsey. Discrete event, continuous time mns. CoRR, abs/1710.041 10, 2017. URL
10

---

Published as a conference paper at ICLR 2021
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased Istm: Accelerating recurrent network training for long or event-based sequences. In Advances in Neural Information Processing Systems, pp. 3182-888233 3890. 2016.
Xueping Peng, Guodong Long, Tao Shen, Sen Wang. J. Jiang, and Michael Blumenstein. Temporal self-attention network for medical concept embedding. 2019 IEEE International Conference on 2019.
Data Mining (ICDM), pp.
Trang Pham, Truyen Tran, Dinh Phung, and Svetha Venkatesh. Predicting healthcare trajectories from medical records: A deep learning approach. Journal of Biomedical Informatics, 69, 04 2017. doi: 10. 1..006/..bb..211.444...
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference Machine Learning, pp.
2014.
Yulia Rubanova, Ricky T. Q. Chen, and David K Duvenaud. Latent ordinary differential equations for irreuulrty-samppeed time series. In Advances in Neural Information Processing Systems 32, pp- 5330- Curran Associates, Inc., 2019.
T. Ruf. The lomb-scargle periodogram in biological rhythm research: analysis of incomplete and unequally spaced time-series. Biological Rhythm Research, 30(2):178 201, 1999.
Jeffrey D Scargle. Studies in astronomical time series analysis. ii-statistical aspects of spectral analysis of unevenly spaced data. The Astrophysical Journal, 263:835-853 --33, 1982.
M. Schulz and K. Stattegger. Spectrum: Spectral analysis of unevenly spaced paleoclimatic time series. Computers & Geosciences, 23(9):929-945, 1997.
Satya Narayan Shukla and Benjamin M Marlin. Interpolation Prediction Networks for Irregularly Sampled Time Series. ICLR, 2019.
Ikaro Silva, George Moody, Daniel Scott, Leo Celi, and Roger Mark. Predicting in-hospital mortality of icu patients: The physionct/computing in cardiology challenge 2012. Computing in cardiology, 39:245-244 248, 2012.
Huan Song, Deepta Rajan, Jayaraman J. Thiagarajan, and Andreas Spanias. Attend and diagnose: Clinical time series analysis using attention models. In 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pp. 4091-4098. AAAI press, January 2018.
Qingxiong Tan, Mang Ye, Baoyao Yang. Siqi Liu, Andy Jinhua Ma, Terry Cheuk-Fung Yip, Grace Lai-Hung Wong, and Pongchi Yuen. Data-gru: Dual-attention time-aware gated recurrent unit for irregular multivariate time series. In AAAI, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefinedukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 1 International Conference on Neural Information Processing Systems, pp. 6000-6010, 2017.
Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Self-attention with functional time representation learning. In Advances in Neural Information Processing Systems 32, pp. 15915-15925. 2019.
Pranjul Yadav, Michael Steinbach, Vipin Kumar, and Gyorgy Simon. Mining electronic health records (ehrs): A survey. ACM Computing Surveys (CSUR), 50(6):85, 2018.
J. Yoon, W. R. Zame, and M. van der Schaar. Estimating missing data in temporal data streams using multi-directional recurrent neural networks. IEEE Transactions on Biomedical Engineering, 66(5): 1477-1490, 2019, doi: 10.1
Jinsung Yoon, William R Zame, and Mihaela Van Der Schaar. Deep sensing: Active sensing using multi-directional recurrent neural networks. ICLR, 2018.
Yuan Zhang, Xi Yang, Julie Ivy, and Min Chi. Attain: ivention-basede time-aware Istm networks for disease progression modeling. In Proceedings of the Twenty-Eiie Eighth International Joint Conference Artificial Intelligence, pp. 4369-43777 2019.

---

Published as a conference paper at ICLR 2021
A
APPENDIX
A.I
ABLATION STUDY
In this section, we perform ablation experiments show the performance gain achieved by learning similarity kernel and time embedding. Table 3 shows the ablation results by substituting fixed positional encoding (Vaswani et al. 2017) in place of learnable time embedding defined in Equation in mTAND-Full model on PhysioNet and MIMIC-III dataset for classification task. We report the average AUC score over 5 runs. As we can see from Table learning the time embedding improves AUC score by 1% as compared to using fixed positional encodings.
Table 3: Ablation with time emhedding
Dataset
Time Embedding
AUC Score
Positional Encoding
0.845 = 0.004 0.858 0.004
PhysioNet
Learned Time Embedding
Positional Encoding
0.843 0.001
MIMIC-III
Learned Time Embedding
0.854 = 0.002
Since mTANs are fundamentally continuous-ting interpolation braepll models, we perform an ablation study by comparing m'TANs with the IP-nets (Shukla & Marlin 2019). IP-Nets use several semi- parametric RBF interpolation layers, followed by a GRU to model irregularly sampled time series. In this framework, we replace the RBK kernel with learnable similarity kernel using m mTTAND module, the corresponding model is mTAND-Enc. Table 4 compares the performance of the two methods on classification task on PhysioNet. MIMIC-III and Human Activity dataset. We report the average AUC score over 5 runs. Table 4 shows that learning the similarity kernel using mTAND module performs well or better than using fixed RBF kemne..
Table 4: Comparing interpolation kernels
Dataset
Model
AUC Score
IP-Nets
0.819 + 0.006
PhysioNet
mTAND-Enc
0.854 = 0.001
0.839 + 0.001 mTAND-Enc
IP-Nets
MIMIC-III
0.842 = 0.001
IP-Nets
0.869 0.007
Human Activity
mTAND-Enc
0.907 + 0.002
A.2 SYNTHETIC INTERPOLATION EXPERIMENTS
To demonstrate the capabilities of our model on the interpolation task, we generate a synthetic dataset consisting of 1000 trajectories each of 100 time points sampled over t [0, 1]. We fix 10 reference points and use RBF kernel with fixed bandwidth of 100 for constructing local interpolations at 100 time points over [0, 1). The values at the reference points are drawn from a standard normal distribution.
We randomly sample 20 observations from each trajectory to simulate a sparse and irregularly sampled multivariate time series. We use 80% of the data for training and 20% for testing. At test time, encoder conditions on 20 irregularly sampled time points and the decoder generates interpolations on all 100 time points. Figure 3 illustrates the interpolation results on the test set for the Multi-Time Attention Network and Latent ODE model with ODE encoder (Rubanova et al., 2019). For both the models, we draw 100 samples from the approximate posterior distribution. As we 3. the ODE interpolations are much smoother and haven't 't been able to capture the local structure as well as TTANS.
from Figure
12

---

Published as a conference paper at ICLR 2021
x
Observed data
Ground truth
Reconstructions
10
x
Figure 3: Interpolations the synthetic interpolation dataset. The columns represent 3 different examples. First row: Ground truth trajectories with observed points, second row: reconstructions on the complete range [0, 1] using the proposed model mTAN, third row: reconstructions on the complete range E [0, 1] using the Latent ODE model with ODE encoder.
Table 5: Synthetic Data: Mean Squared Error
Latent Dimension
Model
Reconstruction
Interpolation
L-ODE-ODE
0.0209
0.0571
10
mTTAND-ulll
0.0088
0.0409
L-ODE-ODE
0.0191
0.0541
20
m TTAND-Full
0.0028
0.0335
Table compares the proposed model with best performing baseline Latent-ODE with ODE encoder (L-ODE-ODE) reconstruction and interpolation task. For both the tasks, we condition on the 20 irregularly sampled time points and reconstruct the input points (reconstruction) and the whole set of 100 time points (interpolation). We report the mean squared error on test set.
A.3 ARCHITECTURE DETAILS
Multi-Time Attention Network (mTAND-Full): In our proposed encoder-decoder framework (Figure 2. we use bi-directional GRU as the recurrent model in both encoder and decoder. In encoder, we use a layer fully connected network with 50 hidden units and ReLU activations to map the RNN hidden state at each reference point to mean and variance. Similarly in decoder, m'TAN embeddings are independently decoded using activations, and the result is used parameterize the output distribution. For classification tasks, use a separate GRU layer on top of the latent states followed by 2-layer fully connected layer with 300 units and ReLU activations to output the class probabilities.
layer fully connected network with 50 hidden units and ReLU
Multi-Time Attention Encoder (mTAND-Enc): As we show in the experiments, the proposed mTAN module can standalone be used for classification tasks. The mTAND-Ene consists of Time attention module followed by GRU to extract the final hidden state which is then passed to a 2-leyer fully connected layer output the class probabilities.
Multi-
13

---

Published as a conference paper at ICLR 2021
Loss Function: For computing the evidence lower bound (ELBO) during training, log- log-likelihood with fixed variance as the reconstruction loss. For all the datasets, we use a fixed variance of 0.01. For computing ELBO, we use 5 samples for interpolation task and 1 sample for classification tasks. We use cross entropy loss for classification. For the classification tasks, we tune the À parameter in the supervised learning loss function (Equation 15). We achieved best performance using À as 100 and 5 for Physionet, MIMIC-III respectively. For human activity dataset, we achieved best results without using the regulaizer ELBO component. We found that KL annealing with coeff 0.99 improved the performance of interpolation and classification tasks on Physionet.
negative
A.4 HYPERPARAMETERS
Baselines: For Physionet and Human Activity dataset, we use the reported hyperparameters for RNN baselines well as ODE models from Rubanova al. (2019). For MIMIC-III dataset, independently tune the hyperparameters of the baseline models on the validation set. We search for GRU hidden units, latent dimension, number of hidden units in fully connected network for ODE function in recognition and generative model over the range {20, 32, 64, 128, 256). For ODEs, we also searched the number of layers in fully connected network in the range 2.3..
mTAN: We learn time embeddings of size 128. The number of embeddings Н E (1,2,4). The linear projection matrices used for projecting time embedding are each dę- dg/h where dę is the embedding size. We search the latent dimension and GRU encoder hidden size over the range {32, 64, 128). We keep GRU decoder hidden size at 50. For the classification tasks, we use 128 reference points. For interpolation task, we search number of reference points over the range {8, 16, 32, 64, 1283. We use Adam Optimizer for training the models. For classification, experiments for 300 iteration with learning rate 0.0001, while for interpolation task experiments for 500 iterations with learning rate 0.001. Best hyperparameters are reported in the code.
A.5 VISUALIZING ATTENTION WEIGHTS
Regularly sampled time points
Irregulrlyy sampled time points
Figure 4: Visualization of attention weights. mTAN learns an interpolation over the query time points by attending to the observed values at key time points. The brighter edges correspond to higher attention weights.
In this section, we visualize the attention weights learned by our proposed model. We experiment using synthetic dataset (described A.2 which consists of univariate time series. Figure ssguss shows the attention weights learned by the encoder mTAND module. The input shown in the figure is the irregularly sampled time points and the edges show how the output reference points attends to the values on the input time points. The final output can be computed by substituting the attention weights in Equation
A.6 TRAINING DETAILS
A.6.1 DATA GENERATION AND PREPROCESSING
All the datasets used in the experiments are publicly available and can be downloaded using the following links:
MIMIC-III: htt.ll/hh...pp/mmmmmm...sssseeee
14

---

Published as a conference paper at ICLR 2021
Human Data-foo--ersoo--cciivyy
We rescale each feature be between 0 and for Physionet and MIMIC-III dataset. We also rescale the time to be in [0, 1] for all datasets. In case of MIMIC-II dataset, for the time series missing entirely, follow the preprocessing steps of Shukla & Marlin (2019) and assign the starting point (time t=0) value of the time series to the global mean for that variable.
A.6.2 SOURCE CODE
The code reproducing the results in this paper available httpp::/github.coo com/
A.6.3 COMPUTING INFRASTRUCTURE
All experiments were run Nvidia Titan X GPU.
15


Reviews:
[{"review_id": "lrbv169PyD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new model (mTANs) for sparse and irregularly sampled multivariate time series. It incorporates the time attention mechanism to learn embedding for continuous time-series based on a kernel smoothing method. Results on real-world dataset such as EHR data has outperformed other baselines.\n\nI have the following comments that I think is worthwhile to consider to improve the paper:\n\n1. I didn't see much on the main method sections that specifically address/discuss sparsity and irregular sample problems, it seems like the proposed method also works for general time-series data.\n\n2. In the discretization part, it would be better to make it clear how to find the set of reference time points r in practice. It may not be efficient enough if one needs to track each time stamp for each variable to obtain this set $r$.\n\n3. How does an extra module of mTAND in encoding and decoding procedure help intuitively? Given that RNN has already captured temporal information.\n\n4. The noise distribution for sparse / irregularly sampled data can often be heavily skewed, so the Gaussian noise assumption in this paper may not hold.\n\nAdditional questions/suggestions:\n1. It is better to specify how to compute the gradient for ELBO, e.g., are there any approximations used, are methods like REINFORCE or Gumbel softmax implemented for non-continuous cases, as well as how does the parameter initialize for unsupervised learning problems.\n\n2. Does the unified supervised and unsupervised objective in Eq (15) mean the proposed method combines imputation/interpolation with learning/inference for time series data?\n\n3. From the time series perspective, the positional encoding in Eq (1) is more like a season and trend decomposition, not sure if this plays a similar role here.\n\n4. It would be very helpful for me to understand the paper if the authors could add a schematic figure to demonstrate the complete structure of the model.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fekomknc7AH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel deep learning framework for handling irregularly sampled time series. \nThe paper is well-written and easy to follow. \nThe key idea of the paper is to learn embeddings for continuous time values and leveraging a time attention mechanism to learn temporal similarity from data instead of using fixed kernels. \nThe time embedding component takes a continuous time point and embeds it into multiple fixed-dimensional spaces. The multi-time attention mechanism takes a query time t and a multivariate sparse and irregularly sampled time series, and returns a fixed dimensional embedding for the query time t. This mechanism is used twice in an encoder-decoder VAE framework with varying reference (input) and query (output) time points.\n\nThe idea of the paper is novel, impactful, and well-explained. \nThe evaluation and benchmarking is proper with significant improvements over the baselines for classification and interpolation tasks. The approach has significant computational gains over the best performing baseline making it more useful in practice while achieving better or similar classification/interpolation performance.\n\nOne query I had is regarding the application of the proposed framework for extrapolation or forecasting tasks. Is the framework directly applicable to such tasks given the way time is handled to get the embeddings as the 0th dimension would keep growing with time (Eqn. 1)? This can have practical implications or challenges even in variable length classification tasks where longer duration time series can be present at test time.\n\nTypos:\ntime series data data\nwith mroe than one\nIn sec 4, input and the generated time series are both denoted by vector s.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SnwZ2GXGuJG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel approach to learn an embedding of continuous time values and use an attention mechanism to produce a fixed-length representation of a time series containing a variable number of observations. In particular, it proposes an mTAN network to leverage the mTAN module in an encoder-decoder framework for both unsupervised and supervised Learning.  The main contribution of this paper is the introduction of Multi-Time Attention Networks to learns a time representation and learns to attend to observations at different time points by computing a similarity weighting by the learning time embedding.  Empirical studies are performed to show the superiority of the proposed model mTANs over several baseline approaches on the tasks unsupervised and supervised learning. \n\nPros: \n \n1. The paper proposes a novel model to learns a continuous time representation and adapt to fixed dimensional vectors or discrete sequences. For me, the problem itself is real and practical. \n2. The proposed mTAN is novel for capturing the time dependencies time-series, sparse, irregularly sampled, and multivariate data. \n3. This paper provides comprehensive experiments, including both unsupervised and supervised learning results, to show the effectiveness of the proposed framework.  \n\nCons: \n \n1.\tThe paper uses a lot of notations in equations and descriptions, which cause a little bit confusion to follow the authors’ idea. Please consider providing a table to list all the import symbols.\n2.\tIt is better to depict this proposed model structure give the audience a main picture to model.\n3.\tAs the model is attention-based, it has the ability to find the relationships among sequential events. Is it possible to provide one or two case studies to demonstrate the dependencies between time points?\n \n\nQuestions during rebuttal period: \n \nPlease address and clarify the cons above \n\nMinor comments: \n1.A few references only list authors, title, and year, but miss publisher or conference, such as “ Michael Mozer, Denis Kazakov, and Robert Lindsey. Discrete event, continuous time rnns. 2017“\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6S1Tt6Vb1H7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This article tackles the analysis of irregular samples time series. The approach is mainly based on interpolation. Thus, the authors can del with both supervised and unsupervised problems.\nThe architecture is made of a sinusoid attention layer, a VAE layer that lead to a fixed size set of landmark in the latent space and a RNN decoder.\nFor the supervised task, the authors add a classification loss.\n\nThey obtain impressive results on the interpolation task and interesting results on the classification task.\n\n* In an interpolation problem, we would like to consider a robust baseline as a linear interpolation or an AR-like modeling. Even if I must admit that the authors already propose a lot of comparisons with models from the -recent- litterature, this would give us a meaningful MSE result to compare other approaches.\n\n* notations should be improved (and/or completed with a schema). The model is really very difficult to understand in this version of the article.\n\n* Results are impressive but I don't get which part of the architecture lead to such a performance\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work proposed a new approach to encode time series that are irregularly sampled and multivariate using time attention module and an encoder-decoder framework based on VAE. All the reviewers find the approach novel and the experiments extensive with encouraging results. Please continue to improve the presentation of the paper. I would  suggest to move the diagram showing the overall architecture to the main text to assist the explanation. Reviewers also would like to see more explanation on the experimental results and some ablation studies to show the importance of each component of the proposed architecture. 
Decision: Accept (Poster)