Heteroscedastic Temporal Variational Autoencoder For Irregularly Sampled Time Series
Authors: Satya Narayan Shukla
Benjamin Marlin
Venue: ICLR 2022 Poster (2022)
Keywords: irregular sampling
uncertainty
imputation
interpolation
multivariate time series
missing data
variational autoencoder

Abstract: Irregularly sampled time series commonly occur in several domains where they present a significant challenge to standard deep learning models. In this paper, we propose a new deep learning framework for probabilistic interpolation of irregularly sampled time series that we call the Heteroscedastic Temporal Variational Autoencoder (HeTVAE). HeTVAE includes a novel input layer to encode information about input observation sparsity, a temporal VAE architecture to propagate uncertainty due to input sparsity, and a heteroscedastic output layer to enable variable uncertainty in the output interpolations.  Our results show that the proposed architecture is better able to reflect variable uncertainty through time due to sparse and irregular sampling than a range of baseline and traditional models, as well as recently proposed deep latent variable models that use homoscedastic output layers.

Published as a conference paper at ICLR 2022
HETEROSCEDASTIC TEMPORAL VARIATIONAL AUTOENCODER FOR IRREGULAR TIME SERIES
Satya Narayan Shukla & Benjamin M. Marlín College of Information and Computer Sciences University of Massachusetts Amherst Amherst, MA 01003, USA
(anshukla, marlin ) Ccs. umass .ddu
ABSTRACT
Irregularly sampled time series commonly occur in several domains where they present significant challenge to standard deep learning models. In this paper, we propose a new deep learning framework for probabilistic interpolation of irregu- larly sampled time series that call the Heteroscedastic Temporal Variational Autoencoder (HeTVAE). HeTVAE includes a novel input layer to encode infor- mation about input observation sparsity, temporal VAE architecture to propa- gate uncertainty due to input sparsity, and a heteroscedastic output layer to enable variable uncertainty in output interpolations. Our results show that the proposed architecture is better able to reflect variable uncertainty through time due to sparse and irregular sampling than a range of baseline and traditional models, as well as recent deep latent variable models that use homoscedastic output layers]
INTRODUCTION
In this paper, propose novel deep learning framework for probabilistic interpolation of ir- regularly sampled time series. Irregularly sampled time series data occur in multiple scientific and industrial domains including finance (Manimaran al. 2006), climate science (Schulz & Stattegger 1997] and healthcare (Marlin et al.] 2212) Yadav et al. 2018). In some domains including electronic health records and mobile health studies (Cheng et al.) 2017), there can be significant variation in inter-observation intervals through time. This is due to the complexity of the underlying observation processes that can include "normal" variation in observation times combined with extended, block- structured periods of missingness. For example, in the case of ICU EHR data, this can occur due to patients being moved between different locations for procedures or tests, resulting in missing phys- iological sensor data for extended periods of time. In mobile health studies, the same problem can occur due to mobile sensor batteries running out, or participants forgetting to wear or carry devices.
In such situations, it is of critical importance for interpolation models to be able to correctly reflect the variable input uncertainty that results from variable observation sparsity so as not to provide overly confident inferences. However, modeling time series data subject to irregular sampling poses a significant challenge to machine learning models that assume -ully-observed, fixed-size feature representations (Marlin et al., 20122] Yadav al. 2018 Shukla & Marlin 2021b). The main chal- lenges in dealing with such data include the presence of variable time gaps between the observation time points, partially observed feature vectors caused by the lack of temporal alignment across different dimensions, dimensions and data cases. Significant recent work has focused on developing specialized models and architectures to address these challenges in modeling irregularly sampled multivariate time se- ries (Li & Marlin| [2015] 2016 [ipton et al.] [016]] Futoma et al. 2017
well as different data cases, and variable numbers of observations across
(0017/Chr Che et al. [0188 [Shukla & Marlin 2019 Rubanova et al.) 2019 Horn et al. 2020 Li & Marlin 2020 Shukla & Marlin 2021a)
De Brouwer et al. 2019 Tan et al. 2020 Kidger al. 2020
Recently, Shukla & Marlin 22021a) introduced the Multi-Time Attention Network (mTAN) model, variational autocncoder (VAE) architecture for continuous-tim interpolation of irregularly sampled
Implementation available at httpt:tt/////. httggggttbb.oomooeeee www.m//mmmaaa//eaae -1ab/metvaaaaeeeeeeeee

---

Published as a conference paper at ICLR 2022
time series. This model was shown to provide state-of- thr classification and deterministic inter- polation performance. However, like many VAEs, the mTAN architecture produces a homoscedastic output distribution conditioned on the latent state. This means that the model can only reflect un- certainty due to variable input sparsity through variations in the VAE latent state. As we will show, this mechanism is insufficient to capture differences in uncertainty over time. On the other hand, (GPR) methods (Rasmussen & Williams] (006) have the abil-
Gaussian Process Regression-basse ity to reflect variable uncertainty through the posterior inference process. The main drawbacks of GPR-based methods are their significantly higher run times during both training and inference, and the added restriction to define positive definite covariance functions for multivariate time series.
In this work, we propose novel recoder-decoder architecture for multivariate probabilistic time series interpolation that we refer to as the Heteroscedlastic Temporal Variational Autoencoder or HeTVAE. VeTTV aims to address the challenges described above by encoding information about input sparsity using an uncertainty-awart multi-time attention network (UnTAN), flexibly capturing relationships between dimensions and time points using both probabilistic and deterministic latent pathways, and directly representing variable output uncertainty via a heteroscedastic output layer.
The proposed UnTAN layer generalizes the previously introduced mTAN layer with an additional intensity network that can more directly encode information about imput uncertainty due to variable sparsity. The proposed UnTAN layer uses attention mechanism produce distributed latent representation of irregularly sampled time series at a set of reference time points. The UnTAN module thus provides an interface between input multivariate, sparse and irregularly sampled time series data and more traditional deep learning components that expect ixxeddiieeniionl or regularly spaced inputs. We combat the presence of additional local optima that arises from the use of heteroscedastic output layer by leveraging an augmented training objective where we combine the ELBO loss with an uncertainty agnostic loss component. The uncertainty agnostic component helps to prevent learning from converging to local optima where the structure in data is explained as noise.
We evaluate the proposed architecture on both synthetic and real data sets, Our approach outper- forms a variety of baseline models and recent approaches in terms of log likelihood, which is our primary metric of interest in the case of probabilistic interpolation. Finally, we perform ablation test- ing of different components of the architecture to assess their impact on interpolation performance.
2 RELATED WORK
Keeping in mind the focus of this work, we concentrate our discussion of related work on determin- istic and probabilistic approaches applicable to the interpolation and imputation tasks.
Deterministic Interpolation Methods: Deterministic interpolation methods can be divided into fil- tering and sroothingg-bseee approaches. Filtering based approaches infer the values at given time by conditioning only on past observations. For example, [Han-Gyu Kim et al. (017)) use a unidi- rectional RNN for missing data imputation that conditions only data from the relative past of the missing observations. On the other hand, smoothing- bbssed methods condition on all possible obser- vations (past and future) infer any unobserved value. For example, Yoon et al. (0118) and Cao et al. 2018) present missing data imputation approach based on multi-directional and bi-directional RNNs. These models typically use the gated recurrent unit with decay (GRU-D) model (Che al., networks lake different approach to interfacing with irregularly sampled data that is based on the use of temporal kernel smoother- based layers (Shukla & Marlin 2019). Shan & Oliva (202I) propose hi- erarchical imputation strategy based on set-based architectures for imputation in irregularly sampled time series. Of course, the major disadvantage of deterministic interpolation approaches is that they do not express uncertainty over output interpolations and thus can not be applied to the problem of probabilistic interpolation without modifications.
2018a) as a base architecture for dealing with irregular sampling. Interpolation-predictin
The two primary building blocks for probabilistic interpola- tion and imputation of multivariate irregularly sampled time series are Gaussian processes regression (GPR) (Rasmussen & Williams 2006) and variational autoencoders (VAEs) (Rezende et al., 2014] Kingma & Welling, 2014). GPR models have the advantage of providing an analytically tractable full joint posterior distribution over interpolation outputs when conditioned on irregularly sampled input data. Commonly used covariance functions have the ability translate variable input obser-
Probabilistic Interpolation Methods:
2

---

Published as a conference paper at ICLR 2022
vation density into variable interpolation uncertainty. GPR-based models have been used as the core of several approaches for supervised learning and forecasting with irregularly sampled data (Ghas- semi et al. 2015 Li & Marlin 2015 2016: Futoma et al.) 2017). However, GPR-based models can become somewhat cumbersome in the multivariate setting due to the positive definiteness con- straint on the covariance function (Rasmussen & Williams 2006). The use of separable covariance functions is one common approach to the construction of GPR models over multiple dimensions (Bonilla et al., 2008), but this construction requires all dimensions to share the same temporal kernel parameters. A further drawback of GP-based methods is their significantly higher run times relative to deep learning- based models when applied to larger-scale data (Shukla & Marlin (2019).
Variational autoencoders (VAEs) combine probabilistic latent states with deterministic encoder and decoder networks to define flexible and computationally efficient class of probabilistic models that generalize classical factor analysis (Kingma& & Welling (0144] Rezende et al. 2014). Recent research has seen the proposal of several new VAE-based models for irregularly sampled time series. CCen et al. (2018) proposed latent ordinary differential equation (ODE) model for continuous-time data using an RNN encoder and a neural ODE decoder. Building on the prior work of Chen et al. (2018), Rubanova et al. (2019) proposed latent ODE model that replaces the RNN with ODE-RNN model as the encoder. Li et al. (2020) replace the deterministic ODEs with stochastic differential equations(SDEs). Norcliffe al. (202T) extends the prior work on neural ode by combining it with neural processes (Garnelo et al. 2018).. [Shukla & Marlin (2021a) proposed the Multi-Time Attention Network (mTAN) model, a VAE-based architecture that uses multi-head temporal cross attention encoder and decoder module (the mTAND module) to provide the interface to multivariate irregularly sampled time series data. Fortuin al. (2020) proposed a VAE-based approach for the task of smoothing in multivariate time series with a Gaussian process prior in the latent space to capture temporal dynamics. Garnelo et al.] (2018); Kim et al. (2019) used heteroscedastic output layers to represent uncertainty in case of fixed dimensional inputs but these approaches are not applicable to irregularly sampled time series.
Similar to the m'TAN model, the Heteroscedastic Temporal Variational Autoencoder (HeTVAE) model proposed in this work is an attention- VAE architecture. The primary differences are that mTAN uses a homoscedastic output distribution that assumes constant uncertainty and that the mTAN model's cross attention operation normalizes away information about input sparsity. These limitations are problematic in cases where there is variable input density through time resulting in the need for encoding, propagating, and reflecting that uncertainty in the output distribution. As we describe in the next section, HeTVAE addresses these issues by combining a novel sparsity-sensitive encoder module with a heteroscedastic output distribution and parallel probabilistic and determinis- tic pathways for propagating information through the model. Another important difference relative to these previous methods is that HeTVAE uses an augmented learning objective to address the underfitting of predictive variance caused by the use of the heteroscedastic layer.
3 PROBABILISTIC INTERPOLATION WITH THE HETVAE
In this section, we present the proposed architecture for probabilistic interpolation of irregularly sampled time series, the Heteroscedastic Temporal Variational Autoencoder (HeTVAE). HeTVAE leverages a sparsity-aware layer as the encoder and decoder in order to represent input uncertainty and propagate it to output interpolations. We begin by introducing notation. We then describe the architecture of the encoder/decoder network followed by the complete HeTVAE architecture.
3.1 NOTATION
We let D = 1= N) represent a data set containing N data cases. An individual data case consists of a aaaaDDDaa D-dimensional, sparse and irregularly sampled multivariate time series s,, Different dimensions d of the multivariate time series can have observations at different times, as well as different total numbers of observations Lan- We follow the series-based representation of irregularly sampled time series (Shukla& & Marlin 2021b) and represent time series d for data case n as a tuple San (tan xan) where tun [idn .... Lan is the list of time points at which observations defined and x,t.. = [z] idn ...; 2 Ing.,d] dnl is the corresponding list of observed values. We drop the data case index for brevity when the context is clear.
3

---

Published as a conference paper at ICLR 2022
3.2 REPRESENTING INPUT SPARSITY
As noted in the previous section, the mTAN encoder module does not represent information about input sparsity due to the normalization of the attention weights. To address this issue, we propose an augmented module that we refer to (UnTAN). The UnTAN module is shown in Figure Ia This module includes two encoding pathways that leverage shared time embedding function and a shared attention function. The first encoding pathway (the intensity pathway, INT) focuses on representing information about the sparsity of observations while the second encoding pathway (the value pathway, VAL) focuses on representing information about values of observations. The outputs of these two pathways are concatenated and mixed via a linear layer to define the final output of the module. The mathematical description of the module is given in Equations T to and is explained in detail below.
Uncertainty Aware Multi-Time Attention Network
(rta(rk, ta) pool (lexp(an (rk, (ra)) bea E ta)) pool (fox (rw)) tro E tt)))
(1)
(rk, Xa) (fop((ex((nn(Tes laa)) aid lia E ta, E xa))
(2)
pool (foxp((((oon tra)) tia E tal)
(n(t,t))= = (dN(()wr" on
(3)
dde
Time Embeddings and Attention Weights: Similar to the mTAAN module, the UnTAN module uses time emhedding functions dh (t) to project univariate time values into higher dimensional space. Each time embedding function is a one-layer fully connected network with a sine function non-linearity (t) sin(w- + 3). We learn H time embeddings each of dimension de. w and are the parameters of the scaled dot product attention function a% (t, t') shown in Equation 3 The scaling factor 1/ de is used to normalize the dot product to counteract the growth in the dot product magnitude with increase in the time embedding dimension de.
The intensity encoding pathway is defined by the function int, (Tk, ta) shown in Equation| [1] The inputs to the intensity function are query time point T'k and a vector ta con- taining all the time points at which observations are available for dimension the intensity function exponentiates the attention weights between тąK and each time point in t, to ensure positivity, then pools over the observed time points. The denominator of this computation identical to the numerator, but the set of time points t,, that is pooled over is the union over all observed time points for dimension d from all data cases.
Intensity Encoding:
The numerator of
Intuitively, if the largest attention weight between F and any element of t, is small relative to attention weights between r2 and the time points in t.. then the output of the intensity function will be low. Importantly, due the use of the non-linear time embedding function, pairs of time points with high attention weights do not necessarily have to be close together in time meaning the notion of intensity that the network expresses is significantly generalized.
We also note that different sets could be used for t. including regularly spaced set of reference time points. One advantage of using the union of all observed time points is that fixes the maximum value of the intensity function at 1. The two pooling functions applicable in the computation of the intensity function are max and sum. If the time series sparse, maz works well because using sum in the sparse case can lead to very low output values. In more densely observed time series, either sum or maz can be used.
Value Encoding: The value encoding function val, (Tk. ta. xx) is presented in Equation 2 in a form that highlights the symmetry with the intensity encoding function. The primary differences are that val, (rk, Eg. Xa) takes as input both observed time points t and their corresponding values Xa, and the denominator of the function pools over ta itself. While different pooling options could be used for this function, in practice we use sum-based pooling. These choices lead to a function val,, ta, x2) that interpolates the observed values at the query time points using softmax weights derived from the attention function. The values of observed points with higher attention weights con- tribute more to the output value. This structure is equivalent to that used in the m module when num-based pooling is used. We can also clearly see that this function on its
not represent
4

---

Published as a conference paper at ICLR 2022
Mexn
Veriance
VLL
Un UnTAND
Cencal
Proh
Sample
MLP
ML.P
UnTAND
(a) Uncertainty Aware Multi-Tima Time Attention Networks (b) Heteroscedastic Temporal VAE
Figure I: (a) Architecture of UnTAN module. This module takes D-dimensional irregularly sampled time points = [t1,... tp] and corresponding observations x [x1 .xp as keys and values and produces a fixed dimensional representation at the query time points r = ['...... .Tk]. Shared time embedding and attention function provide input to parallel intensity (INT)) and value (VAL) en- coding networks, whose outputs are subsequently fused via concatenation and an additional linear encoding layer. (b) Architecture of HcTVAE consisting of the UnTAND module to represent input uncertainty, parallel probabilistic (Prob) and deterministic (Det) encoding paths, and heteroscedas- tic output layer that aims reflect uncertainty due to input sparsity in the output distribution.
information about input sparsity due to the normalization over t.. Indeed, the function is completely invariant to an additive decrease in of the attention weights 0' (Fk, (4a) or, (Tk, (4d) 8.
The last stage of the UnTAN module concatenates the value and intensity path- way representations and then linearly weights them together to form the final J-dimensional repre- sentation that output by the module. The parameters of this linear stage of the model are U and The value of the jth dimension of the output at a query time point r' is given by Equation 4
Module Output:
hdy"
H1
int, (rkss ta) val, (rk, ta, xa)
1"
T
Thitt hdj Uneel Kdj
UnTAN((K t. xxx]
ΣΣ
Finally, we note that the UnTAN module defines continuous function of given an input time series and hence cannot be directly incorporated into standard neural network architectures. We adapt the UnTAN module produce fully observed limeesdimensional discrete sequences by ma- terializing its output at set of reference time points. Reference time points can be fixed set of regularly spaced time points or may need to depend on the input time series. For a given set of reference time points
[r.,..., rx]. the discretized UnTAN module UnTAND(r, t, x) is defined as UnTAND(r, t, [t]] = UnTAN((r, t, x). This module takes as input the time series = (t, x) and the set of reference time points r and outputs sequence of K UnTAN embeddings, each of dimension J corresponding to each reference point. As described the next section, we use the UnTAND module to provide interface between sparse and irregularly sampled data and fully connected MLP network structures.
3.3
THE HETVAE MODEL
In this section, we describe the overall architecture of the HeTVAE model, as shown in Figure Tb
The HeTVAE consists of parallel deterministic and probabilistic pathways for propagating input information to the output distribution, including information about input spar- sity. We begin by mapping the input time series s = x) through the UrTAND module along with a collection of K reference time points r. In the probabilistic path, construct distribu- tion over latent variables at each reference time point using diagonal Gaussian distribution & with mean and variance output by fully connected layers applied to the UnTAND output embeddings
Model Architecture:
5

---

Published as a conference paper at ICLR 2022
he = [hf embeddings he are passed through feed-forward network g to produce a deterministic temporal representation (at each reference point) of the same dimension as the probabilistic latent state.
[ as shown in Equation 6 In the deterministic path, the UnTAND output
The decoder takes as input the representation from both pathways along with the reference time points and a set of query points t' (Eq The UnTAND module produces sequence of embeddings hd = [ ,... peyl corresponding to each time point in t'. The UnTAND embeddings are then
independently decoded using a fully connected decoder fdec and the result used parameterize the output distribution. We use diagonal covariance Gaussian distribution where both the mean f = [11 ... 11 14; E RDD and variance o² [ap.- _² o of E IR. are predicted for each time point by the final decoded representation as shown in Eq sampled from this distribution and is given by $ = (t',x') x') with all data dimensions observed.
9 The generated time series is
The complete model is described below. We define q- (z)r, s) to be the distribution over the proba- bilistic latent variables z [z1.... Zx] induced by the input time series (t, x) at the reference
time points r. We define the prior p(z:) over the latent states to be a standard multivariate normal distribution. We let phet (xr/a 2 ('dg) define the final probability distribution over the value of
time point lt on dimension d given the concatenated latent state zcaz = | cat] ]. and 0
represent the parameters of all components of the encoder and decoder respectively.
h enc UnTTAN (r, x)
(5)
Zk ~ 4r (2b P2, k)) 1h = (rr0(h (mc), o = = Sene(h
pene (h cnc
(6)
2/ cat = concat ((h
(7)
h dec UnTANDdec dec dt'AN r,, (
(8)
het (2/d 1z ('aa) (N u4 [d], of [d], ui = pdee Idee(h (hde)), o, fdee (hg dee P8
z' ~ pb (#/aa cat scat,t thaa
(10)
Compared to the constant output variance used to train the mTTA-bbaeed VAE model proposed in prior work (Shukla & Marlin 2021ap, our proposed model produces a heteroscedastic output dis- tribution that we will show provides improved modeling for the probabilistic interpolation task.
However, the increased complexity of the model's output representation results in an increased space of local optima. We address this issue using augmented learning objective, as described in the next section. Finally, we note that we can easily obtain a simplified homoscedastic ver- sion of the model with constant output variance of using the alternate final output distribution ph(a/a z, Ma)) = (z i P4 [d], o²).
Augmented Learning Objective: To learn the parameters of the HeTVAE framework given data set of sparse and irregularly sampled time series, we propose an augmented learning objective based normalized version of the evidence lower bound (ELBO) combined with uncertainty agnostic scaled squared loss. We normalize the contribution from each data case by the total number of observations so that the effective weight of each data case in the objective function is independent of the total number of observed values. The augmented learning objective is defined below. fm, is the predicted mean over the test time points as defined in Equation [ Also recall that the concatenated latent state z depends directly on the probabilistic latent state Z.
ENVAE(0.
Σ
(log Eg Lan (mIr.s..)) [log pbrt (x, (2t t. ( sl))(()))) (11)
1
(a ||x, M (mll2)
K
=
DKa.
(12)
log Nogrhet hot (x 12 bn))
log
perron hot (azan l2 Ljan)
(13)
We include the uncertainty agnostic scaled squared loss term to counteract the propensity of the heteroscedastic model to become stuck in poor local optima where the mean is essentially flat and

---

Published as a conference paper at ICLR 2022
Figure 2: We show example interpolations on the synthetic dataset. The set of 3 columns correspond to interpolation results with in- creasing numbers of observed points: 3, 10 and 20 respectively. The first, second and third rows correspond to STGP, HeTVAE and HTVAE m mTAN respectively. The shaded region corresponds to - one standard devia- tion. STGP and Vet exhibit variable out- put uncertainty in response input sparsity while mTAN does not.
Observed data
Ground fruhh
Reconeraction
3
n= 10
20
STEP
NOTVAE
mi7Ah
all of the structure in the data is explained as noise. This happens because the model has the abil- ity to learn larger variances at the output, which allows the mean to underfit the data. The extra component (scaled squared loss) helps to push the optimization process to find more informative parameters by introducing a fixed penalty for the mean deviating from the data, As we will show in the experiments, the use of this augmented training procedure has strong positive impact on final model performance. Since, we are focusing on the interpolation task, we train the HeTVAE by maximizing the augmented learning objective (Equation [T] details on training has been provided in the experimental protocols in Section
the interpolated time points (more
4 EXPERIMENTS
In this section, we present interpolation experiments using range of models three real-world data sets. PhysioNet Challenge 2012 (Silva et al. 2012) and MIMIC-III (Johnson et al., (016) consist of multivariate, sparse and irregularly sampled time series data. We also perform experiments on the Climate dataset (Menne et al.] 2016), consisting of multi-rate time series. We also show qualitative results on synthetic dataset. Details of each dataset can be found in the Appendix [1.6.
We randomly divide the real data sets into a training set containing 80% of the instances, and a test set containing the remaining 20% of instances. We use 20% of the training data for validation. In the interpolation task, we condition on a subset of available points and produce distributions over the rest of the time points. On the real-world datasets, we perform interpolation experiments by conditioning on 50% of the available points. At test time, the values of observed points are conditioned on and each model is used to infer single time point marginal distributions over values at the rest of the available time points in the test instance. In the case of methods that make mean predictions. In the case of the synthetic dataset
Experimental Protocols:
do not produce probabilistic outputs, where we have access to all true values, we use the observed points to infer the values at the rest of the available points. We repeat each real data experiment five times using different random seeds to initialize the model parameters. We assess performance using the negative log likelihood, which our primary metric of interest. We also report mean squared and mean absolute error. For all experiments, we select syper-parameter on the held-out validation set using grid search and then apply the best trained model to the test set. The hyper hypee-parameter ranges searched for each model and dataset are fully described in Appendix A.5
Models: We compare our proposed model HeTVAE to several probabilistic and deterministic in- terpolation methods. We compare to two Gaussian processes regression (GPR) approaches. The most basic GP model for multivariate time series fits one GPR model per dimension. This approach known as single task GP model (STGP) (Rasmussen Williams, 2006). A potentially better option is to model data using a Multi Task GP (MTGP) (Bonilla et al. [008). This approach models the correlations both across different dimensions and across time by defining a kernel expressed as the Hadamard product of temporal kernel (as used in the STGP) and a task kernel. We also com- pare to several VAE-based approaches. These approaches use a homoscedastic output distribution with different encoder and decoder architectures. HVAE RNN employs gated recurrent unit net- work (Chung et al.] (0144) as encoder and decoder, HVAE RNN-ODE (Chen et al. 2018) replaces the RNN decoder with neural ODE, HVAE ODE-RNN-ODE (Rubanova et al.
2019) employs
7

---

Published as a conference paper at ICLR 2022
Table Interpolation performance on PhysioNet.
Model
Negative Log Likelihood
Mean Absolute Error
Mean Squared Error
Mean Imputation
0.7396
0.0000
1.1634 1 0.0000
Forward Imputation
0.4840 0.0000
0.9675 = 0.0000
Single-Tase Task GP
0.7875 = 0.0005
0.4075 0.0001
0.6110 = 0.0003
Multi- Task GP
0.9250 = 0.0040
0.4178 = 0.0007
0.7381 = 0.0051
HVAE RNN
1.5220 + 0.0019
0.7634 + 0.0014
1.2061 + 0.0038
HVAE RNN-ODE
1.4946 + 0.0025
0.7372 0.0026
1.1541 + 0.0058
HVAE ODE-RNN-ODE
1.2906 0.0019
0.5334 0.0020
0.7622 + 0.0027
HTVAE mTAN
1.2426 = 0.0028
0.5056
0.0004
0.7167 + 0.0016
HeTVAE
0.5542 = 0.0209
0.3911 + 0.0004
0.5778 + 0.0020
Table 2: Interpolation performance MIMIC-III
Model
Negative Log Likelihood
Mean Absolute Error
Mean Squared Error
Mcan Imputation
0.7507 = 0.0000
0,9842 + 0.0000
Forward Imputation
0.4902 = 0.0000
0.6148 = 0.0000
Single- Task GP
0,8360 0.0013
0.4167 0.0006
0.3913 = 0.0002
Multi-Task GP
0.8722 + 0.0015
0.4121 0.0005
0.3923 + 0.0008
HVAE RNN
1,4380 + 0.0049
0.7804 + 0.0073
1.0382 + 0.0086
HVAE RNN-ODE
1.3464 + 0.0036
0.6864 + 0.0069
0.8330 + 0.0093
HVAE ODE-RNN-ODE
1,1533 = 0.0256
0.5447 0.0228
0.5642 + 0.0334
HTVAE mTAN
1.0498 0.0013
0.4931 0.0008
0.4844 0.0008
HeTVAE
0.6662 + 0.0023
0.3978 0.0003
0.3716 1 0.0001
ODE-RNY encoder and neural ODE decoder. Finally, compare to HTVAE mANN Shukka & Marlin 2021a). temporal VAE model consisting of multi-time attention networks producing homoscedastic output. For VAE models with homoscedastic output, treat the output variance term as a hyperparameter and select the variance using log likelihood on the validation set. Archi- tecture details for these methods can be found in Appendix A.4 As baselines, we also consider deterministic mean and forward imputation-based methods. Forward imputation always predicts the last observed value on each dimension, while mean imputation predicts the mean of all the observations for each dimension.
Synthetic Data Results: Figure 2 shows sample visualization output for the synthetic dataset. For this experiment, compare HTVAE mTAN, the single task Gaussian process STGP, and the proposed HeTVAE model. We vary the number of observed points (3, 10, 20) and each model is used infer the distribution over the remaining time points. We draw multiple samples from the VAE latent state for HeTVAE and HTVAE mTAN and visualize the distribution of the resulting mixture. As we can see, the interpolations produced by HTVAE mTAN have approximately constant uncertainty across time and this uncertainty level does not change even when the number of points conditioned on increases. On the other hand, both HeTVAE and STGP show variable uncertainty across time. Their uncertainty reduces in the vicinity of input observations and increases in gaps between observations. Even though the STGP has an advantage in this experiment (the synthetic data were generated with an RBF kernel smoother and STGP uses RBF kernel as the covariance function), the proposed model HeTVAE shows comparable interpolation performance. We show more qualitative results in Appendix A.3
the interpolation performance of all the approaches on PhysioNet, MIMIC-III and Climate dataset respectively. HeTVAE outperforms the prior ap- proaches with respect to the negative log likelihood score all three datasets. Gaussian Process based methods - STGP and MTGP achieve second and third best performance respectively. We emphasize that while the MAE and MSE values for some of the prior approaches are close those obtained by the HVAE model, the primary metric of interest for comparing probabilistic interpola- tion approaches is log likelihood, where the HVAV performs much better than the other methods.
Real Data Results:
Tables I
8

---

Published as a conference paper at ICLR 2022
Table 3: Interpolation performance on Climate dataset.
Model
Negative Log Likelihood
Mean Absolute Error
Mean Squared Error
Mean Imputation
0.4539
0.0000
0.8403 1 0.0000
Forward Imputation
0.2979
0.0000
0.8426 + 0.0000
Single-Tass Task GP
0.2478 = 0.0016
0.2738 0.0002
0.4886 = 0.0001
Multi- Task GP
HVAE RNN
1.3666 + 0.0674
0.4838 + 0.0474
0.8587 + 0.0863
HVAE RNN-ODE
1.1769 + 0.0032
0.3514 0.0067
0.6076 + 0.0059
HVAE ODE-RNN-ODE
1.1766 0.0053
0.3531 0.0034
0.5953 + 0.0051
HTVAE mTAN
0.9262 = 0.0073
0.2916 0.0046
0,5162 + 0.0060
HeTVAE
0.1287 = 0.0242
0.2813 + 0.0034
0.5013 + 0.0116
Table 4: Ablation Study of HeTVAE Negative Log Likelihood on PhysioNet and MIMIC-III.
Model
PhysioNet
MIMIC-III
HеTVAE
0.5542 0.0209
0.6662 O. 0.0023
HeTVAE - ALO
0.6087 0.0136
0.6869 + 0.0111
НeTVAE - DET
0.6278 0.0017
0.7478 1 0.0028
НeTVAE INT
0.6539 1 0.0107
0.7430 1 0.0011
НeTVAE- -ETT-LO
1.1304 1: 0.0016
0.9272 1 0.0002
We note that the MAE/MSE of the VAE-based models with homoscedastic output can be improved by using small fixed variance during training. However, this produces even worse log likelihood values. Further, we note that the current implementation MTGP is not scalable to the Climate dataset (270 dimensions). We provide experiments on an additional dataset in Appendix A.1
Ablation Results: Table 4 shows the results of ablating several different components of the HeT- VAE model and training procedure. The first row shows the results for the full proposed approach. The HeTVAE ALO ablation shows the result of removing the augmented learning objective and training the model only using the ELBO. This results in an immediate drop in performance on PhysioNet. HeTVAE DET removes the deterministic pathway from the model, resulting in a per- formance drop on both MIMIC-III and PhysioNet. HvTVAE INT removes the intensity encoding pathway from the UnTAND module. It results in a large drop in performance on both datasets. HeT- VAE HET- ALO removes the heteroscedastic layer and the augmented learning objective (since the augmented learning objective is introduced to improve the learning in the presence of heteroscedas- tic layer), resulting a highly significant drop on both datasets. These results show that all of the components included in the proposed model contribute to improved model performance. We provide more ablation results in Appendix A.2 and discuss hyperparameter selection in Appendix A.3
DISCUSSION AND CONCLUSIONS
In this paper, we have proposed the Heteroscedastic Temporal Variational Autoencoder (HeTVAB) for probabilistic interpolation of irregularly sampled time series data. HeTVAE consists of in- put sparsity-aware encoder, parallel deterministic and probabilistic pathways for propagating input uncertainty to the output, and a heteroscedastic output distribution to represent variable uncertainty in the output interpolations. Furthermore, we propose an augmented training objective to combat the presence of additional local optima that arise from the use of the heteroscedastic output struc- ture. Our results show that the proposed model significantly improves uncertainty quantification in the output interpolations as evidenced by significantly improved log likelihood scores compared to several baselines and state-of-the-aa methods. While the HeTVAE model can produce a probability distribution over an arbitrary collection output time points, it is currently restricted to producing marginal distributions. As result, sampling from the model does necessarily produce smooth trajectories as would be the case with -bssed models. Augmenting the HeTVAE model to ac- count for residual correlations in the output layer is interesting direction for future work.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
The source code for reproducing the results in this paper is available pulhttps://gittii com/ teml-Lab/hetvaae It contains the instructions to reproduce the results in the paper including the hyperparameters.. The hyperparameter ranges searched for each model are fully described in Appendix A.3 The source code also includes the synthetic dataset generation process as well as one of the real-world dataset. The other datasets be downloaded and prepared following the preprocessing steps notes in Appendix A.6.1 1
ACKNOWLEDGEMENTS
Research reported in this paper was partially supported by the National Institutes of Health under award number IP41EB028242.
REFERENCES
Mikolaj Binkowski, Gautier Marti, and Philippe Donnat. Autoregressive Convolutional Neural Net- works for Asynchronous Time Series. In Proceedings of the 35th International Conference on Machine Learning, pp. 580-589, 2018.
Edwin V Bonilla, Kian M Chai, and Christopher Williams. Multi-task gaussian process prediction. In Advances Neural Information Processing Systems, pp. 153-160, 2008
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. BRITS: Bidirectional Recurrent Imputation for Time Series. In Advances in Neural Information Processing Systems, pp. 6775- 6785. 2018.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag. and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 8(1), 2018a.
Zhengping Che, Sanjay Purushotham, Guangyu Li, Bo Jiang, and Yan Liu. Hierarchical deep gen- erative models for multi-rate multivariate time series. In Proceedings the 35th International Conference on Machine Learning, pp. 784-793, 2018b.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differ- ential equations. In Advances in Neural Information Processing Systems, pp. 6571-6583. 2018.
Li-Fang Cheng, David Stück, Thomas R Quisel, and L. Foschini. The impact of missing data in user-generated mhealth time series. 2017.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. CoRR, abs/1412.3555, 2014. Presented at the Deep Learning workshop at NIPS 2014.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE-Bayee Continuous Modeling of pooradically-Observee Time Series. In Advances in Neural Information Processing Systems, pp. 7379-7390. 2019.
Vincent Fortuin, Dmitry Baranchuk, Gunnar Raetsch, and Stephan Mandt. GP-VAE: Deep Proba- bilistic Time Series Imputation. In Proceedings of the 23rd International Conference Artificial Intelligence and Statistics, pp. 1651-1661. 2020.
Joseph Futoma, Sanjay Hariharan, and Katherine A. Heller. Learning to detect sepsis with a mul- titask gaussian process RNN classifier. In Proceedings of the 34th International Conference Machine Learning, pp. 1174-1182, 2017.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural processes, 2018.
10

---

Published as a conference paper at ICLR 2022
Marzych Ghassemi, Marco A.F. Pimentel, Thomas Brennan, David A. Clifton, Peter Szolovits, Tristan Naumann, and Mengling Feng. A multivariate timescries modeling approach to severity of illness assessment and forecasting in ICU with sparse, heterogeneous clinical data. In Proceedings of the National Conference on Artificial Intelligence, 2015.
Han- Gyu Kim, Gil-Jin Jang, Ho-Jin Choi, Minho Kim, Young- Won Kim, and Jachun Choi. Recur- rent neural networks with missing information imputation for medical examination data predic- tion. In 2017 IEEE International Conference on Big Data and Smart Computing (BigComp), pp- 317-323, 2017.
Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for time series. In Proceedings of the 25th International Conference on Machine Learning, 2020.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng. Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3, 2016.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equa- tions for irregular time series. In Advances in Neural Information Processing Systems, 2020.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo,, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2019. URL
net.
JC9KX
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Steven Cheng-Xiaa Xian Li and Benjamin Marlin. Learning from irreuulary-sampled time series: A missing data perspective. In Proceedings of the 37th International Conference on Machine Learn- ing. pp. 3937-5940 2020.
Steven Cheng-Xian Xiang Li and Benjamin M Marlin. A scalable end-to-end gaussian process adapter for irregularly sampled time series classification. In Advances In Neural Information Processing Systems, pp. 1804-1812, 2016.
Steven Cheng-Xian Xian Li and Benjmain M. Marlin. Classification of sparse and irregularly sampled time series with mixtures of expected Gaussian kemnlls and random features. In Proceedings of the 31st Conference on Uncertuinty Artificial Intelligence, pp. 484-493, 2015.
Xuechen Li, Ricky Tian Qi Chen, Ting-Kam Leonard Wong, and David Duvenaud. Scalable gradi- ents for stochastic differential equations. In Artificial Intelligence and Statistics, 2020.
Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with rns:: Improved classification of clinical time series. In Proceedings of the 1st Machine Learning for Healthcare Conference, pp. 253 270, 2016.
Manimaran, C. Parikh, K. Panigrahi, S. Basu, C. M. Kishtawal, and M. B. Porecha. Modelling financial time series. In Econophysics of Stock and other Markets, pp. 183-191. 2006.
Benjamin M. Marlin, David C. Kale, Robinder G. Khemani, and Randall C. Wetzel. Unsupervised pattern discovery in electronic health care data using probabilistic clustering models. In Proceed- ings of the 2nd ACM SIGHIT International Health Informatics Symposium, pp. 389-398, 2012.
M. J. Menne, N. Williams, Jr., and R. S. Vose. Long-term daily and monthly climate records from stations across the contiguous united states (u.s.historical climatology network) (ndp-019). 2016. doi: 10.3334/CDIA//CILNDDDD
Alexander Norcliffe, Cristian Bodnar, Ben Day, Jacob Moss, and Pietro Liò. Neural {ode} pro- cesses. In International Conference on Learning Representations, 2021.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. Adaptive computation and machine learning. 2006.

---

Published as a conference paper at ICLR 2022
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Con- ference Machine Learning, pp. 1278-1286, 2014.
Yulia Rubanova, Ricky T. Q. Chen, and David K Duvenaud. Latent ordinary differential equations for irreularly-sampler time series. In Advances in Neural Information Processing Systems, pp- 5320-5330 2019.
M. Schulz and K. Stattegger. Spectrum: Spectral analysis of unevenly spaced paleoclimatic time series. Computers & Geosciences, 23(9):929-945, 1997.
Siyuan Shan and Junier B. Oliva. NRTSI: Non-Recurrent Time Series Imputation for Irregularly- sampled Data. ArXiv, abs/2 102.0330, 2021.
Satya Narayan Shukla and Benjamin Marlin. eeerpolatton-prediiiiii networks for irregularly sam- pled time series. In International Conference on Learning Representations, 2019.
Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations, 2021a.
Satya Narayan Shukla and Benjamin M. Marlin. survey principles, models and methods for learning from irregularly sampled time series. CoRR, abs/2012.00168, 2021b.
Ikaro Silva, George Moody, Daniel Scott, Leo Celi, and Roger Mark. Predicting in-hospital mor- tality of icu patients: The physionet/computing in cardiology challenge 2012. Computing in cardiology, 39:245-244 248, 2012.
Qingxiong Tan, Mang Ye, Baoyao Yang, Siqi Liu, Andy Jinhua Ma, Terry Cheuk-Fung Yip, Grace Lai-Hung Wong, and PongChi Yuen. DATA-GRU: Dual-Attention Time-Aware Gated Recurrent Unit for Irregular Multivariate Time Series. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, pp. 930-937, 2020.
Pranjul Yadav, Michael Steinbach, Vipin Kumar, and Gyorgy Simon. Mining electronic health records (ehrs): survey. ACM Computing Surveys (CSUR), 50(6):85, 2018
Jinsung Yoon, William R. Zame, and Mihacla van der Schaar. Deep Sensing: Active Sensing using Multi-directional Recurrent Neural Networks, In International Conference on Learning Repre- sentations, 2018.
A APPENDIX
A.1 ADDITIONAL RESULTS
We also perform experiments the UCI electricity dataset (described in Appendix A.6. We follow the same experiment protocols described Section [ As we can see from Table 5 the proposed model HeTVAE outperforms the prior approaches across all three metrics.
A.2 ABLATION STUDY
Tables and show the complete results of ablating several different components of the HeTVAE model and training procedure with respect to all three evaluation metrics on PhysioNet and MIMIC- III respectively. We denote different components of the HeTVAE model HET: heteroscedastic output layer, ALO: augmented learning objective, INT: intensity encoding, DET: deterministic path- way. The results show selected individual and compound ablations of these components and indicate that all of these components contribute significantly to the model's performance in terms of the neg- ative log likelihood score. We provide detailed comments below.
12

---

Published as a conference paper at ICLR 2022
Table 5: Interpolation performance on Electricity Dataset.
Model
Negative Log Likelihood
Mean Absolute
Mean Squared
Error
Error
Mean Imputation
0.6765 + 0.0000
1.0311 + 0.0000
Forward Imputation
0.6163 + 0.0000
1.2626 + 0.0000
Single-Task Task GP
0.8972 = 0.0009
0.5456 + 0.0007
0.7827 + 0,0005
Multi-Tass GP
0.7767 = 0.0033
0.5324 + 0.0016
0.7782 + 0.0006
HVAE RNN
1.3981 = 0.0043
0.6267 = 0.0055
0.9577 0.0093
HVAE RNN- ONN-
1.3947 = 0.0054
0.6262 = 0.0074
0.9469 = 0.0111
HVAE ODE-RNN-ODE
1.4089 + 0.0095
0.6453 + 0.0042
0.9792 + 0.0184
HTVAE mTAAN
1.4040 + 0.0148
0.6392 + 0.0250
0.9724 + 0.0366
HeTVAE
0.7055 = 0.0103
0.5049 = 0.0039
0.7503 + 0.0162
Table Ablation Study of HeTVAF on PhysioNet.
Model
Negative Log Likelihood
Mean Absolute
Mean Squared Error
Error
HetVAE
0.5542 + 0.0209
0.3911 0.0004
0.5778 0.0020
HeTVAE ALO
0.6087 + 0.0136
0.4087 0.0008
0.6121 0.0063
НeTVAE DET
0.6278 + 0.0017
0.4089 + 0.0005
0.5950 + 0.0018
HeTVAE INT
0.6539 + 0.0107
0.4013 + 0.0005
0.5935 + 0.0015
HeTVAE HET ALO
1.1304 + 0.0016
0.3990 + 0.0003
0.5871 0.0016
HeTVAE DET -AO
0.7425 + 0.0066
0.4747 0.0024
0.6963 0.0031
HeTVAE PROB -LLO
0.7749 1 0.0047
0.4251 + 0.0029
0.6230 0.0040
НсTVAE - WYT-DET -ALO
0.7866 + 0.0029
0.4857 + 0.0003
0.7120 + 0.0007
HсTVAE -EET-NN-DET ALO
1.2426 + 0.0028
0.5056 + 0.0004
0.7167 + 0.0016
Effect of Heteroscedastic Layer: Since the augmented learning objective is introduced im- prove the learning in the presence of heteroscedastic layer, remove the augmented learning ob- jective (ALO) with the heteroscedastic layer (HET). This ablation corresponds to НеTVAE HET ALO. As we can see from both Table 6 and 7 this results in highly significant drop in the log likelihood performance as compared to the full HeTVAE model on both datasets. However, it results in only slight drop in performance with respect to MAE and MSE, which is sensible as the HET component only affects uncertainty sensitive performance metrics.
Effect of Intensity Encoding: HeTVAE INT removes the intensity encoding pathway from the UnTAND module. It results in an immediate drop in performance on both datasets. We also compare the effect of intensity encoding after removing the deterministic pathway and the augmented learning objective. These ablations are shown in HeTVAE. DET ALO and HeTVAE- INT DET ALO. The performance drop is less severe in this case because of the propensity of the heteroscedastic output layer to get stuck in poor local optima in the absence of the augmented learning objective (ALO).
Effect of Augmented Learning Objective: The HeTVAE ALO ablation shows the result of removing the augmented learning objective and training the model only using only the ELBO. This results in an immediate drop performance on PhysioNet. The performance drop is less severe on MIMIC-III. We further perform this ablation without the DET component and observe severe drops in performance across all metrics on both datasets. These ablations correspond HeTVAE DET and HeTVAE DET ALO. This shows that along with ALO component, the DET component also constrains the model from getting stuck in local optima where all of the structure in the data is explained as noise. We show interpolations corresponding to these ablations in Appendix A.3.1
Effect of Deterministic Pathway: HeTVAE DET removes the deterministic pathway from the model, resulting in performance drop both MIMIC-III and PhysioNet across all metrics. We further compare the performance of both the probabilistic and deterministic pathways in isolation shown by ablation HeTVAE DET ALO and HeTVAE PROB ALO. We observe that the
13

---

Published as a conference paper at ICLR 2022
Table 7: Ablation Study of HcTVAE on MIMIC-III
Model
Negative Log Likelihood
Mean Absolute Error
Mean Squared Error
HetVAE
0.6662 1 0.0023
0.3978 1: 0,0003
0.3716
0.0001
HeTVAE ALO
0.6869 + 0.0111
0.4043 0.0006
0.3840
0.0007
HcTVAE DET
0.7478 + 0.0028
0.4129 + 0.0008
0.3845
0.0009
НсTVAE INT
0.7430 + 0.0011
0.4066 + 0.0001
0.3837 = 0.0001
HeTVAE
0.9272 + 0.0002
0.4014 + 0.0001
0.3765 + 0.0001
HeTVAE - -
0.9005 + 0.0052
0.5177 + 0.0004
0.5325 0.0008
HeTVAE PROB ALO
0.7472 + 0.0056
0.4099 + 0.0006
0.3833 0.0008
HeTVAE
ALO
0.9245 + 0.0021
0.5208 + 0.0009
0.5358
0.0012
HeTVAE НET INT DET ALO
1.0498 + 0.0013
0.4931 + 0.0008
0.4848 + 0.0008
deterministic pathway HeTVAE PROB ALO outperforms the probabilistic pathway HeTVAE DET ALO in terms of log likelihood on MIMIC-III while the opposite is true in case of PhysioNet. However, on both datasets using only the deterministic pathway (HeTVAE PROB ALO) achieves better MAE and MSE scores compared to using only the probabilistic pathway (HeTVAE DET ALO).
A.3 VISUALIZATIONS
A.3.1 INTERPOLATIONS ON PHYSIONET
Figure 3 shows example interpolations on the PhysioNet dataset. Following the experimental setting mentioned in Section the models were trained using all dimensions and the inference uses all dimensions. We only show interpolations corresponding to Heart Rate as an illustration. As we can see, the STGP and НeTVAE models exhibit good fit and variable uncertainty on the edges where there are no observations. We can also see that mTAN trained with homoscedastic output is not able to produce as good fit because of the fixed variance the output (discussed in Section 4
The most interesting observation is the performance of HeTVAE DET ALO, an ablation of HeTVAE model that retains heteroscedastic output, but removes the deterministic pathways and the augmented learning objective. This ablation significantly underfits the data and performs similar to mTAN. This is an example of local optima that arises from the use of a heteroscedastic output layer where the mean is excessively smooth and all of the structure in the data is explained noise. We address this with the use of augmented learning objective described in Section 3.3 As seen the Figure 3 adding the augmented learning objective (HeTVAE- DET) clearly improves performance.
A.3.2 SYNTHETIC DATA VISUALIZATIONS SPARSITY
In this section, show supplemental interpolation results the synthetic dataset. The setting here is same as Section 4 Figure 4 compares HTVVE mTAN, the single task Gaussian process STGP, the proposed TVTE model and ablation of proposed model without intensity encoding HeTVAE- INT. We vary the number of observed points (3, 10, 20) and each model is used to infer the distribution over the remaining time points. We draw multiple samples from the VAE latent state for HeTVAE, HeTVAE INT and HTVAE mTAN, and visualize the distribution of the result- ing mixture. Figure 4 illustrates the interpolation performance of each of the models. As we can see, the interpolations produced by HTVAE m'TAN have approximately constant uncertainty across time and this uncertainty level does not change even when the number of points conditioned on in- creases. On the other hand, both НeTVAF and STGP show variable uncertainty across time. Their uncertainty reduces in the vicinity of input observations and increases in gaps between observations. The HeTVAE-INT model performs slightly better than HTVAE mANN model but it does not show variable uncertainty due to input sparsity like HeTVAE.
A.3.3 SYNTHETIC DATA VISUALIZATIONSS INTER INTER-OBSEREN OBSERVATION GAP
To demonstrate the effectiveness of intensity encoder (INT), we perform another experiment on synthetic dataset where increase the maximum inter-observation gap between the observations.
14

---

Published as a conference paper at ICLR 2022
Observed data
Ground truth
Reconstructions
STGP
HeTVAE
mTAN
In this figure, we show example interpolations of one dimension corresponding to Heart Rate on the PhysioNet dataset. The columns correspond to different examples. The rows corre- spond STGP, HeTVAE, HTVAE mTAN, HeTVAE-DET-ALO and HeTVAF-DET respectively. The shaded region corresponds + one standard deviation. STGP, HeTVAE and HcTVAE-DET exhibit variable output uncertainty and good while mTAN and HETVAE-DET-ALE does not.
Figure 3:
We follow the same training protocol as described Section At test time, condition 10 observed points with increasing maximum observibservation gap. We vary the maximum inter- observation gap from 20% 80% of the length of the original time series. Each model is used to infer single time point marginal distributions over values at the rest of the available time points in the test instance.
Figure showws the interpolations with increasing maximum inter-observation gap. STGP and HeT- VAE show variable uncertainty with time and the uncertainty increases with increasing maximum inter-observation gap. On the other hand, HTVAE mTAN with homoscedastic output shows approx- imately constant uncertainty with time and also across different maximum inter-observation gaps. These results clearly show that HTVAE mTAN produces over-confident probabilistic interpolations over large gaps.
Furthermore, show ablation of the proposed model HeTVAE INT, where remove the intensity encoder and perform the interpolations. As we see from the figure, this leads to approxi- mately constant uncertainty across time as well different maximum inter-observation gaps. This shows that the HeTVAE model is not able to capture uncertainty due to input sparsity as effectively without the intensity encoder.
15

---

Published as a conference paper at ICLR 2022
Observed data
Ground truth
Reconstructions
STGP
HeTVAE
IITAN
HATVAE-INT
Example 1.
STGP
HеTVAE
MTAN
HATVAE-INT
(b) Example 2.
STOP
HeTVAE
ITAAN
HeTVAF-INT
(c) Example 3.
Figure 4: Additional interpolation results on the synthetic dataset. The columns correspond to interpolation results with increasing numbers of observed points: 3, 10 and 20 respectively. The shaded region corresponds to + one standard deviation. STGP and HeTVAE exhibit variable output uncertainty in response input sparsity while mTAN and HeTVAE INT do not.
16

---

Published as a conference paper at ICLR 2022
Observed data
Ground truth
Reconstructions
STGP
HeTVAE
mTAN
HeTVAE-INT
(a) Example 1.
STGP
HeTVAE
HeTVAE-INT mTAN
(b) Example 2.
Figure 5: In this figure. show example interpolations the synthetic dataset with increas- ing maximum observtbse gap. The columns correspond to an inter-observation gap of size 20%, 40%, 60% and 80% of the length of original time series. The rows correspond to STGP, HeT- VAE, HTVAE mTAN and HeT HVAA--INT respectively. The shaded region corresponds to the confi- dence region. STGP and HeTVAE exhibit variable output uncertainty while m'TAN and HeTVAE- INT does not.
17

---

Published as a conference paper at ICLR 2022
A.4 ARCHITECTURE DETAILS
Learnable parameters in the UnTAND architecture shown in Figure la include the weights of the three linear layers and the parameters of the shared time embedding functions. Each time embedding function is a one layer fully connected network with a sine function non-linearity. The two linear layers on top of embedding function are linear projections from time embedding dimension de to de/H where H is the number of time embeddings. Note that these linear layers do not share parameters. The third linear layer performs a linear projection from D * H J. takes as input the concatenation of the VAL encoder output and INT encoder output and produces an output of dimension J. de, H and J are all hyperparameters the architecture. The ranges considered are described in the next section.
HeTVAE:
The HeTVAE model shown in the Figure lb consists of three MLP blocks apart from the UnTAND modules. The MLP in the deterministic path is a one layer fully connected layer that projects the UnTAND output match the dimension of the latent state. The remaining MLP blocks two- layer fully connected networks with matching width and ReLU activations. The MLP in the de- coder takes the output of UnTAND module and outputs the mean and variance of dimension D and sequence length t'. We use a softplus transformation on the decoder output to get the variance σ; = 0.01 + sof (h Similarly, in the probabilistic path, we apply an exponential transformation to get the variance of the distribution of exp(fer (u We use K refer- ence time points regularly spaced between and 1. K is considered to be a hyperparameter of the architecture. The ranges considered are described in the next section.
Baselines: For the HTVAE mTAN, we use a similar architecture as HeTVAE where we remove the deterministic path, heteroscedastic output layer and use the mTAND module instead of the UnTAND module (Shukla & Marlin 2021a). We use the same architectures for the ODE and RNN-based VAES as Rubanova et al. 2019).
A.5 HYPERPARAMETERS
We fix the time embedding dimension to de = 128. The number of embeddings H searched over the range (1, 4... We search the number of reference points K over the range 84, 16, 32), the latent dimension over the range {8. 16, 32, 64, 128), the output dimension of Un- TAND over the range {16, 32, 64, 128), and the width of the two-layer fully connected layers over {128, 256, 512]. In augmented learning objective,
HeTVAE:
search for X over the range {1.0, 5.0, 10.0). We use the Adam Optimizer for training the models. Experiments are run for 2, 000 iterations with learning rate of 0.0001 and batch size of 128. The best hyperparameters We use 100 samples from the probabilistic latent state to compute the evaluation metrics.
reported in the code.
We note that the ablations were not performed with a fixed architecture. For all the ab- lation models, we tuned the hyperparameters and reported the results with the best hyperparameter setting. We also made sure that the hyperparameter ranges for ablated models with just determin- istic/probabilistic path were wide enough that the optimal ablated models did not saturate the end of the ranges for architectural hyper-parameter values including the dimensionality of the latent representations.
Ablations:
VAE Baselines: For VAE models with homoscedastic output, we treat the out- put variance term a hyperparameter and select the variance over the range (0.01, 0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.6, 1.8, 2.0}. For HTVAE mTAN, we search the corresponding hyperparameters over the same range as HeTVAE. For ODE and RNN based VAEs, we search for GRU hidden units, latent dimension, the number of hidden units in the fully connected network for the ODE function in the encoder and decoder over the range {20, 32, 64, 128, 256). For ODEs, we also search the number of layers in fully connected network in the range (1, 3). We use batch size of 50 and learning rate of 0.001. We use 100 samples from the latent state to compute the evaluation metrics.
Gaussian Processes: For single task GP, we use a squared exponential kernel. In case of multi- task GP, experimented with the Matern kernel with different smoothness parameters, and the
18

---

Published as a conference paper at ICLR 2022
squared exponential kemell. We found that Matern kernel performs better. We use maximum marginal likelihood to train the GP hyperparameters. We search for learning rate over the range {0.1, 0.01, 0.001) and run for 100 iterations. We search for smoothness parameter over the range (0.5, 1.5, 2.5). We search for the batch size over the range (32, 64, 128, 256).
A.6 TRAINING DETAILS
A. DATA GENERATION AND PREPROCESSING
Synthetic Data Generation: We generate a synthetic dataset consisting of 2000 trajectories each consisting of 50 time points with values between O and 1. We fix 10 reference time points and draw values for each from a standard normal distribution. We then use an RBF kernel smoother with fixed bandwidth of a 120.0 to construct local interpolations over the 50 time points. The data generating process is shown below:
2A NoO, 1), K.E. E (1, 10]
0.1 k
4 0.02 * i E [1,... .50]
E. exp(-a(t; - rx))) 2b +.V(0.0.13)
22 =
En exp(-a(t; )3)
We randomly sample 10 observations from each trajectory to simulate a sparse and irregularly sampled univariate time series.
The PhysioNet Challenge 2012 dataset (Silva et al. 2012) consists multivariate time series data with 37 physiological variables from intensive care unit (ICU) records. Each record contains measurements from the first 48 hours after admission. We use the protocols described in Rubanova et al. (2199) and round the observation times to the nearest minute resulting in 2880 possible measurement times per time series. The data set consists includes 8000 instances that can be used for interpolation experiments. PhysioNet is freely available for research use and can be downloaded from https: https:::/phyiooette content hhtp//hhhaleeeeeeaa222
PhysioNet:
The MIMIC-III data set (Johnson al. 2016 multivariate time series dataset containing sparse and irregularly sampled physiological signals collected at Beth Israel Deaconess Medical Center. We use the procedures proposed by Shukka & Marlin (2019) to process the data set. This results in 53, 211 records each containing 12 physiological variables. We use all 53, 211 in- stances to perform interpolation experiments. MIMIC-III is available through a permissive data use agreement which can be requested at (ntttts: / //mmmiiico.. mit wttp:iii.ii...iiiiieeeeeee http://i//eeteeinissattteaeeeeee
MIMIC-II:
Once the request is approved, the dataset can be downloaded from htttttts://imic...o mit edu/ lii/gett
The instructions and code to extract the MIMIC-III dataset ht/ wwwlllaannnnteneeee...
is given at http::///fhtttbbboccc/////e/aa//
The U.S. Historical Climatology Network Monthly (USHCN) dataset (Mennel ct al. 2016) is a publicly available dataset consisting of daily measurements of 5 climate variables daily maximum temperature, daily minimum temperature, whether it snowy day or not, total daily precipitation, and daily snow precipitation. It contains data from the last 150 years for 1, 218 meteorological stations scattered over the United States. Following the preprocessing steps of Che et al. (2018b), we extract daily climate data for 100 consecutive years starting from 1910 2009 from 54 stations California. To get multi-rate time series data, we split the stations into 3 groups with sampling rates of 2 days, 1 week, and month respectively. We divide the data into smaller time series consisting yearly data and end up with a dataset of 100 examples each consisting of 270 features. We perform the interpolation task on this dataset where we compute the feature values every day using the multi-rate time series data. The dataset is available for download at Mthttps://ccaiee ess dive. 1bl www.ggoo.tttttt daily/
Climate Dataset:
Electricity Dataset: The UCI household electricity dataset contains measurements of seven dif- ferent quantities related to electricity consumption in household. The data recorded every minute for 47 months between December 2006 and November 2010, yielding over million ob- servations. To simulate irregular sampling, we keep observations only at durations sampled from
19

---

Published as a conference paper at ICLR 2022
exponential distribution with À 20. Following the preprocessing step of [Binkowski et al. 2018), we also do random feature sampling where we choose one out of seven features at each time step. We divide the data into smaller time series consisting of monthly data and end up with a dataset of 1431 examples each consisting of features. We perform interpolation experiments this dataset where we compute feature values every minute using the irregularly sampled data. The dataset is available for download at attttsp:://rrchiee ics ucii. edu fabu/mm//atte /ttp/////daaatss///// viiialiullllo Low.ooooooa.aaooooooooeeeeeeeee. htt.aa/eeeee..eeee.ee.eerooooorr
Dataset Preprocessing: We rescale time to be in [0, 1] for all datasets. We also re-scale all di- mensions. case of PhysioNet and MIMIC-III, for each dimensions we first remove outliers the outer 0.1% percentile region. We then compute the mean and standard deviation of all observations on that dimension. The outlier detection step is used to mitigate the effect of rare large values in the data from affecting the normalization statistics. Finally, z-transform all of the available data (including the points identified as outliers). No data points are discarded from the data sets during the normalization process.
A.6.2 SOURCE CODE
The source code for reproducing the results in this paper is available at uttttps://githuit com/ rem1-lab/hetwaa
A.6.3
COMPUTING INFRASTRUCTURE
All experiments were run on a Nvidia Titan X and 1080 GPUs. The time required to run all the experiments in this paper including hyperparameter tuning was approximately eight days using eight GPUs.
20


Reviews:
[{"review_id": "fPl4Opvamsf", "reviewer": "Reviewer_7Ard", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Clarity: I am confused about the differences between heteroscedastic and homoscedastic after reading the paper. Does the heteroscedastic output means that you learn the variance and the homoscedastic output means you fix the variance? What is the shape of w and v in equation 3? what is g in equation 7?\n\nOriginality: I think the intensity encoding is novel and makes sense to make the model be aware of input sparsity. For the heteroscedastic output layer, based on my understanding (see Clarity above), it just allows the model to learn the variance rather than using a fixed variance. If my understanding is correct, I don't this heteroscedastic output layer is novel because there are lots of works that learn a variance for maximum likelihood estimation. Combining a deterministic path and a probabilistic path is not novel, which has been proposed in [1, 2]. The augmented training objective in equation 11 that encourages the predicted mean to be close to samples x makes sense, but I think there should be some existing works that also use this trick.\n\nExperiments: I am satisfied with the performance this model achieves. However, I think the authors should also compare to Neural Process [1] and Attentive Neural Process [2], which are mentioned in the related work. [1,2] also use attention mechanism and do probabilistic interpolation. This paper also misses citation and comparison to a previous work NRTSI [3] that can also impute irregularly-sampled time series. I suggest the author compare to NRTSI on the irregularly-sampled Billiard dataset introduced in NRTSI.  Also, I am not clear about what the baseline HTVAE mTAN means? Does it mean \"homoscedastic temporal VAE mTAN\"? If my understanding is correct, I think the author should also compare to HeTVAE mTAN (\"heteroscedastic temporal VAE mTAN\") that allows mTAN to learn a variance rather than using a fixed variance. For the ablation study model HeTVAE - DET, because the deterministic path is removed, the model capacity decreases and makes the comparison unfair. For a fair comparison, I suggest increasing the capacity of the probabilistic path after removing the deterministic path. \n\n\n\n[1] Garnelo, Marta, et al. \"Neural processes.\" arXiv preprint arXiv:1807.01622 (2018).\n\n[2] Kim, Hyunjik, et al. \"Attentive neural processes.\" arXiv preprint arXiv:1901.05761 (2019).\n\n[3] Shan, Siyuan, and Junier B. Oliva. \"NRTSI: Non-Recurrent Time Series Imputation.\" arXiv preprint arXiv:2102.03340 (2021).\n\n", "summary_of_the_paper": "This paper introduces several improvements over the previous work mTAN to better support probabilistic interpolation. Specifically, intensity encoding is introduced to make the model be aware of information about input sparsity. Also, the homoscedastic output distribution used by previous work is replaced by a heteroscedastic distribution. Experiments results show that this improved model (HeTVAE) achieves both better likelihood estimation and mean prediction compared to previous works.", "summary_of_the_review": "Given the limited novelty, some unclarities, and lacks of some baselines, I give 5 to this paper now. If the authors can clarify and add these baseline results, I may consider increasing my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VatRgLLnUSp", "reviewer": "Reviewer_i1vq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Contribution\n\n### Advantages\n\nThe tackled problem of probabilistic interpolation of time series is relevant and especially valuable as it has received less attention than the forecasting task, at least in the neural networks community. This is especially the case in the considered setting where observations are not synchronized between dimensions.\n\nThe proposed model is for the most part interesting and well motivated, justifying the additions that are made on top of the prior mTAN. These contributions are, to the best of my knowledge, novel. The so-called heteroscedasticity of the output variance answers a crucial issue of standard VAEs with constant output variance in this context of application - even though this has already been considered elsewhere. The intensity branch of the introduced network clearly addresses the shortcomings of prior state-of-the-art methods. Accordingly, the paper is mostly well written and easy to read.\n\nExperimentally, the performance of HeTVAE is state-of-the-art by a large margin, showing the benefits of the approach. Ablation studies show the individual contribution of each model component; I suggest that the authors include the full ablation of the appendix in the main paper. The experiments are sufficiently well designed and diverse on real-world applications in order to correctly assess this performance. Furthermore, qualitative experiments with examples of interpolation are appealing and highlight the impact of the model and its different components.\n\nFor all these reasons, I think that this paper is interesting and might be ready for publication at ICLR after the revision during the rebuttal. Indeed, I would express reservations, that I detail below.\n\n### Limitations and Potential Improvements\n\nA first limitation of the current version of the paper deals with the significance of the introduced heteroscedasticity. While very few models that are able tackle the same task as HeTVAE seem to leverage temporally varying output variance, it is unclear whether this is an inherent advantage of the described model, or if it is orthogonal to the other architectural considerations. In other words, could heteroscedasticity be reasonably applied to the considered RNN/ODE-based baselines? If it does, I recommend that the authors include these augmented baselines in their experiments to better contextualize the performance of HeTVAE. In any case, a more detailed discussion of the related work is necessary in this regard. In particular, I would advise the authors to consider other works which, to my understanding, could be included in the related work and considered baselines with heteroscedasticity [1, 2].\n\nA second limitation is a partial lack of specification or explication of the modeling choices. In particular, the motivation and intuition behind the deterministic path is missing from the paper, to my understanding; I am wondering about the impact of its introduction in parallel of the usual variational latent variable, since the deterministic variable is unconstrained whereas the stochastic one is constrained by the KL term in the loss. An additional ablation showing the performance of HeTVAE without the stochastic branch would thus be interesting. Moreover, the role of the reference timestamps $\\mathbf{r}$ is unclear and they are only specified in the appendix; could the authors discuss their necessity in the model and the relevance of their choice? Further discussion about how the baselines are adapted for the considered task should also be included. Finally, it is unclear until Section 4 that the learned VAE also learns to predict interpolations besides reconstructing its inputs: this information should be clearly stated earlier, probably in the description of the ELBO.\n\n## Other Remarks and Questions\n\n### Scalability of the Intensity Pathway\n\nCould the authors comment on the scalability of the implementation of the intensity pathway as described in Equation (1)? To my understanding, the numerator pools over all elements of the dataset and it would seem that a large-scale dataset would prevent an efficient computation of this branch.\n\n### Nature of the Augmented Learning Objective\n\nThere seems to be a typo in the Equation (11): the augmented learning objective should be minimized, so it should be negatively added the the ELBO.\n\n### Code and Supplementary Material\n\nBased on my limited review, the experiments of this paper seem to be reproducible. The provided code is appreciated. I recommend the authors to remove the hidden folders in the archive, especially the `.git` which is irrelevant and makes the archive heavier than needed.\n\n\n## References\n\n[1] X. Li et al. Scalable Gradients for Stochastic Differential Equations. AISTATS 2020.\\\n[2] A. Norcliffe et al. Neural ODE Processes. ICLR 2020.", "summary_of_the_paper": "This paper introduces a novel model, HeTVAE, for probabilistic interpolation of time series that are irregularly sampled. HeTVAE builds on prior work by complementing it with a learned time-dependent output variance in the VAE and architectural improvements. The latter include a new branch accounting for the distribution of the sampled timestamps in the series and the addition of a deterministic branch bypassing the stochastic variational latent variable. The performance of HeTVAE is evaluated on multiple datasets against various baselines and via ablation studies.", "summary_of_the_review": "Given the relevance and experimental results of the proposed model, I think that this paper may be valuable to the audience of ICLR. Nonetheless, some substantial limitations about the nature of the heteroscedasticity contribution and unclear points in the paper prevent me from giving a firm positive score, thus choosing a score marginally below the acceptance threshold.\n\nI believe that this paper may be improved in the course of the discussion phase and would be pleased to raise my score, should my concerns be addressed during this time.\n\n### Post-Rebuttal Update\n\nI acknowledge the authors' response and thank them for their detailed answer. As explained in my follow-up response, the rebuttal partly addressed my concerns. While I still find that it could be improved on some aspects, I believe that this paper may be accepted to the conference. Therefore, before discussing with the other reviewers, I choose to raise my score from 5 to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U8tz0z28FUc", "reviewer": "Reviewer_oEK3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ \nNovel method\nWell written\nNotation is clear and organized\nWell evaluated on multiple datasets\nExtensive ablation tests and discussion about the model components\nCode available at review\n\n-\nBig novelty overlap with [1]\n\n[1] Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations, 2021\n", "summary_of_the_paper": "The paper introduces a novel model of Variational Autoencoder that deals with irregularly sampled time series with a probabilistic approach to do time series interpolation.\n\nThe main contribution is the architecture by itself, its components, and the training process. \nThe model was evaluated on both real-world data sets from the medical and climate domain and synthetic data.\n", "summary_of_the_review": "Very good paper with correct methodology. relevant results and a concerning overlap with a recent ICLR paper", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PvyveXtwkTh", "reviewer": "Reviewer_EXpk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS\n\n1. A great deal of modern machine learning literature for time series data often assumes regularly sampled time series, with no missing data and fixed size outputs. This is however rarely the case in many real world applications, which introduces many technical challenges that are not obvious to overcome with more standard architectures. The model presented in this paper is a possible way to tackle these challenges, and as such, I found this paper a very interesting read.\n1. Despite building heavily on the mTAN model, the new ideas introduced in the paper are novel and well motivated.\n2. Empirically, the HeTVAE outperforms competing methods by a large margin, and seem to be able to correctly capture uncertainty over time (as seen for example in figures 2 and 3)\n3. The paper has extensive ablation studies that justify all the new components of the model  \n\nWEAKNESSES\n1. I found the theoretical exposition in section 3 somewhat confusing in its current form, I could only follow it after reading details in the appendix and reading section 3 once again.\n    1. Reference points play a key role in the HeTVAE, but from section 3 it is not clear what their role is as well as how they are chosen. I was only able to really grasp their role after reading the appendix and looking at the code, which should not be the case. Only in appendix A4 I could understand that they are regularly spaced in [0,1], and only in A.6.1 that the time is scaled between 0 and 1 in all datasets (after which the choice of reference points makes sense)    \n    2. related to the above, \"reference points\" are mentioned in the \"intensity encoding\" and \"model output\" paragraphs. But for a reader not familiar with the mTAN it is not obvious why we are interested in them.    \n    3. The prior over z is not defined in the paper \n2. The \"augmented learning objective\" seems quite hacky to me, and I wonder if there are better ways to achieve the same (e.g. better initializations, KL annealing, ..).     \n    1. Learning output variances is normally not a problem in VAEs, why is it a problem in this case?\n    2. How robust is the choice of the scaling factor lambda across different datasets?\n3. Choosing equally spaced reference points means that if one uses as a test point a sample temporally close to the input data (or even a sample from the input data), the imputation of the model might be unnecessarily poor. Could this be improved?\n4. As stated by the authors, the model is only able to provide a marginal distribution at each time step, which practically means that the sampled trajectories might look inconsistent (non-smooth). Exploring the usage of sequential latent variable models for this would be an interesting future research direction. \n5. The baselines for HVAE RNN  and HVAE RNN-ODE are much worse than forward imputation, which makes me question the implementation of the models.", "summary_of_the_paper": "This paper introduces a VAE-based model for interpolation of irregularly sampled time series. The temporal input data is mapped to a latent representation over fixed reference points with an attention mechanism, using an intensity network that allows to encode data sparsity information. This latent representation can then be used to interpolate points at new time steps. Thanks to the intensity network and the heteroscedastic output layer, the proposed HeTVAE model can capture uncertainty estimates over the interpolated points. \nThe model is tested on a number of datasets containing irregularly sampled points, and outperforms competing methods in the interpolation task.", "summary_of_the_review": "This is a paper that can have an impact in real world applications, and as such I think it should be accepted. However, I did not find the paper easy to follow in this form.\nFor this, for now I have set \"marginally above the acceptance threshold\", but I would be happy to revise the score upwards if I see that the authors improve the exposition in the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes a model to handle uncertainties in an irregularly sampling time series setting (HetVAE), built on the VAE framework and the previous work on mTAN (multi-time attention networks), and introduces components to encode sparsity information and heteroscedastic output uncertainty. The paper is clear, well-motivated, and contains extensive ablation studies showing the effect of eaach added components.
I recommend this submission for acceptance.
Decision: Accept (Poster)