Heteroscedastic Temporal Variational Autoencoder For Irregularly Sampled Time Series

1 INTRODUCTION . In this paper , we propose a novel deep learning framework for probabilistic interpolation of irregularly sampled time series . Irregularly sampled time series data occur in multiple scientific and industrial domains including finance ( Manimaran et al. , 2006 ) , climate science ( Schulz & Stattegger , 1997 ) and healthcare ( Marlin et al. , 2012 ; Yadav et al. , 2018 ) . In some domains including electronic health records and mobile health studies ( Cheng et al. , 2017 ) , there can be significant variation in inter-observation intervals through time . This is due to the complexity of the underlying observation processes that can include “ normal ” variation in observation times combined with extended , blockstructured periods of missingness . For example , in the case of ICU EHR data , this can occur due to patients being moved between different locations for procedures or tests , resulting in missing physiological sensor data for extended periods of time . In mobile health studies , the same problem can occur due to mobile sensor batteries running out , or participants forgetting to wear or carry devices . In such situations , it is of critical importance for interpolation models to be able to correctly reflect the variable input uncertainty that results from variable observation sparsity so as not to provide overly confident inferences . However , modeling time series data subject to irregular sampling poses a significant challenge to machine learning models that assume fully-observed , fixed-size feature representations ( Marlin et al. , 2012 ; Yadav et al. , 2018 ; Shukla & Marlin , 2021b ) . The main challenges in dealing with such data include the presence of variable time gaps between the observation time points , partially observed feature vectors caused by the lack of temporal alignment across different dimensions , as well as different data cases , and variable numbers of observations across dimensions and data cases . Significant recent work has focused on developing specialized models and architectures to address these challenges in modeling irregularly sampled multivariate time series ( Li & Marlin , 2015 ; 2016 ; Lipton et al. , 2016 ; Futoma et al. , 2017 ; Che et al. , 2018a ; Shukla & Marlin , 2019 ; Rubanova et al. , 2019 ; Horn et al. , 2020 ; Li & Marlin , 2020 ; Shukla & Marlin , 2021a ; De Brouwer et al. , 2019 ; Tan et al. , 2020 ; Kidger et al. , 2020 ) . Recently , Shukla & Marlin ( 2021a ) introduced the Multi-Time Attention Network ( mTAN ) model , a variational autoencoder ( VAE ) architecture for continuous-time interpolation of irregularly sampled time series . This model was shown to provide state-of-the-art classification and deterministic interpolation performance . However , like many VAEs , the mTAN architecture produces a homoscedastic output distribution conditioned on the latent state . This means that the model can only reflect uncertainty due to variable input sparsity through variations in the VAE latent state . As we will show , this mechanism is insufficient to capture differences in uncertainty over time . On the other hand , Gaussian Process Regression-based ( GPR ) methods ( Rasmussen & Williams , 2006 ) have the ability to reflect variable uncertainty through the posterior inference process . The main drawbacks of GPR-based methods are their significantly higher run times during both training and inference , and the added restriction to define positive definite covariance functions for multivariate time series . In this work , we propose a novel encoder-decoder architecture for multivariate probabilistic time series interpolation that we refer to as the Heteroscedastic Temporal Variational Autoencoder or HeTVAE . HeTVAE aims to address the challenges described above by encoding information about input sparsity using an uncertainty-aware multi-time attention network ( UnTAN ) , flexibly capturing relationships between dimensions and time points using both probabilistic and deterministic latent pathways , and directly representing variable output uncertainty via a heteroscedastic output layer . The proposed UnTAN layer generalizes the previously introduced mTAN layer with an additional intensity network that can more directly encode information about input uncertainty due to variable sparsity . The proposed UnTAN layer uses an attention mechanism to produce a distributed latent representation of irregularly sampled time series at a set of reference time points . The UnTAN module thus provides an interface between input multivariate , sparse and irregularly sampled time series data and more traditional deep learning components that expect fixed-dimensional or regularly spaced inputs . We combat the presence of additional local optima that arises from the use of a heteroscedastic output layer by leveraging an augmented training objective where we combine the ELBO loss with an uncertainty agnostic loss component . The uncertainty agnostic component helps to prevent learning from converging to local optima where the structure in data is explained as noise . We evaluate the proposed architecture on both synthetic and real data sets . Our approach outperforms a variety of baseline models and recent approaches in terms of log likelihood , which is our primary metric of interest in the case of probabilistic interpolation . Finally , we perform ablation testing of different components of the architecture to assess their impact on interpolation performance . 2 RELATED WORK . Keeping in mind the focus of this work , we concentrate our discussion of related work on deterministic and probabilistic approaches applicable to the interpolation and imputation tasks . Deterministic Interpolation Methods : Deterministic interpolation methods can be divided into filtering and smoothing-based approaches . Filtering-based approaches infer the values at a given time by conditioning only on past observations . For example , Han-Gyu Kim et al . ( 2017 ) use a unidirectional RNN for missing data imputation that conditions only on data from the relative past of the missing observations . On the other hand , smoothing-based methods condition on all possible observations ( past and future ) to infer any unobserved value . For example , Yoon et al . ( 2018 ) and Cao et al . ( 2018 ) present missing data imputation approach based on multi-directional and bi-directional RNNs . These models typically use the gated recurrent unit with decay ( GRU-D ) model ( Che et al. , 2018a ) as a base architecture for dealing with irregular sampling . Interpolation-prediction networks take a different approach to interfacing with irregularly sampled data that is based on the use of temporal kernel smoother-based layers ( Shukla & Marlin , 2019 ) . Shan & Oliva ( 2021 ) propose hierarchical imputation strategy based on set-based architectures for imputation in irregularly sampled time series . Of course , the major disadvantage of deterministic interpolation approaches is that they do not express uncertainty over output interpolations and thus can not be applied to the problem of probabilistic interpolation without modifications . Probabilistic Interpolation Methods : The two primary building blocks for probabilistic interpolation and imputation of multivariate irregularly sampled time series are Gaussian processes regression ( GPR ) ( Rasmussen & Williams , 2006 ) and variational autoencoders ( VAEs ) ( Rezende et al. , 2014 ; Kingma & Welling , 2014 ) . GPR models have the advantage of providing an analytically tractable full joint posterior distribution over interpolation outputs when conditioned on irregularly sampled input data . Commonly used covariance functions have the ability to translate variable input observation density into variable interpolation uncertainty . GPR-based models have been used as the core of several approaches for supervised learning and forecasting with irregularly sampled data ( Ghassemi et al. , 2015 ; Li & Marlin , 2015 ; 2016 ; Futoma et al. , 2017 ) . However , GPR-based models can become somewhat cumbersome in the multivariate setting due to the positive definiteness con- straint on the covariance function ( Rasmussen & Williams , 2006 ) . The use of separable covariance functions is one common approach to the construction of GPR models over multiple dimensions ( Bonilla et al. , 2008 ) , but this construction requires all dimensions to share the same temporal kernel parameters . A further drawback of GP-based methods is their significantly higher run times relative to deep learning-based models when applied to larger-scale data ( Shukla & Marlin , 2019 ) . Variational autoencoders ( VAEs ) combine probabilistic latent states with deterministic encoder and decoder networks to define a flexible and computationally efficient class of probabilistic models that generalize classical factor analysis ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) . Recent research has seen the proposal of several new VAE-based models for irregularly sampled time series . Chen et al . ( 2018 ) proposed a latent ordinary differential equation ( ODE ) model for continuous-time data using an RNN encoder and a neural ODE decoder . Building on the prior work of Chen et al . ( 2018 ) , Rubanova et al . ( 2019 ) proposed a latent ODE model that replaces the RNN with an ODE-RNN model as the encoder . Li et al . ( 2020 ) replace the deterministic ODEs with stochastic differential equations ( SDEs ) . Norcliffe et al . ( 2021 ) extends the prior work on neural ode by combining it with neural processes ( Garnelo et al. , 2018 ) . Shukla & Marlin ( 2021a ) proposed the Multi-Time Attention Network ( mTAN ) model , a VAE-based architecture that uses a multi-head temporal cross attention encoder and decoder module ( the mTAND module ) to provide the interface to multivariate irregularly sampled time series data . Fortuin et al . ( 2020 ) proposed a VAE-based approach for the task of smoothing in multivariate time series with a Gaussian process prior in the latent space to capture temporal dynamics . Garnelo et al . ( 2018 ) ; Kim et al . ( 2019 ) used heteroscedastic output layers to represent uncertainty in case of fixed dimensional inputs but these approaches are not applicable to irregularly sampled time series . Similar to the mTAN model , the Heteroscedastic Temporal Variational Autoencoder ( HeTVAE ) model proposed in this work is an attention-based VAE architecture . The primary differences are that mTAN uses a homoscedastic output distribution that assumes constant uncertainty and that the mTAN model ’ s cross attention operation normalizes away information about input sparsity . These limitations are problematic in cases where there is variable input density through time resulting in the need for encoding , propagating , and reflecting that uncertainty in the output distribution . As we describe in the next section , HeTVAE addresses these issues by combining a novel sparsity-sensitive encoder module with a heteroscedastic output distribution and parallel probabilistic and deterministic pathways for propagating information through the model . Another important difference relative to these previous methods is that HeTVAE uses an augmented learning objective to address the underfitting of predictive variance caused by the use of the heteroscedastic layer . 3 PROBABILISTIC INTERPOLATION WITH THE HETVAE . In this section , we present the proposed architecture for probabilistic interpolation of irregularly sampled time series , the Heteroscedastic Temporal Variational Autoencoder ( HeTVAE ) . HeTVAE leverages a sparsity-aware layer as the encoder and decoder in order to represent input uncertainty and propagate it to output interpolations . We begin by introducing notation . We then describe the architecture of the encoder/decoder network followed by the complete HeTVAE architecture . 3.1 NOTATION . We letD = { sn|n = 1 , ... , N } represent a data set containing N data cases . An individual data case consists of a D-dimensional , sparse and irregularly sampled multivariate time series sn . Different dimensions d of the multivariate time series can have observations at different times , as well as different total numbers of observations Ldn . We follow the series-based representation of irregularly sampled time series ( Shukla & Marlin , 2021b ) and represent time series d for data case n as a tuple sdn = ( tdn , xdn ) where tdn = [ t1dn , ... , tLdndn ] is the list of time points at which observations are defined and xdn = [ x1dn , ... , xLdndn ] is the corresponding list of observed values . We drop the data case index n for brevity when the context is clear .

Summary: This paper introduces a VAE-based model for interpolation of irregularly sampled time series. The temporal input data is mapped to a latent representation over fixed reference points with an attention mechanism, using an intensity network that allows to encode data sparsity information. This latent representation can then be used to interpolate points at new time steps. Thanks to the intensity network and the heteroscedastic output layer, the proposed HeTVAE model can capture uncertainty estimates over the interpolated points.  The model is tested on a number of datasets containing irregularly sampled points, and outperforms competing methods in the interpolation task.