Interpolation-Prediction Networks for Irregularly Sampled Time Series

1 INTRODUCTION . Over the last several years , there has been significant progress in developing specialized models and architectures that can accommodate sparse and irregularly sampled time series as input ( Marlin et al. , 2012 ; Li & Marlin , 2015 ; 2016 ; Lipton et al. , 2016 ; Futoma et al. , 2017 ; Che et al. , 2018a ) . An irregularly sampled time series is a sequence of samples with irregular intervals between their observation times . Irregularly sampled data are considered to be sparse when the intervals between successive observations are often large . Of particular interest in the supervised learning setting are methods that perform end-to-end learning directly using multivariate sparse and irregularly sampled time series as input without the need for a separate interpolation or imputation step . In this work , we present a new model architecture for supervised learning with multivariate sparse and irregularly sampled data : Interpolation-Prediction Networks . The architecture is based on the use of several semi-parametric interpolation layers organized into an interpolation network , followed by the application of a prediction network that can leverage any standard deep learning model . In this work , we use GRU networks ( Chung et al. , 2014 ) as the prediction network . The interpolation network allows for information contained in each input time series to contribute to the interpolation of all other time series in the model . The parameters of the interpolation and prediction networks are learned end-to-end via a composite objective function consisting of supervised and unsupervised components . The interpolation network serves the same purpose as the multivariate Gaussian process used in the work of Futoma et al . ( 2017 ) , but remove the restrictions associated with the need for a positive definite covariance matrix . Our approach also allows us to compute an explicit multi-timescale representation of the input time series , which we use to isolate information about transients ( short duration events ) from broader 1Our implementation is available at : https : //github.com/mlds-lab/interp-net trends . Similar to the work of Lipton et al . ( 2016 ) and Che et al . ( 2018a ) , our architecture also explicitly leverages a separate information channel related to patterns of observation times . However , our representation uses a semi-parametric intensity function representation of this information that is more closely related to the work of Lasko ( 2014 ) on modeling medical event point processes . Our architecture thus produces three output time series for each input time series : a smooth interpolation modeling broad trends in the input , a short time-scale interpolation modeling transients , and an intensity function modeling local observation frequencies . This work is motivated by problems in the analysis of electronic health records ( EHRs ) ( Marlin et al. , 2012 ; Lipton et al. , 2016 ; Futoma et al. , 2017 ; Che et al. , 2018a ) . It remains rare for hospital systems to capture dense physiological data streams . Instead , it is common for the physiological time series data in electronic health records to be both sparse and irregularly sampled . The additional issue of the lack of alignment in the observation times across physiological variables is also very common . We evaluate the proposed architecture on two datasets for both classification and regression tasks . Our approach outperforms a variety of simple baseline models as well as the basic and advanced GRU models introduced by Che et al . ( 2018a ) across several metrics . We also compare our model with to the Gaussian process adapter ( Li & Marlin , 2016 ) and multi-task Gaussian process RNN classifier ( Futoma et al. , 2017 ) . Further , we perform full ablation testing of the information channels our architecture can produce to assess their impact on classification and regression performance . 2 RELATED WORK . The problem of interest in this work is learning supervised machine learning models from sparse and irregularly sampled multivariate time series . As described in the introduction , a sparse and irregularly sampled time series is a sequence of samples with large and irregular intervals between their observation times . Such data commonly occur in electronic health records , where they can represent a significant problem for both supervised and unsupervised learning methods ( Yadav et al. , 2018 ) . Sparse and irregularly sampled time series data also occur in a range of other areas with similarly complex observation processes including climate science ( Schulz & Stattegger , 1997 ) , ecology ( Clark & Bjørnstad , 2004 ) , biology ( Ruf , 1999 ) , and astronomy ( Scargle , 1982 ) . A closely related ( but distinct ) problem is performing supervised learning in the presence of missing data ( Batista & Monard , 2003 ) . The primary difference is that the missing data problem is generally defined with respect to a fixed-dimensional feature space ( Little & Rubin , 2014 ) . In the irregularly sampled time series problem , observations typically occur in continuous time and there may be no notion of a “ normal ” or “ expected ” sampling frequency for some domains . Methods for dealing with missing data in supervised learning include the pre-application of imputation methods ( Sterne et al. , 2009 ) , and learning joint models of features and labels ( Williams et al. , 2005 ) . Joint models can either be learned generatively to optimize the joint likelihood of features and labels , or discriminately to optimize the conditional likelihood of the labels . The problem of irregular sampling can be converted to a missing data problem by discretizing the time axis into non-overlapping intervals . Intervals with no observations are then said to contain missing values . This is the approach taken to deal with irregular sampling by Marlin et al . ( 2012 ) as well as Lipton et al . ( 2016 ) . This approach forces a choice of discretization interval length . When the intervals are long , there will be less missing data , but there can also be multiple observations in the same interval , which must be accounted for using ad-hoc methods . When the intervals are shorter , most intervals will contain at most one value , but many intervals may be empty . Learning is generally harder as the amount of missing data increases , so choosing a discretization interval length must be dealt with as a hyper-parameter of such a method . One important feature of missing data problems is the potential for the sequence of observation times to itself be informative ( Little & Rubin , 2014 ) . Since the set of missing data indicators is always observed , this information is typically easy to condition on . This technique has been used successfully to improve models in the domain of recommender systems ( Salakhutdinov et al. , 2007 ) . It was also used by Lipton et al . ( 2016 ) to improve performance of their GRU model . The alternative to pre-discretizing irregularly sampled time series to convert the problem of irregular sampling into the problem of missing data is to construct models with the ability to directly use an irregularly sampled time series as input . The machine learning and statistics literature include several models with this ability . In the probabilistic setting , Gaussian process models have the ability to represent continuous time data via the use of mean and covariance functions ( Rasmussen , 2006 ) . These models have non-probabilistic analogues that are similarly defined in terms of kernels . For example , Lu et al . ( 2008 ) present a kernel-based method that can be used to produce a similarity function between two irregularly sampled time series . Li & Marlin ( 2015 ) subsequently provided a generalization of this approach to the case of kernels between Gaussian process models . Li & Marlin ( 2016 ) showed how the re-parameterization trick ( Kingma et al. , 2015 ) could be used to extend these ideas to enable end-to-end training of a deep neural network model ( feed-forward , convolutional , or recurrent ) stacked on top of a Gaussian process layer . While the basic model of Li & Marlin ( 2016 ) was only applied to univariate time series , in follow-up work the model was extended to multivariate time series using a multi-output Gaussian process regression model ( Futoma et al. , 2017 ) . However , modeling multivariate time series within this framework is quite challenging due to the constraints on the covariance function used in the GP layer . Futoma et al . ( 2017 ) deal with this problem using a sum of separable kernel functions ( Bonilla et al. , 2008 ) , which limit the expressiveness of the model . An important property of the above models is that they allow for incorporating all of the information from all available time points into a global interpolation model . Variants differ in terms of whether they only leverage the posterior mean when the final supervised problem is solved , or whether the whole posterior is used . A separate line of work has looked at the use of more local interpolation methods while still operating directly over continuous time inputs . For example , Che et al . ( 2018a ) presented several methods based on gated recurrent unit ( GRU ) networks ( Chung et al. , 2014 ) combined with simple imputation methods including mean imputation and forward filling with past values . Che et al . ( 2018a ) additionally considered an approach that takes as input a sequence consisting of both the observed values and the timestamps at which those values were observed . The previously observed input value is decayed over time toward the overall mean . In another variant the hidden states are similarly decayed toward zero . Yoon et al . ( 2017 ) presented another similar approach based on multi-directional RNN which operate across streams in addition to within streams . However , these models are limited to using global information about the structure of the time series via its empirical mean value , and current or past information about observed values . The global structure of the time series is not directly taken into account . Che et al . ( 2018b ) focus on a similar problem of modeling multi-rate multivariate time series data . This is similar to the problem of interest in that the observations across time series can be unaligned . The difference is that the observations in each time series are uniformly spaced , which is a simpler case . In the case of missing data , they use forward or linear interpolation , which again does not capture the global structure of time series . Similarly , Binkowski et al . ( 2018 ) presented an autoregressive framework for regression tasks with irregularly sampled time series data . It is not clear how it can be extended for classification . The model proposed in this work is similar to that of Li & Marlin ( 2016 ) and Futoma et al . ( 2017 ) in the sense that it consists of global interpolation layers . The primary difference is that these prior approaches used Gaussian process representations within the interpolation layers . The resulting computations can be expensive and , as noted , the design of covariance functions in the multivariate case can be challenging . By contrast , our proposed model uses semi-parametric , deterministic , feed-forward interpolation layers . These layers do not encode uncertainty , but they do allow for very flexible interpolation both within and across layers . Also similar to Li & Marlin ( 2016 ) and Futoma et al . ( 2017 ) , the interpolation layers in our architecture produce regularly sampled interpolants that can serve as inputs for arbitrary , unmodified , deep classification and regression networks . This is in contrast to the approach of Che et al . ( 2018a ) , where a recurrent network architecture was directly modified , reducing the modularity of the approach . Finally , similar to Lipton et al . ( 2016 ) , our model includes information about the times at which observations occur . However , instead of pre-discretizing the inputs and viewing this information in terms of a binary observation mask or set of missing data indicators , we directly model the sequence of observation events as a point process in continuous time using a semi-parametric intensity function ( Lasko , 2014 ) .

Summary: The authors propose a framework for making predictions on a sparse, irregularly sampled time-series data. The proposed model consists of an interpolation module, and the prediction module, where the interpolation module models the missing values in using three outputs: smooth interpolation, non-smooth interpolation, and intensity. The authors test the proposed method on two different datasets (MIMIC-III and UWave), although only one of the datasets are multi-variate. The proposed method shows comparable training time to other GRU variants, and outperforms all baseline models for mortality prediction and length-of-stay prediction.