RISE and DISE: Two Frameworks for Learning from Time Series with Missing Data

Time series with missing data constitute an important setting for machine learning . The most successful prior approaches for modeling such time series are based on recurrent neural networks that learn to impute unobserved values and then treat the imputed values as observed . We start by introducing Recursive Input and State Estimation ( RISE ) , a general framework that encompasses such prior approaches as specific instances . Since RISE instances tend to suffer from poor long-term performance as errors are amplified in feedback loops , we propose Direct Input and State Estimation ( DISE ) , a novel framework in which input and state representations are learned from observed data only . The key to DISE is to include time information in representation learning , which enables the direct modeling of arbitrary future time steps by effectively skipping over missing values , rather than imputing them , thus overcoming the error amplification encountered by RISE methods . We benchmark instances of both frameworks on two forecasting tasks , observing that DISE achieves state-of-the-art performance on both . 1 INTRODUCTION . Many machine learning settings involve data consisting of time series of observations . Due to various reasons , observations may be missing from such time series . For instance , it may be impossible to observe the data during a given time window , the data-recording system may fail , or measurements may be recognized as noisy and immediately discarded at the source . Historically , the prevalent approach to handling missing data has been to apply a preprocessing step to replace the missing observations with substituted values and then treat the time series as though it were complete ( Schafer & Graham , 2002 ) . Multiple recent works ( Lipton et al. , 2016 ; Yoon et al. , 2017 ; 2018 ; Che et al. , 2018 ; Cao et al. , 2018 ) , however , circumvent this two-step approach and integrate both value imputation and the downstream task in one single model . At their core , these approaches employ recurrent neural networks ( RNN ) ( Hochreiter & Schmidhuber , 1997 ) whose input and hidden states are modified to account for the missing observations . In this paper , we note fundamental commonalities between these prior approaches and define a general framework , Recursive Input and State Estimation ( RISE ) , that encompasses those approaches as specific instances . Instances of RISE operate recursively on all intermediate time steps between two observed values , by imputing the first unobserved value in a row based on the preceding observed values and then working with the imputed value as though it were observed . Recursive approaches suffer from poor long-term performance ( Lamb et al. , 2016 ; Fox et al. , 2018 ) as errors are amplified in feedback loops . To overcome this problem , we propose a novel framework , Direct Input and State Estimation ( DISE ) , also built on recurrent neural networks . In contrast to RISE , DISE learns representations for inputs and hidden states that are enriched with time information and thus allow the framework to model arbitrary future time steps by effectively skipping over missing values , rather than imputing them . Moreover , unlike previous work ( Yoon et al. , 2017 ; Baytas et al. , 2017 ; Binkowski et al. , 2018 ; Che et al. , 2018 ) , DISE integrates not only relative time information ( the time between observations ) , but also absolute time information , which helps the model learn non-stationary properties of the signal . Overall , we make the following contributions : ( i ) We introduce RISE ( Section 3 ) , a general framework for learning from univariate time series with missing data , which captures the fundamental commonalities between the most relevant recursive approaches from the literature . ( ii ) We propose DISE ( Section 4 ) , a non-recursive framework that learns time-enriched input and state representations and is thus able to avoid imputation-induced drift ( a main shortcoming of RISE methods ) by skipping to future hidden states . ( iii ) We benchmark instances of RISE and DISE in two forecasting problems ( Section 5 ) . The best-performing instance of DISE operates on digits as atomic units to derive latent representations for numerical data . The expressiveness of this encoding function allows us to achieve state-of-the-art performance in both problems . 2 NOTATION . We denote a univariate time series x= [ x1 , x2 , . . . , xT ] as a sequence of T observations . Let s be the time vector [ s1 , s2 , . . . , sT ] , each value corresponding to the time—measured as the running time elapsed since the beginning of the time series—when the observation was taken . In practice , some observations may be missing for multiple reasons , for which a T -dimensional masking vector m is defined : mi = 1 if xi is observed and mi = 0 otherwise . We define an alternative notation that will be used in the definition of our framework . We denote o = [ o1 , o2 , . . . , oN ] as the sequence of N observations with non-missing values . The associated timestamp vector is denoted by t= [ t1 , t2 , . . . , tN ] , where ti represents the time—also represented as the running time since the beginning of the time series—of the observed value oi . Moreover , we also define ta , which translates the running time values of t to a certain date or time format . In absence of missing data , o= x and N = T . A time series with missing data can be described with either notation ( see Table 5 in Appendix A ) . 3 RECURSIVE INPUT AND STATE ESTIMATION . Instances of the RISE framework are given by recurrent neural architectures to learn a model for predicting p ( xi+n|x1 : i ) by recursively predicting intermediate conditional terms . While one can find a numbers of variants of recurrent neural architectures in the literature , all variants define a cell whose ( hidden ) state hi ∈ Rdh—dh is the number of units—updates based on the previous state hi−1 and current input xi ∈ R. In time series with missing data , instances of the RISE framework replace the standard input with a transformed input x̂ . Similarly , at the input of the cell the incoming state hi−1 is substituted with a transformed hidden state ĥi−1 to account for the last time a value was observed in the time series . The hidden state hi is updated based on these transformed signals and the equations of the chosen recurrent architecture . The transformed input x̂ depends on the conditionally imputed input xc , defined as follows : xci = ximi + ( 1−mi ) x̃i , ( 1 ) where x̃ is the imputed input , whose computation is instance-specific . It is common to some of these works to define a time gap vector δB , defined as the time gap from the last observed value to the current timestamp . More formally : δ Bi = si− si−1 +δ B i−1 , i > 1 , mi−1 = 0 si− si−1 , i > 1 , mi−1 = 1 0 , i = 1 ( 2 ) The superscript B refers to the backward computation of the time gap vector : the time gap at a certain moment is computed with respect to the previous timestamp . They are used in the computation of the so-called discount factors , denoted as γx ∈ R and γh ∈ Rdh , which in turn are used in the computation of the transformed state and the transformed hidden state , respectively . The discount factors are defined as follows : γxi = γ x ( δ Bi ) = exp ( −max ( 0 , wxγ δ Bi +bxγ ) ) , γhi = γ h ( δ Bi ) = exp ( −max ( 0 , whγ δ Bi +bhγ ) ) , ( 3 ) where wxγ , b x γ , b h γ ∈ R and whγ ∈ Rdh , are parameters trained jointly with all other parameters of the model . The motivation of the discount factor comes from the following intuition : the influence of the past history in the current moment fades away over time . These discount factors allow these models to also learn from time series where consecutive timestamps are not always equally spaced . While in most of the RISE instances authors address related downstream time series classification or regression problems in multivariate time series , in this paper we are interested in autoregression problems in univariate time series , which allow to assess more directly the ability of these methods to learn with missing data . Therefore , a function fθ , parameterized with weights θ , is applied to the hidden state hi to predict the next observation , more formally : ŷi+1 = fθ ( hi ) . The loss at the i-th observation is defined as li = mi+1L ( yi+1 , ŷi+1 ) . Typically , the function fθ is a regression function , L is the mean squared error loss , and yi amounts to xi—i.e . the i-th value of the time series . Alternatively , it can be cast as a multi-class classification problem ( van den Oord et al. , 2016b ; a ; Fox et al. , 2018 ) , wherein the function fθ outputs a probability mass function , L is the cross-entropy loss , and yi is the one-hot encoding of xi . A non-exhaustive list of recent instances of the RISE framework can be found in Table 1 . Simple Recursion ( Fox et al. , 2018 ) This is the simplest instance of the RISE framework . At each timestamp the model is fed with either the input value if there exists , or the imputed value , which is computed in the previous timestamp , if the input value is missing . Zeros/Forward-Filling & Indicators ( Lipton et al. , 2016 ) Lipton et al . ( 2016 ) considers two popular strategies to fill missing values for sequential data : forward- and zeros-filling . The former replaces the missing values with the last non-missing value observed in the sequence . The latter simply fills the missing values with zeros . The input is augmented with a binary indicator , which is set to one if the input value is missing , and set to zero otherwise . Thus , through their hidden state computations , the recursive neural network can use indicators of missing data to learn arbitrary functions of the past observations and missingness patterns . Zero-filling with binary indicators was found to be the best overall model by all considered metrics in their downstream task . GRU-D ( Che et al. , 2018 ) This approach incorporates the previously described discount factors to account for missingness when computing the transformed input value and the hidden state . The decay mechanism applied to the input learns to fade away the impact of the last observation over time and progressively replace it with the empirical mean of the input signal . Similarly when applied to the hidden state , their features are decayed over time . RITS-I ( Cao et al. , 2018 ) Cao et al . ( 2018 ) proposed BRITS , a family of models designed for data imputation in time series . Within these models , RITS-I is the only one that is suitable for the task at hand . It combines aspects of all three previous works . Similar to the simple recursion approach , the imputed value for the current timestamp is the predicted value computed in the previous timestamp . As in Che et al . ( 2018 ) , it is equipped with a decay mechanism over the hidden state , and the input is augmented with a binary indicator . Other instances Kim & Chi ( 2018 ) is another instance of the RISE framework closely related to GRU-D . Recent works ( Yoon et al. , 2017 ; Kim et al. , 2017 ; Luo et al. , 2018 ) on data imputation for time series could also be framed in the RISE framework under certain ( small ) modifications . Different to forecasting , most of data imputation methods replace missing values with substituted values based on previous and subsequent information .

Summary: The authors propose a new univariate time series analysis framework called RISE, which unifies the existing work on adapting RNNs to irregular time series. Building on top of the RISE, they propose a modification called DISE in which the algorithm skips the intervals without any observations. In that sense, DISE can be considered a marked point process analysis algorithm. They quantify the performance of the RISE and DISE on two datasets.