RISE and DISE: Two Frameworks for Learning from Time Series with Missing Data
Authors: Alberto Garcia-Duran
Robert West
Venue:  (2020)
Keywords: Time Series
Missing Data
RNN

Abstract: Time series with missing data constitute an important setting for machine learning. The most successful prior approaches for modeling such time series are based on recurrent neural networks that learn to impute unobserved values and then treat the imputed values as observed. We start by introducing Recursive Input and State Estimation (RISE), a general framework that encompasses such prior approaches as specific instances. Since RISE instances tend to suffer from poor long-term performance as errors are amplified in feedback loops, we propose Direct Input and State Estimation (DISE), a novel framework in which input and state representations are learned from observed data only. The key to DISE is to include time information in representation learning, which enables the direct modeling of arbitrary future time steps by effectively skipping over missing values, rather than imputing them, thus overcoming the error amplification encountered by RISE methods. We benchmark instances of both frameworks on two forecasting tasks, observing that DISE achieves state-of-the-art performance on both.

Under review as a conference paper at ICLR 2020
RISE AND DISE: TWo FRAMEWORKS FOR LEARNING FROM TIME SERIES WITH MISSING DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Time series with missing data constitute an important setting for machine learn- ing. The most successful prior approaches for modeling such time series are based on recurrent neural networks that learn to impute unobserved values and then treat the imputed values as observed. We start by introducing Recursive Input and State Estimation (RISE), a general framework that encompasses such prior approaches specific instances. Since RISE instances tend suffer from poor long-term performance as errors are amplified in feedback loops, we propose Direct Inpur and State Estimation (DisE), a novel framework in which input and state repre- sentations are learned from observed data only. The key to DISE is to include time information representation learning, which enables the direct modeling of arbitrary future time steps by effectively skipping over missing values, rather than imputing them, thus overcoming the error amplification encountered by RISE methods. We benchmark instances of both frameworks on two forecasting tasks, observing that DISE achieves state-of-the-ane performance on both.
INTRODUCTION
Many machine learning settings involve data consisting of time series of observations. Due vari- ous reasons, observations may be missing from such time series. For instance, it may be impossible to observe the data during given time window, the data-reco system may fail, or measurements may be recognized as noisy and immediately discarded at the source. Historically, the prevalent ap- proach to handling missing data has been to apply a preprocessing step to replace the missing obser- vations with substituted values and then treat the time series as though it were complete (Schafer & Graham 2002). Multiple recent works (Lipton et al. 2016 Yoon et al., 2017 2018 Che al,] 2018] Cao et al. 2018). however, circumvent this two-step approach and integrate both value imputation and the downstream task in one single model. At their core, these approaches employ recurrent neu- ral networks (RNN) (Hochreiter & Schmidhuber 1997) whose input and hidden states modified to account for the missing observations.
In this paper, note fundamental commonalities between these prior approaches and define gen- eral framework, Recursive Input and State Estimation (RISE), that encompasses those approaches as specific instances. Instances of RISE operate recursively on all intermediate time steps between two observed values, by imputing the first unobserved value in a row based on the preceding observed values and then working with the imputed value as though it were observed. Recursive approaches suffer from poor long-term performance (Lamb et al.) 2016 Fox et al.) 2018) as errors are ampli- fied in feedback loops. To overcome this problem, we propose novel framework, Direct Input and State Estimation (DISE), also built on recurrent neural networks. In contrast to RISE, DISE learns representations for inputs and hidden states that are enriched with time information and thus allow the framework to model arbitrary future time steps by effectively skipping over missing val- ues, rather than imputing them. Moreover, unlike previous work YYon et al. 2017 Baytas et al, 2017 Binkowski et al. 2018 Che et al. 2018), DISE integrates not only relative time information (the time between observations), but also absolute time information, which helps the model learn non-stationary properties of the signal. Overall, make the following contributions:

---

Under review as a conference paper at ICLR 2020
(i) We introduce RISE (Section general framework for learning from univariate time series with missing data, which captures the fundamental commonalities between the most relevant recursive approaches from the literature.
(ii) We propose DISE (Section 4), a non-recursive framework that learns time-enriched input and state representations and thus able avoid imputation-induced drift (a main short- coming of RISE methods) by skipping to future hidden states.
(iii We benchmark instances of RISE and DISE in two forecasting problems (Section 5). The pest-performing instance of DISE operates on digits as atomic units to derive latent repre- sentations for numerical data. The expressiveness of this encoding function allows us to achieve state-of-the-art performance in both problems.
2
NOTATION
We denote univariate time series = [x1. as sequence of T observations. Let s be the time vector [s1, S2, .... syl, each value corresponding the mimeumeesured as the running time elapsed since the beginning of the time series when the observation was taken. In practice, some observations may be missing for multiple reasons, for which a a T-dimensional masking vector m defined: m; if x; is observed and m; = O otherwise.
We define an alternative notation that will be used in the definition of our framework. We denote (01,02,...,NN) as the sequence of N observations with non-missing values. The associated timestamp vector denoted by //../2...., ..]. where represents the time-also represented as the rumning time since the beginning of the time series of the observed value o,. Moreover, also define t", which translates the running time values of t to certain date or time format. In absence of missing data, and N T'
time series with missing data can be described with either notation (see Table in Appendix A
3 RECURSIVE INPUT AND STATE ESTIMATION
Instances of the RISE framework are given by recurrent neural architectures to learn a model for predicting p(zita|a11)) by recursively predicting intermediate conditional terms. While one can find a numbers of variants of recurrent neural architectures in the literature, all variants define a cell whose (hidden) state h; E R44 d, is the number of units updates based the previous state h;-1 and current input x; E R. In time series with missing data, instances the RISE framework replace the standard input with transformed input f. Similarly, at the input of the cell the incoming state h is substituted with transformed hidden state hạ account for the last time value was observed in the time series. The hidden state h; is updated based on these transformed signals and the equations of the chosen recurrent architecture. The transformed input depends the conditionally imputed input x defined as follows:
x-m- (1 m,)
where ă is the imputed input, whose computation is instance-specific.
It common to some of these works to define a time gap vector 88, defined the time gap from the last observed value to the current timestamp. More formally:
8 8 - Si
5 5
i> 1,m =
8
i 1 [mm--
The superscript B refers to the backward computation of the time gap vector: the time gap at certain moment is computed with respect to the previous timestamp. They are used in the computation of the so-called discount factors, denoted as y E RR and > E Rdh, which in turn are used in the computation of the transformed state and the transformed hidden state, respectively. The discount factors are defined follows:
1/ = exp( max (0, w7,5 8 + bb;)),
(3)
7 = "888) = exp( - exp(-max(O. 5, (,b)),++))))
2

---

Under review as a conference paper at ICLR 2020
where wi. bi, by, E R and w. R are parameters trained jointly with all other parameters of the model. The motivation of the discount factor comes from the following intuition: the influence of the past history in the current moment fades away over time. These discount factors allow these models to also learn from time series where consecutive timestamps are not always equally spaced.
While in most of the RISE instances authors address related downstream time series classification or regression problems in multivariate time series, in this paper we are interested in autoregression problems in univariate time series, which allow assess more directly the ability of these methods to learn with missing data. Therefore, function fo. parameterized with weights 0, is applied to the hidden state h to predict the next observation, more formally: f1+1 = fo(i).. The loss at the i-th observation is defined as 1, = m1-1- 201+1 DV+1.31+1).
Typically, the function fe is regression function, L the mean squared error loss, and y; amounts to i...ee. the i-th value of the time series. Alternatively, it can be cast as multi-class classifi- cation problem (van den Oord et al. 20T Fox et al. 2018), wherein the function fo outputs a probability mass function, L is the cross-entropy loss, and y; is the one-hot encoding of x;.
non-exhaustive list of recent instances of the RISE framework can be found Table
Instance of RISE
Input and State Transformation (Univariate Time Series)
Simple Recursion
8 Wy . 1 + by
f = x
h h111
Zeros& Indicators
8 =
④ = 1
h; = hi-1
Forward-Fillinge Indicators
8 Xy
h h
GRU-D
8 Fxx + (1 V1 Narz
- f
h;-1 % ② hi-1
RITS-I
4 Wx -h1-1 bx
à [xf:mL]]
h 7 h4-1
(Luo al.,
2018)
8
8 x
hi-1 2 ② hi-1
(Kim & Chi
2018)
8
26p + (1 yp yo ()xy
& x;
h hi-1
YYoon et al.
2017)'
8,
4; xx;; 8.0
h;-1 hi-1
BRITS-I
8 Wy hi b,
4; [xfmm)
hi 7 O hi-1
Table I: (Upper) is the dot product. (a:b] b) is the concatenation of terms a and b. @ is the element- wise multiplication.
is the complement of the binary value m. Kay the average of the non- missing values prior to sy of vector æ. Xy is the value of the last time (i' i) the signal was observed. (Lower) Instances to the data imputation problem are indicated with grey background. Luo et al. (2018) is framed in a generative adversarial network wherein the full time series is observed before imputing the missing values. Y/ is a discount factor computed with respect to Xpr, which is the nearest observed value in either time direction, and t is a threshold value. An asterisk * indicates that the described equations are computed in bidirectional manner. Except in cases where it is constant, ñ is always treated as a variable.
Simple Recursion (Fox al., 2018) This is the simplest instance of the RISE framework. At each timestamp the model is fed with either the input value if there exists, or the imputed value, which is computed in the previous timestamp, if the input value missing.
Saaossoooraar-Flllinggg. Indicators (Lipton et al. 2016) Lipton et al. (2016) considers two pop- ular strategies to fill missing values for sequential data: forward- and zeros-filling. The former replaces the missing values with the last non-missing value observed in the sequence. The latter simply fills the missing values with zeros. The input is augmented with a binary indicator, which is set to one if the input value is missing, and set to zero otherwise. Thus, through their hidden state computations, the recursive neural network can use indicators of missing data to learn arbitrary functions of the past observations and missingness patterns. Zero-filling with binary indicators was found to be the best overall model by all considered metrics in their downstream task.
GRU-D D Che et al., 2018 This approach incorporates the previously described discount factors to account for missingness when computing the transformed input value and the hidden state. The decay mechanism applied to the input learns to fade away the impact of the last observation over time and progressively replace it with the empirical mean of the input signal. Similarly when applied to the hidden state, their features are decayed over time.
3

---

Under review as a conference paper at ICLR 2020
RITS-I (Cao al., 2018) Cao et al. (2018) proposed BRITS, family of models designed for data imputation in time series. Within these models, RITS-I is the only one that is suitable for the task at hand. It combines aspects of all three previous works. Similar to the simple recursion approach, the imputed value for the current timestamp is the predicted value computed in the previous timestamp. As in Che al. (2018), it is equipped with decay mechanism over the hidden state, and the input is augmented with binary indicator.
Other instances Kim & Chi (2018) is another instance of the RISE framework closely related to GRU-D. Recent works (Yoon et al. 2017 Kim et al. 2017: Luo et al. 2018) on data imputation for time series could also be framed the RISE framework under certain (small) modifications.
Different to forecasting, most of data imputation methods replace missing values with substituted values based on previous and subsequent information.
DIRECT INPUT AND STATE ESTIMATION
Main (and only) differences across RISE instances lie the equations that alter the standard input and hidden state of the recurrent neural architecture, but everything else remain unchanged. One may define new equations based on expert knowledge or intuition to guide the transformation of input and hidden state, however the DISE framework skip timestamps where the data is missing and only learns from observed data. DISE resorts latent representations enriched with absolute and relative time information to enable the recurrent architecture to update its hidden state so as to jump to certain moment into the future. The learning the underlying "discount" factor up to these representations. Therefore, DISE does not require computing any intermediate term. This implies that the hidden state only updated in timestamps where a value is observed. We resort to the alternative notation defined in Section 2 to define DISE.
RISE
fo
Recument cell
Recurrent call
Recumentt
Recurrett cell
ha
b
call
haz2
2:-2
Agya 2
E5o3 Ezva
DISE
h--14]])
Ro (4)
Recurrett cell
Recurrett
Recurrett call
0,
A
6,147)
G (3 17
To
(23
8 2/
O4+1
04+2
Figure 1: (Upper) segment of time series with missing data. Simplified layouts of (Middle) RIsE and (Lower) DISE. Whereas RISE attends to timestamps whose data points are missing and updates its hidden state accordingly, DISE skips them and only updates its state with observed values.
The DISE framework also defines time gaps between pairs of non- non-missing observations as follows:
8F =
Jitn -i, undefined, otherwise,
which corresponds the time gap between the current non-missing observation, happening at f, and the n-th non non-missing observation after the current one, happening at f In this case, the superscript F' refers to its forward computation.
4

---

Under review as a conference paper at ICLR 2020
Let 4 be shorteut that refers to the time information (f DISE defines 6, b/]4 E R and [ E 124, computed respectively with functions &o and Eh. and that correspond to latent rep- resentations that enriched with the absolute and relative time information in 4/. They the counterparts the transformed input and state of the RISE framework. During training, at a certain time the framework computes:
6 = go(o,. d on (fnn) The dedicated latent representation for the input to predict the value at f;+-- The relative time distance between the two observations is 8
(inn) The dedicated representation for the incoming hidden state h; [ predict the value at Jitn. separated 8 from the the current observation.
h = (h;-1 [4)_]] %/
We refer to [ E IR4 as the A--lagged hidden state, which will be used by the function So to out- put -ie. the prediction for fi-n- It is computed based on 6;(4]] and - and the equations of the chosen recurrent architecture. One can train a model to be good at making predictions not only for the next observed value but also for observations beyond the next one or even for time horizon. In the most general case, the loss at the i-th observation defined as 1, = Σ L (Z++n:&tnn) where D is the length- in number of observations- of the time horizon. As in the RISE framework, L, So and ž are chosen so as to cast the problem as either a regression or a multi-classification one. Figure [ depicts the RISE and DISE frameworks. At inference time, directly target desired future timestamp 1g and query the framework with (/3 - fa-4g) being la the timestamp of the last observed value, which computes the (1+ ff hidden state output prediction.
One key component of the DISE framework is the learning of time-enriched input and hidden state representations. In this work we propose an instance whose functions 8o and gh use the learned representations for relative and absolute time information to act gating mechanisms over the representations of the input and hidden state. This allows effectively allocate some dimensions of the representations to account for time dependencies. They are computed as follows
6,[47] = 8o(o), of ((tn) = (on) @ o (Wafen
(0 (f(m)) o (fLLa)
$'(4))
(5)
h [A7] 3h 2h 8f (( h:-1 oV o of oo (
where all f$) :R -> Rd functions are encoders- we use the same dg for all encoders for numerical values. The W. V matrices map all latent representations to the same dimension, either d, (which is the same as da) or dn. The choice of the encoders will be discussed in Section 4.1 The (AT)) and 3'(A") terms have role similar to the discount mechanisms of RISE, and are computed with both relative and absolute time information. However, these "discount" factors are be learned, and not driven by a certain expression. Compared to the RISE framework, on one side DISE requires to compute additional latent representation for input, relative and absolute time information; but on the other side it does not require to compute intermediate states to make predictions for a future timestamp. This makes our framework also suitable for event series, where the time gap between two consecutive observations is arbitrary. In contrast, RISE instances will require events to be aligned at some chosen timestamps, which generates large number missing observations -thus, being inefficient computationally and increasing its complexity.
The representations for the input 6:[4]] and hidden state [A7] are computed the element- wise product of three representations. Alternatively, 8o and ga could be chosen so as to represent 6([A)) and h [A]] as the concatenation of their three corresponding representations. However, experienced worse performance and longer training times with this instance, the number of parameters of the recurrent cell also grow.
As discussed, instances of RISE and DISE are agnostic to the chosen recurrent neural architecture. In the experimental section we opt for gated recurrent unit (GRU) (Cho et al. 2014), whose details for both frameworks can be found in Appendix B.I An extension of the proposed DISE instance to increase its depth is explained in Appendix B.2 This extension reinforces time information into each of the higher layers. Note that d, might be different to da for higher layers in deeper architectures.
5

---

Under review as a conference paper at ICLR 2020
Most instances of RISE are not appropriate to DISE, as i) they cannot include ddatefoormatted abso- lute time information, and ii) the transformed input and hidden state cannot not directly be computed for arbitrary time into the future. However, it is possible to recycle ideas from RISE instances to only learn from observed data. Inspired by some of these works, we propose RISE2IDISE, which applies corresponding discount mechanisms translate them to any moment into the future. Different to GRU-D, RIsE2DISE only updates the state with observed values. More details to be found in Appendix B.3
Equation (3) to the input and hidden state to
Prediction over a Time Horizon Instances of the the RISE framework predict X by recursively estimating intermediate values until reaching the desired point in time. Therefore, the side effect is that these methods also predict a time horizon-i.e. the values leading up to X;-D- On the other side, multi-output forecasting methods estimate the whole time horizon in one step (Taieb et al.] 2012). Previous work (Fox et al. 2018) has empirically shown that multi-output methods outperform recursive methods in the time horizon prediction task. This is due the feedback loop of recursive methods, which re-use past predictions to predict future values, leading to lower quality predictions as the time horizon increases. However, is not clear how multi-outpu approaches would deal with time series with missing values, as they assume the time series is complete. The DISE framework does not in either of these two paradigms.
4.1 ON THE CHOICE OF THE ENCODERS
Encoders suitable for DISE must be able to map any data that can be decomposed sequence of digits to latent space. One may be tempted to transform these data to one-hot encoding represen- tations, and then learn a latent representation via a linear layer. However, the potential vocabulary size of this approach is infinite. Therefore a data binning scheme should be first applied to the data, which is a challenge in itself.
Feedforward-based encoders A popular option (Li et al. 2017 Pezeshkpour et al. 2018 for mapping single numerical values a latent space is by applying a feedforward neural network over the (typically) log-normalized numerical value. Let m and m be numerical value and its corresponding latent representation, respectively, then this encoding function would amount to: m fiew(m) log log(m)w byw, where the numerical value only scales the weight vector Wf.. A sigmoid activation function is also added when encoding the input signal in Equation (5). Despite this non-linearity, the expressiveness of this encoding function is limited. It is trivial show that the learned latent representations behave monotonically (with respect to m) because of the monotonicity of the sigmoid function and the linearity of the learned representations. To alleviate this limitation we also explore mulli-feedforwar encoding function fnttw with two layers and a sigmoid as an activation function.
Digit-level encoder On the other hand, motivated by character-level architectures for language modeling, alternative would consist of decomposing numerical values sequence of digits and then operate on digits as atomic units to derive latent representations. To our knowledge, using digits as atomic units (tokens) learn representations for numerical values has been previously explored only in García-Durán et al. (2018) for a different problem-link prediction in temporal graphs. Each token of the numerical value is mapped its corresponding embedding via linear layer and the resulting sequence of embeddings is fed into standard recurrent neural network architecture GRU in this work. Let nm be the number of tokens of the numerical value m, its latent representattinn corresponds to the last hidden state of the standard GRU, more formally: m m=rr (m) h In contrast to feEw, the learned representations are not necessarily monotonie. The vocabulary consists of 11 tokens: for digits from 0 to 9, and token ".", which indicates, if there exists, the beginning of the decimal part of the numerical value. On some occasions, positive and negative signs may also be required in the vocabulary. Moreover, this encoding function allows absolute time information l" to be simply decomposed as a sequence of digits following a certain date format.
In any case, dedicated encoding function is used for each of the input signal, relative and absolute time information (see superscripts in Equation (5)). We will refer to the instances whose encoding function is fetu, fufty and fgeu as DISF-FFW, DISR-MFFW and DISF-GRU, respectively.

---

Under review as a conference paper at ICLR 2020
5 EXPERIMENTS
We target two forecasting problems. For the time horizon pre- diction problem we consider time series without missing val- ues, wherein the goal is to be good at predicting the values over a time horizon of length D. For the second one, referred to as next observed value prediction problem, we consider data sets with missing data. This means that the next observed value can happen anytime into the future, and the model makes predic- tion for that specific time. They are illustrated in Figure 2
Figure 2: zon and (Lower) next observed value prediction problem. Pre- dictions (blue triangles) are made based on current (green squares) and past observed values (red cir- cles). Black-fille filled shapes repre- sent missing data.
(Upper) Time hori-
5.1 DATASET DESCRIPTION
5.1.1 BLOOD GLUCOSE
This complete data (Fox et al., 2018) was collected over the course of three years and consists of a large number of continuous glucose readings from 40 patients with type diabetes. Every three months, blood glucose of the patients was monitored over the course of several days with sampling rate of minutes. We do not use date-formatted time information 1 the experiments run in this data set, as many measurements lack reference date. We run experiments in this data set for the time horizon prediction problem. We artificially remove consecutive blocks data- see Appendix D.1-to the next observed value prediction problem, leading to the data sets Blood Glucose (B = {1,5)). The larger the value of B is, the more data is missing. We use the same training. validation and test sets in Fox et al. (2018).
5.1.2 AIR POLLUTION
This incomplete data et al. 2016) consists of measurements from monitoring stations in Beijing. They were hourly collected from 2014/05/10 to 2015/04/30. In this work we consider two particulate matter: 2M2.5 and PM The first particle pollution time series was used in previous work (Yi al., 2016 Cao et al. 2018) for the data imputation problem. We run experiments for the next observed value prediction problem in these data sets.
We have also created a modified version of the PM2.5 time series, called PM2.5-peak. wherein the measurements taken on the 2nd and 15th of each month are increased in 200 oriignaa values range from 1 to 1,000 pg/m². Therefore, the bias of the signal varies sharply in these days.
The first ten months of collected data are used for training, and the data collected in the 11th and 12th month are used for validation and test, respectively. Statistics of the data sets and additional details are shown in Table 5 Appendix D.I
Method
Time Horizon
Minutes
10 Minutes
15 Minutes
20 Minutes
Minutes
30 Minutes
Simple Recursion Fror et 2018
|
1.4
2.99 4.92
4.46 6.97
8.84
7.00
10.61
8.34 12.30
DRRPAA [Fox mn 2013
120
2.64
2.05 4.85
4.25 6:76
3.47 R.3Y
6.63
10.70
7.87 11.97
SEqMo (rox or p010]
1:0
2.52
2.78
4,72
1.08 6.66
3.28 8:45
6.30
10.18
7.63 11.33
Dist-aiw
2.77
2507
5.30
6.88
Disn-Merw [R]
2.78
2.00
4.16
6.71
8.47
6.57
7.60
11.58
DisE daU [R]
1.44
2.48
2.72
4.68
4.00
6.57
5.15
8.31
6.34
10.01
7,51
11.62
Table 2: Blood Glucose: Time horizon prediction.
5.2 SETUP AND BASELINES
As the blood glucose data set is our main benchmark data set, for all models we carefully follow the same evaluation protocol and setup as in Fox et al. (2018), where this data set was introduced. Therefore, instances of RISE are built by stacking two GRUS of 512 hidden units d, in standard manner (Hermans & Schrauwen 2013). Instances of the DISE framework are reinforced with time information at higher layers as explained in Appendix B.2 Following Fox et al. (2018), we use the multi-classification formulation of the problem described in Section 3 and the median and mean APE (absolute percentage error) as evaluation metrics. More training details are found in Appendix C For the air pollution data sets the regression formulation replaces the every other aspect of the evaluation and setup remains the same. We think that this formulation is
one, but
7

---

Under review as a conference paper at ICLR 2020
more common in regression problems, and also helps to assess methods' performance under two different objectives.
For the time horizon problem, we evaluate instance against the two multi-output approached proposed in Fox et al. (2018). We also compare it to the recursive approach previously referred to simple recursion. The performance of these methods is copied from Fox al. (2018). For the next observed value prediction problem we implemented most of the instances of the RISE framework (upper side in Table I and ran experiments under the above-described evaluation and setup. In neither of these two problems we compare to methods that are not based on recurrent neural architectures, related work have shown they perform poorly compared method based on recurrent urchitectures in related problems (Lipton et al. [016]] Fox et al. 2018 Cao et al. (2118).
For all instances of DISE we use a [suffix] that combination of the letters (relative time information) and A (absolute time information) to indicate the use of that data in the learning of the eo and rh terms. For each data set, the 8 values are normalized by the corresponding sampling period.
30-Miiutte Time Horzen Prediction
5.3 RESULTS
Moas ARE
The format "Median APE
Mean APE" is used all tables.
5.3.1 BLOOD GLUCOSE
We show results for the time horizon problem in the blood glu- cose data set in Table 2 This was the problem this data set was originally intended for. As all baselines, our instances were trained for length D of the time horizon of 30 minutes-this amounts observations. DISE-GRU [R] shows the best performance for almost each of the time points within the 30-minute prediction window. The recursive approach matches the performance of
Time Horazr durng Inaveg
Figure 3: Mean APE Vs
DISE-GRU [R] at the first place of the time window. However, its performance degrades as the time horizon increases, being quickly surpassed by the multi-output approaches DEEPMO and SEQMO. The results also indicate the importance on the choice of the encoding function, as the two other instances of DISE are par or even slightly worse than those of RISE.
One may wonder whether increasing the length of the time horizon during training may lead to better forecasts within the next 30-minute. Figure 3 shows the best performance is obtained when training and test objectives are pretty much aligned. As expected, performance degrades when the horizon of the training objective is smaller or larger (enough) than that of the evaluation.
Data Set
Table 3 shows results for the next observed value prediction problem. Motivated by Figure we align training and test objectives: models are trained to be good at predicting the next observed value. GRU-D the best performing recur- sive method, and is competitive with DISE-FFW [R], but outper-
Blood Glucose (B 1)
Blood Glucose (ß =
Method
Simple Recursion
2.43
4.29
7.36
12.43
Indos&Indicators
2.43
4.32
6.73
11.64
GRU-D
2.25
4.06
6.18
11.53
RITS-I
2.32
4.13
6.66
12.23
RISE2DISE
2.48
4.28
6.86
11.98
DISE-FFW [R]
2.31
4.09
6.02
11.31
DISE-MEEW [R]
2.50
4.29
6.31
11.39
DISE-GRU [R]
2.06
3.97
5.55
11.29
Table 3: Blood Glucose: Next observed value prediction.
formed by DISE-GRU [R] in both mean and median APE. This indicates again that the choice of the encoding function is crucial to outperform the instances of RISE. RISE2DISE's performance indi- cates that exponential discount factors do not benefit from learning with only observed data. It is important to note that, contrary to language, similarities across numerical values based on its atomic units is arbitrary and based on the chosen notation scheme. We provide visualizations of the learned representations, and experiments with a hexadecimal notation scheme in Appendix [D2
Once we have validated the choice of the encoding function, for the remaining experiments we only benchmark DISE-GRU against the baselines.
'The two other multi-output approaches proposed in that work are extensions of these ones, wherein the main difference lies additional pre- and post-processing steps applied to the data.
8

---

Under review as a conference paper at ICLR 2020
Method
Data Set
Air Pullution
PM1
Air Pullution
PM25
Air Pollution PMas--pak
5.3.2 AIR POLLUTION
Simple Recumioo
14.27 37.30
12:29
11.66
35T6
Zenod Indicatoos
14.57 33125
12.29
EX.22
11.46]
33.62
GRU-D
14.17 28.85
12.37
23.47
11,02| 32.49
The date-formatted timestamps ta are decomposed as the sequence of the digits that correspond to the month and the day of the month. Results are shown in Table 4 The time series corresponding to the par- ticulate matters 1M and PM2.5 do not largely benefit from including absolute time information. As before, GRU-D is the best performing recursive instance. For the particle PM2.5, whose percent- age of missing data is low, instances of RISE are competitive and even outperform instances DISE in terms of mean APE. In this data set, majority (around 95%) of test data points are spaced one sampling period, so this may explain the very good performance of the RISE instances. One can assess the positive impact of the absolute time information in PM where DISE-GRU [AR] clearly outperforms all methods in mean APE. shows the improvement in performance oc- curs the 2nd, 15th and their both subsequent days- hriis-highhhhtedd with an orange background-,, as it is in these days where the bias value of the time series sharply varies. We show in Appendix D.2 that DISE-GRU [AR] learns allocate dedicated dimensions model this characteristic of the signal. Even though recurrent neural networks are suitable to learn from non-stationary time series, this shows that including absolute time information helps to model certain properties of the signal- in this case, an abrupt change in the bias of the signal tha may be difficult to learn otherwise. To overcome this problem one might preprocess the time series to convert it into stationary, but this might be specially challenging under the existence of missing data in the time series.
14.42
(29.57
12.24
22.08
31.37
DiSE GRU IR] DIS (AR]
13.81 28.25
12.19
28.91
11.12 30.10
13.81
27.60
12.44
28.74
11.54 27.76
Table 4: Air Pollution: Next observed value prediction.
6 DISCUSSION AND FUTURE WORK
We describe RISE, recursion-based framework that encompasses most of the recent works in learn- ing from time series with missing data. We propose a non-recursive counterpart, called DISE, based on latent representations that incorporates relative and absolute absolute time information. We show the benefits of an instance of this novel framework wherein latent representations for numerical data derived from the digits they made up of.
DISE relates to recent advances in the design of recurrent neural networks learn long-term de- pendencies. Pachitariu & Sahani (2013); Chang al. 2017) reinforce dependencies between non- consecutive states by adding connections between them. Different from these works, DISE does not re- quire to manually create such connections. Rather, the model is able to jump to subsequent states be- cause of its time-enriched representations. The continuous- citiiuuou--tmeeeee event prediction problem is a related topic. Event time series attend to sequences of ac- tions that occur asynchronously. Examples are clin- ical visits of patients to the hospital users pur- chasing history in an online marketplace. One major difference is that the input is always an event, which is typically a discrete token, and a single
30-
Dise: [R]
Dise-GRt [AR]
Abuzlute Error
28
Day month
Figure 4: PM2s-peak: MAE vs day.
continuous value. Solutions to this problem involve modifications of recurrent neural architecture Neil et al. 2016 Zhu al. (017) the use of some type of time representation that is typically concatenated to the event latent representation and used as the input to the recurrent architecture (Du
al. 2016 Mei & Eisner 2017 Kazemi et al. 2019). Most related our work is Li et al. (2017). wherein event embeddings are enriched with time information via the encoding function fEEN- A
second related topic is data imputation (Rubin) 1976). The major difference that data imputation methods also use subsequent values to fill the missing values. The study of the DISE framework to these two related problems is left for future work. One might also want to evaluate DISE with recent recurrent architectures whose hidden state dynamics equations (Rubanova et al.] 2019 De Brouwer et al. 2019). Most importantly, future work should make DISE also suitable for learning from multivariate time series.
specified by neural ordinary differential

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large- scale machine learning. In 12th {USENIX) Symposium on Operating Systems Design and Imple- mentation ((OSDI) 16), pp. 265-283, 2016.
Inci M Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K Jain, and Jiayu Zhou. Patient subtyping via time-aware lstm networks. In Proceedings of the 23rd ACM SIGKDD international conference knowledge discovery and data mining, pp. 65-74. ACM, 2017.
Mikolaj Binkowski, Gautier Marti, and Philippe Donnat. Autoregressive convolutional neural net- works for asynchronous time series. In ICML 2018: Thirty-fifth International Conference on Machine Learning. pp. 579-588, 2018
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems, pp. 6775-
2018.
Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson and Thomas S Huang. Dilated recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 77-87, 2017.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag. and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085,
2018.
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In SSST@ EMNLP, 2014.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically-osservedd time series. In Advances neural information processing systems, 2019.
Nan Du Hanjun Dai. Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez Modnez-Rorriguez and Le Song. Recurrent marked temporal point processes: Embedding event history vector. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1555-1564. ACM, 2016.
Ian Fox, Lynn Ang, Mamta Jaiswal, Rodica Pop-Busui, and Jenna Wiens. Deep multi-output fore- casting: Learning to accurately predict blood glucose trajectories. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1387- 1395. ACM, 2018.
Alberto García- Durrán, Sebastijan Dumancic, and Mathias Niepert. Learning sequence encoders for temporal knowledge graph completion. In EMNLP, 2018
Michiel Hermans and Benjamin Schrauwen. Training and analysing deep recurrent neural networks. In Advances in neural information processing systems, pp. 190-198, 2013.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jasprect Sahota, Sanjay Thakur, Stella Z. Wu, Cathal Smyth, Pascal Poupart, and Mareus A. Brubaker. Time2vec: Learn- ing vector representation of time. ArXiv, hts//197.05321.. 2019.
Han-Gyu Kim, Gil-Jin Jang, Ho-Jin Choi, Minho Kim, Young- Won Kim, and Jaehun Choi. Recur- rent neural networks with missing information imputation for medical examination data predic- tion. In 2017 IEEE International Conference on Big Data and Smart Computing (BigComp), pp- 317-323. IEEE, 2017.
Yeo-Jin Kim and Min Chi. Temporal belief memory: Imputing missing data during mnn training. In In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI-2018), 2018.
10

---

Under review as a conference paper at ICLR 2020
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Alex M Lamb, Anirudh Goyal Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor foreing: A new algorithm for training recurrent networks. In Advances In Neural Information Processing Systems, pp. 4601-4609, 2016.
Yang Li, Nan Du, and Samy Bengio. Time-dependent representation for neural event sequence 00065, 2017.
prediction, arXiv preprint
Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with mns: Improved classification of clinical time series. In Machine Learning for Healthcare Conference, pp. 253-270, 2016.
Yonghong Luo, Xiangrui Cai, Ying Zhang, Jun Xu, et al. Multivariate time series imputation with generative adversarial networks. In Advances in Neural Information Processing Systems, pp- 1596-1607, 2018.
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self-modulating mul- tivariate point process. In Advances Neural Information Processing Systems, pp. 6754-6764, 2017.
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased Istm: Accelerating recurrent network training for long or event-based sequences. In Advances in neural information processing systems, pp. 3882-3890, 2016.
Marius Pachitariu and Maneesh Sahani. Regularization and nonlinearities for neural language mod- els: when are they needed? ArXiv, abs/1 301.5500,,, 2013.
Pouya Pezeshkpour, Liyan Chen, and Sameer Singh. Embedding multimodal relational data for knowledge base completion. In EMNLP, 2018.
Yulia Rubanova, Ricky T. Q Chen, and David Duvenaud. Latent odes for imegularly-sampled time series. In Advances in neural information processing systems, 2019.
Donald B Rubin. Inference and missing data. Biometrika, 63(3):581-592 1976.
Joseph L Schafer and John W Graham. Missing data: our view the state of the art. Psychological methods, 7(2):147, 2002.
Souhaib Ben Taieb, Gianluca Bontempi, Amir F Atiya, and Antti Sorjamaa. A review and com- parison of strategies for multi-step ahead time series forecasting based on the nn5 forecasting competition. Expert systems with applications, 39(8):7067-7083, 2012.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In SSW, 2016a.
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In ICML, 2016b.
Xiuwen Yi, Yu Zheng, Junbo Zhang, and Tianrui Li. St-mvl: Filling missing values in geo-sensory time series data. In ISCAI, 2016.
Jinsung Yoon, William R. Zame, and Mihaela van der Schaar. Estimating missing data in temporal data streams using multi-directional recurrent neural networks. IEEE Transactions on Biomedical Engineering, 66: 66:1477-1490, 2017.
Jinsung Yoon, William R Zame, and Mihaela van der Schaar. Deep sensing: Active sensing using multi-directional recurrent neural networks. 2018.
Yu Zhu, Hao Li, Yikang Liao, Beidou Wang, Ziyu Guan, Haifeng Liu, and Deng Cai. What to do next: Modeling user behaviors by time-Istm. In IJCAI, pp. 3602-3608, 2017.
11

---

Under review as a conference paper at ICLR 2020
A
NOTATION: ONE EXAMPLE
An illustration of a time series under the two notations defined in Section is depicted in Figure
Time Series
47 N.A.
N.A.
40
N.A.
Series
Timesstamps
0
2
10
12
14
Timestangs
8
12
14
Masking 278
Daate-Fommaried Timestamps
06:00
14:00
20:00
Figure 5: (Left) A univariate time series with missing values. (Right) Alternative notation for the same time series.
B RISE&DISE
B. LEARNING WITH GRU
As discussed, neither RISE not DISE are restricted to particular type of recurrent network. As in prior work (Che et al. 2018), we opt for gated recurrent unit (GRU).
Within the RISE framework, the equations that define the reset and update gates of the GRU unit to control the hidden state are as follows:
r; a (W,2; + U,
2; O (W. + U.h; +
(6)
G; tanh W U @ (4-1))+++00
h, (1 - 2,) @ + 0c
where a is the sigmoid function, @ is the element-wii product, and all parameters those of the standard setting where the GRU unit operates.
The GRU equations used in the DISE framework
follows:
r:[47] a W +,
2y [ (W + U
(7)
c|4,]] = 6 +0
(A,1) + +b)
h [4" = (1 2, A @ h4
[ + [.A
B.2 STACKED DISE
extension of DISE to increase its depth would consist of just stacking recurrent neural networks on top of each other (Hermans & Schrauwen, 2013). We modify this standard approach with the goal of reinforcing time information into higher layers. do so we also apply gating mechanisms each layer. The time-enriched representations the I-th layer defined
" = (A7 o ow
(1) Senc (6
fenc (f+ta)
(8)
|1 [ (V(" 8 (5%h)) @ (V (enc((
where the input the /-th layer is defined
(o)) h h
1=0
0,47700 =
1> 0
and the Wi (1) matrices map all latent representations to the same dimension. A prediction is computed fo (L) where L is the number of layers.
B.3 RISE2DISE
RIsE2DIsE defines discount factors similar to Equation 33 that are computed with respect to future observed values: Ph. 6/() and 7/t = =8"(8f Since is not able to incorporate late-formatted
12

---

Under review as a conference paper at ICLR 2020
timestamps, 4 simply amounts to (8/)) It learns time-enriched represttttons for input and hidden state as follows:
0, 0, - go 0 T2n (1 -Va uwy
(10)
h |4,4 gn ( 14 (f.)) = -
where Day is the average of the values prior to I; of vector o.
C TRAINING DETAILS
We use the same evaluation protocol and setup as that of Fox et al. (2018). We stack two GRUs of 512 hidden units d, for all approaches. When using the multi-classification formulation, the probability distributions outputted by the function fe are translated to predictions by taking the value represented by the class with maximum probability. Models are evaluated at any point in time in which there are at least ten samples of prior data. We validate performance according to the median APE on the validation set after every epoch. We validate the weight decay between the values {10-2 10 11--3. Models were trained for 100 epochs, but usually the best validated model is obtained within the first 50 epochs. All remaining model details, such as the initialization procedure and the initial learning rate for ADAM (Kingma & Ba 2015). used the Tensorflow (Abadi et al., 2016) default values. The same evaluation protocol and setup has been followed for all trained models.
When using the digit-level GRU fgru as encoding function, we embed tokens of the vocabulary into 64-dimensional space, and the number of hidden units da of the standard GRU is validated between (64, 128]. To avoid increasing complexity during training this hyperparameter is validated globally, and not individually for each dedicated encoding function. Similarly, the dimension da of the feedforward encoders is validated between (64, 128}.
D EXPERIMENTAL SECTION: NUMERICAL AND VISUAL INFORMATION
D.1 DATASETS
Blood Glucose This constitutes the main benchmark data set of our experiments. We run experiments in this data set for the time horizon prediction problem, as the preprocessed data set free missing values. We also run experiments in this data set for the next observed value prediction problem. We assume the existence of probability distribution P(8) that determines the time gap 8, measured in sampling periods, between two observed values. This probability distribution is chosen
to mimic the presence of blocks of consecutive missing values, which is the main driving factor in the design of all above-mentionee methods. Thus, we draw values from
an exponential distribution P(8) exp 33 (with 8 2 1) to determine the time gap in terms of sampling periods from one observation to the next Thus, all intermediate observations dropped. The hyperparameter B controls the average time gap between pairs of observations, the larger the value is, the larger the average time gap between observations is.
Information about time gap values between consecutive observations is displayed in Figure 6 While in the blood glucose data sets these values have been perfectly generated by exponential distribu- tion, one can observe the time gaps for the air pollution data sets are more irregularly distributed.
Following Fox al. (2018), where blood glucose measurements
split into chunks of 101 data points, we also split the air pollution time series into chunks of 96 measurements (4 days).
Statistics of the data sets can be found in Table
13

---

Under review as a conference paper at ICLR 2020
10
Giacoes (2
10-
Pulkdion PMp
Fiood Flurnee 6
PMbs
CAUTT
10
[in sampiing perioda)
samplint
Figure 6: Histogram of time gap values between consecutive non-missing observations.
Data sct
# Observed Values [%6
# Missing Valies [%]]
Range
Sampling Period
Blood Glucose
616,043 [100%)
0 [0%)
40 400
minutes
Blood Glucose (Ø
372,4004 [61%)
245,669 [39%)
40-400
minutes
Blood Glucoss (B 5)
106.975 [18%]
509,068 [822%]
40 4001
minutes
Air Follution PM
273.553 [87%)
41,807 [138]
1 1,000
I hour
Air Follution PM0n
173,243 [55%]]
142,117 [45%]]
5- 1,000
hour
Table 5:
Data set statistics.
D.2 EXPERIMENTS
For the Blood Glucose (B- 5) data set, we show the correlation matrices of the representations learned for a range of values of the input and time gap signals in Figure 7 For small values of the input signal the respective encoding function learns dedicated representations, whereas for larger values the representations are more correlated. Interestingly, the correlation matrix of the time gap representations can be divided into block matrices: one for short-term predictions (up to 10 minutes), one for mid-term predictions (from 10 to 50 minutes) and one for long-term predictions (from 50 minutes onwards).
Convaation Matric - Frru
Conrestioo Matric
15
150
70
125
455
355
143
100
155
115
[jng/d]
[minutes]
Figure 7: (Left) Correlation matrix of the representations learned by the encoding functions fâru and (Right) pgu in Blood Glucose (B = 5).
In both correlation matrices one may observe some abrupt transitions. As an example, for values of the input signal before and after 200 mg/dL the learned representations are negatively corre- lated. Similarly, when the time gap between two observations takes values before and after 50 minutes- thitteess-hhsaaee amounts 10 once normalized by the sampling period the correlation relatively small. Interestingly, in both cases this occurs when there is change in the leftmost digit of the value. Therefore, one may wonder whether the representations learned by the encoding function feru driven by the atomic units the numerical data is decomposed into. We run experiments where numerical data is converted into hexadecimal and decomposed as a sequence of elements of the vocabulary (0- 9, a-). Decimal and hexadecimal notation schemes are compared in Table 6 all blood glucose problems.
The correlation matrices obtained with a hexadecimal notation are depicted in Figure While visually similar to those of Figure / one may observe some differences: the correlation matrix for the input signal is very smooth, being seemingly unaffected by changes in the leftmost digit; while
14

---

Under review as a conference paper at ICLR 2020
Notation
Decimal
Hexadecimal
Data
Blood Glucose (Horizon Prediction)
3.86
7.20
3.89
7.25
Blood Glucose (B =
2.06
3.97
2.02
3.88
Blood Glucose (B = 5)
5.51
11,29
5.61
10,92
Table Blood Glucose: DISE-GRU [R]'s performance with decimal and hexadecimal notation.
the correlation matrix for the time gap signal also presents block matrices, the transition from the second to the third block is much smoother than that of its counterpart with a decimal notation.
Correbtiion
1.00
300
25-
70-
0.00
-191
-175
[mn//L)
[minstm]
Figure 8: (Left) Correlation matrix of the representations learned by the encoding functions fāru and (Right) 1 in Blood Glucose (B 5). Numerical data is converted into hexadecimal.
Experiments in the air pollution data sets have shown that timestamps have large positive impact to model abrupt changes in the bias of the signal. The correlation matrices the representations learned for the timestamps t" are shown in Figure 9 Most of the values in the correlation matrix for the particulate PM2.5 are in the range between 0.95 and On the other side, the correlation matrix for PM2.5-pen peak is directly interpretable and matches the prior knowledge The encoding function feru. learns dedicated representations for the 2nd and 1 15th day of the month, which will serve to allocate some dimensions of the input and state representation to handle this property of the signal.
have of the data.
PMgs
Air Pollution
34/22
05
27/120
00/12
monts
12/12
14/12
22/24
2/22
20/2
Figure 9: (Left) Correlation matrix the representations learned by the encoding function f in PM2.5 and (Right) PM2 peak.
15


Reviews:
[{"review_id": "r1eYpXoJ9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall:\nProvides a nice summary of different methods for dealing with missing values in neural net time series models and proposes a new technique that does not involve running a series of possibly diverging predictions but rather jumps ahead to reason about arbitrary points in the future in a “single hop”, avoiding the risks associated with compounding errors. Also proposes a new method for encoding values that’s quite unusual but appears to work very well.\n\nOverall, the paper is mostly clear, the technique is reasonable, and the best model does indeed appear to work well. I have only one serious reservation about this paper - and it is an extremely serious concern about the experimental setup, and I would ask that the authors clarify this point for me in a response. DISE works poorly or only comparably well to the baselines in all tasks unless the GRU-based input encoding is used. Obviously any RISE model likewise depends on an input encoding, so the question is whether the baseline RISE models were given the benefit of the GRU-based input encoding. If not, please provide this comparison.\n\nMinor comments:\n“replace the standard input with a transformed input ˆx” -> I find this wording awkward. If the input is missing, it cannot be transformed, it can be predicted using a conditional model over data given a representation of past (and/or future) observations, or it can be a (probably learned) dummy value, but please clarify this wording- it’s essential.\n\nSimilarity of DISE to prior work:\nThere are a number of processes that build representations based on measurements that happen at random times without “rolling forward” a single step model, for instance, the neural Hawkes process (Mei and Eisner, 2017 or so), which has also been applied to impute missing values. Some discussion of the relationship to work like this is recommended. Additionally, the idea of learning representations based on predicting values at several time scales into the future comes up in contrastive predictive coding (van Oord et al, 2018).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Ske0bzVTYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies missing value imputation in univariate time series. The paper introduces a framework called RISE which provides a unified framework for existing methods in the domain. The authors further propose DISE which generalizes RISE. Experiments on time series forecasting demonstrate improved performance.\n\n+ It is quite interesting that the unified framework RISE can encompass the existing missing value imputation methods as special cases.\n+ The idea of using learnable factors for relative and absolute time information in DISE makes sense.\n\n- The alternative notations for the proposed framework and the existing framework are very confusing. It is not clear why such alternative notations for the same setting are necessary.\n- The modeling novelty is quite limited. Other than learnable factors for absolute and relative time information, there is very little motivation or theory regarding the modeling choice.\n- The experiments are not well motivated. The paper compares with a lot of missing value imputation baselines but the experimental setup is actually for extrapolation. In this case, a more proper set of baselines should be time series forecasting methods for irregularly sampled data such as Phased LSTM.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxRerZEtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new univariate time series analysis framework called RISE, which unifies the existing work on adapting RNNs to irregular time series. Building on top of the RISE, they propose a modification called DISE in which the algorithm skips the intervals without any observations. In that sense, DISE can be considered a marked point process analysis algorithm. They quantify the performance of the RISE and DISE on two datasets.\n\nTable 1 is a valuable summary of the existing efforts on adapting RNNs to irregular time series. However, the paper overstates its scope. This work only studies RNNs. There are many alternatives for analysis of irregular time series, including Gaussian processes [1], ordinary differential equations [2], convolutional neural networks [3], neural point processes and spiking neural networks [4]. These references are only notable examples from each category and there are many more.\n\nA major limitation of this paper is that it only applies to the univariate time series. Usually in domains such as healthcare, almost always different variables have different missing rates. Multiple works address the multivariate case, see [5] for example.\n\nThe main dataset used for evaluation is the Glucose dataset. However, this dataset is a peculiar and very specific dataset because its goal is to predict glucose level for type-1 diabetics only base on the past glucose measurements. While this task is meaningful, human biology states that forecasting glucose levels without knowledge of insulin injection or carbohydrate consumption is an extremely difficult task. In this setting, the most useful data is the latest data point. This dataset is an extreme forecasting task in absence of major predictors and I do not think it should be the primary dataset for evaluation of a new algorithm. \n\nFinally, the main idea of skipping the intervals without measurements is not very novel given the existing literature on neural point processes. Also, it is not enough contribution for a full conference paper.\n\n[1] Shukla, Marlin (2019) Interpolation-Prediction Networks for Irregularly Sampled Time Series. In ICLR.\n\n[2] Chen, T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. K. (2018). Neural ordinary differential equations. In NeurIPS.\n\n[3] Nguyen, P., Tran, T., Wickramasinghe, N., & Venkatesh, S. (2016). Deepr: a convolutional net for medical records. IEEE BHI.\n\n[4] Islam, K. T., Shelton, C. R., Casse, J. I., & Wetzel, R. (2017). Marked point process for severity of illness assessment. In Machine Learning for Healthcare Conference.\n\n[5] Che, Z., Purushotham, S., Li, G., Jiang, B., & Liu, Y. (2018). Hierarchical deep generative models for multi-rate multivariate time series. In ICML.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper attacks the important problem of learning time series models with missing data and proposes two learning frameworks, RISE and DISE, for this problem. The reviewers had several concerns about the paper and experimental setup and agree that this paper is not yet ready for publication. Please pay careful attention to the reviewer comments and particularly address the comments related to experimental design, clarity, and references to prior work while editing the paper.
Decision: Reject