Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Overall:
Provides a nice summary of different methods for dealing with missing values in neural net time series models and proposes a new technique that does not involve running a series of possibly diverging predictions but rather jumps ahead to reason about arbitrary points in the future in a “single hop”, avoiding the risks associated with compounding errors. Also proposes a new method for encoding values that’s quite unusual but appears to work very well.

Overall, the paper is mostly clear, the technique is reasonable, and the best model does indeed appear to work well. I have only one serious reservation about this paper - and it is an extremely serious concern about the experimental setup, and I would ask that the authors clarify this point for me in a response. DISE works poorly or only comparably well to the baselines in all tasks unless the GRU-based input encoding is used. Obviously any RISE model likewise depends on an input encoding, so the question is whether the baseline RISE models were given the benefit of the GRU-based input encoding. If not, please provide this comparison.

Minor comments:
“replace the standard input with a transformed input ˆx” -> I find this wording awkward. If the input is missing, it cannot be transformed, it can be predicted using a conditional model over data given a representation of past (and/or future) observations, or it can be a (probably learned) dummy value, but please clarify this wording- it’s essential.

Similarity of DISE to prior work:
There are a number of processes that build representations based on measurements that happen at random times without “rolling forward” a single step model, for instance, the neural Hawkes process (Mei and Eisner, 2017 or so), which has also been applied to impute missing values. Some discussion of the relationship to work like this is recommended. Additionally, the idea of learning representations based on predicting values at several time scales into the future comes up in contrastive predictive coding (van Oord et al, 2018).
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The paper studies missing value imputation in univariate time series. The paper introduces a framework called RISE which provides a unified framework for existing methods in the domain. The authors further propose DISE which generalizes RISE. Experiments on time series forecasting demonstrate improved performance.

+ It is quite interesting that the unified framework RISE can encompass the existing missing value imputation methods as special cases.
+ The idea of using learnable factors for relative and absolute time information in DISE makes sense.

- The alternative notations for the proposed framework and the existing framework are very confusing. It is not clear why such alternative notations for the same setting are necessary.
- The modeling novelty is quite limited. Other than learnable factors for absolute and relative time information, there is very little motivation or theory regarding the modeling choice.
- The experiments are not well motivated. The paper compares with a lot of missing value imputation baselines but the experimental setup is actually for extrapolation. In this case, a more proper set of baselines should be time series forecasting methods for irregularly sampled data such as Phased LSTM.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The authors propose a new univariate time series analysis framework called RISE, which unifies the existing work on adapting RNNs to irregular time series. Building on top of the RISE, they propose a modification called DISE in which the algorithm skips the intervals without any observations. In that sense, DISE can be considered a marked point process analysis algorithm. They quantify the performance of the RISE and DISE on two datasets.

Table 1 is a valuable summary of the existing efforts on adapting RNNs to irregular time series. However, the paper overstates its scope. This work only studies RNNs. There are many alternatives for analysis of irregular time series, including Gaussian processes [1], ordinary differential equations [2], convolutional neural networks [3], neural point processes and spiking neural networks [4]. These references are only notable examples from each category and there are many more.

A major limitation of this paper is that it only applies to the univariate time series. Usually in domains such as healthcare, almost always different variables have different missing rates. Multiple works address the multivariate case, see [5] for example.

The main dataset used for evaluation is the Glucose dataset. However, this dataset is a peculiar and very specific dataset because its goal is to predict glucose level for type-1 diabetics only base on the past glucose measurements. While this task is meaningful, human biology states that forecasting glucose levels without knowledge of insulin injection or carbohydrate consumption is an extremely difficult task. In this setting, the most useful data is the latest data point. This dataset is an extreme forecasting task in absence of major predictors and I do not think it should be the primary dataset for evaluation of a new algorithm. 

Finally, the main idea of skipping the intervals without measurements is not very novel given the existing literature on neural point processes. Also, it is not enough contribution for a full conference paper.

[1] Shukla, Marlin (2019) Interpolation-Prediction Networks for Irregularly Sampled Time Series. In ICLR.

[2] Chen, T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. K. (2018). Neural ordinary differential equations. In NeurIPS.

[3] Nguyen, P., Tran, T., Wickramasinghe, N., & Venkatesh, S. (2016). Deepr: a convolutional net for medical records. IEEE BHI.

[4] Islam, K. T., Shelton, C. R., Casse, J. I., & Wetzel, R. (2017). Marked point process for severity of illness assessment. In Machine Learning for Healthcare Conference.

[5] Che, Z., Purushotham, S., Li, G., Jiang, B., & Liu, Y. (2018). Hierarchical deep generative models for multi-rate multivariate time series. In ICML.
review_title: Official Blind Review #1

Meta-review: The paper attacks the important problem of learning time series models with missing data and proposes two learning frameworks, RISE and DISE, for this problem. The reviewers had several concerns about the paper and experimental setup and agree that this paper is not yet ready for publication. Please pay careful attention to the reviewer comments and particularly address the comments related to experimental design, clarity, and references to prior work while editing the paper.
Decision: Reject