Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a modified gated RNN caled GRU-D that deals with time series which display a lot of missing values in their input. They work on two fronts. The first deals with the missing inputs directly by using a learned convex combination of the previous available value (forward imputation) and the mean value (mean imputation). The second includes dampening the recurrent layer not unlike a second reset gate, but parametrized according to the time elapsed since the last available value of each attributes.

Positives
------------
- Clear definition of the task (handling missing values for classification of time series)
- Many interesting baselines to test the new model against.
- The model presented deals with the missing values in a novel, ML-type way (learn new dampening parameters).
- The extensive tests done on the datasets is probably the greatest asset of this paper.

Negatives
-------------
- The paper could use some double checking for typos.
- The Section A.2.3 really belongs in the main article as it deals with important related works. Swap it with the imprecise diagrams of the model if you need space.
- No mention of any methods from the statistics litterature.

Here are the two main points of this review that informs my decision:

1. The results, while promising, are below expectations. The paper hasn’t been able to convince me that GRU-simple (without intervals) isn’t just as well-suited for the task of handling missing inputs as GRU-D. In the main paper, GRU-simple is presented as the main baseline. Yet, it includes a lot of extraneous parameters (the intervals) that, according to Table 5, probably hurts the model more than it helps it. Having a third of it’s parameters being of dubious value, it brings the question of the fairness of the comparison done in the main paper, especially since in the one table where GRU-simple (without intervals) is present, GRU-D doesn’t significantly outperforms it.

2. My second concern, and biggest, is with some claims that are peppered through the paper. The first is about the relationship with the presence rate of data in the dataset and the diagnostics. I might be wrong, but that only indicates that the doctor in charge of that patient requested the relevant analyses be done according to the patient’s condition. That would mean that an expert system based on this data would always seem to be one step behind. 
The second claim is the last sentence of the introduction, which sets huge expectations that were not met by the paper. Another is that “simply concatenating masking and time interval vectors fails to exploit the temporal structure of missing values” is unsubstantiated and actually disproven later in the paper. 
Yet another is the conclusion that since GRU models displayed the best improvement between a subsample of the dataset and the whole of it means that the improvement is going to continue to grow as more data is added. This fails to consider that non-GRU models actually started with much better results than most GRU ones. 
Lastly is their claim to capture informative missingness by incorporating masking and time intervals directly inside the GRU architecture. While the authors did make these changes, the fact that they also concatenate the mask to the input, just like GRU-simple (without intervals), leads me to question the actual improvement made by GRU-D. 

Given that, while I find that the work that has been put into the paper is above average, I wouldn’t accept that paper without a reframing of the findings and a better focus on the real contribution of this paper, which I believe is the novel way to parametrize the choice of imputation method.
review_title: Interesting model, overreaching conclusions

final_score: rating: null
review_content: Interesting work!

Another simple baseline for dealing with missing values would be to add a standard NN layer (instead of a handcrafted one) for preprocessing the input consisting of the data and the mask,  feeding into the GRU layer. It is conceivable that if the preprocessing layer happens to be some sort of gated, recurrent layer (GRU/LSTM), that it would learn to impute the missing values and compute important statistics of the mask. Therefore, it would important to address the following question:
  
How does the performance of the GRU-simple baseline change if it is given more hidden layers (with same number of parameters and with increasing number of parameters)? 
review_title: Comparison to multi-layer RNNs

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a RNN-method for time-series classification with missing values, that can make use of potential information in missing values. It is based on a simple linear imputation of missing values with learnable parameters. Furthermore, time-intervals between missing values are computed and used to scale the RNN computation downstream. The authors demonstrate that their method outperforms reasonable baselines on (small to mid-sized) real world datasets. The paper is clearly written.
IMO the authors propose a reasonable approach for dealing with missing values for their intended application domain, where data is not abundant and requires smallish models. I’m somewhat sceptical if the benefits would carry over to big datasets, where more general, less handcrafted multi-layer RNNs are an option. 
review_title: null

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a way to deal with supervised multivariate time series tasks involving missing values. The high level idea is still using the recurrent neural network (specifically, GRU in this paper) to do sequence supervised learning, e.g., classification, but modifications have been made to the input and hidden layers of RNNs to tackle the missing value problem. 

pros: 
1) the insight of utilizing missing value is critical. the observation of decaying effect in the healthcare application is also interesting;
2) the experiment seems to be solid; the baseline algorithms and analysis of results are also done properly. 

cons:
1) the novelty of this work is not enough. Adding a decaying smooth factor to input and hidden layers seems to be the main modification of the architecture. 
2) the datasets used in this paper are small. 
3) the decaying effect might not be able to generalize to other domains. 
review_title: null

Meta-review: This paper presents a modification of GRU-RNNs to handle missing data explicitly, allowing them to exploit data not missing at random. The method is presented clearly enough, but the reviewers felt that the claims were overreaching. It's also unsatisfying that the method depends on specific modifications of RNN architectures for a particular domain, instead of being a more general approach.
Decision: Reject