Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a deep generative model for multivariate time series with missing data (in the most general setup of missing-not-at-random, MNAR). The development of the model follows the same core design principles presented in [1], [2], and [3], extending it and proposing a joint hybrid model for the time series, the missing mechanism, and a classifier. The inference of such a model can be intractable in general, thus an approximation for the log-likelihood is proposed, adapting the Importance Weighted AutoEncoder (IWAE) lower bound for the proposed model (which defines the loss function to be optimized). Furthermore, it is proposed a regularization for this lower bound by dropping more portions of the observations, encouraging imputations that contribute towards the classification tasks. The empirical validation is done using multiple datasets and comparing with multiple baselines and state-of-the-art.

[1] P. Mattei and J. Frellsen. MIWAE: deep generative modelling and imputation of incomplete data sets. ICML 2019.
[2] N. B. Ipsen, P. Mattei, and J. Frellsen. not-MIWAE: deep generative modelling with missing not at random data. ICLR 2021.
[3] N. B. Ipsen, P. Mattei, and J. Frellsen. How to deal with missing data in supervised deep learning?. ICLR 2022.



strength_and_weaknesses: ## Strength
* The modeling choices and inference method is well motivated and principled
* The problem of imputation with NMAR in the multivariate time-series setup with a hybrid deep generative model is novel and potentially impactful
* Empirical results demonstrate the strength of the proposal

## Weakness:
* The methodology is incremental

typos:
Irrelvent -> irrelevant (page 3)

clarity,_quality,_novelty_and_reproducibility: Clarity:  The paper is well-written and clear. The method and model are well-motivated and clear

Quality: The derivations are technically correct (to the best of my knowledge). The modeling choices are appropriate for the different “modules” and their purpose (missingness, time series, classifier). The empirical validation is convincing

Novelty: Somewhat incremental.

Reproducibility: The paper is accompanied by code for the model and experiments (which I did not go through)


summary_of_the_review: Overall, this a solid and well-motivated work, tackling an relevant problem (NMAR imputation for time-series). My only concern is the limited technical novelty. That being said, I recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on imputation and prediction for time series with missing data. It extends an existing deep generative model to impute the missing values in the time series and make predictions given both the observations and imputations. A regularization method "obsDropout" is proposed to randomly drop some observed entries to force the prediction model to focus more on the imputed entries to drive the generation of good imputations.
strength_and_weaknesses: Strengths:
- The proposed model is reasonable and sound. Detailed analysis of the problem as well as existing work is provided.
- Although the proposed supnotMIWAE is largely a variant of supMIWAE, it adds new elements to tackle the MNAR and prediction and the formulation is overall sound.
- Overall, the paper is organized and easy to follow.

Weaknesses:
- My main concern regarding the method is that the prediction task may not be enough to drive the learning of imputation, especially MNAR. Even with the obsDropout regularization incorporated, the model only generates imputation that helps most the prediction but may not be the most accurate imputation.
- The empirical evaluation, compared against existing methods like GRU-forward and GRU-D, seems weak. The improvement over the SOTA baselines is quite marginal.
clarity,_quality,_novelty_and_reproducibility: The paper is overall organized and easy to follow. Regarding technical novelty, this paper presents some non-trivial extensions to existing work, but some assumptions (like driving the imputation using prediction only) might need stronger justifications. The detailed model specification, hyperparameter setting, and training details are lacking, making the paper less reproducible.
summary_of_the_review: Overall, this paper presents some interesting extensions of the existing deep generative model for time series missing data imputation. The formulations are sound. However, some methodological design needs stronger justification and the empirical evaluation seems weak.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: THis paper tackles time series classification problems with missing data. A deep generative model is used to impute missing data first before classification. The classifier is trained to capture the predictive uncertainty due to the multiple possibilities of imputations. A novel regularization technique is proposed that can promote the model to produce useful imputation values that actually
help classifcation. 
strength_and_weaknesses: Pros:
1. This paper is probabilistic and uncertainty is considered for classification
2. The regularization techniques makes sense

Cons: 
1. Several recent works have shown that GRU-D is not good at handle irregularly sampled data compared to ODE based methods, is it possible to extent this paper to handle irregularly-sampled data well?
2. In Table1 and Table2, the improvement over GRU-D is marginal, so I am not sure this method is as effective as claimed.


clarity,_quality,_novelty_and_reproducibility: Good
summary_of_the_review: Can't handle irregularly sampled data well, improvement is marginal
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors suggest a sequential VAE/IWAE framework for imputation and classification of time series data that is particularly applicable
 under the missing not at random (MNAR) setting. To improve the imputation performance, the authors suggest to mask some observed values and subsequently reconstruct them. The method is compared agains imputation approaches using GRUs that are not fully probabilistic. 


---
Update after the authors' response:
The authors have added an additional probabilistic imputation benchmark model which has been found to be outperformed by the approach suggested here. This improves the empirical validation of the new approach.

---
strength_and_weaknesses: Positives:
- The paper proposes a probabilistic model for time series data that allows for very flexible assumptions for the missing values. Imputation for the time series setting/for non-linear latent state-space models has not been explored in the literature, as far as I am aware. 
- The method appears to yield better empirical performance on two benchmark datasets, compared to imputation approaches using GRUs.

Negatives:
-There appears to be a lack of empirical comparison to other probabilistic imputation approaches, such as Fortuin et al., 2020. Even if such approaches make strong assumptions on the missingness patterns, it would be insightful to see how this translates into the empirical forecasting performance.
- The missingness model appears relatively simple compared to the other components (encoder, classifier) in that it does not depend on the full history of the observations via the RNN latent state. A more flexible model should not require 'data-specific design' choices. 

Comments/Actionable feedback:
-For the suggested bound with the dropout variables in (22), does this constitute a lower bound on something sensible?
- My feeling is that even without dropout, there should be an incentive to learn non-trivial imputations, because I am not only minimising the classification error, but also the log-probability of the observed time series.
- A semi-supervised approach to regularise the missing data has been suggested in Sahra Ghalebikesabi, Rob Cornish, Chris Holmes, Luke Kelly: Deep Generative Missingness Pattern-Set Mixture Models. AISTATS 2021. I was wondering how this relates to the dropout procedure suggested here?




clarity,_quality,_novelty_and_reproducibility: The paper is generally well written. The presentation is clear. 
Accounting for MNAR for latent state-space models is novel, as far as I am aware.
summary_of_the_review: The proposed modelling setup is new and addresses a relevant question that has not yet been addressed previously. The model (encoder, decoder) is somewhat standard. Including the missingness model is new, as well as the dropout procedure, along with the variational bound. The empirical performance appears favourable compared to imputations with GRUs, but in my opinion, a more detailed empirical comparison with other probabilistic imputation approaches would improve the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This interesting paper has been thoroughly reviewed by four knowledgeable reviewers. Three of them assessed it below the threshold for acceptance, one suggested an accept. Even the strongest review points out the key deficiency of the paper in its current form: its limited novelty versus the prior work. Others point the novelty as marginal, and suggest that more comprehensive empirical studies would need to be conducted to reinforce the claims being made. In summary, I find this work interesting and very promising, but at its current stage it is below the threshold of ICLR acceptance. However, I would like to encourage the authors to continue maturing it.
justification_for_why_not_higher_score: Marginal novelty.
justification_for_why_not_lower_score: n/a
Decision: Reject