Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes a probabilistic data imputation model for multivariate time series. The proposed method uses a conditional diffusion model to estimate the (masked) conditional distribution of missing data given observed values. To utilise information from observed values, the authors employ a neural attention mechanism to capture spatiotemporal dependencies of time series. The authors use self-supervised learning for training the model.  In experiments, the author demonstrate competitive performance of their method against a number of comparable methodologies on time series imputation and forecasting tasks.

main_review: The proposed model is an extension of the previously proposed denoising diffusion probabilistic model (DDPM), where the Markov chain reverse process of DDPM is conditioned on observed data. More specifically, the denoising function that parameterises the mean of the conditional Markov chain distributions in the DDPM reverse process is provided with the observed data as an additional argument. 

The paper is well written and easy to follow. The proposed method builds on a methodology that has been shown to be competitive in a variety of domains including time series modelling. The task addressed by the authors is quite relevant. The utilisation of attention mechanism for capturing both spatial and temporal dependencies appears interesintg. In comparison to recent relevant methodologies, the method has been shown to be state-of-the-art on time series imputation and forecasting tasks involving a number of benchmark datasets.  

limitations_and_societal_impact: My main concern with the work is that methodologically, its contribution is rather limited. The main methodological contribution of this work is the modification of denoising function parameterising mu in eq (5), which in my view is a straightforward and minimal extension of DDPM. The architectural modification involving the use of neural attention layer appears to be interesting, but without any comparison it is difficult to say how crucial it is to the performance of the model as opposed to for instance using (bidrectional) RNNs.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on the problem of the probabilistic imputation of missing data in time series. This work proposes a novel approach for probabilistic imputation based on the score-based diffusion models. Experiments on time series imputation tasks show that the proposed method achieves better performance than the current SOTA methods in this space.  
main_review: ### Novelty and Significance 
- The problem addressed is very important in many domains. The proposed approach is novel.

### Quality
- The paper is missing comparisons with some standard baselines for probabilistic imputation/interpolation such as Single-Task [1] and Multi-task Gaussian Processes [2] and SOTA methods based on neural ODEs [3]. ODE methods also perform particularly well on forecasting tasks so should be compared in Table 5 as well. Furthermore, the authors are also missing a comparison with mTANs in Tables 2 and 3 considering mTANs can produce probabilistic interpolation by sampling multiple latent states and passing through the decoder. 
- Negative log-likelihood is a standard metric for evaluation uncertainty computations. What is the reason behind omitting such a standard metric when evaluating probabilistic interpolations and imputations?
- The authors should perform ablation to show the advantage of using temporal and feature transformer layers.

### Clarity
- The paper is well written and easy to follow.

### Additional Comments
- How are the interpolation and imputation tasks on the PhysioNet dataset different after performing discretization on the irregularly sampled dataset? Is the proposed approach applicable for handling irregularly sampled sequences directly since the discretization loses a lot of information about the time series?

#### References
1. C. E. Rasmussen and C. K. I. Williams. Gaussian processes for machine learning. Adaptive computation and machine learning. 2006.
2. E. V. Bonilla, K. M. Chai, and C. Williams. Multi-task Gaussian process prediction. In Advances in Neural Information Processing Systems, pages 153–160, 2008.
3. Y. Rubanova, R. T. Q. Chen, and D. K. Duvenaud. Latent ordinary differential equations for irregularly sampled time series. In Advances in Neural Information Processing Systems, pages 5320–5330. 2019.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors use diffusion probabilistic models for imputation / conditional generation and propose self-supervised training schemes for this task.

The authors demonstrate their method outperforms other state-of-art approaches for time-series data

main_review: - Originality:
  - This paper explains a novel albeit incremental application of existing techniques.
  - Inpainting/ imputation/ infilling using score-matching has previously been explored for images and recently also for time-series, specifically for audio data [1]. Similarly [1] uses a mask for conditional generation, though this is not the focus of the paper.
  - Class conditioned generative modeling has been explored previously [2], where the score network is augmented by a class-dependent embedding. This paper’s approach may be seen as an extension of this but instead based on an embedding with inputs of observed/conditional data rather than class.
  - Modeling missing data as masked is standard for inpainting [1,3,4,5] and generating masks for self-supervised training has been performed before in other domains e.g. NLP. but not in combination with score-matching or time-series imputation as far as I am aware.

- Quality
  - The authors’ experimental section is excellent and they compare their approach to many state of the art methods in a variety of domains.

- Clarity
  - The paper is in general very well written. 
  - A few important details are left for the appendices but are helpful for clarity and understanding the paper, specifically:
    - Definition of $\mu^{DDPM}$, line 97
    - The details on padding to ensure being same dimension inputs into the score network

- Significance
  - Although the individual techniques are not novel, the combination may be novel
  - The method is general, shows state of art performance and may be used in practice

Summary

Strengths:
 - Excellent experimental section and comparisons, the proposed method demonstrates state of art performance
 - Well written and clear paper

Weaknesses:
 - The work lacks significant novelty and is essentially an application of two existing methods 1) self-supervised training and 2) diffusion probabilistic models.  Score based models have previously been applied to time-series imputation and masked self-supervised training is fairly standard in training sequential models in NLP. I find it surprising that self-supervised techniques have not before been applied to train models for time-series imputation, as it would be an obvious extension of previous data imputation methods using pre-selected masks.

- Missing reference [1]

In summary, although there is nothing incorrect with the paper and the paper demonstrates good empirical results, unfortunately I believe the work is too incremental and lacks sufficient novelty to justify acceptance. 

References
[1] Mittal, Gautam, et al. "Symbolic music generation with diffusion models." arXiv preprint arXiv:2103.16091 (2021). \
[2] Nichol, Alex, and Prafulla Dhariwal. "Improved denoising diffusion probabilistic models." arXiv preprint arXiv:2102.09672 (2021). \
[3]  Jinsung Yoon, James Jordon, Mihaela Schaar “GAIN: Missing Data Imputation using Generative Adversarial Nets” PMLR 80:5689-5698, 2018. \
[4] Luo, Yonghong, et al. "Multivariate time series imputation with generative adversarial networks." Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018. \
[5] Guo, Zijian, Yiming Wan, and Hao Ye. "A data imputation method for multivariate time series based on generative adversarial network." Neurocomputing 360 (2019): 185-197.
limitations_and_societal_impact: The authors discuss societal / ethical impacts in Appendix H.
review_title: null

Meta-review: This paper uses conditional score-based diffusion models for probabilistic time series imputation. The authors use an attention mechanism to capture the temporal and feature dependencies. A self-supervised training method is developed. The paper is overall well-written and the method appears to outperform alternative imputation techniques on some metrics.

The proposed method is novel but was also perceived as a rather straightforward combination of existing techniques (conditional score ideas have already been used in different domains and the self-supervised technique has been applied in related areas). That said, it is by no means trivial to put this combination in practice and the empirics are compelling.

After the rebuttal of the authors, some of the reviewers increase their score. With scores of 6, 6, 6, there remains a lack of enthusiasm for the paper. However, I recommend acceptance of the paper as it is a useful contribution to probabilistic time series imputation.

Decision: Accept (Poster)