Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to model time series data as discretized observations from an underlying continuous stochastic process using a diffusion framework. The main contribution of the paper is to extend the method proposed in [1] and [2] to the case where a stochastic noise process is used for the forward process. Based on the proposed method, the paper also developed approaches for multivariate probabilistic forecasting and imputation tasks.  The effectiveness of the proposed method is demonstrated through synthetic and real-world datasets.


Reference:

[1] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." Advances in Neural Information Processing Systems 33 (2020): 6840-6851.

[2] Song, Yang, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. "Score-Based Generative Modeling through Stochastic Differential Equations." In International Conference on Learning Representations. 2020.
strength_and_weaknesses: Strength: the paper is well-written and easy to follow. The proposed methodology appears to be reasonable.
Weakness: I am not so sure about the novelty of the proposed methodology.

1. Although the authors repeatedly emphasized about the denoising approach is designed for stochastic processes, the loss function (13) and (8) with score function (16) are still vector-based loss functions. It would make more sense to generate the noise process from a stochastic process instead of a multivariate normal distribution. Even if one needs to discretize the stochastic process, the dimension of the noise process should be higher than the original observed data. Under the current setting, it is really difficult to tell how much difference is there between the proposed method and those in[1] and [2], from the implementation point of view.

2. When considering the stochastic process, it is not as simple as "Gaussian+Gaussian" is still Gaussian. Even Gaussian processes with different covariance kernels have different properties such as smoothness and differentiability. This brings another question: what noise processes to use in practice? The paper proposes two candidates: a Gaussian process with a radial basis kernel and the Ornstein-Uhlenbeck process. Any guidelines on how to choose? Do parameters in the kernel function need to be estimated? Is it possible that a noise process with similar smoothness/differentiability/strength-of-dependence properties to the original data will yield better performance? These issues need to be investigated.


Reference:

[1] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." Advances in Neural Information Processing Systems 33 (2020): 6840-6851.

[2] Song, Yang, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. "Score-Based Generative Modeling through Stochastic Differential Equations." In International Conference on Learning Representations. 2020.
clarity,_quality,_novelty_and_reproducibility: The paper is very well written and easy to follow. I am not sure about the level of novelty of the paper.
summary_of_the_review: The paper is very well written and easy to follow. I am not sure about the level of novelty of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper describes a novel diffusion model extending the concept to modelling irregularly sampled time series by utilizing correlated noise obtained from noise processes to produce continuous samples. The paper provide an elegant framework to combine forecasting, and imputation tasks.
strength_and_weaknesses: One of the main advantage of the method is the elegant framework to combine all kinds of inference tasks for possibly sporadically observed time series. Forecasting (prediction in temporal causal direction), smoothing (prediction in both direction,) and imputation tasks can be formalized as conditional generation tasks in the framework.

One of the weakness is that little guidance is provided on how to select the noise process, and that not many benchmark method is provided, see later.
clarity,_quality,_novelty_and_reproducibility: The paper is quite well written, and easy to follow. The method is clearly novel, generalizing the diffusion models to time series is a nontrivial task. In one hand the proper noise process need to be used, and algorithms for interpolation, imputation and forward prediction need to be given.

Can you please give the used definition of continuity in case of stochastic process samples. What is the exact constraint you have on the noise process?

Do you have any idea on the effect of the stochastic process kernel on the model? We see the difference between OU and GP in the results. Do you have any observations you can share with the reader on the effect of the shape of the noise curve, effect of gamma parameter etc. ?

It would be useful to compare with more methods as it is not clear that the Latent ODE (Rubanova et al. 2019)  is the best alternative method. There is a concurrently published work (De Brouwer et al. 2019) that applies Bayesian filtering in the NODE framework, and a later paper (Kideger et al 2020) using the formulation of controlled differential equations and in some situation outperforms the previous two,
It is clearly a biased list given I am familiar with the NODE literature more than other similar methods.

(De Brouwer et al. 2019) De Brouwer, Edward, et al. "GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series." Advances in neural information processing systems 32 (2019).
(Kideger et al 2020)  Kidger, Patrick, et al. "Neural controlled differential equations for irregular time series." Advances in Neural Information Processing Systems 33 (2020): 6696-6707.

Note that I feel the work is novel enough that it is useful for the community even if it will not over-perform every single baseline, it would still be quite useful to see a bit more benchmark, even for just a selected dataset.

summary_of_the_review: The paper is quite well written and the method is novel. A little bit more detailed comparison would benefit the paper, but in general I find the paper a valuable addition to state of the art.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposes to learn a diffusion based generative model to model temporal data. The model is able to handle They test their model on various tasks such as forecasting and interpolation.
strength_and_weaknesses: Strength:
- The abstract and introduction give a nice overview and motivation for the problem.
- Overall the paper is clear.
- The results are good and evaluation is somehow reasonable.

Weaknesses:
- I can't find the meaning of the bold format in your tables.
- You should run a statistical test to compare the different methods.
- I am not sure the editors' names are needed in the bibliography, removing them would ease the reading of it
clarity,_quality,_novelty_and_reproducibility: Overall the writting is good. Addressing irregular sampled data with diffusion models and modeling the time series as a continuous function is, to the best of my knowledge, new.
summary_of_the_review: Overall I think this is a good paper tackling a well motivated and realistic task with diffusion models in the setting where temporal data are not sampled regularly.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a diffusion model focusing on time series data where the diffusion function takes the predefined matrix from Cholesky decompositions of Gaussian processes. Experiments show some interesting results on time series predictions and imputations.

strength_and_weaknesses: Strength:
- The paper presents an interesting idea as the covariance is introduced in diffusion processes to express some degree of correlation between temporal data points.
- For prediction tasks, the method is able to produce estimations at multiple time points unlike the previous work [Rasul et al 2021a] which follows autoregressive approaches. The key contribution here is the temporal correlation defined by covariance functions..

Weakness:
- The paper explains the approach as replacing the noise vector as a stochastic process. In fact, in a much simpler view, this way is the same with the diffusion process that just has a diffusion function as a matrix $L$ from Cholesky decomposition. In light of these observations, the contribution of formulating discrete and continuous versions of diffusion models is not significant. 
- The justification for the loss function in Appendix A.2 is subtle without evidence to support the claim. In my opinion, $\Sigma^{-1}$ plays an important role in the curvature of loss functions, containing information of temporal dependencies.

clarity,_quality,_novelty_and_reproducibility: Question:
- On the scalability of the method, to compute the loss, the proposed method needs to compute Cholesky decompositions.The number of data points in the studied data sets seems quite big. I agree that Cholesky computaions can be done one time at the beginning of trainning. However, it can take some time to compute $\epsilon$ (line 5) in Algo. 1. How does it affect the training time?
- How to select hyperparameters ($\gamma$) for Gaussian process kernels?

Minor point:
- I’m curious about the title using the term “process diffusion”. I feel somewhat strange about this choice as I’m mostly familiar with the diffusion process as a stochastic process.


In general, the paper is fairly easy to follow. The paper mentions that the source code is available to reviewers but not found.
summary_of_the_review: Although the paper presents a nice idea, I feel the technical contribution of the paper is limited as stochastic process for noise part is not different than matrix-valued diffusion functions for diffusion models.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
In this paper, the authors extend the diffusion model for modeling functional data, which are represented as a set of points with different length. 

The paper is well-written and easy-to-follow. 

The major concerns raised by the reviewers:

- Although the paper is trying to establish the functional generalization of diffusion model, the current presentation does not emphasize this, which makes the reviewers confused.  

- The experiment part is relatively weak. As the reviewer mentioned, the current competitors are limited. More competitors, e.g., GRU-ODE and other neural processes, should be included to demonstrate the benefits. 


Minors: 

- The current derivation eventually leads to Gaussian processes. A more detailed discussion might be required to show the benefits over the classic Gaussian processes, especially with parametrized kernel learning. 

- The limited distribution still follows Gaussian, according to Eq. 15. Does this mean the model can only be applied to regression? How this model can be used for classification?

justification_for_why_not_higher_score: In sum, the paper provides an interesting and promising model for temporal data. However, there are still several points not clarified. I encourage the authors to consider the reviewers' suggestion to improve the paper.
Decision: Reject