Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper rigorously shows how to calculate a variational lower bound on the marginal log density of a continuous time diffusion model. The bound contains implicit score matching, and it is calculated using Girsanov’s theorem from a Feynman-Kac representation of the marginal density obtained from the Fokker-Planck equation.
main_review: This is an excellent paper. I appreciate that the authors are rigorous for continuous time processes, and I like how the authors derive a marginal density without an emission probability.
My only concerns are regarding its framing in the context of prior work. My suggestion is to be more precise in describing the contributions as showing that *implicit* score matching arises from a variational lower bound on log likelihood, as opposed to existing and concurrent work showing that *denoising* score matching arises from a variational bound derived differently, starting with discrete time argument in DDPM (which, in section 7, should also be credited for showing that denoising score matching is learning to reverse an inference process).

If the goal of deriving implicit score matching is made clear throughout the paper, I think it will clarify to the reader the advantages of their method of deriving the variational lower bound as opposed to more traditional approaches, e.g. the approach from the original DPM paper, or modifying the approach from Tzen and Raginsky. And from this perspective, it is quite remarkable that the implicit score matching formula arises from Girsanov and Feynman-Kac, and I commend the authors on this finding.

It would be nice if the authors could discuss their ELBO in a more general setting, e.g. in which both forward and reverse processes have learnable drifts, or the setting of Tzen and Raginsky. How will training work in that case? Furthermore, can the authors discuss exactly when their method has advantages or disadvantages over other bounds, for example in lines 286-293? In the variational inference setting of Tzen and Raginsky, what does the authors’ proposed ELBO look like, and how does it compare to Tzen and Raginsky’s ELBO?

It appears that sections 3 and 4 use notation that refers to general SDEs (e.g. equation 4), but sections 2 and 6 use the same notation to refer to score-based generative modeling SDEs (e.g. equation 2). The change in notation is a bit confusing to the reader.

limitations_and_societal_impact: Limitations are adequately addressed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents a framework for studying continuous diffusion models. Thanks to this it demonstrates the plug in generative model corresponds to a continuous version of the discrete diffusion models and is equivalent to minimising an ELBO on the likelihood of the model. 
It does so by first transforming the change of density associated to the SDE into a second order linear differential equation and then expressing its solution as an expectation over another SDE. Solving this expectation is performed via variation inference on the Brownian paths and yields the continuous-time ELBO.
This CT-ELBO is then compared with its discrete time counter part. Finally it is compared to the score matching loss which demonstrates the soundness of such method for learning the generative model and continuous time diffusion models.

main_review: Originality:
To the best of my knowledge this paper present completely new results. The approach to derive the different expression is novel as well. Overall I think the proposed CT-ELBO could lead to many interesting future works for variation of the current continuous diffusion models.

Quality:
I have to admit that my background and the time I was able to spend on this review does not let me judge accurately the quality of the methods presented in this paper. However I think the methodology proposed by the authors to express the CT-ELBO is very nice and I don’t see fundamental flaws in those steps.
I think the authors could have expressed the role of lambda explicitly in the manuscript as it is discussed later in the experiments. I also believe authors should have discussed more thoroughly why the claims in Durkan and Song are not accurate.

Clarity:
Overall I think the paper is quite clear and make a really great at presenting complex materials into a comprehensive way.  However I also have to admit section 6 was a bit harder to follow on my side (except the first paragraph) and could maybe be improved.

Significance:
This work is very important as 1) It provides formal results for using plug-in score matching in diffusion generative models. 2) It provides the tools for creating new SDE based generative models.

Additional remarks:
• EQ 13: ds missing?
• I did not get how the first part of 15 disappear into 16.
limitations_and_societal_impact: /
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper develops a variational framework for general diffusion processes. A continuous-time ELBO is developed that extends the ELBO of discrete-time diffusion processes. They use this to show that optimizing the score matching loss corresponds to maximizing a lower bound on the log marginal likelihood.

More specifically, the paper makes use of the Fokker-Planck/Kolmogorov forward equation along with the Feynman-Kac formula to express the marginal density of the generative diffusion process. Since the resulting expression is intractable, a change-of-measure is applied, introducing a variational approximation $\mathbb{Q}$ to $\mathbb{P}$. $\mathbb{Q}$ is chosen such that the Radon-Nikodym derivative $\frac{d\mathbb{Q}}{d\mathbb{P}}$ can be expressed using Girsanov theorem. From this, a continuous-time ELBO is obtained. Finally, through a specific choice of the drift coefficient $a$ in the inference SDE, a connection between the ELBO and score matching is established.
main_review: This paper provides a variational framework for general continuous-time diffusion processes, which to the best of my knowledge is novel. The continuous-time ELBO is shown to consistently extend the usual ELBO for discrete-time diffusion processes, which is very elegant. Further for a specific choice of the inference SDE, the ELBO is shown to correspond (up to a constant) to a score-matching loss with a specific weighting ($\Lambda = gg^T$). This provides a connection between the training of continuous-time score-based diffusion models [1] and maximum likelihood training, concurrent with [2] which uses a different approach.

For the negatives: The experimental results are quite poorly presented. The plots in Fig. 3 and 4 and difficult to read. I would suggest 1) zooming in and 2) considering a log scaling also of the x-axis. Furthermore, a proper discussion of the results is missing (especially for MNIST/CIFAR-10), making it more difficult for the reader to get the take-aways from the experiments. Finally, details on the experiment setups appear to be missing.

This is a technically strong paper with somewhat heavy, but elegant theory. It adds to our theoretical understanding of score-based diffusion models and opens doors for further theoretical and experimental work. I would therefore like to see this paper accepted.

Minor questions/comments:
- A reference or appendix section on the relations in Eq. (3) would be useful.
- Line 71: You refer to $\mathcal{I}(q) = \mathbb{E}\left[ \| \nabla \log q \|_{\Lambda}^2 \right]$ as the Fisher information. Isn't the Fisher information commonly defined as the matrix $\mathcal{I}(q) = \mathbb{E}\left[ (\nabla \log q)(\nabla \log q)^T \right]$?

[1] Score-based generative modeling through stochastic differential equations (Song et al. 2020)
[2] Maximum Likelihood Training of Score-Based Diffusion Models (Song et al. 2021)
limitations_and_societal_impact: In the checklist, the authors claim to describe limitations of their work, but with no reference to where. A dedicated paragraph to this could be beneficial. Potential negative societal impacts are not discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors derive a general variational formulation for likelihood estimation in diffusion based generative modeling and draws an equivalence relation between maximising the associated continuous time (CT) ELBO and minimizing specific score-matching losses.

The authors also provide a more formal interpretation to “diffusion models can be viewed as a continuous limit of hierarchical VAEs” and interesting commentary on bias/ variance trade-off for weighted losses in score-matching.
main_review: Although previous attempts have been made to connect score-matching and likelihood objectives, this paper provides stronger results with more insight.
Formal interpretation to “diffusion models can be viewed as a continuous limit of hierarchical VAEs” is novel, though incremental by itself.
The results appear correct. Although not the focus of this work, experiments are limited however, see below.
Work is clear, though perhaps poorly organised. Perhaps re-stating existing theorems (Girsanov/ Feynman Kac) is unnecessary and the excess space may be used to provide more discussion on the paper’s contributions.
This work provides insights into score-matching losses used in practice and the link to likelihood, I believe this will be used and built upon by researchers.

Strengths:
Derive novel CT-ELBO and connect score matching to ELBO in continuous time. This is a general ELBO and allows comparison for various ODE and SDE reverse diffusion generative models. 
Unlike previous work [2], the derived connection to specific score-matching losses is exact rather than an upper bound and provides insight into how various score-matching losses used in practice relate to likelihood and family of plug-in reverse SDEs. This formulation is elegant and provides better understanding. 
Maximising ELBO in continuous time using ODE/ SDE integrators is computationally prohibitive, this paper derives an equivalency to a more scalable, albeit noisier approach using score-matching. I believe this will be of interest to practitioners as it gives a better understanding of score matching losses used in practice.
The authors formally confirm that limit of likelihood of discretized diffusion (i.e. infinite hierarchical VAE) matches continuous time likelihood
Interesting discussion on bias-variance trade-off via weighting scheme in score matching losses

Weaknesses:
The authors spend a significant proportion of the paper stating existing theorems (Girsanov, Keynman Kac representation), yet leave original Theorem 5 and discussion on plug-in reverse SDEs to the appendices. 
Section 6 discussing the implications of main theoretical contribution is overly brief. 
Although the bias/ variance trade-off from weighting is explored for images MNIST/ CelebA, the main contribution regarding likelihood based learning is only verified on low dimensional Swiss Roll data. 
No experiments comparing numerical SDE integrators, just ODE. 

Minor:
I do not think Durkan [2] claims score matching is equivalent to max likelihood as you state in line 195, just that score matching upper bounds likelihood and is a proxy.
Missing references:
Reverse SDE [1] (for instance)
Notation undefined:
Hessian H_p not defined in equation (7) line 88

[1] U. G. Haussmann. E. Pardoux. "Time Reversal of Diffusions." Ann. Probab. 14 (4) 1188 - 1205, October, 1986. 
[2] Durkan, Conor, and Yang Song. "On maximum likelihood training of score-based generative models." 

limitations_and_societal_impact: Limited societal impact, this paper is mainly theoretical.
review_title: null

Meta-review: The main pros and cons that came up during reviews and discussions are:

pros:
* excellent paper with a elegant approach (k6w1, dz2y, xxp9, 99PV)
* new results opening up many interesting future works (dz2y, xxp9)


cons:
* framing in the context of prior work can be improved. (k6w1, dz2y)
* The advantages and disadvantages of proposed bound over other bounds could be clarified.  (k6w1)
* notation could be clarified. (k6w1)
* Section 6 can be improved in terms of clarity and can be expanded.(dz2y, 99PV)
* presentation of results can be improved. figures are hard to read and a discussion is missing. (xxp9)
* limited experimental results, although these are not the focus of the paper. (99PV)
* paper re-states too many existing theorems. Authors disagree, state it is important for a self-contained paper. (99PV) 

The authors and reviewers engaged in discussions during the discussion period and most of the concerns raised by the reviewers were addressed. 3/5 reviewers raised their scores, leading to a unanimous recommendation of acceptance. The recommended decision by the meta-reviewer is to accept the paper.
Decision: Accept (Spotlight)