Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper generalizes a family of score-based generative models that rely on sequences of noise scalings of the data and extends them to the continuous domain, which leads to an SDE-based framework. By using score-matching, the forward SDE, which transforms data into a tractable noise distribution, can be reversed and thus used as a generative model. This is then improved by employing a two-phase algorithm with a prediction step, followed by a tunable number of correction steps. Further, reformulating the problem as a neural ODE allows for exact likelihood computations and reduces the number of required function evaluations. The framework enables unconditional, as well as conditional samples.

I find the paper to be very well written and straightforward. As someone who does not have neural SDEs or Langevin Samplers as a core competence, I was able to follow all of the writeup, which is remarkable. I think the framework is nice and there is substantial novel innovation to justify accepting this paper. The experiments are convincing.

A few questions and remarks:
- You claim that you unify current methods into a common framework. While I see that you attempt to do this (i.e. putting the algorithms side-by-side, etc.), but in essence, you still handle VE and VP SDEs separately throughout. My suggestion would be to either really try to unify them into a single formulation, or alternatively, tone down the claims of unification, maybe just say that you show commonalities.
- In Figure 2, you claim that the results are best when computation is "split" between the predictor and corrector. However, this is a very imprecise statement. An equal split would mean just 1 step of corrector, but I don't see clear evidence that that's best. Do you have numerical evidence that a 1-to-1 split is best, or what do you mean by "split"? Otherwise, you could just say that M is a tunable hyperparameter.
- Also in Figure 2, it seems that there is a clear shift at some point where the samples go from low to high quality. Do you have any numerical indication (without looking at a test set, FID, etc.) of how a practicioner could notice that running for more steps would or wouldn't help?
- In Table 1, what is the last row? I guess it's just employing the corrector, but maybe a label for the row would be nice.
- Also in Table 1, it looks like the corrector helps for VE SDEs, but hurts for VP SDEs. Do you have an explanation for this? Maybe it's somewhere in the text, but if it is, I've missed it, so maybe point me to it.
- Given that you can compute exact likelihoods, is it possible to compute the exact NLL for any real dataset, like a test dataset?
review_title: Review

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Summary: This paper presents a generative model based on stochastic differential equations (SDEs), which generalizes two other score-based generative models score matching with Langevin dynamics (SMLD) and denoising diffusion probabilistic modeling (DDPM). The recipe to sample from the data distribution is based on (i) the observation that both SMLD and DDPM can be formulated as the discretization of an SDE, (ii) the finding from Anderson (1982) about the reverse of an Ito process, (iii) a score model. A novel aspect of the presented technique is the use of the score model as "predictor", which gives the initial sample from the MCMC sampler that serves as "corrector". Finally, the Ito process induced by the reverse SDE is formulated in a deterministic manner, leading to a neural-ODE based generative model.

Pros:
- The authors did a good job at showing connections between the previous score-based generative models and their model. I believe on its own this is a nice contribution.
- The method is thoroughly analyzed. I went through the derivations and didn't find any errors.
- Experiments show that combining the predictor and corrector routines leads to better performance, a nice validation of the theoretical claims.
- As promised, the model achieves SOTA on several tasks.

Cons:
- I'm having difficulty seeing the transformation of the reverse SDE into an ODE (from eq.10 to eq.12). Is it as simple as multiplying the second term with 1/2 and discarding the Brownian motion? Also, eq.12 is a simple ODE system, which has nothing to with a process as far as I understand. Maybe more explanation or pointers in Maoutsa et al., 2020 would be nice.
- The paper lacks the discussion on the benefits/downsides of different SDE solvers, discretization time steps, etc.
- As such, the paper lacks a "toy example" experiment, for example, on a simple 2D dataset like half-moons. A visual demonstration of the SDEs and probability flow (maybe corresponding vector fields and Brownian motion over time) would be interesting.

Additional comments:
- A typo (intead) right below eq.9
- Best performing rows can be bold in Table 1.
review_title: Fantastic Paper!

final_score: rating: 7: Good paper, accept
review_content: SUMMARY

The submission proposes a score-based generative model, which uses an SDE to map the
data distribution to a simple noise distribution and the corresponding reverse-time SDE to
generate samples by mapping the noise to the data space. The proposed model builds
upon and generalises two existing models (SMLD and DDPM) by transforming the data
using a continuous SDE dynamics as opposed to perturbing the data with a finite number
of noise distributions utilized by these models.

##################################################################

REASON FOR SCORE

The paper provides a clear motivation for the proposed modifications to SMLD and DDPM,
as well as a clear technical description of these modifications, their analysis and discussion.
I think the proposed model and sampling algorithms offer substantial conceptual
improvements to the existing models and should be of interest to the community. The
paper is well written and structured.

##################################################################

PROS

+ Clear motivation for the work.
+ Detailed technical description of the proposed models.
+ Interesting discussion of similarities and differences between SMLD, DDPM, and the SDE
based model, as well as corresponding sampling algorithms.
+ Extensive experiments.

##################################################################

CONS

- I found the discussion of equivalent neural ODE and its differences to reverse SDE
 a bit short, especially given that it is used in multiple experiments.
- The case of using general SDEs (not only those derived from SMLD and DDPM) is
mentioned only briefly, leaving it unclear if using a general SDE would require
relatively simple changes, or if the proposed model is limited to SDEs derived from
SMLD and DDPM.

##################################################################

QUESTIONS and COMMENTS

- Is it correct that the function \sigma(t) in Eq. (6) is assumed to be monotonic
and bounded by \sigma_max, while \beta(t) in Eq. (8) is bounded by 1, but doesn't have to
be monotonic?

- In the case of general SDE for noise perturbations in Eq. (9), are there any assumptions on
drift and diffusion function (such as monotonicity or boundedness)?

- In the case of SDEs (6) and (8), \nabla_x p_{0t} (x(t)) is available in closed-form. Is it always
necessary to have such a closed form expression in order to compute the objective (11), or
can it be estimated somehow without it? (I guess for a general SDE, there is typically no
closed-form \nabla_x p_{0t} (x(t)) available)

- Why do you think the FID values for the PC sampler with corrector are higher than without it
for VP SDE? (Table 1b)

- In section 4.2: "[...] PC samplers significantly outperform the corrector-only method, and can
improve over predictor-only approaches for most cases without extra computation." Why does
full PC sampler (with predictor and corrector) not incur extra computation in comparison to
prediction-only approaches. Don't we need to evaluate the approximate score function
s_\theta(x, i) for each of the M steps in the corrector sampler?

- In section 4.3: "[...] deterministic process whose trajectories induce the same evolution
of densities". Does it mean that ODE (12) and reverse SDE (10) map the same noise
distribution p(x(T)) to the same distribution in the data space? If so, are there any
advantages of using a reverse SDE instead of equivalent ODE if the latter is easier to solve
numerically and admits an exact computation of likelihoods?

review_title: Interesting generative model and good paper

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: #### Summary and contributions
This paper proposes a generalized framework for score-based generative modeling (SBGM). The proposed method subsumes previous SBGM techniques of score matching with Langevin dynamics (SMLD aka NCSN) and denoising diffusion probabilistic modeling (DDPM) and shows how they correspond to different discretizations of Stochastic Differential Equations (SDEs). The  continuous-time SDE generalizes the idea of a finite number of perturbation kernels used by previous methods to a continuum of them. The authors propose a forward SDE that transforms the data distribution into a known noise distribution and the corresponding reverse-time SDE that converts samples from this noise distribution to the data distribution. A predictor-corrector sampling framework is studied that leads to improved performance of both NCSN and DDPM frameworks. The paper also shows the equivalence of the proposed SDE to Neural ODEs which allows exact computation of the log-likelihood using the continuous change of variables formula. Quantitative experiments on the CIFAR10 dataset show that the proposed framework leads to significant improvements over previous SBGMs. Qualitative results on the CelebA-HQ dataset demonstrate the ability of the method to scale to high resolution images.

#### Strengths
This paper makes significant technical and empirical contributions to the emerging area of score-based generative models. The generalized SDE framework subsumes recent works in this area and is also connected to Neural ODEs, enjoying exact likelihood calculation, which may be relevant to the normalizing flows and generative modeling community is general. The empirical evaluation is particularly well-done. It bridges the gap between the performance of NCSN and DDPM models leading to state-of-the-art performance. The authors also demonstrate the ability of the method to generate high quality images of human faces when trained on CelebA-HQ dataset. Preliminary experiments on class conditional generation, imputation, and image colorization demonstrate the wide applicability of the proposed method.

#### Weaknesses
The paper does not suffer from any obvious weaknesses. The quantitive experiments could be strengthened by the addition of results on another dataset but the empirical evaluation is sufficient in its current state.

#### Additional feedback
Questions:
- In equation 11, how is the weighting function $\lambda$ chosen?
- In equation 11, apart from being able to sample from the transition kernel, it should also have a closed-form density for the evaluation of the score. Is my understanding correct?
- In equation 21, should there be a discretization step-size corresponding to $\Delta t$?
- In table 1 (a), why does SMLD with corrector only perform so poorly? As far as I understand it is equivalent to NCSN. Can the authors clarify if I misunderstood something?

-------

Post Rebuttal: I thank the authors for clarifying on my questions and updating the manuscript. 
review_title: Official Blind Review #3

Meta-review: All reviewers agree that this is a well-written and interesting paper that will be of interest to the ICLR and broader ML community.
Decision: Accept (Oral)