Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel class of continuous-time diffusion-based generative models. Specifically, the paper proposes forward diffusions that are simple forms of stochastic Hamiltonian dynamics, unlike the previous methods, where linear stochastic differential equations (linear SDE) diffuse the data distribution. Thus, the proposed forward diffusion transport a joint distribution of data $x_0$ and auxiliary random variables $v_0$ to a prior joint distribution. Then, the generative models are their reverse-time diffusion processes on the joint space whose initial condition is the prior joint distribution.

In the Hamiltonian dynamics-type forward diffusion, the auxiliary random variable can be interpreted as velocity (as in physics), while data corresponds to position. The forward diffusion updates data by the velocities without any noise, while the velocities are updated via linear SDEs, similarly to deterministic Hamiltonian dynamics. In particular, the paper shows that the data distribution will be transported to a prior distribution and that velocities will map to a prior distribution similar to previous approaches. The prior joint distributions are commonly chosen to simple distributions, such as standard Normal distributions.

The authors emphasize two interesting properties of the proposed method. First, the paper points out that the reverse-time SDEs should only contain $\nabla_{v_t} \log p_t(v_t| x_t)$, but not logarithmic gradient wrt $x_t$, which implies that the model complexity (the size of models) is almost similar to the previous approaches. Second, the authors shows that $p_t(v_t| x_t)$ is Gaussians for all $t \in [0, T]$, including $p_0(v_0|x_0) = p_0(v_0)$. This indicates that $\nabla_{v_t} \log p_t(v_t| x_t)$ is potentially bounded unlike previous diffusion-based generative models, where the scores closer to data are possibly unbounded. As a result, esp. with common practice to learn scores at all time $t$'s by a single neural networks, training the proposed model will be more stable.

In order to learn the $\nabla_{v_t} \log p_t(v_t| x_t)$, the paper first propose to use denoising score matching (DSM) following previous diffusion-based generative models. However, observing that training the proposed model via DSM is unstable, the paper proposes a modified objective called hybrid score matching (HSM). HSM can be obtained by marginalizing out $v_0$ from score matching loss similarly to deriving DSM; the marginalization is possible as the auxiliary random variable defined to follow a known distribution, such as Normal distribution.

In addition, as the reverse-time diffusion is also a Hamiltonian SDE, the paper proposes a new integrator for generations, benefitting from the symplectic structure of the Hamiltonian dynamics. Note that it has been well appreciated that the discretization methods that utilize symplectic structure are more accurate than the Euler-Maruyama method under similar computational costs. Thus, the proposed integrator will have a better quality of samples, esp. when the number of discretization is small.

In order to show the effectiveness of the proposed method, the paper runs three main experiments: First, with toy experiments, the authors demonstrate that learning $\nabla_{v_t} \log p_t(v_t| x_t)$ is less difficult in comparison to learning the scores wrt the data directly. Then, the authors evaluate the generation qualities of the proposed models on two image modeling benchmark datasets; CIFAR-10 and CelebA-256. Finally, the paper demonstrates the proposed integrator's effectiveness compared to the Euler-Maruyama method, including additional analysis ablating the effect of hyperparameters of the proposed integrator.
main_review: The paper claims the following contributions.
1. First, the paper proposes a novel class of continuous-time diffusion-based generative models, called critically-damped Langevine diffusion (CLD).
2. Second, the authors show that a score matching objective for the proposed method requires only $\nabla_{v_t} \log p_t(v_t | x_t)$, not $\nabla_{(x_t, v_t)} p_t(x_t, v_t)$.
3. Third, for the proposed model, the paper proposes a variant of denoising score matching, called hybrid score matching (HSM), observing that the training of CLD-based models is unstable by using DSM.
4. Fourth, the authors propose a leapfrog-style integrator designed for CLD-based models, benefitting from Hamiltonian dynamics' symplectic structure.
5. Finally, the paper claims to provide novel insights into continuous-time diffusion-based generative models and their connections to statistic mechanics.


**Strengths of the paper**

In general, the paper's contributions wrt the novelty are clear for several reasons:
- First, the paper provides novel diffusion-based models inspired by stochastic Hamiltonian dynamics. Moreover, the paper provides sufficient proofs to support that the proposed methods are well defined. I find that the proposed method will be very interesting contribution to the generative model community.
- Second, the authors demonstrate the scalability of the proposed methods by (1) showing that the increment of the model complexity is marginal thanks to reverse-time SDEs only requires log-density gradient of conditional distributions of velocities given data, and (2) proposing a modified DSM loss suited for CLD-based models.
    
In addition, I found that the paper has a well-organized structure so that it is clear to understand the proposed and other practical techniques to improve training.

**Weaknesses of the paper**  
The following aspects of the paper can be improved.

- First, the discussion about the proposed integrator can be improved. The authors emphasize that the proposed leapfrog-style integrator for CLD-based models can generate better quality samples even when the number of discretization steps is small compared to the Euler-Maruyama method. However, it isn't easy to assess the statement based on the results described in Tables 2 and 3. In the experiments, the results are dependent on both the integrators and the qualities of learned models. To resolve this issue, one can add toy experiments to solidify the argument, where ground truth reverse-time SDEs are (approximately) accessible; thus, the resulting analysis can highlight the difference of the integrators.

- Second, the paper claims that the proposed method only needs to learn smoother signals, and thus the models can perform better. However, in the current submission, the discussion about this seems limited.  
    
  While the authors have already discussed the property by using toy experiments (such as in Fig. 2), the benefits of the proposed model seem marginal in high-dimensional datasets.  
    
  I assume that dropping weights to improve the generation qualities has diluted the benefits of the proposed method. Thus I believe that discussion about the proposed methods wrt maximum likelihood training can strengthen the significance of the proposed method. In my understanding, dropping weights for training diffusion-based generative models prevents modeling unbounded scores at $t$'s close to the data, where the weights are extremely high. As a result, the previous methods don't learn unbounded scores in practice, which can be a potential reason for the marginal benefits of the proposed method in high-dimensional settings. Therefore, the proposed method may benefit from not dropping weights, possibly resulting in high-quality samples being achieved with maximum likelihoods training. I anticipate that the comparison can be more observable if the models are trained in real space after logit transformation + uniform dequantization.  
  
  Moreover, it can be helpful to add discussions about the trade-off between the variance of the DSM/HSM objective estimates and the variance of the $p_0(v_0)$. For example, I imagine that the high variance of $p_0(v_0)$ results in the high variance in $p_t(v_t|x_t)$, implying that the models only need to learn smoother scores. Again, I assume this aspect can be more observable if the models are trained to maximize likelihoods.
    
- Third, the motivation of Hamiltonian dynamics-style generative models can be improved. Aside from the applicability of the leapfrog-style integrator for the proposed method, the authors claim that the velocity in CLDs accelerates mixing in the diffusion process (for example, in Sec 4). However, it is unclear how important is this "mixing" in the context of diffusion-based generative models.  
    In standard MCMC methods, the mixing properties are important since such methods aim at matching the distribution of recorded states (from a generated chain) to target distributions. On the other hand, the diffusion-based generative models rely on the distributions of collection of chains, esp. the chains' last states.  
Knowing that the previous diffusion-based generative models are already expressive and capable of generating diverse samples, it is difficult to motivate the proposed method while its novelty.   
    I believe that improving the discussion about the proposed integrators will resolve the issue, regardless of the proposed method's connection to statistic mechanics.
    
- Finally, the discussion about the motivation for HSM can be improved. In Sec 5.3, the authors state that for the proposed method, training with regular DSM is unstable. While the authors state some reasoning about the instability, but the analysis about it is limited. I believe that the single sample Monte Carlo estimate for the regular DSM objective can have a high variance for the proposed method, and thus I believe that the marginalizing out $v_0$ in HSM has reduced the variance. I consider that additional toy experiments that discuss the difference between DSM vs. HSM wrt their variance will further motivate the importance of HSM for CLD-based models.

summary_of_the_review: In general, the paper's contributions wrt the novelty are clear, and the proposed methods are well-defined. In addition, I found that the paper has a well-organized structure so that it is clear to understand the proposed and other practical techniques to improve training. Thus, I'm inclined to accept the paper. However, I found that several aspects of the submission can be improved. I'm inclined to improve my evaluation if the aforementioned weak points are well addressed.  

  
===== POST-REBUTTAL COMMENTS ======== 

The rebuttal had addressed most of my concerns. Consequently, I raised my score from 6 to 8, and I also raised the "Empirical Novelty And Significance" score from 2 to 3.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a critically-damped version of score-based diffusion models, by extending the inference process to an augmented state-space and diffusing the data coupled with an auxiliary velocity variable. The authors further proposed a hybrid score matching loss for training the reverse generative process, which provides an empirical advantage of learning a conditional score function that does not blow up (e.g. to infinity) and therefore stabilizes training. For sampling, the authors derived a new numerical integration method based on first principles of statistical mechanics and MD, which they found has a better convergence behavior in practice if the computing budget is limited. 

main_review: This is a paper well-done. Diffusion models require designing (or sometimes training) an inference process that brings the data distribution closer to some tractable prior. Working on an augmented state-space to introduce a momentum (velocity) variable to simulate damped Langevin dynamics is very a natural extension that complements well the current score-based diffusion-based models literature. The authors then went on exploring several benefits of such an extended framework, including a smoother (conditional) score function to approximate, a novel smoother (score-matching) loss function to optimize, and a specialized numerical integrator for efficient sampling. I believe the novelties in all different aspects introduced in this work could largely benefit the community.  

Overall, this is a great paper that is easy to follow and has multiple neat ideas. Strengths of the paper include 
* Novelty: although it is a straightforward extension, the authors explored different aspects of the central idea of augmentation/acceleration and proposed a new loss that is smooth and a numerical method that works better for certain settings.
* Clarity: the paper is well written, ideas presented with clear theoretical development. Most of the time the design choices are explained, leaving room for future research on some relevant topics (such as optimizing for NLL, and adaptive solvers).  
* Impact: the problem the proposed method solves is very specialized, particularly for SGDMs. The idea of state-space augmentation could potentially mitigate the exploding score of DSM-type training and can be a principled way of designing an SGDM that naturally admits faster numerical simulation.
  
I do not have any serious criticisms, but I do have the following (more detailed) questions / comments:
1. There is a paragraph in Section 3 describing the trade-off between being overdamped and underdamped. The explanation is quite intuitive, but is there any more quantitative evidence for opting for critical damping (either in theory or empirically)? I suppose an optimal ratio between $\Gamma$ and $M$ will be problem-specific. Would such generality be useful in practice?
2. A missing point about using damped Langevin dynamics is the potential speedup in convergence to the prior distribution (A la Ma et al. 2019, Proposition 1). Diffusing along with the momentum variable could allow for significant acceleration, which might reduce the integration time T needed for sampling as well. 
3. For HSM, is marginalizing out $v_0$ the key to smoothing out the loss function (compared to DSM)? If so, is this a form of Blackwellization, and does the gradient estimate always have a smaller variance compared to DSM?
4. In 3.3, Leimkuhler & Reich, 2005 was cited for Euler’s methods not being suitable for Hamiltonian dynamics. Is there any intuition behind this? Can the standard midpoint integration scheme be employed which is well suited for Hamiltonian systems?
5. For SSCS, can the second half step of the current iteration be merged with the first half step of the next iteration, similarly to leap-frog integration. 
6. Experiment-wise, are the numbers presented in the tables with one random seed (for experiment and for evaluation)? Just reading the number, I wouldn’t say the difference between EM and SSCS for large n in Table 3 is negligible, but perhaps that’s more due to the stochasticity of the nature of FID? 
7. About $p(v_0)$, the reason for worse NLL for small $\gamma$ could be due to the fact that the upper bound requires subtracting the entropy of the augmented distribution (which will be smaller). It’d be worth noting that this could be potentially optimized for NLL. Aside from that, smaller $\gamma$ would also mean the magnitude for the score at $t\rightarrow0$ is also larger? 

Ma et al 2019: Is There an Analog of Nesterov Acceleration for MCMC?

summary_of_the_review: This is a timely paper that explores acceleration methods for diffusion-based models. The authors also explored different aspects / benefits of the proposed framework which I think will benefit the community of researchers working on SGDMs, including the smoothness of loss function and a new numerical integration method. 

My recommendation is on the lower band of 8: accept, good paper.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The authors propose critically-damped Langevin dynamics (CLD) for score-based generative modeling. This consists of a higher-order Langevin dynamics scheme with particle velocity and position coupled to each other, as in Hamiltonian dynamics. The Langevin dynamics is critical in the sense that it is neither over- nor under-damped. A corresponding score matching objective is derived as an objective, with proof given that it is simply necessary to approximate the score of the velocity given the position. Empirical evidence is provided that this score is easier to estimate on a synthetic example. As DSM is analytically intractable for the higher-order scheme, Hybrid Score Matching (HSM) is proposed and the integration integral to this objective is addressed with a new numerical integration scheme. This approximation scheme, called Symmetric Splitting CLD Sample (SSCS), decomposes the SDE to be integrated into a tractable expression and a (hopefully small) Euler-Maruyama integration for improved accuracy (although still first order) overall. Synthetic examples are used in both the main text and the supplementary material to motivate the theory. Benchmark image datasets exhibit exceptionally strong performance, with improved sample efficiency after training and robust hyperparameters.
main_review: **Strengths**

- Section 1 and 4 provide a very thorough overview of recent work, with a fantastic visualisation of the method.
- I found the arguments and derivations throughout Section 3 compelling, with the appropriate mathematics reserved for the appendix. The synthetic experiment demonstrating the ease of numerical approximation is also clearly explained and useful.
- Particular praise should be afforded to the consistent use of appropriate references in Section 3, placing the derivations in context, as well as the connections drawn to the high-friction limit in Section A.2.
- Model evaluation is in-line with literature and on-par with the very best performing models. Up to extreme levels of compute, the method is essentially SoTA. Given the higher-order nature of the sampling, evaluation of NFEs is also well-motivated and is a clear improvement for constant compute. Finally, the robust hyperparameters are of practical relevance.

**Weaknesses**

There are very few weaknesses beyond typos and personal preferences for presentation.

My only comment would be the absence of references to the line of work on higher-order MCMC by Michael I Jordan and others which I believe to be pertinent:

[1] Ma, Yi-An, et al. "Is there an analog of Nesterov acceleration for MCMC?." arXiv preprint arXiv:1902.00996 (2019).

[2] Mou, Wenlong, et al. "High-order Langevin diffusion yields an accelerated MCMC algorithm." arXiv preprint arXiv:1908.10859 (2019).

Minor comments:
- The reference for Song et al.'s SDE paper is dated as ICLR 2020, rather than 2021.
- Penultimate para of sec 1: "diffusionand" -> "diffusion and"
- Section B.2: "SGMs" does not need to be repeated in sentence 1.
- One of the $\Sigma_{t}^{xv}$ matrices in Equation (9) should $\Sigma_{t}^{vt}$? This notation is used at the end of Section B.2, so consistency would make things a little smoother.
- Sec 5.1: "We are significantly outperforming this model" could be rephrased.
- Sec 5.1 "outperforms" -> "outperform"
- Sec 5.2, para 2: "as backbone" -> "as a/the backbone"
- Tab. 2 caption: "denotes" -> "denoted"
- A.1: "very inefficient too" -> "very inefficiently too"
- Sec B2, penultimate para. and Equation (67): What is $\Sigma_{t}^{zz}$? I assume this is a typo?
summary_of_the_review: This paper is exceptionally well put together, in terms of derivations, presentation, and experimentation. The narrative is well-written, well-motivated, and logical throughout. To the best of my abilities, the proofs are entirely correct. The only way I can see to improve the experiments is to offer the authors more compute or data.

As a result, I believe this paper to be of the highest quality and recommend that it is given particular attention at the conference.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors introduce a novel approach for training a score-based generative model.  With prior score based networks, generation is performed by solving a stochastic differential equation (SDE) based on Langevin Dynamics, using an estimate of gradient of the log likelihood of the underlying signal distribution.  In the present paper, the authors present a novel forward process, where diffusion is run in a joint data-velocity space.  The noise term is only applied to the velocity component.  This reduces the learning problem to only needing to learn the score of the conditional distribution of velocity given data, which is easier than learning the score of the data distribution directly.  The paper shows that the novel scheme (CLD) yields higher quality for image generation when compared to prior models of similar capacity and number of neural network evaluations.  

main_review: Strengths of the paper:
- The paper uses tools from physics to design a novel score based generative model, allowing it to leverage existing insights in that field
- The paper results in training of generative networks that outperform existing approaches when making sure to balance compute budgets and model size
- The paper derive a SDE integrator that allows for efficient sampling from the proposed class of models.


My only question to the authors is: Do they envision that a class of methods of this sort are also possible?  For example, would the method work if they had an acceleration term in which the noise was injected?

summary_of_the_review: I recommend the paper be accepted.  It presents a potentially significant improvement to the training of score-based generative models which is well grounded in theory from physics and demonstrates strong empirical performance for sample generation when compared to baselines of similar complexity and compute budgets.
review_title: null

Meta-review: The paper develops a diffusion-process based generative model that perturbs the data using a critically damped Langevin diffusion. The diffusion is set up through an auxiliary velocity term like in Hamiltonian dynamics. The idea is that picking a process that diffuses faster will lead to better results.The paper then constructs a new score matching objective adapted to this diffusion, along with a sampling scheme for critically damped Langevin score based generative models. The idea of a faster diffusion to make generative models is a good one. The paper is a solid accept.  

Reviewer tK3A was lukewarm as evidenced by their original 2 for empirical novelty that moved to a 3. From my look, it felt like a straightforward application of ideas in one domain, sampling, to another, generative modeling. It's a good paper, but it does not stand out relative to other accepts.
Decision: Accept (Spotlight)