Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This is a theoretical paper on score-based generative models. The main result states that the Wasserstein distance between the data distribution and the generated distribution of such models is upper bounded by the square root of their objective function up to multiplicative constants and a fixed constant. This suggests that such models tend to minimize the Wasserstein distance between the target distribution and the estimated one, and complements another recent similar result on the Kullback-Leibler divergence [31]. Another theorem connecting this result to diffusion models is also provided. The findings are supported by numerical experiments on toy data (Gaussian mixtures).
strengths_and_weaknesses: ## Strength
- The paper provides a strong contribution towards a better theoretical understanding of the dramatic empirical success demonstrated by score-based and diffusion-based generative models in recent years.

## Weakness
- The paper is hard to follow for a reader who is not deeply familiar with PDE formulations of scored-based models, diffusion models, and Wasserstein metrics.

Unfortunately I do not have this background nor the time to acquire it, and hence was not able to check the details of the proofs in the paper. My assessment of the paper is based on the assumption that the proofs are all correct.
questions: - The analysis is conducted in the continuous time domain, whereas in practice, diffusion model are always trained using a discrete, finite number of time step. A comment on how this theoretical analysis translates to the discrete setting would be welcome. 
- In a similar vein, it is not clear how the various continuous-time integrals were numerically calculated in practice by the authors in the various experiments.
- The loss function defined in eq. (10) is not very well explained or motivated.
- Please check the bibliography, it contains many mistakes. For instance, bibliographical reference [31] appears to be incorrect (it seems to include a long list of "editors"). Moreover, many bibliographical references are duplicate, eg., [31] and [32],  [33] and [34], [36] and [37]. 

Typos:
- L29: the upper bound => an upper bound
- L59: neither variables J_SM or J_DSM have been properly defined at this point of the paper
- L94: from in the above => from the above
- L269: makes us hard => makes it hard

limitations: The limitations of the proposed theoretical analysis are adequately addressed in Section 6.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - This paper discusses properties of the PDF realized by the data generating process of score-based models (denoising diffusion models), which have received a great deal of attention recently.
The three main theorems are as follows
  1. If the true density is $p_0$ and the density of generated data is $q_0$, then its Wasserstein2 distance $W_2(p_0, q_0)$ is bounded above by 
$$W_2(p_T, q_T) \times \text{(a constant dependent on Lipschitz constants)} + \sqrt{\text{(training loss)}} \times \text{const}.$$
Therefore, under ideal conditions such that $p_T=q_T$, the loss function is an upper bound of the Wasserstein2 distance between the true and generated densities.
  2. The loss function commonly used in the training of score-based model is just an approximation of the true score-matching loss. In this paper, a theorem that gives the relation betweem them is provided. This gives a theoretical backborn  that the approximate loss function used in practice can be used as a surrogate function for the true objective function which is difficult to evaluate exactly.
  3. Perturbing around the initial value (eliminating singularities at the time origin) is a fundamental technique in practical situations. A theorem is provided to evaluate the upper bound of the Wasserstein distance between the true density and the perturbed density. This is a direct consequence of the triangular inequality of the Wasserstein distance and the above theorem.

- It is then demonstrated numerically using a toy model and toy data that the upper bound given by these theorems actually valid. It is also argued that the main factors of the gap between the upper bound and the loss are related to the the Lipschitz constant of the neural network.

- The proof begins with a review of the fundamentals that the Fokker-Planck equation is analogous to the equation of continuity (or the law of mass conservation) which is the fundamental relationship between density and velocity field. Based on that argument, a equation is presented that states that the time derivative of the Wasserstein2 distance is expressed by the velocity field, the proof of which is based on outside literature.

strengths_and_weaknesses: - These theorems are interesting because they guarantee fundamental properties of the score-based generative models. (However, I did not follow all the technical details of the proofs, and am not still confident in my understanding.)

- It is an interesting claim that there is a theoretical guarantee that the same algorithm minimizes KL and Wasserstein distances simultaneously, as opposed to GAN, which leads to different algorithms depending on its distance measure.

- These evaluations are strictly valid only for special cases, and for more general cases, theoretical evaluation is an open question. This is discussed in Section 6.

questions: - The discussion in this paper is based on a continuous stochastic process, but the numerical simulations appear to be dealing with discretized processes with the step size of $t = 1$. Is my understanding correct that that this is justified by Cor 2?

- (The issue that interested me most while reading the text was the behavior in higher dimensions rather than the 2-dim toy data. However, this was discussed in the appendix, albeit to a limited extent.)

limitations: - It is argued in section 6 that more general conditions make theoretical evaluation more difficult.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This article uses optimal transport theory to prove that score-based generative models not only minimize the Kullback-Leibler divergence but also the Wasserstein distance between the generated samples distribution and the data distribution. In addition, it establishes that the Wasserstein distance can be upper-bounded by the square root of the objective function (up to multiplicative and additive constants) for both score-based models and denoising diffusion models. 
This paper then discusses the impact of weight decay and the number of timesteps $T$ on the upper bounds. Theoretical results are experimentally checked and illustrated on simple synthetic datasets. Beyond that, the article also discusses the robustness of score-based models to noise perturbation on the datasets using the Wasserstein distance.  
strengths_and_weaknesses: - The results proved in this article bring new interpretations on the score-based models learning properties, which were, to the best of our knowledge, not known before. The results are clearly stated and discussed thanks to numerous remarks sections. 
- The article is well written and easy to follow, despite some paragraphs that could be made clearer. Previous works, both in the field of generative models and optimal transport, are referenced and well-integrated in the narration.
Notations are clearly defined and respected, even if prior knowledge about the related works is sometimes needed. Also, note that objective functions $J_{SM}$ $J_{DSM}$ are referenced (line 59) before definitions, or that the variables with respect to which variance and expectation are taken in Eq. 15 should be precised.
- The experiments and figures help with the comprehension of the article and provide illustrations of the theoretical statements.

- The motivation of the article could be stated more clearly. Besides the theoretical result, the article could mention the benefits of minimizing the Wasserstein distance between the model and the data distributions.
- My biggest concern is about the Experiments part: This is mostly a theoretical paper and the presented experiments look somewhat superficial or confusing. The fact that the upper bound of Eq. 19 depends on the training procedure is rather unusual. This could have been controlled before training by devising a dedicated architecture. There are numerous resources on this subject, so that controlling the network's Lipschitz constant only via weight regularization does not seem convincing in this case. Also, it would have been interesting to discuss if the upper bound of Eq. 19 could suggest new ways to select the forward SDE. In the paper, only the dependence on the network's Lipschitz constant $L_s$ is discussed.
The assumption $W_2(p_T, q_T)=0$ depends on the choice of the forward SDE and is only approximately true. Completely removing it from Eq. 19 does not show the balance between low Wasserstein distance at $T$ and small $I(t)$.
- It is not very clear how the theoretical results of this article can motivate improvement in the case of score-based model training on more complex datasets in practice.
- Some details could be added regarding the training and evaluation of the network: T=10 timesteps seems like a too small value compared to what is regularly used in the literature. The number of samples used to compute $W_2$ is also missing.
- Some adjustments regarding the overall organisation of the paper could be made: "The score matching and diffusion probabilistic models" subsection in §2.4 could be included in the Introduction. This could help understanding that there is not much distinction between score-based generative models and DDPMs, as suggested in the Introduction, and would help understanding Sect. 2. The sketch of the proof could follow the theorem.

- l. 219 $L_s(t)$ instead of $L_s(r)$

questions: - The article discusses casts light on reducing the intercept $I$ by applying weight decay (since it allows to reduce the Lipschitz constant $L_s$). Considering that $I$ also depends on the SDE used through $L_f$ and $g^2$, did the authors study the effect of the choice of the SDE on the intercept value given a fixed dataset? 
- On figure 6 (Appendix E), we can notice that the denoising score matching loss does not decrease much during training. Can the authors explain why the observed gap is equal to the theoretical one and not the consequence of the neural network of the denoising score matching model underfitting the distribution? (since it is a very small network).

limitations: There is no direct potential negative societal impact of the article proposed in this article.

[Edit] - I raised my score
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper shows that score-based models minimize the Wasserstein distance between them and that the Wasserstein distance is upper bounded by the square root of the objective function up to multiplicative constants and a fixed constant offset.
strengths_and_weaknesses: Strengths: This paper shows that score-based models minimize the Wasserstein distance between them and that the Wasserstein distance is upper bounded by the square root of the objective function up to multiplicative constants and a fixed constant offset.

Weaknesses:
1. The assumptions in the main theorems seem to be strong. 
2. The abstract indicates the claim such that score-based models minimize the Wasserstein distance between them. However, the main theorems cannot follow the claim. 
questions: 1. The main question is whether or not the theorems in this paper lead to the finding that score-based models minimize the Wasserstein distance between them, i.e., $W_2(p_0,q_0) = \min W(p,q) = 0$. The theorems in this paper are based on the upper bounds of $W_2$. Remark 1 indicates that Corollary 1 implies that $W_2$ converges to $0$ under some assumptions. Hence, it is not guaranteed that score-based models minimize the Wasserstein distance between them.
2. The assumptions in the main theorems seem to be strong. For example,
- Remark 1: the conditions $J_{SM} \to 0$, $p_T = q_T$, and $\lambda(t) = g(t)^2$ seem to be strong. Provide some practical examples satisfying all the conditions. 
- Theorem 2: Provide some practical examples of $p_{0t}$ satisfying (15). 
3. As a result, I cannot understand what the contribution of this paper is and why the results in this paper are significant. In particular, I cannot understand why the upper bounds of $W_2$ are needed for deep machine learning and its related areas. 
limitations: 1. There is a gap between the claim in the abstract and the main theorems in this paper. 
2. The assumptions in this paper seem to be strong.
3. I do not know that numerical results in section 4 are appropriate since the condition $W_2(p_T, q_T) = 0$ is assumed. Also, I do not know why the setting in Page 6 (e.g., a 3-layer neural network, AdamW with lr = 0.001) are appropriate. The numerical results are based on factitious setting, and hence, limited results.
review_title: null

Meta-review: metareview: This paper considers a reverse process (4) of the diffusion process (1), and provides an upper bound (Theorem 1) of the Wasserstein distance $W_2(p_0,q_0)$ of a data distribution $p_0$ and the distribution $q_0$ defined by the reverse process at time $t=0$, in terms of the approximation error $J_I$ of the score function $\nabla\log p_t$ by $s_\theta(t)$ and the Wasserstein distance $W_2(p_T,q_T)$. It implies that, provided that $W_2(p_T,q_T)$ is zero, approximating the score function by $s_\theta$ via training will make $W_2(p_0,q_0)$ small, allowing generative modeling of $p_0$ by $q_0$. A more practical upper bound (Theorem 2), as well as an analysis on effects of perturbations of the data distribution $p_0$ (Theorem 3), is also provided.

The review ratings/confidences were 8/2, 7/1, 6/4, and 5/3. The average is above the acceptance threshold. Upon reading the reviews, the author responses, as well as the paper itself, I found that the comments by the reviewers with lower ratings were mostly reasonable given the initial version of this paper which was difficult to follow in some places, as well as the fact that one would need simplifications like the continuous-time formulation and assumptions to guarantee integrability of relevant quantities, for theoretical development like the one conducted in this paper, but I found that most of them, especially the issue of the competition between $I(T)$ and $W_2(p_T,q_T)$ in the case of the Denoising Diffusion Probabilistic Models as summarized in Remark 4 in the revised version, were properly addressed in the revised version. I would therefore like to recommend acceptance, which would encourage further discussion among the attendees of the conference on more technical details.

A few minor points: 
- Notational inconsistency: The authors used the notation $g^2(.)$ to mean $(g(.))^2$ and $I(.)^2$ to mean $(I(.))^2$, which seem inconsistent to me.
- Equation (11): The integrand $L_f(r)+L_s(r)g^2(r)$ should be put in parentheses.
- The term $\frac{1}{2}\log2$, which comes from the prefactor 2 in the square root in Equation (16), should be added to the right-hand side of Equation (19).
confidence: Certain
Decision: Accept