Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper is a theoretic investigation on how to learn a manifold with a score-based generative model (SGM). SGM uses an approximated score of the distribution in the middle of the reverse diffusion. This score estimation is only an approximation by a neural network, and this paper provides the study on error bound for this approximation. Also, this paper analyzes the impact of the limited sample population in learning the score function. Another story line is the limitation of diffusion in T because the diffusion step needs to be infinite to make the diffused distribution be a standard Normal distribution. This becomes infeasible in actual implementation, so there should be discrepancy between the final timestep distribution and the prior distribution. This discrepancy is analyzed, and its bound is suggested.
strengths_and_weaknesses: Strength:

1.
This paper provides a theoretic discussion on many mathematical assumptions suggested by SGMs.

2.
This paper clearly shows the necessary and sufficient assumptions to make the SGM identify the low-dimensional manifold.

Weakness:

1.
This paper is very difficult to understand because of its eccentric structure. I partly understand the authors' effort because this paper is dedicated to the theoretic analyses. However, this paper is really needs a restructuring to draw more attention from possible audience.

1) Assumptions are referred before their appearances. 
2) The motivation of Theorem 1 and 2 need to be further provided by formally throwing a research question.
3) Need a further clear statement on error bounds. Assumption 2 shows the characteristics of e(x,t), but I expected the explicit error bound for e. 

2.
The score approximation error will grow as we bring the diffused distribution to the standard Normal distribution, i.e. t->T. This is briefly mentioned in line 206-211, and I think that authors might produce further discussions and plain explanation on the interaction of score approximation quality and closeness between the prior and the diffused distributions. 

questions: See the weakness section
limitations: I think that this paper would need further toy example experiments and plain explanations to secure audience who may use diffusion models without enough knowledge on authors' discussions background.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper is a theoretic paper that solidifies the well-definedness of sample measure and proves the equivalence of the sample measure and the learnt measure under Assumption 2.
strengths_and_weaknesses: $\textbf{Strengths}$
- Until now, there is no guarantee that the reverse diffusion is well-defined at $t=0$ if $p_{data}$ is embedded in a low-dimensional manifold. This paper is the first to prove that we can safely solve the reverse diffusion up to $t=0$ when we use the previously suggested diffusion strategies (VE, VP, CLD).
- On top of that, this paper shows the well-definedness of the generative measure if Assumption 2 holds. Theorem 2 additionally proves that the sample measure and the trained measure have identical support.

$\textbf{Weaknesses}$
- It would have been much novel work if the author proved that Assumption 2 is the necessary and sufficient condition in Theorem 2. Theorem 2 only proves one direction, and it is difficult to conclude that uniform integrability is the key diagnosis of indicating the overfitting of the trained network.

- It would be much better to move the proof of theorems to the appendix and put more explanations and experiments. In particular, this paper significantly lacks the empirical validation of their claims. Although I believe the claims are novel, the one-point lesson is missing to the community of diffusion models. What would be the "concrete truth" observed from the high-dimensional experiment?

$\textbf{Notes}$
- This is not a weakness, but I should note that Theorem 1-(ii) is nothing but Theorem 1 of [**Song21Maximum**]. In Theorem 1 of [**Song21Maximum**], they used the data processing inequality for KL divergence, but the data processing inequality is well known for its $f$-divergence extension, and Theorem 1-(ii) is merely a straightforward generalization of the work of [**Song21Maximum**]. With all this, nevertheless, proving Theorem 1-(i) is a significant improvement, so I value Theorem 1.

- Again, this is not a weak point, but I think it would be preferable to describe the meaning of equivalent measure for the general audience. A measure $\mu$ is equivalent to $\nu$ if $\text{supp}(\mu)=\text{supp}(\nu)$, and this can be framed in an everyday language as "support matching force". The general audience will much more value this paper if Theorem 2 is well-understood.

- Not a weak point, but it seems that Assumption 2 is akin to the assumption of Theorem 1 of [**Bortoli21Diffusion**]. Theorem 1 of [**Bortoli21Diffusion**] explicitly bounds the total variation distance between the data distribution and the generative distribution under the uniformly bounded score estimation. Under such previous research, it would be very interesting to compare Theorem 1 of [**Bortoli21Diffusion**] and the authors' Theorem 2.

- To get a generalizable model, the authors claim that Assumption 2 should be violated. [**Kim22Soft**] introduced Unbounded NCSN and Unbounded DDPM as $\mathbf{s_{\theta}}(x_{t},\eta(t))$ with $\lim_{t\rightarrow 0}\eta(t)=\infty$ instead of NCSN and DDPM that parameterize the score estimation as $\mathbf{s_\theta}(x_t,t)$, to enable the score network to successfully estimate the unbounded score function. But it seems that NCSN and DDPM are more appropriate to violate Assumption 2, from their network design. It could be an interesting topic to investigate the network architectures with respect to the generalization power.

[**Song21Maximum**] Song, Yang, et al. "Maximum likelihood training of score-based diffusion models." Advances in Neural Information Processing Systems 34 (2021): 1415-1428.

[**Bortoli21Diffusion**] De Bortoli, Valentin, et al. "Diffusion Schrödinger bridge with applications to score-based generative modeling." Advances in Neural Information Processing Systems 34 (2021): 17695-17709.

[**Kim22Soft**] Kim, Dongjun, et al. "Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation." International Conference on Machine Learning (2022).
questions: - Is $Q$ in the proof of Theorem 1 well-defined? Does $\frac{\mathrm{d}Q}{\mathrm{d}P}(w)=\frac{\mathrm{d}\mu_{prior}}{\mathrm{d} p_{T}}$ enough to define $Q$? It seems that I am missing some parts, and it would be greatly appreciated if the authors let me know what I am missing.
limitations: - I am a practitioner in the community of diffusion models. In my perspective, a paper in this venue is better to contain intuitive illustrations and related explanations. In the current form, the submitted manuscript is flawless in its solidness, and I believe this is good work, but I doubt if this version could be valued in this field simply because it is difficult to understand the contents for the audience. Please remember that the expected audience of this paper is mostly the practitioners, and most of them are short-lesson-seekers. For instance, what is the meaning of uniformly integrable martingale? NeurIPS is not Annals of Mathematics, and sufficient interpretation of mathematical concepts is required. With the kind and satisfactory illustrations and insights, practitioners could find the true value of this paper.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper considers the approximation errors in the sampling process of score-based models. Specifically, the errors are in two folds: the estimation error of neural networks and the approximation of priors. The authors study the terminal distribution of the reverse SDE under such errors. They first show that the terminal measure is absolutely continuous w.r.t the data measure if we have an imperfect prior. Next, they show that the terminal distribution has the same support as the data distribution, under mild assumptions of the estimation error. Toy experiments demonstrate the validity of the theoretical results. 
strengths_and_weaknesses: ## Strengths

- The problem of approximation errors in score models is interesting and the theoretical understanding is lacking. The paper takes a step in this direction and investigates the terminal distribution of reverse sampling with the existence of various sources of errors.

- The paper verifies the correctness of the main results on toy experiments, and examines popular SDEs to see whether they fulfill the assumptions. 

**Questions**

Before going into the concrete questions, I would like to double check some terms and the overall proof ideas with the authors. Please correct me if I'm wrong. (1) In line 71 "its distribution is equivalent to the distribution of ..": Does the equivalence here only mean that the two distributions have the same support? (2) The general proof idea of theorem 2: To my understanding, the paper uses the Girsanov theorem with $Z_T$ as the ratio $\frac{\mu_{sample}}{\mu_{data}}=Z_T$. The uniformly integrable martingale assumption is to ensure the boundness of $Z_T$. Together the result manifests itself.

## Weaknesses

- The main result (theorem 2) in the paper is straightforward. It's a direct implication of Girsanov theorem. 

- The results seem tangent to the actual score-based generative models. For the image score-based generative models, the terminal $T$during sampling is set to $1e-3\sim1e-5$ due to discontinuity at $T=0$. Critically, the support of $p_{T}(x), T>0$ is $\mathbb{R}^n$, and the results trivially hold in this case. In addition, I ran some quick experiments on CIFAR-10 by adding 1 to the predicted score (the paper also does similar things), and the final results are noisy samples, with very large pixel values.



Due to the limited impact of the theoretical results and the gap to practical models, I think the paper has a lot of room for improvement.
 






questions: I copy and paste the questions above. 

Before going into the concrete questions, I would like to double check some terms and the overall proof ideas with the authors. Please correct me if I'm wrong. (1) In line 71 "its distribution is equivalent to the distribution of ..": Does the equivalence here only mean that the two distributions have the same support? (2) The general proof idea of theorem 2: To my understanding, the paper uses the Girsanov theorem with $Z_T$ as the ratio $\frac{\mu_{sample}}{\mu_{data}}=Z_T$. The uniformly integrable martingale assumption is to ensure the boundness of $Z_T$. Together the results manifests itself.


limitations: The authors have adequately addressed the potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This is a pure theoretical study of the score based generative model. The authors discover the conditions under which the sample distribution is equivalent to the data distribution.
strengths_and_weaknesses: 1. The theorems provided here could help researchers better understand those score based generative models, which is significant to the whole community.
2. The assumptions proposed here can guide future design of score based models.
3. As a pure theoretical work, the authors provide detailed proofs, which is helpful. But the lack of experimental verification could make it hard to digest. Although, I would not penalize the authors for this, since numerical verification on high dimension is a challenge on itself.
questions: Move Theorem 1&2 to later sections, especially after assumption 1&2 are introduced, may make it flow better.
limitations: No societal impace for a theoretical work.
review_title: null

Meta-review: metareview: This paper presents a theoretical analysis of score-based generative models (SGMs) [diffusion models]. Specifically, the paper theoretically studies the effect of approximations used by SGM [1. approximating p_T by µ_{prior} and 2. approximating ∇log(p_t) by a neural network], which currently lacks a solid understanding. The paper presents conditions that assures SGMs can sample from the underlying data manifold and also analyzes conditions under which an SGM memorizes the training data (the latter relates to understanding the generalization properties of SGMs).
Besides technical discussions and clarifications during the rebuttal period, the authors overhauled the introduction section and also added some experiments with CIFAR-10 dataset to support their theory, both of which were requested by the reviewers to enhance the paper. Reviewers were satisfied with the responses and the improvements in the revision. In concordance with them, I believe the paper provides a solid theoretical contribution to our understanding of SGMs and recommend accept.

confidence: Certain
Decision: Accept