Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The paper introduces a family of diffusion-based generative models for density estimations on image datasets. Here, the forward diffusion is directly defined by a set of conditional densities of diffused random variables for a given data. In particular, the paper proposes that $t$-th density for a given data follows a normal distribution; its mean and variance will be determined by data and a monotonic function of $t$, called $SNR(t)$. The generative models aim at learning diffusion processes reversed in time by maximizing its ELBO.

The paper first draws connections between the proposed models and previous diffusion-based models. For example, the authors show that the proposed method's discrete-time versions include previous finite-length Markov chain models as special cases. The paper then demonstrates that the ELBOs of the proposed continuous-time models exist. In here, the resulting ELBOs include an integration (wrt time) of a reconstruction loss weighted by the time-derivative of $SNR(t)$. Based on this, the paper shows that the losses of the previous continuous-time diffusion-based models follow the same integration form but with different weighting functions.

Second, the paper presents interesting properties of the proposed method due to $SNR(t)$. In particular, remind that $SNR(t)$ is invertible because of its monotonicity, and the continuous-time loss's integration contains the time-derivative of $SNR(t)$. The paper shows that the integral can be re-written wrt $SNR(t)$'s output by change-of-variable. In this form, the integral only depends on the range of the integration but doesn't depend on the shape of $SNR(t)$ within the range. As a result, acknowledging the invariant of the loss on the $SNR(t)$'s output space, the paper proposes to parameterize the score models conditioning on log-scaled $SNR(t)$ instead of $t$.

Third, the paper proposes to train $SNR(t)$ for the discrete and continuous-time models. For the monotonicity, $SNR(t)$ is defined as a monotonic network wrt $t$. For discrete-time models, the paper proposes to train $SNR(t)$ networks by maximizing the ELBO together with the diffusion models. For continuous-time models, $SNR(t)$ is trained by minimizing the variance of the Monte-Carlo estimation of the continuous-time ELBOs.

Furthermore, the paper introduces various practical techniques to improve training the models. For example, instead of iid sampling of $t$, the authors propose to use a low-discrepancy sampler of $t$ within a mini-batch. The paper also proposes to use the Fourier feature of inputs: feeding the concatenation of inputs and its Fourier feature to diffusion models.

In the experiments, the paper demonstrates that the proposed models achieve SOTA likelihoods on image datasets, such as CIFAR-10 and ImageNet.
main_review: (Originality & Significance)  
In my understanding, the paper's contributions are clear, and I also consider that the results are essential for several reasons.

First, it introduces a novel family of diffusion-based models whose diffusions are characterized by $SNR(t)$. In particular, with the proposed method, the paper demonstrates the SOTA results on density estimations which autoregressive models have dominated.

Second, the novel $SNR(t)$-based characterization of diffusions helps to understand similar models. For example, the proposed models include some popular diffusion-based models as special cases; thus, one can evaluate the continuous-time ELBOs for those cases. On the other hand, one can apply similar proof techniques for more general cases. Another example includes that for continuous-time cases, the invariance-to-the-noise-schedule property clearly motivates the efficient formulation of score models: models should condition on $SNR(t)$ (or its log-scale), especially when $SNR(t)$ can vary during training.

Third, the paper discusses the benefits of various practical techniques applicable to similar family models with ablation studies. For example, the low-discrepancy sampler of $t$ or Fourier features.

(Quality & Clarity)  
In general, the paper has a well-organized structure to motivate the proposed diffusion-based models and other practical techniques to improve training.


(Minor comments)  
In Lines 209-210 or Appendix B, I consider some previous works on monotonic neural networks should be cited; for example, J. Sill 1997, C-W. Huang et al. 2018, and A. Wehenkel & G. Louppe 2019.  

J. Sill, '97, Monotonic Networks  
C-W. Huang et al., '18, Neural Autoregressive Flows  
A. Wehenkel & G. Louppe, '19, Unconstrained Monotonic Neural Networks  
limitations_and_societal_impact: (Limitations)  
N/A

(Societal Impact)  
N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper tackles density estimation with denoising diffusion-based generative models and achieves very strong, state-of-the-art likelihoods (bounds) on standard image modeling benchmarks. The formulation largely relies on previous denoising diffusion probabilistic models (DDPMs), but uses the exact ELBO objective without dropping weighting coefficients in the mixture of losses, as was done in DDPM. The paper also proposes to learn the noise schedule, either using the ELBO objective or a variance reduction objective, and adds additional Fourier features into the model, which brings a significant benefit. On the theoretical side, the paper demonstrates how a change of variables in the time variable can lead to an equivalence between models trained with different noise schedules and shows how the important aspect of the noise schedule is only its behaviour at its endpoints. 
main_review: __Overall Impression:__ The *Fusion* model is mostly based on DDPM [1], while introducing additional tricks and architecture modifications (learning the noise schedule, adding Fourier features) that benefit the quantitative performance. The paper primarily stands out by its extraordinarily strong quantitative results, although the computational resources required for reproducing the results seem intense.

__Originality:__ As mentioned, the core model formulation (section 3) is largely based on DDPM [1], while reformulating certain terms and defining the noise schedule in terms of a signal-to-noise-ratio SNR(t), but it is essentially the same model, I think. The fact that more steps is better, leading to a tighter bound (section 4.2), is well-known and has been pointed out already by [2], which proposed this class of models. The originality and conceptual novelty up to this point seems limited. 

The demonstration that in the continuous time formulation the only thing that truly matters about the noise schedule are its endpoints is quite interesting, in particular considering that multiple previous and concurrent works explore different noise schedules. The idea to learn the noise schedule in itself is not novel, as it has been pointed out before [1], but this paper seems to be the first one to actually do it, proposing meaningful objectives for that. Overall, it feels the originality and theoretical/methodological novelty is not impressive, but there are some interesting contributions.

__Clarity:__ The paper is well written and easy to read.

__Specific Questions and Feedback:__
1. In Section 5, in the discretized setting, the noise schedule is learned using the ELBO objective. In that case, I think the model essentially behaves as a VAE. However, VAEs suffer from the prior-hole problem, explaining their poor sample quality. One major advantage of diffusion models is that they usually avoid this problem, diffusing each individual data point to full noise by construction. But by learning the noise schedule with the ELBO, don't we re-introduce this issue? What if the noise schedule learns to add virtually no noise to the data, so the diffusion objective (which is sort of a bunch of reconstruction objectives, given the noisy data) can be easily minimized, while still encoding data points under the prior, as in a VAE, to also satisfy the prior term? I admit that if we are purely targeting strong likelihood this argument doesn't matter. Nevertheless, it seems we may lose one of the main advantages of diffusion models here and sample quality may suffer. I feel this aspect is discussed insufficiently.
2. The authors also propose to instead learn the noise schedule with a variance reduction objective. This makes a lot of sense, but I think this is conceptually analogous to importance sampling techniques proposed by previous work: In principle, we can either optimize the noise schedule as proposed here considering uniform sampling from $t$, or alternatively we can use any arbitrary fixed noise schedule (fixing only the endpoints) and instead use importance sampling that oversamples the important $t$, similarly reducing variance. This seems basically equivalent, explained by the change-of-variables discussed in the paper. But this conceptual relation of learning the noise schedule to previously proposed importance sampling schemes is not discussed in the paper at all.
3. Regarding the theoretical equivalence between different models with different noise schedules: Does this only hold for noise schedules corresponding to a similar underlying SDE? This is, does this mean, for example, that the $\beta(t)$ in the VPSDE [3] is basically arbitrary? Or does this mean that also the underlying form of the SDE itself does not matter (apart from endpoint behavior and for training objective variance)? For example [3] proposed the SubVPSDE. Would it still be helpful to explore other SDEs, considering the continuous time framework, in light of the results?
4. The strongest results are obtained using the continuous setting for both training and evaluation (see Tab. 2). How is evaluation done in this case? Via black-box ODE solvers, leveraging the connection of continuous diffusion models to SDEs and ODEs and therefore continuous Normalizing flows [3]? I feel like I am missing some details regarding the model evaluation. I also didn't find this in the appendix.

__Conclusion:__ I think the paper meets the bar for publication, considering the importance of likelihood-based generative modeling and the paper's very strong quantitative results on that. It also has some minor theoretical contributions that help the understanding of diffusion models and the proposed architectural innovations may be adopted by the community.

[1] Ho et al., "Denoising Diffusion Probabilistic Models";
[2] Sohl-Dickstein et al., "Deep Unsupervised Learning using Nonequilibrium Thermodynamics";
[3] Song et al. "Score-Based Generative Modeling through Stochastic Differential Equations"
limitations_and_societal_impact: __(Negative) Societal Impact:__ Has been discussed very briefly, but in a satisfactory manner.

__Limitations:__ I think limitations could have been discussed more thoroughly. The proposed *Fusion* model is a typical denoising diffusion model, and as such has well-known and important limitations, I think. For example:
1. Very slow, iterative synthesis. The model samples shown in the appendix use 1000 synthesis steps, for example.
2. Other types of likelihood-based generative models, such as VAEs and autoregressive models, can be applied on arbitrary data types (language, graphs, discrete/binary data, etc.). Most current diffusion models, including the one in this paper, are limited to continuous data types where we can easily define meaningful diffusion processes. 
3. While the quantitative results of the paper are without doubt very strong, there seems to be a trade-off between achieving strong FID scores (or perceptual image quality more generally) and NLL. This can be considered a limitation.

I am aware that this paper does not aim to address these aspects, but I think it would not hurt to acknowledge these important limitations.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduce a new insight to diffusion generative models. Mainly, the paper re-write the forward and backward processes in new formulation which incorporate the SNR value into the diffusion process. 

Previous diffusion process like DDPM can be regard as instance of the proposed model. The snr value in the proposed method is learned jointly with the model whereas previous approach use fix snr value. 

The paper showed various theoretical results: (i) More steps is better, (ii) for continues time loss the model is invariance to the noise schedule, (iii) the training is impacted the start and end snr values.   
The paper introduce to main novelties: (i) learning noise schedule and (ii) insert Fourier features as additional features

main_review: The paper is well written as show big progress in understanding diffusion process. However i concern with the following issues:

(1) The paper introduce results only for low-mid resolution datasets (64x64<). Does the proposed model improved the results for larger resolution datasets, like LSUN 256x256?

(2) The results in the paper is given with bits per dim. Since most of the literature in diffusion process use FID score, it would be beneficial if the paper also show the improvement in terms of FID scores. 

(3) The paper should add ablation analysis to see the contribution of the learned noise schedule.

(4) In the ablation study, it seems that adding the Fourier features give the major part of the bits per dim improvement. Therefore, it will be beneficial to check what is the performance of other diffusion method when adding the Fourier features, e.g. Improved DDPM with Fourier features

(5) There are some typo in Eq22 (2nd line), v-->v' at the beginning of the line.

limitations_and_societal_impact: The authors addressed the limitations and the potential negative social impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposed an improvement to DPM. The main contribution is summarized below.

- Sec 3: The diffusion and reverse processes in DPM are re-written with the proposed SNR function. It can be seen as a high-level framework of the noise schedule. 
- Sec 4: The ELBO (for both discrete and continuous DPMs) is then re-written in terms of SNR. The authors had a few other findings: e.g., a larger T is better, and equivalence between continuous DPMs. 
- Sec 5: The SNR function is parameterized by a neural network and targets at minimizing the estimation variance of ELBO. 
- Sec 6: Additional features computed from Fourier transformation are added to data.
- Sec 7: Experiments on CIFAR and ImageNet show improvements on BPD. 


main_review: The paper is original to my knowledge. The research problem is significant, as DPMs are recently found to be very powerful in a number of generative tasks. As for quality and clarity, generally speaking, the theoretical results are correct, the experimental results are reasonable, but the writing is not very clear (detailed comments below).

Pros:
- The SNR function is the most interesting and important contribution of the paper. It provides a high-level view of both discrete and continuous DPMs. Learning SNR is meaningful as most current noise schedules are heuristic and it is unclear how to setup a best schedule.
- There are some interesting findings (e.g. "more steps is better" and "equivalence between continuous DPMs") and other useful tools (e.g. unbiased estimators and numerically stable computation).
- Experimentally, it is shown that the proposed Fusion, with the SNR learned and additional Fourier features, lead to SOTA BPD.

Cons / main concerns:

My first major concern is that neither the theoretical or the empirical contribution is sufficient. Do the authors consider this paper as a theoretical paper or an empirical paper? Currently, it looks only like a somewhat smart algorithm, with limited theoretical understanding and some experimental validation. 
- If the authors want to write a **theoretical** paper, then there should be a **complete** theory for the proposed method and relative findings. However, there is no theorem/proposition in the main text or the appendix. More importantly, many findings are not formally justified, so they are only insights. Below are some suggestions for writing a theoretical paper.
    - Line 104-105: small SNR(1) -> $q(z_1|x)\approx$ the standard Gaussian (need theoretical analysis).
    - Line 109-111: large SNR(0) -> $p(x|z_0)\approx q(x|z_0)$ (need theoretical analysis).
    - Line 152-155 and the paragraph after eq(48): $L_{2T} < L_T$. It makes sense that loss tends to be larger because $z_{t}$ is noisier than $z_{t'}$. However, a formal theoretical justification, even under some more assumptions, is necessary. Then, a tighter (non-zero) bound on $L_T - L_{2T}$ in terms of SNR is ideal.
    - Section 5: The SNR function is learned so that it minimizes the ELBO estimation variance. To make the theory sound, I suggest providing an upper bound on the variance term, which should be relevant to the SNR function or the general noise schedule. Then, provide a theoretical bound on the possible improvement of the variance estimation. 
    - Appendix E: a sensitivity analysis for the numerical stable computation is recommended.
- If otherwise, the authors want to write an **empirical** paper, then there should be **extensive** experiments. Below are some suggestions for making it an empirical paper.
    - Evaluation: The BPD improvement is marginal, given that those for prior DPMs are just upper bounds. The FID on CIFAR is worse than prior DPMs. Beating prior methods is not absolutely necessary; however, given these results, it is too early to call the proposed method "SOTA". There should be at least some aspect that your method clearly outperforms other models. In addition, in line 267-268, those DPMs are not *specifically* designed for FID, and if you can get much better FID then you should report the setting and numbers.
    - More datasets should be surveyed, because different datasets have different characteristics. In this paper, both CIFAR and ImageNet are very diverse datasets. How about datasets that are less diverse but have higher quality (resolution), such as CelebA-HQ and LSUN? It would be more interesting to look at other domains, such as audio or 3d point cloud generation. 
    - Figure 4 shows that the learned SNR leads to very small variance. However, it is necessary to directly compare BPD between learned SNR and prior (linear/cosine) schedules. What is the improvement of BPD if we just learn the SNR and keep all other settings the same? In addition, why does reducing the variance leads to improvement of BPD, if that is the case?
    - I notice that in Figure 3, having "variance minimization" very marginally improves BPD, whereas having Fourier features significantly improves BPD. How about applying these features to prior DPMs? It seems this is the main reason for the improvement.
    - In Table 2, when $T_{\mathrm{eval}}$ is 100 or 1000, the discrete DPM is better than the continuous DPM. Does that conflict with line 157-158?

Another major concern is about writing. The whole paper is very hard to follow. 

- Intro is too short and lacks some key discussion. Why are we interested in image density estimation benchmarks (rather than e.g. FID)? What are the main drawback of current methods? What are the challenges / difficulties, and how do you get around with it? How is the proposed method meaningful for the community? What can people learn from it?
- Always define a symbol **before** using it. (Minor concern for equations: remember to add commas or dots after an equation, such as in eq 8 and 9.)
- Eq(1) seems coming from nowhere. More background is needed. You could start with either diffusion in physics or DDPM by Ho et al 2020. Also, $\mathcal{N}(z_t;\alpha_tx,\sigma_t^2I)$. Why don't you model the covariance matrix in a similar as improved DDPM?
- Line 80-85: The correct order is defining the parameters first and then proving the diffusion process is Markov. Then, why do we care about $q(z_s|z_t,x)$? To make it clear, the ELBO expansion should be introduced here. And similarly, a proof is needed. To make this part clearer, I suggest start with DDPM and show how to adapt the theory here (by talking about connections and modification). 
- Line 86-90: It is okay to postpone theory to section 4, but more explanation on the SNR is preferred. Briefly talk about its meaning and how it affects the diffusion process.
- Eq (7): Why do you separate $x$ and $z_0$? In DDPM they are the same thing. What is the benefit here? (Minor concern for notations $s(i)$ and $t(i)$: sometimes the index is ignored, e.g. in eq17 and eq47. I suggest use subscriptions, and also remember to apply it to $t'$ as in eq47.)
- Line 106-111: this part is confusing. Where did you define $p(x|z_0)$? Why can you factorize it according to dimensions? Why does it approximate the $q(x|z_0)$?
- Line 112-116: What is the network and what are the re-parameterizations? Why does $\hat{x}_{\theta}$ predict $x$? And similarly, it is preferred to connect eq11 and eq12 to DDPM as well as eq4. 
- Eq 13: what is $W_t$?
- A general problem for section 3 is that too many things are postponed to later sections without any further explanation. This makes it very hard to read for general readers.
- Section 4.2: Do $L_T$ and $L_{2T}$ use the same SNR, or you are taking the inf over all SNR? And as mentioned before, line 151-154 is not rigorous.
- Line 221-222: What random variables are the $\mathbb{E}$ and VAR over? $t$, $\epsilon$, or both? (Minor concern: in eq 50, it should be $\frac{\partial}{\partial\eta}$.)
- Section 5: As mentioned before, why does reducing variance help? A deeper discussion is needed.


-----------------------------------------

In the rebuttal, the authors added insights and preliminary experimental results (or descriptions) to support their methods. These sound reasonable to me so I will raise my score to 6.

limitations_and_societal_impact: Limitations and social impact are discussed in the paper. More questions on limitations are in the main review.
review_title: null

Meta-review: This paper presents a novel perspective on denoising diffusion models. The main contributions of this submission include (i) a new formulation of ELBO using $SNR(t)$ and showing invariance to the noise schedule under this formulation, (ii) training noise schedule, and (iii) several new architecture improvements. The paper executes these ideas extremely well and obtains SOTA likelihood results on several images. Although the majority of the improvements in likelihood results seem to be obtained from architectural improvements (i.e., Fourier features), this paper provides very valuable insights into the diffusion models and can be of great interest to the community. Given the contributions, I am pleased to recommend this paper for acceptance. 
Decision: Accept (Poster)