Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents likelihood weighting to improve the log-likelihood performance of the score-based model. It is widely known that the score-based model is an excellent generative model to sample an instance from the high-dimensional data distribution. However, its encoding capability has remained question. This paper argues that the presented likelihood weighting can be universally applicable to the score-based models to improve such encoding capability without hurting the generation performance. There were series of theoretic derivations to induce the likelihood weighting, and a benchmark experiment was conducted. 

main_review: 1)
The main question is improving negative log-likelihood without damaging FID. Table 2 shows that "DEEP+LW+IS" achieves the best NLL with mediocre FID because the best NLL case results in 5.70 FID when the best FID is 2.86. When we observe Table 2, FID score almost doubles whenever we introduce likelihood weighting. There is no discussion about FID in Table 2 except "with only slight degradation of sample quality as measured by FID" in line 299.

Then, these questions arise.
- Why do you need additional importance sampling to limit the damage on FID? What is the theory behind of this observation?
- Still, the degradation of the FID is notieable. Can you verify this sample quality in other metrics, i.e. IS?
- Is there any trade-off coefficient that you can introduce to barter NLL with FID?


2)
Why do you report only NLL in Table 3 without FID? To empirically verify your claim, don't you need to observe the two aspects at the same time? Or, is it because your score model has definately better FID over the flow, transformer, and other models?


3)
Likelihood weighting seems to increase the loss variance as you mentioned in line 252. It seems that importance weighting is introduced to limit this side effect in an adhoc manner. To reason thie phenomena further, there should be inequality checking of loss variances between the case of using $\lambda$ as before and $\lambda=g(t)^2$. 

4)
VE-SDE uses the same weighting of likelihood weighting and the weighting in the prior work. Then, people may just use the VE-SDE variation of score-based models to optimize the NLL. 


limitations_and_societal_impact: 
There is no discussion at all. I understand that authors may not discuss the societal impact, but they can surely include the limitations that they expect from their models.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors show that for a specific weighting of score matching losses at various noise levels, the objective upper bounds the negative log-likelihood. They further bound the log-likelihood of individual datapoints to develop a continuous-time generalization of the ELBO.

Using the proposed objective the authors optimize score-based diffusion models towards likelihood.
The authors further make use of 1) importance sampling of time steps for reduced variance, 2) variational dequantization for a tighter bound on the likelihood and 3) the use of the equivalent ODE during evaluation, which allows exact likelihood computation (apart from the dequantization) for an approximation to the SDE model.
As a result of this, state-of-the-art likelihood performance is presented on CIFAR-10 and ImageNet 32x32, beating even the best autoregressive models.
main_review: Explicitly outlining the connections to maximum likelihood estimation (MLE) for continuous-time score-based diffusion models is novel, concurrent with [1], which was an important missing piece. It is nice to see a continuous-time extension of the ELBO. One thing I found missing was a connection to the discrete-time case as in DDPM [2] where a link between the ELBO and the score-matching loss is also established. I feel a discussion on this would clarify connections and thus strengthen the paper.

As for the results, the ablation study in Table 2 is informative, showing that 
1. likelihood weighting tends to improve likelihood, empirically backing up the choice, while degrading the FID score.
1. the performance of the equivalent ODE tends to perform better than the SDE despite this not being guaranteed.
1. the proposed importance sampling improves performance, along with the use of variational dequantization and depth.

Furthermore, the best numbers reported, as summarized in Table 3, are quite impressive, setting new records on two highly benchmarked datasets, which have been dominated by autoregressive models for years. This demonstrates clearly that score-based diffusion models have great potential, not just for image synthesis, but also for tasks relying on the likelihood (e.g. compression).

As for the clarity: I believe parts of Section 4 could be compressed and simplified to improve clarity. Please correct me if I'm wrong. 
Thm. 1 seems superfluous given Thm. 2 which turns the inequality into an equality (obtained through Eq. 25-27 with $q:=p_{\theta}^{\text{SDE}}$). Also, for Thm. 2, it seems $q_t$ is defined as the process solving Eq. (5) and thus $q := q_0 := p_{\theta}^{\text{SDE}}$ trivially, so it seems $q$ could be dropped?  
Finally, would not the requirement for $p_{\theta}^{\text{ODE}} = p_{\theta}^{\text{SDE}}$ be that $s_{\theta}(x,t) = \nabla_x \log p_t(x)$ instead of $\nabla_x \log q_t(x)$?

Minor fix on Line 127: $s_{\theta}(x) \rightarrow s_{\theta}(x, t)$.

[1] A Variational Perspective on Diffusion-Based Generative Models and Score Matching (Huang et al., 2021)  
[2] Denoising Diffusion Probabilistic Models (Ho et al., 2020)
limitations_and_societal_impact: Some limitations of the work are discussed in Sec. 4 on Line 202-210 and Line 244-247.
Potential societal impacts of the work are not discussed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper shows that, with a specific weighting of the score matching loss, one can upper bound the negative log-likelihood of the density induced by the reverse SDE parameterized by the score function. The authors then propose to use importance sampling to minimize this weighted score matching loss and obtain state-of-the-art likelihood estimates on CIFAR10 and ImageNet 32x32. 
main_review: It is a strong submission with some theory and impressive empirical results, which furthermore bridges the long-standing gap between autoregressive models and non-autoregressive likelihood-based models. 

In general, the paper is quite well written and quite easy to follow. Below I have some nits, which I hope the authors could address. 

1. Thm 1 & 3: The main proof technique can be summarized by an application of data processing inequality (which upper bound marginal KL divergence with the joint, functional KL divergence) and Girsanov's theorem (for computing the functional KL). My main critique for Theorem 1 is that it does not hold for empirical distribution, as the differential entropy will be unbounded. Then the authors derive Theorem 3 to bound the likelihood of individual datum. However, I do not quite follow the step where the expectation E_p(x) is removed. Do you mean by taking p(x) to approximate a point mass centered at x? If so please make the statement rigorous. Also, I'm not sure why in the LHS of the equation after line 649, there is the conditional expectation p0t and x', is this a typo?

2. Thm 2: in the remark after the theorem, it's said that the densities induced by the SDE and ODE will be close if s\approx score of the data. There seems to be some logical leap here, since this will only hold true provided that the score of the model (\nabla log q, not s) approximates the score of the data (\nabla log q). It is still unclear how the two densities differ from or relate to each other, quantitatively and non-asymptotically (asymptotically it's clear when the score function s is perfect, then everything is fine), and why the ODE's likelihood is improved. 

3. Bias in importance sampling: g^2/sigma^2 is not integrable as it blows up too fast as t->0, which means defining p(t) to be proportional to it will not result in a valid probability density. In Appendix B, it's said that the integration starts from some t_0, which should resolve the integrability issue, but this may result in bias in the ELBO estimation for both training and testing. Does the reported ELBO have this bias? I imagine this bias is negligible if t_0 is sufficiently small, has it been tested? Also, I'd suggest somehow include this in the main text so that people who want to build on the framework of this paper can be made aware of it. 

4. VE SDE: Line 284, it is said that the weighting for VE SDE is the same as in [1]. But if I understand correctly, in [1], lambda is taken to be 1/E[nabla log p_0t], which will correspond to the "conditional" variance, not the diffusion coefficient sigma^2(t), aka likelihood weighting.  
5. SOTA claim: It'd be better to include some other SOTA results with data augmentation in the table, such as [1,2]. It wouldn't harm the contribution of the paper as long as the SOTA claim is within the category of standard training with random horizontal flips.

[1] Score-Based Generative Modeling Through Stochastic Differential Equations.
[2] Distribution Augmentation for Generative Modeling
[3] Consistency Regularization for Variational Auto-Encoders
limitations_and_societal_impact: As pointed out in the main review above.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies score-based denoising-diffusion generative models and makes one major contribution. It shows how the training objective, a mixture or score-matching losses defined along different times of the diffusion, can be adapted to enable maximum likelihood training (using a lower bound objective). This is rigorously derived and proven analytically. The result is a specific weighting of the score-matching losses for different times t. The paper validates the derived methodology and empirically demonstrates very strong likelihood performance, state-of-the-art among models without data augmentations. 
main_review: __Overall impression:__ The main contribution of the paper, demonstrating how to train score-based diffusion models in a maximum likelihood fashion, using a lower bound objective, is a very important contribution. Score-based diffusion models recently generated much attention and demonstrated very high image quality synthesis, but their likelihoods were worse than those of some other models (autoregressive models, VAEs, etc.). This work solves this problem, demonstrates how to actually train the model directly towards good likelihood, and outperforms all previous models (when comparing to models without data augmentation). Consequently, I think this is a significant contribution and I expect future works on score-based diffusion models to often rely on the weightings derived in this paper. That being said, regarding the overall quality the paper lacks some discussions of important questions, and could discuss the context and limitations better. More specific feedback below.

__Clarity:__ I think the paper is well written and easily readable.

__Originality:__ The main contribution is original. The main contribution consists of several theorems that have not been appeared before, to the best of my knowledge. The proofs of the theorems seem solid and elegant. The paper also proposes importance sampling techniques and uses variational dequantization during training. However, these aspects are rather incremental.

__Weaknesses, Open Questions, Suggestions:__
1. The authors distinguish between $p^{ODE}$ and $p^{SDE}$, presenting it as if these are different generative models (e.g. in line 291 $p^{ODE}\neq p^{SDE}$). However, the authors also write that both $p^{ODE}$ and $p^{SDE}$ share the same marginal $p_t$ for all $t$ (line 117). Doesn't this imply that they define the same generative distribution and therefore model? Could we argue that $p^{ODE}$ and $p^{SDE}$ are merely two different ways of writing the *same* generative model, with one allowing us to sample with black-box ODE solvers, and the other, for example, with ancestral sampling (and differences in practice mainly being due to numerical issues and different solvers used)? With this interpretation it is also not surprising that the likelihoods/bounds of $p^{ODE}$ and $p^{SDE}$ improve together. Or is the issue that the statement "$p^{ODE}$ and $p^{SDE}$ share the same marginal $p_t$ for all $t$" is only true for valid score functions modeling proper distributions, while the learnt score function $\mathbf{s}_\theta$ may not satisfy these conditions? To me it rather feels the crucial difference is in the way they are evaluated - in one case via actual integration with error guarantees due to ODE solver tolerances, as in Normalizing flows, in the other case using a stochastic ELBO-like estimator. I got a bit confused and feel this point could be explained more clearly.
2. The models that achieve the strongest likelihoods tend to have worse FIDs than models trained without the likelihood weighting. There is a trade-off between NLL and FID. This is not discussed at all in the paper. How can we overcome this? However, looking at it from another perspective, the achieved FID scores, even for the models trained with the likelihood-weighting, are much better than typical FIDs obtained by previous autoregressive models, VAEs, flows, etc., which were also trained with likelihood objectives - and this is great! But why are FIDs so much better here? I think these aspects are quite interesting and important and deserve discussion.
3. One of the main drawbacks of current score-based diffusion models is their slow synthesis speed. This aspect is not discussed at all in the paper. What is the number of function evaluations, i.e. network calls, of the ODE solver when sampling? I think this needs to be reported and discussed as it is an important metric when working with diffusion models and can vary wildly. Ideally, even wall-clock time could be measured. It's also important when comparing to other types of likelihood-based models.
4. There is no thorough discussion of Related Works. I think contrasting the proposed *ScoreFlow* and its advantages and disadvantages to other types of likelihood-based generative modeling techniques and putting it into a broader context a bit more thoroughly would be nice.
5. The authors say they do not compare to the VESDE, since the VESDE likelihood weighting is the same as the weighting in [1]. It seems, however, that [1] did not evaluate likelihood for their models. Also, [1] obtained very strong FID score (although when using predictor-corrector sampling) even though they trained with a weighting corresponding to the likelihood-weighting derived in this work. This stands in contrast to the worse FID scores obtained here whenever using the likelihood weighting. I feel that exactly because the weighting used in [1] already is the maximum likelihood weighting, it would be very interesting to evaluate VESDE models towards likelihood or at least to discuss this.
6. Just a suggestion: If space is an issue for addressing some of my comments, I think the explanation of Variational Dequantization can be put in the Appendix entirely, as this is a standard technique. Also the proposed importance sampling is straight-forward and could be cut much shorter.

__Conclusion:__ In conclusion, I think this is a good paper that deserves to be published, as I think its main contribution is very important and it quantitative results are strong. However, I think the paper would significantly profit from addressing the above concerns and also considering the feedback regarding "Limitations and Societal Impact".

[1] Song et al. "Score-Based Generative Modeling through Stochastic Differential Equations"
limitations_and_societal_impact: Limitations and Societal Impact were not addressed in the paper in an optimal manner.

__Regarding limitations:__ I think obvious and well-known limitations of current score-based diffusion models, including the models and methods proposed in the paper, are: 
1. The slow synthesis speed, since a deep network needs to be called very often to obtain the final results (by the ODE solver). This stands in contrast to models such as GANs or VAEs, from which one can sample very efficiently.
2. The used diffusion processes are defined for continuous data (or dequantized discrete data, if applicable, like for typical images), but cannot be used easily for truly discrete or categorical datatypes (e.g. language data, graphs, binary data, etc.). 
3. As discussed, there seems to be some trade-off between strong FIDs and NLLs. 

I think such limitations could be acknowledged and discussed better.

__Regarding Societal Impact:__ The authors state that they did not discuss societal impact, since the method per se does not add any new negative impacts. I basically agree with this statement. However, this work essentially tackles generative modeling, which indeed can have negative societal impacts, such as synthesizing fake images, etc. This is well-known, but I think a pointer in this direction to acknowledge this would not hurt.
review_title: null

Meta-review: This paper shows how training continuous-time denoising diffusion models (a.k.a. score-based modes) can be converted to maximum-likelihood training with a proper weighting of the training objective. Although this connection is known to some degrees from classical literature (see below), the paper does an excellent job of showing how following this connection for training yields "SOTA" likelihood results with modern score-based generative models and proper importance sampling. Given the unanimously positive reviews and the potential impact of this work, I am very happy to recommend this paper for acceptance. 

That said, I'd like to encourage the authors to make the following changes for the camera-ready version:
- I appreciate coming forward with the updated numbers after discovering the numerical problems. Please make sure that the claims in this submission are adjusted after updating the numbers in the final camera-ready version. 
- The connection between the KL divergence and score matching objective was also discussed in classical papers such as [1]. Please add a small section discussing these earlier works.

[1] Lyu, Interpretation and Generalization of Score Matching. 
Decision: Accept (Spotlight)