Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper analyzes the denoising score matching objective used to train diffusion models, which have recently become the dominating class of deep generative models. The paper argues that at intermediate perturbation levels the denoising score matching objective is particularly noisy and "unstable", because one perturbed data sample could correspond to many different original clean data samples (the paper calls this phase 2; at the same time, we aren't yet in the regime where every data point is almost entirely destroyed and the ground truth score is effectively almost the same standard normal score for all data points -- phase 3 in the language of the paper). Standard denoising score matching uses only a single data sample among the plausible ones to estimate the score in each iteration. This effectively leads to high variance training (although unbiased). The paper therefore proposes to include a reference batch of additional samples to calculate a more stable score estimate. This corresponds to a form of importance sampling, introducing an unknown normalization factor. This problem is circumvented with self-normalized importance sampling, which, however, introduces bias. Ultimately, the paper proposes "Stable Target Field" (STF), a biased but lower variance score estimator for training score-based generative models.

Experimentally, the paper shows superior performance of "Variance-Exploding" SDE models when trained with STF compared to standard denoising score matching training. Also, training tends to converge quicker. These advantages primarily manifest when using ODE samplers to generate data.
strength_and_weaknesses: **Strengths:**

- The idea to use an additional reference batch to create a more stable and lower variance target score estimate for denoising score matching is novel. It is an interesting, well-motivated and smart idea.

- I like how the paper derives the STF approach and constructs its main method. The analysis of the original denoising score matching objective, followed by the introduction of the reference batch leveraging an importance sampling framework, and then resorting to self-normalized importance sampling to avoid the intractable normalization factor, is elegant.

- The work includes explicit technical analyses on the bias and variance of the proposed STF objective.

- The variance reduction analyses (Figure 3) in the different phases are quite interesting and insightful.

**Weaknesses:**

- It is nice that the paper provides some analytical analyses on the bias and variance of the STF approach. However, how tight is the bound on the STF variance in Theorem 2? I agree that when $p_{0|t}(x|x(t))\approx p_0(x)$, then the bound is meaningful. But this only really happens in "phase 3" (essentially, phase 3 is defined by $p_{0|t}(x|x(t))\approx p_0(x)$). But the variance reduction also works well in the more relevant "phase 2", as empirically shown. But in this relevant regime, where $D_f$ is not yet 0, the bound is very hard to interpret and provides little insight.

- The paper states that Variance Exploding SDEs (VESDE) are the state-of-the-art SDEs used for training score-based generative diffusion models. It is correct that Song et al. [1] achieved strong results with the VESDE when combined with predictor-corrector SDE sampling. However, almost the entire literature on diffusion models uses Variance Preserving SDEs (VPSDE). Moreover, it has recently been shown how to subsume all different common diffusion model SDEs under one umbrella in Karras et al. [2]. This paper then shows how a VESDE-based model can be improved significantly simply by using appropriate model and network parametrizations and preconditionings (their "EDM" scheme). Consequently, I would consider this EDM the state-of-the-art approach. Moreover, the main question is, does the STF approach also help when applied to more modern and widely used diffusion models, this is, (a) VPSDE and (b) Karras et al.'s EDM approach? The paper focuses on the vanilla VESDE with ODE-based sampling, even though it is well-known that the vanilla VESDE does not play well with ODE solvers and isn't widely used like that in the literature. In fact, already when sampling stochastically with predictor-corrector sampling, the benefit by STF is very marginal and almost not significant (Table 1, PC sampler (SDE)). Also note that the Karras et al. paper is not cited or discussed at all.

- Note that Karras et al. [2] also discuss different perturbation regimes (see paragraph "Loss weighting and sampling" in their Section 5), similar to the different phases discussed in this work. The fact that there are effectively different phases during SGM training is known. This should be acknowledged.

- In Section 3, the paper points to the multimodality of the distribution $p_{0|t}(x|x(t))$. Note that this is extensively discussed also in Xiao et al. [3]. It would make sense to acknowledge that work.

- Regarding "The STF objective stabilizes the model performance at convergence.": I think this is overclaiming. The model only shows a significantly lower variance on the evaluation metric for the setup VESDE + RK45 (already when using the SDE framework with a PC sampler, the variance isn't much lower; Table 1). To verify this claim more broadly, again, experiments on other diffusion models should be run (VPSDE, EDM).

- The paper mentions the computational overhead induced by the reference batch. How small or big is this overhead actually? My understanding is that it is small, because the samples from the reference batch are only used to estimate the target score, but we do not have to call the expensive neural network on them. However, what about memory? Some more details would be helpful here. Note that this comment did not influence my rating significantly.

[1] Song et al., Score-Based Generative Modeling through Stochastic Differential Equations, 2021.

[2] Karras et al., Elucidating the Design Space of Diffusion-Based Generative Models, 2022.

[3] Xiao et al., Tackling the Generative Learning Trilemma with Denoising Diffusion GANs, 2022.
clarity,_quality,_novelty_and_reproducibility: **Clarity:** I think the paper is clearly written and was easy to follow for me, despite some fairly technical content.

**Quality:** Overall, the quality of the paper is okay. The paper is well written and motivated with interesting ideas and insights. Most experiments are appropriate, although insufficient. There are some concerns regarding the thoroughness of the experiments. Also, there is very relevant work that is not properly discussed and taken into account (see above).

**Novelty:** Technically, the method clearly is novel, to the best of my knowledge. It is creative and original work. I do have some concerns regarding the relevance and significance, though (see above; benefits only shown clearly for VESDE + ODE sampling).

**Reproducibility:** I think the approach is reproducible. The method is described clearly, experiment details seem to be provided in the appendix, and the submission also includes code.
summary_of_the_review: In summary, I think the main idea and the proposed method of the paper, the STF for training diffusion models, is interesting, well-motivated, and also novel. I do not have any concerns with respect to the technical and methodological novelty.

However, my concern is that the method has been primarily validated on experiments using a standard VESDE scheme, together with ODE solvers for sampling. This setup is known to be challenging and in practice, almost all diffusion models actually rely on a different setup, either the VPSDE or the recent EDM scheme. It isn't clear whether there is any benefit when applied to these methods. If there isn't, then the new method is not particularly important or significant. Consequently, I cannot recommend the paper for publication at this point. I would recommend the authors to run experiments with different SDEs and setups and demonstrate the benefit of STF more broadly.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Score-based generative models are learned via denoising score matching at many time steps during a diffusion process. This paper identifies an important problem with such loss function---estimating this loss via simple monte carlo could have high variance in the middle stage of the diffusion. The authors then propose to use self-normalized importance sampling to reduce the variance, which leads to a new objective function called stable target field. Empirical results show that this new objective consistently improves the performance of score-based generative models. 
strength_and_weaknesses: ## Strengths

* I find the paper very interesting, address a relevant problem in score-based model training, and potentially have high impact. Simply observing the variance problem in the middle stage of training is nontrivial and requires a deep understanding of the denoising score matching idea and the training of score-based models.

* Moreover, the proposed fix is simple and very practical. Both the theory and empirical results suggest the modified loss has lower variance.

## Weaknesses

I do not see much weakness except a few minor points below:

* It would be nice to give a derivation of eq. (7) in appendix.
* It would be nice to see if there a connection between the modified denoising score matching loss and an importance-weighted version of the ELBO. See, e.g., 

Bornschein, J., & Bengio, Y. (2014). Reweighted wake-sleep. arXiv preprint arXiv:1406.2751.
Burda, Y., Grosse, R., & Salakhutdinov, R. (2015). Importance weighted autoencoders. arXiv preprint arXiv:1509.00519.

* since this paper focus on reducing variance of score matching loss, the following paper that proposes a control variate for DSM could be relevant

Wang, Z., Cheng, S., Yueru, L., Zhu, J., & Zhang, B. (2020, June). A wasserstein minimum velocity approach to learning unnormalized models. In International Conference on Artificial Intelligence and Statistics (pp. 3728-3738). PMLR.
clarity,_quality,_novelty_and_reproducibility: This paper is of high quality with an original observation, theoretically sound fix, and sufficient theoretical and empirical evidence to support the claim. 

The technical part of this paper is very well-written. It clearly states the variance problem, and introduces all necessary backgrounds to understand the fix. Nevertheless, I feel the introduction part can be improved. It frequently mentions terminology like stable targets, direction of inverse path, reference batch, etc. It is hard to get the idea by just looking at the first two pages. I suggest to revise them and point out the variance problem more directly. 

Minor:

* At the beginning of section 4: "The vanilla denoising score-matching approach (Equation 3) can viewed as", missing a verb

summary_of_the_review: I find this paper identify an important problem, propose a nice solution, and is sufficiently practical to have large impact. Therefore, I recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to reduce the sample variance for
training score-matching models. The method is based on using multiple
reference samples and weighted importance sampling to compute more
stable targets. Small-scale experiments show improvement when using
the proposed scheme.

strength_and_weaknesses: **Strengths**
- I found the paper interesting to read, with the teorethical
  part well  explained.
- The paper demonstrates in simple examples the high target variance
  of vanilla score-matching, and how it is reduced when using the
  proposed approach.
- The approach improves the sampling of the baseline methods, without
  incurring in extra training time. In the case of CIFAR-10, it
  significantly improves the training efficiency.

**Weaknesses**

- The empirical evidence in favor of the method is rather weak, with
  experiments showing some improvements only for the small CIFAR-10
  benchmark.
- In Figures 6 and 7, FID is computed on only 1K samples, yet the
  Inception features are typically of higher dimension, so the Gaussian
  approximation for the FID would be ill-conditioned. I wonder if the
  authors can comment on that (Curves show scores of ~40, while ~2 is
  reported in Table 1.)

clarity,_quality,_novelty_and_reproducibility: The paper is clear, has good writing quality and novelty. Source code was provided for reproducibility.
summary_of_the_review: The paper proposes an interesting idea for improving score-matching training. On the flip side the paper presents limited
experimental evidence in favor of the proposed approach.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: It is known that learning score-based generative models via denoising score matching can suffer from high variance training objectives. This work circumvents this problem by introducing a low-variance yet biased training objective, incorporating an importance-weighted denoising objective. The introduced objective is well-motivated and it is analyzed theoretically and empirically in different settings. The experimental results show that the proposed approach can marginally improve the performance of score-based diffusion models. The rebuttal successfully addressed some of the minor issues raised by reviewers. Given this, I am happy to recommend accept. 
justification_for_why_not_higher_score: Albeit the nice theoretical variance reduction results, the empirical results are marginal and whether the proposed technique will be widely used is questionable. 
justification_for_why_not_lower_score: The submission provides valuable insights into variance reduction in diffusion models.
Decision: Accept: poster