Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper the authors propose an improvement for score-matching based
generative modeling [1] resembling low temperature sampling as in GANs or
flow-based models. Similarly to [2] they propose to modify the drift function
used in the sampling step of the diffusion model by including the gradient of
some classifier. However, contrary to [2] the classifier considered by the
authors is implicit in the sense that it is purely defined by a conditional and
an unconditional generative model. The authors show that using such a classifier
allows to control a trade-off between IS and FID on the ImageNet dataset. The
use of such implicit classifier also provides intuition on the guidance
influence in score-based generative modeling: the model tries to reduce the
unconditional likelihood while increasing the conditional likelihood.

[1] Song, Sohl-Dickstein, Kingma, Kumar, Ermon, Poole -- Score-based Generative Modeling through Stochastic Differential Equations
[2] Dhariwal, Nichol -- Diffusion Models Beat GANs on Image Synthesis
main_review: Strengths:

-The paper is clear and overall well-written. The experiments are easy to read
 and illustrate the announced trade-off between FID and IS.

-The idea of using implicit classifiers is interesting and provides a new step
 toward defining guided score-based generative models using only score-based
 models. Similarly, I found the intuitive explanation for how guidance works to
 be well motivated and illustrated with the implicit classifier guidance.

-The experiments are interesting and clearly illustrate the trade-off between IS
 and FID. 

Weaknesses:

-In my opinion, the contribution is quite incremental in the sense that [2]
 already used classifiers for the guidance of score-based models. The authors
 justify the use of unconditional classifiers by claiming that the use of the
 classifiers of [2] can be interpreted as an adversarial attack and that
 therefore the (guided) score-based generative modeling behaves like GANs. I am
 not fully convinced by this explanation as I think that even though the model
 in [2] incorporate a guidance classifier term there are still quite far from
 being close to GANs. Also, I am not clear as of why the classifier used in [2]
 would constitute an adversarial attack against the model. As of now, the
 motivation for considering implicit classifier as presented in the introduction
 seems a bit superficial. I think the paper would truly benefit an investigation
 of the shortcomings of the score-based generative models using classifier
 guidance. Then, we could fully appreciate the benefits of using unconditional
 guidance diffusion models.

-Even though the paper is mostly experimental I think that the theoretical part
 of the paper could have been more developed. In particular it is not clear to
 me what are the properties the model given in the equation at the end of
 p.3. Also, I think that in order to be able to say that the classifier guidance
 model appoximately sample from $\tilde{p}_\theta(z_\lambda|c)$ one needs to
 explicitly write down what is the joint forward model and how to derive the
 backward model. The approximation can then be obtained similarly to Equation 4
 in [3].

General comments:

-In Table 1, I would have expected non-guided model to have better FID than the
 guided model with w=0.1. However, this does not seem to be the case. Similarly,
 in Table 2 while the IS score keeps increasing with the parameter w it is not
 the case for FID. Is there a reason why the best FID score is reached for w=0.3?

-I think it would have been interested to show the nearest neighbors in the
 ImageNet dataset depending on the parameter w. How close is the model to the
 original dataset for large values of w?

-For large values of w the score given by Equation (6) is approximately scaled
 by w. Thinking in terms of diffusions this amounts to scale the drift by a
 parameter w. The Lipschitz constant of the drift is also multiplied by w and
 therefore in order to obtain a stable discretization we need to divide the
 stepsize in the Euler-Maruyama discretization by a parameter w. However in the
 experiments it seems that the authors choose the same stepsize for every value
 of w. Could you comment on this?

[1] Song, Sohl-Dickstein, Kingma, Kumar, Ermon, Poole -- Score-based Generative Modeling through Stochastic Differential Equations

[2] Dhariwal, Nichol -- Diffusion Models Beat GANs on Image Synthesis

[3] De Bortoli, Thornton, Heng, Doucet -- Diffusion Schrodinger Bridge with Applications to Score-Based Generative Modeling 
summary_of_the_review: The main idea of the paper is interesting and the experiments are quite
convincing.  However, I feel that the motivation behind this improvement is
quite superficial. I think that the authors should better motivate their study,
especially putting an emphasis on the limitations of classifier guidance in
score-based generative models. I will increase my score if the authors address
my concerns.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposed a method to trade-off sample diversity for sample quality in diffusion models, which is termed unconditional guidance. Different from the prior work called classifier guidance (Dhariwal & Nichol 2021) that relies on a classifier for providing the guidance signal, the proposed unconditional guidance mixes the score estimates of a conditional diffusion model and an unconditional diffusion model for a trade-off between sample quality and sample diversity. Experiments on ImageNet 64x64 and 128x128 showed that the proposed model can achieve the claimed quality-diversity tradeoff regarding FID and IS scores. 

main_review: Strengths:
(1) The paper is well written and easy to read. The method is clearly stated and I particularly like the way the authors posit some hypotheses on classifier guidance (i.e., being adversarial to the classifier) to motivate the proposed method and the resulting discussions.
(2) The idea of constructing an implicit classifier from the conditional and unconditional generative model to provide a guidance signal in diffusion models is new to me (although I have some concerns about its significance in the below comments).
(3) Experiments demonstrate the effectiveness of the proposed method in trading off sample diversity for sample quality. 

Weaknesses:
(1) My first major concern is about the significance of the proposed method: is the unconditional guidance of more practical significance than the classifier guidance? The proposed unconditional guidance relies on a mixture of a conditional diffusion model and an unconditional diffusion model for the quality-diversity trade-off. 1) It means we have to train a conditional diffusion model from scratch to apply the proposed method. Wouldn’t it be much less flexible than training a noisy image classifier from scratch and directly applying it to the pre-trained unconditional diffusion model (in a plug-n-play manner)? Plus, the proposed method has worse sampling speed and no better sample quality and diversity trade-off compared to classifier guidance. 2) The authors mentioned the truncation trick in GANs as their motivating example, but the truncation trick is a post-hoc process that plugs in a pre-trained GAN while the proposed method has to train a new generative model (with labels!) to get the trade-off. On the contrary, the classifier guidance enjoys the same post-hoc property as the truncation trick (though it also needs labels during inference). In this sense, the classifier guidance more resembles the truncation trick than the proposed method.

(2) In experiments, to show the trade-off between sample quality and sample diversity, I would recommend adding the precision and recall metrics rather than solely focusing on FID and IS scores. Because the FID score does not only capture the sample diversity but also is affected by the sample quality, a precision-recall curve will be more convincing.

(3) In experiments, only ImageNet 64x64 and 128x128 are considered. I think adding experiments on ImageNet 256x256 and ImageNet 512x512 (as the classifier guidance paper did) will make the results in the paper more impressive.

(4) When introducing the equation $\epsilon(z_\lambda, c) = \sigma_\lambda \nabla_{z_\lambda}\log p(z_\lambda|c)$, is the $\sigma_\lambda$ assumed to be negative? It seems that a negative sign is missing before $\sigma$. 

(5) The method name "unconditional guidance" is inappropriate because it requires training a conditional diffusion model with labels.

summary_of_the_review: Although the idea is new in the context of diffusion models and the experiments support the major claim to some extent, I think 1) the proposed method is of less practical significance and less flexibility, compared with the prior work on classifier guidance, 2) the experiments can be improved to better support the claim and to make the results more impressive. Thus, my initial recommendation is not accepting the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper belongs to the class of diffusion-based generative models for generating synthetic images using class labels as guidance. It starts with a view of a recent work Dhariwal-Nichol (2021) where a classifier model is trained jointly with the diffusion-based generative model and the score (gradient of log probability) of the classifier is magnified and added to the score of the generative model to increase fidelity and the expense of diversity. In the view, Dhariwal-Nichol's approach resembles adversarial attacks of GAN-based approaches, motivating the key question that the paper addresses: whether we can train a generative model without a classifier but still enjoying the ability to trade diversity for fidelity.

The answer presented in the paper is to replace the (explicit) classifier with an implicit classifier modeled as a conditioned generative model (conditioned on the class), and the weights between the conditioned generative model than the unconditioned generative model are shared by viewing the unconditioned generative model as part of the conditional model but with an additional null class.

Experiments on ImageNet shows that the results compare favourably to Dhariwal-Nichol (2021) and Ho et al (2021), and in some cases slightly outperforming.
main_review: ### Positive Points

* Quite an impressive background review and explanation of Dhariwal-Nichol (2020)'s approach resembling GAN.
* Although not clearly stated in the paper, but I find replacing an explicit classifier with a generative-based implicit classifier has a potential of improving classification performance on noisy latent spaces because of better model capacity and more resemblance to the already accurate unconditional generative model.

### Negative Points

* The motivation for removing a classifier is not entirely clear. For understanding? Yes but that is not too difficult to see from the last equation/line of page 3. If the reason is to avoid adversarial attacks altogether then the reason leads to another question. Why do we need to avoid adversarial attacks? The mathematical formulation of GAN may be flawed for optimisation but the idea of increasingly harder examples to learn is beneficial. This tactic is very common in imbalanced classification problems.  More discussion on the motivation of the paper is needed.
* The paper introduced a clever trick to allow sharing weights between a conditioned model and an unconditioned model, by introducing an additional label representing the unconditional case. This works well for class labels and allows for deriving implicit classifiers directly from the model via Bayes rules. However, there is an efficiency cost at inference. For every iteration at inference, the model has to run twice, to generate conditioned and unconditioned probabilities. Using an explicit classifier, as pointed out by the authors, is faster. While I think speed is a concern, what would be more interesting is discussion, both theoretically and experimentally, of the positive point 2 above.
* The title "unconditioned diffusion guidance" is misleading. The presented model is clearly conditioned diffusion guidance because you still condition on the labels. In the end, you just replaced an explicit classifier with an implicit one. It is more appropriate to call the approach as conditioned diffusion guidance without a classifier, as mentioned in a few places in the paper.
* The explanation on page 4 that claims that a (w+1)-weight guided unconditioned model should theoretically lead to the same result as a w-weighted guided conditioned model is somewhat not entirely true. We are roughly discussing trading an unconditioned generative model and a classifier model for a conditioned generative model. If the capacity of the generative models outweighs that of a classifier model, it would be fair to say the latter would be dominated by the former.
* The experiments are somewhat lacking. Apart from the replacement of the explicit classifier with an implicit classifier sharing weights with the generative model, there is a switch from discrete-time training in Dhariwal-Nichol (2021) to continuous-time training here. The experimental results are on the total performance. However, how much of improvement of FID and IS in best-case scenarios really comes from the classifier replacement (i.e. the implicit classifier model has larger capacity than the explicit classifier, and shares weights with the generative model)? How much comes from the switch from discrete-time training to continuous-time training?
* In section 5, the argument that the proposed unconditional-guided sampler "...cannot be interpreted as a gradient-based adversarial attack on a classifier" and the explanation on how guidance works "it decreases the unconditional likelihood of the sample while increasing the conditional likelihood" somewhat contradicts with each other, since the latter can be viewed as a form of implicitly injecting adversarial attacks into the computation of the score (i.e. last equation on the 2nd-last line of page 4, section 3.2).


summary_of_the_review: The paper has some novelties but they are not well-motivated and not well backed up by the provided discussion and evidences.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an unconditional guidance, which discards the classifier in previous work.Under such unconditional guidance, it is also able to obtain a trade-off between sample quality and diversity as the classifier guidance.
main_review: Even though the effectiveness of the proposed unconditional guidance is justified, I cannot admit its advantages comparing with the classifier guidance. Specifically, the proposed method need to simultaneously train an unconditional model and a conditional one, while the classifier guidance method need to simultaneously train a conditional model and a classifier. In my opinion, both of them need to train two models, so the computational cost to train is similar.  What's more, for the sampling phase, the proposed method is much slower as pointed out in Sec.5.
Therefore, I'm confused with the usefulness and the advantages of the proposed unconditional guidance.

As for the experiments, all the network architecture and hyper-parameter settings are following the previous work (Dhariwal & NIchol 2021). Through the experiments, the authors verify that the unconditional method can also achieve the goal of balancing the sample quality and diversity. However, as a complete work, it should also sufficiently exploits its best potential performance and gives some optimal settings for it, so as to provide conveniences for others to follow. 
summary_of_the_review: In summary, I give positive comments to this work though some aspects of this method can be improved.
review_title: null

Meta-review: This paper modifies the conditional diffusion model guided by a classifier, as introduced by Dhariwal & Nichol 2021, by replacing the explicit classifier with an implicit classifier. This implicit classifier is derived under Bayes' rule and combined with the conditional diffusion model. This combination can be realized by mixing the score estimates of a conditional diffusion model and an unconditional diffusion model. A trade-off between sample quality and diversity, in terms of the IS and FID scores, can be achieved by adjusting the mixing weight. The paper is clearly written and easy to follow. However, the reviewers do not consider the modification to be that significant in practice, as it still requires label guidance and also increases the computational complexity. From the AC's perspective, the practical significance could be enhanced if the authors can generalize their technique beyond assisting conditional diffusion models.
Decision: Reject