Unconditional Diffusion Guidance
Authors: Jonathan Ho
Tim Salimans
Venue: ICLR 2022 Submitted (2022)
Keywords: diffusion
score
guidance
generative

Abstract: Classifier guidance is a recently introduced method to trade off mode coverage and sample fidelity in conditional diffusion models post training, in the same spirit as low temperature sampling or truncation in other types of generative models. Classifier guidance combines the score estimate of a diffusion model with the gradient of an image classifier and thereby requires training an image classifier separate from the diffusion model. It also raises the question of whether guidance can be performed without a classifier. We show that guidance can be indeed performed by a pure generative model without such a classifier: in what we call unconditional guidance, we jointly train a conditional and an unconditional diffusion model, and we combine the resulting conditional and unconditional score estimates to attain a trade-off between sample quality and diversity similar to that obtained using classifier guidance.

Under review as a conference paper at ICLR 2022
UNCONDITIONAL DIFFUSION GUIDANCE
Anonymous authors Paper under double-blind review
ABSTRACT
Classifier guidance is a recently introduced method to trade off mode coverage and sample fidelity in conditional diffusion models post training, in the same spirit as low temperature sampling or truncation in other types of generative models. Classifier guidance combines the score estimate of diffusion model with the gradient of an image classifier and thereby requires training an image classifier separate from the diffusion model. It also raises the question of whether guidance can be performed without a classifier. We show that guidance can be indeed performed by pure generative model without such a classifier: in what we call unconditional guidance, we jointly train a conditional and an unconditional diffusion model, and we combine the resulting conditional and unconditional score estimates to attain trade-off between sample quality and diversity similar to that obtained using classifier guidance.
INTRODUCTION
Diffusion models have recently emerged expressive and flexible family of generative models, delivering competitive sample quality and likelihood scores on image and audio synthesis tasks (Sohl-
Dickstein al. 2015 Song & Ermon 2019 Ho et al. 2020 Song al. 2021b; Kingma ct al. 2021
Song et al. (0211). These models have delivered audio synthesis performance rivaling the quality of autoregressive models with substantially fewer inference steps (Chen et al.) 2021 Kong al.)
2021). and they have delivered ImageNet generation results outperforming BigGAN-decy (Brock al.
2019) and VQ-V (Razavi et al.) 2019) in terms of FID score and classification accuracy
score Ho et al., 2021 Dhariwal & Nichol 2021
Dhariwal Nichol (202I) proposed classifier guidance, a technique boost the sample quality of a diffusion model using an extra trained classifier. Prior to classifier guidance, it was not known how to generate "low temperature" samples from diffusion model similar to those produced by truncated BigGAN (Brock et al., (209)) or low temperature Glow (Kingma & Dhariwal] (018)): naive attempts, such as scaling the model score vectors decreasing the amount of Gaussian noise added during diffusion sampling, are ineffective (Dhariwal & Nichol] 2021). Classifier guidance instead mixes a diffusion model's score estimate with the input gradient of the log probability of classifier. By varying the strength of the classifier gradient, [Dhariwal & Nicholl can trade off Inception score (Salimans et al. 2016) and FID score (Heusel et al. 2017) (or precision and recall) in manner similar varying the truncation parameter of BigGAN.
Figure I: Unconditional guidance on the malamute class for a 64x64 ImageNet diffusion model. Left to right: increasing amounts of unconditional guidance, starting from non-guided samples the left.

---

Under review as a conference paper at ICLR 2022
Figure 2: The effect of guidance on a mixture three Gaussians, each mixture component represent- ing data conditioned on a class. The leftmost plot is the non- guided marginal density. Left to right are densities of mixtures of normalized guided conditionals with increasing guidance strength.
non-guided
We interested in whether classifier guidance be performed without classifier. Because classifier guidance mixes score estimate with a classifier gradient during sampling, classifier-guides diffusion sampling can be interpreted attempting to confuse an image classifier with gradient- based adversarial attack. This raises the question of whether classifier guidance is successful at boosting classifier-based metrics such as FID and Inception score (IS) simply because it is adversarial against such classifiers. Stepping in direction of classifier gradients also bears some resemblance to GAN training, particularly with nonparameteric generators; this also raises the question of whether classifier-guided diffusion models perform well on classifier-based metrics because they are beginning to resemble GANs, which are already known to perform well on such metrics.
To resolve these questions, present unconditional guidance, guidance method which avoids any classifier entirely. Rather than sampling in the direction of the gradient of an image classifier, unconditional guidance instead mixes the score estimates of conditional diffusion model and jointly trained unconditional diffusion model. By sweeping over the mixing weight, we attain a FID/IS tradeoff similar to that attained by classifier guidance. Our unconditional guidance results demonstrate that pure generative diffusion models are capable of synthesizing extremely high fidelity samples possible with other types of generative models.
BACKGROUND
We train diffusion models in continuous time (Song et al. 2021): letting x p(x) and z = {zx A [Amin Amax] for hyperparameters Amin Amax R.
2021b Chen al. 2021: Kingma et al.)
the forward process (z x) is the variance- peeenccerreeevinggg Markov process (Sohl-Dickstein al. 2015):
<(>x|> = (T) where o = 1/(1 + + of = of ((>l|r))
(1)
where X, ON (1
)oー
(2)
We will use the notation p(z) (or p(zx)) to denote the marginal of z (or zx) when x ~ p(x) and can be interpreted the log signal-to-noise ratio of 2x.
(z(xx). Note that À log a / and the forward process runs in the direction of decreasing X.
Conditioned the forward process can be described in reverse by the transitions gizx [Zx. x) A (PPxpx (2x, x), 2 I), where
jxy A2 x) x (ax/ax)2x (1 6- (1-c~~N)axxx JaxXx, /xIa -
(3)
The reverse process generative model starts from pa (Z)mmn)) = -O. We specify the transitions:
(z (2x) = (N xo()))) "
(4)
During sampling, we apply this transition along increasing sequence Amin = <..... Amax for timesteps; in other words, we follow the discrete time ancestral sampler of Sohl-DDissteenn al. (2015); Ho et al. (022)). If the model Xy is correct, then as T - co, obtain samples from an SDE whose sample paths are distributed as p(z) )) (Song et al. 2021b), and we use pz(z) to denote the continuous time model distribution. The variance log-space interpolation of x and o AA as
2

---

Under review as a conference paper at ICLR 2022
suggested by Nichol & Dhariwal 2021); y; we found it effective to constant hyperparameter
rather than learned Z ox-dependent v. Note that the variances simplify to 211 as X - X, so v has an effect only when sampling with non-infinitesimal timesteps as done in practice.
The reverse process mean comes from an estimate Xy (zx)
plugged into g(zx/ |2, (Ho et al. 2020; Kingma et al.) 2021) (Xe also receives À as input, but we suppress this to keep our notation clean). We parameterize Xy in terms of e-prediction (Ho al. 2020 Xg (ZA) (zx (xxee(z)))) and we train on the objective
Be, B. (lEoo((((x))
(5)
where E (0, ZA OAX + O2E, and À is drawn from distribution p()) over Dminn Amax] This objective is denoising score matching (Vincent 2011 Hyvärinen & Dayan 2005) over multiple noise scales (Song & Ermon, 2019), and when p()) is uniform, the objective is proportional variational lower bound on the marginal log likelihood of the latent variable model [ p(x z p (z dzz
the
ignoring the term for the unspecified decoder p(x k) and for the prior at ZAmin (Kingma et al. 2021).
If p(A) is not uniform, the objective can be interpreted as weighted variational lower bound whose
weighting can tuned for sample quality (Ho al., 2020 Kingma et al. 2021). ). We use a p(A)
inspired by the discrete time cosine noise schedule of Nichol & Dhariwal (2021): :: we sample X
via À = lol tan + b) for uniformly distributed u E [0, 1], where = arctan (e-ras// /mmas/2 and
a arctan(e Amiiu / - b. This represents a hyperbolic secant distribution modified to be supported on a bounded interval. For finite timestep generation, we use values corresponding to uniformly spaced u E [0, 1]. and the final generated sample is Xo (Z
Because the loss for co(z)) denoising score matching for all X, the score (0(2)) learned by our model estimates the gradient of the log-density of the distribution of our noisy data Za, that is (2x) & σ Væ, log p(z)). Sampling from the learned diffusion model resembles using Langevin diffusion to sample from sequence of distributions p( x that converges to the conditional distribu- tion pxx) of the original data x.
In the case of conditional generative modeling, the data is drawn jointly with conditioning informa- tion c, i.c. a class label for class-conditional image generation. The only modification to the model is that the reverse process function approximator receives e as input, in co (Z,, C).
3 GUIDANCE
An interesting property of certain generative models, such as GANs and flow-based models, is the ability to perform truncated or low temperature sampling by decreasing the variance or range of noise inputs to the generative model at sampling time. The intended effect is to decrease the diversity of the samples while increasing the quality of each individual sample. Truncation in BigGAN (Brock) ee al. 2019), for example, yields a tradeoff curve between FID score and Inception score for low and high amounts of truncation, respectively. Low temperature sampling in Glow (Kingma & Dhariwal 2018 has similar effect.
Unfortunately, straightforward attempts of implementing truncation or low temperature sampling in diffusion models are ineffective. For example, scaling model scores decreasing the variance of Gaussian noise in the reverse process cause the diffusion model to generate blurry, low quality samples (Dhariwal & Nichol 2021).
3.1 CLASSIFIER GUIDANCE
To obtain a truncation-like effect in diffusion models, Dhariwal & Nichol (2021) guidance, where the diffusion score e Ez c) x d Vz, log p(zx(())) modified to include the gradient of the log likelihood of an auxiliary classifier model po po(clz)) as follows:
introduce classifier
Eo (Zx, c) (z c) + wo Vz, log2z(c(zz) & Vz, [log ((x/c() + w log Po (c|zx))
where is parameter that controls the strength of the classifier guidance. This modified score (z c) is then used in place of E (zx, c) when sampling from the diffusion model, resulting in approximate samples from the distribution
Do (2x/c) x pa po(zx(c)po(c(zz>)
3

---

Under review as a conference paper at ICLR 2022
Algorithm Joint training diffusion model with unconditional guidance
Require: Puncond probability of unconditional training
1: repeat
2:
(x, c) p(x, c)
Sample data with conditioning from the dataset Randomly discard conditioning to train unconditionally Sample log SNR value
3:
c c Ø with probability Puncoud
4:
A~ p())
5:
E E~ (0,I)
6:
AA mxx++xx OXE
Corrupt data to the sampled log SNR value D Optimization of denoising model
7:
Take gradient step on Vo (E E|}
8:
until converged
The effect is that of up-west up-weighting the probability of data for which the classifier pa (c Zx) assigns high likelihood the correct label: data that can be classified well scores high on the Inception score of perceptual quality (Salimans et al., 2016), which rewards generative models for this by design. Dhariwal & Nichol therefore find that by setting w > 0 they can improve the Inception score of their diffusion model, at the expense of decreased diversity in their samples.
on toy 2D example of three classes, in which the conditional distribution for each class is an isotropic Gaussian. The form of each conditional upon applying guidance is markedly non-Gaussian. As guidance strength is increased, each conditional places probability mass farther away from other classes and towards directions of high confidence given by logistic regression, and most of the mass becomes concentrated in smaller regions. This behavior can be seen simplistic manifestation of the Inception score boost and sample diversity decrease that occur when classifier guidance strength increased in an ImageNet model.
Figure illustrates the effect of numerically solved guidance (z c) x po(z)/c)pu(clz)
Applying classifier guidance with weight w + 1 to an unconditional model would theoretically lead to the same result as applying classifier guidance with weight conditional model, because wo po(z, (c) (c)c xx pe (e (2))))p(c(2z)) w+1, ( or in terms of scores,
< + (w + VAAA,, logpa(clza Ro σ VN [(gg(( + (w + 1) log -A 22 [logp(zx(c) w log (c/zx)),
but interestingly, DDhriwwl& Dhariiaal & Nichol obtain their best results when applying classifier guidance already class-conditionn model, as opposed to applying guidance to an unconditional model. For this reason, we will stay the setup of guiding an already conditional model.
3.2 UNCONDITIONAL GUIDANCE
While classifier guidance successfully trades off IS and FID as expected from truncation or low temperature sampling, it is nonetheless reliant on gradients from an image classifier and we seek to eliminate the classifier for the reasons stated in Section I Here, we describe unconditional guidance, which achieves the same effect without such gradients. Unconditional guidance is an alternative method of modifying e (2,,, c) to have the same effect as classifier guidance, but without classifier.
Instead of training separate classifier model, we choose to train an unconditional denoising diffusion model Po (z) parameterized through a score estimator 6 (0x))) together with the conditional model pe(z(c) parameterized through e (Z,, c). We use single neural network to parameterize both models, where for the unconditional model we can simply input a null token Ø for the class identifier when predicting the score, i.e. (z(z)) (z c 2). We jointly train the unconditional and conditional models simply by randomly setting с to the unconditional class identifier Ø with some hyperparameter. (It would certainly be possible to train separate models instead of jointly training them together, but we choose joint training because it is extremely simple to implement, does complicate the training pipeline, and does not increase the total number of parameters.)
probability Puncond,
We can then apply Bayes' rule to obtain an implicit classifier (c/zA) pe(zxlc)/po(z,) The score of this implicit classifier will then be given by Va, lo _e(ea(z, Es (z.)). Applying classifier guidance with this implicit classifier yields the following modification to the
4

---

Under review as a conference paper at ICLR 2022
Algorithm 2 Conditional sampling with unconditional guidance
Require: w: guidance strength
Require: c: conditioning information for conditional sampling
Require: A1....; Ar: increasing log SNR sequence with A Amin+ AT Amxx
1: 21 ~ (O
2: for = I....,T' do
> Form the unconditional-guidee score at log SNR X
3: é, (1 (Zz; c) weol (Z+)
> Sampling step (could be replaced by another sampler, e.g. DDIM)
4:
Xr x- (Zr aA, Ex,/eaaax
5:
1141~ " - 1e (Ze, &c), (o (62e41l)e)
(0 (odAA if < T else Zi+1 = 8c
6: end for
7: return ZT+1
diffusion score estimator:
ox (0(2), = (1 + w)eo(2x. weo(2x) = ax Ve, [lo poxxp(zzlc) + w log p/(c/2x)
(6)
We then use Eo(zx. c) to sample from our diffusion model as usual, thus producing approximate samples from po (zxle) pe (zx c)pp (c/zx) Algorithms and describe training and sampling with unconditional guidance in detail.
Note that in some cases, may be possible to entirely avoid training an unconditional model. If know the class distribution and there are only few classes, we can use the fact that > p(x[c)p(c) = p(x) obtain an unconditional score from conditional scores without explicitly training for the unconditional score. Of course, this would require as many forward passes as there are possible values of and would be inefficient for high dimensional conditioning signals.
It is not obvious priori that inverting generative model using Bayes' rule yields good classifier that provides useful guidance signal. For example, Grandvalet & Bengio (2004) find that discriminative models generally outperform implicit classifiers derived from generative models, even in artificial cases where the specification of those generative models exactly matches the data distribution. In cases such as ours, where expect the model be misspecified, classifiers derived by Bayes' rule can be inconsistent (Grünwald & Langford 2007) and we lose all guarantees on their performance. Nevertheless, in Section we show empirically that unconditional guidance is able to trade off FID and IS in the same way as classifier guidance. In Section 5 we discuss the implications of unconditional guidance in relation classifier guidance.
4 EXPERIMENTS
We train diffusion models with unconditional guidance on area-downsampled class-conditional ImageNet (Russakovsky et al. 2015), the standard setting for studying tradeoffs between FID and Inception scores starting from the BigGAN paper (Brock et al. 2019).
The purpose of our experiments is to serve as a proof of concept to demonstrate that unconditional guidance able attain a FID/IS tradeoff similar to classifier guidance and understand the behavior of unconditional guidance, not necessarily to push sample quality metrics to state of the art on these benchmarks. For this purpose, we use model architectures and hyperparameters as the guided diffusion models of Dhariwal & Nichol (2021) (apart from continuous time training as specified in Section 2): those hyperparameter settings were tuned for classifier guidance and hence may be suboptimal for unconditional guidance. Furthermore, since we amortize the conditional and unconditional models into the same architecture without an extra classifier, we in fact are using less model capacity than previous work. Nevertheless, unconditional guuuddd models still produce competitive sample quality metrics and sometimes outperform prior work, as can be seen in the following sections.
5

---

Under review as a conference paper at ICLR 2022
Figure Unconditional guidance on 128x ImageNet. Left: non-guided samples, right: unconditional- -guided samples with w = 3.0. Interestingly, strongly guided samples such these display saturated colors. See Fig. g for more.
4.1 VARYING THE UNCONDITIONAL GUIDANCE STRENGTH
Here we experimentally verify the main claim of this paper: that unconditional guidance is able trade off IS and FID in manner like classifier guidance GAN truncation. We apply our proposed unconditional guidance to 64 64 and 128 x 128 flass-conditioaal ImageNet generation. In Table I and Fig. we show sample quality effects of sweeping over the guidance strength w on our 64 x 64 ImageNet models; Table 2 and Fig- 5 show the same for our 128 x 128 models. We consider w E (0, 0.2,.... 4} and calculate FID and Inception Scores with 50000 samples for each value following the procedures of Heusel et al. (201)) and Salimans al. (016)). All models used log SNR endpoints Amin -20 20 and Amax = 20. The 64 64 models used sampler noise interpolation coefficient 0.3 and were trained for 400 thousand steps; the 128 128 models used v 0.2 and were trained for 2.7 million steps.
We obtain the best FID results with small amount of guidance (w 0.1 the dataset) and the best IS result with strong guidance (w 2 4). Between these two extremes we see clear trade-off between these metrics of perceptual quality, with FID monotonically decreasing
0.3. depending

---

Under review as a conference paper at ICLR 2022
and IS monotonically increasing with Our results compare favorably to Dhariwal & Nichol (2021) and Ho et al. (2021), and in fact our 128 x 128 results are the state of the art in the literature. At
0.3, our model's FID score on 128 x 128 ImageNet outperforms the classifier-guide and at w = 4.0, our model outperforms BigGAN-deee at both FID and IS when BigGAN-deepp is evaluated its best-IS truncation level.
ADM-G,
Figures 1 and 6 to g show randomly generated samples from our model for different levels of guidance: here we clearly see that increasing unconditional guidance strength has the expected effect of decreasing sample variety and increasing individual sample fidelity.
Model
FID (1)
IS (t)
ADM Dhariwal & Nichol 2021
2.07
CDM (Ho et al.
2021
1.48
67.95
Ours
Puncond
0.1 0.1/0.2/0.5
0.0
1.8/ 2.21
53.71 / 52.9 47.66
0.1
1.55 1.62 1.91
66.11 64.58 56.1
w 0.2
2.1 2.08
78.91/76.99 /6..6
w = 0.3
/.93 2.65 4.3/4/3.44
92.8/ 88.64 74.92
0.4
106.2/ 101. 84.27
0.5
5.74/ 4.334
119.3 112.15 92.95
0.6
6.49 5.27
131.1 122.13 102
W = 0.7
8.62/7.73 /
141.8/ 131.6 109.8
w = 0.8
10.08 / /
151.6/ 140.82 / 116.9
0.9
11.41 / 10.09 8.21
161 / 150.26 / 124.6
1.0
12.6 11,21 /9.13
170.1 158.29 131.1
2.0
21.03 18.79 16.16
225.5 212.98 183
w 3.0
24.83 22.36 19.75
250.4 / 237.65 208.9
= 4.0
26.22 23.84 21.48
260.2 / 248.97 225.1
Table 1: ImageNet 64x64 results (w 0.0 refers non-guided models).
Puncond 0.1
Puncond 0.2
20
Puncond 0.5
FID
10
50
100
150
200
250
IS
Figure 4: IS/FID curves over guidance strengths for ImageNet 64x64 models. Each curve represents a model with unconditional training probability Puncond- Accompanies Table
4.2 VARYING THE UNCONDITIONAL TRAINING PROBABILITY
The main hyperparameter of unconditional guidance is Puncond. the probability of training unconditional generation during joint training of the conditional and unconditional diffusion models. Here, study the effect of training models varying Puncoud 64 x 64 ImageNet.
Table I and Fig. show the effects of Puncoud on sample quality. We trained models with Puncoud E (0.1, 0.2, 0.5), all for 400 thousand training steps, and evaluated sample quality across various guidance strengths. We find Puncond = 0.5 consistently performs worse than Puncond {0.1, 0.2) across the entire IS/FID frontier; Puncond E (0.1, 0.2} perform about equally well each other.
7

---

Under review as a conference paper at ICLR 2022
Based on these findings, conclude that only relatively small portion of the model capacity of the diffusion model needs to be dedicated to the unconditional generation task in order to produce unconditional- -guided scores effective for sample quality. Interestingly. for classifier guidance. Dhariwal & Nichol report that relatively small classifiers with little capacity are sufficient for effective classifier guided sampling, mirroring this phenomenon that found with unconditionally guided models.
4.3 VARYING THE NUMBER OF SAMPLING STEPS
Since the number of sampling steps T is known to have major impact the sample quality of diffusion model, here we study the effect of varying T on our 128 x 128 ImageNet model. Table 2 and Fig. 5 show the effect of varying 7 { {128, 256, 1024) over range of guidance strengths. As expected, sample quality improves when 7 is increased, and for this model 7 256 attains a good balance between sample quality and sampling speed.
Note that 7' = 256 is approximately the same number of sampling steps used by ADM-G (Dhariwal & Nichol 2021). which outperformed by our model. However, it is important to note that each sampling step for our method requires evaluating the denoising model twice, once for the conditional (o c) and once for the unconditional co (zx). Because we used the same model architecture as ADM-G, the fair comparison in terms of sampling speed would be our T 128 setting, which underperforms compared to ADM-G in terms of FID score.
Model
FID (4)
IS (1)
BigGAN-deep. max IS (Brock el al. 2019)
25
253
BigGAN-deep (Brock el al.
2019
5.7
124.5
CDM (Ho 2021
3.52
128.8
LOGAN [wu
2019
3.36
148.2
ADM-G (Dhariwal & Nichol
2021
2.97
Ours
128/256/ /1024
W 0.0
8.11/ / 7.22 5.31 /4.53 4.5
81.46 / 82.45 / 81,54
w 0.1
105.01 106. 12. 104.67
0.2
3.7/ 3
130.79/ / 132.54 130.09
0.3
/.04 2.43
156.09 158.47 156
w 0.4
/ 2.48
183.01 183.41 180.88
w 0.5
/ / 2.96
206.94 207. 207.98 204.31
0.6
4.09 3.76 3.73
227.72 / 228.83 226.76
0.7
4,96 4.67 4.69
247.92 249.25 247.89
W
5.71
263.54 267.99 265.52
W 0.9
6.89 / 6.8 / 6.81
280. 283.41 281.14
W 1.0
7,88 7.86 / 778
295.29 / 297.98 294.56
2.0
15.9 / 15.93 / 15.75
378.56 377.37 373.18
3.0
19.77 / 19.77 19.56
409.16 407.44 405.68
W 1.0
21.55 21.53 21.45
422,29 421.03 419.06
Table 2: ImageNet 128x128 results (w = 0.0 refers to non non-guided models).
DISCUSSION
Since unconditional guidance is able to trade off IS and FID like classifier guidance without needing an extra trained classifier, we have demonstrated that guidance can be performed with pure generative model. We confirm that it is possible to maximize Inception scores using classifier- free unconditional guidance (and improve FID score for small amount of guidance), thus providing evidence that classifier-based sample quality metrics can improved using methods that are not adversarial against ImageNet classifiers using classifier gradients. Our diffusion models are parameterized by unconstrained neural networks and therefore their score estimates do not necessarily form conservative vector fields, unlike classifier gradients (Salimans & Ho 2021). Therefore, unconditional -uidee sampler follows step directions that do not resemble classifier gradients at all and thus cannot be interpreted as gradient-based adversarial attack classifier; hence results show that boosting
8

---

Under review as a conference paper at ICLR 2022
x T 128
20
-- = 256
- 512
FID
10
50
100
150
200
250
300
350
400
450
IS
Figure 5: IS/FID curves over guidance strengths for ImageNet 128x 128 models. Each curve represents sampling with different number of timesteps T. Accompanies Table
the classifier-based IS and FID metrics can be accomplished with pure generative models with a sampling procedure that adversarial against image classifiers.
We also have arrived at an intuitive explanation for how guidance works: decreases the uncondi- tional likelihood of the sample while increasing the conditional likelihood. Unconditional guidance accomplishes this by decreasing the unconditional likelihood with negative score term, which to our knowledge has yet been explored and may find uses in other applications.
On the practical side, unconditional guidance is extremely simple to implement and does not compli- cate the training pipeline of diffusion model, unlike classifier guidance which requires training an additional classifier model. This model has to be trained on noisy data Zy, so it is not possible to plug standard pre-trained classifier.
A potential disadvantage of classifier-free guidance is sampling speed. Generally, classifiers can be smaller and faster than generative models, so classifier guided sampling may be faster than classifier- free guidance because the latter needs to run two forward passes of the diffusion model, one for conditional score and another for the unconditional score. The necessity to run multiple passes diffusion model might be mitigated by changing the architecture inject conditioning late in the network, but we leave this exploration for future work.
Finally, any guidance method that increases sample fidelity at the expense of diversity must face the question of whether decreased diversity is acceptable. There may be negative impacts in deployed models, since sample diversity is important to maintain in applications where certain parts of the data are underrepresented in the context of the rest of the data. would an interesting avenue of future work to try boost sample quality while maintaining sample diversity.
CONCLUSION
We have presented unconditional guidance, a method increase sample quality while decreasing sample diversity in diffusion models. Unconditional guidance can be thought of as classifier guidance without classifier, and our results showing the effectiveness of unconditional guidance confirm that pure generative diffusion models are capable of maximizing classifier-based sample quality metrics while entirely avoiding classifier gradients. We look forward further explorations of unconditional guidance in wider variety of settings and data modalities.
REPRODUCIBILITY STATEMENT
Our model architectures and training hyperparameters are from Dhariwal & Nichol (2021). which can be referenced for implementation details. Hyperparameters related to guidance are are described in Section 4 In Algorithm I and Algorithm 2 we provide detailed pseudocode that further describes

---

Under review as a conference paper at ICLR 2022
implementation. We plan to open source code and model checkpoints with the release of the final version of this paper.
ETHICS STATEMENT
Like other machine learning methods, generative models can suffer from bias if applied data sets that are not carefully curated. As mentioned in Section 5 any guidance truncation method such as ours will drop sample diversity in favor of quality, and this may introduce more biases. We did not study the influence of our proposed method on this property of generative models, and careful evaluation advised before using our algorithm in practice.
REFERENCES
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- Grad: Estimating gradients for waveform generation. International Conference on Learning Representations, 2021.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. arXiv preprint arXiv:2105.05233, 2021.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Proceedings of the 17th International Conference on Neural Information Processing Systems, pp. 529-536, 2004.
Peter Grünwald and John Langford. Suboptimal behavior of bayes and mdl in classification under misspecification. Machine Learning, 66(2-3):119 19-149, 2007.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to local Nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pp. 6840-6843 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.
Cascaded diffusion models for high fidelity image generation. arXiv preprint arXiv:2106.1222 2021.
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Diederik Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions.
In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint arXiv:2 107 07.000600 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A Versatile Diffusion Model for Audio Synthesis. International Conference on Learning Representations, 2021.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. International Conference on Machine Learning, 2021.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In Advances in Neural Information Processing Systems, pp. 14837-14847, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 11(3):211-253) 2015.
10

---

Under review as a conference paper at ICLR 2022
Tim Salimans and Jonathan Ho. Should EBMs model the energy or the score? In Energy Based Models Workshop-ICLR 2021, 2021.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in Neural Information Processing Systems, pp. 2234-2242, 2016.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning. pp. 2256-2265, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances Neural Information Processing Systems, pp. 11895-11907, 2019.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. arXiv e-prints, pp. arXiv-2101, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben
Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021b.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computa- tion, 2377):1611-1674 2011.
Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy Lillicrap. Logan: Latent optimisation for generative adversarial networks. arXiv preprint arXiv:1912.00953, 2019.


Reviews:
[{"review_id": "yJTBJ5z4Kj2", "reviewer": "Reviewer_Y1QD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Positive Points\n\n* Quite an impressive background review and explanation of Dhariwal-Nichol (2020)'s approach resembling GAN.\n* Although not clearly stated in the paper, but I find replacing an explicit classifier with a generative-based implicit classifier has a potential of improving classification performance on noisy latent spaces because of better model capacity and more resemblance to the already accurate unconditional generative model.\n\n### Negative Points\n\n* The motivation for removing a classifier is not entirely clear. For understanding? Yes but that is not too difficult to see from the last equation/line of page 3. If the reason is to avoid adversarial attacks altogether then the reason leads to another question. Why do we need to avoid adversarial attacks? The mathematical formulation of GAN may be flawed for optimisation but the idea of increasingly harder examples to learn is beneficial. This tactic is very common in imbalanced classification problems.  More discussion on the motivation of the paper is needed.\n* The paper introduced a clever trick to allow sharing weights between a conditioned model and an unconditioned model, by introducing an additional label representing the unconditional case. This works well for class labels and allows for deriving implicit classifiers directly from the model via Bayes rules. However, there is an efficiency cost at inference. For every iteration at inference, the model has to run twice, to generate conditioned and unconditioned probabilities. Using an explicit classifier, as pointed out by the authors, is faster. While I think speed is a concern, what would be more interesting is discussion, both theoretically and experimentally, of the positive point 2 above.\n* The title \"unconditioned diffusion guidance\" is misleading. The presented model is clearly conditioned diffusion guidance because you still condition on the labels. In the end, you just replaced an explicit classifier with an implicit one. It is more appropriate to call the approach as conditioned diffusion guidance without a classifier, as mentioned in a few places in the paper.\n* The explanation on page 4 that claims that a (w+1)-weight guided unconditioned model should theoretically lead to the same result as a w-weighted guided conditioned model is somewhat not entirely true. We are roughly discussing trading an unconditioned generative model and a classifier model for a conditioned generative model. If the capacity of the generative models outweighs that of a classifier model, it would be fair to say the latter would be dominated by the former.\n* The experiments are somewhat lacking. Apart from the replacement of the explicit classifier with an implicit classifier sharing weights with the generative model, there is a switch from discrete-time training in Dhariwal-Nichol (2021) to continuous-time training here. The experimental results are on the total performance. However, how much of improvement of FID and IS in best-case scenarios really comes from the classifier replacement (i.e. the implicit classifier model has larger capacity than the explicit classifier, and shares weights with the generative model)? How much comes from the switch from discrete-time training to continuous-time training?\n* In section 5, the argument that the proposed unconditional-guided sampler \"...cannot be interpreted as a gradient-based adversarial attack on a classifier\" and the explanation on how guidance works \"it decreases the unconditional likelihood of the sample while increasing the conditional likelihood\" somewhat contradicts with each other, since the latter can be viewed as a form of implicitly injecting adversarial attacks into the computation of the score (i.e. last equation on the 2nd-last line of page 4, section 3.2).\n\n", "summary_of_the_paper": "The paper belongs to the class of diffusion-based generative models for generating synthetic images using class labels as guidance. It starts with a view of a recent work Dhariwal-Nichol (2021) where a classifier model is trained jointly with the diffusion-based generative model and the score (gradient of log probability) of the classifier is magnified and added to the score of the generative model to increase fidelity and the expense of diversity. In the view, Dhariwal-Nichol's approach resembles adversarial attacks of GAN-based approaches, motivating the key question that the paper addresses: whether we can train a generative model without a classifier but still enjoying the ability to trade diversity for fidelity.\n\nThe answer presented in the paper is to replace the (explicit) classifier with an implicit classifier modeled as a conditioned generative model (conditioned on the class), and the weights between the conditioned generative model than the unconditioned generative model are shared by viewing the unconditioned generative model as part of the conditional model but with an additional null class.\n\nExperiments on ImageNet shows that the results compare favourably to Dhariwal-Nichol (2021) and Ho et al (2021), and in some cases slightly outperforming.", "summary_of_the_review": "The paper has some novelties but they are not well-motivated and not well backed up by the provided discussion and evidences.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RCBjCujcXl", "reviewer": "Reviewer_4V2e", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Even though the effectiveness of the proposed unconditional guidance is justified, I cannot admit its advantages comparing with the classifier guidance. Specifically, the proposed method need to simultaneously train an unconditional model and a conditional one, while the classifier guidance method need to simultaneously train a conditional model and a classifier. In my opinion, both of them need to train two models, so the computational cost to train is similar.  What's more, for the sampling phase, the proposed method is much slower as pointed out in Sec.5.\nTherefore, I'm confused with the usefulness and the advantages of the proposed unconditional guidance.\n\nAs for the experiments, all the network architecture and hyper-parameter settings are following the previous work (Dhariwal & NIchol 2021). Through the experiments, the authors verify that the unconditional method can also achieve the goal of balancing the sample quality and diversity. However, as a complete work, it should also sufficiently exploits its best potential performance and gives some optimal settings for it, so as to provide conveniences for others to follow. ", "summary_of_the_paper": "This paper proposes an unconditional guidance, which discards the classifier in previous work.Under such unconditional guidance, it is also able to obtain a trade-off between sample quality and diversity as the classifier guidance.", "summary_of_the_review": "In summary, I give positive comments to this work though some aspects of this method can be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QJV0Vc-AD8r", "reviewer": "Reviewer_U6n2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n(1) The paper is well written and easy to read. The method is clearly stated and I particularly like the way the authors posit some hypotheses on classifier guidance (i.e., being adversarial to the classifier) to motivate the proposed method and the resulting discussions.\n(2) The idea of constructing an implicit classifier from the conditional and unconditional generative model to provide a guidance signal in diffusion models is new to me (although I have some concerns about its significance in the below comments).\n(3) Experiments demonstrate the effectiveness of the proposed method in trading off sample diversity for sample quality. \n\nWeaknesses:\n(1) My first major concern is about the significance of the proposed method: is the unconditional guidance of more practical significance than the classifier guidance? The proposed unconditional guidance relies on a mixture of a conditional diffusion model and an unconditional diffusion model for the quality-diversity trade-off. 1) It means we have to train a conditional diffusion model from scratch to apply the proposed method. Wouldn’t it be much less flexible than training a noisy image classifier from scratch and directly applying it to the pre-trained unconditional diffusion model (in a plug-n-play manner)? Plus, the proposed method has worse sampling speed and no better sample quality and diversity trade-off compared to classifier guidance. 2) The authors mentioned the truncation trick in GANs as their motivating example, but the truncation trick is a post-hoc process that plugs in a pre-trained GAN while the proposed method has to train a new generative model (with labels!) to get the trade-off. On the contrary, the classifier guidance enjoys the same post-hoc property as the truncation trick (though it also needs labels during inference). In this sense, the classifier guidance more resembles the truncation trick than the proposed method.\n\n(2) In experiments, to show the trade-off between sample quality and sample diversity, I would recommend adding the precision and recall metrics rather than solely focusing on FID and IS scores. Because the FID score does not only capture the sample diversity but also is affected by the sample quality, a precision-recall curve will be more convincing.\n\n(3) In experiments, only ImageNet 64x64 and 128x128 are considered. I think adding experiments on ImageNet 256x256 and ImageNet 512x512 (as the classifier guidance paper did) will make the results in the paper more impressive.\n\n(4) When introducing the equation $\\epsilon(z_\\lambda, c) = \\sigma_\\lambda \\nabla_{z_\\lambda}\\log p(z_\\lambda|c)$, is the $\\sigma_\\lambda$ assumed to be negative? It seems that a negative sign is missing before $\\sigma$. \n\n(5) The method name \"unconditional guidance\" is inappropriate because it requires training a conditional diffusion model with labels.\n", "summary_of_the_paper": "This work proposed a method to trade-off sample diversity for sample quality in diffusion models, which is termed unconditional guidance. Different from the prior work called classifier guidance (Dhariwal & Nichol 2021) that relies on a classifier for providing the guidance signal, the proposed unconditional guidance mixes the score estimates of a conditional diffusion model and an unconditional diffusion model for a trade-off between sample quality and sample diversity. Experiments on ImageNet 64x64 and 128x128 showed that the proposed model can achieve the claimed quality-diversity tradeoff regarding FID and IS scores. \n", "summary_of_the_review": "Although the idea is new in the context of diffusion models and the experiments support the major claim to some extent, I think 1) the proposed method is of less practical significance and less flexibility, compared with the prior work on classifier guidance, 2) the experiments can be improved to better support the claim and to make the results more impressive. Thus, my initial recommendation is not accepting the paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5K0JW1ZNVyS", "reviewer": "Reviewer_zw1c", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n-The paper is clear and overall well-written. The experiments are easy to read\n and illustrate the announced trade-off between FID and IS.\n\n-The idea of using implicit classifiers is interesting and provides a new step\n toward defining guided score-based generative models using only score-based\n models. Similarly, I found the intuitive explanation for how guidance works to\n be well motivated and illustrated with the implicit classifier guidance.\n\n-The experiments are interesting and clearly illustrate the trade-off between IS\n and FID. \n\nWeaknesses:\n\n-In my opinion, the contribution is quite incremental in the sense that [2]\n already used classifiers for the guidance of score-based models. The authors\n justify the use of unconditional classifiers by claiming that the use of the\n classifiers of [2] can be interpreted as an adversarial attack and that\n therefore the (guided) score-based generative modeling behaves like GANs. I am\n not fully convinced by this explanation as I think that even though the model\n in [2] incorporate a guidance classifier term there are still quite far from\n being close to GANs. Also, I am not clear as of why the classifier used in [2]\n would constitute an adversarial attack against the model. As of now, the\n motivation for considering implicit classifier as presented in the introduction\n seems a bit superficial. I think the paper would truly benefit an investigation\n of the shortcomings of the score-based generative models using classifier\n guidance. Then, we could fully appreciate the benefits of using unconditional\n guidance diffusion models.\n\n-Even though the paper is mostly experimental I think that the theoretical part\n of the paper could have been more developed. In particular it is not clear to\n me what are the properties the model given in the equation at the end of\n p.3. Also, I think that in order to be able to say that the classifier guidance\n model appoximately sample from $\\tilde{p}_\\theta(z_\\lambda|c)$ one needs to\n explicitly write down what is the joint forward model and how to derive the\n backward model. The approximation can then be obtained similarly to Equation 4\n in [3].\n\nGeneral comments:\n\n-In Table 1, I would have expected non-guided model to have better FID than the\n guided model with w=0.1. However, this does not seem to be the case. Similarly,\n in Table 2 while the IS score keeps increasing with the parameter w it is not\n the case for FID. Is there a reason why the best FID score is reached for w=0.3?\n\n-I think it would have been interested to show the nearest neighbors in the\n ImageNet dataset depending on the parameter w. How close is the model to the\n original dataset for large values of w?\n\n-For large values of w the score given by Equation (6) is approximately scaled\n by w. Thinking in terms of diffusions this amounts to scale the drift by a\n parameter w. The Lipschitz constant of the drift is also multiplied by w and\n therefore in order to obtain a stable discretization we need to divide the\n stepsize in the Euler-Maruyama discretization by a parameter w. However in the\n experiments it seems that the authors choose the same stepsize for every value\n of w. Could you comment on this?\n\n[1] Song, Sohl-Dickstein, Kingma, Kumar, Ermon, Poole -- Score-based Generative Modeling through Stochastic Differential Equations\n\n[2] Dhariwal, Nichol -- Diffusion Models Beat GANs on Image Synthesis\n\n[3] De Bortoli, Thornton, Heng, Doucet -- Diffusion Schrodinger Bridge with Applications to Score-Based Generative Modeling ", "summary_of_the_paper": "In this paper the authors propose an improvement for score-matching based\ngenerative modeling [1] resembling low temperature sampling as in GANs or\nflow-based models. Similarly to [2] they propose to modify the drift function\nused in the sampling step of the diffusion model by including the gradient of\nsome classifier. However, contrary to [2] the classifier considered by the\nauthors is implicit in the sense that it is purely defined by a conditional and\nan unconditional generative model. The authors show that using such a classifier\nallows to control a trade-off between IS and FID on the ImageNet dataset. The\nuse of such implicit classifier also provides intuition on the guidance\ninfluence in score-based generative modeling: the model tries to reduce the\nunconditional likelihood while increasing the conditional likelihood.\n\n[1] Song, Sohl-Dickstein, Kingma, Kumar, Ermon, Poole -- Score-based Generative Modeling through Stochastic Differential Equations\n[2] Dhariwal, Nichol -- Diffusion Models Beat GANs on Image Synthesis", "summary_of_the_review": "The main idea of the paper is interesting and the experiments are quite\nconvincing.  However, I feel that the motivation behind this improvement is\nquite superficial. I think that the authors should better motivate their study,\nespecially putting an emphasis on the limitations of classifier guidance in\nscore-based generative models. I will increase my score if the authors address\nmy concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper modifies the conditional diffusion model guided by a classifier, as introduced by Dhariwal & Nichol 2021, by replacing the explicit classifier with an implicit classifier. This implicit classifier is derived under Bayes' rule and combined with the conditional diffusion model. This combination can be realized by mixing the score estimates of a conditional diffusion model and an unconditional diffusion model. A trade-off between sample quality and diversity, in terms of the IS and FID scores, can be achieved by adjusting the mixing weight. The paper is clearly written and easy to follow. However, the reviewers do not consider the modification to be that significant in practice, as it still requires label guidance and also increases the computational complexity. From the AC's perspective, the practical significance could be enhanced if the authors can generalize their technique beyond assisting conditional diffusion models.
Decision: Reject