Year: 2021

Abstract:
Classifier guidance is a recently introduced method to trade off mode coverage and sample fidelity in conditional diffusion models post training, in the same spirit as low temperature sampling or truncation in other types of generative models. Classifier guidance combines the score estimate of a diffusion model with the gradient of an image classifier and thereby requires training an image classifier separate from the diffusion model. It also raises the question of whether guidance can be performed without a classifier. We show that guidance can be indeed performed by a pure generative model without such a classifier: in what we call unconditional guidance, we jointly train a conditional and an unconditional diffusion model, and we combine the resulting conditional and unconditional score estimates to attain a trade-off between sample quality and diversity similar to that obtained using classifier guidance.

heading: 1 INTRODUCTION
text: Diffusion models have recently emerged as an expressive and flexible family of generative models, delivering competitive sample quality and likelihood scores on image and audio synthesis tasks (SohlDickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020; Song et al., 2021b; Kingma et al., 2021; Song et al., 2021a). These models have delivered audio synthesis performance rivaling the quality of autoregressive models with substantially fewer inference steps (Chen et al., 2021; Kong et al., 2021), and they have delivered ImageNet generation results outperforming BigGAN-deep (Brock et al., 2019) and VQ-VAE-2 (Razavi et al., 2019) in terms of FID score and classification accuracy score (Ho et al., 2021; Dhariwal & Nichol, 2021).
Dhariwal & Nichol (2021) proposed classifier guidance, a technique to boost the sample quality of a diffusion model using an extra trained classifier. Prior to classifier guidance, it was not known how to generate “low temperature” samples from a diffusion model similar to those produced by truncated BigGAN (Brock et al., 2019) or low temperature Glow (Kingma & Dhariwal, 2018): naive attempts, such as scaling the model score vectors or decreasing the amount of Gaussian noise added during diffusion sampling, are ineffective (Dhariwal & Nichol, 2021). Classifier guidance instead mixes a diffusion model’s score estimate with the input gradient of the log probability of a classifier. By varying the strength of the classifier gradient, Dhariwal & Nichol can trade off Inception score (Salimans et al., 2016) and FID score (Heusel et al., 2017) (or precision and recall) in a manner similar to varying the truncation parameter of BigGAN.
We are interested in whether classifier guidance can be performed without a classifier. Because classifier guidance mixes a score estimate with a classifier gradient during sampling, classifier-guided diffusion sampling can be interpreted as attempting to confuse an image classifier with a gradientbased adversarial attack. This raises the question of whether classifier guidance is successful at boosting classifier-based metrics such as FID and Inception score (IS) simply because it is adversarial against such classifiers. Stepping in direction of classifier gradients also bears some resemblance to GAN training, particularly with nonparameteric generators; this also raises the question of whether classifier-guided diffusion models perform well on classifier-based metrics because they are beginning to resemble GANs, which are already known to perform well on such metrics.
To resolve these questions, we present unconditional guidance, our guidance method which avoids any classifier entirely. Rather than sampling in the direction of the gradient of an image classifier, unconditional guidance instead mixes the score estimates of a conditional diffusion model and a jointly trained unconditional diffusion model. By sweeping over the mixing weight, we attain a FID/IS tradeoff similar to that attained by classifier guidance. Our unconditional guidance results demonstrate that pure generative diffusion models are capable of synthesizing extremely high fidelity samples possible with other types of generative models.

heading: 2 BACKGROUND
text: We train diffusion models in continuous time (Song et al., 2021b; Chen et al., 2021; Kingma et al., 2021): letting x ∼ p(x) and z = {zλ |λ ∈ [λmin, λmax]} for hyperparameters λmin < λmax ∈ R, the forward process q(z|x) is the variance-preserving Markov process (Sohl-Dickstein et al., 2015):
q(zλ|x) = N (αλx, σ2λI), where α2λ = 1/(1 + e−λ), σ2λ = 1− α2λ (1)
q(zλ|zλ′) = N ((αλ/αλ′)zλ′ , σ2λ|λ′I), where λ < λ ′, σ2λ|λ′ = (1− e λ−λ′)σ2λ (2)
We will use the notation p(z) (or p(zλ)) to denote the marginal of z (or zλ) when x ∼ p(x) and z ∼ q(z|x). Note that λ = logα2λ/σ2λ, so λ can be interpreted as the log signal-to-noise ratio of zλ, and the forward process runs in the direction of decreasing λ.
Conditioned on x, the forward process can be described in reverse by the transitions q(zλ′ |zλ,x) = N (µ̃λ′|λ(zλ,x), σ̃2λ′|λI), where
µ̃λ′|λ(zλ,x) = e λ−λ′(αλ′/αλ)zλ + (1− eλ−λ ′ )αλ′x, σ̃ 2 λ′|λ = (1− e λ−λ′)σ2λ′ (3)
The reverse process generative model starts from pθ(zλmin) = N (0, I). We specify the transitions:
pθ(zλ′ |zλ) = N (µ̃λ′|λ(zλ,xθ(zλ)), (σ̃2λ′|λ) 1−v(σ2λ|λ′) v) (4)
During sampling, we apply this transition along an increasing sequence λmin = λ1 < · · · < λT = λmax for T timesteps; in other words, we follow the discrete time ancestral sampler of Sohl-Dickstein et al. (2015); Ho et al. (2020). If the model xθ is correct, then as T →∞, we obtain samples from an SDE whose sample paths are distributed as p(z) (Song et al., 2021b), and we use pθ(z) to denote the continuous time model distribution. The variance is a log-space interpolation of σ̃2λ′|λ and σ 2 λ|λ′ as
suggested by Nichol & Dhariwal (2021); we found it effective to use a constant hyperparameter v rather than learned zλ-dependent v. Note that the variances simplify to σ̃2λ′|λ as λ
′ → λ, so v has an effect only when sampling with non-infinitesimal timesteps as done in practice.
The reverse process mean comes from an estimate xθ(zλ) ≈ x plugged into q(zλ′ |zλ,x) (Ho et al., 2020; Kingma et al., 2021) (xθ also receives λ as input, but we suppress this to keep our notation clean). We parameterize xθ in terms of -prediction (Ho et al., 2020): xθ(zλ) = (zλ−σλ θ(zλ))/αλ, and we train on the objective
E ,λ [ ‖ θ(zλ)− ‖22 ] (5)
where ∼ N (0, I), zλ = αλx + σλ , and λ is drawn from a distribution p(λ) over [λmin, λmax]. This objective is denoising score matching (Vincent, 2011; Hyvärinen & Dayan, 2005) over multiple noise scales (Song & Ermon, 2019), and when p(λ) is uniform, the objective is proportional to the variational lower bound on the marginal log likelihood of the latent variable model ∫ p(x|z)pθ(z)dz, ignoring the term for the unspecified decoder p(x|z) and for the prior at zλmin (Kingma et al., 2021). If p(λ) is not uniform, the objective can be interpreted as weighted variational lower bound whose weighting can be tuned for sample quality (Ho et al., 2020; Kingma et al., 2021). We use a p(λ) inspired by the discrete time cosine noise schedule of Nichol & Dhariwal (2021): we sample λ via λ = −2 log tan(au+ b) for uniformly distributed u ∈ [0, 1], where b = arctan(e−λmax/2) and a = arctan(e−λmin/2)− b. This represents a hyperbolic secant distribution modified to be supported on a bounded interval. For finite timestep generation, we use λ values corresponding to uniformly spaced u ∈ [0, 1], and the final generated sample is xθ(zλmax). Because the loss for θ(zλ) is denoising score matching for all λ, the score θ(zλ) learned by our model estimates the gradient of the log-density of the distribution of our noisy data zλ, that is θ(zλ) ≈ σλ∇zλ log p(zλ). Sampling from the learned diffusion model resembles using Langevin diffusion to sample from a sequence of distributions p(zλ) that converges to the conditional distribution p(x) of the original data x.
In the case of conditional generative modeling, the data x is drawn jointly with conditioning information c, i.e. a class label for class-conditional image generation. The only modification to the model is that the reverse process function approximator receives c as input, as in θ(zλ, c).

heading: 3 GUIDANCE
text: An interesting property of certain generative models, such as GANs and flow-based models, is the ability to perform truncated or low temperature sampling by decreasing the variance or range of noise inputs to the generative model at sampling time. The intended effect is to decrease the diversity of the samples while increasing the quality of each individual sample. Truncation in BigGAN (Brock et al., 2019), for example, yields a tradeoff curve between FID score and Inception score for low and high amounts of truncation, respectively. Low temperature sampling in Glow (Kingma & Dhariwal, 2018) has a similar effect.
Unfortunately, straightforward attempts of implementing truncation or low temperature sampling in diffusion models are ineffective. For example, scaling model scores or decreasing the variance of Gaussian noise in the reverse process cause the diffusion model to generate blurry, low quality samples (Dhariwal & Nichol, 2021).

heading: 3.1 CLASSIFIER GUIDANCE
text: To obtain a truncation-like effect in diffusion models, Dhariwal & Nichol (2021) introduce classifier guidance, where the diffusion score θ(zλ, c) ≈ σλ∇zλ log p(zλ|c) is modified to include the gradient of the log likelihood of an auxiliary classifier model pθ(c|zλ) as follows:
̃θ(zλ, c) = θ(zλ, c) + wσλ∇zλ log pθ(c|zλ) ≈ σλ∇zλ [log p(zλ|c) + w log pθ(c|zλ)], where w is a parameter that controls the strength of the classifier guidance. This modified score ̃θ(zλ, c) is then used in place of θ(zλ, c) when sampling from the diffusion model, resulting in approximate samples from the distribution
p̃θ(zλ|c) ∝ pθ(zλ|c)pθ(c|zλ)w.
Algorithm 1 Joint training a diffusion model with unconditional guidance Require: puncond: probability of unconditional training
1: repeat 2: (x, c) ∼ p(x, c) . Sample data with conditioning from the dataset 3: c← ∅ with probability puncond . Randomly discard conditioning to train unconditionally 4: λ ∼ p(λ) . Sample log SNR value 5: ∼ N (0, I) 6: zλ = αλx+ σλ . Corrupt data to the sampled log SNR value 7: Take gradient step on ∇θ ‖ θ(zλ, c)− ‖2 . Optimization of denoising model 8: until converged
The effect is that of up-weighting the probability of data for which the classifier pθ(c|zλ) assigns high likelihood to the correct label: data that can be classified well scores high on the Inception score of perceptual quality (Salimans et al., 2016), which rewards generative models for this by design. Dhariwal & Nichol therefore find that by setting w > 0 they can improve the Inception score of their diffusion model, at the expense of decreased diversity in their samples.
Figure 2 illustrates the effect of numerically solved guidance p̃θ(zλ|c) ∝ pθ(zλ|c)pθ(c|zλ)w on a toy 2D example of three classes, in which the conditional distribution for each class is an isotropic Gaussian. The form of each conditional upon applying guidance is markedly non-Gaussian. As guidance strength is increased, each conditional places probability mass farther away from other classes and towards directions of high confidence given by logistic regression, and most of the mass becomes concentrated in smaller regions. This behavior can be seen as a simplistic manifestation of the Inception score boost and sample diversity decrease that occur when classifier guidance strength is increased in an ImageNet model.
Applying classifier guidance with weight w + 1 to an unconditional model would theoretically lead to the same result as applying classifier guidance with weight w to a conditional model, because pθ(zλ|c)pθ(c|zλ)w ∝ pθ(zλ)pθ(c|zλ)w+1; or in terms of scores,
θ(zλ) + (w + 1)σλ∇zλ log pθ(c|zλ) ≈ σλ∇zλ [log p(zλ) + (w + 1) log pθ(c|zλ)] = σλ∇zλ [log p(zλ|c) + w log pθ(c|zλ)],
but interestingly, Dhariwal & Nichol obtain their best results when applying classifier guidance to an already class-conditional model, as opposed to applying guidance to an unconditional model. For this reason, we will stay in the setup of guiding an already conditional model.

heading: 3.2 UNCONDITIONAL GUIDANCE
text: While classifier guidance successfully trades off IS and FID as expected from truncation or low temperature sampling, it is nonetheless reliant on gradients from an image classifier and we seek to eliminate the classifier for the reasons stated in Section 1. Here, we describe unconditional guidance, which achieves the same effect without such gradients. Unconditional guidance is an alternative method of modifying θ(zλ, c) to have the same effect as classifier guidance, but without a classifier.
Instead of training a separate classifier model, we choose to train an unconditional denoising diffusion model pθ(z) parameterized through a score estimator θ(zλ) together with the conditional model pθ(z|c) parameterized through θ(zλ, c). We use a single neural network to parameterize both models, where for the unconditional model we can simply input a null token ∅ for the class identifier c when predicting the score, i.e. θ(zλ) = θ(zλ, c = ∅). We jointly train the unconditional and conditional models simply by randomly setting c to the unconditional class identifier ∅ with some probability puncond, set as a hyperparameter. (It would certainly be possible to train separate models instead of jointly training them together, but we choose joint training because it is extremely simple to implement, does not complicate the training pipeline, and does not increase the total number of parameters.)
We can then apply Bayes’ rule to obtain an implicit classifier as piθ(c|zλ) ∝ pθ(zλ|c)/pθ(zλ). The score of this implicit classifier will then be given by ∇zλ log piθ(c|zλ) ≈ 1σλ [ θ(zλ, c) − θ(zλ)]. Applying classifier guidance with this implicit classifier yields the following modification to the
Algorithm 2 Conditional sampling with unconditional guidance Require: w: guidance strength Require: c: conditioning information for conditional sampling Require: λ1, . . . , λT : increasing log SNR sequence with λ1 = λmin, λT = λmax
1: z1 ∼ N (0, I) 2: for t = 1, . . . , T do . Form the unconditional-guided score at log SNR λt 3: ̃t = (1 + w) θ(zt, c)− w θ(zt) . Sampling step (could be replaced by another sampler, e.g. DDIM) 4: x̃t = (zt − σλt ̃t)/αλt 5: zt+1 ∼ N (µ̃λt+1|λt(zt, x̃t), (σ̃ 2 λt+1|λt) 1−v(σ2λt|λt+1) v) if t < T else zt+1 = x̃t 6: end for 7: return zT+1
diffusion score estimator:
̃θ(zλ, c) = (1 + w) θ(zλ, c)− w θ(zλ) ≈ σλ∇zλ [log pθ(zλ|c) + w log piθ(c|zλ)]. (6)
We then use ̃θ(zλ, c) to sample from our diffusion model as usual, thus producing approximate samples from p̃θ(zλ|c) ∝ pθ(zλ|c)piθ(c|zλ)w. Algorithms 1 and 2 describe training and sampling with unconditional guidance in detail.
Note that in some cases, it may be possible to entirely avoid training an unconditional model. If we know the class distribution and there are only a few classes, we can use the fact that ∑ c p(x|c)p(c) = p(x) to obtain an unconditional score from conditional scores without explicitly training for the unconditional score. Of course, this would require as many forward passes as there are possible values of c and would be inefficient for high dimensional conditioning signals.
It is not obvious a priori that inverting a generative model using Bayes’ rule yields a good classifier that provides a useful guidance signal. For example, Grandvalet & Bengio (2004) find that discriminative models generally outperform implicit classifiers derived from generative models, even in artificial cases where the specification of those generative models exactly matches the data distribution. In cases such as ours, where we expect the model to be misspecified, classifiers derived by Bayes’ rule can be inconsistent (Grünwald & Langford, 2007) and we lose all guarantees on their performance. Nevertheless, in Section 4, we show empirically that unconditional guidance is able to trade off FID and IS in the same way as classifier guidance. In Section 5 we discuss the implications of unconditional guidance in relation to classifier guidance.

heading: 4 EXPERIMENTS
text: We train diffusion models with unconditional guidance on area-downsampled class-conditional ImageNet (Russakovsky et al., 2015), the standard setting for studying tradeoffs between FID and Inception scores starting from the BigGAN paper (Brock et al., 2019).
The purpose of our experiments is to serve as a proof of concept to demonstrate that unconditional guidance is able to attain a FID/IS tradeoff similar to classifier guidance and to understand the behavior of unconditional guidance, not necessarily to push sample quality metrics to state of the art on these benchmarks. For this purpose, we use the model architectures and hyperparameters as the guided diffusion models of Dhariwal & Nichol (2021) (apart from continuous time training as specified in Section 2); those hyperparameter settings were tuned for classifier guidance and hence may be suboptimal for unconditional guidance. Furthermore, since we amortize the conditional and unconditional models into the same architecture without an extra classifier, we in fact are using less model capacity than previous work. Nevertheless, our unconditional-guided models still produce competitive sample quality metrics and sometimes outperform prior work, as can be seen in the following sections.

heading: 4.1 VARYING THE UNCONDITIONAL GUIDANCE STRENGTH
text: Here we experimentally verify the main claim of this paper: that unconditional guidance is able to trade off IS and FID in a manner like classifier guidance or GAN truncation. We apply our proposed unconditional guidance to 64× 64 and 128× 128 class-conditional ImageNet generation. In Table 1 and Fig. 4, we show sample quality effects of sweeping over the guidance strength w on our 64 × 64 ImageNet models; Table 2 and Fig. 5 show the same for our 128 × 128 models. We consider w ∈ {0, 0.1, 0.2, . . . , 4} and calculate FID and Inception Scores with 50000 samples for each value following the procedures of Heusel et al. (2017) and Salimans et al. (2016). All models used log SNR endpoints λmin = −20 and λmax = 20. The 64 × 64 models used sampler noise interpolation coefficient v = 0.3 and were trained for 400 thousand steps; the 128× 128 models used v = 0.2 and were trained for 2.7 million steps.
We obtain the best FID results with a small amount of guidance (w = 0.1 or w = 0.3, depending on the dataset) and the best IS result with strong guidance (w ≥ 4). Between these two extremes we see a clear trade-off between these two metrics of perceptual quality, with FID monotonically decreasing
and IS monotonically increasing with w. Our results compare favorably to Dhariwal & Nichol (2021) and Ho et al. (2021), and in fact our 128 × 128 results are the state of the art in the literature. At w = 0.3, our model’s FID score on 128× 128 ImageNet outperforms the classifier-guided ADM-G, and at w = 4.0, our model outperforms BigGAN-deep at both FID and IS when BigGAN-deep is evaluated its best-IS truncation level.
Figures 1, 3 and 6 to 8 show randomly generated samples from our model for different levels of guidance: here we clearly see that increasing unconditional guidance strength has the expected effect of decreasing sample variety and increasing individual sample fidelity.

heading: 4.2 VARYING THE UNCONDITIONAL TRAINING PROBABILITY
text: The main hyperparameter of unconditional guidance is puncond, the probability of training on unconditional generation during joint training of the conditional and unconditional diffusion models. Here, we study the effect of training models on varying puncond on 64× 64 ImageNet. Table 1 and Fig. 4 show the effects of puncond on sample quality. We trained models with puncond ∈ {0.1, 0.2, 0.5}, all for 400 thousand training steps, and evaluated sample quality across various guidance strengths. We find puncond = 0.5 consistently performs worse than puncond ∈ {0.1, 0.2} across the entire IS/FID frontier; puncond ∈ {0.1, 0.2} perform about equally as well as each other.
Based on these findings, we conclude that only a relatively small portion of the model capacity of the diffusion model needs to be dedicated to the unconditional generation task in order to produce unconditional-guided scores effective for sample quality. Interestingly, for classifier guidance, Dhariwal & Nichol report that relatively small classifiers with little capacity are sufficient for effective classifier guided sampling, mirroring this phenomenon that we found with unconditionally guided models.

heading: 4.3 VARYING THE NUMBER OF SAMPLING STEPS
text: Since the number of sampling steps T is known to have a major impact on the sample quality of a diffusion model, here we study the effect of varying T on our 128× 128 ImageNet model. Table 2 and Fig. 5 show the effect of varying T ∈ {128, 256, 1024} over a range of guidance strengths. As expected, sample quality improves when T is increased, and for this model T = 256 attains a good balance between sample quality and sampling speed.
Note that T = 256 is approximately the same number of sampling steps used by ADM-G (Dhariwal & Nichol, 2021), which is outperformed by our model. However, it is important to note that each sampling step for our method requires evaluating the denoising model twice, once for the conditional θ(zλ, c) and once for the unconditional θ(zλ). Because we used the same model architecture as ADM-G, the fair comparison in terms of sampling speed would be our T = 128 setting, which underperforms compared to ADM-G in terms of FID score.

heading: 5 DISCUSSION
text: Since unconditional guidance is able to trade off IS and FID like classifier guidance without needing an extra trained classifier, we have demonstrated that guidance can be performed with a pure generative model. We confirm that it is possible to maximize Inception scores using classifier-free unconditional guidance (and improve FID score for a small amount of guidance), thus providing evidence that classifier-based sample quality metrics can be improved using methods that are not adversarial against ImageNet classifiers using classifier gradients. Our diffusion models are parameterized by unconstrained neural networks and therefore their score estimates do not necessarily form conservative vector fields, unlike classifier gradients (Salimans & Ho, 2021). Therefore, an unconditional-guided sampler follows step directions that do not resemble classifier gradients at all and thus cannot be interpreted as a gradient-based adversarial attack on a classifier; hence our results show that boosting
the classifier-based IS and FID metrics can be accomplished with pure generative models with a sampling procedure that is not adversarial against image classifiers.
We also have arrived at an intuitive explanation for how guidance works: it decreases the unconditional likelihood of the sample while increasing the conditional likelihood. Unconditional guidance accomplishes this by decreasing the unconditional likelihood with a negative score term, which to our knowledge has not yet been explored and may find uses in other applications.
On the practical side, unconditional guidance is extremely simple to implement and does not complicate the training pipeline of a diffusion model, unlike classifier guidance which requires training an additional classifier model. This model has to be trained on noisy data zλ, so it is not possible to plug in a standard pre-trained classifier.
A potential disadvantage of classifier-free guidance is sampling speed. Generally, classifiers can be smaller and faster than generative models, so classifier guided sampling may be faster than classifierfree guidance because the latter needs to run two forward passes of the diffusion model, one for conditional score and another for the unconditional score. The necessity to run multiple passes of the diffusion model might be mitigated by changing the architecture to inject conditioning late in the network, but we leave this exploration for future work.
Finally, any guidance method that increases sample fidelity at the expense of diversity must face the question of whether decreased diversity is acceptable. There may be negative impacts in deployed models, since sample diversity is important to maintain in applications where certain parts of the data are underrepresented in the context of the rest of the data. It would be an interesting avenue of future work to try to boost sample quality while maintaining sample diversity.

heading: 6 CONCLUSION
text: We have presented unconditional guidance, a method to increase sample quality while decreasing sample diversity in diffusion models. Unconditional guidance can be thought of as classifier guidance without a classifier, and our results showing the effectiveness of unconditional guidance confirm that pure generative diffusion models are capable of maximizing classifier-based sample quality metrics while entirely avoiding classifier gradients. We look forward to further explorations of unconditional guidance in a wider variety of settings and data modalities.
REPRODUCIBILITY STATEMENT
Our model architectures and training hyperparameters are from Dhariwal & Nichol (2021), which can be referenced for implementation details. Hyperparameters related to guidance are are described in Section 4. In Algorithm 1 and Algorithm 2 we provide detailed pseudocode that further describes
our implementation. We plan to open source our code and model checkpoints with the release of the final version of this paper.
ETHICS STATEMENT
Like other machine learning methods, generative models can suffer from bias if applied to data sets that are not carefully curated. As mentioned in Section 5, any guidance or truncation method such as ours will drop sample diversity in favor of quality, and this may introduce more biases. We did not study the influence of our proposed method on this property of generative models, and careful evaluation is advised before using our algorithm in practice.

Review summaries:
In this paper the authors propose an improvement for score-matching based generative modeling [1] resembling low temperature sampling as in GANs or flow-based models. Similarly to [2] they propose to modify the drift function used in the sampling step of the diffusion model by including the gradient of some classifier. However, contrary to [2] the classifier considered by the authors is implicit in the sense that it is purely defined by a conditional and an unconditional generative model. The authors show that using such a classifier allows to control a trade-off between IS and FID on the ImageNet dataset. The use of such implicit classifier also provides intuition on the guidance influence in score-based generative modeling: the model tries to reduce the unconditional likelihood while increasing the conditional likelihood.  [1] Song, Sohl-Dickstein, Kingma, Kumar, Ermon, Poole -- Score-based Generative Modeling through Stochastic Differential Equations [2] Dhariwal, Nichol -- Diffusion Models Beat GANs on Image Synthesis

This work proposed a method to trade-off sample diversity for sample quality in diffusion models, which is termed unconditional guidance. Different from the prior work called classifier guidance (Dhariwal & Nichol 2021) that relies on a classifier for providing the guidance signal, the proposed unconditional guidance mixes the score estimates of a conditional diffusion model and an unconditional diffusion model for a trade-off between sample quality and sample diversity. Experiments on ImageNet 64x64 and 128x128 showed that the proposed model can achieve the claimed quality-diversity tradeoff regarding FID and IS scores.  

The paper belongs to the class of diffusion-based generative models for generating synthetic images using class labels as guidance. It starts with a view of a recent work Dhariwal-Nichol (2021) where a classifier model is trained jointly with the diffusion-based generative model and the score (gradient of log probability) of the classifier is magnified and added to the score of the generative model to increase fidelity and the expense of diversity. In the view, Dhariwal-Nichol's approach resembles adversarial attacks of GAN-based approaches, motivating the key question that the paper addresses: whether we can train a generative model without a classifier but still enjoying the ability to trade diversity for fidelity.  The answer presented in the paper is to replace the (explicit) classifier with an implicit classifier modeled as a conditioned generative model (conditioned on the class), and the weights between the conditioned generative model than the unconditioned generative model are shared by viewing the unconditioned generative model as part of the conditional model but with an additional null class.  Experiments on ImageNet shows that the results compare favourably to Dhariwal-Nichol (2021) and Ho et al (2021), and in some cases slightly outperforming.