Towards Understanding the Mechanisms of Classifier-Free Guidance
Authors: Xiang Li
Rongrong Wang
Qing Qu
Venue: NeurIPS 2025 spotlight ()
Keywords: diffusion model
classifier free guidance
mechanism interpretation
Primary area: deep_learning

Abstract: Classifier-free guidance (CFG) is a core technique powering state-of-the-art image generation systems, yet its underlying mechanisms remain poorly understood. In this work, we first analyze CFG in a simplified linear diffusion model, where we show its behavior closely resembles that observed in the nonlinear case. Our analysis reveals that linear CFG improves generation quality via three distinct components: (i) a mean-shift term that approximately steers samples in the direction of class means, (ii) a positive Contrastive Principal Components (CPC) term that amplifies class-specific features, and (iii) a negative CPC term that suppresses generic features prevalent in unconditional data. We then verify these insights in real-world, nonlinear diffusion models: over a broad range of noise levels, linear CFG resembles the behavior of its nonlinear counterpart. Although the two eventually diverge at low noise levels, we discuss how the insights from the linear analysis still shed light on the CFG's mechanism within the nonlinear regime.

Towards Understanding the Mechanisms of Classifier-Free Guidance
Xiang Li', Rongrong Wang", Qing Qu' 'Department of EECS, University of Michigan, DDepartment of CMSE and Mathematics, Michigan State University, forkobe @ umich.edu, wangronó @ msu.edu, qingqu @umich.edu
Abstract
Classifier-free guidance (CFG) is a core technique powering state-of-the-un image generation systems, yet its underlying mechanisms remain poorly understood. In this work, we begin by analyzing CFG in simplified linear diffusion model, where we show its behavior closely resembles that observed in the nonlinear case. Our analysis reveals that linear CFG improves generation quality via three distinct components: (i) mean-shift term that approximately steers samples in the direction class means, (ii) a positive Contrastive Principal Components (CPC) term that amplifies class-specific features, and (íii) a negative CPC term that suppresses generic features prevalent in unconditional data. We then verify these insights in woold, nonlinear diffusion models: over broad range of noise levels, linear CFG resembles the behavior of its nonlinear counterpart. Although the two eventually diverge at low noise levels, discuss how the insights from the linear analysis still shed light on the CFG's mechanism in the nonlinear regime.
Introduction
Diffusion models [1-4] generate samples from data distribution (x), where IRd by reversing a forward noising process. This forward process, defined in (1), progressively corrupts the clean data until p( (x;; maxx) becomes indistinguishable from Gaussian distribution N(0,, of maxx)
[ pox (xr (rlzoo)Pann (xoo)ooo pd'
plæ; or (f))
=
Following the ofte--of-the-ar EDM framework [4, 5], the forward transition kernel is set to Por (x)) NZ (t) The reverse process can then be expressed probabilistic ODE:
dæ; v. log p(æ; (())dl,
such that p(a;o(()) for every (t)) E (0, m In practice, the function be approximated as Va log p(æ; (t)) = (Do(æ: att) x) (t) where Dø is a deep network-based denoiser with parameter 0 optimized by minimizing the denoising score matching objective [6]:
C= [ (e + t;a(()))
x 1
To sample from conditional distribution p(æ c), the deep denoiser Dø (t), c) receives an auxiliary embedding e specifying the target class or other conditions during training such that conditional sampling can be performed with:
dx, - log (())
where V. p(xp(( (t)) = (Do(#; (t), c) (t). However, the naive (standard) conditional sampling (4) alone often results images with incoherent structures and fail to align well with the
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

target condition [7]. Classifier-free free guidance (CFG) [8] addresses this issue by steering the naive conditional sampling trajectory with a guidance term:
g(x, t) = Ve logp(z)(; - Vv logp(æ;;
(5)
so that (4) becomes:
dx, = - (t) logp(zle; 70( t))
where y 2 O controls the strength of guidance. With properly chosen % CFG substantially improves sample quality, albeit with reduced diversity. Since its invention, CFG and its variants [9-15] have become the backbone that powers the most advanced image generation systems [16-18].
Despite the practical success of CFG, its underlying mechanism remains largely unknown. As shown in [7], the CFG-perturbed reverse trajectory does not correspond to any known forward process, therefore. analyzing the effects of CFG requires case-by-case studies with explicit assumptions on the data distribution. For example, work [19] proves that under an isotropic Gaussian mixture data assumption, CFG boosts classification accuracy at the cost of sample diversity. The work [20] shows that under either 1-D mixtures of compactly supported distributions or 1-D isotropic Gaussian data assumptions, CFG guides the diffusion models towards sampling more heavily from the boundary of the support. Despite providing invaluable insights, these analyses rely on oversimplified assumptions that neglect critical aspects of real data, particularly the covariance structures of natural images. Consequently, it remains unclear how well these theoretical results generalize to diffusion models trained on complex image datasets.
In this work, we pursue deeper understanding CFG's mechanism, focusing on two core questions: (i) What the failure mode of naive conditional sampling, i.e., in what aspect is the generated images subpar compared to the training images? and (ii) how does CFG mitigate this problem?
To answer the first question, show that the naive conditional suffers from lack of class- saas-sppeifiictty issue: images conditioned on different labels often share similar structures and lack distinct class features. We posit that this issue can be partially attributed the covariance structures of different classes being insufficiently distinct. Recent studies [21, 22] observe that over a broad range of noise levels, diffusion models can be unreasonably approximated by the optimal linear denoisers for the multivariate Gaussian distribution defined by the empirical mean and covariance of the training set. Consequently, the data covariance (and particularly its principal components, or PCs) heavily influences the generation. However, as we will demonstrate, different classes can share overly similar covariance structures, resulting in generated images that lack class-specific patterns.
Based on this intuition, we posit that CFG must identify the unique features of the target class. To understand how this is achieved, we study the prototypical setting of the optimal linear diffusion model, where we show that CFG guidance naturally decomposes into three components with distinct effects: (i) a mean-shift term that approximately pushes the samples towards the direction of the class mean, (ii) a positive contrastive principal components (CPC) term that enhances the target class's unique features and (iii) negative contrastive principal components (CPC) term that suppresses the features prominent in the unconditional dataset. Despite the simplicity of the linear model, the linear CFG greatly improves the visual quality of generated samples in a way reminiscent of real-world, nonlinear deep diffusion models, implying that nonlinear CFG share a similar underlying working mechanism. We then investigate how well the insights derived from the linear setting extend to actual diffusion models. We first show that at high to moderate noise levels, linear CFG yields highly similar effects as those of the nonlinear CFG. As noise decreases further and the diffusion model enters highly nonlinear regime, the effects of linear CFG and actual nonlinear CFG begin to diverge. Nevertheless, by interpreting denoising as weighted projection onto an adaptive basis, the insights from linear analysis can still shed light on the CFG's ss mechanism in the nonlinear regime.
Contributions. Our main contributions
follows:
We identify the lack of class caasssppeii issue of naive conditional sampling, linking it to the non- distinctiveness of class covariances. Under a linear model assumption, we show CFG overcomes this issue by amplifying class- specific features, suppressing unconditional ones and shifting the samples in the direction of class mean.
We validate these insights derived in the linear model on real diffusion models, demonstrating that: (i) at high to moderate noise levels, linear CFG closely matches the effects of nonlinear CFG, and (ii) at low noise levels, the insights from the linear analysis can still shed light on the mechanism of CFG in this nonlinear regime.

---

2 Preliminaries
2.1 Optimal Linear Diffusion Model
Suppose (a) has mean la and covariance Σ. Under the constraint that D(x; (t)) is a linear
model (with a bias term), the optimal solution to (3) has the analytical form:
Dr(2; - + U U UZ (æ
-4),
(7)
where Σ = UAUT is the full SVD of the covariance matrix, A = diag " Aa) is the singular values and Ao( diag
With this linear denoiser, the reverse diffusion ODE (2) has the following closed-form expression (see appendix B.1 for the proof):
A ( :. A
X + u (ØT A x²
p)u.:
where 7' is the starting timestep and u, the 7th singular vector of E, which also the ith principal component. Note that in this linear setting, the generated samples are largely determined by the data covariance.
Recent studies [21, 22] show that for a wide range (high to moderate) of noise levels, deep network-based diffusion models can be well approximated by the linear model (7), with la and Σ set to the empirical mean and covariance of the training data. As shown in Figures and 14, the sampling
Oea Sumplng Trelectory
trajectories of the deep diffusion model (EDM) and the linear model share high similarity at high to moderate noise levels. Although the models begin to diverge at lower noise levels- EDM exhibits strong nonlinearity and realistic image content begins to form- form-their final sam- ples still share a similar overall structure. More- over, it is shown in [21] that this similarity is particularly obvious when the deep network has limited capacity or the training is insufficient.
+I1 INL
21.
C
Use
Figure 1: Comparison of Sampling Trajectories. For high to moderate noise levels (a1)) (4, 80)., the linear denoisers well approximate the learned deep denoisers. Though the two models diverge in lower noise reigmes, their final samples still match in overall structure.
Since Dr. oz: (t)) ) is the optimal denoiser for p(a;; (t)) induced by Paata (x) [M, E), sampling with Dr is equivalent to sampling from (H, E). Hence, we refer to Dr. as the linear Gaussian model.
2.2 Contrastive Principal Component Analysis
Principal component analysis (PCA) [23, 24] identifies directions that capture the most variances in a dataset. These principal components (PCs), which are equivalent the singular vectors of the data covariance matrix, are widely used for data exploration and visualization. However, large variance alone does not guarantee that PC captures the unique patterns tied to the dataset; it may instead reflect more general patterns such as foreground-background variations.
To discover low-dimensional structure that is unique to a dataset, the work [25] proposed the contrastive principal component analysis (CPCA), which utilizes a background (or reference) dataset to highlight patterns unique to the largef dataset. Let X and Y be two datasets with covariance matrices Ex and Σy, respectively. For a unit vector v E gd-1 its variances Varx (v) and Vary (v)
in the two datasets are:
Varx = -'xV,
(=)======²>>
(9)
If v corresponds to a unique class speciicc pattern of X, we expect Var (v) > Vary (u) explains significantly more variance in X than in Y. Such directions, called the contrastive principal components (CPCs), can be found by maximizing:
arg max T T(Ex Ey)o.
(10)
vEgd-1
which are essentially the top eigenvectors of Σx Ey. Geometrically, the first k CPCs span the k-dimensional subspace that best fits the dataset X while being as far as possible from Y (see appendix A for details). Conversely, directions for which Varx(v) Vary (v) represent either universal structures shared by both X and Y or meaningless features lying in the null space of the data covariances and are thus discarded as less interesting. Finally, scalar factor can be introduced in (10) to control the strength of the contrast.
3

---

EDM
Linear Gaussian Model
Classt mushroam
chenseburger
coffee
Class: mushroom
rolden rabrieuer
Nalve conditional
samplea
CPG Samples
Figure 2: Efffett of CFG. Left and right figures compare the naive conditional sampling (top rows) versus CFG-guided sampling (bottom rows) for deep diffusion models (EDM) and linear Gaussian diffusion models, respectively. Each grid ceil corresponds to the same initial noise. While naive conditional samples lack class-specific clarity, CFG significantly improves both visual quality and distinctiveness. The conditional linear models are built with class-specific means and covariances. Please refer to appendix D) for more experiment results.
2.3 Posterior Data Covariance
Consider (aa and æ; p(z](z,) denoted by Cov(ar [ov[ 2z). is proportional to the denoiser's Jacobian [26]:
(t)e, where (0, I). Then the posterior covariance of
Cov[2 = a² (t)))(
(11)
where VD(x;; (t)) = ( is the Jacobian of the optimal denoiser D(x; (t)) at input x Bas
Analogous to PCs, the singular vectors of Cov [ar [z,] are the posterior PCs, representing directions of maximal variances of all clean images that could have generated the noisy observation æt, In the
case that Pdata = (H, Σ), we have Cov [ar [a,] = o² (t(t) U", matching VDL(x (t)), the Jacobians of the optimal linear denoiser (7). and independent of æ;. In more general scenarios, one can approximate Cov| [ ar] by computing the network Jacobian at a via automatic diffcrentiation.
3 Analyzing CFG in Linear Model
In this section, we first show that naive conditional sampling often produces low low- qww-quality samples lacking clear spesifie features, which attribute to the con-iistinetivenss of class covariance matrices (section 3.1). We then theoretically analyze how CFG in the context of linear diffusion models alleviates this issue (section 3.2).
3.1 Naive Conditional Generation Lacks Class-Speciffcity
Figure 2(left) (top row) shows samples gener- ated via naive conditional sampling (4). Qual- itatively, these samples often exhibit poor vi- sual fidelity, with incoherent features that blend into the background and the class-specific image structures can be hard to discriminate. More- over, even when conditioned labels, images generated from the same initial noise share high structural similarity, suggesting that naive conditional sampling fails to capture discriminative, class-dependent patterns.
betivesn ciansee tralting conditonal sumpler Cew.--gdddddddrerrooeoeo
different class
Figure 3: Class-to-Clas Similarity. Each cell reports the FID between datasets of classes, built with (i) training data (ii) data generated by naive conditional sampling and (iii) data gener- ated by CFG sampling (refer to appendix D.2 for experiment details and more results.)
To quantify this loss of specsfspec,, we com- pute the pairwise inter-class similarity with the FID metric [27]. For each pair of classes, we construct two datasets X and Y and evaluate the FID between them. As shown in Figure 3, when X and Y are built with images generated with naive conditional sampling, the FID (colored in orange) is consistently lower than when they are built with the training data (colored in blue). Since lower FID indicates higher similarity, this result confirms that compared with the training images, which represent the ground truth data distribution, images generated by naive conditional sampling are less distinguishable across classes.
This issue is especially pronounced in linear diffusion models. As shown in Figure 2(right, top row), samples generated with linear diffusion models built with class-specific means and covariances appear

---

highly similar. From (8), we see that the linear sampling trajectory is governed by the data covariance: 2y a linear combination of PCs, weighted by (i) the correlation u/" (xT- (1) between the mean-
subtracted initial noise Tr and the i-th PC, and (ii) scaling factors V Ax+o"(2) that emphasize leading PCs. Consequently, if class-conditional covariances lack sufficiently discriminative structures (which is indeed true as shown in appendix D.3), generated samples will appear similar regardless of class label. This lack of class-specificity aligns with prior findings [25], which shows that PCs often capture generic image variations (c.g, brreground-bacgrrounn
A
rather than class-specific patterns.
The existence of the class- class-specificity gap implies these models fail to fully capture the higher- order statisties of the training data: if they did, naive conditional sampling, which by construction samples from the target conditional distribution, would already produce high quality samples, and CFG would only distort the target distribution. Linear diffusion models represent an extreme case: due to the linear constraint, they can only learn the first and second-order moments (mean and covariance) of the training data, which despite being fundamental data statistics, cannot capture the rich, nonlinear dependencies necessary for realistic generation. In particular, when covariances across classes share high similarity, samples initialized from the same noise become visually alike regardless of label.
We hypothesize that real-world diffusion models inherit similar limitations. Although nonlinear diffusion models surely learn beyond second-order statistics, as discussed in section 2.1, for high to moderate noise levels, they can be well approximated by linear models, especially under limited model capacity or insufficient training. Indeed, Figure (top row) and Figure 13, 14 demonstrate that linear models reproduce the coarse-graing structures of nonlinear diffusion samples, implying that the covariance structure plays a significant role in shaping the high-level features of the generated samples. These observations reflect well-known simplicity bias, where deep networks favor learning low-order, linearly structured representations over complex, higher-order dependencies [28]. Hence, if the covariances are indistinct across classes, sample quality can be limited even in nonlinear models (see appendix D.3 for more discussion).
As quantitatively shown in Figure 3, CFG significantly increases the inter-class separation: FID (colored in green) between different generated classes rises. Qualitatively, Figure (bottom row) shows that CFG substantially improves both linear and nonlinear models, producing visibly better samples with enhanced class-specific structures. Similar effects of CFG across both linear and nonlinear models motivate us to use a linear model as simplified prototype to analyze how CFG reshapes the generation process and why effective.
3.2 How Linear CFG Leads to Distinet Generations
We now dissect how CFG, in the linear diffusion models, produces samples with distinct class-specific features. Consider two independent optimal linear denoisers, DL. (I);; (t), c) for conditional data and (r,(;;;((() for unconditional data, with means pes Pue and covariances Σ, = U,A,U? and En.. Uu..A... ue respectively. Substituting the optimal linear denoiser (7) into (6), the CFG-guided sampling process can be decomposed into three terms:
dx: (f)) t) + Iepo t) + Imean (t))
(12)
where by letting E., = Undo ((A, UU. and Enc, = Uoc UTo each term takes the following form: (i) Sa(ate () = न (E,t T) - Me), (ii) gepo L) नं (Ec,t - Euc,)) pe),
and (iii) Imean (t) नo. -E Eue
Muc). (Mc)).
Here, fa t) is the standard conditional score, and Vepe(Es:) plus Imeon (t) form the CFG guidance (derivation for the decomposition provided in appendix B.2). Let A (8) be the eigen
decomposition of E.,t Euc.o whose spectrum contains both positive and negative eigenvalues (see Figure 18), Pepe (x,, t) can be split accordingly into positive and negative CPC components:
(2( (Vo (Vo(o. Aoto, Vola (xx Mc) = 02(1) Σ
S (of,
(0.. -Mo)),
(13)
positive CPC guidunce
0² (Vo(),- Sow Volo, M2) Mc) = 7 Σ
"i,00. (x,, - -Mo)),
(14)
[2(1)
negative CPC guidance
5

---

where Va(t).+ and Va(t), contain eigenvectors 44,i and v-.: corresponding to positive and negative eigenvalues +t) and Ar(),- respectively. As discussed in section 2.3, Ee, and Eue, are up to a scaling factor (1) equivalent to the conditional and unconditional posterior covariances of Pdata (x)c) (pte Σe) and Pdata (x) = ( Euc). Hence, Va(e) are the CPCs which contrast between X ~ Pdata ( ( c) and Y ~ (x)a(()/=)) Specifically, Va(t).+ captures directions of higher conditional variance (class-specific features), while Vo(t).- captures directions of higher unconditional variance (features more prevalent in the unconditional data).
Distinctive Effects of the CFG Components. Figure 4(a) shows that for both nonlinear (EDM) and linear models, CFG significantly enhances the characteristic pattern-a person holding a fish-of the "tench" class from ImageNet [29]. Next, we isolate the roles of each CFG term by selectively enabling only one at time within the linear model. In the following discussion, we omit the negative sign in (1)) since the ODE runs backward time:
The positive CPC term (13) projects i.e., the eigenvectors 04,7 associated with positive eigenvalues, with each component scaled by its eigenvalue At,; and the guidance strength 7. Since d4.5 2 O, (13) is added to Lt, i.e., the components of at, - - that align with the positive CPCs, which represent the class-specific features, are amplified. Figure 4 (b) (second column) show the first 25 positive CPCs of Σ, Eue and the resulting samples. Compared to the conditional PCs of the dataset, the positive CPCs better capture the unique patterns of the class, which emerge visibly in the generated images.
pe onto the subspace spanned by the positive CPCs,
Similarly, the negative CPC term (14) projects H; - He onto the negative CPC directions v-,6- Since A_, < O, these components are subtracted from T,, suppressing features associated more strongly with the unconditional data. Figure 4 (b) (third column) shows the first 25 negative CPCs and the resulting generations. Although visually less interpretable, these directions represent common but target-class-irees irrelevan features the unconditional data. Suppressing them reduces background clutter and irrelevant content, making class-relevant structures more salient.
In the context of linear diffusion model, it can be shown that (see proof in appendix B.2):
Imean (t) = -
porl( (1) - V log po:
(15)
Thus, the Mean-shift term gmean (t) can be interpreted as the beababbblity-eiihtte average the steepest ascent direction that maximizes the difference (log- -igeliidl ratio) of the noise-mollified conditional and unconditional distributions. Note that when (t) is large, where I - Euc,t Re I, Imean (t) approximately shifts x, in the direction of μlc puc. which the difference between conditional and unconditional mean. As (l) decreases, I Enc, progressively shrink the components μ. Mhe lying in the column space of Un. (the covariances of image datasets are typically low-rank), while preserving its energy in the null space.
Figure 4(b), fourth column, shows that the mean-shift term enhances the structure of class mean in the generated samples. However, unlike the positive CPC term, Imean()) is independent of ær. thus producing more homogeneous samples with reduced diversity.
Analytical Solution to the CFG Trajectory. To better understand the distinct effects of the CFG components, we aim to examine the global solution to the linear ODE system (12). However, the variables in the general solution of (12) are coupled and difficult to interpret. To obtain a more tractable expression, we follow [30, 31] to make the following assumption:
The covariance matrices E. and Enc are simultaneously diagonalizable, iee. Euc U.A.cU. where U. E Hd are the singular vectors (principal components) of the conditional covariance. Here Aue is not necessarily ordered by the magnitude of the singular values.
Assumption
Assumption 1, known as the Common Principal Components Assumption is widely applied to analyze structural relationships across data groups. Under this assumption, the relative importance of the i-th principal component uee in the conditional and unconditional datasets is determined by the relative magnitudes of its associated singular values. If Xe, > Aue,5, then ue,, is the positive CPC as it captures more variance in the conditional distribution, while X < Ame,i implies that U is more relevant to the unconditional distribution; therefore, it is a negative CPC.
Although Vote) depends on (t). over wide range of noise levels (especially high ones), it remains close the eigenvectors of Σ. Euc. We provide its full evolution across time in Figure 19.

---

of (K Mell and (Paa Pc]
Nahre Slonal surples
OTO srrpas
Natre www.craeaerrrraa
Usear Gieesien that Caluens
maper Reperta ueser
Examrel
(a) Qualitative Effects of CFG
Univariate Distribution of of De. Pell and af Ho
Univariata Distribution of (Pfe Mic--Me Maz and Pac) (Tata Rc)
(b) Decomposing the effects of CFG
(c) Quantitative Effects Components
Figure 4: Distinct effects of different CFG components. (a) CFG substantially enhances class- specific features (in both EDM and linear diffusion). (b) Top row: PCs, positive/negative CPCs, and N. Muc- Bottom row: generated samples when each component is applied in isolation. (c) dineniina densities of generated samples after projection onto key directions. The left column corresponds to the linear diffusion model, whereas the right column corresponds the EDM model. Top row: project onto leading positive CPC. Middle row: project onto negative CPC. Third row: project onto the mean-shift direction. Here only plot the resulting histograms for the first positive and negative CPCs but the same patterns hold for subsequent CPCs. For experimental details and more results, please refer to appendix E.3.
Theorem 1. Under Assumption 1, the solution to the linear CFG process (12) is:
de, oz " Mo)ucs + /U.B. (Me pue)s
Σ le(Ae,s )uc,e)
V +
where h Ae Nue,s) (e t and B., diag()o(0),...., bo(t),a) has diagonal ( tocn+t=21)
entries bo()) depending only on Auc,es de, and (t).
The proof is postponed appendix B. Compared to the solution of the standard conditional sampling (8), the CFG guidance introduces the following two effects:
CPC guidance (z () introduces an additional scaling factor h f((e,,i Auc. ) for each com- ponent Ur,; æt Since h( (AAr,i, Auc.s) Z 1 only Ao,i 2 Aue,i the positive CPCs are enhanced. Conversely, the negative CPCs are suppressed. The guidance strength 7 serves as an additional control over the degree of enhancement or suppression.
Mean-shift guidance term Jreen (4) shifts æ, by B., U (n.- -ue), a direction determined by the class-conditional mean offset f4 p've- Crucially, this shift is independent of the initial noise ær (and intermediate state x2) and is thus applied consistently to all samples, promoting canonical class features but reducing diversity.
Empirical Verification. In Appendix B.3, we provide empirical validation of Theorem I using a 2D synthetic dataset that satisfies Assumption Here, we further verify the CFG's effects of enhancing (suppressing) CPC components and shifting samples towards the mean-shift direction in natural image datasct through the following experiment:
For a chosen class, generate 1,000 samples using naive conditional sampling (denoted by x;)) and 1,000 samples using CFG (denoted by Feig), and center both sets by subtracting the class mean Pe-
Project each sample onto the positive CPCs (denoted as 04), the negative CPCs (denoted by v-), and the mean-shift vector (denoted by p. Hue) obtain series of univariate distributions. The above experiments are conducted on both linear and nonlinear (EDM) diffusion models. The resulting univariate distributions are shown in Figure 4(c). Compared with naive conditional sampling,
7

---

Linear Regime
Class: cheeseburger
Nonlinear regime
Pgy 20.24
16:44
A.
(6) Effects of nontinnee CFO at individust timestop
0i Effects linea CPO at otindividual timestoop
Figure 5: Linear-to-nonlinear transition in diffusion models. (a) and (b) compare nonlinear CFG and linear CFG applied to a deep diffusion model (EDM). The leftmost column shows unguided samples; subsequent columns show final samples when guidance is applied only specific noise level, with y = 15 (See Figure 25 for more examples).
CFG shifts probability mass toward higher projection values along the positive- and mean-shit directions, and toward lower values along the negative-CPC direction, indicating that the first two are amplified whereas the third suppressed.
4 Investigating CFG in Nonlinear Models
We now explore how the findings from the linear analysis extend to real-world diffusion models. Recent studies [32, 21, 22] show that diffusion models transition from linear regime to a nonlinear regime as the noise level decreases. In the linear regime, where (t)) is large, the learned diffusion denoisers Do can be well approximated by the optimal linear denoiser D1 (7) (see both qualitative and quantitative verification appendix F.1). As (() decreases, the diffusion model enters the nonlinear regime where Do diverges from Dr.. Interestingly, this Aieea-to-nnonliee the coarse-to-fi effects CFG. As shown in Figures 5 and 25, in the linear regime, linear and nonlinear CFG produce similar effects, substantially reshaping the global structure of the samples. In contrast, in the nonlinear regime, nonlinear CFG primarily refines local details while preserving the overall structure, leading to different effects as those of linear CFG. This coarse-to- tarreeto-ffne fine transition motivates our separate analyses of CFG behavior in each regime.
transition correlates with
4.1 CFG in the Linear Regime
Figure 6 illustrates the effects of separately applying (i) nonlinear CFG, (ii) linear CFG, (iii) mean-shift guidance, (iv) positive CPC guidance, and (v) negative CPC guidance within the linear regime of EDM over a broad range of 7. As expected, linear CFG² produces re- sults that closely match those of nonlinear CFG, both significantly altering the overall struc- tures of unguided samples. Notably, decomposing linear CFG provides further insights: Mean-shift guidance dominates CFG in the linear regime. As shown in Figure 6(a), qualita- tively, mean-shift guidance alone replicates the effects of both linear and nonlinear CFG. Con- sistent with this observation, FDDINOV2 scores confirm that the mean-shift term is the main contributor to CFG's overall behavior. Because mean-shift term is independent of the sampling trajectory, it can reduce sample diversity. As shown in Figure 6(b), mean-shift guidance im- proves generation quality only within limited range of 7, after which further increases in y degrade FDDINOW2 scores, reflecting loss of diversity.
36
Figure 6: Effects of CFG in Linear Regime. Each row in (a) demonstrates the impact of dif- ferent guidance types applied to EDM within the linear regime (specified in the subtitles), with vary- ing guidance strength 7. (b) shows the FD scores computed over 50,000 samples. The re- ported values are relatively high because the scores are computed separately per class, which often has limited number of training images. It is well known that FDDINOV2 scores can appear inflated when the reference dataset size is small.
CPC guidance also improves generation qual- ity. Although overshadowed by the mean-shift term, applying CPC guidance independently of- fers notable benefits as well. Qualitatively, posi-
Nott that the "linear CFC" here differs from the "linear CFG" in section 3, where both the naive conditional score and the cfg guidance are linear. In contrast, the linear guidance in this section, along with its components, is applied wolld deep diffusion model.
8

---

tive and negative CPC terms preserve the global structure of unguided samples while refining existing features, remaining effective over a broader range of y, Moreover, CPC guidance sometimes mitigate the artifacts introduced by the mean-shift term, such as color oversaturation in the golden retriever example at y = 20. Lastly, we note that the effects of CPC guidance can vary by class. As shown in Figure 30, negative CPC term improves DDDINO scores for "golden retriever" but has minimal effect on "sports car". These findings are verified on 10 classes, with additional results presented in appendix F.2, Figures 27 to 30.
4.2 CFG in the Nonlinear Regime
In the nonlinear regime where o(t) is small, as shown in Figure 5(b), the effects of linear CFG diverge from those of the actual nonlinear CFG. By Tweedie's formula, the CFG guidance (5) be
expressed as g( 1) (tee))rt))))-P(((ee) where D(æ; (f), c) and o denote the optimal
o²(?)
conditional and unconditional denoisers minimizing (3). Unlike in the linear setting, these denoisers do not admit foosed-frmm expressions in the nonlinear regime, making analytical study difficult.
Nevertheless, when denoisers are parameterized by deep networks with no additive 'bias' parameters, their input-output mappings are locally piecewise linear [33, 34), satisfying:
o(x;o(t).c) = (aD(z;a()).))) x, ((): = V=D(zz;(t)) x,
(16)
where V=D( (t)) c) and Di a (t)) are the local Jacobians of the denoisers. In this case, the CFG guidance becomes (V. (V. D(asa(())) V. D(mna()))) which shares a similar form as (epe t)
(2())
defined under the linear setting in (12), since E., E = V Di ((),, V Dr. (æ;; o (4)). where PL is the optimal linear denoiser. Thus, the guidance can again be decomposed into positive and negative CPC components, enhancing the former and suppressing the latter. The key distinction from the linear setting is that here, the CPCs are adaptive to the current input æ.
The bias-free denoisers belong to the broader class of pseudo-linear denoisers [35, 36), which ad- mit the form ((=; (t)) = W(:; (t))ax where W(x; (t)) is symmetric and input-dependent Importantly, it is shown in [35] that if the origin is a stationary point of the log-density, i.e., V loggo((ee p(æ;; (()) (-))))]m -
O, then the optimal denoiser must possess such piecewise linear structure. Even if the diffusion models are not bias-free and the locally linear property does not hold exactly, (16) still serves
reasonable proxy. As discussed in section 2.3, the Ja- cobian (ix; is proportional to the posterior covariance. Its leading singular vectors capture the dominant structures shared by all plausible clean images corresponding to the noisy input æ, while directions associated with near-zero singular values span a null space irrelevant to the image content. Hence, (16) performs a weighted projection onto the subspace encod- ing the most informative image structures effectively functioning as valid denoising operator. Indeed, as shown in Figure (c)) both con- ditional and unconditional Jacobians effectively denoise the input, although their outputs appear brighter and sharper than those from the actual denoisers in (d)- Comparing Figure 7(b) and (c), we find that the conditional and uncon- ditional Jacobians yield denoised outputs with similar global structure, which implies both cap- ture the generic structure of the current sample. However, the conditional Jacobian additionally preserves finer, class-specific details. A similar Figure 7(d) and (e).
Figure 7: Denoising Results. (a) Noisy input im- age. (b)-e) show the denoised outputs generated with (i) conditional Jacobian, (ii) unconditional Ja- cobian, (iii) actual conditional denoiser, and (iv) actual unconditional denoiser, respectively.
pattern holds for the actual denoisers shown in
For guidance purposes, our goal to selectively enhance these fine, class-specific details that the conditional denoiser captures but the unconditional one does not. Achieving this requires identifying directions that encode class-dependent information from those represent generie structures. Empirically, as shown in Figure 8, the following guidance, inspired by the positive CPC guidance (13), can lead to similar effects as the actual nonlinear CFG, sharpening image details:
o2(1) Ed+. U+t (uT, Do(x.oo(t)))))))
(17)
where X4.5 and u4ts denote the positive eigenvalues and eigenvectors of (() VDo(x;;o((t) Unlike (13), this guidance applies projection to the denoiser's output rather than the

---

noisy input Ft, which we find leads to better qualitative results. For comparison, we also test the following non-selective guidance that enhances all conditional posterior PCs:
Σ Nos
Mas thss (u., (o(;)));))(()))))))
(18)
where Ae,5 and Uee are the singular values and vectors of (D)( c). As shown in Figure this approach fails to improve image quality and frequently produces images with oversaturated colors, indicating that not all conditional posterior PCs correspond to class-specific features- effective guidance must selectively amplify only those that do.
We note that our heuristic guidance serves conceptual approximation and may not always perfectly align with actual CFG behavior; in practice, the actual nonlinear CFG yields more stable and consistent results. Due to the black-box nature of deep networks, fully characterizing this mechanism remains challenging, and we regard this as an important direction for future research.
Figure 8: Effects of CFG in the Nonlinear Regime. Different guidance methods, each with a fixed strength of y 15, are applied at individual timesteps in the nonlinear regime. Each image shows the final output when guidance is applied solely at the timestep indicated at the top. Note that (17) matches the effects of CFG by enhancing finer image details, whereas (18) does not improve generation quality. For additional experimental results, please refer to Figure 36.
Discussion and Conclusion
The experiments in the main-text are conducted extensively using the EDM-1 model [4], which operates directly in pixel space with 64 x 64 resolution. In appendix G, we present complementary results on the EDM-2 [5] latent diffusion model, which generates images at 512 x 512 resolution.
The key insight of this work is that CFG enhances generation quality by amplifying class-specific features while suppressing generic ones. In the linear setting, this effect emerges from the interplay of three guidance components. Different from previous works which mainly focus on analyzing isotropic Gaussian distributions, our study probes the covariance structures of image data, revealing that salient class-specific features emerge from contrast between class covariances.
Although our analysis is based on linear diffusion model (Gaussian data) assumption, the results remain noteworthy since: (i) CFG significantly enhances the generation quality of linear diffusion models, making the linear setting meaningful stand-alone testbed for studying CFG and (ii) real- world diffusion models can be well-approximated by their linear counterparts for a wide range of noise levels. We note that the dynamics of linear setting is by itself complex: an interpretable solution to the linear reverse ODE is unattainable unless additional assumptions are imposed on the covariance structures (e.g., the common principal components assumption). A natural next step is to extend the analysis to Gaussian mixtures. We have made some initial attempts in appendix H, showing that CFG guidance in the Gaussian mixture setting can be decomposed in similar manner as the single Gaussian case.
We believe our findings open several promising directions for future research. First, the observed lack of class-specificity issue implies the current training procedures for diffusion models remain suboptimal. This highlights the need for developing principled training objectives that explicitly encourage the model to learn class-specific patterns. Second, beyond the context of CFG, PCA has been widely utilized for extracting visual features or semantic concepts from diffusion models [37- 39]. Our results suggest that extending these approaches with Contrastive PCA can be a promising next step for more controllable and interpretable generation.
10

---

Data Availability Statement
The code and instructions for reproducing our experiments will be released at: https://github. theeMMechaismaa of of--s--f--aassfieeeeee Pwwe-reeedaaee
git.
Acknowledgment
We acknowledge funding support from NSF CCF-2212066, NSF CCF- 2212326. NSF IIS 2402950, and ONR N000142512339, This research used the Delta advanced computing and data resource which is supported by the National Science Foundation (award OAC 2005572) and the State of Illinois. Delta is joint effort of the University of Illinois Urbana-Ciin -haamapaa and its National Center for Supercomputing Applications(40) We thank Mr. Yixiang Dai for fruitful discussions and valuable feedbacks.
References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances nearul information processing systems, 33:6840-6851, 2020.
[2] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Reprexentations, 2021.
(3] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[4] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
[5] Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 24174-24184, 2024.
[6] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661-1674 2011.
[7] Arwen Bradley and Preetum Nakkiran. Classifier-free guidance is predictor-corrector.. arXiv preprint 240i 09000, 2024.
[8] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[9] Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1462-7471, 2023.
[10] Tuomas Kynkäänniemi. Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in limited interval improves sample and distribution quality in diffusion models. arXiv preprint 4rXiv:2404.07724, 2024.
[11] Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a bad version of itself. arXiv preprint arXiv:2406.025000 2024.
[12] Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M Weber. Cads: Unleashing the diversity of diffusion models through condition-anneall sampling. In The Twelfth International Conference on Learning Representattions 2024.

---

[13] Seyedmorteza Sadat, Manuel Kansy, Otmar Hilliges, and Romann M Weber. No train- ing, no problem: Rethinking classifier-free guidance for diffusion models. arXiv preprint arXiv:2407.026677 2024.
[14] Hyungjin Chung, Jeongsol Kim, Geon Yeong Park, Hyelin Nam, and Jong Chul Ye. Cfg++: Manifold-constrainee classifier free guidance for diffusion models. arXiv preprint arXiv:2406.08077) 2024.
[15] Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-attention guidance. In European Conference on Computer Vision, pages 1-17. Springer, 2025.
[16] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
[17] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
[18] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 2022.
[19] Yuchen Wu, Minshuo Chen, Zihao Li, Mengdi Wang, and Yuting Wei. Theoretical insights for diffusion guidance: A case study for gaussian mixture models. In Forty-first International Conference on Machine Learning, 2024.
[20] Muthu Chidambaram, Khashayar Gatmiry, Sitan Chen, Holden Lee, and Jianfeng Lu. What does guidance do? fine-graines analysis in simple setting. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[21] Xiang Li, Yixiang Dai, and Qing Qu. Understanding generalizability of diffusion models requires rethinking the hidden gaussian structure. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[22] Binxu Wang and John Vastola. The unreasonable effectiveness of gaussian score approximation for diffusion models and its applications. Transactions on Machine Learning Research, 2024.
[23] Karl Pearson. Liii, on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11):559-572, 1901.
[24] Harold Hotelling. Analysis of complex of statistical variables into principal components. Journal of educational psychology. 24(6):417, 1933.
[25] Abubakar Abid, Martin Zhang, Vivek K Bagaria, and James Zou. Exploring patterns enriched in dataset with contrastive principal component analysis. Nature communications, 911:2113, 2018.
[26] Hila Manor and Tomer Michaeli. On the posterior distribution in denoising: Application to uncertainty quantification. In The Twelfth International Conference on Learning Representations, 2024.
[27] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[28] Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. Advances in neural information processing systems, 32, 2019.
[29] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEFE conference on computer vision and pattern recognition, pages 248-255 Iece, 2009.
12

---

[30] Bernhard N Flury. Common principal components in k groups. Journal of the American Statistical Association, 79(388):822-895 1984.
[31] Bernhard Flury. Common principal components related multivariate models. John Wiley & Sons, Inc., USA, 1988.
[32] Gabriel Raya and Luca Ambrogioni. Spontaneous symmetry breaking in generative diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[33] Zahra Kadkhodaie, Florentin Guth, Fero Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representations. In The Twelfth International Conference on Learning Representations, 2024.
[34] Sreyas Mohan, Zahra Kadkhodaie, Eero P Simoncelli, and Carlos Fernandzz-Granda Robust and interpretable blind image denoising via bias-free convolutional neural networks. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
[35] Peyman Milanfar and Mauricio Delbracio. Denoising: A powerful building-bloce for imaging, inverse problems, and machine learning. arXiv preprint arXiv:2409.06219, 2024.
[36] Yaniy Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal Imaging Sciences, 10(4):1804-1844, 2017.
[37] Yong-Hyun Park, Mingi Kwon, Jaewoong Choi, Junghyo Jo, and Youngjung Uh. Understanding the latent space of diffusion models through the lens of riemannian geometry. Advances in Neural Information Processing Systems, 36:24129 1244119--1422, 2023.
[38] Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. Freecontrol: Training- spatial control of any text-to-image diffusion model with any condition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7465-7475, 2024.
[39] Rohit Gandikota, Zongze Wu, Richard Zhang, David Bau, Eli Shechtman, and Nick Kolkin. Sliderspace: Decomposing the visual capabilities of diffusion models. arXiv preprint arXiv:2502.01639, 2025.
[40] Timothy J Boerner, Stephen Deems, Thomas R Furlani, Shelley L. Knuth, and John Towns. Access: Advancing innovation: Nsf's advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Research Computing 2023: Computing for the Common Good, pages 173-176. 2023.
[41] Didong Li, Andrew Jones, and Barbara Engelhardt. Probabilistic contrastive principal compo- nent analysis. arXiv preprint arXiv:2012.09977, 2020.
[42] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. 2009.
13

---

Appendices
Contents
1
Introduction
2
Preliminaries
2.1
Optimal Linear Diffusion Model
2.2
Contrastive Principal Component Analysis
2.3
Posterior Data Covariance
3
Analyzing CFG in Linear Model
3.1
Naive Conditional Generation Lacks Class-Specificity
3.2 How Linear CFG Leads to Distinct Generations
Investigating CFG in Nonlinear Models
4.1 CFG in the Linear Regime
4.2 CFG in the Nonlinear Regime
5 Discussion and Conclusion
10
Contrastive Principal Component Analysis
15
Analytical Solution the Reverse Diffusson ODE
16
B.1 Naive Diffusion Reverse ODE
16
B.2 CFG-Guided ODE
17
B.3 Empirical Verification on Synthetic Data.
19
C
Constructing Linear Denoisers
19
D Naive Conditional Generation Lacks Class-Specificity
20
D.1
Qualitative Results
20
D.2
Quantitative Results
20
D.3
Covariance Matrices of Different Classes Lack Class- Shess-Specifictty
25
Mechanism of Linear CFG
25
E.1
Mean-Shift Guidance
25
E.2
CPC guidance
26
E.3
Distinct Effects of the CFG Components
27
CFG in Nonlinear Deep Diffusion Models
34
F1 Linear to Nonlinear Transition in Diffusion Models
34
F2 CFG in the Linear Regime
34
14

---

F.3 Mean-Shifted Noise Initialization
36
F.4 CFG in the Nonlinear Regime
36
Experimental Results on Latent Diffusion Models
46
H
CFG in Gaussian Mixture Model
46
I Computing Resources
48
A Contrastive Principal Component Analysis
Principal component analysis (PCA) finds the features that explain the most variances in the dataset, however, features with high variance do not necessarily correspond to distinct patterns of the target class.
To discover low-dimensional structure that unique dataset, the work [25] proposes the con- trastive principal component analysis (CPCA), which includes a background (or reference) dataset to highlight patterns unique to the targer dataset. Let X and Y be two datasets with covariance matrices Ex and Σy, respectively. For unit vector v E gd- its variances Var (v) and Vary (v) in the two datasets are:
Varx (v) = Exu,
Vary = vJ Eyo.
(19)
If corresponds to unique class saes-speccfiic pattern of X, expect Varx >> Vary (v), i.e. explains significantly more variance in than Y. Such directions, called the contrastive principal components (CPCs), can be found by iteratively solving:
arg max v" (Ex Ey)e,
(20)
vES4-1
where, at each iteration, the resulting v is subtracted from Σx Ey. These directions are essentially the eigenvectors of E(X) Σ(Y). Conversely, directions e for which Varx (v) z Vary (v) represent either universal structures shared by both X and Y or meaningless features lying in the null space of the data covariances-ant and are thus discarded less interesting.
Geometric Interpretation of CPCA. Geometrically, the first k CPCs span the k-dimensional subspace that best fits the dataset X while being far as possible from Y [41]. This is proved by the following theorem:
Theorem 2.
Without loss of generality, assume px (æ) and pv (y) have zero means (i.e., the data is centered). Then the following objective is equivalent to (20):
arg min Em-px BE
|| vv x012 Eysor 11 vo v112
(21)
Proof:
arg min Ex 1/ vo 2012 vE
-0n-----en Ryyoy 111 vo vo
arg min Eg (x -
xo vot Enyyy - y wo v)
vCS
arg min E_ - x vv x) E. (-y² vou y)
wES#-1
arg min - (x T vv voTTT x) + E, ( vo yT) y)
ve$4-1
arg max E, (x vv- x)) Ey ( vv ) vesu-1
arg max of TT (xx ) - Ey (yy vegd-1
arg max x (Ex - Ey)u. vggd
This proof is adapted from [41].
15

---

B Analytical Solution to the Reverse Diffusion ODE
In this section, we examine the solutions both the naive reverse diffusion ODE (2) and the CFG- guided reverse diffusion ODE (6) in the context of linear diffusion models. We follow the EDM formulation [4], which uses time schedule (t) = t. Notice that this same schedule also used by the well-known DDIM sampler [2].
B.1 Naive Diffusion Reverse ODE
We begin by analyzing the diffusion ODE (2) with no guidance applied. The proof below is borrowed from [22].
Let la and E be the mean and covariance of Past( respectively. Suppose UAUT is the full SVD of Σ with U E Bddd being orthonormal and A = diag(Aı, Ad) contains the singular values. For image datasets, the covariance is often low-rank implying some singular values are Under the constraint that D(æ;ar (t)) is linear (with a bias term), the optimal solution to (3) has the closed-form [21]:
D( (1)) = p UA (1) U (xx p4),
(22)
where Ao(t) diag A1
( This optimal linear solution obtained by setting
the derivative of (3) with respect to the weight and bias to zero, leveraging the fact that the objective
convex under the linear constraint.
Following the EDM framework, this optimal linear denoiser yields the sampling trajectory for (2) as:
dæ -V log (2;
(23)
dx (I- ---A U-AAUS xx H²) H)
do
(24)
U(I A AT) (x 11) do, 3 d(x - p1) where we omit the subscript / for simplicity.
(25)
Define (x( un (æ: 14) for k (1,...d) we have:
dcx(a)= =
du + oz 1oz (o))o
o
(26)
de (a)
(o) d + tz20o
(27)
Integrating both sides (27), we get:
d (o))) = d dlog v λ + 0
(28)
+ cx( A + oCC.
(29)
where C' is the integral constant. Using the initial condition cx(oT) (eT 1)), we have:
= u (xT 1)
(30)
Av + oo
d + 0²
(o(
u
T (xT - 11)
(31)
d + of
A + of
u (IT 11) u
(32)
b + oz
where the last equality holds because x; f4 ビ Ch (o,
Notice that the generated samples are primarily determined by the data's covariance structure. However, since the covariance may not capture the most distinctive features of a specific class, the resulting images often lack sufficient class specificity.
16

---

B.2 CFG-Guided ODE
To apply CFG, we need two separate models corresponding conditional and unconditional data respectively. Let pe: Mue be the means of the conditional and unconditional data, and let Σ. U.A.U. Luc Uue be their corresponding covariances, where A. diag(X..1," Ac,a) and Auc diag (au,(,,,,1."". Auc.a). Then the conditional and unconditional optimal linear denoisers take the following forms:
(i( Me UcAco U? (a Mo))
(33)
a1. (1)) Mue + U UnoAuc,o Une ue (x - pua),
(34)
Then the CFG sampling trajectory (6) can be expressed in terms of the optimal linear denoisers:
dx dx;= = -o(()( Dr. (2:;; a (1), c) at + Dr. (r, (I), c) - Dr. (2: o(1)
dt
(35)
(t)
(²
1
(U. (1)) Uº [x Me)dt
(36)
(tt)
? (U (U,Ao()). U² - UueAr(t) U (Ee Me)dt
(37)
2 (I Une Und (H Mua)dt
(38)
(tt)
where (36) is the naive conditional score while (37) and (38) together form the CFG guidance direction. Note that under the setting of linear diffusion model, we have
o(); N(Hae Eve + o())),,
(39)
p(alc;((t)) N(r., Eg + ot))),
(40)
V x logp(z;; (Euo + +t)(t) (0)1)~(((r)) (Muo x),
(41)
V = logp(z/c, (E. + +o² 0 x))
(42)
E Bes-(ale (og)(z));
v logp(æ;; (())) (Eno +o o²((() (o:0)~((pp)) (He pmo)
(43)
o2(t) - Euc,t) (I-Suc.
Mac)
(44)
Therefore, have:
Imonn (1) 7 e (x) V log log Va logp(a;(((1)
(45)
pp= (t) log p(zle
(46)
p(z;o(t))
which implies the mean-shift guidance term can be interpreted as the probability-weighee average of the steepest ascent direction that maximizes the log-likelihood ratio of the noise mollified conditional and unconditional distributions.
Since (35) is a first-order non-homogeneous differential equation, its closed-form solution can in principle, be expressed through integrals. However, these integrals cannot be explicitly evaluated or decoupled in the general case. To obtain tractable, interpretable solution, we must impose additional assumptions on the structures of E. and Euc. Therefore, we make the following assumptions:
The covariance matrices E, and Ene are simultaneously diagonalizable, i.e., Enc U.A. where U. E Bd are the principal components (singular vectors) of the conditional data. Here Anc is not necessarily ordered by the magnitude of the singular value.
Assumption.
This is well-know analyze structural relationships across data groups Under this assumption, the relative importance of the h principal component u in the conditional and unconditional datasets is fully determined by the relative magnitudes its associated singular values:
the Common Principal Components assumption [30, 31], widely utilized to
If )e,k Auakk then u,, explains more variance in the conditional dataset than in the unconditional dataset, i.e., it is more distinct for the conditional data. This corresponds the positive CPC discussed in the main text.
17

---

Conversely, if Ac < Auc,k, then uec explains more variance in the unconditional dataset than in the conditional dataset, making it more distinct for the unconditional data and therefore it is negative CPC.
Under the assumption, the CFG guided ODE (35) can be simplified as:
dx (I U U Mo) do Uc(Aa,c - Aa,uo) U2 (x: - (oe) do
(47)
Vcl1 Ut -uu)) do.
(48)
Define (x(0) = =u[ "²,1(2 Mc), we have:
o(Ackk - )mock)
dcx(o) = de, + 0² (o(oddo
(Mec + o²) + 02) Ca (o)
(49)
Auank + 0² U o,k ( - Hoo
(50)
Therefore, the dynamics of dcx (o) can be expressed as:
(ox + ((o))x(o)da = g(o)do,
(51)
(dae -huck) and g(o) - Xue, u (Ho Muo) where (() -( (Ac,t (Aucc
Homogeneous ODE. We first consider the homogeneous counterpart of (51): dcx(o) = - f(r) (o))do, where (o(d) corresponds to the combination of the standard conditional score (36) and the CPC guidance term (37). Integrating over both sides of (52), we get:
(52)
Ck (x) Cel-Cel- -((o)d
(53)
Notice that:
/ - = / Aayk + 02 do -7 I aDack -Nuek))
+ o2) (2) + + do
(54)
2 In(Ac, + +²2)+ -- In( de + t
(55)
2 due,k + +
which implies:
dd, + o2 (0 = C(dc,b + 02)
(56)
Auc,k + o²
Applying the initial condition that c* (oT) = u¹ (xT
-Mc), we have:
C = CNekk + - de, + 07
)- 3 uu u (xT Me)
(57)
ue + ox
(58)
d
de, of Aue,t + oo oz do, + of Auc + o²
dd of uh u 1e)
= cx(or)
Σ
(59)
de + oo
de + of T de of u Mc)
Me +
he (1 duckk)
V
(60)
N=
where h(Aa,kr )uo.k) =
Compared with the solution to the naive reverse process with no guidance (32), each component of x, differs only by scalar factor h()c,kr )uc,k) 7 Specifically:
Ac (e)) Auc +o
Aex+o<(IT Auc.kto
h()c,ke ) 2 if and only if Ae,k 2 Aue,ks meaning positive CPCs are enhanced (scaled up).
h()a,k ) < 1 if and only if Ac,k S Auc,k, meaning negative CPCs are suppressed (scaled down).
Note that the guidance strength y provides additional control, amplifying or reducing the degree of enhancement suppression for each component.
18

---

Mean-shift: [0.0, 0.01
Mean Mean-shift: (5.0, 5.0]
Mean-shift: [10.0, 10.0]
Mean shift: [20.0, 20.0]
Figure 9: CFG effects in 2D. Each subplot differs by the class mean M4c, indicated in the titles.
Blue, orange and green points show 1,000 samples generated from conditional sampling, naive unconditional sampling and CFG sampling, respectively.
Non-Homogeneoos ODE. Let (x( be the solution to the homogeneous ODE (52), then the solution to the non-homogeneous ODE (51) takes the form:
1 (x(o) = (x(o) + I(o)
/ I(d')(o')dd''
(61)
where ((o) e! f(a')do' is the integrating factor. Since:
((0) = C( + - Ae, + 0²
(62)
Auchb + 0²
we have:
de, o²
07 (Nuc, + +
=()()) 7(Da, + 0z)
uo Huclōdō (63)
duckk +
(Ne, + 02) 2
=(x(o) v T (the Mue))
(64)
where bo,k (De,k + 02)²
Ao Anato? xuo,)toz) ) for (mnckto2) ādō. Therefore have: (Ao,)t2)) 2
Ae, + to² u (MT MeJuakk
pe +
Meaa, )uc,k) 2
V
b 2e (Me
Hue)
de, to
1=1
k=1
(65)
B. Anc + oz T (xT Mc)uc UU.... U (Mc -uee,, (66)
Σ h(dackk &
Me +
V
de,b + of
where Bd, = diag (bo, a)). Here by depends only on Aua.k Ae and (1). Hence the mean- shift guidance term (38) has the effect of adding constant perturbations that are independent of the initial noise DT to the sampling trajectory.
B.3 Empirical Verification on Synthetic Data.
We validate Theorem on a 2D toy model with U. =
V2 v2
A. = 1 9] and Auc = [3 1
/7
Figure shows the effects CFG under different class mean Pe (with y and Pue 0). As predicted, CFG enhances variation along the positive CPC
1/ l" suppresses variation along the negative CPC [ (1 - and shifts samples roughly toward Mue at a rate proportional to
[
allle Puell2.
C Constructing Linear Denoisers
Constructing the linear denoisers (7) requires estimating the data means and covariances. We perform our experiments on CIFAR-I0 [42] and ImageNet dataset [29], estimating the linear denoisers for each dataset in different ways:
19

---

CIFAR-10. This dataset consists of 10 different classes, each with 5000 images. We obtain the unconditional linear diffusion model by computing the empirical mean and covariance across all 50000 images. For conditional linear diffusion model, we construct a separate linear model for each class, using that class's 's mean and covariance estimated from the 5000 images.
ImageNet. This dataset contains 1000 classes, each with approximately 1000 images-4 smaller per-class sample size that can introduce bias when estimating means and covariances directly from the training set. Although such direct estimation still yields linear denoisers aligned with the actual diffusion models in the linear regime, these denoisers tend generate noisier images. We hypothesize that, in the conditional setting, each class's 's diffusion denoiser may implicitly leverage information from other classes, meaning the true mean and covariance learned by the deep diffusion model can differ (albeit slightly) from the those estimated solely from that class's 's data. To obtain more accurate línear approximation, trained diffusion model, then compute the empirical mean and covariance from these generated samples. Nonetheless, all of our main conclusions remain valid even if build the linear models using the actual ImageNet training data.
therefore generate 50,000 samples per class with the
D Naive Conditional Generation Lacks Class-Specificity
In section 3.1 we argue that naive conditional generation lacks class-specificity and in the linear model setting, such issue can be partially attributed the non-distinctiveness of the class covariance matrices. In this section, we provide comprehensive experiments to support our claim both qualitatively and quantitatively.
D.1 Qualitative Results
We generate samples using naive conditional sampling (4) and CFG sampling (6) for all 10 classes of CIFAR-10, 10, as well as for 10 selected ImageNet classes: including (i) class O: tench, (ii) class 31: tree frog, (iii) class 64: green mamba, (iv) class 207: golden retriever, (v) class 430: basketball, (vi) class 483: castle, (vii) class 504: coffee mug, (viii) class 817: sports car, (ix) class 933: cheese burger and (x) class 947: mushroom. CFG is applied to the entire noise interval (t) E [0.002, 80), with guidance strength = 4. The results for CIFAR-10 and ImageNet are shown in Figure 10 and Figure respectively.
Our key observations are as follows:
Despite being built from class-specific means and covariances, the conditional linear diffusion models produce visually similar samples that lack distinguishable class features. From (8), we see that the generated samples are largely shaped by each class's 's covariance structure; hence, their indistinct and lua-quality generations suggest that these covariance matrices are insufficiently distinctive.
Linear Diffusion Models.
Deep diffusion models (EDM) These models inherit similar limitations. The generated samples often exhibit poor image quality, with incoherent features that blend into the background and the class-specific image structures can be hard to discern. Furthermore, images generated from the same initial noise can appear structurally similar even under different class labels, indicating that naive conditional sampling fails to capture distinct, class-specific patterns. Lastly, comparing the generations from linear model and EDM reveals they match in terms of the overall structures, underscoring the key role of covariance in shaping higher-level features. Consequently, when class- specific covariance matrices are not sufficiently distinct, sample quality remains limited even in nonlinear models.
D.2 Quantitative Results
To quantify the class-specificity gap, we compare the pairwise class similarity with FID score [27], which measures the similarity between two datasets X and Y in the Inception embedding space. For every ordered pair of different classes (c4, Cy) we build two datasets (X, Y) and compute FID(X,) Y) under three settings:
Real data. X and Y contain all training images from classes e; and cj, respectively.
20

---

Deep Diffusion Model
Classe
automabile
homee
CFO Samples
ship
trudit
Samples
(a) Naive conditional and CFG generations for EDM
Linear Diffusion Model
Narve conditional
sample
CPO Sumples
Naive conditional samples
CPO Samples
(b) Naive conditional and CFG generations linear diffusion model
Figure 10: Effects of CFG on CIFAR-10. (a) and (b) demonstrate the naive conditional samples and the CFG-guided samples of deep diffusion model and linear diffusion model respectively. Each grid corresponds to the same initial noise.
Naive conditional sampling. X and Y contain images generated by vanilla conditional sampling (4) with the EDM model. We generate approximately the same number images as the corresponding training images.
Classifier-free guidance (CFG). X and Y contain images generated from the same EDM model us- ing CFG sampling (6). We generate approximately the same number of images as the corresponding training images.
The results are presented in Figure 12, which shows that for most pairs of classes, when X and Y are built with images generated with naive conditional sampling, the FID (colored in orange) is consistently lower than when they are built with training data (colored in blue). Because lower FID indicates higher similarity, this results confirms that images produced by naive conditional sampling
21

---

Deep Diffusion Model(EDM)
Linear Diffusion Model
frog
een
bosketholt
Clase:
frog
con mamba
baoketball
Neive conditional
samples
CFQ Samples
Natvv conditional sumples
CPO Somples
conditional gonerations for EDM
Naive conditional and CFG gencrations diffusion model
Figure 11: Effects of CFG on ImageNet. (a) and (b) demonstrate the naive conditional samples and the CFG DFG-guided samples of deep diffusion model and linear diffusion model respectively. Each grid corresponds the same initial noise. Here we only display 6 classes since the other 4 classes are presented in fig. 2.
FID between classes: training data / naive conditional samples / CFG-guided samples
quen namia
gelzen nereoer-
antic
D13136 7130
cilerr sxeg
21.21 1
denneurger
reach
SreeK AFA//0A//224
gHiden 20A4ORAT22 1973
204/242/228
Figure 12: Class-to- -o-Clsss Similarity (Measured with FID). Each cell reports the FID between datasets of two classes, built with (i) training data (ii) data generated by naive conditional sampling and (iii) data generated by CPG sampling.
are less distinguishable across classes than the real data. In contrast, CFG greatly improves the FID score, implying an increased inter-class separation.
The samples used for calculating FID in Figure 12 are generated using 20 steps of Euler method (first-order sampler). Increasing the number of steps or switching to higher-order sampler only marginally narrows the gap. Table shows the inter-class FID averaged over 10 selected classes as described in appendix D.1 with different sampling steps and sampler. Note that even when using 100 steps and second-order Heun samplers, the average inter-class FID is still considerably smaller compared to the training data (ground truth). Figure 13 qualitatively visualizes the samples generated from the same initial noise but different class labels. Despite conditioned on different labels, the generated images share high structural similarity. For certain classes, such as tree frog, green mamba and golden retriever, the class features are even hard to discern. contrast, CFG greatly reduces the structural similarity, yiclding images with clear, class features.
22

---

Class: tench
tree frog
mamba
golden retriever
cheeseburger
mushroom
Gaussian
model
Naive conditional sampling with 10 steps (Euler)
Diffusion
model
(b) Naive conditional sampling with 10 steps (Euler)
Diffusion
model
(c) Naïve conditional sampling with 20 steps (Euler)
Diffusion
model (EDM)
(d) Naive conditional sampling with 100 steps (Euler)
Diftusion
model
(e) Naïve conditional sampling with 100 steps (Heun)
Dittusion (EDM)
(f) CFG guided sampling with 20 steps (Euler)
Figure 13: Naive conditional sampling lacks class-specific features. Figure (a) shows the samples generated with naive conditional sampling using linear diffusion models. Figures (b)-(e) show the samples generated with naive conditional sampling using the actual diffusion models with different steps and samplers. Figure (f) shows the generated samples with CFG guided sampling. Note that generated images from linear models of different classes share high visually similarity, implying the covariance structures different classes are not distinctive enough. Similar structural similarity can be observed in the samples of nonlinear diffusion models. CFG greatly alleviates this issue of lack of class- specisscce,,,,tt, leading to images with clear class specific features.
23

---

D (()) along Diffusion Sampling Trajectory
Class: golden retriever
Noise level e00 88.0
42. 415
21 109
066
1.502
0.47
1:717
0.02
002
Diffusion
(EDM)
Linnar Gauzzian
Class: cheeseburger
Neiso lovel el0 - 88.0
42.4 415
21
066
1,502
0.47
11117
002
Diffuian
mode!
(EDM)
Linear
Gauesian
model
Class: tree frog
Diffusion model (EDN))
Linear
Gauesian
model
Class: coffee mug
Nolee Invel
e(f) 88.0
109
0.42
0.02
(EDM)
Linear
model
Class: mushroom
Noine lnvel
+(f)
$2.435
Diffuxion model
(EDM)
Gausxian
model
Class: Basketball
Nolse level
80.0
42.415
066 1.502
0,002
Diffusion
model
(ED0))
Gaunsian modol
Figure 14: Similarity between Linear and Nonlinear Models. For high to moderate noise levels (o(t) E (4, 80)), the linear denoisers well approximate the learned deep denoisers. Though the two models diverge in lower noise reigmes, their final samples still match in overall structure. Although the linear models are built separately for each class according to (7), they generate highly similar samples when starting from the same initial noise. The same similarity also exists in the samples of real-world diffusion models.
24

---

Table 1: Average inter-class FID for training data and various sampling settings (10-class average).
Method
Steps
Sampler
Avg. FID
Training (ground truth)
226.6
Naive conditional
10
Euler
210.7
20
Euler
214.6
Fuler
215.8
50
Fuler
215.9
100
Fuler
216.2
100
Heun
216.3
CFG guided (y 4)
20
Euler
258.9
D.3 Covariance Matrices of Different Classes Lack Class-Specificity
The lack of class-specificity is especially pronounced in linear diffusion models. As shown in Fig- ure 13(a) and Figure 14, although the linear diffusion models are separately parameterized with the class-specific means and covariances for each class, the resulting samples share high similarity. Since the generated samples of the linear models are governed by the data covariances, the observed inter-class similarity implies that the covariance structures of different classes are not distinct enough.
Next, we quantitatively demonstrate that the class-specifie covariance matrices are insufficiently distinct. To do this, we take Uwe. the principal components (PCs) of the unconditional dataset (i.c.. the singular vectors of the unconditional covariance), as a baseline. We then compare Uue to Uo- the PCs of each conditional dataset. As shown in Figure 15 and Figure 16, the correlation matrices U.e for 10 classes (5 from CIFAR-10 10 and 5 from ImageNet) reveal that the leading PCs of each class share high similarity with those of the unconditional data. Thus, the PCs do not necessarily capture the distinctive features of individual classes though they represent the dominant variations of dataset. Instead, these PCs often reflect global intensity or beegrrondd-backgron
variations.
Why Covariance Structure Matters? Covariance structures are fundamental statistics of target distribution, and we would expect a robust diffusion model learn them accurately. However, because these covariance structures are not sufficiently distinct, linear diffusion models-relying heavily on covariance for generation-struugll fidelity, models must leverage higher-order information beyond covariance. Recent works [21, 22] observe that deep diffusion models can be approximated unreasonably well by linear diffusion models, especially when the model capacity is limited or the training is insufficient [21]. Qualitatively, we have demonstrated the similarity between linear models and the actual diffusion models by showing that linear models replicate the coarse (ow-frequency) features of samples generated by deep diffusion models. These results suggest that deep diffusion models may have an implicit bias toward learning simpler structures such as covariance, and thus the suboptimal nature of data covariance for generation task can limit their generative quality.
produce high-quality images. To achieve better
E Mechanism of Linear CFG
In the setting of linear diffusion model, (37) and (38) together form the CFG guidance. For the following discussion, we let Σ = U,Ao(,,, and Ene,t = UncAa(() UT
E.1 Mean-Shift Guidance
Equation (38) is the mean-shift guidance term that shifts o, towards (I (I- Euc,; Muc), direction independent of the specific sample æ,, At sufficiently large (()), (I (mee)))( mhe) RS phe phuss indicating the mean-shift term approximately shifts 2; towards the direction of the difference between class mean and unconditional mean μ.e Mue. As a(t) decreases. the components of Me pue within the subspace spanned by the unconditional PCs (Uwe) progressively shrink to O. Figure 17 demonstrates μe Mue and the evolution of the mean-shift guidance term (I - (mo Mue) across different noise levels. Notice that for wide range of noise levels (t)) (I Ene. Muc)
25

---

Mea
Mact
M,a
Mack
Maa
Muct
Maca
0.984
0.989
0.965
0.966
green mamba 0/: U.e
M4.1
Mec3
M22
Menz
14.4
Macs
Me.a
Mur
0.990
0.914
0.975
coffee
Ma.
Mac.
M2a
Mac2
Max.5
Mac4
0.908
0.902
0.905
0.978
cheeseburger U.Ugcc
Met
Mecc
M4,3
Muc3
Mec
Mac.5
W4.34
Mac37
0.956
0.979
0.853
retriever
0.976
0.981
0.914
Figure 15: Covariance Structures CIFAR-10. Each row corresponds different class. On the left, we show the correlation matrix between conditional and unconditional principal components (PCs), visualizing only the first 25. The subsequent images depict several highly correlated PCs, with correlation values displayed underneath. These results illustrate that the leading PCs do not always capture class-specific patterns.
remains close M4 Mlae before it becomes uninformative small (t). Hence, as stated in the main text, the mean-shift guidance term has the effect of approximately shifting æ; in the direction He
Mac-
E.2 CPC guidance
a(t) be the cigendecomposition of E., Eua.t whose eigen spectrum is demonstrated in Figure 18, the CPC guidance term be further decomposed into the positive CPC and negative CPC guidance:
Equation (37) is the CPC guidance term. Let Vo(1) VT
(t) (Vat),+ Aa(). Vo He)dt.
7
(67)
(2( (Vo()) A Vo())-)(as - we)dt,
(68)
where Va(t.. and Vo(),- contain eigenvectors corresponding positive and negative eigenvalues Ao(e) and Ag(t),- respectively. As discussed in section 2.3, Σe,t and Euc,t are (up to a factor 21) the conditional and unconditional posterior covariances of Pana(ar/c) = N(uue, E,) and Pdata (x) (puee Eve). Hence, Va(t) are the CPCs which contrast between X ~ Paato (x) x c) and Y Pasaa Pasau((/a)) (x/x:)) Specifically, Va(e). captures directions of higher conditional variance (class- specific features), while Va(t).- captures directions of higher unconditional variance (features more relevant to the unconditonal data). Figure illustrates the evolution of positive CPCs (Vo())+) and PCs (U.) across different noise levels. It is evident that the positive CPCs better capture the class-specific patterns compared to PCs. Here we choose not to display negative CPCs since they correspond to generic features that explain more variances for the unconditional dataset, which
26

---

airplane
automobile
U.U.
M4,1
Mact
M4,2
M4,
Muxc
14,10
Mucto
0,980
0.912
0.842
0.002
bird
0.868
M,.1
Mar.1
M4,2
0.984
0.882
0.091
0.954
deer
USU..
0.989
0.800
0.856
0.897
Figure 16: Covariance Structures of ImageNet. Each row corresponds different class. On the left, we show the correlation matrix between conditional and unconditional principal components (PCs), visualizing only the first 25. The subsequent images depict several highly correlated PCs, with correlation values displayed underneath. These results illustrate that the leading PCs do not always capture class-specific patterns.
less visually interpretable. Nevertheless, will show next, suppressing these directions is beneficial.
F.3 Distinct Effects the CFG Components
As we discussed in the main text, the three CFG components have the following effects respectively:
The positive CPC guidance term amplifies the components æ; that lie the subspace spanned by the positive CPCs, thereby enhancing class-specific patterns.
The negative CPC guidance term suppresses components x; that lie in the subspace spanned by the negative CPCs, mitigating background clutter and irrelevant content. As a result, the class-relevant sstructures become more salient.
The mean-shift term approximately shifts æ; in the direction μ4. puue enhancing the structure of class mean within the generated samples. However, since this perturbation is independent of the specific 3i, tends to reduce sample diversity.
Qualitative Results. Figures 20 and 21 qualitatively demonstrates the effects of each CFG compo- nent in linear diffusion models over 10 different ImageNet classes.
Quantitative Results. The distortion effects of the CFG components can be quantitatively verified through the following experiment:
For a chosen class, generate 1,000 samples using naive conditional sampling (denote the samples as ax,) and 1,000 samples using CFG (denote the samples as Fefzy), and center both sets by subtracting the class mean Me-
27

---

(1 -- A..)
Me. Pur
o(O) 80.00
o(0)= 42.415
o(1) 21.109
((t) 9.723
o(0) 4.066
o(O) 1.502
ott) 0.470
free fog
mug
sports car
room
Figure 17: Evolution of Mean-shift Guidance. The leftmost image shows μ.c. Muc while the subsequent images illustrate (I (pe - pue) various noise levels ott). Note that over wide range of (t), (I- (Mo (ue)) remains close to Ma Puc.
frog CPC Bisemaie Rote Leve!
golden retriever
CPC Usermatue mate Lave!
CPC Lneetvetee Netee Levee
Figure 18: Eigenvalues of E., Eme,t The matrix Σ., Eme, exhibits both positive and negative cigenvalues, whose corresponding eigenvectors correspond positive and negative CPCs respectively. Though we only show the spectrum for three classes, this behavior remains consistent across other classes.
For a chosen positive (or negative) CPC v. compute the projection magnitudes |o (xc p4o) and |o" (Iag 142) to obtain a series of univariate distributions along v.
Project the same samples onto the mean-shift direction μec Muae by performing (Me M pe) and (pe Muc) (xxdg - p.).
The resulting univariate distributions quantify the amount of energy the samples have along these directions. The above experiment are performed on both linear and nonlinear (EDM) diffusion models. The samples generated using 20 steps and the guidance strength is set to 2. We focus on the first class of ImageNet (tench) and present the results on the first 5 positive CPCs and negative CPCs. As shown in Figures 22 and 23, compared to the samples with no CFG, the distributions of the CFG-guided samples have higher density on the positive CPC directions but lower density on the negative CPC directions, implying the former enhanced while the latter is suppressed. The univariate distribution of the projection onto the mean-shin shift direction is presented in Figure 4(c) (bottom row), from which clear that the density is shifted in the direction of p.c puc-
28

---

Positive CPC ((t)
Positive CPC
Positive CPC 21.109
Positive CPC e(() 727
Positive CPC ett) 4. 066
Positive CPC o 1 502
(a) tench
(b) mushroom
cheeseburger
DVOGE
VO00E
UOUTE
DOOTE
TOOTE
POTUO
PI1T T
DUTE T
(d) coffee mug
(e) golden retriever
Figure 19: Visualization of PCs and Positive CPCs. Compared to principal components (PCs), the positive CPCs better capture class class-specific patterns. Although only five classes shown here, similar trends appear across other classes as well.
29

---

conditiona sumples
positive CPC guidance
negative CPC guidance
mean-shift guidence
CFG guidence
tench
(b) tree frog
green mamba
(d) castle
(e) coffee mug
Figure 20: Distinct Effects Different CFG Components. Each row shows (from left to right) the samples generated with (i) naive conditional sampling, (ii) guided with positive CPC term only (ii) guided with negative CPC term only, (iii) guided with mean-shift term only and (iv) guided with full complete CFG. Each row corresponds to different class.
30

---

condiional samples
positive CPC guidance
negative CPC guidance
mean-shift guidence
CFG guidence
golden retriever
(b) mushroom
(c) besketbell
(d) sports car
(e) cheeseburger
Figure Distinct Effects Different CFG Components. Each row shows (from left to right) the samples generated with (i) naive conditional sampling, (ii) guided with positive CPC term only (ii) guided with negative CPC term only, (iii) guided with mean-shift term only and (iv) guided with full complete CFG. Each row corresponds to different class.
31

---

Univariate Distribution of (x, R2)l and |ao (xesg - He)l
1" positive CPC
Linear Gaussian
Rusenn seroa pairoe
Pestive CPC onherrd
(a)
positive CPC
Linear Gaussian
EDM
Prog now podeed conp coefitin
(b)
positive CPC
Linear Gaussian
EDM
(c)
positive CPC
Linear Gaussian
EDM
Peggrt sahe ueteed
(d)
5th positive CPC
Linear Gaussian
EDM
(e)
Figure 22: CFG enhances positive CPCs. For both linear and deep diffusion models, randomly generate 1,000 naive conditional samples æ. and CFG-guided samples Zefg, center them by subtracting the class mean fhe and project them onto the top 5 positive CPCs (v4) to obtain series univariate distributions. In both model types, the distributions of CFG- samples have greater density at higher projection values, suggesting that CFG amplifies the positive CPCs.
32

---

Univariate Distribution (x, (c)) and iro (x - ue)l
negative CPC
Linear Caussian
we temeree segarin frenc uetied sanole repulte
Prsset
Heg Copaoeprene
(a)
2d negative CPC
Linear Gaussian
EDM
Presed
Prggon nus sorggtss repaltre
(b)
negative CPC
Linear Gaussian
EDM
(c)
negative
Linear Gaussian
EDM
(d)
5t negative CPC
Linear Gaussian
EDM
(e)
Figure 23: CFG suppresses negative CPCs. For both linear and deep diffusion models, we randomly generate 1,000 naive conditional samples a, and CFG-guided samples &efg. center them by subtracting the class mean Pe, and project them onto the top 5 negative CPCs (v_) to obtain series univariate distributions. In both model types, the distributions of CFG-guided samples have greater density lower projection values, indicating that CFG suppresses the negative CPCs.
33

---

e
80.0
59. 66
13.92
31. 88
22.79
16.02
11.05
7.16
3.93
3.17
0.69
(a) sports car
(b) golden retriever
(c) cheeseburger
Figure 24: Correlation between (2 and U.- The leading singular vectors of (my; (t), ()) c)
well align with Ug for high to moderate (t). Each plot shows the average correlation computed over 10 randomly initialized sampling trajectories measured for three different classes.
CFG in Nonlinear Deep Diffusion Models
In this section provide additional experimental results for section 4, where investigate how well the insights derived from linear diffusion models extend to real-world, nonlinear deep diffusion models. In this work, study the state-of-the-ard EDM models [4].
F.1 Linear to Nonlinear Transition in Diffusion Models
Recent studies [21, 22] observe that at high to moderate noise levels, deep diffusion models Da (t)) can be well approximated by the corresponding linear diffusion models (x (f)) defined in (7). As the noise level decreases, De (æx; (t)) becomes nonlinear. We verify this transition by the following experiment:
Let (x(,,) be the left singular vectors the network Jacobians ( c)) along the sam- pling trajectories, and let Ue be the left singular vectors of (DL (t), c). Since (i (t)) He UaAaato (a: pe),, if Do Dr. then VDo (tt,, e) (D) s U U,A UT, implying U,(2,) = U.. independent of x,- As illustrated in Figure 24, for large (t), the leading singular vectors of VD (Z (t), c) indeed align with Ue. Note that since A A = diag( (o0 \c-tto-(t) (DL( is highly low-rank at large (t)). Thus, primary interest is in the leading singular vectors, and the non-leading singular vectors biguous. contrast, for small (t), the alignment no longer holds and (D) (t)) starts to adapt to individual samples, reflecting the model's nonlinear behavior. Figures 5, 25 and 26 qualitatively demonstrates this linear to nonlinear transition.
Ac.1
As,d
F.2 CFG in the Linear Regime
We provide additional experimental results for section 4.1 in Figures 27 30. Because the precise transition time from the linear to the nonlinear regime-as well as the influence of each CFG component-varies. across classes, we empirically choose the interval for applying guidance and calculate the FDDINOv2 score with 50,000 generated images for each class separately. We summarize our observations as follows (see also section 4.1):
Linear vs. Nonlinear CFG. Applying linear CFG to deep diffusion models produces effects that closely resemble those the actual (nonlinear) CFG.
Dominance of Mean-Shift. In most the 10 classes studied, the mean-shift guidance term dominates CFG behavior, as it alone can generate results visually similar to full CFG. However, for the coffee mug class, the positive CPC term takes precedence, becoming the primary driver of CFG.
34

---

Lineer Regime
Class: tree frog
Nonlinear regime
1W
(e) Effcots of nontinear CRO individual tinestop
H6) Effects tinaa CPC indivduall timestep
Linear Regime
golden retriever
Nonlinear regime
indiviiddal timestop
(M Effeots Uineer CF0 indiviiiuu Gimestop
Linear Regime
Class: sports car
Nonlinear regime
Regime
Class: cheeseburger
regime
(ei Effecis ofnontinear CFG at inchvisust tmeette
(b) linor CTG individdal timestep
Linear Regime
Class: golden retriever
Nonlinear regime
Effects nontinaar CTG individual timestap
(h) Dffects of Uneer CPG Individtual timeshep
Figure Linear-to-nonlinear transition in diffusion models. (a) and (b) compare nonlinear CFG and linear CFG applied to deep diffusion model (EDM). The leftmost column shows unguided samples; subsequent columns show final samples when guidance applied only specific noise level, with = 15.
Role of CPC Guidance. CPC guidance generally improves generation quality, though its benefits can sometimes be less pronounced. For instance, in the tree frog and castle classes (Figure 29), the CPC term does not enhance FDDINO much as the mean-shift term. Nevertheless, CPC guidance
35

---

Noise
42.415
1.502
Diffusion model (EDM)
Gausstan
model
Dt., Diffusion Sampling Trajectory
(g(Kg)
PC
ait) = 22.79
18.02
7.47
3.17
1.19
Principal Components
singular 9(,, ((1) slong Sampling Trajectory
Figure 26: Evolution of Denoiser Jacobian During Sampling (a) demonstrates reverse diffusion trajectory. The left most image of (b) demonstrates the leading PCs of the data covariance. The subsequent images visualize the singular vectors, (2+)), of the denoiser Jacobian at different noise levels. Note that at early timesteps (x,,) match the PCs but gradually adapt to the geometry of the sample xt.
operates effectively over wider range of guidance strengths and noise intervals. For the green mamba and basketball classes, we show results within the prescribed noise interval as solid curves, and extend beyond this interval dashed curves. While mean-shift becomes highly detrimental once outside the linear regime, CPC guidance remains beneficial.
F,3 Mean-Shifted Noise Initialization
The observation that the indeeedend mean-shift guidance alone leads to improved FDpINOC2 score implies that simply initializing the sampling process from mean-shifted Gaussian, Ør N(7( o² (T) I)) with no additional guidance applied, can improve the generation quality, which we verify through the following experiment:
For a chosen class and positive scalar y, generate 50,000 samples via naive conditional sampling initialized from mean-shifted Gaussian N (T((e o (T) Then evaluate the sample quality with FID) and FDDINOs2 scores.
Repeat the above procedure across several classes and range of values.
We perform the above experiments on 5 classes, where (T) is set to 31.9. The results are shown in Figures 31 35. Note that the sample quality improves with properly chosen
F.4 CFG in the Nonlinear Regime
We provide additional experimental results for section 4.2 in Figure 36. We argue that effective guidance in this regime should satisfy two key properties:
Capture local structure of specitic sample. As shown in Figure 25, when ott) is small, the model diverges considerably from its linear approximation, and linear CFG deviates from the actual nonlinear CFG. this regime, CFG does not alter the overall image structure but instead refines existing features to produce crisper images. Consequently, effective guidance must adapt to each specific sample. We propose that such guidance can derived from the network Jacobians VDa (æy; (l), e) evaluated at æt. Prior work [33] shows that the singular vectors of these Jacobians, which are equivalent the posterior covariances, adapt to the input 2;,-
Capture class-specific patterns. in the linear case, the guidance must also capture class-specific patterns. This can be achieved by contrasting the conditional Jacobian VDa (Zx; (f), c) with the
36

---

Classc grean mambe
ett e [22. 80)
Class: coffee mug
403
116. 16.022 80)
F-O
yoa
yob 7 y you y 2-
yee
y=1
F-3
y-2
y-7
y-3
15
T-
Class muabroom r=
e (1L. 05,80)
Class: tench
e 11..05 211.0,,00 80)
T=1
y=3
y=?
y-s
y=0
T-1
y-3
y-5
y=3
y+9
T- M
WI2
Class: castle
e E [22 79,80)
Class: basketball T-5
d (22.79,80 90)
y=9
7
y
y=3
7
35
Sunie
Seperu
Figure 27: Effects of CFG in Linear Regime. Each row demonstrates the impact of different guidance types applied to EDM within the linear regime, with varying guidance strength 7. The guidance is applied only within intervals specified in the subtitles, where the model exhibits linear behavior.
unconditional Jacobian VDa (ay; (1)). Figure shows that guidance built using CPCs-i.e. difference between these two acobians-yyille effects similar to actual CFG. In contrast, guidance derived solely from the conditional Jacobian does not improve image quality.
Note that (17) inspired by linear positive CPC guidance (13). We also test other guidance such as
? O2(1) S4. (of, pe)),,
(69)
but find it less effective than (17), likely due to additional noise in æ,, Moreover, observe that negative CPCs and mean-shift terms are not as effective in the nonlinear regime.
Lastly, we'd like to remark that our goal here is not suggest that CFG in the nonlinear regime is exactly equivalent to (17); rather, we note that both approaches exhibit similar behaviors, implying they may share features. The exact analytico form of CFG in the nonlinear setting remains challenging to derive due to the complexity of deep networks, leaving promising direction for future work.
mechanism: identifying and amplifying sample-s speciffcc and class-specific
37

---

#
Fandee
caleree
Caber
Class: cheeseburger y
ocly [22 79,00)
TT OZ
Ueer
Frobe DN
Facie
Figure 28: Effects of CFG in Linear Regime. Each row demonstrates the impact of different guidance types applied to EDM within the linear regime, with varying guidance strength y. The guidance is applied only within intervals specified in the subtitles, where the model exhibits linear behavior.
38

---

Closs: tenoh
= 80)
#03 [1.. 80)
Beseine the guicencl)
2033
CT2
Liner CTG
1290
3754
FAEE Guitence
Mann Cu dance
Negetive CPC Guddance
1300
2506
FD
1.22
3006
033
13
23
Guldlance
Strengtth
Guldance Strength
frog
(22.79.800 79.9
grean #0 (22.79, 98)
Basetine (ne guatence)
Easeline guiderce
PorBtve ESC Guidence
Rochin CSC GuEdares
Negative CPC Gateenc
3460
Nequiive CTC Guslence
Meaa' Silt Ganzonue (.J.0r.000
Pcs live Curdarce (31.07.35
Negative Guidance [11.0,,000
xmn
202 Guidance Strungth
Guidancc Strength
Class: basketball
c (22.79, 88)
castla
Baseire tht guicencel
Baceline (to guiderct)
CMS
CPC Guldance
Negative Gardenee
FD D00000
13300
Noan Cuntunse: 11.01.00 20.0)
Pooithe CPC Guidence [11,01 05.01
Nogathen Guisanen (31 (0.3)
122
120
13
16
10
Guillance Strength y
Guidance Strength
Clane: coffen mag
#tt' e (16. 80)
Clans: chenteitunger
ett [22. 80)
1000
CGG
Sayeline
2300
CTG
Buadin
Liner ETC
Uneer CTG
Vesh Siit Guidance
Mean Shift Guidence
Fosdiv Guacence
2000
lee Cundarce
Galsance Strangtt
Strength
Figure 29: FDDINOv2 Scores. The guidance is applied to the interval specified in the subtitles. For green mamba and basketball, we find it beneficial apply CPC guidance beyond the linear regime, with results demonstrated by the dashed curves.
39

---

Baseline retrlerer
Class: sports
7300
F56
Literr ETG
Unear CRG
Moan Guitanze
Mean Shin Guidence
dence
PoSifrre CPC Gu dance
Megglliv Guidaric
FD2M02
300
Guldance
Guidance
Figure 30: DDDINO Scores. The reported values are relatively high because the scores are computed separately per class, which often has a limited number of training images. It is well known that FDDINO22 scores can appear inflated when the reference dataset size is small.
Class: tench
FID and F DDruav2 scores for samples initialized from N(y (Je, - Muc), (T)+()
244
212
250
200
Gudsece Strength
Guicance Strangthy Y
(8)
(b)
==3 3
y 5
y
y 9
10
15
y 20
Sample
Sample
Sample 3
Figure 31: Effects of initializing with mean-shift. For every [0, 1,3, 7,9, 10, 15, 20). we generate 50,000 images from initial nosies sampled from mean-shifted Gaussian distribution (( Mwe), (T))1) and compute the FID scores (a) and FDDINOV2 scores (b). The samples are visualized in (c). Note that adding mean-shift to the initial distribution leads to improvement of standard metrics.
40

---

Class: tree frog FID and FDpiovv scores for samples initialized from N (u. - o
Gussance Strengh
Gustance Strenghh
(8)
(b)
y
y=1
==3
y y=5 5 y= y=
7 10
y = 15
y = 20
Sample
Sample
Sample 3
(c)
Figure 32: Effects of initializing with mean-shift. For every [0, 3, 7,9, 10, 15, 20). we generate 50,000 images from initial nosies sampled from mean-shifted Gaussian distribution V(7(e ) (T)²1) I) and compute the FID scores (a) and FDDINOV2 scores (b). The samples are visualized (c). Note that adding mean-shift the initial distribution leads improvement of standard metrics.
41

---

Class: green mamba
FID and F DDDIO scores for samples initialized from N(y + (Je. - (u)).
800
200
800
320
36
Guidance Strengthy Y
Guidarce Strength y
(8)
(b)
==
y=1
==3
y=5
y=
y
y
y
y
Sample
Sample
Sample
(c)
Figure 33: Effects of initializing with mean-shift. For every [0, 10, 15, 20). we generate 50,000 images from initial nosies sampled from mean-shifted Gaussian distribution V(7(e ) (T)²1) I) and compute the FID scores (a) and FDDINOV2 scores (b). The samples are visualized (c). Note that adding mean-shift the initial distribution leads improvement of standard metrics.
42

---

Class: golden retriever
FID and F FDpnov scores for samples initialized from N(y + (u. - (T)))
50
302
Strength
Cuidlere Strongth
(a)
(b)
==
y=1
==3
==5
y=7
y=
y 10
y= 15
Y = 20
Sample
Sample
Sample 3
(c)
Figure 34: Effects of initializing with mean-shift. For every [0, 3, 10, 15, 20). we generate 50,000 images from initial nosies sampled from mean-shifted Gaussian distribution V(7(e ) 2 and compute the FID scores (a) and FDDINOV2 scores (b). The samples are visualized (c). Note that adding mean-shift the initial distribution leads improvement of standard metrics.
43

---

Class: mushroom
FID and DDinov samples initialized from Ny (H. o
XC
30
2
Gusdance Strengh
Guidance Strength
[a]
(b)
y=1
y=3
==5
y=7
y=9
10
15
= 20
Sample
Sample
Sample
(c)
Figure 35; Effects of initializing with mean-shift. For every [0, 1, 7,9, 10, 15, 20). we generate 50,000 images from initial nosies sampled from mean-shifted Gaussian distribution V(7(e ) 2T)2TT T) and compute the FID scores (a) and FDDINOV2 scores (b). The samples are visualized (c). Note that adding mean-shift the initial distribution leads improvement of standard metrics.
44

---

No Degidance
Megadeen
Galara
Adligun
Figure 36: Effects of CFG in the Nonlinear Regime. Different guidance methods, each with fixed strength of y 15, are applied individual timesteps in the nonlinear regime. Each image shows the final output when guidance is applied solely at the timestep indicated at the top. Note that (17) closely matches the effects of CFG by enhancing finer image details, whereas (18) does not improve generation quality.
45

---

Class: golden retriever (t)) [16. 02,
p=0
7=5
==10
7
15
=
Nonlinear CFG
Linear
Beeine piinect
Mean-Shift CFG
Positive CPC Quidance
Negative CPC Guidanee
Gultinnc Sowrggh
(a)
(b)
Figure 37: Effects of CFG in the Linear Regime (EDM-2). Each row in (a) demonstrates the impact of different guidance types applied to EDM-2 within the linear regime (specified in the subtitles), with varying guidance strength 7. (b) shows the FDDINO scores computed over 50,000 samples.
G Experimental Results on Latent Diffusion Models
In the main text, conducted experiments using the EDM-1 model [4], which operates directly in pixel space with 64 x 64 resolution. Here, we present complementary results on the EDM-2 [5] latent diffusion model, which generates images at 512 x 512 resolution.
Linear Regime. We evaluate multiple guidance crrateiess-incudiin actual CFG, linear CFG, Mean-shift guidance, positive CPC guidance, and negative CPC guidance-within the high-noise intervals (the linear regime). For each method, we generate 50,000 images conditioned on the class label "golden retriever" and compute the FDDINOv2 metric. The results, shown in Figure 37, are consistent with the observations reported in the main text.
Nonlinear Regime. We next examine guidance effects in the nonlinear regime using (17) and (18). As shown in Figure 38, guiding with CPCs produces visual effects similar those of actual CFG enhancing image sharpness and structure wwheress guidance with conditional PCs often leads to oversaturated colors. This highlights the importance of selectively amplifying class-specift features. We note that our heuristic guidance serves as conceptual approximation and may not always perfectly align with actual CFG behavior. Additional failure cases will be provided in our code release.
H CFG in Gaussian Mixture Model
Thus far we've been focusing on the setting of linear diffusion models, in which the learned score functions are equivalent to those of Multivariate Gaussian distribution. From complementary perspective, several works [20, 19, 7] have studied CFG under the Gaussian mixture model data assumption. However, these works assume each Gaussian cluster has isotropic covariance, which is oversimplified for natural image dataset. this section, we demonstrate that CFG guidance under Gaussian mixture model can be decomposed in similar way the case of linear diffusion model.
Consider unconditional data distribution:
K
Pultta (x) = Σ T<; E.),
(70)
im1
46

---

Apply guidance at ((t) = 1. 5020 y =
No guidence
Actual
Gulding with positive CPCS
Guiding with conditional
No quidance
Actual
Guiding posttive CPC
Gudding conditional
No dundcce
Actull CFG
Guiking with positive CPCs
Guiding with conditional NSs
300
No guidance
Actual
Gulding with positive CPCs
Guiding with conditional
Figure 38: Effects of CFG in Nonlinear Regime (EDM-2), Different guidance methods, each with fixed strength of ? 8, are applied (t) = 1.502. The samples in each row generated from the same initial noise.
where 14, and E, are the mean and covariance of the 5th cluster with weight n.. The noise-mollified data distribution then takes the following form:
K
p(z:a(t))
Σ
E, (11)
(71)
Let E Ea(()s = E; o²()),, then the score function of p(x; (t)) is:
Vplæ;o(t) pl (1))
V Vlogp(z;o(())
(72)
Σ D mi VN(x:; pss Eo(r,))
(73)
1=1 T:N( 14 Eo(0)
TNN(x; pis 20(0))) (m: x)
(74)
M Tx; p4i Do().))
(x)x))_ (14 x),
(75)
where uz(xx EK T. (m: Eottn.)) representing the posterior probability that belongs to the 8th
a.N(w: Eo(n.))
cluster and EK wx(x) 1. Let Σ, U,A,U, be the full SVD where A, = diag(Ar, A.a).
47

---

by Tweedie's formula, the optimal denoiser of the noise-mollifies Gaussian mixture model takes the following form:
D(z;o(t)) Vt)Vloggp o
(76)
x + (z)
uy (x: (1) [ -a) σ))
(77)
K
Wi ( p4 +
wi w;(x:) (, p4),
(78)
where (o()) diag
Furthermore, under the Gaussian mixture model assumption, each conditional distribution is a Gaussian distribution and from (7) we know the conditional optimal denoiser of the 7th cluster is:
A
(
D(æ; (1), c,) = p. U (a pi).
(79)
Without loss of generality, we the target condition C₁. Then the CFG guidance at timestep takes the form:
g(x, t) = Va, (og((z)c( Va, logp(z;o(t))
(80)
(2( (D(zi;o(t), e1) - D(z(;o(t)))
(81)
K
1
(U (U(Ar(t), UT Σ
wx( p1)
(82)
(²(
+ σ 2²(t)
w. (I -UAA (H4 mi).
(83)
1m2
Note that:
Guidance (82) resembles the CPC guidance (epe defined in (12). Different from the linear setting-whers the CPC guidance contrasts the posterior covariance of the target class with single unconditional posterior covariance, here it contrasts the posterior covariance of the target class with a meetmmax-weightee average of the posterior covariances of all classes.
Guidance (83) resembles the mean-shift guidance Imean (t) defined in (12). Different from the linear setting where the mean-shift guidance approximately aligns with μ4: Ple. the difference between the conditional and unconditional means, here it instead approximately aligns with softmax-weighted average of the pairwise differences between the conditional mean (mean of the target class) and the means of every other class.
I Computing Resources
All experiments are performed A100 GPUs with 80 GB memory. The diffusion model used in this work typically requires days for sufficient training.
48

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the abstract and introduction, we claim we (i) analyze how CFG works in linear setting and (ii) investigate how the insights from linear analysis extend to real- world nonlinear models. The former claim studied in section while the latter is studied in section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In section 4.2 we explicitly mention that the analysis in linear regime is only proxy. The exact mechanism remains unknown due the black-box nature deep networks.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
49

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We provide the assumption and the theory completely in section 3.2.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We'v provided the general information for the experiment setups in both main text and appendix. We promise to release code upon publication to enable reimplementation.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
50

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: Although do not provide code in the submission, promise release it upon publication of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co co/
public/ hubblcc/giies... wtwggg/e/codduuueeeeeeeee hatps//ooeSSSussseeeeee wwiiiisssiipyyyyoo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. wwtppcccc//ccccccccccccccccccclt wtwic//gg//////eieeieiiiccciise/ guwdee//odeeuumiii Cwweeoomiisisiiieoooooo wwwssssssssoioooooooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provide detailed information the datasets use, models and other hyperparameters including number of sampling steps, type of sampler and many others.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Although we do not provide error bar, all the quantitative experiments are performed according the standard. For example, all the FD scores are calculated over 50,000 randomly generated images.
Guidelines:
The answer NA means that the paper does not include experiments.
51

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the related information appendix I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http////ww....ppplll/tttttceeeee
Answer: [Yes]
Justification: All the authors have read the NeurIPS Code of Ethics. We confirm the research conform with the code in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiciio))
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is an empirical and theoretical work on understanding the mechanism of CFG. It has more scientific contributions rather than socictal impacts.
52

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work is an empirical and theoretical work understanding the mechanism CFG. doesn't 't have a high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We use the public released dataset and models, which properly credited the license.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
53

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work is an empirical and theoretical work on understanding the CFG mechanism. We don't release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
54

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: /tt////eurippppepe http://ripp..cccmeeeeeree/////// wctcConfeeeencc///25//////////// for what should or should not be described.
55


Reviews:
[{"review_id": "SczqTkN7P3", "reviewer": "Reviewer_225p", "summary": "The paper aims to understand classifier-free guidance (CFG) from a theoretical perspective. To simplify the analysis, the authors first study a linear model and then explore how the insights might extend to real-world diffusion models. They find that linear CFG improves generation quality through three key components: (i) a mean-shift term that roughly guides samples toward the class means, (ii) a positive contrastive principal components (CPC) term that enhances class-specific features, and (iii) a negative CPC term that suppresses generic features commonly present in unconditional data. The experiments are conducted on simple linear models.", "questions": "Please address the weakess.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Analyzing CFG theoretically to understand its mechanism is interesting and motivating.\n\n2. The theoretical formulation would be helpful to propose new guidance strategies.\n\nWeaknesses:\n\n1. While I appreciate the analysis in linear models, the extension to non-linear regime and real-world diffusion models seems weak. E.g., the argument in L349: “empirically ..” not not convincing, since I don’t see much improvement in the visual results in Fig. 7, with PCs, or CPC. Guiding with PCs even decrease the quality compared to CFG, why that's the case? \n\n2. In Fig.7, sample 3, guiding with CPC, PC, change the color of the car, can you explain this? Do PCs capture different visual attributes? But in that case, how the prompt alignment is performed, it's not clear to me.\n\n3. The qualitative results are not convincing. E.g., Fig. 5, I see non-linear CFG, performs better than CPCs. It can be useful to understand CFG, but how it's helping to improve CFG in terms of generation quality and prompt fidelity remains unclear.\n\n3. CFG is an inference time strategy, not involving any training, then how class statistics are affected? Please clarify.\n\n4. Experiments need to be done on recent diffusion models, with higher quality and resolutions to actually see any meaningful improvement.\n\n5. Can this analysis be extended or applied to classifier guidance?\n\n6. In L294, whats FD _{DINOv2} ?\n\n7. The flow and format of the paper needs to be improved. The figures should be improved overall, both in quality and formatting (labels etc).", "was_revised": true, "final_justification": "My concerns are mostly addressed!", "rebuttal": "We really appreciate you find our work interesting and motivating. We are happy to have the chance to address your questions.\n\n---\n**Q.1:  I don’t see much improvement in Fig.7, guiding with PCs even decrease the quality, why that is the case?**\n\nA.1: We thank the reviewer for this insightful question, which gets to the heart of our contribution. The reviewer is correct that naively guiding with Principal Components (PCs) can degrade image quality, and explaining this phenomenon is the central motivation for our proposed CPC guidance. \n\n**Question 1: Not much improvement.** Fig. 7 focuses on the low‑noise regime where guidance mostly refines finer details rather than producing dramatic changes. Comparing row 1 (CFG) and row  2 (eq.17), both guidances sharpen image details: e.g. in sample 2 the texture of French fries become finer, and in sample 3 the car separates more clearly from the background with the guidance whereas the unguided one blends into the background.\n\n**Question 2: Why naive PC guidance fails?** On the other hand, guiding with PCs indeed harms quality—precisely the point we highlight (in line 355‑357):\n\n*  **PC guidance fails because it is a mix of both (1) generic, coarse structures and (2) the desired class specific, finer details.** From Figure 6 (b), it can be seen that by projecting the noisy image onto the conditional posterior PCs, the noisy images can be denoised. To achieve this denoising result, the PCs not only need to capture the class-specific information or finer image details, but also need to capture generic information such as the overall coarse structure of the sample. Guiding with PCs simultaneously amplifies coarse and class‑specific patterns, adding redundant information and causing oversaturation.\n\n*  **Actual CFG only enhances class-specific, finer details.** In contrast, CFG in the nonlinear (low noise) regime mainly makes the image sharper, implying that it selectively enhances class‑specific details, leaving coarse structure largely unchanged. To do so, CFG must distinguish those desired structures from the generic, coarser ones. \n\n* **How CPC selectively enhances class-specific finer details.** Our heuristic CPC guidance (eq.17) formalises this selectivity: by contrasting conditional and unconditional PCs it retains the fine, class‑unique directions while suppressing generic ones, yielding higher fidelity than naïvely boosting all PCs. Note from Figure 6 (c), the unconditional PCs also lead to decent denoising results, implying that the coarse structure is captured, but it lacks some finer details, for example, the structures of the cheese burger and fries are blurrier. The guidance in eq.17 basically contrasts the conditional and unconditional PCs, get rid of the redundant coarse structures and enhance the finer ones, hence achieving better quality compared to naively enhancing PCs.\n\n---\n\n**Q.2:  In Fig.7., why is the color of the car changed? What visual attributes are captured by PCs? How is the prompt alignment performed?**\n\nA.2: The guiding effects of eq.17 result from the combined effects of all the positive CPCs and some might correlate with color, others with shape, texture, etc. While the color change is visible, the more important effect is that the car’s silhouette becomes sharper and better separated from the background.\n\nAlthough we don’t know the exact attribute of a particular positive CPC, mathematically, they are the directions where conditional variance exceeds unconditional variance, i.e., they represent the unique class patterns (see appendix A). Ordinary PCs, in contrast, capture the largest overall variance, often coarse foreground–background structure. Prompt alignment therefore arises by enhancing directions unique to the prompt and attenuating generic ones.\n\n---\n\n**Q.3: How linear CFG improves CFG remains unclear in Fig.5.**\n\n A.3: In this work, we didn’t propose the linear CFG as a method that improves upon the standard CFG, but rather to use it as an analytical tool to better understand the underlying mechanism of CFG.\n\n In Figure 5 (b), although the linear CFG is slightly better for relatively large guidance $\\gamma$, it can be seen that for $\\gamma<5$, it performs worse than the actual CFG. Whether linear CFG performs better or not  depends on the specific class. For example, in Figure 29., for class ‘coffee mug’ and class ‘cheeseburger’, the actual CFG performs better compared to the linear CFG. \n\nThe key observation is the close quantitative match between the linear and actual CFG curves across various guidance strengths. This supports our claim that the linear CFG can serve as a good approximation of the actual CFG in the high‑noise regime. \n\nRegarding the slight outperformance of linear CFG in Figure 5, our conjecture is that for high noise levels, it is possible that the network fails to learn the PCs perfectly. As can be seen in Figure 1, compared to the linear model, where we compute the PC exactly, at $\\sigma$=80.0 and 42.415, the generated samples of deep diffusion models tend to be noisier. This imperfect learning might be a factor explaining where the improvement comes from.\n\n---\n\n__Q.4: How can sample statistics be influenced by CFG?__\n\nA.4: CFG steers the diffusion trajectory; the resulting samples differ substantially from unguided ones, so their aggregate statistics (means, covariances, FID, etc.) also shift.\n\n---\n\n__Q.5:  Experiments need to be done in more advanced diffusion models__\n\nA.5: During rebuttal we repeated all experiments in section 3-4 on EDM-2 [1], which is a latent space diffusion model with latent dimension 4x64x64, and results in images with size 3x512x512. Moreover, the network architecture is also more advanced compared to the EDM-1 model we used in the original submission. The results remain consistent with those presented in the original submission and we summarized them as follows:\n\n* CFG improves the generation quality of the linear diffusion models constructed with covariances and means of the conditional and unconditional latent variables.\n\n* In high noise regime of the EDM-2 latent diffusion mdoel, linear CFG achieves qualitatively and quantitatively similar effects as the actual nonlinear CFG. Below we present the FD_DinoV2 scores for the various types of guidance applied to the noise range [11.05, 80.0], for class 'golden retriever':\n\n| Method / $\\gamma$ |   0 |    5 |    10 |    15 |    20 | \n|------------|----:|------:|------:|------:|------:|\n| Nonlinear‑CFG    |631.89   | 421.147 | 434.49 | 540.767 | 653.437\n| Linear-CFG  | 631.89 | 444.712 | 420.143 | 495.995 |658.213 \n| Mean-shift Guidance | 631.89 | 489.578 | 470.133 |553.23 | 728.914 \n| Positive CPC Guidance| 631.89 |  588.354 | 554.723 | 537.513 | 532.699\n| Negativ CPC Guidance     | 631.89 |  597.438 | 559.37 | 546.788 | 528.656 \n\nThe trends match Fig. 5 of the submission. \n\n* In low noise regime, the heuristic CPC guidance in eq.17 is capable of refining the finer details of the unguided samples.\n\n---\n\n__Q.6. Can the analysis be extended to classifier-guidance?__\n\n\nA.6.: Yes. Classifier guidance is mathematically equivalent to classifier‑free guidance; only the implementation differs: \nThe primary difference between the two methods is in their implementation:\n\n* Classifier Guidance uses an external, separately trained classifier to compute the guidance term.\n\n* Classifier-Free Guidance (CFG) derives the guidance term from the diffusion models themselves (conditional and unconditional scores) with Bayes rule.\n\n\n---\n\n__Q.7.: What is FD_Dinov2?__\n\nA.7: FD_Dinov2 [2] is similar to FID, which measures the similarity between two datasets using the Frechet distance (FD). The lower FD is, the more similar two datasets are. FID computes FD using the inception features while FD_Dinov2 computes FD with Dinov2 features. The work [2] has shown that FD_Dinov2 more accurately reflects human-perceptual quality compared to FID, and advocates the use of FD_Dinov2. In our experiments, we also tested with FID score and the trend still holds. We can include these extra results in the final version.\n\n---\n\n__Q.8. The flow and futures should be improved.__\n\nA.8: Due to the page limit, we have to condense some texts and figures. We will definitely improve them with the extra page given in the final version.\n\n---\n\n__References__:\n\n[1] Karras, Tero, et al. \"Analyzing and improving the training dynamics of diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Stein, George, et al. \"Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models.\" Advances in Neural Information Processing Systems 36 (2023): 3732-3784."}, {"review_id": "M69NJ0xESk", "reviewer": "Reviewer_uHYW", "summary": "The paper provides an in-depth analysis of Classifier-Free Guidance (CFG) in diffusion models, uncovering a bias caused by mean shift when the zero-mean assumption in data samples is violated. It proposes tuning the CFG coefficient to compensate for this bias. Empirical experiments support these findings, demonstrating the mean-shift phenomenon and offering a simple, practical adjustment to correct it.", "questions": "## Questions:\n* In line 171, as discussed in Section 2.1: For high to moderate noise levels, deep diffusion models can be well approximated by linear models, particularly when network capacity is limited and training is insufficient. Could you provide both an intuitive and a quantitative justification for why the linear approximation is considered sufficiently accurate in this regime?\n\n* In line 178, it is stated that \"CFG significantly increases the inter-class separation.\" Given that FID is computed from the overall sample mean and covariance, how does increased inter-class separation affect the overall FID score? Could this separation inadvertently inflate FID despite visual improvements?\n\n* Based on the insights presented in this paper, do you envision any new directions or improvements to the design of existing CFG methods?", "limitations": "## Limitations:\n\n* The study evaluates only EDM as the diffusion model; it remains unclear whether the observed effects generalize to other diffusion architectures.\n\n* The paper does not offer actionable insights or design improvements for existing Classifier-Free Guidance (CFG) methods based on its analysis.\n\n* There is a lack of quantitative evaluation assessing how different CFG configurations influence generation quality.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\nThe analysis of CFG is thorough. The authors first demonstrate that naive conditional sampling tends to yield low-quality samples that lack distinct class-specific features, which they attribute to the overlap in class covariance matrices.\n\n\n## Weakness\n\n* The setup for linear Gaussian diffusion is not clearly defined—how well does it approximate real-world diffusion processes?\n\n* The rationale for applying Contrastive Principal Component Analysis (CPCA) is unclear; further justification for its suitability would strengthen the analysis.\n\n* The study evaluates only EDM as the diffusion model. It's uncertain whether the observed effects generalize to other types of diffusion models.\n\n* The paper lacks actionable insights or design implications for improving existing Classifier-Free Guidance (CFG) based on its findings.\n\n* There is insufficient quantitative evaluation of how varying CFG configurations affect generation quality.", "was_revised": true, "final_justification": "The paper presents an in-depth analysis of Classifier-Free Guidance (CFG) in diffusion models, revealing a bias from mean shift when the zero-mean assumption is violated. It proposes tuning the CFG coefficient to correct this bias, supported by empirical evidence demonstrating the phenomenon and a practical adjustment.\n\nIn the rebuttal, the authors justify the use of linear approximation and provide additional experiments with the EDM-2 model to reinforce their claims.", "rebuttal": "We really appreciate your thoughtful feedback and are encouraged that they found our analysis thorough. We address each question in turn.\n\n___\n\n__Q.1: The setup for the linear model is not clear-how well does it approximate real-world diffusion models.__\n\nA.1: \n\n**Setup:** The setup is as follows: For a given dataset, we can analytically construct the optimal linear model with the empirical data covariance and mean based on equation 8. Once constructed, this linear model is used in the exact same manner as a deep diffusion model for sampling and guidance. More details can be found in Appendix C. \n\n**How well linear model approximates deep diffusion model.** The similarity between linear and nonlinear diffusion models has been extensively studied in several recent works [1-3]. In summary, linear diffusion models well approximate the actual diffusion models in high noise levels to moderate noise levels but differ in low-noise levels.\n\n---\n\n__Q.2: The rationale of applying Contrastive Principal Component Analysis (CPCA) is unclear.__\n\nA.2: The CPCA framework was not a priori choice for out study. Rather, the mathematical structure of CPCA emerged naturally from our analysis of classifier-free guidance in a linear model.\n\n* Inspired by the CFG’s similar effects on both linear and nonlinear models, as illustrated in Figure 2, we adopted the linear model as a prototype for understanding CFG. \n\n* When we derived guidance term in this linear setting, it naturally contains the CPC guidance $g_\\text{cpc}$ in eq.12, which involves $\\Sigma_c-\\Sigma_{uc}$, the difference between the conditional covariance and unconditional covariance. \n\n* Given this matrix, the most direct way to understand its effect is through its eigendecomposition. The eigenvectors with positive eigenvalues mathematically spans the subspace that best fits the conditional data while being as far as possible from the unconditional dataset, hence representing the class-specific feature while the eigenvectors with negative eigenvalues represent directions irrelevant to the given condition. \n\nThis specific analysis—examining the eigenvectors of a covariance difference matrix—is what has been formalized in the literature as CPCA. Therefore, our citation is intended to **acknowledge the established terminology** for this mathematical technique, not to indicate that we are simply applying a pre-existing method. Our contribution is the demonstration that this structure is fundamental to the mechanism of CFG. \n\n\n---\n\n__Q.3: It is unclear how the findings generalize to other diffusion models.__\n\nA.3:  During rebuttal we repeated all experiments in section 3-4 on EDM-2 [5], which is a latent space diffusion model with latent dimension 4x64x64, and results in images with size 3x512x512. Moreover, the network architecture is also more advanced compared to the EDM-1 model we used in the original submission. The results remain consistent with those presented in the original submission and we summarized them as follows:\n\n* CFG improves the generation quality of the linear diffusion models constructed with covariances and means of the conditional and unconditional latent variables.\n\n* In high noise regime of the EDM-2 latent diffusion mdoel, linear CFG achieves qualitatively and quantitatively similar effects as the actual nonlinear CFG. Below we present the FD_DinoV2 scores for the various types of guidance applied to the noise range [11.05, 80.0], for class 'golden retriever':\n\n| Method / $\\gamma$ |   0 |    5 |    10 |    15 |    20 | \n|------------|----:|------:|------:|------:|------:|\n| Nonlinear‑CFG    |631.89   | 421.147 | 434.49 | 540.767 | 653.437\n| Linear-CFG  | 631.89 | 444.712 | 420.143 | 495.995 |658.213 \n| Mean-shift Guidance | 631.89 | 489.578 | 470.133 |553.23 | 728.914 \n| Positive CPC Guidance| 631.89 |  588.354 | 554.723 | 537.513 | 532.699\n| Negativ CPC Guidance     | 631.89 |  597.438 | 559.37 | 546.788 | 528.656 \n\nThe trends match Fig. 5 of the submission. \n\n* In low noise regime, the heuristic CPC guidance in eq.17 is capable of refining the finer details of the unguided samples.\n\n---\n\n__Q.4: The paper lacks actionable insights to how to improve CFG based on the findings. Do you envision any new directions or improvements to the design of existing CFG methods ?__\n\nWe'd like to emphasize that our work primarily aims to provide a theoretical understanding of CFG—i.e., to offer insights into an observed phenomenon—rather than to propose practical improvements for model performance. We believe that an investigation of the mechanisms of CFG carries significant value on their own. Nevertheless, our findings suggest several potential avenues for future research both in application side and theoretical side:\n\n* Our observation that applying the mean-shift guidance to the diffusion trajectory yields improved conditional samples implies the existence of local class-specific clusters in the noisy data manifold learned by diffusion models. Specifically, in our extra experiments after the submission of the original draft, we find that simply initializing noise $x_T$ from a mean-shifted Gaussian distribution $\\mathcal{N}(\\gamma(\\mu_c-\\mu_{uc}), \\sigma^2(t) I)$, with no additional guidance applied, can improve the overall generation quality. The existence of the class-specific clusters has not been widely discussed or known in the literature, and we hope our results will motivate further theoretical investigation into the geometry of conditional diffusion models.\n\n* The main insight is that CFG improves the sample quality by (i) enhancing class-specific information and (ii) suppressing generic features prevalent in the unconditional data. We hypothesize that explicitly incorporating this dual objective into the diffusion model’s training process may lead to models capable of generating high-quality images without the need for guidance. Indeed, we have made progress in this direction in our ongoing project.\n\n* Our findings also reveal that PCA may not be robust in capturing class-specific patterns. As mentioned in the conclusion section of our paper, many recent works (e.g., [2], [3]) apply PCA in an embedding space to extract class-related concepts or structural information. Given that our results suggest CPCA is more effective for uncovering class-specific patterns, we believe investigating CPCA’s utility in these existing methods could be a promising research direction.\n\n---\n\n__Q.5: There is insufficient quantitative evaluation of how varying CFG configurations affect generation quality.__\n\nA.5: The main hyperparameters of CFG is the guidance strength $\\gamma$, and most of our experiments are conducted over a wide range of $\\gamma$'s. If “configuration” refers to something else, please clarify and we will be happy to provide additional results.\n\n---\n\n__Q.6: Could you provide both an intuitive and a quantitative justification for why the linear approximation is considered sufficiently accurate in this regime?__\n\nA.6: \n\n* Quantitatively, the similarity between linear diffusion model and nonlinear models have been extensively studied in previous works [1-3]. In particular, it is shown in [3] that this similarity is most pronounced when the model capacity is low or the training is insufficient. \n\n* Theoretically, given finite number of training data, which matches the practical setting where the model is trained on a fixed dataset with finite samples, the optimal diffusion denoiser admits a closed-form expression as a kernel density estimator, which is approximately linear for high noise levels (please refer to section D.8 of [2] for a proof).\n\n* However, in practice, the linear diffusion models and nonlinear diffusion models share similarity not only in very high noise levels, but also in moderate noise levels. Why this happens is not clear yet, but intuitively, the deep networks, especially when its size is small and the training is not sufficient, can favor learning simple structures of the data, such as linear structure, data covariance, rather than learning more complex structures such as higher order momentum. Such bias is known as the simplicity bias of deep networks [4], observed in many different settings such as classification and regression and has been well-documented. \n\n---\n\n__Q.7.:Could this separation inadvertently inflate FID despite visual improvements?__\n\nA.7: Yes, you are absolutely correct. A very high class-separation can reduce sample diversity and inflate FD metrics. However, since the naive conditional sampling leads to samples that lack separation compared to training data (the ground truth), if we choose the guidance strength carefully to improve inter-class separation by the right amount, the FID can be improved. \n\n\n\n\n\n\n\n---\n__References__:\n\n[1] Wang, Binxu, and John J. Vastola. \"The hidden linear structure in score-based models and its application.\" arXiv preprint arXiv:2311.10892 (2023).\n\n[2] Wang, Binxu, and John J. Vastola. \"The unreasonable effectiveness of gaussian score approximation for diffusion models and its applications.\" arXiv preprint arXiv:2412.09726 (2024).\n\n[3] Li, Xiang, Yixiang Dai, and Qing Qu. \"Understanding generalizability of diffusion models requires rethinking the hidden gaussian structure.\" Advances in neural information processing systems 37 (2024): 57499-57538.\n\n[4] Kalimeris, Dimitris, et al. \"Sgd on neural networks learns functions of increasing complexity.\" Advances in neural information processing systems 32 (2019).\n\n[5] Karras, Tero, et al. \"Analyzing and improving the training dynamics of diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024."}, {"review_id": "6JZNxgu1up", "reviewer": "Reviewer_VtnW", "summary": "This paper investigates the phenomenon of classifier-free guidance via the lens of linear diffusion models. The authors first show empirically that a linear diffusion model can approximate the results of diffusion models, then start the analysis from the linear diffusion model. The main proposition from the authors is that the CFG enables the sampling process towards the following three terms: (1) mean-shift towards the class mean; (2) a positive Contrastive Principal Components (CPC) term amplifying the class-specific features; and (3) a negative CPC term that suppresses generic features prevalent in unconditional data. Finally, the authors also discuss the extension to non-linear cases.", "questions": "See weaknesses above", "limitations": "Yes, the authors have discussed limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I have worked on some practical aspects of diffusion models, but not the theoretical side. I tried my best to follow the maths and compared them with my observations in actual experiments. Based on these, I have the following comments.\n\n# Strengths\n\n- It is intriguing to see the connection between diffusion models and a simplified linear diffusion formulation (Fig. 1).\n- The observation and statistics of class specifity is reasonable and make the paper well-motivated (Sec. 3.1).\n- The authors proposed the three terms explaining the intra-class and inter-class relationships, which aligns well with the intuition of how CFG assists generation.\n- The experiments conducted by the authors, ruling out the assumptions and factors gradually, further reveal the insights of understanding the effects of the three terms.\n\n# Weaknesses\n\nI am not working on the theoretical side of the diffusion models, so my question might not apply to the theory researchers. Specifically, I have reserved opinions toward the Equation 12, the ODE formula splitting into the three terms proposed by the authors.\n\nMy concern is that: under the assumptions of the linear model and Gaussian distribution, it seems intuitive that a CFG formula SHOULD correspond to the three terms: $p(x|c)$ --> mean shift, $p(x|c) - p(x| \\phi)$ the components between class-specific features and unconditional features. It is true that calculating the weights of the terms is challenging, but I am curious about how do the theory researchers judge that a conclusion is non-trivial?", "was_revised": true, "final_justification": "The author's rebuttal address my concerns and better clarifies the contributions. Therefore, I would increase my score.", "rebuttal": "Thanks for the thoughtful feedback!\n\nWe agree that, once one adopts the optimal linear assumption, the three‑term decomposition in Eq. (12) follows naturally. Our contribution, however, does not rest on the algebraic split alone but on the framework and the empirical validation that follow from it:\n\n(i) **We Introduce the optimal linear score model as a controlled test‑bed for guidance.** Our first key finding is that CFG manifests similar visual perceptual improvement on the linear diffusion models, making it a meaningful stand‑alone playground for analysis.\n\n(ii) **We show that the linear model accurately mimics real diffusion models at high–to-moderate noise levels.** We believe a good model need not be complex but it must match practice in certain aspects. The linear model does so in high-to-moderate noise levels, yet remains analytically transparent.\n\n(iii) **We Map linear‑world insights back to nonlinear models and delineate the failure cases.** We identify where the linear view breaks (low‑noise levels). But by showing that the proposed heuristic guidance in eq.17 could lead to similar effects as the actual CFG, we believe the insights from the linear model can still shed light on the practical settings.\n\n(iv) **The Insights Generate New, Actionable Research.** We believe our findings suggest several potential avenues for future research both in application side and theoretical side:\n\n* Our observation that applying the mean-shift guidance to the diffusion trajectory yields improved conditional samples implies the existence of local class-specific clusters in the noisy data manifold learned by diffusion models. Specifically, in our extra experiments after the submission of the original draft, we find that simply initializing noise $x_T$ from a mean-shifted Gaussian distribution $\\mathcal{N}(\\gamma(\\mu_c-\\mu_{uc}), \\sigma^2(t) I)$, with no additional guidance applied, can improve the overall generation quality. The existence of the class-specific clusters has not been widely discussed or known in the literature, and we hope our results will motivate further theoretical investigation into the geometry of conditional diffusion models.\n\n* The main insight is that CFG improves the sample quality by (i) enhancing class-specific information and (ii) suppressing generic features prevalent in the unconditional data. We hypothesize that explicitly incorporating this dual objective into the diffusion model’s training process may lead to models capable of generating high-quality images without the need for guidance. Indeed, we have made progress in this direction in our ongoing project.\n\n* Our findings also reveal that PCA may not be robust in capturing class-specific patterns. As mentioned in the conclusion section of our paper, many recent works (e.g., [2], [3]) apply PCA in an embedding space to extract class-related concepts or structural information. Given that our results suggest CPCA is more effective for uncovering class-specific patterns, we believe investigating CPCA’s utility in these existing methods could be a promising research direction.\n\n**In summary, our contribution is the systematic hypothesis, test and refine loop that bridges a tractable model and real-world diffusion models, clarifying what transfers, what doesn't and why. We therefore see the work less as pure theory and more as theory‑guided experimentation that advances practical understanding of CFG.**"}]

Author rebuttal:

Meta-review: (a) summary: The paper revisits the classifier-free guidance (CFG) in diffusion models with the aim to provide some insights and justification for its good performance. To do so, it relies on a simplified linear denoising model which allows for analysis of the mean and covariance components of the (de-noised) data. It attributes the low quality of pure conditional sampling to the lack of class-specificity in the class-conditional covariance and argues that this is mitigated in CFG by amplifying class specific features (positive contrastive principal components) and suppressing generic features (negative contrastive PC). It supports this argument analytically by decomposing the CFG guidance in the denoising step into the positive/negative CPC and a mean-shift terms. Though analytical study of nonlinear denoisers is challenging, the authors argue that nonlinear CFG guidance shares similar underlying mechanism. The study is supported by both proofs of the analytical derivations and experimental exploration of the guidance effects.

(b) strengths: Theoretical analysis of the CFG helps to improve the understanding of the underlying mechanism. The decomposition into the three terms (mean shift, positive CPC and negative CPC) is in retrospect natural and intuitive. The theoretical analysis of the CFG decomposition is supported by empirical exploration of both linear and non-linear CFG denoising regimes.

(c) weaknesses: Though the provided theoretical results are interesting, it is unclear how they can be utilized in practice. Experimental exploration limited to EDM-type of diffusion model and it is unclear if/how it translate to other more recent/advanced types of models.

(d) reasons to accept: The paper explores an important type of model (CFG) whose functioning is so far not well understood. The theoretical 
analysis is solid and is supported by relevant empirical evidence, the outcomes are well formulated and presented in an manner supporting natural intuitions. The provided insights can trigger further research into conditional diffusion generations leading to improved models.
I propose a spotlight presentation since CFG is a well known method in the diffusion modelling area and hence the results may of interest for a substantial part of the audience. 
They are, however, not so groundbreaking to deserve an oral.

(e) review discussions: Initial reviews had been cautiously positive recognizing the technical quality and the potential of the paper in terms of interest for the community. In addition to few clarifying technical questions, the main concerns were about applicability to modern more complex diffusion models and practical usability of the provided insights. The authors actively interacted during the rebuttal, clarified the technical questions, provided additional results over more complex diffusion models and argued that the primary purpose of the paper is not proposing practical improvements but providing better understanding of the CFG with the potential to be elaborated upon in future research. All reviewers increased their scores to 5 in the follow up of the discussion. The authors summarized the main concerns and how they will address these in the camera-ready version of the paper in their final remark. I find these appropriate, realistically doable and improving the paper in the right direction. Overall, a solid paper worthy the interest of the community.
Decision: Accept (spotlight)