Inner Classifier-Free Guidance and Its Taylor Expansion for Diffusion Models
Authors: Shikun Sun
Longhui Wei
Zhicai Wang
Zixuan Wang
Junliang Xing
Jia Jia
Qi Tian
Venue: ICLR 2024 poster (2024)
Keywords: diffusion models
classifier-free guidance
Primary area: generative models

Abstract: Classifier-free guidance (CFG) is a pivotal technique for balancing the diversity and fidelity of samples in conditional diffusion models. This approach involves utilizing a single model to jointly optimize the conditional score predictor and unconditional score predictor, eliminating the need for additional classifiers. It delivers impressive results and can be employed for continuous and discrete condition representations. However, when the condition is continuous, it prompts the question of whether the trade-off can be further enhanced. Our proposed inner classifier-free guidance (ICFG) provides an alternative perspective on the CFG method when the condition has a specific structure, demonstrating that CFG represents a first-order case of ICFG. Additionally, we offer a second-order implementation, highlighting that even without altering the training policy, our second-order approach can introduce new valuable information and achieve an improved balance between fidelity and diversity for Stable Diffusion.

Published as a conference paper at ICLR 2024
INNER CLASSIFIER-FREE GUIDANCE AND ITS TAYLOR EXPANSION FOR DIFFUSION MODELS
Shikun Sun 1,, Longhui Wei", Zhicai Wang", Zixuan Wang" 12, Junliing Xing Jia Jia 1.2 & Qi Tian
Tsinghua University, 2BRist,, HHuawei Inc., UUniversity of Science and Technology of China (ssk21, wangzixul xx21 jemails. tsinghus. edu, en, weil weilh2568@gmat com.
wangzhicemail. ustc. edu. cn, {ilxing, jjiajetsinghus. edu. cn tian. baan.gi@hhuawei.
ABSTRACT
Classifier-fre guidance (CFG) is a pivotal technique for balancing the diversity and fidelity of samples in conditional diffusion models. This approach involves utilizing a single model to jointly optimize the conditional score predictor and unconditional score predictor, eliminating the need for additional classifiers. It delivers impressive results and can be employed for continuous and discrete con- dition representations. However, when the condition is continuous, it prompts the question of whether the trade-off can be further enhanced. Our proposed inner classifier-free guidance (ICFG) provides an alternative perspective on the CFG method when the condition has specific structure, demonstrating that CFG rep- resents a first-order case ICFG. Additionally, we offer a second-order imple- mentation, highlighting that even without altering the training policy, our second- order approach can introduce new valuable information and achieve an improved balance between fidelity and diversity for Stable Diffusion.
INTRODUCTION
Diffusion models have garnered significant achievements in tasks involving image and audio gener- ation (Sohl-Dickstein et al., 2015; Ho et al., 2020; Rombach et al., 2022; Podell et al., 2023; Huang et al., 2023; Wang et al., 2024). These models exhibit comparable, and in some cases, superior performance to GAN-based models (Brock et al., 2019) and autoregressive models (Razavi al., 2019) regarding diversity and fidelity. Notably, text-based image generation models have emerged as particularly successful examples, including Stable Diffusion (Rombach et al., 2022), SDXL (Podell et al., 2023), DALL-E 2 (Ramesh et al., 2022).
There are primarily two approaches to introducing or enhancing guidance in diffusion models: using classifiers (Dhariwal & Nichol, 2021) and employing classifier-free guidance (Ho & Salimans, 2022) (CFG). In the case of classifiers, an external trained classifier is employed to guide the diffusion model at each timestep towards achieving a higher probability according to the classifier's 's judgment, which is also be extended to energy-based guidance (Zhao et al., 2022: Lu et al., 2023; Sun et al., 2023), On the other hand, classifier-free guidance involves utilizing single model to optimize the conditional score predictor and unconditional score predictor jointly. The discrepancy between these predictors is then employed as guidance, which is subsequently added to the score function. This approach eliminates the need for additional classifiers and has delivered impressive results (Ho & Salimans, 2022).
Nevertheless, none of these methods impose specific constraints on the condition space, resulting in the underutilization of the benefits of continuity when the condition space continuous. Our focus lies in exploring whether the nature of continuity can be effectively applied to CFG. Moreover, in the case of widely used text-based image diffusion models like Stable Diffusion (Rombach al., 2022), the complexity of the text encoder (Radford et al., 2021) raises the question of whether a structured continuous space exists. This space could potentially enhance the balance between fidelity and diversity in generated samples.
"Corresponding author

---

Published as a conference paper at ICLR 2024
To address these concerns, propose novel interpretation of CFG. Our approach assumes that the condition space possesses a (local) cone structure. Under this assumption, CFG can be seen as a first-order Taylor expansion of our proposed inner classifier-fre free guidance (ICFG) method. Building on this interpretation, we further introduce a second-order Taylor expansion of ICFG and reveal alternative energy-based formulation. Surprisingly, we discover that the second-order Taylor expansion of ICFG yields enhancements for Stable Diffusion, even without modifying the training policy. This finding suggests the existence structured continuous condition space that has the potential to enhance the sample performance of Stable Diffusion further.
To summarize, our main contributions are three-fold as follows:
We introduce ICFG and analyze the convergence of its Taylor expansion under specific conditions.
We demonstrate that CFG can be regarded as first-order ICFG and propose second-order Taylor expansion for our ICFG.
We apply the second-order ICFG to the Stable Diffusion model and observe that, remark- ably, our new formulation yields valuable information and enhances the trade-off between fidelity and diversity, even without modifying the training policy.
2 BACKGROUND
2.1 DIFFUSION MODELS
Diffusion models encompass various formulations, and we will provide a brief overview of score- based diffusion models (Song & Ermon, 2019; Song et al., 2021b;a; Sun al., 2023; Ni et al., 2023) (SBDMs) as they offer a solid foundation for guidance methods. SBDMs employ a stochastic differential equation (SDE) to diffuse the data distributions towards known distributions, typically Gaussian distributions. By learning the necessary information to reverse the diffusion process while preserving the marginal distribution, can sample from the known distribution and subsequently reverse the SDE. This process is equivalent to sampling directly from the data distribution.
Denote the unknown dataset distribution as g(xa), where X Rd. We aim to sample from g(xo) We also have the terminal distribution (xx)), where Xy E Rd To connect these two distributions, we introduce a forward diffusion process {x) where (xx;) or (x(x) represents the distribu- tion of x,. We assume that this diffusion process follows a SDE:
dx = f(x, t) + gt)
where f(x. t) is the drift term, gt) is the diffusion coefficient, and w is the standard Wiener process. In the work by Song al. (2021b), it has been shown that to reverse the SDE while maintaining the marginal distribution, the score function s(x, 1) is the only required information. The score function defined as follows:
s(x, 1) V xx log (x:))
Then, the reverse-time SDE is:
dx = [f(x () s(t))s(xx))))))))))))))))) t)]dt + g(t)dw,
(3)
where the symbol w represents another standard Wiener process that is independent of the Wiener process in the forward diffusion process. The reverse-time SDE is equivalent the reverse-time diffusion process {x, (T([o. Then we can sample from the known distribution g(xr), and reverse the SDE to get the sample from the data distribution g(xo).
The works of Vincent (2011); Song et al. (2021b) present feasible method to estimate the score functions of complex datasets using a neural network sx, t). The optimization objective is defined as follows:
/'
8- arg min
Egtxe (oo (so) (A(e) |>( t) Vx. log goo (xp xo)| dt,
(4)
(x,|Xx)) xn) is the transition probability from Xo Xt- where X(1) weighting function, and (x;
2

---

Published as a conference paper at ICLR 2024
Furthermore, consider Eq. (1) where we have x; 0,X0 + 8iz, with being standard Gaussian distribution, and Ct and Bg representing the corresponding coefficients. In many scenarios, the diffu- sion score is parameterized as (x, = -3,s"(x, t). Moreover, in the case of conditional diffusion models, where the data x is accompanied by conditioning variable c, the only modification is to include an input to e,, resulting in e cx( c, t).
2.2 CLASSIFIER GUIDANCE FOR DIFFUSION MODELS
Dhariwal & Nichol (2021) introduce classifier guidance for diffusion models to enhance control over conditions. Assume that the learned conditional diffusion score is denoted as e" (xx c, t), and we have a set of classifiers p (c|x,) that predict the condition с based on the diffused data x, at various time steps In this case, we can modify the diffusion score as follows:
E (x,,,,, = e (Xx, c, Xx, log (c|x,) = -B,Vx (x=lc( + w log (c|x:)] (5) where w represents a weighting factor that controls the strength the guidance. The modified diffusion score, denoted as e" (xx, c, L), replaces the original diffusion score e (x c, in the sam- pling process. At each time step I, E (xe, c,t) serves as the diffusion score for a new conditional distribution.
(x(((c x q" (x
(6)
the conditional distribution (x multiplied by P (c(x))" This indicates that the modified diffusion score assigns a higher probability to the data x, that is more likely to be associated with the condition Consequently, the modified diffusion score allows for trade-off between sample diversity and sample fidelity.
It can be observed that 7" (x/|c) can be
unconditional diffusion score. For the unconditional diffusion score e (x, t), using the same set classifiers, the modified diffusion score is given by:
The classifier guidance also be applied
(x,, t) cx -)- (w+ V., log p. (c)x,) - Vx, [log g (xx) (w + 1) log log (:)
The corresponding guided intermediate distribution is: 7 (x+(c) x xg (c(x))
While the experiments on guiding the unconditional diffusion score may not have yielded as re- markable results as the conditional case initially (Dhariwal & Nichol, 2021), this formula can still be further connected to the concept of classifier-free guidance.
2.3 CLASSIFIER-FRE GUIDANCE FOR DIFFUSION MODELS
To avoid training classifiers, Ho & Salimans (2022) propose an alternative approach called classifier- free guidance (CFG) for diffusion models. The main idea behind CFG is to use single model to simultaneously fit both the conditional score predictor and the unconditional score predictor. This achieved by randomly replacing the condition e with Ø (an empty value). By doing so, one can obtain the conditional score predictor e" c, t) and the unconditional score predictor t). which is equivalent to e e((x 2x,0,1) Then, because
Vx, [cox)( = Vx, (log((x:/0) log ((x:) + [o)]
= Vx. [log ((x:(c) - log q(x)),,
which indicates that after applying the operator Vx,, we can replace the last term of Equation (7) with log 9d (xx|c) log 9º (x:) to achieve similar effect. Then we get the enhanced diffusion score:
el (xx, c, t) = (w + 1)e"
c, we (x),
-3, V x. [lgg d (x+|c) + w(log 9 (xelc) log 9d (x:)) = logg d (xx) + (w + 1)(log 9e (xx/c) log 9d (xx))),
(10)
whose enhanced intermediate distribution is:
ja (x/(c) x xx (x(x) (x(
w+1
(11)
(x,,)
3

---

Published as a conference paper at ICLR 2024
It is worth noting that both of these guidance methods can be regarded more general form of energy-based guidance (Zhao et al., 2022; Lu et al., 2023). In this case, the formulation of the intermediate time distribution is:
de (x x o (xx)e (w+1)((x)) (xxx)e-
(12)
where ((x,) is an arbitrary energy function.
3 INNER CLASSIFIER-FRES GUIDANCE
Firstly, consider the enhanced intermediate distribution o (xx/c) obtained by Eq. (8) or Eq. (11), given the condition The question is whether these enhanced distributions follow the same diffu- sion forward process the original intermediate distribution q" (xx). We have:
Theorem 3.1. Given condition the enhanced transition kernel go. (x4 (x, c) by Eq. (8) Eq. (11) equals to the original transition kernel foe (X, [X/XX,, c) = 1or (Xp Xn) does not hold trivially. Specifically, when = 0, the equation holds.
The proof and discussions of Theorem 3.1 is in the Appendix A. Theorem 3.1 suggests that in the majority of cases, the enhanced intermediate distribution o (xxelc) at different timesteps / does not adhere to the same SDE as the original intermediate distribution (x(,) or the original conditional intermediate distribution q" (x+|c). which is also mentioned the work of Lu et al. (2023); Du et al. (2023).
There are two important clarifications to make regarding Theorem 3.1:
qd (xo xx) appears never be 8 distribution. However, under certain conditions, such as a low noise level or when Xy is sparse, it can be approximately close to a 5 distribution.
Even if oor [x, Xn, c) deviates from the original diffusion process, The sampling process may still be effective based the underlying principles of Langevin dynamics.
To address this issue, we incorporate the guidance strength w and represent the enhanced interme- diate distribution more clear form as y (x+|c, w + 1). For instance, in the context CFG, we can express (xxlc, + 1) x (x::)
(xx lc)
9" (x.)
Let 8 w + 1. It is worth noting that in Theorem 3.1, when 3 1, have gh (xr Xxo, 3) Lue (xr (x)). The question arises: Can we always ensure that = 1? other words, can we treat and the same variable, such that di. [xxx c, 3) due (x/xx) holds consistently? By doing so, we incorporate the condition strength into the condition variable itself, and we refer to this approach inner classifier-fie guidance (ICFG).
To establish a well-defined ICFG, it is necessary to impose certain structural assumptions on the space C of the condition The following assumptions made:
Assumption 3.1.
C is cone, which means 43 E R Ve E C.
For each E C. ||c|| represents the guidance strength and ||] represents the guidance direction.
For implementation purposes, as the origin C does inherently form cone, it is common practice to extend the existing meaningful space C to conform to a cone structure. This extended space is denoted as C = (Co + B(cc ccc- E R cc C}, where the vertex of the cone Cy not necessarily 0. Further details can be found in Appendix D.
Under Assumption 3.1, we define g (x+(cc = gº (xicc, (x,1BBc) Based on this definition, we can state the following Corollary 3.1.1:
Corollary 3.1.1. Given condition and the guidance strength w + we have:
8 (x. Xo. c. (8) dbc (x, (x0)-
4

---

Published as a conference paper at ICLR 2024
Corollary 3.1.1 indicates that for ICFG, the forward diffusion process consistently remains the same as the original forward diffusion process, given condition с and guidance strength B3.
We propose training policy that allows the model to assess the guidance strength based relation metric r(x, c). which measures the relationship between the condition and the data. The training policy is outlined in Algorithm 1.
Algorithm Training policy for ICFG
Require: r(x, c): similarity metric
Require: Puncond: probability of unconditional training
1: F Ex,er(x, c)
2: repeat
3:
(x. c(x.c)
4: с c/r(x,c)
5:
+- Ø with probability Puncond
6:
U
7:
(0.1))
8:
X, = a;Xo + B,e
9:
Take gradient step Volle - cc12
10:
until converged
TAYLOR EXPANSION OF ICFG AND ITS CONVERGENCE
In most cases, we use the extended cone C = [Bc]3 E Rt.c E Cj. Consequently, we need to consider the Taylor expansion of ICFG at 8 = 1 to estimate the score under conditions within C/C. The n-th order Taylor expansion of <x(x (3c) at / = 1 is given by:
2 (x,
हc (x, = (x, +
(3- - 1)a + +R(())
(13)
k!
03
1=
8=1
where (3) represents the remainder term. It is evident that CFG is a first-order Taylor expansion of ICFG at 3 = without R₁ (8) and with the following estimation:
Oea (x.|33)
(x:(c) (x, ( - c((x)
(14)
03
B=1
Then we will discuss the convergence of the Taylor expansion of ICFG in Eq. (13).
Firstly, considering this problem from the model space {τ6 (x, Vx, Ve), we judi- ciously choose the components of the neural network such that the function 7" (Bc) becomes analytic with respect to 3, then the convergence of the Taylor expansion near 8 = is guaranteed trivially.
Secondly, if we desire a bound for (3) for any 3 E 10, B), certain assumptions must be intro- duced. Before that, consider the specific quantitative relationships of Stable Diffusion, as depicted in Figure I. It becomes apparent that the estimation of 0e" (8c) quently, propose the following Assumption:
is relatively small. Conse-
03 A=1
(' at E [] is
Assumption 4.1. For all E N+, the k-th order partial derivative bounded by MA.
0
Then, we have:
Theorem 4.1. Under Assumption 4.1, the remainder term RB( of the n-th order taylor expansion of (x. 9c) at 1 is bounded by Mn11 Bat1, This bound converges to 0 when the sequence (w41). (
MA+1 ~o [v++I a+1 c
m+1)
By Stirling's formula, the proof will be discussed in the Appendix B.
5

---

Published as a conference paper at ICLR 2024
Figure 1: The quantitative relationships of Stable Diffusion during sampling. These figures show the relationship between the norm of the predicted score E" (x, ε (xx) and the distance norm between them during the 50 sampling timesteps CFG. The guidance strength for the first two figures is set to 3, while the last figures have guidance strength of 80. The caption for all figures states, "A photograph of an astronaut riding a horse."
IMPLEMENTATION OF SECOND-ORDER ICFG FOR STABLE DIFFUSION
For the pretrained popular Stable Diffusion, we present two approaches to implement the second- order ICFG. The first method follows straightforward Taylor expansion, outlined in Algorithm 2. Notably, when we use f two points (I1, V1), (322. 1/2) to estimate the gradient at #1t-22 the second-order term is unique. Further details will be discussed in the Appendix
Algorithm 2 Strict sample algorithm for second-order ICFG
Require: m: middle point for estimate second-order term
Require: wi guidance strength on conditional score predictor
Require: c: condition for sampling
Require: Require (f1, 12, [x increasing timestep sequence of sampling
Require: Sample Zz, <t): sample algorithm for diffusion models given Ze and Et
1: ZN ~ I)
2: for N....,
3: EL e (Z,, c) w(c"(zi, c) -e((z))
w m(1-m) (11 m)) + c) e (0, mc))
4: 2 Sample(z,,t))
5: end for
6: return %o
However, in practical scenarios, the second-order term may suffer from a large bias due to the un- changed training policy, and this bias is further amplified by the coefficient w². To address this issue, we propose an alternative approach called the non-strict sample algorithm, presented in Al- gorithm 3. This algorithm offers practical solution and be effectively applied to mitigate the aforementioned problem.
The only distinction between Algorithm and Algorithm 2 lies in assigning completely unre- stricted hyperparameter to the second-order term. This modification allows for more flexible control over the second-order term.
EXPERIMENTS
The primary motivation behind our experiments twofold. Firstly, we seek to showcase the efficacy of our new sampling algorithm in leveraging the continuity of C and incorporating valuable informa- tion to achieve an improved balance between diversity and fidelity, even in cases where does not exhibit "cone" structure apparently. Secondly, we aim to demonstrate that new training policy enables the model to capture better the inherent "cone" structure of To accomplish the first ob- jective, utilize the pretrained Stable Diffusion v1.5 model and apply our Algorithm 3 generate sampled images. More implementation details are in Appendix Nevertheless, our second-order ICFG continues demonstrate its advantages in these experiments. For the second target,

---

Published as a conference paper at ICLR 2024
Algorithm 3 Non-strict sample algorithm for second-order ICFG
Require: m: middle point for estimate second-order term
Require: w: first-order guidance strength on conditional score predictor
Require: et second-ordes guidance strength on conditional score predictor
Require: c: condition for sampling
Require: Require (11, l2, (N) increasing timestep sequence of sampling
Require: Sumple <t): sample algorithm for diffusion models given and
1: NN ~ I),
2: for i N N......1.. do
3: Et = = + w(ee(( (z; c) 15----
(zi))
+U mi] ((1 m)e" (zi) + me (zi, c) - -E) mc))
4:
2--1 Sample(zi,
5: end for
6: return Zo
Table 1: The results of varying the guidance strength and the condition space on the MS-COCO validation set.
Metei&Settings
FID J
CLIP Score (%)f
CFG
1.0
17.24
25.03
w 2.0
15.42
25.80
3.0
16.68
26.12
w 4.0
18.18
26.34
W 5.0
19.53
26.45
Ours 1.0, C Call
V
0.25/0.5/11 5/.0
16.40/17.34/20.70
25.46/25.71/255
w 2.0, C = Call
55.28/15,42/16.31
26.11/26.30/26.33
3.0, C Call
16.59/16.660 6..59/16.69/66.88
26.37/26.54/266
w 4.0, C = Call
17.98/18 06/18.20
w 5.0, C Call
19.35/19. 99////999.//9445
26.59/26.69/266
to = 1.0, C = Cnouns
16.33/17 2,2/1177.
83.02/24.88/24.23
w 2.0, C Coouna
15.22/15.23/1B
25.86/25.81/25..
w 3.0, C
Coous
16.59/16 1.666666661166
26.19/26. 18/26.09
w 4.0, C Cnouns
18.08/18 8.7/18.002
w 5.0, C Cnouns
19.33/19.31/19.47
06.48/26.49/26.44
clusively fine-tune the U-Net (Ronneberger et al., 2015) of Stable Diffusion v1.5 (Rombach al., 2022) with Low-Rank Adaptation (Hu et al., 2022; Ruiz et al., 2023).
6.1 SAMPLING WITH SECOND-ORDER ICFG
We employ the pretrained Stable Diffusion v1.5 model directly and utilize our Algorithm to gen- erate sampled images. The settings we follow consistent with those provided in the official repository of Stable Diffusion v1.5. The sampling algorithm employed is PNDM (Liu et al., 2022), and the default number of timesteps is 50. The evaluation the results, presented in Table 1, Table and Table 3 is based on two metrics: the Fréchet Inception Distance (FID) (Heusel et al., 2017) and the CLIP Score (Radford al., 2021). The FID metric is calculated by comparing 10,000 generated images with the MS-COCO (Lin et al., 2014) validation dataset, measuring the distance between the distribution generated images and the distribution of the validation dataset. On the other hand, the CLIP Score is computed between the 10,000 generated images and their corresponding captions by the model ViT-L/14 (Radford et al., 2021), reflecting the similarity between the images and the textual descriptions. In tables, the default configuration is set We systematically vary the guidance strength w and v the middle point m, and the condition space C to observe the impact of the second-order term. The designs of C.ll and Cnouns are in Appendix D.
2.0, 0.25.
1.1.

---

Published as a conference paper at ICLR 2024
FID-CLIP Score (w (2.0,2.0,3.00 (2.02200.330A40....0. 0,4.0,50)
FID-CLP Score (V 10, 0.25, 0.5, 0.2,, 1.01)
CeC_all
wm3.00 CHCCneous
19
x-
vmo.s 5 CmC
16.85
CnC_nouns
16.75
17.
16.70
16.00
Figure 2: The FID-CLIP Score of varying w, and C.
Table 2: The results of varying the middle points the MS-COCO validation set. Here
0.5,C C = Can.
Middle points
FID 4
CLIP Score (%)f
0.5
15.59
26.02
m = 0.8
15.58
26.28
0.9
15.54
26.27
m = 1.05
15.47
26.31
1.1
15.42
26.30
m = 1.2
15.43
26.28
6.1.1 VARYING THE GUIDANCE STRENGTH AND THE SPACE OF CONDITION
Here we experimentally validate the first primary claim in this paper: that second-order ICFG can improve the balance between FID and CLIP Score. By varying the guidance strength and space of condition, determine that the optimal balance between FID and CLIP Score achieved at 2.0, 0.25. The best is the same many other diffusion models (Bao et al., 2023b) Beyond this point, we observe a discernible trade-off between FID and CLIP Score as w increases. In the condition space Call, note the presence of trade-off between FID and CLIP Score increases. However, such a trade-off is not evident in the conditional space Cnouns- Additionally, observe that while the best FID score is obtained in Cnouns. superior balance between FID and CLIP Score is achieved in Canl- This finding suggests that the condition space Call exhibits a more favorable "cone" structure for processing.
6.1.2 VARYING MIDDLE POINTS
One of the key hyperparameters in the second-order ICFG is the selection of middle point, which is utilized to estimate the second-order term. Two primary factors influence the outcome in this regard. Firstly, if the chosen points are too close to each other, the estimated second-order term fails capture long-term changes adequately. Secondly, if the middle points relatively distant from either or 1, the model struggles to estimate the corresponding score. The observed "U" shape of the FID results presented in Table serves validate our analysis.
6.1.3 VARYING THE NUMBER SAMPLING STEPS
The number of sampling steps also influences the quality of the generated samples. We have ob- served that the CLIP Score remains relatively stable across different numbers of sampling steps, while the FID score improves as the number of sampling steps increases. This suggests that the sample quality improves with an increase in the number of sampling steps. However, it is worth
8

---

Published as a conference paper at ICLR 2024
Table 3: The results of varying the sampling steps the MS-COCO validation set.
Model& Settings
FID
CLIP Score (%)T
Ours
C Gall/noous
T 10
15.80/15.86
26.13/25.88
T 20
15.39/15.40
26.15/25.88
T 30
15.29/15.28
26.13/25.80
T 40
15.29/15.23
26. 1/25.86
T 50
15.28/15.22
26. 14.11/25.86
noting that even with small number of sampling steps, the initial matching degree between the generated text and the image is already quite good.
6.2 FEW-SHOT TININGUNING FOR STABLE DIFFUSION
To validate the efficacy of our training algorithm, we employ fine-tuning process on the pretrained Stable Diffusion v1.5 model using Algorithm We then compare the outcomes with those obtained through traditional fine-tuning By generating cases with varying inner 3 values, aim assess the capacity of the model to capture the inherent "cone" structure C. The results, presented in the Appendix E, demonstrate that our training algorithm yields improved tolerance when coupled with more substantial inner guidance.
DISCUSSION
ICFG offers a novel perspective for comprehending CFG and can be seen as an extension of CFG. One significant advantage of our ICFG approach is its simplicity in implementation. Furthermore, integrating second-order ICFG into complex conditions trained diffusion models is straightfor- ward, involving adding a few lines of code. By selecting suitable space to exploit continuity, we can effectively implement second-order ICFG. In cases where the condition space C exhibits a well-defined structure, extending the second-order ICFG higher-order ICFG becomes feasible.
We also offer an intuitive explanation of how the Taylor expansion operates. When we extend the CFG to higher-order Taylor expansion of ICFG, the corresponding enhanced transition kernel The (xx Xo, c) gradually aligns with the original transition kernel more smoothly. In simpler terms, the enhanced transition kernel &or (x Xo, c) becomes increasingly similar to the original transition kernel dbe (x, Xx)) This explains why the second-order ICFG can enhance the FID and CLIP Score balance.
Despite its advantages, ICFG also has few potential disadvantages. Firstly, the training policy of ICFG relies on more precise data pairs to accurately capture the guidance strength during training. However, this requirement can be alleviated by incorporating the similarity function r(x, c). Sec- ondly, the second-order ICFG necessitates three forward passes of the diffusion model to estimate the second-order term, which can lead to increased sampling time. Nevertheless, this issue can be mitigated by reusing the points for the pre-order term, thereby reducing the computational overhead.
CONCLUSION
We introduce ICFG, novel perspective on CFG. ICFG is an extension of CFG and can be read- ily implemented in training and sampling processes. We further propose the Taylor expansion of ICFG and analyze its convergence properties. By incorporating second-order ICFG, we mitigate the mismatch issue in the diffusion process that arises from CFG. Through our experiments on Stable Diffusion, validate the efficacy of our second-order approach. In future work, intend in- vestigate higher-order ICFG and anticipate further investigations into the application of ICFG in a wide array of diffusion models across diverse data modalities.

---

Published as a conference paper at ICLR 2024
9 ACKNOWLEDGEMENTS
This work is supported by the National Key R&D Program of China under Grant No. N0.22021Q1100
REFERENCES
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In CVPR, 2023a.
Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion scale. In Proceedings of International Conference on Machine Learning, 2023b.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In Proceedings of International Conference on Learning Representations, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Pro-
ceedings of Advances Neural Information Processing Systems, 2021.
Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jaseha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and memc. In International con- ference on machine learning, pp. 8489-8510. PMLR, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Proceedings of Advances in Neural Information Processing Systems, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In Proceedings of International Conference on Learning Representations, 2022.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffiusion probabilistic models. Proceed- ings of Advances in Neural Information Processing Systems, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In Proceedings of International Conference on Learning Representations, 2022.
Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to- tuuoooio generation with prompt-enhanced diffusion models. In Proceedings of International Conference on Machine Learning, 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of European Conference on Computer Vision. Springer, 2014.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models manifolds. In Proceedings of International Conference on Learning Representations, 2022.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy- energy-guided diffusion sampling in offline reinforcement learning. In Pro- ceedings of International Conference on Machine Learning, 2023.
Zixuan Ni, Longhui Wei, Jiacheng Li, Siliang Tang, Yueting Zhuang, and Qi Tian. Degeneration- tuning: Using scrambled grid shield unwanted concepts from stable diffusion. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 1900-8909, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiw:2307.01952. 2023.
10

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of International Conference Ma- chine Learning, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 2022.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Proceedings of Advances in Neural Information Processing Systems, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Proceedings of Medical Image Computing and Computer-Assisted Intervention, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings International Conference on Machine Learning, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Proceedings of Advances Neural Information Processing Systems, 2019.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training score- based diffusion models. In Proceedings of Advances in Neural Information Processing Systems, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proceedings of International Conference on Learning Representations, 2021b.
Shikun Sun, Longhui Wei, Junliang Xing, Jia Jia, and Qi Tian. Sddm: Score-decomposed diffusion models on manifolds for unpaired image-to-image translation. Proceedings of International Conference Machine Learning, 2023.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- 2011.
tation,
Zixuan Wang, Jia Jia, Shikun Sun, Haozhe Wu, Rong Han, Zhenyu Li, Di Tang, Jiaqing Zhou, and Jiebo Luo. Dancecamera3d: 3d camera movement synthesis with music and dance. arXiv preprint arXiv:2403. 13667, 2024.
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. EGSDE: Unpaired image-to-image translation via cncrgy-guided stochastic differential equations. In Proceedings of Advances in Neural Informa- tion Processing Systems, 2022.

---

Published as a conference paper at ICLR 2024
A PROOF OF THEOREM 3.1 AND COROLLARY 3.1.1
A.1 PROOF OF THEOREM 3.1
Proof. Firstly, we ignore the symbol 0. Classifier guidance and CFG with guidance strength have the following enhanced conditional probability:
1 ((x:(c)= Z, ((xx))(()xx))*t
(15)
where = w -dx dx,.
Suppose the enhanced transition kernel Tue (xe Xo. c) equals the original transition kernel goe (X1) xo), we have:
Te (x+|c)
(xr(xx|xx: c) (xolc)dxo
1
/ (xx w +
Zo
(16)
/ ((xe,x))((c(xx) nr dxo
=
Zo
1
(xxx) /
((x01x))(((xx0) " "xx
Zo
1
((xx))(((x)))
Z,
Because g(c)x,) (()x)x)))(((x)))))) We take the last two terms and then get the following equation:
1
/ 10x0
1
w+1
Zo
Z,
J ) -
Zo
[ q(xn[xx)))(())))))()))))) (cc)x)) w+1
Ze
(17)
+ xx (0) ) wu+1
Zoo
(xxx0//x0/xx((((())0)) 1+
Ze
To enhance clarity, let's consider straightforward scenario. Suppose X comprises only x with label c and x² with label c², Given с = c', the left side of Eq. (17) is as follows:
((8)) ( ||x x1 /63?)
10+1 q |x]) wr + |x q(c² |x])
-x:- (2166))))
((()))))())))))))))))))) |xx)) |rg
(2008)
- ((3?)
0
/ (e xxxxx=xaxx]|////000 x - 2/(07) x x0) w+1
(/(())
ell-xi-on x x.1 (1)9) + ell x311 (18)))
xx-x-aaxa (/182)
(18)
which is function of x4.
12

---

Published as a conference paper at ICLR 2024
For more general situation: We have N data pairs (x), c²). Given c', then the left side of Eq. (17) is
1
N - --xxxx-xxxx q((c (
Σ el x
w+1
Ei e |xx x 12000 )
e -xx x (2))2) x
>> 17/03
(19)
w+1
||x x 1)
En xx X 1(
Σ |== xx x /1 ((0))
el -X- - 2/(02)
And the radio of two enhanced intermediate distributions is
Ze Zo
(20)
x (13/(())
Then reconsider the context Eq. (17), it observed that the equation does not maintain universal validity. A contradiction between the two sides becomes apparent, as the left side is function of the stochastic variable x,, while the right side remains constant. However, it is easy to check equation holds when w 0, because when = 0, the left side and the right side of the last line of Eq. (17) are 1.
We specifically discuss the case of ((x))x;) being 5 distribution because (xx)) xx,)
like mexpll
--- dxoo which approaches an approximation of 8 distribution when is small or when the values of xo are highly sparse
A.2 PROOF OF COROLLARY 3.1.1
Proof. In this case, we have:
g(x:/c) gxxx133,, = =xxx|| Sc).
(21)
Treat the 8C an entire e, which is special case of 0 in Theorem 3.1.
B PROOF OF THEOREM 4.1
Proof. Consider the Lagrange form of (3)
ax+ [8c)
Rx(3)
03411
(B 1)++1
(22)
(k: 1)!
where & E [0,3]. Then get the upper bound of [R
Ma+1 (B 1)
(B)Il
(n + 1)!
(23)
Mat1 Br+1
< (n + 1)!
To establish more relaxed condition for the convergence of the sequence (M, In E N), we utilize Stirling's formula in Eq. (23) to obtain:
Mn+1 B (n + 1)!
M
eB
n+1
(24)
Vn
1n+1
n n+
13

---

Published as a conference paper at ICLR 2024
which indicates when n +oo, the sequence (3) In N) converges to 0 if
n+1 n
Mn+1 Vn++1 +1+11
eB
C THE UNIQUENESS OF THE SECOND-ORDER TERM
Proof. When we use 91- 21 of two points (x1,8)), (22, y2) to estimate the gradient at #120, For three data pairs (ro, yo), (I1, v (22 1/2), where (Iy - 2o (o) x1)("0 -22) o 0, no matter how we organize them, the estimated second-order gradient is uniquely determined as:
2 Toll + T13o I28/1 Tolh T13/2 T230
(x) (0) 3i (1n 52)
Let define 2o 0, 2j m, and 22 With these values, proceed to estimate the second-order gradient, which is given by:
2 m m)
((1 mmmm 90 + myy v2)
D IMPLEMENTATION DETAILS
D.1 THE CONDITION SPACE
In this paper, we have designed "cone" structures for the conditions of Stable Diffusion. All two kinds C' are extended from the tensors after CLIP model, whose dimension is 77 x 768.
Cal: We use the pretrained CLIP model to extract the text embedding Ctext the captions. We also get the extract embedding сœ of empty caption, then with the inner coefficient 3, we get the enhanced embedding Cœ + (Creet Co)
Cnouns: We utilize the pretrained CLIP model to extract the text embedding Ctext from the captions. Additionally, we obtain the embedding Cœ for an empty caption. By in- corporating the inner coefficient 3 and the indicator function Inouss, embedding. Specifically, we set the values corresponding to the positions of nouns to in I nouns. while the remaining values set O. The resulting enhanced emhedding is given by Cø + nouns (Crext Co).
modify the
D.2 DETAILS FIN-TUNING PROCESS
We set rank 4 and apply the Low-Rank Adaptation (Hu et al., 2022; Ruiz et al., 2023) to modify the attention layers of the U-Net (Ronneberger et al., 2015) of Stable Diffusion v1.5 (Rombach et al., 2022). We the Adam optimizer with learning the model for 300 epochs on small part of MS-COCO (Lin et al., 2014) dataset, which contains 30 images and their corresponding captions. We use the pretrained Stable Diffusion vI.5 model the initialization of the U-Net. We compare our fine-tuning policy with default fine-tuning policy.
of 1e and batch size of 8. We fine-tune
E SAMPLES AFTER FINE-TUNING
After the fine-tuning process, we proceed to compare the samples generated using different training policies. The corresponding results are presented in Figure 3 and Figure 4. Each sample is generated with the caption "a brown and white giraffe values of inner B set to 1.0, 1.2, 1.4, and 1.6. Notably, performance compared to the default training policy when 3 assumes relatively larger values. This observation suggests that training policy effectively captures the inherent "cone" structure of C.
field of grass" and arranged from left to right, with training policy demonstrates superior
14

---

Published as a conference paper at ICLR 2024
Figure 3: Generated images of different inner 33 of our training policy.
Figure 4: Generated images of different inner of default training policy.
15

---

Published as a conference paper at ICLR 2024
Table 4: FID results U-VIT of CFG and ICFG.
Steps
5w
10w
15w
20w
25w 30w 35w 40w 45w 50w 55w 60w 65w 70w 75w 80w
CFG 34.23
13.64
11.26 10.38 9.78 8.98 8.98 8.76 8.58 8.52 8.37 8.37 8.27
8.32 8.39 8.10
ICFG 24.69
13.51
11.00
10.13
9.69
9.09
8.82 8.68
8.54 8.41 8.35
8.21
8.29
8.15
8.10 7.92
Table Experiments about the speedup.
Method
Time (seconds)
Extra Time
U-Net Computation
FID
CLIP Score
CFG
10.43 = 0.23
0%
100%
15.42
25.80
2nd-order ICFG
15.01 0.31
43.91 %
150%
15.28
26.11
0.2-0.8 2nd-order ICFG
13.17 = 0.26
26.27%
130%
15.29
26.03
F VISUAL RESULTS COMPARISON
We conducted a comparison between CFG and second-order ICFG with 5.0 and 0.25.
The visual comparisons presented in Figure 5. It is evident from the images that second-order ICFG outperforms CFG, producing images with better-rendered hands and closer alignment to provided prompts.
G EXPERIMENTS ANOTHER FRAMEWORK
train another framework, U-VIT (Bao al., 2023a), with resolution of 256x256 the COCO dataset from scratch to fully explore the capabilities of our ICFG. The FID results are listed in Table 4.
H EXPERIMENTS ABOUT THE SPEEDUP
We conducted experiments performed 100 samplings to calculate the timings and utilized 10,000 images for the computation of FID and CLIP scores, The results shown in Table 5.
NVIDIA GeForce RTX 3090, using a batch size of 4. We
We have the following findings.
Due to the text encoder and VAE decoder, the real-time consumption of the 2nd-order ICFG is less than the estimated extra computation of the U-Net.
Through a preliminary selection of key timesteps (0.2-0.8) for applying the 2nd-order ICFG, we achieve nearly full FID benefits and 74% improvement CLIP scores, with a reduced extra inference time of 26.27%. We anticipate further enhancements in extra inference time by refining the selection of key timesteps.
16

---

Published as a conference paper at ICLR 2024
hatsune miku
Ibo,, bishoumen, casual, indoors, sitting, coffee shop, bokch
scenery, village, outdoors, sky, clouds
Prompt
CFG
nd order ICFG
Figure 5: The generated images presented here compare the outputs CFG and second-order ICFG with 5.0 and 0.25, utilizing the model anything-v4.0 (htps:/huugiingface co..yyn-ccoyoooooe ai/anything-v44
In the first two rows, is evident that our second-order ICFG produces superior results in hand generation. In the last row, second-order ICFG generates images that align more closely with the provided prompts.
17


Reviews:
[{"review_id": "0JDy5bGcUt", "reviewer": "Reviewer_cPdv", "summary": "This work presents a new perspective on classifier-free guidance (CFG) for diffusion models imposing specific assumptions on the space of condition. Assuming that the condition space has a cone structure, the previous CFG can be seen as the first-order Taylor expansion of the proposed ICFG, and this work further presents a second-order ICFG that improves the Stable Diffusion model.", "questions": "- What is the main reason we should use ICFG instead of CFG?\n\n- How much does the enhanced transition kernel deviate from the original transition kernel (as in Theorem 3.1?) How much does the deviation affect the generation quality?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The observation that there exists a mismatch between the transition distribution of the original forward process and the conditional forward process is new to the best of my knowledge. \n\n- The idea that the mismatch is alleviated when assuming that the condition space has a cone structure is interesting.", "weaknesses": "- The contribution of the proposed ICFG is not clear: Although the authors state that the second-order ICFG introduces new valuable information, it is not clear which additional information it provides, and further the experimental results show a marginal improvement over previous CFG, e.g., FID improvement from 15.42 (CFG) to 15.22 (ICFG) and CLIP Score from 26.45 (CFG) to 26.86 (ICFG), on only single dataset (MS-COCO). Especially, ICFG does not seem to provide an improved balance between FID and CLIP Score. What is the main reason we should use ICFG instead of CFG?\n\n- As the main motivation of this work is the mismatch between the transition kernels (in Theorem 3.1), this should be further analyzed, for example, how much difference in these kernels and how much it affects the generation quality. The proposed method should be evaluated after these validations.\n\n- The assumptions (Assumptions 3.1 and 4.1) made to achieve the proposed method do not seem realistic and were not verified in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RMV9cTzYc7", "reviewer": "Reviewer_LfUU", "summary": "This paper generalizes CFG for diffusion model guidance by adapting the guidance strength according to the relevance between the condition and a given sample. Conditions more relevant to a sample require weaker guidance strength.", "questions": "(see weaknesses)\n\nWilling to increase score if issues are adequately handled.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper identifies an issue with the deviating SDE when guidance is added and proposes a solution with clearly defined assumptions. The paper strikes a good balance between theoretical analysis and empirical results. The theories are relevant to the technique and justify the design choice. Extensive ablation studies on hyperparameters of the method yield insight to the adoption of the technique in practice.", "weaknesses": "**Cone assumption**\n\nIt seems like the key point of ICFG working is for the conditional space to be a cone. Is there any method to check whether a conditional space is a cone beforehand for practioners to decide whether ICFG should be adopted? Are there any metrics for characterizing how cone-shaped a conditional space is?\n\n**Algorithm 3 relaxing 2nd order term**\n\nThe relaxation of the 2nd order term is concerning. The argument for ICFG with 2nd order Tayler Expansion working better than typical CFG is the additional information provided by the 2nd order term. However, if an additional hyperparmeter is introduced and optimized over, does this argument still hold? Does the performance gain of ICFG truly come from the 2nd order information or is it the mere additional of another tuning knob $v$?\n\n**3 forward passes for naive 2nd order ICFG implementation**\n\nThe last paragraph of the discussion section mentions one major drawback of 2nd-order ICFG, which would require 3 forward passes to estimate the 2nd order term. The authors mention a solution of \"reusing the points for the pre-order term\". Elaboration on how exactly this can be done is crucial for actual adoption of this technique. Paying a computation penalty of 3 forward passes is definitely not feasible. The authors should also compare theoretically and/or empirically how this approximation would affect the efficacy of ICFG.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xkqbvcRjXN", "reviewer": "Reviewer_jYKn", "summary": "This paper presents a generalized version of classifier-free guidance (CFG), i.e., inner classifier-free guidance. By exploiting the continuity of the generation condition, the author propose an interesting taylor expansion formulation to interpret CFG, where the classic CFG is viewed as the first order case of the proposed formulation. Given such novel formulation, the author proposed higher order version of CFG to achieve better image generation results.", "questions": "1. Is the few-shot training of ICFG on a small dataset generalizable to large scale setting? For example, as I understand, in this work, the authors trained with the ICFG policy on COCO dataset. I was wondering if this trained LoRA is readily available for generation beyond COCO dataset (e.g., on laion-level dataset)?\n\n2. Is the proposed ICFG applicable on other fine-tuned variation of Stable Diffusion such as anything-V4 (https://huggingface.co/xyn-ai/anything-v4.0), etc.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed formulation is novel and insightful.\n2. The paper is easy-to-follow.\n3. The hyper-parameters introduced by the method is well-studied.", "weaknesses": "1) In the reviewer's viewpoint, the major weakness of the current submission is that the empirical validation is not sufficient. More qualitative results should be provided to justify the effectiveness of the method. One example is that it would be better if the provided qualitative samples could corroborate with the numerical experimental results. Another example is that the author could provided some examples showing that ICFG can resolve some of the well-known failure cases of Stable Diffusion.\n\n2) In addition, it would be good to show the effectiveness of ICFG on other fine-tuned variation of Stable Diffusion such as anything-V4 (https://huggingface.co/xyn-ai/anything-v4.0), etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0zH6TJ4hGc", "reviewer": "Reviewer_UdAr", "summary": "This paper introduces an enhancement of classifier-free guidance (CFG) for diffusion models, called inner classifier-free guidance (ICFG). The paper claims that CFG can be extended to ICFG when the condition is continuous, leading to further improvements. They provide a theoretical analysis based on the condition space assumption and Taylor expansion. They present the experimental results comparing CFG and ICFG.", "questions": "Please answer the questions in the Weaknesses section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The idea of combining the guidance strength and condition is interesting because adjusting guidance strength is the effect of a trade-off between image fidelity and condition information.", "weaknesses": "1. Concerns about the theoretical analysis in Section 3\n* In the proof of Theorem 3.1, there is doubt about the validity of the third equality in Eq. 16. It seems that it may come from Eq. 15, which might change the equality to the proportion.\n* Additionally, the relationship between the last two terms in Eq. 16 is the proportional relationship, but I'm not sure why these two terms are equal after the logarithm, as shown in Eq. 17.\n* In the proof of Theorem 3.1, it would be helpful to provide a detailed explanation of why there is a contradiction unless $w=0$ or $q(x_0|x_t)$ is a Dirac delta distribution.\n* Theorem 3.1 claims necessary and sufficient conditions, but the proof only demonstrates one direction.\n* In the paper, all cases of the enhanced intermediate distribution are denoted $\\bar{q}$. (In addition, $\\beta(x_t)$ in Eq. (12) is not defined.) This is confusing, especially for the definition $\\bar{q}(x_t|c,\\beta):=\\bar{q}(x_t|\\beta c)$.\n* In the proof of Corollary 3.1.1, it is unclear whether the first proportion holds. It seems that $q$ is modified by $\\bar{q}$. Detailed derivation and explanation would be helpful.\n\n2. Not significant experimental results\n* The quantitative results suggest that the performance gain is marginal, and this may require very careful hyperparameter tuning. It would be beneficial to include results from other datasets to assess the tuning problem.\n* The implementation of ICFG in Algorithms 2 and 3 requires three network evaluations for each timestep ($\\epsilon(z_i), \\epsilon(z_i,c), \\epsilon(z_i,mc)$), which is 1.5 times more network evaluations than CFG. Consequently, ICFG has 1.5 times higher sampling cost compared to CFG.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)