Towards Understanding the Working Mechanism of Text-to-Image Diffusion Model
Authors: Mingyang Yi
Aoxue Li
Yi Xin
Zhenguo Li
Venue: NeurIPS 2024 poster (2024)
Keywords: text-to-image generation; working mechanism;
Primary area: diffusion_based_models
TL;DR: In this paper, we reveal the working mechanism of text-to-image diffusion model

Abstract: Recently, the strong latent Diffusion Probabilistic Model (DPM) has been applied to high-quality Text-to-Image (T2I) generation (e.g., Stable Diffusion), by injecting the encoded target text prompt into the gradually denoised diffusion image generator. Despite the success of DPM in practice, the mechanism behind it remains to be explored. To fill this blank, we begin by examining the intermediate statuses during the gradual denoising generation process in DPM. The empirical observations indicate, the shape of image is reconstructed after the first few denoising steps, and then the image is filled with details (e.g., texture). The phenomenon is because the low-frequency signal (shape relevant) of the noisy image is not corrupted until the final stage in the forward process (initial stage of generation) of adding noise in DPM. Inspired by the observations, we proceed to explore the influence of each token in the text prompt during the two stages. After a series of experiments of T2I generations conditioned on a set of text prompts. We conclude that in the earlier generation stage, the image is mostly decided by the special token [\texttt{EOS}] in the text prompt, and the information in the text prompt is already conveyed in this stage. After that, the diffusion model completes the details of generated images by information from themselves. Finally, we propose to apply this observation to accelerate the process of T2I generation by properly removing text guidance, which finally accelerates the sampling up to 25\%+.

Towards Understanding the Working Mechanism of Text-to-Image Diffusion Model
Mingyang Yi Aoxue Li Yi Xin 3M Zhenguo Li² Renmin University of China
2 Huawei Noah's Ark Lab Nanjing University (yimingyang@ruc edu cn) (liaoxue2,1i. zhenguo) (hennuu @huawei. com xinyi@smail .nju. edu.
Abstract
Recently, the strong latent Diffusion Probabilistic Model (DPM) has been applied to high-quality Text-to-Iman Iomae (T2I) generation (e.g., Stable Diffusion), by injecting the encoded target text prompt into the gradually denoised diffusion image genera- tor. Despite the success of DPM in practice, the mechanism behind it remains to be explored. To fill this blank, we begin by examining the intermediate statuses during the gradual denoising generation process in DPM. The empirical observations indicate, the shape of image is reconstructed after the first few denoising steps, and then the image filled with details (e.g., texture). The phenomenon is because the low-frequency signal (shape relevant) of the noisy image is not corrupted until the final stage in the forward process (initial stage of generation) DPM. Inspired by the observations, token in the text prompt during the two stages. After a series of experiments of T21 generations conditioned on a of text prompts. We conclude that in the carlier generation stage, the image mostly decided by the special token [EOS] the text prompt, and the information in the text prompt is already conveyed in this stage. After that, the diffusion model completes the details of generated images by information from themselves. Finally, we propose apply this observation to accelerate the process of T2I generation by properly removing text guidance, which finally accelerates the sampling up to 25%+.
adding noise proceed to explore the influence of each
Introduction
In real-world application, the Text-to-Image (T2I) generation has long been explored owing to its wide applications 47 31 32
2 [5], whereas the Diffusion Probabilistic Model (DPM) [12] 32] stands out as promising approach, thanks its impressive image synthesis capability. Technically, the DPM is hierarchical denoising model, which gradually purifies noisy data from standard Gaussian to generate an image. In the existing literature [31 28 G. the framework of (latent)
Stable Diffusion model 31 is backbone technique in T21 generation via DPM. In this approach, the text prompt is encoded by a CLIP text encoder [30], and injected into the diffusion image decoder condition to generate target image (latent encoded by VQ-GAN in (31) that is consistent with the text prompt. Though this framework works well in practice, the working mechanism behind it, especially for the text prompt, remains to be explored. Therefore, in this paper, we systematically explore the working mechanism of stable diffusion.
"qqual contribution corresponniigg to Mingyang Yi: @imingyang@ruc..dd.ccm
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Our investigation starts from the intermediate status of the denoising generation process. Through an experiment (details are in Section 4 we find that in the early stage of the denoising process, the overall shapes of generated images (latent) are already reconstructed. In contrast, the details (e.g.. textures) are then filled at the end of the denoising process. To explain this, we notice that the overall shape (resp. semantic details) is decided by low-frequency (resp. high-frequency) signals (9. We both empirically show and theoretically explain that in contrast to the high-frreet high-frequency signals, the low-frequency signals of noisy data are not corrupted until the end stage of the forward noise-adding process. Therefore, its reverse denoising process firstly recovers the wwwfrequency signal (so that overall shape) in the initial stage, and then recovers the nigh-frequency part in the latter stage.
Following the phenomenons, we investigate the effect of encoded tokens in the text prompt of T21 generation during the two stages, where each token is encoded by an auto-regressive CLIP text encoder. The text prompt has length of 76, and is enclosed by special tokens [SOS] and [EOS]. at the beginning and end of the text prompt, respectively. Therefore, we categorize the tokens into three classes, i.e., [SOS], semantic tokens, and [EDS]. Notably, the special token [SOS] does not contain information, due to the auto-regressive encoding of the text prompt. Thus, our investigations into the influence of tokens will primarily focus on the semantic tokens and [EOS]. Surprisingly, we find that compared with semantic tokens, the special token [EOS] has a larger impact during generation.
Concretely, under a set of collected text prompts, we select 1000 pairs "[SDS] Prompt A (B) + [BOS] A(B)) " from it. Then, replace the special token [EOS]A in the text prompt A with [EOS] from prompt to observe the generated data under this condition. Interestingly, we find that the generated images are more likely to be aligned with text prompt B (especially for the shape features) instead of A, so that [EOS] has larger impact compared with semantic tokens. Besides that, we further find that the information in [EOS] is already conveyed during the early shape reconstruction stage of the denoising process. Exploring along the working stage of [EOS], we further verify and explain that the whole text prompts (including semantic ones) primarily work on the early denoising process, when the overall shapes of generated images are constructed. After that, the image details are mainly reconstructed by the images themselves. This phenomenon is explained by "first shape then details", as the injected text prompt implicitly penalizes the generated images to be consistent with it. Therefore, the penalization quickly becomes weak, when the overall shape of image is reconstructed.
Finally, we apply our observations in one practical cases: Training-free free sampling acceleration, as the text prompt works in the first stage of denoising process, we remove the textual prompt-related model propagation (co (f, F4, C) in (3) during the details reconstruction stage, which merely change the generated images but save about 25%+ inference cost.
We summarize our contributions as follows.
1. We show, during the denoising process of the stable diffusion model, the overall shape and details of generated images are respectively reconstructed in the early and final stages of
2. For the working mechanism of text prompt, we empirically show the special token [EOS] dominates the influence of text prompt in the early (overall shape reconstruction) stage of denoising process, when the information from text prompt is also conveyed. Subsequently, the model works filling the details of generated images mainly depending on themselves.
3. We apply our observation to accelerate the sampling of denoising process 25%+.
2 Related Work
Diffusion Model. In this paper, our exploration is based on the Stable Diffusion (32. which now terms to be standard T2I generation technique based on DPM [12 36), and has been applied into various computer vision domains e.g., 3D
29 33 19 and video generation (26 4 In practice, the goal of T21 is generating an image that is consistent with a given text injected into the cross- attention module [42 of the image decoder. Therefore, understanding the working mechanism of stable diffusion potentially improves the existing techniques [.] Unfortunately, to the best of our knowledge, the problem is limited explored. expected in [46] 35 where they similarly observe the low-frequency signals are firstly recovered in the denoising process. However, further explanations for this phenomenon are neglected in these works.
[EOS] contains the overall information in text prompt due the auto-regressive CLIP text-encoder

---

Influence of Tokens.
Understanding the working mechanism of encoded (by a pretrained language
model) text prompt 30 43 27 helps us understanding T2I generation [38 16 2TI. For example,
45 finds that in LLM,, the first token primarily decides the weights in the cross-attention map, which similarly appeared in the cross-modality text-image stable diffusion model as we observed. 48 explores the influence of individual tokens in counterfactual memorization. However, in the multi-modality models e.g., [30, 18 17 231, whereas the textual information interacted with the image in the cross-attention module as in stable diffusion, the working mechanism of tokens interacts with cross-modality data is limited explored, expected in [.. They find in a single case that the influence of text prompts may decrease during the denoising process, while they do not proceed to study or apply this phenomenon as in this paper. Recently, [49] finds that the cross-attention map between the text prompt and generated images converges during the denoising process, which is also explained by our observations that the information conveyed during the first few denoising steps. Besides that, unlike ours, their observations are lack of theoretical explanation.
3 Preliminaries
We briefly introduce the (latent) stable diffusion model [31] which transfers a standard Gaussian noise into target image latent that aligns with pre-given text prompts. Here, the generated data space is a low-dimensional Vector- Qurioo-euattzeed (VQ) 77] image latent to reduce the computational cost of generation. One may get the target natural image by decoding the generated image latent. In this paper, the original data (image latent) is denoted by æo, and the encoded textual prompt (by CLIP text encoder (30) is represented by C. The noisy data
3t Vanzo v1 aiEi;
(1)
is used as input to diffusion model Eg trained by
min [|] me C, - =
(2)
0
with 0 <ttss e ~ NO0. I) independent of Tu, ā + 0 (resp. ā. + 1) for t - 0 (resp. t - T). Here, the noise prediction model eø æe. C, () is constructed by classifier-free guidance [3 with
(l zz, C, 0) (t æ., 0) + w (t,( C) c0(l, æ,,
(3)
where (t, I; 0) is an unconditional generative model, and the w 2 0 is guidance scale. As the conduct
model is trained to predict noise e; in æt, and æy approximates standard Gaussian, the reverse denoising process (DDIM (36)) transfers a standard Gaussian to target image Wo
ō. co æt, 3,, C)O).
(4)
act1
Finally, the diffusion model (usually UNet) takes the text prompt as input the cross-attention module in each basic block fl of diffusion model with output Attention(), K, V) = Softmax(QKT///D) (d is dimension of image feature), where d(x;) is the feature of image, and
Woo(x:); K WxC; V WvC.
(5)
4 First Overall Shape then Details
In this section, we first explore the image reconstruction process of the stable diffusion model. As noted in M. the generated image's overall shape is difficult to be alterted in the final stage of the denoising process. Inspired by this observation, and note that the low- fwwffreuuecyy and frequency signals of image determine its overall shape and details, respectively [9]. We theoretically and empirically verify that the denoising process recovers the low and high-frequence signals in its initial and final stages, respectively, which explains the phenomenon of "first overall shape then details".
4.1 Two Stages of Denoising Process
Settings (PromptSet). As in [4], we use 1600 prompts following the template "a fattribute) [noun)", with the attribute as an adjective of color or texture. We create 800 text prompts respectively
'The model is stacked by such basic blocks sequentially with residual module, self attention module, and cross-attention module in each of it.
3

---

Generate Image
Text Prompt: The square coaster was next to the circular mig.
b.
t=0
12.5
t=0
0.6
b=
Rabalive Fi
t=10
0.2
H
0.-
0
10
30
30
-0
50
[S0S]
The squurr coedler
WGS
to the circular mug (BOS)
Time Stepe
(a) Cross-Attention Map
(b) Convergence of Cross-Attention Map
Figure I: Figure la is the averaged cross-attention over denoising steps. The two generated images the bottom with whiter
the top, and the weights in cross-attention maps of each tokens pixels correspond to larger weights in cross-attention map. Figure Tb is obtained by taking average over tokens and prompts in PromptSet, which compares the shapes of cross-attention map and final generated images, Measured by relative Fi-score F1,/FI, over different denoising steps.
under each of the two categories of attributes. Besides that, we add another extra 1000 complex natural prompts in [14] without a predefined sentence template. These prompts consist of the text prompts (abbrev PromptSet) we used. The classes of nouns, colors, and textures are respectively 230, 33, and 23 in these prompts. In this paper, we generate images under PromptSet by Stable Diffusion v1.5-Base [31]. Finally, without specification, we use 50 steps DDIM sampling (36].
From 40) though stable diffusion generates encoded VQ image latents 7 These latents preserve semantic information transformed by text prompt through cross-attention module (5). Notably, in the cross-attention module, the pixel is a weighted sum of token embedding with cross-attention map Softmax /vd) as weights. The weights reveal the semantic information of token, as they are the correlations between image query Q and textual key K. To check the correlation, we visualize the averaged cross-attention map over all layers of model Eg under different time steps L, from 50 to 1.
Interestingly, the cross- attention map of each token already has semantic shape in the early stage of the denoising process, e.g., for = 40 in example Figure Ial This can hold only if the overall shape of the image is constructed in this early stage, that each pixel can correspond the correct token. To further investigate this, we compute the average cross-attention map of each token under the aforementioned PromptSet. We compare the shape of the rross-attention map and the final generated image quantitatively by transforming them into canny images (9) and computing the F1-score [39 (F1, for each between these canny images. To compare the difference over different time steps more clearly, we plot the relative Fl-score F1,/F1, (1 = 1 the image has been recovered). The result in Figure [Tb shows the shape of the cross-attention map rapidly close the ones of the generated image in the early stage of denoising, which is consistent with our speculation and the result in 44). where they conclude that the pross-attention map will converge during the denoising process.
4.2 Frequency Analysis
To further explain the above phenomenon, we refer to the frequency-signal analysis. It has been well explored that the low-frequency signals represent the overall smooth areas slowly varying components of an image (related to the overall shape). On the other hand, the high-frequency signals correspond to the fine details or rapid changes in intensity (related to attributes like textures) [.]. Thereafter, to explain the "first overall shape then details" in the denoising process, it is natural to refer to the variations in frequency signals of images during the denoising process.
Mathematically, suppose the clean data (image latent) Dy has M x N dimensions for each channel with defined in (I. Then the Fourier transformation Fz, (u, v) (with u E [M], E [N] of a, is
ku M
Iv
Fs, (), MN Σ
Σ
kl xi exp
-2ri
N
(6)
= VarFa (u, v) + vI antee

---

C-0
0.13
0.
04
Hgl. 141
Hil Drone
0.00
08
101
20
03
22
70
(a) Noisy data and its high, low frequency parts (b) Norm of features Vā,zz(() Ratio of high / low fre- quency parts variation
and /1 GGE
Figure 2: Figure 2a visualizes the completed noisy data and its high-frequency, and low-frequency parts over different time steps, listed from top bottom. Figures 2b 2 the low/high- frequency signals of æ, In Figure 2b, "Low Add Noisy Data/eps" means the norm of Vor,z and vI a vise versa for "High...." On the other hand, Figure 2c measures the variation ratio of high/low frequency parts of images during the noising/denoising process. For example, [1 1|/||=/ ( during noising process.
"High Add Noise" represents || ||ahh - vo high
where V-I=i i, and zz! is the (k. () component of Ht. As we do not now the distribution of zo. we explore the Fe, (u, 2) sequel. The result the following proposition proved in Appendix [
Proposition 1. For all u E [M]. E [N], with high probability, the complex mumber Fe, (u, v) satisfies
1 MN
[l (u, w)ll 2 O
This proposition indicates that under large image size (MN), the strength of frequency signals (no matter low or high) of standard Gaussian are equally close zero. Thus, the frequency signal of æo in noisy data æ; is mainly corrupted by the shrink factor a, due to (6). instead of the noise in it.
However, visualized in Figure 2a in contrast low-frequency parts [ are more robust than the ones of figh-freeuency.. For example, for t == 20 in Figure 2a] the shape of the clock is still perceptible in the low-frequent wwwffreuenccy part of the image. If this fact is generalized to image latent, then it explains the two stages of generation as observed in Section Because the low-frequency parts
part of image, the image's
not corrupted until the end of the adding noise process. Then, will be recovered at the beginning of the reverse denoising process.
To investigate this, in Figures 2b and [c] we plot the averaged results over time steps of variation of frenhigy-requencyy parts in images generated by PromptSet. In these figures, a is the low-
frequency part of æ; and vice-versa for high-frequency part æ, high As can be seen, in Figure 2c the behavior of v; is similar under add/de noise processes, and the reconstruction of low-frequency signals is faster than the high-frequency signals. On the other hand, by comparing "Low...
( [ ()) and "High... Biigh....DDai Data" (I) (lvā hight in Figure 2b we observe the strength of high- frequency signals are significantly lower than the low-frequency signals, which seems to property adopted from natural image [9]. However, the relationship oppositely holds for Gaussian noise, which implied by Proposition]] I. as the frequency signals of noise e, under each spectrum are all close to zero, while the
high is larger than the coo parts contain 80% spectrum, so that e,
These observations explain the phenomenon "first overall shape then details". Since the low- low-frequency parts of the image (decide overall shape) are not totally corrupted until the end of the noising process. Thus, they will be firstly recovered during the reverse denoising process, while the phenomenon does not hold for low-frequency parts of the image, as they are quickly corrupted during the noising process, so they will not be recovered until the end of denoising.
Please note that the t in Figure and and the other parts of this paper refers to the corresponding t-th time step of 50-steps DDIM sampling, which corresponds to 20t steps in [12]
Wee distinguish 'This fact is also observed in [35] 46). while they leverage this phenomenon to refine the architecture of
frequency by threshold 20%. i.e., lowest 20% parts spectrum are low oreeureuenccy.
UNet, and the rationale behind the phenomenon is unexplored.
5

---

Time Step
10 20
50
0.75
S08
0.50
BOS
0.25
Figure 3: Averaged weights in cross-attention map over pixels of three classes of tokens. For each prompt in PromptSet, the result is obtained by taking average over tokens in each class. The final result the average over PromptSet. Notably, the weights on [SOS] all larger than 0.9.
Prompt [BOS], Proupt
[EOS]a Proupt 4 [205]] Pronpp [EOS]
Prompt A:A bhae bird Prompt B:A chair
Prompt A: The sharp, sugular odges of the city skyline piereed the clouds, symbol ofhmman innovation and progress.
Prompt B: The delicate, flottering wings the butterfyy signaled the arrisal of spring, symbol ofrebirth reaewal.
Figure 4: Images under prompts from S-PromptSet with switched [EOS]. The objects are consistent with the ones conveyed by [EOS], while some information in semantic tokens still conveyed.
5 The Working Mechanism of Text Prompt
We have verified that the denoising process has two stages i.c., "first overall shape then details". Next, we explore the working mechanism of text prompts during these stages. Our main observations are two fold, I): The special tokens [EOS] dominate the influence of text prompt. 2): The text prompt mainly works on the first overall shape reconstruction stage of the denoising process.
5.1 [EOS] Contains More Information
In T21 diffusion, the text prompt is encoded by auto-regressive CLIP text encoder, with semantic tokens (SEM) enclosed with special tokes [SOS] and [EOS]. For such three classes of tokens, the information these tokens is conveyed by the cross-attentie weights over pixels in the cross-attention map for each class. The weights
module, we first compute the averaged
computed by taking the average over PromptSet and presented in Figure 3 As can be seen, the weights of [SOS] are significantly larger than the other classes. However, due the CL.IP text encoder is auto-regressive model, [SOS] does not contain any semantic information. Therefore, we conclude that the influence of [SOS] mainly adjusting the whole cross-attention map i.c., weights on the other tokens. To further verify this conclusion, we conduct experiments in Appendix [.I] A similar phenomenon is observed single-modality LLM 45 As the information of text prompt is conveyed by semantic tokens and [EOS], we will focus on them instead of [SOS] in the sequel.
As both SEM and [EOS] contain the semantic information in the text prompt, first explore which of them has larger impact on T2I generation. To this end, we select 3000 pairs of text prompts from PromptSet (2000 pairs follow the template, the other 1000 pairs have complex prompts), where the two text prompts are represented as "[SDS] + Prompt (B) + [EOS] AA())" " For each pair, we switch their [EOS] construct the new text prompt pairs as "[SDS] Prompt A (B) + [BO))((4)"
We examine the generated images under these artificially constructed text prompts (namely Switchedd-PromttSe (S-PromptSet)). We call A from Prompt "source" and B from [EOS] as "target" for "[SOS] Prompt + [EOS] ", and vice versa. For the generated images under these prompts, measure their alignments with the source and target prompts, respectively. The used metrics are the three standard ones in measuring text-image alignment: CLIPScore B0 [0], BLIP-VQA [18 [4], and MiniGPT4-CoT [51 [4] (details in Appendix [B).
The results are in Table T Surprisingly, the generated images under the constructed text prompts more likely to be aligned with the target prompt instead of the source prompt. That says, even

---

A: Source; B: Target
move of start step a switches [EOS]
t T
a
t E (a, TI: Prompt A + [EOS]s
t [0, a]: Prompt A + [EOS]
Figure 5: Desnoising process under text prompt with switched [EOS] in [a, 50].
Figure 6: Relative text-image alignments ("current minus worst" over "best minus worst") with source (or target) prompt under switched [EOS] (substitution in [Start Step, 50J. Figure Align- ments with source and target prompts are respectively solid and dot lines.
Table 1:
The alignment of gen- erated image with its source and target prompts. The prompts are constructed with switched [EOS].
1.0
SernSonn N.P.VONTTON
Relaiive
Prompe
Alignment
Source
Target
CLIPSSCee
D.2563
0.2758
BLIP-VQAT
0.3525
0.4441
MiniGPT-CaTT
0.6473
0.7213
0.0
Start Step soutch [BO55]
with prefixed irrelevant semantic tokens, the information contained in [EDS] dominates the denoising process (especially for overall shape) as in Figure 4 Thus, we conclude that the special tokens [EOS] have a larger impact than semantic tokens in prompt during T2I generation. We have two speculations about this phenomenon. 1): Owing to the ruto-regressive encoded text prompt, unlike semantic tokens. [EOS] contains complete textual information, so that decides the pattern of the generated image. 2): The number of [EOS] is usually larger than semantic tokens, as the prompt is enclosed by [EOS] to length 76. An ablation study Appendix verifies this speculation.
In summary, our conclusion in this subsection can be summarized as: In T2I generation, the special token [EOS] decides the overall information (especially shape) of the generated image.
Remark 1. For the generated images under S-Promp tSet, we find some information in semantic tokens is also conveyed, especially for the altribule information in it, e.g. "brown" color in the last image of the first row in Figure A We explore this in Appendix DiiixDDannn and explain this as: unlike noun information. attributes in semantic tokens may conftict with the contained information in [EDS) (which quickly decides the overall shape of the generated image), so that has potential to be conveyed.
5.2 The Text Prompt Mainly Working on the First Stage
In Section 1.1 we have conclude that the denoising process is divided into two stages "first overall shape then details".. Next, we explore the relationships between text prompts and the two stages. We start with special tokens [EOS] which contain major information in T2I generation. During the whole 50 denoising steps of T21 generation under prompts from S Setootttttt. we vary the starting point of substituting [EOS] i.e., the used text prompt is "[SOS] + Prompt
[EOS] (resp. [EOS] for
t e [Start Step, 50] (resp. E [0, Start Step)) with "Start Step" e [o, 50), i.e.. Figure We compare the alignments of generated images with source / target prompts as in Figure
In Figure 6 the alignment with the target prompt slightly decreases, until the "Start Step" of substitution close to 50. This shows that the information in [EOS] has been conveyed during the first few steps of the denoising, which is the overall shape reconstruction stage according to Section [
Following the revealed working stage of [EOS], we explore whether the whole text prompt also works this stage. If so, the T2I generation will only depends on 6g 2,, 0) in (3) for small I. To see this, we vary the w in (3) to control the injected information from the text prompt during the denoising process. Concretely, for a as the starting step of removing text prompt, i.c., during t E [0, a), use only works for E [0, a). We generate target images 50 under PromptSet with standard denoising process (a 50), and compare them with the ones æ generated under varied
7.5, and w 0 for t E [a, 50), where E [0, 50). Then, the text prompt
[0, 50] (Figure The image- image aligmments are measured by standard metrics CLIPScore and L -isttanne 9 To eliminate magnitude, report relative results, i.e., "current minus worst" over "best minus worst".
The results are in Figure Sa. During generation, the text information is absence for / E (a, 50), while Figure 8a indicates that alignments between ag and target a will quickly be small only for large (from 30 to 50). This shows that only if removing the textual information under large i, its influence to generated image is removed. Therefore, we can conclude: The information of text prompt is
7

---

move start step removes text prompt
[a, without text prompt [0, a] with text prompt
Figure 7: Desnoising with text prompt injected in a].
1.0
le-4
Imvge-Level Ll-aorm
deniiiiiinall.cmmmm Novse Norrs
Difference Norm/ Dim
0.8
Image-Level CLIP Score
Relative Seate
0.6
5-
6-
0.4
4-
0.2
0.0
10
40
50
20
30
40
50
20
30
40
S0
Start Stop to set w-0
step
step
(a) Relative Alignment
(b) Norm/Dim of (Un)/Conditional Model
Figure 8: Figure Balis the relative difference "current minus worst" over "best minus worst" under different start step a of Denoising process Figure 7 The last two figures 8b are per-dimensional norm of unconditional noise Eb(L, wt, 0) and noise difference (1, 2z, C)) Eo Wr, 0))
conveyed during the early stage of denoising process. Therefore, the overall shape of generated image is mainly decided by the text prompt, while the its details are then reconstructed by itself.
Discussion. Next, let explain the phenomenon. Technically, in (3). the co az C, 0) was proposed to approximate Va log (x C)/VT - O₃ with decomposition (pr. is the density of xx))
V (ax C) = V. (r Va log x+).
(8)
Comparing (3) and (8) it holds (t, xz, 0) oc V log p: and w<e((t, xz, C)x ct x 0)) x aims maximize likeliilldood log Po (Zo C). Then, moving along the direction Va log (C xx.) (leads to large log Pe xx))
V log pe 2t). From (37) the denoising process
push 2t to be aligned with the text prompt C during the decreasing of Adding such moving direction is standard in conditional generation [25] 36 g 24]. As shown in Figure [b] during the denoising process, 2,, will gradually to be consistent with C. that V log Pc(C œe) will decrease with L. Thereafter, we observe the impact of text prompt conveyed by this term decreases with / - 0. Notably, owing the quickly reconstructed overall shape of image in Section 4 the generated æ; will quickly be consistent with C, so that explain the quickly decreasing v log Pc xt).
Remark 2. In this section, we verify the injected textual information are all conveyed in the first stage of diffusion process. In fact, this phenomenon is also generalized to the other types of information, e.g, conditional image information subject-drive generation 44 501, we verify this Appendix
H
6
Application
Acceleration of Sampling. Since the information contained in text prompt is mainly conveyed by the noise prediction with condition (t, C), first few steps of denoising process. This because the information text prompt has been conveyed in this stage, and the computational cost can be significantly reduced without evaluating eo eo x., C).
consider removing the evaluation of after the
Therefore, we substitute the noise prediction c0(1, me. C, 0) as
(f. æı, 0) (ee(t, me, C) - e (t. m., 0)) c (t, xt, 0)
a S t
(t æ., C, 0) =
(9)
0 0 <
This can be verified by the training strategy of it in where co oc, 0) is used to predict noise in noisy data without condition injected.

---

move start step a adds text prompt
T
a
t E [a, T] with text prompt
E
[0, a] without text prompt
Figure 9: Desnoising under Eg The text prompt is injected in [a, T], instead of [0, a] in Figure]
Start point
0 (baseline) a
a
25 w/o
dog standing bench during the day.
text
SD v1.5
SD v2.1
Pixart-Alpha
Figure 10: The generated images with 25 steps DPM-Solver under eg in 9 (Figure 9 The textual
information is removed during [0, a). With + 25, the inference cost is decreased.
Table The difference between images generated under varied with the ones of a = 0. The results averaged over 30K generated images, and saved latency is evaluated one V100 GPU.
point i
('hhselime)
Sumpler Backbone
DDIM/SDD<
DPMM-Solver
v1.
Cing-CLIPScorce
1000
0.99%
(13955
0.97
0.838
0.330
[.000
03550
01504
03356
0.705
0.533
fi-distance
0000
0.011
(1012
0.943
0.087
0.195
0.000
0.015
0.027
0.050
0.100
0.188
Saved Latency
0.00%
7.34%
18.10%
27.18%
35.245%
48.47%
0.00%
8.36%
17.95%
26.11%
34,86%
47.00%
13.772
13,770
15.048
14,725
15,860
Text-CT IPScore 1
31.000
31,0011
28.172
16.682
30,993
30.772
36,543
16.721
Sumpler / Backbone
DDIM
/D
v2.
DPM-Sallvr
7 S2.
Image-C [PPooee
1000
0399
019955
0.966
0.902
0.350
1,000
01999
03998
0.885
0.901
0.343
1y-distancee
Saved Latency
18.28%
35.48%
FID 1
13.04
13.011
13.065
13.247
14.242
18.472
13,507
13.500
13.914
14.159
15.015
18.983
Test-CI IPSeare 1
31.413
31,405
31 362
51.113
29.717
16.706
31.339
31 31.326
31.202
31 045
29.653
16.639
Sumpler/ / Backbene
PDLM / Fesart Alpha
DPP
Solver
Psaat-Alpha
Impge-CCIPSSe
(1935
0.643
1,000
0.515
Ii-distance
0.00
0.058
0.103
8.169
0.347
0.000
0.022
0.008
Saved Latency
0.00%
8.24%
17.98%
27.20%
$4.95%
49.15%
0.00%
7.92%
15.77%
25.18%
33.80%
10.0
FID J.
22.661
22.884
23.258
25. 48.488
29.760
36,525
18.669
18.520
73.798
19.358
20.494
26.159
Text-CL.IPSoore
28.157
27.979
25.719
19.640
14.986
14,386
30.733
30.731
30.745
29.416
20,629
14,928
By varying - T in 99) the inference cost is reduced as evaluation of ct, 2r, C) is saved.
To evaluate the saved computational cost of using noise prediction p during inference and the quality of generated data, we consider applying it on two standard samplers DDIM (36] and DPM-Solvee
[22] on a benchmark dataset MS-COCO (0 in T2I generation. We consider backbone models Stable-Diffusion (SD) vl.5-Base, SD v2. -ase 31,. and Pixart-Alpha [5]. Concretely,
apply noise prediction with varied a to generate 30K images from 30K text prompts in the test set of MS-COCO, for each sampler and backbone model. We compare the difference (measured by Li-distance and Image-Level CLIPScore) between the generated images under a > 0 and a = 0 (the standard noise prediction). The results are in Table 2 where also report the Frechet Inception Distance (FID) score [[]] under each a to evaluate the quality of generated images.
The Table 2 indicates that proper in 9 significantly reduces the computation cost during the inference stage without deteriorate the quality of generated images. For example, SD v1.5 with 20 saves 27+% computational cost, but generates images close to the bascline method (a 0).

---

7 Conclusion
In this paper, we investigate the working mechanism of T2I diffusion model. By empirical and theoretical (frequency) analysis, we conclude that the denoising process firstly constructs the overall shape then details of the generated image. Next, we explore the working mechanism of text prompts. We find its special token [EOS] has significant impact on the overall shape in the first stage of the denoising process, in which the information in the text prompt is conveyed. Then, the details of images are mainly reconstructed by themselves in the latter stage of generation. Finally, we apply our conclusion accelerate the inference of T21 generation, and save 25%+ computational cost.
Acknowledgement
We gratefully acknowledge the support of Mindspore, CANN(Compute Architecture for Neural Networks) and Ascend Al Processor used for this research.
References
[1] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Mika Aittala, Timo Aila, Samuli Laine. Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. Preprint arXiv:221 1.001224 2022.
[2] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Conference on Computer Vision and Pattern Recognition, 2023.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. 2020.
[4] Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. Stablevideo: Text-driven consistency- aware diffusion video editing. In International Conference Computer Vision, 2023.
[5] Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-alpha: Fast training of diffusion transformer for photorcalistic text-to-image synthesis. In International Conference Learning Representations, 2024.
[6] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[7] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Conference on Computer Vision and Pattern Recognition, 2021.
[8] Karim Farid, Simon Schrodi, Max Argus, and Thomas Brox. Latent diffusion counterfactual explanations. Preprint 2xXiv:2310006666 2023.
[9] Rafael C Gonzales and Paul Wintz. Digital image processing. Addison- -eessyn-Wesley Longman Publishing Co., Inc., 1987.
[10] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Conference on Empirical Methods in Natural Language Processing, 2021.
[11] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. 2017.
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances Neural Information Processing Systems, 2020.
[13] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
10

---

[14] Kaiyi Huang. Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A compre- hensive benchmark for open-words world compositional text-to-image generation. Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[15] Minguk Kang, Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Conference on Computer Vision and Pattern Recognition, 2023.
[16] Kalpesh Krishna, Yapei Chang, John Wieting, and Mohit lyyer. Rankgen: Improving text generation with large ranking models. In Empirical Methods in Natural Language Processing, 2022.
[17] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: 2: Bootstrapping language-image ure-training with frozen image encoders and large language models. Preprint arXiv:2301.12597, 2023.
[18] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International Conference Machine Learning, 2022.
[19] Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Yi Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Conference on Computer Vision and Pattern Recognition, 2023.
[20] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014.
[21] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Perey Liang. Lost in the middle: How language models use long contexts. Preprint arXiv:2307.03172, 2023.
[22] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems, 2022.
[23] Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Conference on Computer Vision and Pattern Recognition, 2022.
[24] Nishtha Madaan, Inkit Padhi, Naveen Panwar, and Diptikalyan Saha. Generate your counterfac- tuals: Towards controlled counterfactual generation for text. In Association for the Advancement of Artificial Intelligence, 2021.
[25] Petr Marek, Vishal Ishwar Naik, Anuj Goyal, and Vincent Auvray. Oodgan: Generative adversarial network for out-of-domain data generation. In Conference of the North American Chapter of the Association for Computational Linguistics, 2021.
[26] Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen. Dreamix: Video diffusion models are general video editors. Preprint arXiv:2302.01329) 2023.
[27] OpenAl. Gpt-4 technical report. Preprint arXiv:2304.1059) 2023.
[28] William Peebles and Saining Xie. Scalable diffussion models with transformers. In International Conference on Computer Vision, 2023.
[29] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In International Conference Learning Representations, 2022.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning,
2021.
11

---

[31] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. Preprint urXiv:2204.06125, 2022.
[32] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution mmgge synthesis with latent diffusion models. In Conference on Computer Vision and Pattern Recognition, 2022.
[33] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Ktir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Conference on Computer Vision and Pattern Recognition, 2023.
[34] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, 2022.
[35] Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. Preprint arXiv:2309. 11497 2023.
[36] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2022.
[37] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[38] Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit lyyer. Do long-range language models actually use long-range context? In Conference Empirical Methods in Natural Language Processing, 2021.
[39] Abdel Aziz Taha and Allan Hanbury. Metries for evaluating 3d medical image segmentation: analysis, selection, and tool. BMC Medical Imaging, 15:29, 2015.
[40] Raphael Tang, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Jimmy Lin, and Ferhan Ture. What the daam: Interpreting stable diffusion using cross attention. 2023.
[41] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[43] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. 2022.
[44] Yu Liu Yujun Shen Deli Zhao Hengshuang Zhao Xi Chen, Lianghua Huang. Anydoor: Zero-shot object-level image customization. In Conference on Computer Vision and Pattern Recognition, 2024.
[45] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. Preprint arXiv:2309.114533 2023.
[46] Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Difffusion probabilistic model made slim. In Conference on Computer Vision and Pattern Recognition, 2023.
[47] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfci Yang. Bureu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research, 2022.
[48] Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. Preprint 12. 2021.
12

---

[49] Wentian Zhang, Haozhe Liu, Jinheng Xie, Francesco Faccio, Mike Zheng Shou, and Jürgen Schmidhuber. Cross-attention makes inference cumbersome text-to-image diffusion models. Preprint ar arXiv:2404.02747,, 2024.
[50] Xintao Wang Zhongang Qi Ming-Ming Cheng Ying Shan Zhen Li, Mingdeng Cao. Photomaker: Customizing realistic human photos via stacked embedding. In Conference on Computer Vision and Pattern Recognition, 2024.
[51] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. Preprint N0W:2330.11592.. 2023.
13

---

A Proofs of Proposition |
Proposition 1. For all [ [M], with high probability, the complex number Fe, (u, v)
satisfies
1 MN
| (u, w)|l O
(7)
Proof. Note that kll we have
(abbreviated as e are i.id. Gaussian random variable for each of k Thus
o)(U,
MN Σ Σ Ethh
exp
2
ku M
+ Iv
)))
M-1N-1
Σ
dl exp (-i0ft).
(10)
MN
with 6l is the (Is, 1)-th angle in complex value space, and we may simplify as 6 for ease of notations.
Next, we will show the proposition is direct consequence of the concentration inequality of Gaussian distribution. We prove our results under one-dimensional Fourier transformation under dimension M, where the proof can be easily generalized to two-dimensional case. Owing to the definition of the norm of complex value, for any specific u.
[x)(1l = (.) 1
M2 A11 Āe,
(11)
..... i0M "). Then let P = (V1/MI".....) ,A,, where (VI/MTT TT is constructed by vector V1/M1 and its orthogonal complement. We can verify that P is an orthogonal matrix. Then, let E P y. so that y has the same distribution with e. Thus
where A diag(e 100
M T
A11 AE= 1 PAMI"AP" y e vy
M²y PA11
1
M (3')²
(12)
M
where y² is a standard Gaussian. Thus, by the Berstein's inequality to sub-exponential random variable i.e., xi. we have
(() - [[F(())]] 6) P -I [ 2 s 2ep (- min 18 (8
min
(13)
Since E (0, 1). Thus with probability at least - 5/M, we have
1
2M
2M 8log
M V log
& $I < M M
(14)
M
which proves that the target |eee, (u, o)l is of order O (++),, so that verify our conclusion.
B Text-Image Alignment Metrics
In this paper, we mainly use three metrics as in [4] measure the alignment between text prompt condition and generated image. Next, we give a brief introduction to the these metrics.
CLIPScore (Text). After extracting the features of generated images and text prompt respectively by CLIP encoder [30], CLIPScore (Text) the cosine similarity between the features. Similarly, the CLIPScore (Image) is the cosine-similarity between two image features.
BLIP-VQA. To improve the limited details capturing capability of the CLIP encoder, 14] propose BLIP-VQA which leverages the visual question answering (VQA) ability of BLIP model [8]. They compare the generated with the target text prompt separately described by several questions. For example, the prompt "A blue bird" can be separated into questions bird ?", "a blue bird ?" etc. Then BLIP-VQA outputs the probability of "Yes" when comparing generated images and these questions.
MiniGPT4-CoT. The MiniGPT4-CoT 14 combines strong multi-modalle question answering model MiniGPT-4 (51 and Chain-of-Thought 43. The metric computed by feeding the generated images to MiniGPT-4, then sequentially asking the model two questions "describe the image" and "predict the image-text alignment score". By constructing such CoT, the multi-modal model will not ignore the details in generated images.
14

---

C Ablation Study on Number of [EOS]
lapi Ause Iota:
26
Capal hocd biial
Sant pers lomen)
Sunt pers trone
Tepaltany TOREE
herp
C2
C2
Figure 11: Relative BLIP-VQA Combine color and texture add complex, CLIP score, MiniGPT-CoT with source (or target) prompt under different number of [EOS]. Here the y-axis is the current BLIP- VQA, CLIP, MiniGPT-CoT over the maximum ones, which is used alleviate the bias brought by the metric itself.
In this section, we conduct an ablation on the number of [EOS]. Concretely, for each text prompt in S-PromptSet. repeat the semantic tokens in the text prompt e.g., "[SOS] + yellow cat yellow + [EOS]", so that the number [EOS] is reduced in these constructed text prompts. Then, generate images under these reconstructed text prompts and compare the text-image alignments between generated images and source or target prompts.
The results are in Figure I As can be seen, with the increasing of semantic tokens (so that decreasing of [EOS]), the generated images tend be consistent with the source prompt, instead of the target prompt. Therefore, we speculate that the domination of [EDS] may be partially originated from its larger number, compared with semantic tokens. the other hand, we observe that [EDS] in the forward positions have larger impacts compared to the latter ones, the alignments between generated images with target prompts significantly decreased along the x-axis from right to left, in Figure II] This trends further indicate that the [EOS] may contain more information compared with the latter ones, which indicates the domination of [EOS] originates from their larger number, but also the more information in the first few [EDS].
D Conveyed Information in Semantic Tokens
During discussion in Section 5 conclude that the [EOS] has larger impact than the ones of semantic tokens during T2I generation. However, observed in Figure 4] under text prompt with switched [EOS], some information in semantic tokens are still conveyed in the generated images, c.g.. blue color in the last image of the first row in Figure 4 Therefore, we explore how this information is conveyed this section, which also reveals the working mechanism of text prompt.
we visualize the cross-attentie map of each tokens under text prompt from S-PromptSet, similar to Figure [b] Surprisingly, find that in coss-attention map of semantic tokens and [EOS] are all visually similar to the shape of the final generated images. The similarity is reasonable for [EOS] as it contains the overall information, that it perceptible and transfer their information according to constructed cross-attention map.
Firstly, in Figure 12
On the other hand, when semantic tokens convey their information according to the similar cross- attention, for attributes (color or texture), unlike object information, they are potentially not contradict to overall shape decided by [EDS]. Thus, the information of attributes is more likely to be conveyed in its corresponding pixels. However, this does hold for object/noun tokens whose information is very likely related to shape, which has already been decided by [EDS].
This discussion explains the phenomenon of information semantic tokens are appeared in generated images under prompt from S-PromptSet.. Combining the observations to the working stage of text prompt in Section 5 we can conclude that the semantic tokens also work the T21 generation, though it has less impact compared to [EOS].
15

---

Prompt A [EOS]
Prompt B [EOS]A
t= 50
== 30
20
t=
[SOS]
leather chair [EOS]
[SOS]
fabric pants [EOS]
Figure 12: The visualization of cross-attention map under text prompt with switched [EOS] from S- S-PromptSet. The pixels corresponding to semantic tokens are in the shape of the final generated data as in text prompt B provide [EOS]. For example, the token "chair" corresponds pixels in the shape of paint, so its information can not be conveyed, while this phenomenon does exist in the attribute token "leather".
Generated Image
SOS
EOS
Generated Image
SOS
while plate
a brown elephant
A magnificou how was taking leisurely walk the forest, its steady powerful, its eyes muaeesty king.
cutle pomeranian dog, is playing in the green grass. Rt hatris is brown fanzy highly detailed. eyes round black it like furry doll,
Figure 13: Generated images with prompts only contain information from [SOS] or [EOS).
E [SOS] Contains no Textual Information
As mentioned in Section 5.1 the special token [SOS] is supposed to contain no textual information, due to the auto-regressive textual prompt encoder. To further verify this, we conduct the following two types of prompts. 1): all 77 tokens are [SOS] from the given text prompt, 2) except the first [SOS] token, all other tokens are [EOS] from given text prompt. Then, we generate images under these prompts with SD v1.5. The generated images as in Figure [3] As can be seen, for the first type of prompt, no textual information conveyed, while the phenomenon disappears in the second type of prompt. This observation verifies our conclusion that [SOS] contains no textual information.
16

---

Sem+EOS
Sem+Zero
Zero+EOS
Sem+
Rand+EOS
a red sunset
a plastic plate
a white bench
Figure 14: Generated images with zero or random vectors substitution.
Table 3: The alignment of generated results image under different constructed text prompt sets. Here
"Sem + EOS" is the original text prompt, and serves as baseline here. Besides that, the CLIPScore
(Image) is the image-level alignment of generated images with the under "Sem EOS".
Text Prompt CLIPScore (Text)f CLIPScore (Image)t
BLIP-VQAT MiniGPT4-CoT1
Sem Zero
0,2407
0.6732
0.5392
0.6757
Sem Rand
0,2153
0.6038
0.3606
0.5428
Zero + EOS
0.2999
0.8887
0.7467
0.6982
Rand EOS
0.3008
0.8791
0.7669
0.7000
Sem + EOS
0,3110
1.0000
0.8999
0.7412
F
More Evidences on [EOS] Contains More Information
In this section, further verify that the impact of [EDS] is larger than the ones of semantic tokens in T2I generation. To further verify this conclusion, under the text prompts with the format of "[SOS] + Sem [EOS]" from our dataset PremptSet,
substitute all semantic tokens or [EOS] with zero vectors or random Gaussian noise. As a result, we get the 4 sets of text prompts, i.e., "[SOS] + Sem 7ero" (abbrev Sem Zero), "[SOS] Sem Random" (Sem Rand), "[SOS] Zero [EOS]" (Zero + ROS), and "[SOS] + Random + [EOS]" (Rand + EOS). These constructed text prompts ideally contain complete semantic information, and we verify the alignment of the generated images with the corresponding text prompt conditions. The alignments are measured by text-image alignment metrics: CLIPScore [30 10), BLIP-VQA [18, 14), and MiniGPT4-CoT [1 [4).
The results are summarized Table 3] As can be seen, as expected for baseline combination "Sem + EOS", the alignments under text prompts with "EOS" preserved significantly better than the ones with "Sem" preserved. Thus, the observations further verify our conclusion that the [EOS) has larger influences than semantic tokens during the denoising process.
Moreover, we find the generation somehow robust, as involving random noise text prompts still generates semantic meaningful images. We visualize some generated images under constructed text-
17

---

Table 4: The alignment of generated image with its source and target prompts, under switched [EOS] on Key or Value substitution. Here VV-Sub is the complete substitution as in Section which serves baseline here.
Attribute
K-Sub
V-Sub
VKV-uub
Alignment CLIPScore (Text)
Source
Target
Source
Target
Source
Target
0.3132
0.1875
0.2538
0.2628
0.2363
0.2758
BLIP-VQAT MiniGPT-CoTT
0.7127
0.2379
0.3724
0.3984
0.3325
0.4441
0.8025
0.6215
0.6749
0.7071
0.6473
0.7213
prompts from Table in Figure 14 which indicates the images under "Zero + EOS" indeed visually have the best quality in alignment, so that consist with Table 3) Besides that, the other combinations generate semantic meaningful images as well, expected for "Sem + Rand" Thus semantic tokens do not contain enough information for generation.
G Key or Value Dominates the Influence?
Prompt.4 A a brown car
Prompe blue girafle
Prompt A K-[EOS]B Prompt A v-[BO]] Prompt B K-[EOS]A Prompt B V-[BOS]
A: a brown backpack
B: a blue horse
A: fabric bag
B: a leather shoes
A: leather jacket
8: a glass jar
Figure 15: Generated examples of Key Value substitution.
As mentioned in Section 3 the information from [EOS] is conveyed by the cross cooss-ttention module. More concretely, the Key (K) and Value (V) in it, respectively decide the weights and features the output of the cross-attention module (a weighted sum of features). Next, explore their individual influence for [EOS] to further reveal the working mechanism of
we generate images under the constructed text prompt set S- S-PromptSet. However, the substitution of [EOS] only conducted on computing Key or Values cross-attention module, which are respectively denoted as K K-Substitution (Sub) and V-Sub. For such two substitu- tions, similar to Table T. we compare the image-text alignment of generated images with source and target prompts.
Concretely, as in Section 5]
18

---

Loprifbnic renilts of dild-degneeeee
1.5-
1.0-
Scbetininng the (BCS) in K
0.5-
Unitocu Distribution
3.5
100
20
Time Stegs
Figure 16: The averaged KI--dieegeece over pixels and layers.
Sulject Image
poine
We implement our sampling strategy subject-driven generation model, AnyDoor 444 We remove the condition image from different time steps (denote as a) during denoising process. The generated images still preserve the specific details as baseline model (start point 0) when start removing time steps set to 20.
Figure 17:
Start Point
10
2-20
a=30
Reference Face
Figure 18: We implement our sampling strategy on human face generation model, PhotoMaker. We remove the condition (lext prompts and reference face) from different time steps (denote as a) during denoising process. The generated images and faces still preserve the specific details as baseline model (start point a = 0) when start removing time steps is set to 20.
The results are summarized in Table As can be seen, substituting the [EOS] in V has a larger influence than the substitutions in K. To explain this, as we have observed in Figure 3 the weights on semantic tokens and [EOS] are significantly smaller than the ones on [SOS]. However, the K of [EOS] is only related to these small weights, which have limited influence. In contrast to K, the V of [EOS] contains information on features, which can directly conveyed generated images. So that we can conclude the value of [EOS] dominates the influence in generation observed in Table We present some generated images under text prompts from S-PromptSet, but with only Key or Value substituted in Table 4 The generated images are in Figure 15
19

---

Prompt A: The warm, golden glow of the sumrise lit up the horizon, natural wonder breathtaking beauty.
Prompt B: The sharp, pointed teeth the shark contrasted with its sleek, streamlined body, powerful and graceful predator of the ocean.
Prompt A: brown
Prompt B: Ard suitcase.
Prompt A: A fabric towel.
Prompt B: A glass vase.
Prompt A: A fabric pants. Prompt B: A leather wallet.
Figure 19: More generated examples under tokens from S S-PromptSet.
We further verify the variation of the cross-attention map after substituting [EOS]. For each pixel, the cross-attention map of it is discrete probability distribution. Thus, we compute the KI.- divergence [1 between probability distributions under substinuted/ussaasiitredd K. The averaged KL-divergence over pixels and layers in models under different denoising steps is presented in Figure [6] where we add the KL-divergence of coos-atttnntion map distribution between uniform distribution as a baseline. The result shows that even with substituted [EOS], the cross-attention map does not vary much. We speculate that this because as in Figure 3 the weights [SOS] dominate the cross-altention map. Thus, in the cross-attention module, altering K has slighter influence compared with altering V.
H Firstly Conveyed Information for Subject-Driven Generation
injected textual information conveyed in the first stage of diffusion model. Since the information is conveyed in the cross-attention module of model, we speculate this phenomenon may be generalized the other conditional generation task, e.g., subject-driven generation with an extra image condition.
As mentioned in Section 5.1
20

---

To see this, we conduct experiments under two tasks: zero-shot subject-driven generation and human face generation. For such two tasks, there extra reference image (given subject and human face) used as condition to guide image generation. We use the sampling strategy as in Figure 10 for the two tasks, respectively follow the backhone methods AnyDoor in Figures] 17 and 18 respectively.
44 and Photomaker [50]. The results are
As can be seen, the generation results verify the conclusion that for conditional information from in the other modality than text, they will be also firstly conveyed in during the diffusion process.
I More Generated Images
L.1 [EOS] Substitution
In this subsection, we first present the generated images under text prompts from S-PromptSet in Figure T9 As can seen, most overall shape of generated images are consistent with the ones conveyed by [EOS].
1.2
Generated Images in Paragraph "Acceleration of Sampling" of Section 6
Next, we present more generated images under noise prediction ( with varied a in Figure 20 and
Start point
a = 0 a = 10 a = 20
= 30
a = 40
a 50
Prompt A: green vase with white flowers in sitting counter.
Prompt B: A tray with coffee and pastry it.
SD v1.5
SD v2.1
Pixart-Alpha
Figure 20: The generated images with steps DDIM under eg L where the textual information are C removed during time steps E a). With + 50, the inference cost is decreased.
21

---

Start point
a a=5 a 10 a 15 a 20
a 25
Prompt A: A kitchen counter is lined with multiple objects.. Prompt B: A plate with some vegetables and meat on it.
SD v1.5
SD v2.1
Pixart-Alpha
Figure 21: More generated images with 25 steps DPM-Solver under ep 9 where the textual information are C removed during time steps / E [0, a]. With - 25, the inference cost is decreased.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main contributions of this paper have been clarified in Abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long clear that these goals not attained by the paper.
22

---

2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation of this paper discussed in the main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
23

---

Justification:
The experimental details are in main part of this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will release the code in future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
cnosiinyy)yy)) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ https//ccccbbiiiguuieeeeee/////e Stteee//CooSuuuiisiiiii.. wwt.ssuwissssoip.... ssooiolliciii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
24

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
They are specified in main part of this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: The results have no error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:////.aaaaaa.coooo/o///aaaaa
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
26

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
27

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "rc1WLAYS2g", "reviewer": "Reviewer_KEXx", "summary": "This paper study how the EOS token plays a role in the generation process of diffusion model. In particular, this paper finds that diffusion models tend to first generate low frequency part of the image at the beginning of the generation process, then gradually add high frequency signal to it. Experiments show that the low frequency signal is conditional on the EOS token while the high frequency signal can be generated without text guidance. In combined with the aforementioned observation, this paper proposes to remove $\\epsilon_\\theta$ in classifier-free guidance once the low frequency signal has been generation to improve generation efficiency.", "questions": "- Please clarify how the computation cost is defined in this paper and how the efficiency gain is computed.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper offers a new perspective for understanding the role of textual condition in diffusion models.  By exploring how the EOS influence the generation process of diffusion model, this paper argues that the conditional part $\\epsilon_\\theta$ in classifier-free guidance (CFG) might be unnecessary after certain denoising step $t_w$.  \n- Most experiments are inspirational and interesting. By swapping the EOS token and the sentence body, it demonstrates that diffusion models rely on the EOS token to synthesize low frequency part of the image. \n- This paper explains the tendency of generating image from low-to-high frequency in diffusion models.", "weaknesses": "- It is not clear that how the \"computational cost\" is defined in this paper. If the computational cost is GPU VRAM, then the claimed efficiency improvement might be invalid, as the required GPU VRAM for computing $\\epsilon_\\theta(x_t, C)$ or $\\epsilon_\\theta(x_t, \\emptyset )$\n is unchanged. \n\n- This paper mainly focus on the role of EOS token in T2I diffusion models while neglecting the SOS token. Despite the weight of SOS token is significantly higher than SEM and EOS token (see Figure 3). However, the authoer(s) claims that the SOS carries no information due to the autoregressive nature CLIP text encoder.  Since this claim is not yet supported by other works, the author(s) should have conducted experiments to support this claim, as there is a chance that EOS and SOS tokens altogether influence the generation process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable comments. Here we address your concerns as follows.\n\n**Q1**: “It is not clear that how the \"computational cost\" is defined in this paper. If the computational cost is GPU VRAM, then the claimed efficiency improvement might be invalid, as the required GPU VRAM for computing $\\epsilon_{\\theta}(x_{t}, C)$ or $\\epsilon_{\\theta}(x_{t}, \\emptyset)$ is unchanged.” \n\n**A1**: In this paper, the “computational cost” we mentioned is used for computing the noise prediction $\\epsilon_{\\theta}(t, x_{t}, C, \\emptyset) = \\epsilon_{\\theta}(t, x_{t}, C) + w(\\epsilon_{\\theta}(t, x_{t}, C) - \\epsilon_{\\theta}(t, x_{t}, \\emptyset))$ ($w > 0$) defined in (3), to conduct the diffusion process. Clearly, to get $\\epsilon_{\\theta}(t, x_{t}, C, \\emptyset)$, one need **two number of model evaluations** on $\\epsilon_{\\theta}(t, x_{t}, C)$ and $\\epsilon_{\\theta}(t, x_{t}, \\emptyset)$. \n\nHowever, as we have empirically verified that text prompt has conveyed their information after a certain time step $t_{w}$, then for $t > t_{w}$, we suggest to substitute $\\epsilon_{\\theta}(t, x_{t}, C, \\emptyset)$ with $\\epsilon_{\\theta}(t, x_{t}, \\emptyset)$ (as in (9)), which means for diffusion steps $t > t_{w}$, we only **one number of model evaluation** to conduct diffusion step, which reduces the computational cost (no matter measured under which metric), compared with the original noise prediction requires two number of model evaluations. \n\nTo see the saved computational cost clear, please check “Saved Latency” in Table 2, where we report the saved latencies of our method under different settings, compared with the original method. \n\n**Q2**: “This paper mainly focuses on the role of EOS token in T2I diffusion models while neglecting the SOS token. Despite the weight of SOS token is significantly higher than SEM and EOS token (see Figure 3 in attached file of General Response). However, the author(s) claims that the **SOS carries no information** due to the autoregressive nature CLIP text encoder. Since this claim is not yet supported by other works, the author(s) should have conducted experiments to support this claim, as there is a chance that EOS and SOS tokens altogether influence the generation process.” \n\n**A2**: Thank you for pointing out this. First, due to the auto-regressive encoding process of CLIP, [SOS] should contain no textual information, while serves as a “dummy variable” to adjust weights of the other tokens in cross-attention module [1]. \n\nMoreover, following your suggestion, we design an experiment to verify that [SOS] indeed contains no textual information. For Text-to-Image generation, given a text prompt, we constructs two prompts, **1)** all 77 tokens are [SOS] from the given text prompt, **2)** expected for the first [SOS] (used to adjust attention map), the other all 76 tokens are [EOS] from the given text prompt. For such two constructed prompts, **they respectively only contain textual information from [SOS], and [EOS]**. We generate images under these prompt, and the generated results are in the Rebuttal Figure 1 in attached file of General Response.\n\nAs can be seen, the images generated with information from [EOS] can be consistent with the target text prompt. However, this phenomenon does not happen for prompt with only information from [SOS] injected. This further verify our conclusion that [SOS] only adjusts the cross-attention map but contains no textual information in line 193.\n\nThanks for inspiring us for such experiment, and we have added this in the revised version. \n\n**Q3**: “Please clarify how the computation cost is defined in this paper and how the efficiency gain is computed.”\n\n**A3**: Please check A1. \n\nReferences: \n\n[1] Xiao et al., 2023. Efficient streaming language models with attention sinks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k6sNoAPxO7", "reviewer": "Reviewer_P41E", "summary": "The paper investigates the denoising process in DPM, identifying that the overall shape of the image is formed early in the process while details are added later. It further examines the influence of different text prompt tokens, finding that the end-of-sequence token [EOS] plays a crucial role in shaping the initial stages of image generation. The authors propose a method to speed up the generation process by removing text guidance after the initial stages, achieving a significant reduction in computational cost.", "questions": "- Can you provide a more detailed explanation of the theoretical aspects of frequency signal analysis used in your study? Specifically, how do the low and high-frequency components influence the denoising process? Including more accessible explanations or visual aids to illustrate the frequency signal analysis could help readers better understand this aspect of your work.\n- Your experiments are primarily based on a specific set of text prompts and Stable Diffusion model versions. How do you ensure that your findings generalize across different models and broader text prompt sets?\n- The paper uses various metrics like CLIPScore, BLIP-VQA, and MiniGPT4-CoT for evaluation. Can you provide a more detailed explanation of why these particular metrics were chosen and how they comprehensively assess the text-image alignment?", "limitations": "- Authors should discuss the robustness of their findings and the need for further experiments across various models and more complex or diverse text prompts to validate their conclusions.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Comprehensive analysis of the denoising process stages in DPM.\n- Detailed exploration of the influence of different tokens in the text prompt.\n- Practical application of findings to accelerate the T2I generation process.\n- Empirical and theoretical support for the proposed acceleration method.", "weaknesses": "- The paper might lack clarity in explaining the theoretical aspects of frequency signal analysis.\n- Limited exploration of potential biases introduced by the dominance of the [EOS] token.\n- The study may benefit from a broader range of experiments to validate the generalizability of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable comments. Here we address your concerns as follows.\n\n**Q1**: The paper might lack clarity in explaining the theoretical aspects of frequency signal analysis.\n\n**A1**: The theoretical aspects of frequency signal are in Proposition 1, where we have actually proved that the added standard Gaussian noise has almost the same magnitude under each frequency spectrum. This means Gaussian noise has more high frequency signals compared with low frequency signals (clarified in line 172 and Figure 2b), since the high-frequency parts contain 80% spectrum. \n\n**In a word, the theoretical aspects of frequency signal analysis Proposition 1 show the added noise mainly contain high-frequency part**, so that the high-frequency part of noisy data will be quickly destroyed during the adding noise process, and will not be recovered until the end of the reverse denoising process. The conclusion is oppositely hold for low-frequency part. Altogether, it explains the “first overall shape then details” phenomenon. \n\n**Q2**: “Limited exploration of potential biases introduced by the dominance of the [EOS] token.”\n\n**A2**: In line 215, we said “an ablation study in Appendix C verifies the influence to the dominance of the [EOS] token.” In Appendix C, we verify the impact of [EOS] partially originates from its dominance number, but also the more information contained in it. \n\n**Q3**: “The study may benefit from a broader range of experiments to validate the generalizability of the findings.”\n\n**A3**: In this paper, we verify the conclusion on simple and complicated text prompts and evaluated on T2I models based on UNet Stable Diffusion 1.5, 2.1 and DIT based model Pixel-Art. To further address your concerns, we conduct experiments on two tasks: subject driven generation and human face generation [1,2]. For such two tasks, we verify whether the conclusion “textual information is conveyed in the first few steps”. The results are in the Rebuttal Figure 2 and Rebuttal Figure 4 in the attached file of General Response. Please refer A3, A4 to Reviewer xbNR for more details.    \n\nBesides, as you suggested, we further verify the conclusion “[SOS] contains more information” on dataset MS-COCO. Please see A5 for details.  \n\n**Q4**: “How do the low and high-frequency components influence the denoising process? Including more accessible explanations or visual aids to illustrate the frequency signal analysis could help readers better understand this aspect of your work.”\n\n**A4**: How does low and high-frequency components influence the denoising process is clarified in A1, such that the add-noise process takes noise with more high-frequency components so that the high-frequency parts in original image will quickly destroyed and will not be recovered until the end of the reverse denoising process. The conclusion is oppositely hold for low-frequency parts. This phenomenon is also visualized in Figure 2b. We will make this more clearly and readable in the revised version. \n\n**Q5**: “Your experiments are primarily based on a specific set of text prompts and Stable Diffusion model versions. How do you ensure that your findings generalize across different models and broader text prompt sets?” \n\n**A5**: As mentioned in A3, we further verify the conclusion that text prompts convey information at the first few steps of denoising process. It shows the conclusion can be generalized to the other tasks. The models we used in this paper includes three T2I generative models (Stable Diffusion 1.5, 2.1 and Pixel-Art). The three models are SOTA of open-sourced diffusion-based generative models. \n\nAs for text prompts, the results in Section 5 are verified on the constructed PromptSet in line 120, which consists of prompts from a benchmark dataset T2I commonbench [3] used to verify the quality of T2I generation. Notice that the prompts contain 1000 natural complex text prompts generated LLM. The prompts are actually diverse. \n\nBesides, the experiments for verifying our second conclusion “The information of text prompt is conveyed during the early stage of denoising process.” in **Section 6 are conducted under 30K text prompts from MS-COCO**, which are diverse enough, and consist with our conclusion. \n\nTo further address your concerns, we conduct additional experiments on 15K pairs of text prompts from MS-COCO dataset with switched [EOS] to further support the conclusion in Section 5, “[EOS] Contains More Information”. The experiment is similar conducted as in Section 5.1 (with switched [EOS]), the results are shown in Table 1, and the generate images are in the Rebuttal Figure 3 in the attached file of General Response. The experimental results show the conclusion is still hold for MS-COCO dataset. \n\n\n|                | Source Prompt | Target Prompt |\n|----------------|---------------|---------------|\n| Text-CLIPScore | 0.2086        | **0.2696**    |\n| BLIP-VQA       | 0.3735        | **0.5655**    |\n| MiniGPT-COT    | 0.6512        | **0.7479**    |\n\n\n**Q6**: “Can you provide a more detailed explanation of why CLIPScore, BLIP-VQA, and MiniGPT4-CoT are chosen as metrics?”\n\n**A6**: As mentioned in line 206, we give an explanation to the three chosen metrics in Appendix B, where we have described these metrics. The three metrics are proposed in [3], which fully utilized the strong text-to-image alignment capability of multi-modality models CLIP, BLIP, MIiniGPT4. They are standard metrics in measuring text-image alignment as mentioned in [3]. Please see more details in Appendix B or [3].  \n\n\nReferences:\n\n[1] Chen et al., 2023. AnyDoor: Zero-shot Object-level Image Customization\n\n[2] Li et al. 2024. Photomaker: Customizing realistic human photos via stacked id embedding. \n\n[3] Huang et al., 2023. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uzQWZrgmFI", "reviewer": "Reviewer_xbNR", "summary": "This paper explores the mechanism in the text-to-image diffusion model, including the generation order of image components, the influence of various tokens, and the steps in which tokens work.\nThese observations bring some insight into understanding the diffusion model.\nBesides, the authors also design a sampling strategy that accelerates the sampling of the denoising process by 25%+.", "questions": "1. Is the proposed sampling strategy still feasible for generating tasks that require preserving specific details, e.g., subject-driven generation?", "limitations": "Suggest the author to discuss the applicability and limitations of the proposed sampling scheme.\nFor example, can it be applied to human face generation without losing human identity?", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. The conclusion of the [EOS] token is interesting and has been rarely investigated in previous papers.\n2. The analytical experiments in this article are sufficient and strongly support its conclusion.\n3. The writing expression of this article is very clear.", "weaknesses": "1. The other conclusions in this paper, e.g., shape first then details, have been discussed in previous works.\n2. The sampling strategy is more like a sample trick than a method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable comments. Here we address your concerns as follows.\n\n**Q1**: The other conclusions in this paper, e.g., shape first then details, have been discussed in previous works.\n\n**A1**: Yes, and we have mentioned in line 161, footnote 5. However, the existing literature only observe this phenomenon by a single case, while we conduct a frequency analysis to systematically explore this phenomenon, and our contributions also include the exploration to the working mechanism of text prompt. \n\n**Q2**: “The sampling strategy is more like a sample trick than a method.”\n\n**A2**: The sampling strategy is an application of the conclusion that text prompt convey their information at first few steps of denoising process. It saves computational cost in a simple yet efficient way. \n\n**Q3**: “Is the proposed sampling strategy still feasible for generating tasks that require preserving specific details, e.g., subject-driven generation?”\n\n**A3**: Thanks for your suggestion, we implement the proposed sampling strategy on a recent subject driven generation method, Anydoor [1]. Here, the visual features of reference image are used as condition to guide image generation. We remove the condition from different time steps during denoising process similar to Figure 10 in our paper. The generated images are given in the Rebuttal Figure 2 of attached file of General Response, and still preserve the specific details as baseline model (start point a=0) when start removing time steps is set to 20. Thus our sampling strategy is suitable for this task.    \n\n**Q4**: Suggest the author to discuss the applicability and limitations of the proposed sampling scheme. For example, can it be applied to human face generation without losing human identity?\n\n**A4**: Following your suggestion, we conduct an experiment on the human face generation task using PhotoMaker [2] to verify our sampling strategy are still applicable. Unlike the T2I task, this experiment includes both text prompts and reference faces as condition information. We applied the proposed sampling strategy (removing all condition information) and find that the generated images, including faces (a=20), are similar to those produced by the baseline method (a=0). The generated images similar to Figure 10 in our paper are in the Rebuttal Figure 4 of the attached file of General Response. \n\nReferences:\n\n[1] Chen et al., 2023. AnyDoor: Zero-shot Object-level Image Customization\n\n[2] Li et al. 2024. Photomaker: Customizing realistic human photos via stacked id embedding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tcuXG63FXo", "reviewer": "Reviewer_zfAn", "summary": "This paper aims to understand two mechanisms of diffusion models. First, the denoising process is analyzed, and it is found that shapes in an image are constructed in the beginning of the denoising process, while textures and details are filled in later. This empirical observation is justified with a mathematical frequency analysis. Second, the role of text conditioning is analyzed and it is found that the [EOS] token, which captures global information of the prompt, is relied on more heavily by the diffusion model. It is also observed that the text prompt is utilized more in the earlier stages of the denoising process. This finding is utilized to speed up diffusion sampling by ~25% while maintaining the image quality and prompt alignment. This is done by only injecting conditional information in the beginning of the denoising process.", "questions": "* What are some of the limitations and implications of these findings?\n* I did not take this into account for my review, but there are many typos in the text and figures which can be corrected for the next version.\n* This (https://arxiv.org/pdf/2404.07724) is a concurrent work, so it is not expected for this paper to compare against it. Their finding is applying guidance in the middle denoising steps improves image quality and distribution coverage. I am curious to hear how the findings this paper being reviewed can be connected to the phenomena observed there. It might be something to keep in mind for the next version, although it will not be used in assessing this paper.", "limitations": "Although there are no societal implications, a discussion of limitations is lacking.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Although the finding that shape is constructed in the first few timesteps has been observed many times before, it is nice to have a more principled study with various experiments and mathematical justification. \n* The finding that the special [EOS] token is the most relied upon during generation rather than the prompt tokens is an interesting finding that can be used in later studies. For instance, improving prompt alignment, attribute binding, etc. \n* The observation that the text prompt is used more in the early denoising process lends itself to a practical application of speeding up inference. \n* Multiple architectures and samplers are used in this study, suggesting the generality of these findings.", "weaknesses": "* As mentioned in the Strengths section above, the findings are not completely surprising (for instance, the shape reconstruction or reliance on text in the early denoising steps, then detail-filling in the later steps). However, this work takes a principled approach in studying these phenomena which have largely been used in diffusion application literature (e.g., [1, 2])\n* Limited to no mention of broader impact or limitations. Furthermore, the Conclusion section is just a summary of the paper but does not discuss the implications of these findings. \n\n[1] @inproceedings{mengsdedit,\n  title={SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations},\n  author={Meng, Chenlin and He, Yutong and Song, Yang and Song, Jiaming and Wu, Jiajun and Zhu, Jun-Yan and Ermon, Stefano},\n  booktitle={International Conference on Learning Representations}\n}\n[2] @inproceedings{hertzprompt,\n  title={Prompt-to-Prompt Image Editing with Cross-Attention Control},\n  author={Hertz, Amir and Mokady, Ron and Tenenbaum, Jay and Aberman, Kfir and Pritch, Yael and Cohen-or, Daniel},\n  booktitle={International Conference on Learning Representations}\n}", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable comments. Here we address your concerns as follows.\n\n**Q1**: “As mentioned in the Strengths section above, the findings are not completely surprising (for instance, the shape reconstruction or reliance on text in the early denoising steps, then detail-filling in the later steps). However, this work takes a principled approach in studying these phenomena which have largely been used in diffusion application literature”\n\n**A1**: We thank you for appreciating our “principled approach in studying these phenomena”. As you mentioned and our claim in line 161, these phenomena has been previously observed by single cases. However, our contributions are conducting a systematically exploration to them by frequency analysis. Besides, we also study the working mechanism of text prompts, and link it with these observed phenomena. \n\n**Q2**: What are some of the limitations of these findings? \n\n**A2**: For the limitation of this paper, owing to the auto-regressive textual encoder used in stable diffusion explored in this paper, it is natural for [EOS] contains more information. However, under bidirectional textual encoder, every token should have the same information magnitude. Thus, the conclusion “[EOS] contains more information” may not hold anymore. \n\nIn fact, when for T2I model Pixel-Art [1] with bidirectional text encoder T5 [2], the conclusion indeed does not hold anymore. However, the other conclusions “first overall shape then details” and “text prompt convey their information in the early stage of diffusion process” are still hold and has been verified in Table 2, where we have conducted the proposed sampling strategy on Pixel-Art. \n\nWe will clarify these limitations in the revised version. \n\n**Q3**: What are some of the implications of these findings? \n\n**A3**: The straightforward implication of our finding is designing the efficient sampling strategy as we proposed in Section 6. Besides, we wonder is there any possibility to combine these findings during the training stage of T2I model, e.g., using our noise prediction (9) during training stage to accelerate training. For the other conditional generation tasks e.g., human-face generation or subject-driven generation, we find that our sampling strategy is still applicable. Please see General Response for more details.   \n\nWe will clarify these implications in the revised version. \n\n**Q4**: “I did not take this into account for my review, but there are many typos in the text and figures which can be corrected for the next version.”  \n\n**A4**: We will carefully check the typos and revise them accordingly in the revised version. \n\n**Q5**: “This (https://arxiv.org/pdf/2404.07724) is a concurrent work, so it is not expected for this paper to compare against it. Their finding is applying guidance in the middle denoising steps improves image quality and distribution coverage. I am curious to hear how the findings this paper being reviewed can be connected to the phenomena observed there.”\n\n**A5**: Thank you for pointing out such interesting reference [3]. **Their main conclusion is actually consistent with ours.** They divide the denoising process into three stages: early, middle, and late. They claim that the guidance (textual prompt in this paper) of conditional generation in late stage is useless, which is consistent with our conclusion that textual prompts convey their information in early stage of diffusion process. Moreover, they claim the guidance existence in early stage of diffusion is strong and make the generation saturates into several modes, so that the guidance should only applied in the middle stage of diffusion process. This is also consistent with our observation that the overall shape is quickly decided by the text prompt in early stage of diffusion process, owing to the strong guidance of text prompt during this stage. \n\nThe main difference between our methods is they propose to remove the guidance in early stage of diffusion process to improve diversity, while we add guidance in this stage. Our method improves the alignments of generated images with target prompts, especially for small or middle size model with relatively poor alignments, while for model with poor diversity, their method seems fix the issue. \n\nWe will add this comparison in the revised version.  \n\nReferences:\n\n[1] Chen et al., 2024. PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis.\n\n[2] Raffel et al., 2019. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. \n\n[3] Kynkäänniemi et al., 2024. Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models。", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
General Response:

We thank all reviewers for their valuable comments. It seems a common question is whether our sampling strategy can be applied to the other conditional generation tasks. 

To verify this, we further apply our sampling strategy to the other two conditional generation tasks: subject-driven generation and human face generation. For such two tasks, there is extra reference image (given subject and human face) used as condition to guide image generation. Our sampling strategy are conducted on backbone methods AnyDoor [1] and Photomaker [2] respectively. 

The generated results of these two tasks are in the Rebuttal Figure 2 and Rebuttal Figure 4 in attached file, and the results show that our sampling strategy is still applied for such two tasks. Because the images generated with conditions removed in the final stage of diffusion are consistent with baseline method (similar to Figure 10 in our paper).

**The Figures for extra experiments are in the attached file.**

References 

[1] Chen et al., 2023. AnyDoor: Zero-shot Object-level Image Customization

[2] Li et al., 2024. Photomaker: Customizing realistic human photos via stacked id embedding.

Meta-review: This paper provides a comprehensive analysis of the denoising process stages in DPM, such as how the shape of the image is reconstructed after a few denoising steps, then filled with details. The final image is mostly determined by the special token [EOS]. The paper also proposes accelerating the T2I generation process by removing text guidance, which could speed up sampling by over 25%.

Initially, this paper received mixed reviews. However, after discussions between the authors and reviewers, the paper received unanimous acceptance from all reviewers, leading to an "Accept" decision.
Decision: Accept (poster)