Zigzag Diffusion Sampling: Diffusion Models Can Self-Improve via Self-Reflection
Authors: Bai LiChen
Shitong Shao
zikai zhou
Zipeng Qi
zhiqiang xu
Haoyi Xiong
Zeke Xie
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion model
Semantic Information
Classifier Guidance Gap
Primary area: generative models
TL;DR: We propose a novel guidance mechanism that leverages guidance gap to inject semantic information step-by-step throughout the entire generation process.

Abstract: Diffusion models, the most popular generative paradigm so far, can inject conditional information into the generation path to guide the latent towards desired directions. However, existing text-to-image diffusion models often fail to maintain high image quality and high prompt-image alignment for those challenging prompts. To mitigate this issue and enhance existing pretrained diffusion models, we mainly made three contributions in this paper. First, we propose **diffusion self-reflection** that alternately performs denoising and inversion and demonstrate that such diffusion self-reflection can leverage the guidance gap between denoising and inversion to capture prompt-related semantic information with theoretical and empirical evidence. Second, motivated by theoretical analysis, we derive Zigzag Diffusion Sampling (Z-Sampling), a novel self-reflection-based diffusion sampling method that leverages the guidance gap between denosing and inversion to accumulate semantic information step by step along the sampling path, leading to improved sampling results. Moreover, as a plug-and-play method, Z-Sampling can be generally applied to various diffusion models (e.g., accelerated ones and Transformer-based ones) with very limited coding and computational costs. Third, our extensive experiments demonstrate that Z-Sampling can generally and significantly enhance generation quality across various benchmark datasets, diffusion models, and performance evaluation metrics. For example, DreamShaper with Z-Sampling can self-improve with the HPSv2 winning rate up to **94%** over the original results. Moreover, Z-Sampling can further enhance existing diffusion models combined with other orthogonal methods, including Diffusion-DPO. The code is publicly available at
[github.com/xie-lab-ml/Zigzag-Diffusion-Sampling](https://github.com/xie-lab-ml/Zigzag-Diffusion-Sampling).

Published as a conference paper at ICLR 2025
ZIGZAG DIFFUSION SAMPLING: DIFFUSION MODELS CAN SELF-IMPROVE VIA SELF-REFLECTION
Lichen Bai' Shitong Shao'
Zikai Zhou' Zipeng Qi
Zhiqiang Xu Haoyi Xiong' Zeke Xiel
*xLeaF Lab, The Hong Kong University of Science and Technology (Guangzhou) BBehhang University *Mohamed bin Zayed University of Artificial Intelligence {lichenbai, sshao213, zikaizhon zekexie/@hkust -z edu.cn
"Baidu Inc
DIFFFFUISSON
Standdrrd Sumpling
DIFFUSION
Z-SamplingttOnm
Postion:
Texe
right
ploto table
two
MinoCruf Syyl.
dog
Figure 1: The qualitative results of Z- Z-Sampling demonstrate the effectiveness of our method in various aspects, such as style, position, color, counting, text rendering, and object co-occurrence. We present more cases in Appendix D.2.
ABSTRACT
Diffusion models, the most popular generative paradigm far, inject con- ditional information into the generation path to guide the latent towards desired directions. However, existing text-to-image diffusion models often fail to main- tain high image quality and high prompt-image alignment for those challenging prompts. To mitigate this issue and enhance existing pretrained diffusion mod- els, we mainly made three contributions in this paper. First, we propose disfusion self-reflection that alternately performs denoising and inversion and demonstrate that such diffusion self-refiection can leverage the guidance gap between denois- ing and inversion to capture prompt-related semantic information with thcoreti- cal and empirical evidence. Second, motivated by thcoretical analysis, we derive Zigzag Diffusion Sampling (Z-Sampling). a novel self-reflection- based diffusion sampling method that leverages the guidance gap between denosing and inversion to accumulate semantic information step by step along the sampling path, leading improved sampling results. Moreover, can be generally applied to various diffusion models (e.g., accelerated ones and Transformer-basee ones) with very limited coding and computational costs. Third. our extensive experiments demonstrate that Z- Z-Sampling can generally and signif- icantly enhance generation quality across various benchmark datasets, diffusion models, and performance evaluation metrics. For example, DreamShaper with Z-Sampling can self-improve with the HPSv2 winning rate up to 94% over the original results. Moreover, Z-Sampling can further enhance existing diffusion models combined with other orthogonal methods, including Diffusion-DDP The code is publicly available at ggthuu..com/xi-llle wawiaiiaaaaam//iiia---aaaaa Diffusion-Sampiaapie
plug-and-play method, Z-Sampling
*Corespoondigg author

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Diffusion models, known for its powerful generative capabilities and diversity, have become main- stream generation paradigm in images (Podell et al. 2023 Lin et al., 2024bb Qi et al. 2025 videos (Ho al. 2022 Blattmann et al. 2023); and 3D objects (Luo Hu 2021 Voleti et al.
2024) and beyond. One key ubility of diffusion models is to guide the sampling path based on some conditions (e.g., texts), leading to conditional or controllable generation (Ho & Salimans 2022).
However, while strong guidance may improve semantic alignment to those challenging prompts, it often causes significant decline in image fidelity, leading to mode collapse, and resulting inevitable accumulation of eπrors during the sampling process (Chung et al. [024). To mitigate this issue, some studies apply additional manifold constraints to the sampling paths (Chung et al.] [0224] Yang et al.; He et al.). which compromises the diversity of generated outputs. Others design varying guid- ance scales across different denoising regions to mitigate this issue (Shen et al., 2024). but such explicit strategies often lead to unnatural outputs. Thus, enhancing high generation quality while maintaining prompt alignment effectively during sampling remains a crucial challenge, especially for those challenging prompts. This challenge may require more controllable prompt guidance be- yond classical guidance like classifer-free guidance (Ho & Salimans 2022)
Fortunately, discover that semantic information may be inherently embedded in the random latent space, influencing the quality of image generation (Xu et al. 2024b Po- Yuan et al. 2023 Mao et al., 2023b Wu et al., 2023c). In Figure 2 demonstrate the following phenomenon: if latent can generate images aligned with a specific concept under no conditional prompt, it will generate high- quality results with as the conditional prompt. This implies that the latent naturally carries relevant semantic information and can align with relevant semantic prompts very well. Figure 3 intuitively illustrates that the green initial point with certain semantic information is usually superior the red initial point for the prompts associated with the semantic information.
Low Guidanos Sale
Latent without senatic information
unconditooal
Prompe reloned "Flowe"
Prompe reloted to "Mue"
High Gaithnee
Latent sematic infirmatio
Seed:
21,8685
212177
Seed: 14
19,5408
21A4744
Figure 2: Semantic-rich latents effectively generate images aligned with intended semanties. For in- stance, the random latent (seed 21) is better suited for generating images related the concept of "flow- ers". We present more cases in Appendix C.I
Figure 3: If the latent carries seman- tic information, we can obtain prompt- related results from this latent even without conditional guidance.
We fortunately discover that employing strong guidance during denoising process and weak guid- ance during inversion process establishes guidance gap between denoising and inversion that can inject prompt semantic information to the latent. We present more examples and discussion in Ap- pendix C.2 Can this insight lead to improved sampling methods? We note that large language models (LLMs) can self-improve reasoning through self-reflection (Ji et al. 2003) Shhin et al.) 2024). However, the self-reflection mechanism that self- self-improve diffusion sampling has not been reported in previous studies.
Motivated by observation and self-reflcction in LLMs, the best of our knowledge, the first to formulate diffusion self-reflection that let a latent denoise a zigzag manner, namely a denoising step and an inversion step alternately, step by step along the sampling path. As Figure
2

---

Published as a conference paper at ICLR 2025
illustrates, we propose Zigzag Diffusion Sampling, Z-Sampling. which capture semantic information with such repeated zigzag self-reflection operations and move to more desirable results along the sampling path. Through each zigzag self-reflection operation, the latent accumulates more semantic information.
The contributions of this work be summarized follows.
First, we theoretically and empirically demonstrate that the guidance gap between denoising and capture the semantic information embedded in the latent
inversion of diffusion self-reflection space, which matters to generation quality and prompt-image alignment.
Second, motivated by the theoretical results, derive Z-Sampling, novel seffrreflection-basee diffusion sampling method that can leverage the guidance gap to accumulate semantic information through each zigzag self-reflection step and generate more desirable results. allows flexible control over the injection of semantic information and is applicable across various diffusion architectures with very limited coding costs.
Third, extensive experiments demonstrate the effectiveness and generalization of Z-Sampling across various benchmark datasets, diffusion models, and evaluation metrics. As theoretical analysis sug- gests, diffusion models with 7-Sammling especially self-improve for challenging complex or fine- grained prompts, such as position, counting, color attribution, and multi-object, breaking through the performance peak of pretrained diffusion models. Moreover, orthogonal methods, such as Diffusion- DPO (Wallace et al., 2024), can further self-improve with Z-Sampling. Importantly, as training- free method, Z-Sampling can still exhibit significant improvements over the baselines with limited computational cost, which suggests its efficiency and practical value. In the efficiency study, even with 36% less computational time, Z-Sampling can reach the best performance of standard sampling.
Low Guidence Scale
Grussai Dismbbttio
Algorithm Z-Sampling
High
Taraet Datribution
I: Input:
Denoising at timestep t: Ø', In- version at timestep t: VV, text prompt: c, denoising guidance: 71. inversion guidance: 72, inference steps: T", zigzag
optimization steps: À
b)
2: Output: Clean image To
3: Sample Gaussian noise 2r
a)
for to
5: 1T-1 = 4 =0'male.
6: if >T T then
7:
#Inversion by equation
5. 4/ (x
T-1
8: #Denoising by equation?
J. = 46 (ā,lc m)
T
9: end if
10: end for
Figure 4: The illustration of our method: a) weak guidance sampling: b) strong guidance sampling; c) Z-Sampling (with diffusion self-reflection.
return 200
PRELIMINARIES
In this section, we formally introduce prerequisites and background.
Diffusion Model. We define the total numeber of denoising steps T and conditional prompt Given the denoising procss N + D and guidance scale 71, starting from øT N, we can generate Zo ( 7)) D, where A represents the distribution of Gaussian and D represents the distribution target data. We note that the mapping function corresponds to the probability simplify only the initial input 2y in Φ. Similarly, we also
P(zolc, c, M1 F. ). For simplicity, reverse this process, given the inversion process
C + N under guidance scale 72, we obtain inverted data दr V (5olc, 72) E N from in D.

---

Published as a conference paper at ICLR 2025
Following Ho et al. 2020), treat diffusion model as Monte Carlo process and decompose + into times single- step denoising mappings as
(arle, tx (xrle, o" (oall
(122)e) o o (milem))
(1)
And we define Φ as
Tt = Φ (x () 7) va--1 St v (w))
+ (1-a--r((ma))
(2)
where de = I (1- 3:) and 3, are the pre-defined parameters for scheduling the scales of adding noises in DDIM scheduler (Song et al. (220)). we denote c) as the predicted score by the denoising network at timestep t, with further details provided in the next paragraph.
Similarly, for the inversion process Vr we can also perform this decomposition as
M ( c, 722 = V (øulc. c, 72) o v (31lc, 12) o Ti TT where we obtain - 1 via & as
V (ăT-2lc, (2,))) v (#r-1le,77))
(3)
Vai (
1 01-1
t. V (i2-11e,7)) =
E1
(0)-)))
(4)
04-1
In equation 4 approximate the score predicted at timestep t with timestep t along the inversion path, i.e, set c 2 = (,,)). If this approximation error is negligible, and v can be proven to be inverse functions (Mokady al. 2023 meaning that V --11
Classifier free guidance. Controllable generation typically involves guiding or constraining the semantic representation. In classifier free guidance (Ho & Salimans (022). score prediction net- work is trained both conditionally and unconditionally. During inference, denoising scores are computed by interpolating between conditional and unconditional scores predicted by ug. thus en- abling the adjustment of guidance scale across various levels.
Specifically, for denoising and inversion process, we use guidance scales "/ and 72, with the corre- sponding scores as
(xx)) = (1 + (1) (
ruul 0,1)
(5)
(),, (1 + (ā, 72uo 2,t)
where ug is the noise predictor, and Ø is the null prompt, representing the denoising result under unconditional settings.
METHODOLOGY
In this section, discuss how to encode semantic information into latents through the guidance gap and derive Z-Sampling according to theoretical analysis.
3.1 LATENTS WITH RELEVANT SEMANTIC INFORMATION
Our inspiration stems from the question: what makes a good latent in the diffusion process? As Figure 3 illustrates, we argue that a latent with relevant semantic information (green point) can align with the prompt under weak or sometimes even negative conditional guidance. In contrast, a latent lacking semantic information (red point) necessitates strong conditional guidance to attain comparable aligmment and may remain unaligned under unconditional generation.
To verify this, we generate images using different latents (seeds) under unconditional settings, shown in Figure 2 We observe that if latent can generate a image certain concept e unconditionally. then, under certain prompt guidance, this latent usually performs higher in generating images re- lated to c compared to other latents. For example, in Figure 2 if the latent (seed 21) generates the images of flowers unconditionally. it yields higher-quality images when used with flower-related prompts in conditional generation. Previous studies also argued that the properties latents par- tially predetermine image composition contents during generation, affecting object position, size, and depth (Wu et al. 2023c Guttenberg/ 2023) Lin et al.) [2224a] Xu et al.) 2024b) Mao et al. (2023)). However, they did not formally explore how encode semantic information into the latents.
4

---

Published as a conference paper at ICLR 2025
3.2 CAPTURE SEMANTIC INFORMATION FROM THE GUIDANCE GAP
Considering a denoising process Φ : .N's x C + D, under text condition с E C, we sample an initial latent vт N, and obtain the generated data x5
no
where ? is condition guidance scale during denoising. Now, we further perform inversion operation on Zo under the guidance scale of 7/2 as
PT V e, 12)-
the approximation error in the inversion process negligible, meaning T- 4b, then equation can be equivalently inverted as
= 72) @(ārle, 72).
Generally, the denoising guidance scale Yi set common value (e.g., 71 5.5) to maintain standard generation and alignment to the prompt (Ho & Salimans, 2022). Conversely, the inversion guidance scale "/2 usually set to small value (e.g., 72 0) to achieve inversion with weak guidance (Mokady et al., 2023). By comparing equation 6 and equation 8 we note that starting from Ëт, we can generate Zo under weak or even unconditional guidance scale 7/2 = In contrast, starting from fy requires strong conditional guidance scale 7/1 5.5 to produce similar results.
According to the insight discussed in Section 3.1 if a initial latent can generate results related to prompt under weak guidance, it indicates this latent contains more semantic information related to Since guidance scale 7/2 is less than 71, we argue that the corresponding inverted latent दy contains more semantic information compared to TT. We present more empirical evidence in Appendix C.2
3.3 ZIGZAG DIFFUSION SAMPLING
Now we know that the guidance gap can capture additional semantic information. The next question how to effectively leverage this property to inject semantic information into the sampling process.
Vanilla Inversion A vanilla way is to use the inverted latent är in place of 2r as the starting point to generate semantically aligned results the denoising process (see Algorithm 2). We pro- vide Theorem [ and show that the difference between the original øт and the inverted TT, namely dendlend = (IT )²)²² may reveal how significant the vanilla end-to-end information injection is. An illustrative diagram of the latents' difference is provided in Figure 27 (a) of Appendix
Theorem (See the proof in Appendix F.1 For random latent xT N and an inverted latent ïT given by equation] the latent difference SendZend between or and öT
Sendlend (FT )² = aT
hel
((())))
+ (zr())))))
(9)
Tv (fysemantic information gain term
T2 =appppxx evvor term
where by V//a- 11/0-11 - and (-) is the predicted score given by equation
Here, 7i (t) represents the semantic information gain induced by the guidance gap at timestep t, whereas (t()) represents the approximation error inherent in the inversion process, which may be neglected for semantic information. We note that in equation 9 the end-to-end aggregation may let the sum of the semantic information TI over each step be small and fail to accumulate the desired semantic information gain step-by-step.
Z-Sampling To let Ti of each step be accumulated step-by-step instead of being canceled out in the vanilla sum, we decompose & into { $2 .4..... "" as defined in equation | We first
denoise to obtain It-1 4' (zy 71) and then we invert 2t-1 1 to get ï, = (1' each timestep
4' [T. We may call such zigzag benoising-and-invee diffusion sampling path as diffusion self-reflection. The proposed Z-Sampling method is presented in Algorithm [ and illustrated in Figure 4 Note that Z-Sampling injects semantic information by replacing 2, with _, at each timestep. We prove Theorem 2 and demonstrate the cumulative latent
(#z-1 (2) for operation along the
difference oz - Sampling Σtn ( ). 2, depicted in Figure 27 (b) of Appendix F
5

---

Published as a conference paper at ICLR 2025
carrot on the left of a broccoli.
A blue colored dog.
One car on street.
blue bird and brown beur.
Z.San Z-Sarpiin Sundand Sampine
Figure 5: The cross-attention map highlights the interaction between the entity token (red color) and latent variables. 7-Sampling optimizes the latent so that it is more suitable for generating concepts the promme. For example, in the zigzag path of the second column, semantically injected latents exhibit sharper attention on "dog" with relatively clear boundaries.
Suppose Tt is the denoised latent at step I, and Tt be the corresponding inverted latent given by equation Then the cumulative latent difference in 7-Sampling can be written as
Theorem (See the proof in Appendix F.2)
ÖZ-Sampling Σ (oo--ai 1ec-aa zz)²
arhi(
- (Ttr
((6 (10)
Ty (1):semantic information gain term
73 (t) (t)aaprrx crna term
where be and (.) consistent with Theorem
Again, focusing on the semantic information gain term, we report that dendZend O (ET (t)) holds for vanilla inversion and 8z Sampling Et (1)) holds for Z-Sampling. Given the Jensen's inequality, we have Σ (t)) E (t))) showing that the cumulative semantic information gain 8z Sammpln is larger than the end-to-end semantic information gain ōandZend- The se- mantic information gain induced by the guidance gap in Z-Sampling can be effectively accumulated, solving the previous issue of the semantic information gain cancellation.
We further prove Theorem 3 and show the significant impact of the guidance gap 6, on fzzzzzzsaan -Sammplin
Theorem (See the proof in Appendix semantic information gain Z-Sampling can be written as
F3
Under the conditions of Theorem 2 the cumulative
8z Sampling
Σ one (5, (uo(se, c,t) uo( (0,),,0,)))) 2))))
(11)
where the guidance gap is defined as 8., = 71 7/2-
We note that the larger dy, the more pronounced the effect of 7-Sampling. When 8, 0, it is approximately equivalent to standard sampling. This is also empirically verified in Figure
In Figure 5 visualize the cross-attention map of Z-Sampling during the early stages (i.e, t/T 49/50) of the generation process. observe that Z-Sampling indeed makes the attention regions corresponding to entity tokens Z-Sampling on the semantic information gain. Mao et al.] (2023b) reported that certain regions in random latents induce objects representing specific concepts, which aligns with observation that Z-Sampling enhances the association of certain regions with the prompt. Additionally, we discuss the impact of the approximation error 72 Appendix E.2 and E.3
semantically focused, further illustrating the effectiveness of
EMPIRICAL ANALYSIS
In this section, we conduct extensive experiments to demonstrate the effectiveness of our method.
4.1 EXPERIMENTS SETTING
Datasets Pick-a-Pic Kirstain et al. 2023). Draw Bench dataset (Saharia et al. 2022). and
GenEval (Gosh et al. 2024). We leave more details in Appendix A.

---

Published as a conference paper at ICLR 2025
Metrics We multiple evaluation metrics, including HPS v2 (Wu al. 2023c PickScore (Kirstain et al. 2023), and ImageReward (IR) (Xu et al. large-scale human preference datasets, providing reliable indication of genuine human prefer- ences. Furthermore, we also employ the traditional metric AES (Schuhmann et al. 2022). which
2024a). They are trained on
purely evaluate image quality. More details are found in Appendix A.2 A.2
Diffusion Models We use various diffusion models as the generation backbone in main experi- ments. For SD2.1 (Rombach al., 2022). SDXL (Podell al. 2023). and Hunyuan- DiT (Li al. 2024), we perform 50 denoising steps. For SaaunShape-xx-v2-tttee
which achieves efficient and high-quality generation by fine-tuning SDXI. Turbo (Sauer et al., 2023), we set denoising step T only to 4. And we set "1 5.5 in SDXI/SD221, "1 = 6.0 in Hunyuan-DiT, and SaauuShhapr-xl-2-uuue
3.5 in all to the recommended default values. We set the zigzag operation to be executed throughout the entire path (A T and inversion guidance scale "22 zero, unless specify them otherwisely.
Baselines We validate the effectiveness of 7-Sampling and compare it against the following base- lines: (a) standard sampling, we first select four models: SD-2.11 SDXL. Hunyuan-Dif, and PreamShaper-xl-22-uffe
(b) resampling (lug, 2022), repeatedly performs denoising at the same timestep by adding random noise to maintain the latent on the data manifold. Due to the page limit, other baselines such as AYS Sampling (Sabour al. 2024). Diffusion DPO (Wallace et al. 2024).
SEG (Hong. 2024), and CFG++ (Chung et al. 2024) are discussed in Appendix A.3
4.2
MAIN EXPERIMENTS
Pick-a-Pic
DrawBench
18
70
18
75.5
Pak Score
Score
ABS
AES
F1.5
NPS
HPS
Preforon.co/1s
Proference/95
Figure 6: The winning rates of 7-Sampling over standard sampling. The blue bars represent the side of our method, The orange bars represent the side of the standard sampling. Model: DreamShaper- xl-v2-turbo. We present more results in Appendix D.3
Table 1: The quantitative results of Z-Sampling on Pick-a-Pic and Draw
Method
Pick-a-Pic
DrawBench
HPS
AES
PickScore
IR
HPS
AES
PickScore
IR
Standard
23.05
5.28
19.08
-43.66
23.90
5.20
20.49
-4.34
SD-2.1
Resumpling
24.46
5.46
19.51
-18007
23.94
5.08
20.40
-3.900
7. 7-Samliing(uurs)
-23.61
Standard
29.89
6.09
21.63
58.65
28.81
5.56
22.31
60.75
SDXI
Resampling
30.54
6.04
21.73
78.60
29.62
5.58
22,52
72.69
Z. S-Smplingg
31.28
21.85
79.22
30.50
5.68
22.46
79.97
Standard
30.04
21.59
21.77
DreanShaper
Resampling
31.42
6.04
21.95
82.43
28.55
5.39
22.32
64.69
-X--2-turbo
Z Z-Sampling(ours)
32.38
6.15
22.11
90.87
29.90
5.64
22.35
73.51
Standard
30.82
21.88
94.22
30.22
5.70
22.29
82.63
Hunyuan-DiT
Resampling
31.10
21.87
95.51
30.72
22.32
95.88
Z. (oump)
31.12
6.31
21.90
97.88
30.53
5.75
22.40
96.13
In Table I we evaluate our method against standard sampling and Resampling across various dif- fusion architectures, including U-Net, DIT, and distillation architectures. Z-Sampling achieves top
7

---

Published as a conference paper at ICLR 2025
Table 2: The quantitative results of Z-Sampling on GenEval. Model: SDXL
Method
Single object t
Two object
Counting
Colors f
Position f
Color attribution
Overall
Standard
97.50%
69.70%
33.75%
86.71%
10.00%
18.00%
52.52%
Resampling
98. 75%
76.77%
38.75%
88.30%
5.00%
20.00%
54.59%
Z- Z-Sampling(ours)
100.00%
74.75%
46.25%
87.23%
10.00%
24.00
57.04%
Table 3: The quantitative results of 7-Sampling and Semantic-CFG. Model: SD-2.1. For fairness, we follow the default settings of Semantie-CHC with the 768 768 resolution and SD-2.1.
Pick-a-Pic Pic
DrawBench
Method
HPS v2f
AES
PickScoret
IR1
HPS v2f
AEST
PickScoref
IR
Standard
25.67
5,66
20.20
0.53
25.98
5.37
21.39
6.77
Semantic-aware CFG
26.02
5.65
20.28
2.03
26.03
5.37
21.38
9.39
Z-Sampling(ours)
27.05
5.74
20.41
36.89
26.71
5.45
21.55
25.42
Table 4: Z-Sampling can enhance the training-free AYS. Model: DreamShappr:xt-2-tutto.
Pick-a-Pic
DrawBench
Method
HPS v2f
AESt
PickScoret
IR1
HPS v2
AES1
PickScoret
IRT
Standard
32.80
6.05
22.31
91.48
30.94
5.57
22.68
77. 17.44
Z- (-Sampling(ours)
33.53
6.16
22.45
103,95
31.92
5.71
22.78
95.82
AYS
32.78
6.05
22.32
91.88
30.95
5.57
22.68
77.85
AYS + M (ampllinng(()))))
33.57
6.15
22.45
104,22
31.93
5.72
22.75
94.82
Table 5: Z-Sampling enhance the training-based Diffusion-DPO. Model: SDXL.
Pick-a-Pi Pic
DrawBench
Method
HPS v21
AES1
PickScore?
IR1
HPS v21
AES1
PickScore
IR1
Standard
29.89
6.09
21.63
58.65
28.81
5.56
22.31
60.75
Z-Sampling(ours)
31.28
6.13
21.85
78.22
30.50
5.67
22.46
79.97
Diffusion-DPO
31.41
5.60
22.00
90.28
29.80
5.66
22.47
85.94
Z-Sppliin(oours
31.60
6.08
22.18
94.48
30.35
5.67
22.47
93.34
performance across nearly all metrics and Figure the winning rates across these two bench- HPS v2. Furthermore, for more detailed comparison, present results on GenEval (Ghosh et al. 2024). which serves as a challenging benchmark. As Table 2 show, Z-- Sampling significantly enhances aligmment in aspects such as counting, two-object relations, and color attribution, further demonstrating the effectiveness of our method.
marks, exceeding 88
We also compare our method with recent sampling technique designed enhance semantic in- jection. Shen et al.] (2024) proposed Semantic- CFG, dividing the latent into independent semantic regions each denoising step and adaptively adjusting their guidance, thereby unifying the effects across regions. While the setting is different from previous experiments, this results still underscore the effectiveness of Z-Sampling remains unaffected. As shown in Table 3 we observe that 7-Sampling demonstrates higher improvement.
Moreover, we present more quantitative experimental results in Appendix DDI and more qualitative comparison across various dimensions (e.g, color, style, and etc.) in Appendix D.2
Specifically, we also discuss the effect of Z-Sampling under extremely high CFG guidance in Ap- pendix D.4 demonstrating its ability to achieve favorable balance between image quality and prompt adherence, suppressing artifacts and oversaturation.
Z-Sampling can be combined with other orthogonal methods to further enhance diffusion models. In Table 4 Z-Sampling further enhances AYS-Sampling, sampling strategy that optimizes the denoising scheduler, leading to improved overall performance. Note that AYS-Sampling only released the 10-step scheduler, which is more applicable DreamShaper-
Orthogonal Methods
8

---

Published as a conference paper at ICLR 2025
HPS
Pick Score
Image Reward
0.32
22.0
1.00
6.20
208074
0.75
0.30
21.5
6.35
0.28
6.30
21.0
0.26
0.00
6.05
20.5
0..2
0.00
0.50
20.9
5.95
405
19.5
1.00
5.90
V
Figure 7: Robustness the inversion guidance scale. When the gap is zero, i.., the inversion guidance equals the denoising guidance (c.g. 71 = 72), the positive gains almost disappear.
Standard Sampling
8y =-5 5
6, =- 3
8y =
6y
6,
HPS v2 02445
HPPS +2 0.71
0.3856
0.7544
0.3394
0.342
cinematic still highly reflective stainless in the desert, sunset.
HPS +2
HPS 8:M0
HES +2 12286
HPS +2:
HPS +2 35399
HTS +2
0.167
A woman with trained muscles.
Figure 8: The guidance gap 8, between 71 and ?2 influences both the magnitude and direction of semantic injection. When 6, is large (6,=5), the gain of Z-Sampling becomes pronounced. Con- versely, when d, is zero even negative, it approximately degenerates into standard sampling or significantly break generation.
v2-turbo. Additionally, Table shows that Z-Sampling can also be combined with training- -ased methods, further enhancing the generation quality of Diffusion-DPO. We leave more quantitative results of enhancing orthogonal methods in Table
shows
The Guidance Gap We first examine the impact of guidance scale. In Section 3.1 we show that the guidance gap between denoising and inversion dictates the degree of semantic information gain. To further verify this, we fix the guidance scale 71 as 5.5 following standard sampling. By varying 72, we control the guidance gap 6, = 71 72 to observe its impact. As shown Figure 7 when 722 increases and the guidance gap 8, narrows, the benefits of Z-Sampling diminish. According to the theoretical results of semantic information gain, a zero guidance gap can approximately lead standard sampling. When the gap is below (72 7)) it result in negative gain. In Figure 8 we present a qualitative analysis showing that when the zero guidance gap indeed yields very similar results to standard sampling.
Zigzag Diffusion Steps We note that À indicates the first steps using the zigzag operation. For example, when À is 0, it reverts standard SDXL. When À is 25, it means the first 25 steps of the denoising process use the zigzag operation. We conducted experiments on Pick-a- a-ckk using SDXL (50 steps), as shown in Figure 9, when À increases from 0 to 25, the winning rate rises from 50% to 75%. However, when À increases from 25 to 50 steps, it only rises from 75% to 80%. This indicates that the zigzag operation is more effective during the early stages of denoising process.

---

Published as a conference paper at ICLR 2025
0,319
80.0%
Samplnng
0,301
Winiegg Ratos
70.0%
HPS V22
0.303
65.0%
0.293
55.0%
0.241
50.0%
0.299
20 10
0.a
10s
12s
14
16a
Zigrag Step
Time Spent per Image (s)
Figure 9: Robustness to the zigzag diffusion steps A. The horizontal axis shows the number of zigzag operations, and the vertical axis rep- resents the winning rate over HPS v2 on Pick- a-Pic. As À increases, generation quality im- proves, indicating effective semantic informa- tion gain throughout the whole path.
Figure 10: 7-Sampling outperforms standard sampling with the same time consumption and significantly enhance the performance peak of pretrained SDXL. The horizontal axis shows the average time per image with various denoising steps, while the vertical axis shows the average HPS v2 the Pick-a-Pic benchmark.
Efficiency Comparison When the denoising steps are fixed (e.g., T=50), 7-Sampling naturally incurs additional time consumption due to the zigzag step. Suppose the timestep lengths for Stan- dard Sampling and Z-Sampling are T, and Tg, respectively, then the corresponding noise prediction operation times are T, and 2 + 2A. To facilitate a fairer comparison terms of computation time, we set T- = IT, and À I7. This allows to compare evaluation scores under the same gener- ation time consumption per image. Figure I0 indicates that Z-Sampling significantly outperforms standard sampling and enhance the performance peak. Even with 36% less computational time, Z. 7-Sampling can surpass the best performance of standard sampling.
5 DISCUSSION AND CONCLUSION
Discussion We further discuss the limitations and future directions of our work. First, we note that 7-Sampling relies the semantic information gain through deterministic inversion, limiting its applicability to deterministic samplers, such as DDIM. Extending it to the SDE-based diffusion framework is important direction for future work (see Appendix E.II) I.. Second, while Z-Sampling exhibits strong generalization, we only studied text-to-image diffusion models in this work. There- fore, exploring its applications areas such as video generation, 3D generation, and molecular synthesis is naturally another promising research direction. However, due to the different natures of latent space and sampling schedulers, this direction may require further algorithm design and theoretical understanding. Third, while the extra computational cost is acceptable according to the experiments, Z- E-Sampling takes more computational time on its zigzag steps. It will be interesting to distill the path of Z- Z-Sampling into the model itself.
Conclusion To the best of our knowledge, this work is the first to theoretically and empirically discover that the guidance gap between denoising and inversion can inject semantic information into the latent space, which can lead to improved generation. By theoretically investigating how the semantic information gain depends on the guidance gap, we naturally derive Z-Sampling. novel self-reflection-ber based diffusion sampling method that can accumulate semantic information through zigzag self-refled self-reflection operation and, thus, generate more desirable results. The extensive experi- ments not only demonstrate that various models can self-improve significantly with Z-Sampling in various settings, but also suggest that Z-Sampling can further enhance other orthogonal methods. In summary, Z-Sampling is flexible, additive, and powerful with limited coding computation costs. Given the theoretical mechanism and empirical success of ZaSmpling and diffusion self- self-reflection, believe this work motivate better theoretical understanding of visual generation and inspire more advanced sampling methods. Moreover, this approach will soon incentivize video generation, 3D generation, and beyond.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
We propose Z-Sampling, a novel guidance mechanism designed to enhance the quality of diffusion model generation. Although it does not directly involve human subjects issues related dataset privacy, we have carefully considered its potential ethical and moral implications. We ensure the transparency of all datasets used for debugging and developing the algorithm, and their randomness guarantees the absence of bias in the ethical domain, which is of utmost importance. Additionally, all models used comply with the terms of open-source licenses. Given Z-Sampling's 's significant commercial potential, we strive to apply this technology responsibly, ensuring that its applications yield positive societal benefits.
ACKNOWLEDGEMENT
This work supported by the Science and Technology Bureau of Nansha District Under Key Field Science and Technology Plan Program No. 20247I002. This work was supported by Guangdong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Thing(NNo22233312200000000
REFERENCES
Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11661-11471 2022.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 843-852, 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:231..15127 2023.
Hyungjin Chung, Jeongsol Kim, Geon Yeong Park, Hyelin Nam, and Jong Chul Ye. Cfg++: Manifold-constraneed classifier free guidance for diffusion models. arXiv preprint arXiv:2406.08077 2024.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt, Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang. Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. arXiv preprint arXiv:2307.04725. 2023.
Nicholas Guttenberg. Diffusion with offset noise, 2023.
Yutong He, Naoki Murata, Chich Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim. Wei- Hsiang Liao, Yuki Mitsufuji, J Zico Kolter, Ruslan Salakhutdinov, et al. Manifold preserving guided diffusion. In The Twelfth International Conference on Learning Representations.
Jonathan Ho and Tim Salimans. arXiv:2207.12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi. and David J Fleet. Video diffusion models. Advances Neural Information Processing Systems, 35:8633- 8646, 2022.
11

---

Published as a conference paper at ICLR 2025
Susung Hong. Smoothed energy guidance: Guiding diffusion models with reduced energy curvature of attention. arXiv preprint 24Ki::2008.00660, 2024.
Susung Hong. Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. arXiv preprint arXiv:2210.000 00939, 2022.
Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating hallucination in large language models via self-reflection. arXiv preprint arXiv:2310.06277 2023.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-3666,, 2023.
Kunchang Li. Yali Wang. Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19948-19960, 2023.
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint 2458722405.08748, 2024.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVE winter conference on applications of computer vision, pp. 5404-5411, 2024a.
Shanchuan Lin, Anran Wang, and Xiao Yang. Sáxl-lightning: Progressive adversarial diffusion distillation. arXiv preprint arXiv:2402. 13929....,9,,, 2024b.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Buhua Liu, Shitong Shao, Bao Li, Lichen Bai, Zhiqiang Xu, Haoyi Xiong. James Kwok, Sumi Helal, and Zeke Xie. Alignment of diffusion models: Fundamentals, challenges, and future. arXiv preprint MXXv:2409900255: 2024a.
Yuanxin Liu, Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen, Xu Sun, and Lu Hou. Fetv: A benchmark for fine-grained evaluation of open-domain text-to-video generation. Ad- vances in Neural Information Processing Systems, 36, 2024b.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2837-2845, 2021.
Jiafeng Mao, Xueting Wang, and Kiyoharu Aizawa. Guided image synthesis via initial image editing in diffusion model. In Proceedings of the st ACM International Conference on Multimedia, pp- 5321-5329, 2023a.
Jiafeng Mao, Xueting Wang, and Kiyoharu Aizawa. Semantic-driven initial image construction for guided image synthesis in diffusion model. arXiv preprint arXiv:23/2.08872, 2023b.
Ron Mokady, Amir Hertz, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6038-6047 2023.
Mao Yu-Y Shashank Kotyan, Tham Yik Foong, and Danilo Vasconcellos Vargas. Synthetic shifts to initial seed vector exposes the brittle nature of latent-based diffusion models. arXiv preprint arXiv:2312.1144 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
12

---

Published as a conference paper at ICLR 2025
Zipeng Qi. Lichen Bai, Haoyi Xiong, and Zeke Xie. Not all noises created equally: Diffusion noise selection and optimization. arXiv preprint arXiv:2407.14041, 2024.
Zipeng Qi, Guoxi Huang, Chenyang Liu, and Fei Ye. Layered rendering diffusion model for con- trollable zero-shot image synthesis. In European Conference Computer Vision, pp- 426-443. Springer, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10699 2022.
Amirmojtaba Sabour, Sanja Fidler, and Karsten Kreis. Align your steps: Optimizing sampling schedules in difffusion models. arXiv preprint 1rX1::2404.14507, 2024.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, (136479-36494, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik, Generating images of rare concepts using pre-trained diffusion models. 2024.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation. arXiv preprint 1xXi::2311.10442 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294 2022.
Maximilian Seitzer. pytorch-fit fid:: FID Score for PoTchh. httta:///sitaub..commm....t pytorch- fid. fid August 2020. Version 0.3.0.
Shitong Shao, Zikai Zhou, Lichen Bai, Haoyi Xiong, and Zeke Xie. Iv-mixed sampler: Leveraging image diffusion models for enhanced video synthesis. arXiv preprint arXiv:2410.04171, 2024a.
Shitong Shao, Zikai Zhou, Tian Ye, Lichen Bai, Zhiqiang Xu, and Zeke Xie. Bag design choices for inference of high-resolution masked generative transformer. arXiv preprint arXiv:2411.10781. 2024b.
Dazhong Shen, Guanglu Song, Zeyue Xue, Fu- Yun Wang, and Yu Liu. Rethinking the spatial inconsistency in classifier-free diffusion guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9370-993 9379, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint htt::20000025002. 2020.
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Chris- tian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. arXiv preprint 24X1:24403.2000 2024.
13

---

Published as a conference paper at ICLR 2025
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Viston and Puttern Recognition, pp. 9228-8238, 2024.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7623-7633, 2023a.
Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Freeinit: Bridging initial- ization gap in video diffusion models. arXiv preprint arXiv:23/2.07537, 2023b.
Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Freeinit: Bridging initializa- tion gap in video diffusion models. In European Conference on Computer Vision, pp. 378-394. Springer, 2025.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences text-to- image synthesis. arXiv preprint (eXiv:23060093// 2023c.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024a.
Katherine Xu, Lingzhi Zhang, and Jianbo Shi. Good seed makes good crop: Discovering secret seeds in text-to-image diffusion models. arXiv preprint N.NV:22405.18222 2024b.
Lingxiao Yang, Shutong Ding, Yifan Cai, Jingyi Yu, Jingya Wang, and Ye Shi. Guidance with spherical gaussian constraint for conditional diffusion. In Forty-first International Conference on Machine Learning.
Xiaofeng Yang, Cheng Chen, Xulei Yang, Fayao Liu, and Guosheng Lin. Text-to-image rectified flow as plug-and-play priors. arXiv preprint arXiv:2406.03299 2024.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. Transactions on Machine Learning Research.
Shenghai Yuan, Jinfa Huang, Yongqi Xu, Yaoyang Liu, Shaofeng Zhang, Yujun Shi, Ruijie Zhu, Xinhua Cheng, Jiebo Luo, and Li Yuan. Chronomagic- A benchmark for metamorphic evaluation of text-to-time-lapse video generation. arXiv preprint arXiv:2406. 18522, 2024.
Zikai Zhou, Shitong Shao, Lichen Bai, Zhiqiang Xu, Bo Han, and Zeke Xie. Golden noise for diffusion models: A learning framework. arXiv preprint 2. 2024.
A EXPERIMENTAL DETAILS
In this section, we introduce the details of the metrics and benchmarks used in the experiments.
A.1 DATASETS
Pick-a-Pic. The Pick-a-Pic dataset (Kirstain et al., 2023) was generated by logging user inter- actions with the Pick-a-Pic web application for text-to-image generation. Each entry includes prompt, two generated images, and a label indicating the preferred image or a tie if neither is signif- icantly favored. Here we use only the first 100 prompts as the test set, which is sufficient to reflect the model's capabilities.
Drawbench. DrawBench is comprehensive and challenging benchmark for text-to-image mod- els, introduced by the Imagen research team (Saharia et al.) (022). contains 11 categories, includ- ing aspects such color, counting, and text, with approximately 200 text prompts.
14

---

Published as a conference paper at ICLR 2025
GenEval. Geneval (Ghosh et al. 2024 is object-focused framework designed evaluate com- positional properties of images, including object co-occurrence, position, count, and color. It incor- porates 553 prompts, achieving an 83% agreement with human judgments regarding the correctness of the generated i
PartiPrompts. PartiPrompts (Yu et al.) is collection of over 1,600 diverse prompts in English, designed to assess the capabilities models across different categories and challenges. The prompts cover wide range of topics and styles, helping evaluate the strengths and weaknesses of models in areas like language understanding, creativity, coherence. Here we randomly select 100 prompts from Part for evaluation.
A.2 METRICS
AES. Aesthetic score (AES) (Schuhmann et al. 2022) refers to mechanism for evaluating the vi- sual quality generated images, which assigns quantitative score based on attributes like contrast, composition, color, and detail, reflecting alignment with human aesthetic standards.
PickScore. Kirstain et al. (0023) developed Pick-a-Pic, a large open dataset consisting of text- to-image prompts and real user preferences for generated images. They then utilized this dataset to train a CLIP-based scoring function, PickScore, for the task predicting human preferences.
ImageReward.
(2024a) developed ImageReward, the first general-purpose text-to-image human preference reward model. which is trained based on systematic annotation pipeline, including rating and ranking and has collected 137,000 expert comparisons to date.
Xu et al.
HPS v2.
Wu al. (202c)) first introduced the Human Preference Dataset (HPD v2), a large- scale dataset comprising 798,090 human preference choices 433,760 pairs of images. By fine- tuning CLIP using HPD v2, they developed the Human Preference Score (HPS v2), a scoring model that more accurately predicts human preferences for generated images.
A.3 BASELINES
Semantic-aware CFG (Sen et al. 2024). adaptively adjust the CFG scales across different se- mantic regions to mitigate the undesired effects caused by guidance.
Riffusion-DPO (Wallace et al. rated high quality images and captions to improve visual appeal and text alignment.
2024), finetune pretrained Diffusion model using carefully cu-
AYS-Sampling (Sbour et al. for the dataset, model, and sampler enhance image quality.
2024 a strategy for optimizing sampler timesteps, which accounts
Semantic-Aware CFG Shen et al. 2024). a strategy dividing the latent into independent se- mantic regions each denoising step and adaptively adjusting their guidance, thereby unifying the effects across regions.
Smoothed Energy Guidance (Hong 2024), which employs energy landscape perspective and intermediate self-attention maps to achieve higher quality samples.
CFG++ CChng et 2024), which optimizes the classifier-fres guidance mechanism from the
perspective of manifold constraints. It only replaces the conditional latent with the unconditional latent during the classifier free guidance mechanism, effectively addressing the oversaturation issue caused by high CFG values.
"To ensure consistency with other experiments, used denoising guidance scale of 5.5, differing from the default 9.0 in GenEval.
15

---

Published as a conference paper at ICLR 2025
B RELATED WORKS
In this section, we discuss existing work related to Z-Sampling.
Semantic Information in Latent Space Recent works have shown that the prior information present in the noise latent can significantly impact the quality of image generation (Xu et al. 2024bb]
Mao al. 2023a Samuell et al. 2024; Zhou al. 2024 Liu al. (024a). For example, Mao al. (023b) found certain regions in random latents can induce objects representing specific concepts. And Zhou et al. (2024) propose Golden Noise by leveraging a semantic accumulation approach. And YPo-Yuan et al. (202) found slight perturbations can lead to significant changes in the diffusion model's 's generated results. And injecting semantic information (e.g., low-frequency wavelengths) into Gaussian noise can enhance image quality, particularly improving alignment performance (Wu jet al., 2023c Guttenberg 2023] [in et al. 2024a Qi et al., 2024). IRFDS (Yang al.) 2024) utilizes pretrained rectified flow model provide prior, optimizing the initial latent for image editing task. Building on these studies, we investigate semantic information from the guidance perspective, implicitly integrating it into the generation process without requiring explicit reference data.
Sampling Strategies of Diffusion Model To improve the sampling process, lug (2022) proposed Resampling that involves adding random noise and performing multiple back-and-fort samples at each timestep. Subsequent studies adopted this paradigm for tasks such video generation (Wul
it al., (023b) and universal classifier guidance (Bansal et al. (023). IRFDS (Yang et al. 2024)
utilizes pretrained rectifying flow model to provide a prior, optimizing the initial latent bet- ter image editing. However, they overlooked the importance of inverted latent and simply applied random noise, which does not effectively enhance prompt adherence. Tune-a- Video, to ensure structural consistency, Wu et al. (2023a) incorporate the denoising-ing -nvversonn paradigm as subcom- ponent. However, their end-to-end approach is not optimal and overlooks the importance of the guidance gap. To reduce spatial inconsistency in different latent regions under the same guidance scale, Shen et al. (2024) developed adaptive guidance based on semantic segmentation. relies on attention-level changes, limiting adaptability to other algorithms, and its robustness is influenced by semantic segmentation effectiveness. Constraint-based approaches aim to improve sampling, for example, (Cenn al. (2024) substitutes conditional noise with unconditional noise enhance gen- cration quality from an image manifold perspective, though improvements are minimal. Yang et al. applies spherical gaussian constraint during guidance, but it requires a reference data, limiting its applicability. Finally, we note that Z-Sampling can be effectively transferred to other generative paradigms, such as Masked Generative Models (Shao et al., (024b) and IV-Mixed Sampler (Shao) et al. 2024a).
C MOTIVATION AND OBSERVATION
C.1 LATENTS WITH SEMANTIC INFORMATION
In Figure I we present additional cases illustrating that random latents encode relevant semantic information. For instance, for prompts related to the concept "Jeep Cars", the latent corresponding to seed 20 achieves the highest performance, with PickScore of 23.4784, whereas latents from other seeds fail exceed PickScore of 23.
C.2 INVERSION MAKES GOOD LATENTS
In this section, we show that the inverted latent inherently carries semantic information related to the conditional prompt These extra semantic information gain leads to superior generation outcomes.
First, we choose images of "cats" and "spiders" as depicted in Figure [2 Employing the DDIM inversion algorithm with guidance scale set O, we obtatin latent and latent We hypoth- esize that latent,ne encapsulates semantic information associated with "cat" whereas latent, inherently relates more closely to "spiders".
Next, we use these two latents to generate images conditioned on text prompts "cats" and "spiders" respectively, as illustrated in Figure 13 We observe that latent performs better when condi-
16

---

Published as a conference paper at ICLR 2025
starrdtioonl roill
Prinpt vslatd "Ropy
Pronge rdated "Spanw
Phoppi "Living
rclite to "Human Scultture
Seed.
23,7844
21.4400
21.6255
20JII
22,0294
11,9588
19,8488
Seed:: 47
19,2551
Seed.
21.9288
20.4304
Figure 11: Latents with relevant semantic information about specific concept can generate images more effectively from prompts related to that concept. Each row shows the results of the same latent across different prompts, while each column shows results from different latents under the same prompts. For each cell, we compute the PickScore. For example, the latent related seed 20 achieves an PickScore of 23.4784 when generating images related to "Jeep Cars".
Prompe
Prozzee: "a spide?"
late
laters...
Figure 12: Given two natural images and their corresponding prompts, we perform DDIM inversion reverse them and obtain the corresponding initial noise latents.
latest....
latent..
Diflusion
Difusion
saker
scone: 21,9907
score: 21.1783
setre:
seoee
Figure 13: Generate images related to "cat" and "spider" using two latents respectively, and calculate the PickScore.
tioned on text related to "cats" while latent performs better when conditioned on text related to "spiders". This phenomenon empirically validates our hypothesis that inverted latent does matter.
17

---

Published as a conference paper at ICLR 2025
D SUPPLEMENTARY EXPERIMENTAL RESULTS
In this section, we present more quantitative and qualitative results of Z-Sampling.
D.1 SUPPLEMENTARY QUANTITATIVE RESULTS
Results of Z-Sampling other benchmarks In Table 6 we evaluate 100 randomly selected prompts from PartiPrompts using the SDXL model, with Z-Sampling demonstrating the higher per- formance. Additionally, we also compare classical metrics such as FID (Seitzer) 2020), IS (Sali-) mans et al.) 2016). and clip-score (Radford et al. (2221) MS-COCO 2014 (Lin et al. 2014). Due to numerous evaluation prompts (30K), we employ the distilled model, preamshhapr--1-v2tum with 4 denoising steps, showing the higher generation quality in Table 7 We also report additional comparative results on Geneval in Table [ including Resampling and Iiffusiin-DPO. showcasing Z-Sampling's superiority in average scores.
Table 6: The quantitative results of Z-Sampling on Table 7: PartiPrompts. Model: SDXL.
The quantitative results of Z-
Sampling on MS-COCO 2014.
Model:
DreanShape-xt----uubo.
Method
HPS v2 f
AES
PickScore f
t
Standard
29.34
5.81
22.27
72.53
Method
IS-30K
FID-30K
Clip-Score f
Resampling
30.21
5.78
92.34
Standard
34.0745
24.1420
0.3267
Z-Sampling(ours)
31.00
5.85
22.43
97.32
7-Smpliing(ours
34.4173
23.4958
0.3288
Table 8: The additional quantitative results of Z-Sampling on GenEval. Model: SDXL
Method
Single object f
Two object
Counting
Colors
Position
Color attribution
Overall
Standard
97.50%
69.70%
33.75%
86.71%
10.00%
18.00%
52.52%
Diffusion-DPO
100.00%
80.81%
45.00%
38.30%
10.00%
31.00%
59.18%
DP0+2-Samplinggour)
100.00%
82.83 %
46.25%
89.36 %
10.00%
29.00%
59.57%
Results of Z-Sampling in other baselines and tasks We also compare Z-Sampling with other methods that improve the effect of guidance. Specifically, Hong et (2022) proposed SAG, which employs blur guidance and intermediate self-attention maps achieve higher quality samples. Fur- thermore, SEG (Hong) 2024) further optimized SAG from the energy landscape perspective. Here we report the comparison results with SEG in Table g Additionally, We have also compared Z- Sampling with CFG++ (Chung et al., 2024). which optimizes the classificr-free guidance mecha- nism from the perspective of manifold constraints. since it restricts the cfg scale to the range from 0.0 to 1.0, while the classic Z-Sampling is larger, a fair comparison is not possible. Given this, we use w = 0.5 in CFG++, corresponding to a cfg scale of 5.5 in Z-Sampling.
Table 9: The quantitative results of Z-Sampling and SEG. Model: SDXL.
Pick-a-Pic
DrawBench
Method
HPS v2t
AES
PickScoref
IRt
HPS v2f
AES1
PickScoref
IRt
Standard
29.89
6.09
21.63
58.65
28.81
5.55
22.31
60.75
SEG
30.53
6.12
21.42
61.57
29.60
5.66
22.15
60.42
Z-Sampling(ours)
31.28
6.13
21.85
78.22
30.50
5.67
22.46
79.97
Finally, general method, we test Z-Sampling's 's performance the video generation task. We choose DnimateDiff (Guo et al., 2023 as the baseline model and test it on Chronomagic-Bene Bench- 150 (Yuan et al. 2024). and set ?1 7.5 and 7/2 = 0 in Z-Sampling. With the results shown in Table 11 we note that Z-Sampling outperforms both AnimateDiff and another train-free sampling method FreeInit FreeInit (Wu al. 2025) in UMT-FVD (Liu et al. 2024b), UMT-SCORE (Li al. 2023
GPT40-MTSCORE (Achiam et al. 2023).
18

---

Published as a conference paper at ICLR 2025
Table 10: The quantitative results of Z-Sampling and CFG++ Model: SDXL. worth noting that in the official implementation of CFG++, the VAE encoder uses madebyollin/skkl-vaeffff fix checkpoint. For fair comparison, we follow this setting, so the results reported for SDXL and Z-Sampling are slightly different from the previous results.
Pick-a-Pic
DrawBench
Method
HPS v2f
AES1
PickScore"
IR
HPS v2f
AEST
PickScoref
IR
Standard
30.04
6.11
21.80
60.07
28.85
5.62
22.42
67.61
CFG++
30.28
6.09
21.83
67.30
28.65
5.62
22.38
62.66
Z-Sampling(ours)
31.24
6.12
21.85
78.55
30.35
5.66
22.44
79.11
Table 12:
The quantitative results of Z- Sampling under different backtracking stepsize
Table 11: The quantitative results of Z-Sampling on Chronomatic-BBnchh-155 Model: Animatel)- iff.
k. Model: SDXI.
HPS v2
AES
PickScore
1
UMT-FVD J
GPPH+-MTSCORE
D (SDXL)
29.89
6.09
21.64
58.65
Suadardd
275.18
2.82
2,83
31.28
6,13
21885
79.22
FREFINIT
268 31
282
2.50
31.11
6.08
21.72
84.53
Z-Sampliggooss)
243.26
2.97
2.8
3
30.75
6.09
21.48
78.54
4
30.59
6.09
21.34
78.60
Multiple steps of denoising and inversion operation in Z-Sampling We have explored the one-step scenario, i.e, It - It-1 - Fy. Here, we extend multiple steps scenario, i.e., It + Σt-k + Σt. As shown in Table [2] the best performance is achieved when k=1. As k increases, the performance of 7-Sampling deteriorates, which aligns with the Theorem [ and Theo- rem 2 where increasing k gradually brings the step-by-step approach closer to end-to-end, thereby increasing the error term T2. Specifically, when k=T-1 and the zigzag operation is only performed on the initial latent, it corresponds the scenario in Table 17
D.2 SUPPLEMENTARY QUALITATIVE RESULTS
In Figure 14 we note Z-Sampling can better recognize the stylistic descriptions prompts. For example, can generate "Mario characters" that more realistic and lifelike.
Standard Sampling
Z-Sampling Ours))
photoggaphy of super mario, 8k Ultm HD.
sculpture piniff,
moga Lego space station inside a kcid's bedroom.
McDonald Church.
January in Now England, stands in shadow of maple
tripping over a eat.
Figure 14: Qualitative comparison in terms of style.
In Figure 15 we note Z-Sampling accurately interprets object positional relationships, e.g., "under- neath'. 'on top of", 'on the right of", etc.
In Figure [6 Z-Sampling enhances the binding of color attributes, aligning images more closely with prompts and improving quality.
19

---

Published as a conference paper at ICLR 2025
Stundard Sampling
Z-Samplingt Ours)
underneath broccoli.
right lemnis nikket.
bicycle top bost.
lapiop top leddly bear.
lire hydrunt.
chair.
Figure 15: Qualitative comparison in terms of position.
Standard Sampling
i
Z-Sampling(
photo of yellow sports ball and a green bost
A photo of orange snowboard and green
photo yellow bird and black motoreycle
photo white and purple skaticbourd
A photo of green souch and an orange umbrella
A photo purple dog and a black dining table
Figure 16: Qualitative comparison in terms of color.
In Figure 17 we note Z-Sampling demonstrates enhanced capability in understanding quantitative relationships, effectively addressing the persistent challenge in diffusion models. For example, can effectively understand and generate images such as 'three suitcases', 'four buses', and two beds'.
Standard Sampling
Z-Sompling Ours)
A photo three suitcases
photo of four buses
photo of three kites
A photo of four fire hydrants
photo of three sinks
A photo of ruoo heds
Figure 17: qualitative comparison in terms of counting.
In Figure 18 we find that Z-Sampling aids in generating Multi-object composite (c.g., mouse and bowl) or counterfactual (e.g., an elephant in the sea) images, manifested in its enhanced 'co- occurrence" capability.
20

---

Published as a conference paper at ICLR 2025
Standard Sampling
Zigzag Diffusion Sumping
3
elephant panda making the sea.
monkey.
green clock.
gray backpacks.
purple bowl.
Figure 18: Qualitative comparison in terms of object co-occurrence.
21

---

Published as a conference paper at ICLR 2025
D.3 WINNING RATES COMPARISON
Here, present comparative analysis of winning rates under various settings, such as different models and denoising steps. The blue bars represent 7-Sampling (ours), while the orange bars represent the standard sampling method. Winning rates of method exceeds 50% in all metrics. Especially HPS v2, which is much better than standard method.
Ricka-Piic
DrawBench
ABS Nok: Score
79.5
AES
HPS
HP5
390
15
Pr
Praference[5]
Figure 19: Comparison of Winning Rates with 10 Denoising Steps in the SDXL.
DrawBench
Rick-a-Pic
Pik Score
AES
AES
HPS V2
HOS
81
100
Preferen:ce/95
Preference|99
Figure 20: Comparison of Winning Rates with 50 Denoising Steps in the SDXL.
Pick-a-Pic
DrawBench
5S
Pick Score
67.5
AES
HPS +2
Proforoncol/5
Proforence15)
Figure 21: Comparison of Winning Rates with 50 Denoising Steps in the SD 2.1.
22

---

Published as a conference paper at ICLR 2025
Rick-a-Pic
Pick-a-Pic
50
Rok Scor
Rick
ABS
AES
NPC 12
Profarenco/[S]
Preference(5)
Figure 22: Comparison of Winning Rates with 10 Denoising Steps in the Hunyuan- DiT.
D.4 PERFORMANCE OF Z-SAMPLING UNDER HIGH CFG SCALE
We also report the performance of Z-Sampling under different intensities of classifier free guidance 71 during denoising process.
as the base model. As shown in Table 13 the standard sampling performs best at 3.5, which is also the official recommended guidance sclae. When 2 3.5, the standard sampling begins to exhibit issues such as oversaturation and artifacts.
We use DreamShaper-xltturoor
However, Z-Sampling consistently yields positive gains, indicating that our method can still work effectively under high guidance scales. And we present the winning rate of Z-Sampling over Stan- dard sampling HPS v2 across different guidance sclae 7/1 in Figure 23 further validating this point.
Table 13: Performance of Z-Sampling under different guidance 7/1 Model: DreamShaper-xl xI turbor
v2. We note that the official recommended guidance scale = 3.5. When >> 3.5, the quality of standard sampling gradually declines, while 7--Smpling still shows improvement on this basis.
Method
HPS v2
AES
PickScore
IR f
Winning Ratef
Standard Sampling
1.5
28.51
5.83
21.37
43.25
Z-Sampling
1.5
29.51
6.02
21.66
55,89
73%
Standard Sampling
3.5
30.04
5.94
21.59
66.18
Z-Sampling
32.38
6.15
22.11
90.87
Standard Sampling
5.5
29.96
5.97
21.37
64.46
Z-Sampling
5.5
31.42
6.05
21.83
76.00
85%
Standard Sampling
7.5
29.10
5.88
21.02
60.26
7-Sampling
7.5
30.90
5.96
21.59
74.18
86%
Standard Sampling
27.98
5.76
20.59
41.70
7-Sampling
9.5
29.95
5,88
21.28
63.40
92%
Standard Sampling
11.5
26.93
5.60
20,30
31.45
Z- Z-Sampling
11.5
28.97
5.77
20.97
55.69
91%
Generally, classifier-free guidance serves as a mechanism for semantic control, balancing image quality and prompt adherence, with excessive guidance scale causing deviations and artifacts. Z- Sampling, as a similar semantic enhanced mechanism, employs an iterative approach (unlike the vanilla CFG mechanism, which directly alters the latent distribution) to more effectively explore this balance. And we presents some visual cases in Figure [4]] showcasing Z-Sampling 's capability maintain image quality even under high guidance scale.
23

---

Published as a conference paper at ICLR 2025
90.0%
87.5%
Winning Rate
85.0%
82.5%
80.0%
77.5%
75.0%
72.5%
1.5
2.5
3.5
4.5
5.5
6.5
7.5
8.5
9.5
10.5
11.5
Guidance Scale V1
Figure 23: Comparison of Winning Rates under different guidance scale n1. Model: DreamShaper- xl-turbo-v2. Horizontal axis: guidance scales 71.. Vertical axis: Z-Sampling vs Standard Sampling winning rates on Pick-a-Pic.
Yi = 3.5
= 7.5
Y1 = 9.5
= 3.5
=.7775 7.5
Y: 9.5
Standard Sampling
Sampling
Z-Sampling Ohrrs)
ZzSmmp(ing(Omm)
Portmic Close Up. Intricate Realistic Render, Portnit of year old extremely handsome boy.
Black and white 1905 year funuristic poetrait of professional photographer with camera hand a desert sadly covered by mushroom..
Y, =
Y1 =
Y; = 9.5
3.5
Y: 9.5
Standrd Sampling
Sampling
Z-Samplingl Ougr)
Z-Sumpling(Omm)
ghoto,
Caucesian woman, dressed dress, model face.
cat in space suit wall on the moon
Qualitative comparison under high guidance scale. When 71 = 3.5 (the official recom- mended guidance scale), both Z-Sampling and Standard exhibit artifacts or degradation in image quality. As 71 increases, standard sampling exhibits artifacts and oversaturation, while Z-Sampling less affected.
Figure 24:
24

---

Published as a conference paper at ICLR 2025
Yi y1 15.5
7: 25.5
= 5.5
15.5
25.5
Sandard Sumpling
Samplingg
HEATSONE REALL
HAT SUNE REAL
2-SmmpingOOrr))
Z-SimpiinfOOmm
A
HATTUAA REMA
A blue coloeed dog.
sign that says "Hatsune Miku es real"
5.5
Y, 15.5
V, 25.5
Y1 =
Yi
Y1
Standan Samplnng
Stundrrd Sumpling
Z-SamplingtOees)
Z-Sampling
official postrait itarian president aller in 1960, the style of pan am adverisements, looking up,
An image of ginger inside tea blde runner.
Figure 25: Qualitative comparison under high guidance scales. Standard sampling suffers from supersaturation more significantly than Z-Sampling under the high guidance scales. Model:SDXL.
D.S ADDITIONAL EXPERIMENTS VARIOUS GUIDANCE SCALES
We report more visual cases in Figure 25 under different guidance scales 7i- It can be observed that the guidance scale increases, the phenomenon of artifacts and oversaturation for standard sampling become more pronounced, while Z-Sampling effectively mitigates these issues. The similar observation also holds in Figure 24 with DreamShaper.
showcasing the performance Z-Sampling SDXL
To further investigate the performance improvement with various high CFG scales, present the conclusive quantitative experimental results of SD 2.1, SDXL, and DreamShaper together in the additional Table 14 We searched the best guidance scales for each model in terms of HPS v2 and present the results. For SDXL/SD2.1, 1, the seached guidance range was set from 3.5 to 25.5, and for DreamShaper-a xl-turbo-v2, do not fine-tune the guidance scale hyperparameter.
from 1.5 to 1.5. Note that existing relevant studies commonly
The conclusive quantitative results demonstrate that Z-Sampling can significant improve the best performance of all three diffusion models with various choices of the guidance scales. Moreover, the results indicate that the distilled DreamShaper with Z-Sampling can even outperform SDXL, while DreamShaper with standard sampling cannot match SDXL.
E ANALYSIS OF THE APPROXIMATION ERROR TERM
In this section, undertake in-depth analysis of the approximation error term within Equation I0, first demonstrate Z-Sampling's results under the uncertainty scheduler. Then, we analyze how this approximation error affects the performance of Z-Sampling.
25

---

Published as a conference paper at ICLR 2025
Table 14:
Quantitative comparison of Standard Sampling and Z-Sampling with the best grid searched guidance scale on Pick-a-Pic Dataset. Z-Sampling consistently outperforms across SDXL, SD2.1, and DreamShaper-xle -lurturox indicating significantly better performance, even with grid searched results. Note that previous studies commonly use the defaulted guidance scale instead of fine-tuning for diffusion models. This further demonstrated that the advantage of Z-Sampling is robust to various settings.
Model
Method
HPS v2 f
AES t
PickScore t
IR f
Winning Rate 1
Standard
26.86
5.70
20.39
23.87
SD-2.1
Z-Sampling(ours)
27.29
5.72
20.38
28.85
62%
Standard
31.00
6.10
21.72
79.17
SDXL
Z-Sampling(ours)
31.28
6.13
21.85
79.22
61%
DreamShaper
Standard
30.16
5,99
21.53
68.99
-xl-v2-turbo
Z-Sampling(ours)
32.38
6.15
22.10
90.87
86%
E. UNCERTAINTY AND STOCHASTIC SAMPLERS
To assess the impact of different inversion algorithms on generation quality, we test various inversion methods. Specifically, we use SDXL--Turbo (4 steps) (Sauer et al. 2023) an adversarial distilla- tion diffusion model. Notably, SDXL-Turbo's default sampler is an ancestral Euler sampler, which introduces random noise at each denoising step, leading to highly inaccurate inversion.
Table 15: With stochastic samplers (e.g., Euler(a)), inversion inaccuracies reduce Z- Z-Sampling's effectiveness. In contrast, deterministic samplers (e.g., Euler) yield better results with Z-Sampling.
Method
HPS v2 1
AES f
PickScore
IR 1
Standard Samplingealetaa)
31.23
5.95
21.63
82.24
Z-Samplingraaetta
30.78
5.95
21.65
80.60
Standard Samplingeuter
27.05
5.60
20.36
41.44
7-Samplingsuler
28.57
5,85
20.96
39.54
From Table [5] it can be seen that when using the Euler ancestral sampler, e.g., Euler(a), which in- troduces randomness in the denoising process, most metrics show decline. This is because Euler(a) leads to inaccuracies in the inversion process, causing the approximation error term in equation 23 increase significantly. As result, Z-Sampling diverges from the data manifold, leading to reduced effectiveness.
However, when using deterministic Fuler samplers, although the overall performance does not match that the Fuler(a) sempler--ckkowlleee introduce blurring and related issues-22 Z-seepii still demonstrates performance improvements over the corresponding baseline. For example, the PickScore increase from 20.3643 to 20.9639 This highlights the importance of the inversion algorithm and presents opportunities for improving Z- Sampliig under stochastic samplers
that other sampling methods on the turbo model may
Corresponding to equation I0 deterministic sampler implies that the inversion process is impre- cise, leading to an increase in (t). We note that end-to-end inversion amplifies the approximation error MMkady al.] 2023), risking latents deviating from the data manifold. Z-Sampling. on the other hand, truncates the error at each step, reducing making semantic injection more efficient.
E.2 THE INCREASE APPROXIMATION ERROR RESULTS IN NEGATIVE GAINS
To focus solely on the approximation error 72 in Equation 10
we need to eliminate the influence of the semantic term 71- So set 7 = 72 5.5, which means o,, 0 and 7 0. Then Equation 10
26

---

Published as a conference paper at ICLR 2025
can be transformed
8 Sampling
Σ at 0,)²
Σ
-
(12)
(tp):pppmxx enror term
Similarly, Equation 9 can be transformed
Sendlend (IT TT)² 2 or
h4 (cg(() (
(13)
appp:pp error term
Since the semantic term тi no longer contributes, only the effect of 72 remains, as shown in Table I6 and Figure 26 both the end-to-end and step-by-step approaches result in negative gains. Notably, the approximation error introduced by the end-to-end method that of the step-by-step method, significantly degrading the image quality. This demonstrates that:
orders of magnitude higher than
An increase in the error term degrades the sampling effect.
The step-by-step approach helps reduce the error term 72, mitigating this negative gain.
SDXL
Step-by-Step
End-to-End
curious exploring
a haunted mansion
IE
A sign that says 'Hatsune Milk es real!
Figure 26: When the semantic term тi removed (c.g., Ti 0), the presence of only the error term T2 degrades the quality of generation results, and this negative gain effect is more pronounced in the end-to-end method.
Additionally, we test the performance of end-to-end and step-by-step methods in the presence of the semantic term Tı, as shown in Table 17 Since in this case, Ti and T2 are mixed together, so we only report the PickScore exact Approx Error. It can be observed that with the presence of the semantic term, both methods yield positive gains, and the step-by-step method performs better.
reflect the quality of the generated results,
unable report the
Table 17: The results Pick-a-Pick, in- cluding semantic term T1- Model: SDXL.
Table 16:
The results on Pick-a-Pick, excluding se- mantic term тi. Model: SDXL.
Method 8,
PickScore 1
Approx Error
Method
8y
PickScore
SDXL
21.63
0
SDXL
21.63
End-to-Fnd
0
18.82
160.3313
End-to-End
5.5
21.65
Step-by-Step
0
21.52
0.9919
Step-by-Step
5.5
21.85
27

---

artificially simulating and Published as a conference paper at ICLR 2025
E.3 ARTIFICIALLY INTRODUCING GAUSSIAN ERROR
Specifically, to further illustrate that the approximation error 72 leads to negative gains, we consider adding an additional random Gaussian term errorgs Equation 12 controlling the inversion approximation error as
T
hn - (() +s* norm(c((Z1)) error crror (14)
Σ 1=1
21 x2)²
O7-Sampling
norm(error...
t=1
(2))):approx error term
where s used control the magnitude of the error. As seen in Table 18 the larger the value of the worse the performance of 7-Sampling, further illustrating that reducing the error term introduced by inversion is a direction that warrants attention.
Table 18: As the coefficient of the Gaussian error term increases, the quality of generation decreases.
S
HPS v2 +
AES +
PickScore 1
IR 1
0
29.95
6.1889
21.53
51.12
0.5
29.93
6.15
21.51
45.53
1.0
28.12
6.01
20.78
28.74
F PROOFS
In this section, derive the relationship between the end-to-end semantic injection approach and Z- Sampling, proving Z-Sampling's s superiority. Then we formalize how Z-Sampling injects semantics via the guidance gap.
Proof F.1 (Theorem
Given inference timesteps T. T'T.froo from equation we can obtain the inverted
latent अ7 as
QT ST
VTT
(Vo--1- V ar-1
1
e (ĀT-1)-
(15)
aT-1
For the sake of convenience, sel
OCT
1
MT
nT VT
Vor=
(16)
QT-1
aT-1
So, equation
15 could also be written
PT my To T (TT-1)-
(17)
Through iterative and combinatorial processes in equation
ØT could be expressed
äT mrčr- 1 ( (ĀT-1)
-- T- -1 + NTEg (ĀT-1)
T- (iT-a) + mTNT-16T 1 T-' (ZT-2) + =n (iT-1)
myp + mT
T
T
T
m, +
Σ nl
II
m (Tr-1).
(18)
1=0
1=1
k=++1
28

---

Published as a conference paper at ICLR 2025
Similarly, based on equation 7 aand equation equivalent form of 2T as
can perform iterative derivations to obtain the
T
T
JT
m330
+
Th
II
(x))
(19)
==
k=t+1
We can determine the difference between and ir, representing the gainfrom end-to-end semanlic injection as
BendZend = (IT - ITT
T
T
my (ro
000) +
Σ
f
II my (cb(ar) ca (6(0e-1))
t=l
k=(+1
T
(20)
1
1 - -1 (eb(ere) <b((tr-))
v
V
Ct
0t-1
1=1
QT
-1
(cg(urs) c (b((-=1))
0t
04-1
where we set h1 = Jai and further refine equation 20 to yield the semantic injection term Tι and the approximation error term T2 as
BendZend OT
Σ
h
(eglate) - <b((Te)
1=1
2
T
OT
h4
(
+ +cb(Te) - - (
(21)
TL:semantic information goin term
Ta:apprex evrur term
Proof F.2 (Theorem)
Unlike end-to-end approaches, in Z- Sampling. we focus solely on the local Ft ". Substituting equation into equation 4 yields 3..
cycle of "It It-1
a1-1)ai (x))))
2, = 24 - VI (x)) +
at-1
1
E (Trz-)
V Okt
04-1
(1 -1-ay (1-an-1)on
Et + v1 - oie (ch()t-1)= - cb(e)) +
(cg(zz) cb(āx-1))
a411
(1 - (1-ar-))ar
+
v1 V1-
(cb(are) c(((-=1))
a-11
1 1
Σt Va
(c)(xz) c6(()=1))
(22)
Qt-1
29

---

Published as a conference paper at ICLR 2025
T-1
Bendzend (Xp x2)2
a
T-I
8y Samptingg 2n (x2 22)²
(b)
Figure 27: The EandttoEEnd injection risks semantic cancellation across stages, leading to suboptimal results. In contrast, Z-Sampling captures and injects semantic information at each step in timely manner along the sampling path, resulting in stronger injection effect.
The latent difference of Z-Sampling is accumulated as
T
8z-Sampling
(xt
zz)²
T
Σ
a (<b(xz) cb(()=1))
[=1
Σ
anh
( - (
(23)
Ts:semantic information gain term
T2 :approsimation error term
In Figure 27
we visually represent the effect of equation 21 and equation 23
Z-Sampling clearly
injects semantic information each step in timely manner, leading to more pronounced effect and deeper level of semantic injection.
We note in Equation 24 that ci (#t,) actually represents the denoising result of latent 2; under low guidance 72, written this way for consistency with Equation [3] Therefore, the only difference be- tween e (Txx and e is the guidance scale: (2x) the guidance scale of 7r while ej (ā+) uses the guidance scale of 72. The latent input to the denoising network is the same for both zt.
Proof F.3 (Theorem) 3 Excluding the approximation error introduced by inversion algorithm,
can rewrite equation 23 as
87 Sampling Σ anh (c)(2)) ²
(24)
Although the step-by-step approach results in x, and 7, being the same at each timestep t, from equation 5 we note that (x.) and (fx) obtained under guidance scaley " and "22 respec-
30

---

Published as a conference paper at ICLR 2025
rively. Thus, the effect of Z-Sampling is further equivalent as
8z Sampling
oghy ((7 72) (0 (11 -2)) us (3re, 20, 1))²
ag ((71 - 72) (uo( e, uo(oi, (, 20,)))²
T Σ an (ô, (wo( C, - us (Zt, Ø,()))²
(25)
Here, 8, represents the guidance gap between denoising and inversion, i.e., V/1 72.
From equation 25 we note that the effectiveness of Z-Sampling primarily depends on:
1. The guidance gap 6y, which we can control to regulate the magnitude and intensity of the optimization.
2. The difference between the conditional branch (Ie, c, t) and unconditional branch u0(2,, Ø, t), which is determined by the prompt and the model parameters 6.
As mentioned in the end of Proof in the absence of inversion approximate errors, the only differ- ence between c (2x)) and (fx)) in Equation 24 is they the different guidance scale. Therefore, even when = 0, our focus remains on the invariant, which is the difference between the network outputs of the conditional and unconditional branches uo(2;, c, t) us (0 Ø, t).
G THE END-TO-END SEMANTIC INJECTION ALGORITHM
In this section, we show how inject semantic information end-to-end as described in Section 3.3
Algorithm End-to-End Semantic Injection
1: Input: Denoising Process: 4, Inversion Process: V, text prompt: c, denoising guidance: YY, inversion
guidance: 72, inference steps: T
Output: Clean image Ig
3: Sample Gaussian noise ør
4: = @(rl/
#see equation
5: ê (zulc, 72)
#see equation #sce equation
6: dt = ((rlc, (a)
7: return 20
31


Reviews:
[{"review_id": "BAYvsWhSxV", "reviewer": "Reviewer_uffk", "summary": "This paper introduces Z-Sampling, a novel sampling method for diffusion models that improves image generation quality and prompt alignment. The paper investigates that if the latent variable contains the sementic information, the sampling result would be enhanced. The key insight is that the guidance gap between denoising and inversion processes can capture semantic information. Rather than doing end-to-end semantic injection, Z-Sampling accumulates semantic information step-by-step through zigzag sampling. At each step, the denoising process is followed by inversion. The authors provide a theoretical analysis showing why this approach is more effective than end-to-end injection. The paper demonstrates empirical improvements across multiple diffusion models, datasets, and evaluation metrics. The method is particularly effective for challenging prompts involving style, position, counting, and multiple objects.", "questions": "1. Could you clarify how the semantic information gain term and approximation error term interact? While equation 10 separates these terms, it's unclear how they might counteract or reinforce each other in practice.\n2. Can the method be extended to Video generation or 3D generation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Interesting observation and reasonable method: The paper found that the latent variable contains semantic information, and the sampling result would be enhanced. The paper does the denoising and inversion process iteratively by using different semantic scales to add the semantic information in the latent variable. The method is intuitive and reasonable.\n\n2. Strong Theoretical Foundation: The paper provides rigorous theoretical analysis explaining why Z-Sampling works, including detailed proofs showing how semantic information accumulation differs between end-to-end and zigzag approaches. \n\n3. Comprehensive Evaluation: The authors evaluate their method across multiple diffusion models, various datasets and different metrics. They also provide extensive ablation studies examining the impact of different parameters.\n\n4. Practical Applicability: Z-sampling is a plug-and-play method that can enhance existing diffusion models without requiring model retraining. It can also be combined with other optimization techniques like Diffusion-DPO and AYS-Sampling for further improvements.", "weaknesses": "1. Large computation: Z-sampling requires additional computation due to the zigzag steps. While they show that it can achieve better results with fewer steps, the method still requires more computation than standard sampling for the same number of steps.\n\n2. Limited to Deterministic Samplers: The method's effectiveness is reduced when used with stochastic samplers due to inversion inaccuracies. This limitation restricts its applicability to deterministic sampling methods like DDIM. Besides, the inversion such as DDIM have approximate errors. The paper should investigate how it impacts the results of some experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IbTQ1ZMqIV", "reviewer": "Reviewer_A4t2", "summary": "This paper identifies that the guidance gap between denoising and inversion captures prompt-specific semantic information and proposes Z-Sampling to gradually accumulate this information throughout the generation process, through iteratively denoising and reversing. Extensive experiments demonstrate that  Z-Sampling significantly improves image quality and prompt alignment, outperforming other baseline models.", "questions": "1. In Tables 1 and 2 of the main results, what guidance scales were used for each model, and what was the specific setting for $\\gamma_1$?\n2. In Figure 10, how is the x-axis varied? My understanding is that the time taken for a model to generate a single image should be constant.\n\n3. How might Z-Sampling be integrated into the training phase to achieve improved results? Currently, the oversaturation effect seems too pronounced.\n\n4. $\\gamma_2$ is set to 0 in most cases, suggesting that the reverse process disregards unconditional information entirely. Could a more detailed theoretical explanation be provided for this finding?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper demonstrates strong problem identification and analysis, accurately pinpointing the reasons behind guidance failures with certain seeds in text-to-image diffusion models.\n\n2. The presentation of the paper is clear and well-structured, making it easy to follow and understand the theoretical insights and proposed methods.\n\n3. The experimental results are thorough, covering multiple baselines and benchmarks to robustly validate the effectiveness of the approach.", "weaknesses": "1. My major concern is that the qualitative results appear unsatisfactory. In almost all qualitative examples, Z-Sampling exhibits a pronounced **oversaturation** effect, with excessively vivid subject colors and significantly blurred backgrounds. While this approach may excel on certain benchmarks, these results are unlikely to be perceived as high-quality by users in practical evaluations.\n\n2. Following from weakness 1, this oversaturated, unrealistic effect often occurs at very high CFG scales. As shown in Figure 3, Z-Sampling may essentially function as a high-CFG sampling method to enforce guidance. The paper does not provide a detailed comparison between Z-Sampling and high-CFG sampling, either theoretically or experimentally, and the results suggest that both approaches share similar drawbacks.\n\n3. The paper lacks deeper theoretical analysis. The mathematical proofs provided rely on basic formula applications, but Z-Sampling alters the sampling path, creating a shift between the denoising trajectories of inference and training. In contrast, standard CFG sampling has rigorous theoretical backing. Revisiting the score function to analyze how training and inference could align under Z-Sampling conditions would strengthen the paper’s theoretical foundation.\n\n4. Limited technical innovation. The proposed iterative inverse-and-denoise approach has already been used in various image editing applications, and also, in image to video methods like Tune-A-Video[1] to enhance structural coherence. The paper would benefit from further elaboration to clarify its unique technical contributions.\n\n5. Lack of comprehensive analysis. CFG sampling incorporates not only semantic information but also an unconditional branch that helps with structural aspects. By attributing improvements solely to semantic response, the paper misses an analysis of structural and visual enhancements. A more detailed examination here could strengthen the paper’s argument and overall impact.\n\n6. There are some typographical errors. For instance, in Algorithm 1, the steps for denoising and inversion are mistakenly reversed. \n\n[1] Wu, Jay Zhangjie, et al. \"Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ak8BjwkiEq", "reviewer": "Reviewer_BMZe", "summary": "The paper introduces a novel approach called Zigzag Diffusion Sampling (Z-Sampling) to enhance the performance of existing diffusion models, particularly for complex and fine-grained prompts in text-to-image generation tasks. This method employs a zigzag-like alternating denoising and inversion process, which progressively accumulates semantic information. Z-Sampling enables flexible control over the injection of semantic information, is applicable across various diffusion models, and requires minimal code modifications. Extensive validation on multiple benchmark datasets, diffusion models, and evaluation metrics demonstrates its effectiveness, especially in handling complex prompts such as style, position, counting, and multiple objects. As a training-free approach, Z-Sampling significantly improves baseline performance with lower computational cost, showing substantial practical value.", "questions": "- See Weakness\n- In each zigzag optimization, only one step of denoising and one step of inversion is performed. Have you considered performing multiple steps of denoising and multiple steps of inversion?\n- I am also curious about Figure 9. Theoretically, using the zigzag algorithm indeed quick and effective alignment, but as the number of optimization steps approaches infinity, the accumulated prediction errors should lead to a gradual downward trend. However, Figure 9 does not reflect this; it even shows an accelerating upward trend. How can this phenomenon be explained?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Through zigzag sampling, semantic information in the latent space is progressively accumulated, effectively enhancing the alignment between generated content and prompts, especially for complex prompts.\n\n- Z-Sampling is a plug-and-play method that can be seamlessly applied to different types of diffusion models with minimal code modifications.\n\n- As a training-free sampling method, Z-Sampling reduces computational costs while maintaining high generation quality, with experiments demonstrating its performance superiority over standard sampling within the same time frame.", "weaknesses": "- The authors’ insight—that the semantic information contained in the latent affects generation results—is quite evident. \n- The method’s process is very simple and straightforward, which reminds me of time travel. The difference is that, in time travel, noise is added when tracing back, while zigzag uses inversion. The improvement in time travel comes from the inherent bias in the sampling process. By performing additional noise addition and denoising steps, the distribution can be pulled back onto the intermediate data manifold, thereby improving quality. Thus, to imbue $x_t$ with more semantic information of $c$, using inversion instead of time travel’s random noise to pull the data distribution back onto the intermediate data manifold $p_t(x_t|c)$ for enhancing conditional sampling quality seems like an obvious approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D0BXU0djJ1", "reviewer": "Reviewer_9XhW", "summary": "The paper presents a novel sampling method, Zigzag Diffusion Sampling (Z-Sampling), designed to enhance the generation quality of text-to-image diffusion models. The key contribution lies in leveraging the guidance gap between the denoising and inversion processes to accumulate semantic information throughout the generation process. This leads to improved image quality and prompt-image alignment, especially for challenging prompts. Z-Sampling is a plug-and-play method that can be easily integrated into various diffusion models. Extensive experiments demonstrate its superior performance across different datasets, models, and evaluation metrics.", "questions": "- Would you compare Z-sampling with other training-free enhancement techniques?\n- Would you compare Z-sampling with \"Text-to-Image Rectified Flow as Plug-and-Play Priors\"?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear and well-organized paper: The paper is well-structured, making it easy to follow the authors' arguments.\n- Theoretical and empirical analysis: The authors provide a strong theoretical foundation for the guidance gap between the denoising and inversion processesand  their approach and support it with extensive empirical evidence.\n-  Z-Sampling is a unique contribution to the field of diffusion models, offering a new way to improve generation quality.\n- Plug-and-play nature: The method's simplicity and ease of integration make it attractive for researchers and practitioners.\n- Strong experimental results: The authors demonstrate the effectiveness of Z-Sampling through comprehensive experiments.", "weaknesses": "- Comparison to other training-free enhancement techniques: While the paper mentions other training-free enhancement techniques, a more in-depth comparison with methods like SAG[1], SEG[2], and CFG++[3] would strengthen the paper. This would provide a clearer understanding of Z-Sampling's unique advantages.\n- Comparision to \"Text-to-Image Rectified Flow as Plug-and-Play Priors\"[4]: The proposed method has similarity with this paper in terms of utilizing the forwad and reverse process to enhance the quality. \n\n\n[1] Hong, Susung, et al. \"Improving sample quality of diffusion models using self-attention guidance.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Hong, Susung. \"Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention.\" arXiv preprint arXiv:2408.00760 (2024).\n\n[3] Chung, Hyungjin, et al. \"CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models.\" arXiv preprint arXiv:2406.08070 (2024).\n\n[4] Yang, Xiaofeng, et al. \"Text-to-Image Rectified Flow as Plug-and-Play Priors.\" arXiv preprint arXiv:2406.03293 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)