Zigzag Diffusion Sampling: Diffusion Models Can Self-Improve via Self-Reflection

Authors: Bai LiChen, Shitong Shao, zikai zhou, Zipeng Qi, zhiqiang xu, Haoyi Xiong, Zeke Xie
Primary area: generative models
Keywords: Diffusion model, Semantic Information, Classifier Guidance Gap

Abstract:
Diffusion models, the most popular generative paradigm so far, can inject conditional information into the generation path to guide the latent towards desired directions. However, existing text-to-image diffusion models often fail to maintain high image quality and high prompt-image alignment for those challenging prompts. To mitigate this issue and enhance existing pretrained diffusion models, we mainly made three contributions in this paper. First, we propose **diffusion self-reflection** that alternately performs denoising and inversion and demonstrate that such diffusion self-reflection can leverage the guidance gap between denoising and inversion to capture prompt-related semantic information with theoretical and empirical evidence. Second, motivated by theoretical analysis, we derive Zigzag Diffusion Sampling (Z-Sampling), a novel self-reflection-based diffusion sampling method that leverages the guidance gap between denosing and inversion to accumulate semantic information step by step along the sampling path, leading to improved sampling results. Moreover, as a plug-and-play method, Z-Sampling can be generally applied to various diffusion models (e.g., accelerated ones and Transformer-based ones) with very limited coding and computational costs. Third, our extensive experiments demonstrate that Z-Sampling can generally and significantly enhance generation quality across various benchmark datasets, diffusion models, and performance evaluation metrics. For example, DreamShaper with Z-Sampling can self-improve with the HPSv2 winning rate up to **94%** over the original results. Moreover, Z-Sampling can further enhance existing diffusion models combined with other orthogonal methods, including Diffusion-DPO. The code is publicly available at
[github.com/xie-lab-ml/Zigzag-Diffusion-Sampling](https://github.com/xie-lab-ml/Zigzag-Diffusion-Sampling).

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729088820310

thread_type:
official_review

<thread_object>
creator: reviewer_9xhw

<content>
summary:
The paper presents a novel sampling method, Zigzag Diffusion Sampling (Z-Sampling), designed to enhance the generation quality of text-to-image diffusion models. The key contribution lies in leveraging the guidance gap between the denoising and inversion processes to accumulate semantic information throughout the generation process. This leads to improved image quality and prompt-image alignment, especially for challenging prompts. Z-Sampling is a plug-and-play method that can be easily integrated into various diffusion models. Extensive experiments demonstrate its superior performance across different datasets, models, and evaluation metrics.

strengths:
- Clear and well-organized paper: The paper is well-structured, making it easy to follow the authors' arguments.
- Theoretical and empirical analysis: The authors provide a strong theoretical foundation for the guidance gap between the denoising and inversion processesand  their approach and support it with extensive empirical evidence.
-  Z-Sampling is a unique contribution to the field of diffusion models, offering a new way to improve generation quality.
- Plug-and-play nature: The method's simplicity and ease of integration make it attractive for researchers and practitioners.
- Strong experimental results: The authors demonstrate the effectiveness of Z-Sampling through comprehensive experiments.

weaknesses:
- Comparison to other training-free enhancement techniques: While the paper mentions other training-free enhancement techniques, a more in-depth comparison with methods like SAG[1], SEG[2], and CFG++[3] would strengthen the paper. This would provide a clearer understanding of Z-Sampling's unique advantages.
- Comparision to "Text-to-Image Rectified Flow as Plug-and-Play Priors"[4]: The proposed method has similarity with this paper in terms of utilizing the forwad and reverse process to enhance the quality. 


[1] Hong, Susung, et al. "Improving sample quality of diffusion models using self-attention guidance." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Hong, Susung. "Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention." arXiv preprint arXiv:2408.00760 (2024).

[3] Chung, Hyungjin, et al. "CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models." arXiv preprint arXiv:2406.08070 (2024).

[4] Yang, Xiaofeng, et al. "Text-to-Image Rectified Flow as Plug-and-Play Priors." arXiv preprint arXiv:2406.03293 (2024).

questions:
- Would you compare Z-sampling with other training-free enhancement techniques?
- Would you compare Z-sampling with "Text-to-Image Rectified Flow as Plug-and-Play Priors"?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730377319532

thread_type:
official_review

<thread_object>
creator: reviewer_bmze

<content>
summary:
The paper introduces a novel approach called Zigzag Diffusion Sampling (Z-Sampling) to enhance the performance of existing diffusion models, particularly for complex and fine-grained prompts in text-to-image generation tasks. This method employs a zigzag-like alternating denoising and inversion process, which progressively accumulates semantic information. Z-Sampling enables flexible control over the injection of semantic information, is applicable across various diffusion models, and requires minimal code modifications. Extensive validation on multiple benchmark datasets, diffusion models, and evaluation metrics demonstrates its effectiveness, especially in handling complex prompts such as style, position, counting, and multiple objects. As a training-free approach, Z-Sampling significantly improves baseline performance with lower computational cost, showing substantial practical value.

strengths:
- Through zigzag sampling, semantic information in the latent space is progressively accumulated, effectively enhancing the alignment between generated content and prompts, especially for complex prompts.

- Z-Sampling is a plug-and-play method that can be seamlessly applied to different types of diffusion models with minimal code modifications.

- As a training-free sampling method, Z-Sampling reduces computational costs while maintaining high generation quality, with experiments demonstrating its performance superiority over standard sampling within the same time frame.

weaknesses:
- The authors’ insight—that the semantic information contained in the latent affects generation results—is quite evident. 
- The method’s process is very simple and straightforward, which reminds me of time travel. The difference is that, in time travel, noise is added when tracing back, while zigzag uses inversion. The improvement in time travel comes from the inherent bias in the sampling process. By performing additional noise addition and denoising steps, the distribution can be pulled back onto the intermediate data manifold, thereby improving quality. Thus, to imbue $x_t$ with more semantic information of $c$, using inversion instead of time travel’s random noise to pull the data distribution back onto the intermediate data manifold $p_t(x_t|c)$ for enhancing conditional sampling quality seems like an obvious approach.

questions:
- See Weakness
- In each zigzag optimization, only one step of denoising and one step of inversion is performed. Have you considered performing multiple steps of denoising and multiple steps of inversion?
- I am also curious about Figure 9. Theoretically, using the zigzag algorithm indeed quick and effective alignment, but as the number of optimization steps approaches infinity, the accumulated prediction errors should lead to a gradual downward trend. However, Figure 9 does not reflect this; it even shows an accelerating upward trend. How can this phenomenon be explained?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730543992565

thread_type:
official_review

<thread_object>
creator: reviewer_a4t2

<content>
summary:
This paper identifies that the guidance gap between denoising and inversion captures prompt-specific semantic information and proposes Z-Sampling to gradually accumulate this information throughout the generation process, through iteratively denoising and reversing. Extensive experiments demonstrate that  Z-Sampling significantly improves image quality and prompt alignment, outperforming other baseline models.

strengths:
1. The paper demonstrates strong problem identification and analysis, accurately pinpointing the reasons behind guidance failures with certain seeds in text-to-image diffusion models.

2. The presentation of the paper is clear and well-structured, making it easy to follow and understand the theoretical insights and proposed methods.

3. The experimental results are thorough, covering multiple baselines and benchmarks to robustly validate the effectiveness of the approach.

weaknesses:
1. My major concern is that the qualitative results appear unsatisfactory. In almost all qualitative examples, Z-Sampling exhibits a pronounced **oversaturation** effect, with excessively vivid subject colors and significantly blurred backgrounds. While this approach may excel on certain benchmarks, these results are unlikely to be perceived as high-quality by users in practical evaluations.

2. Following from weakness 1, this oversaturated, unrealistic effect often occurs at very high CFG scales. As shown in Figure 3, Z-Sampling may essentially function as a high-CFG sampling method to enforce guidance. The paper does not provide a detailed comparison between Z-Sampling and high-CFG sampling, either theoretically or experimentally, and the results suggest that both approaches share similar drawbacks.

3. The paper lacks deeper theoretical analysis. The mathematical proofs provided rely on basic formula applications, but Z-Sampling alters the sampling path, creating a shift between the denoising trajectories of inference and training. In contrast, standard CFG sampling has rigorous theoretical backing. Revisiting the score function to analyze how training and inference could align under Z-Sampling conditions would strengthen the paper’s theoretical foundation.

4. Limited technical innovation. The proposed iterative inverse-and-denoise approach has already been used in various image editing applications, and also, in image to video methods like Tune-A-Video[1] to enhance structural coherence. The paper would benefit from further elaboration to clarify its unique technical contributions.

5. Lack of comprehensive analysis. CFG sampling incorporates not only semantic information but also an unconditional branch that helps with structural aspects. By attributing improvements solely to semantic response, the paper misses an analysis of structural and visual enhancements. A more detailed examination here could strengthen the paper’s argument and overall impact.

6. There are some typographical errors. For instance, in Algorithm 1, the steps for denoising and inversion are mistakenly reversed. 

[1] Wu, Jay Zhangjie, et al. "Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

questions:
1. In Tables 1 and 2 of the main results, what guidance scales were used for each model, and what was the specific setting for $\gamma_1$?
2. In Figure 10, how is the x-axis varied? My understanding is that the time taken for a model to generate a single image should be constant.

3. How might Z-Sampling be integrated into the training phase to achieve improved results? Currently, the oversaturation effect seems too pronounced.

4. $\gamma_2$ is set to 0 in most cases, suggesting that the reverse process disregards unconditional information entirely. Could a more detailed theoretical explanation be provided for this finding?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730782827250

thread_type:
official_review

<thread_object>
creator: reviewer_uffk

<content>
summary:
This paper introduces Z-Sampling, a novel sampling method for diffusion models that improves image generation quality and prompt alignment. The paper investigates that if the latent variable contains the sementic information, the sampling result would be enhanced. The key insight is that the guidance gap between denoising and inversion processes can capture semantic information. Rather than doing end-to-end semantic injection, Z-Sampling accumulates semantic information step-by-step through zigzag sampling. At each step, the denoising process is followed by inversion. The authors provide a theoretical analysis showing why this approach is more effective than end-to-end injection. The paper demonstrates empirical improvements across multiple diffusion models, datasets, and evaluation metrics. The method is particularly effective for challenging prompts involving style, position, counting, and multiple objects.

strengths:
1. Interesting observation and reasonable method: The paper found that the latent variable contains semantic information, and the sampling result would be enhanced. The paper does the denoising and inversion process iteratively by using different semantic scales to add the semantic information in the latent variable. The method is intuitive and reasonable.

2. Strong Theoretical Foundation: The paper provides rigorous theoretical analysis explaining why Z-Sampling works, including detailed proofs showing how semantic information accumulation differs between end-to-end and zigzag approaches. 

3. Comprehensive Evaluation: The authors evaluate their method across multiple diffusion models, various datasets and different metrics. They also provide extensive ablation studies examining the impact of different parameters.

4. Practical Applicability: Z-sampling is a plug-and-play method that can enhance existing diffusion models without requiring model retraining. It can also be combined with other optimization techniques like Diffusion-DPO and AYS-Sampling for further improvements.

weaknesses:
1. Large computation: Z-sampling requires additional computation due to the zigzag steps. While they show that it can achieve better results with fewer steps, the method still requires more computation than standard sampling for the same number of steps.

2. Limited to Deterministic Samplers: The method's effectiveness is reduced when used with stochastic samplers due to inversion inaccuracies. This limitation restricts its applicability to deterministic sampling methods like DDIM. Besides, the inversion such as DDIM have approximate errors. The paper should investigate how it impacts the results of some experiments.

questions:
1. Could you clarify how the semantic information gain term and approximation error term interact? While equation 10 separates these terms, it's unclear how they might counteract or reinforce each other in practice.
2. Can the method be extended to Video generation or 3D generation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732339188479

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All

comment:
We are thankful for the feedback and suggestions from all the reviewers. We are glad that the reviewers recognize our work (Z-Sampling), acknowledge our **interesting and reasonable findings** and **strong theoretical foundation** (uffk, 9XhW), **comprehensive evaluation experiments** (uffk, A4t2, BMZe, 9XhW). Additionally, its **ease of integration and practical applicability** make Z-Sampling inspiring for the community and researchers (uffk, BMZe, 9XhW).

---

Here, we would like to highlight and clarify the novelty and contributions of Z-Sampling as public comments:

- **What** signifies a better latent variable in the Diffusion Model? (Sec 3.1)

  Observing the latent's generation results under weak conditions to judge its performance under specific strong conditions.

- **How** to improve the properties of an initial latent variable to enhance generation quality? (Sec 3.2)
  
  Demonstrating that the cfg gap is the key component, controlling this gap can determine the effect of optimization.

- **Why** end-to-end methods is inferior to step-by-step approaches? (Sec 3.3)

  Conducting theoretical and experimental analysis to reveal the impact of semantic term $\tau _{1}$ and error terms $\tau _{2}$,  further improving the performance of Z-Sampling.

---

We duly address the reviewers' concerns in the individual responses. 

Here, we summarize the supplementary experiments and contents for addressing the concerns in the revision.
- In Appendix D.4,  we demonstrate that Z-Sampling is still powerful under extremely high cfg guidance and can even mitigate oversaturation caused by strong cfg guidance.   (A4t2)

- In Appendix D.1, we presents the results of Z-Sampling on multi-step denoising and multi-step inversion settings, further demonstrating the superiority of the step-by-step approach. (BMZe)

- In Line 478-483, we provide a more detailed explanation for Figure 9. (BMZe)

- In Appendix E.2/E.3, we conduct extra experimental analysis on the error term $\tau_{2}$, which indicates that: (uffk, BMZe)

  - An increase in $\tau_{2}$ leads to negative gain.

  - A step-by-step approach can reduce the magnitude of $\tau_{2}$.

- In Appendix D.1, we demonstrate Z-Sampling remains effective in other generative tasks such as video generation. (uffk)

- In Appendix D.1, we show that Z-Sampling can also outperform recent methods for improving CFG mechanisms, such as CFG++ and SEG. (9XhW)

- In Appendix B, we discuss the relationship between Z-Sampling and other methods, such as Time-Travel, Tune-a-video, further clarifying the novelty and contributions of our work. (A4t2, BMZe)

---

Finally, we hope that these clarifications and explanations resolve the reviewer’s concerns and questions. We are more than happy to discuss if the reviewer has further questions or comments.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a sampling method for conditional diffusion models that alternates between denoising and inversion to improve the alignment of text prompts and images for T2I models. Although alternating between denoising steps and inversion steps is not new, this paper is the first to demonstrate its effectiveness for conditional generation. The Zigzag scheme is simple, effective, and generally applicable to any off-the-shelf diffusion models.
One weakness is the extra computational cost compared to standard sampling due to inversion steps. However, Zigzag can be applied to selected timestamps which makes it faster if achieving the same level of image quality and fidelity. Zigzag is also limited to deterministic samplers, and it doesn't improve image generation for stochastic samplers like DDIM.
Overall, this paper is well-motivated by interesting but somewhat evident observations. The proposed Zigzag scheme is simple, modular, and effective for image generation as well as video generation.