SANA: Efficient High-Resolution Text-to-Image Synthesis with Linear Diffusion Transformers

Authors: Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, Song Han
Primary area: generative models
Keywords: efficient ai, diffusion models, text to image generation

Abstract:
We introduce Sana, a text-to-image framework that can efficiently generate images up to 4096$\times$4096 resolution. Sana can synthesize high-resolution, high-quality images with strong text-image alignment at a remarkably fast speed, deployable on laptop GPU. Core designs include: (1) Deep compression autoencoder: unlike traditional AEs, which compress images only 8$\times$, we trained an AE that can compress images 32$\times$, effectively reducing the number of latent tokens. (2) Linear DiT: we replace all vanilla attention in DiT with linear attention, which is more efficient at high resolutions without sacrificing quality. (3) Decoder-only text encoder: we replaced T5 with modern decoder-only small LLM as the text encoder and designed complex human instruction with in-context learning to enhance the image-text alignment. (4)  Efficient training and sampling: we propose Flow-DPM-Solver to reduce sampling steps, with efficient caption labeling and selection to accelerate convergence. As a result, Sana-0.6B is very competitive with modern giant diffusion model (e.g. Flux-12B), being 20 times smaller and 100+ times faster in measured throughput. Moreover, Sana-0.6B can be deployed on a 16GB laptop GPU, taking less than 1 second to generate a 1024$\times$1024 resolution image. Sana enables content creation at low cost. Code and model will be publicly released upon publication.

Decision: accept (oral)
Reviewer scores: 10, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730179533406

thread_type:
official_review

<thread_object>
creator: reviewer_dipw

<content>
summary:
The paper introduces Sana, a novel framework for efficient text-to-image generation, capable of producing 4K images while maintaining strong text-image alignment at remarkable speed. Sana’s core innovations include: a deep compression autoencoder, a Linear DiT architecture to enhance efficiency and a LLM-based text encoder. The model achieves competitive results compared to much larger models like Flux-12B, boasting significantly lower computational requirements.

strengths:
1. Efficiency is a really big selling point. With a very high compression rate of deep autoencoder and linear DiT, SANA can generate high-quality images even on a laptop is very impressive. 

2. The fact that using a small but good LLM can outperform a big but bad text encoder is another interesting point. This boosts text-image alignment a lot when used with their proposed complex human instruction (CHI) pipeline.

3. They provide extensive ablation experiments on each components and measure their impact on efficiency and perform of the model, which is very helpful for future work to build upon.

weaknesses:
1. Although the results are impressive, the contribution of this paper on the technical side is somewhat limited. Their three main components have been already explored in the literature, deep autoencoder in [1], linear DiT in [2] and using LLM as text encoder in [3, 4]. So combining them is not really a significant novel idea.

2. In section 2.3, the author do not fully explain their CHI pipeline, is it same as [5] or completely different. If it is similar then this even reduces the novelty of the work further.

Reference:

[1] Chen et. al. Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models. 2024

[2] Liu et. al. LinFusion: 1 GPU, 1 Minute, 16K Image. 2024

[3] Hu et.al. ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment. 2024

[4] Liu et.al. LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation. 2024

[5] Ma et. al. Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models. 2024

questions:
I do not have any further question
</content>
</thread_object>

</thread>
<thread>
creation date:
1730248439665

thread_type:
official_review

<thread_object>
creator: reviewer_s4e6

<content>
summary:
This paper proposes a text-to-image framework that can efficiently generate images up to 4096 x 4096 resolution. Comparing to existing works, the paper proposes four designs to facilitate fast speed, high quality text-to-image generation: (1) An auto encoder that down samples to input image by 32 times. (2) Replacing the quadratic self attention with linear attention. (3) Applies decoder-only LLM as text encoder. (4) Adopts efficient caption labelling and Flow-DPM-Solver for training and inference. Experimental results show that the proposed model achieves better or comparable results in terms of image quality, text image alignment comparing to existing methods, while having much higher throughput and lower latency

strengths:
1. The paper is well motivated, as developing method that facilitates high quality, high resolution text-to-image generation in resource limited scenarios is of great application value.
2. The paper addresses the problem from several aspects, including auto encoder design, light weight self attention module, sampling method, and better caption labeling procedure. 
3. The experimental results on resolutions of 512 x 512, 1024 x 1024 show that the proposed indeed achieve better or comparable performance comparing to previous state of the art methods, while having much higher throughput and lower latency. 
4. The ablation experiments of different design components are sufficient and convincing.

weaknesses:
1. The design of linear attention block needs to be explained in more detail. For example, the original self attention helps modelling long range dependency between tokens, here in the linear attention block, how is the dependency / contextual relation between tokens modeled through the new operations ?
2. The paper claims that the proposed method is able to generate images up to 4096 x 4096 resolution, however, in terms of quality comparison with existing methods, there are only ones with resolution of 512, 1024. For 4096 resolution, only the speed comparison is reported (Table 14), it would be more convincing to include FID and other image-text alignment metrics comparison for 4096 resolution as well.

questions:
Please see weaknesses section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730443351863

thread_type:
official_review

<thread_object>
creator: reviewer_mbm3

<content>
summary:
The paper "SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers" introduces SANA, a high-efficiency text-to-image generation framework capable of producing high-resolution images up to 4096×4096 pixels. SANA employs multiple innovations, including a deep compression autoencoder (compressing images 32× for latency reduction), linearized Diffusion Transformer (DiT) attention to improve high-resolution efficiency, and the use of a decoder-only LLM as a text encoder with "Complex Human Instruction" for enhanced image-text alignment. The model also implements a Flow-DPM-Solver to reduce sampling steps, achieving competitive quality at 100× faster speeds compared to state-of-the-art diffusion models. SANA's improvements are demonstrated on high-resolution benchmarks and support deployment on a laptop GPU.

strengths:
1. Innovative Efficiency Strategies: SANA’s integration of linear attention, a high-compression autoencoder, and a unique sampling solver enables fast high-resolution generation, which holds practical value for many applications.
2. Effective Use of Large Language Model (LLM) for Text Encoding: SANA’s implementation of a decoder-only LLM with CHI to refine prompts improves text-to-image alignment, enhancing quality without incurring high latency.

weaknesses:
Limited Ablation in Model Design: While SANA combines existing methods effectively, many techniques—such as linear attention, high-compression auto-encoders, and Flow-based solvers—are iterative upon recent advancements in diffusion transformers. Explicit comparisons with recent models like PixArt-Σ or Playground v3, which also incorporate high-efficiency strategies or decoder-only LLMs, with component by component comparisons would better attribute SANA’s unique contributions.

questions:
1. SANA uses the Gemma-2 LLM with Complex Human Instruction (CHI) to improve text-image alignment. While similar works like Playground v3 uses Llama3-8B, this paper only compared T5 with Gemma, do we know if larger LLMs can further improve the generation quality?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730469407153

thread_type:
official_review

<thread_object>
creator: reviewer_9t32

<content>
summary:
The paper presents Sana, a cutting-edge text-to-image synthesis framework that can generate high-resolution images up to 4096×4096. Here are the key highlights:
Unlike conventional autoencoders that compress images by a factor of 8×, Sana's autoencoder achieves a remarkable 32× compression, drastically reducing the number of latent tokens.

Sana innovates by replacing the standard attention mechanisms in the Diffusion Transformer (DiT) with linear attention. This change enhances efficiency at high resolutions without sacrificing image quality.

Decoder-Only Text Encoder: Instead of relying on T5, Sana uses a modern, decoder-only small language model (LLM) as its text encoder. This model, combined with sophisticated human instructions and in-context learning, significantly improves text-image alignment.

Sana introduces Flow-DPM-Solver to cut down on the number of sampling steps and employs efficient caption labeling and selection techniques to speed up training convergence.

These advancements allow Sana to produce high-quality images at a significantly faster pace than existing models.

strengths:
This work demonstrates its originality through several innovative contributions, including the Deep Compression Autoencoder, Linear DiT, and impressive 4K generation ability. These innovations enhance the quality and efficiency of high-resolution image generation while reducing computational requirements. The paper is well-written and clearly structured, with detailed experiments and results that validate the effectiveness of the proposed methods. Additionally, the significance of the work lies in its practical applications and the removal of limitations from prior models, making high-resolution image synthesis more accessible and scalable.
Overall, I find this work to be a breakthrough for diffusion text-to-image models.

weaknesses:
1. The Gemma2-2B-IT model in Table 9 needs to be explained, as not all readers have a solid background in LLMs.

2. Further comparisons between Gemma2 and T5-XXL are needed, such as showing HPSv2 metrics on complex prompt benchmarks. The current FID scores in Table 9 are insufficient to highlight the true advantages of Gemma2.

3. Why does SANA perform significantly better than LUMINA-Next, even though both use the Gemma-2B model?

4. I believe some relevant works should be cited or compared, such as high-resolution generation work: UltraPixel: Advancing Ultra-High-Resolution Image Synthesis to New Peaks (NeurIPS 2024) and efficient generation work: Stable Cascade.

questions:
1. The Gemma2-2B-IT model in Table 9 needs to be explained, as not all readers have a solid background in LLMs.

2. Further comparisons between Gemma2 and T5-XXL are needed, such as showing HPSv2 metrics on complex prompt benchmarks. The current FID scores in Table 9 are insufficient to highlight the true advantages of Gemma2.

3. Why does SANA perform significantly better than LUMINA-Next, even though both use the Gemma-2B model?

4. I believe some relevant works should be cited or compared, such as high-resolution generation work: UltraPixel: Advancing Ultra-High-Resolution Image Synthesis to New Peaks (NeurIPS 2024) and efficient generation work: Stable Cascade.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732300054747

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers

comment:
We sincerely thank all reviewers for their thorough reviews and constructive feedback. We are encouraged that the reviewers recognize SANA's innovations in efficient high-resolution image generation, practical value for resource-limited scenarios, and significant performance improvements while maintaining lower computational requirements (all reviewers).

## Key Improvements and Clarifications

1. Technical Novelty and Component Analysis
- Provided detailed comparisons with recent works like PixArt-Σ 
- Demonstrated unique innovations in VAE design, training schedule, and model architecture
- Explained how SANA significantly improves upon existing techniques through Linear Attention Mix-FFN, LLM and CHI designs

2. Extended Evaluation and Analysis
- Added comprehensive metrics comparing Gemma2 and T5-XXL using GenEval and HPSv2
- Explained linear attention's efficiency while maintaining global receptive field
- Clarified evaluation challenges at 4K resolution due to benchmark limitations

3. Safety and Practical Considerations
- Integrated ShieldGemma for content moderation to prevent misuse
- Focused on consumer GPU compatibility and efficiency

Please see our detailed responses below each review and all the changes in the revised paper is high-lighted with orange font. We appreciate your feedback and welcome any additional questions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work presents SANA, a cutting-edge text-to-image synthesis framework that can generate high-resolution images up to 4096×4096.  

It has the integration of linear attention, a high-compression autoencoder, and a unique sampling solver that enables fast, high-resolution generation, which holds practical value for many applications. Moreover, it also shows the effective use of the Large Language Model (LLM) for text encoding, which is simple yet effective. Extensive experiments on various Gen benchmarks show the effectiveness of SANA.

All reviewers feel positive before the rebuttal, with most concerns about more extensive experiment results and ablation studies.
After the rebuttal and discussion stage, all the issues are solved, and all reviewers raise their scores.

The area chair checks the refined version of this submission and suggests the authors merge the discussion results in their camera-ready version. 

In addition, the author should also include a paragraph to discuss the potential misuse of SANA like creating harmful AI-generated contents in their final draft.

In the end, due to the effectiveness of the model and the practical potential of efficient generation, the area chair recommends this work as an oral presentation.