SANA: Efficient High-Resolution Text-to-Image Synthesis with Linear Diffusion Transformers
Authors: Enze Xie
Junsong Chen
Junyu Chen
Han Cai
Haotian Tang
Yujun Lin
Zhekai Zhang
Muyang Li
Ligeng Zhu
Yao Lu
Song Han
Venue: ICLR 2025 Oral (2025)
Keywords: Efficient AI
Diffusion Models
Text to Image generation
Primary area: generative models
TL;DR: Sana can synthesize high-resolution, high-quality images with strong text-image alignment at a remarkably fast speed.

Abstract: We introduce Sana, a text-to-image framework that can efficiently generate images up to 4096$\times$4096 resolution. Sana can synthesize high-resolution, high-quality images with strong text-image alignment at a remarkably fast speed, deployable on laptop GPU. Core designs include: (1) Deep compression autoencoder: unlike traditional AEs, which compress images only 8$\times$, we trained an AE that can compress images 32$\times$, effectively reducing the number of latent tokens. (2) Linear DiT: we replace all vanilla attention in DiT with linear attention, which is more efficient at high resolutions without sacrificing quality. (3) Decoder-only text encoder: we replaced T5 with modern decoder-only small LLM as the text encoder and designed complex human instruction with in-context learning to enhance the image-text alignment. (4)  Efficient training and sampling: we propose Flow-DPM-Solver to reduce sampling steps, with efficient caption labeling and selection to accelerate convergence. As a result, Sana-0.6B is very competitive with modern giant diffusion model (e.g. Flux-12B), being 20 times smaller and 100+ times faster in measured throughput. Moreover, Sana-0.6B can be deployed on a 16GB laptop GPU, taking less than 1 second to generate a 1024$\times$1024 resolution image. Sana enables content creation at low cost. Code and model will be publicly released upon publication.

Published as a conference paper at ICLR 2025
SANA: EFFICIENT HIGH-RESOLUTION IMAGE SYN- THESIS WITH LINEAR DIFFUSION TRANSFORMERS
Enze Xie Junsong Chen'r, Junyu Chen 2.3 Han Cai',
Haotian Tang",
Yujun Lin",
Zhekai Zhang", Muyang Liº, Ligeng Zhu', Yao Lu', Song Han
INVIDIA MIT
TTsinghua University
Project Page: mvlabs.github..i
Res:4352xx584
SANA
SO FAST
SANA
(a). Generation results from our 1.6B models
Flux-Dev
1,023.0
SD3
4.4
SD3
230.0
PixArtE
2.7
PixArt-Z
186.0
SANA-1.88
1.2
SANA-1.68
25x
106x
SANA-0.68
0.9
SANA-0.6B
9.6
12
24
275
(b). Generation latency (s) on 1024x1024
(c). Generation latency (s) on 4096x4096
Figure 1: An overview of generated images and inference latency of Sana.
ABSTRACT
We introduce Sana, a text-to-image framework that efficiently generate images up to 4096x4096 resolution. Sana can synthesize high-resolution, high-quality images with strong text-image alignment remarkably fast speed, deployable on laptop GPU. Core designs include: (1) Deep compression autoencoder: un- like traditional AEs, which compress images only 8x, we trained an AE that can compress images 32x, effectively reducing the number of latent tokens. (2) Lin- ear DiT: we replace all vanilla attention in DiT with linear attention, which is efficient high resolutions without sacrificing quality. (3) Decoder-only text encoder: we replaced T5 with modern decoder-only small LLM as the text encoder and designed complex human instruction with in-context learning hance the image-text alignment. (4) Efficient training and sampling: we propose Mow--DMM-Solver reduce sampling steps, with efficient caption labeling and selection to accelerate convergence. As a result, Sana-0.60 6B is very competitive with modern giant diffusion model (e.g. Flux-12B), being 20 times smaller and 100+ times faster in measured throughput. Moreover, Sana-0.6B can be deployed on 16GB laptop GPU, taking less than second to generate 1024x 1024 reso- lution image. Sana enables content creation at low cost. Code and model will be publicly released.
Project

---

Published as a conference paper at ICLR 2025
INTRODUCTION
In the past year, latent diffusion models have made significant progress in text-to-image research and have generated substantial commercial value. On one hand, there a growing consensus among researchers regarding several key points: (1) Replace U-Net with Transformer architectures (Chen et al., 2024b;a; Esser et al., 2024; Labs, 2024), (2) Using Vision Language Models (VLM) for auto-labelling images (Chen et al., 2024b; OpenAI, 2023; Zhuo et al., 2024; Liu et al., 2024a) (3) Improving Variational Autoencoders (VAEs) and Text encoder (Podell et al.. 2023; Esser et al., 2024; Dai al., 2023) (4) Achieving ultra High- nighhteesolutin image generation (Chen et al., 2024a), etc. On the other hand, industry models are becoming increasingly large, with parameter counts escalating from PixArrts 0.6B parameters to SD3 8B, LiDiT 10B, Flux 1 12B, and Playground v3 at 24B. This trend results in extremely high training and inference costs, creating challenges for most consumers who find these models difficult and expensive use. Given these challenges, pivotal question arises: Can we develop high quahiiyutttyy and high-resolutton image generator that is computationally efficient and runs very fast on both cloud and edge devices?
Flux-dev
1022
Params: 12B, GenEval: 0.67, DPG: 84.0;
Sana Basellne
469
Deep Compression
AutoEncader
11.4x
1.7x
Linear DIT
1.1x
Kernel Fusion
21
Flow DPM-Solver
2.2x
(Sana final version)
Paraire: 0.68, GenEvat 0.84, DPG:83.8
106X
Figure 2: Algorithm and system co-optimize reduce the inference latency of 4096x4096 image generation, from 469 seconds to 9.6 seconds, and achieve 106x faster than the current SOTA model, FLUX. The numbers are measured with batch size I on an A100 GPU.
This paper proposes Sana, pipeline designed efficiently and cost-effectively train and synthesize images at resolutions ranging from 1024 x 1024 to 4096x4096 with high quality. To our knowl- edge, no published works have directly explored 4K resolution image generation, except for PixArt- Σ (Chen et al., 2024a). However, PixArt-E is limited to generating images close to 4K resolution (3840x2160) and is relatively slow when producing such high- high-resolution images. To achieve this ambitious goal, we propose several core designs:
Deep Compression Autoencoder: We introduce new Autoencoder (AF) in Section 2.1 that ag- gressively increases the scaling factor to 32. In the past, mainstream AEs only compressed the image's ' length and width with factor of 8 (AE-F8). Compared with AE-F8, our AE-F32 outputs 16 x fewer latent tokens, which is crucial for efficient training and generating ultra-bigh-resolltion images, such as 4K resolution.
linear DiT to replace vanilla quadratic attention modules (Section 2.2). The computational complexity of the original DiT's self-attention is O(N3), which increases quadratically when processing high-resolution images. We replace all vanilla attention with linear attention, reducing the computational complexity from O(N2) to O(N). At the same time, we propose Mix-FFN, which integrates 3x3 depth-wise convolution into MLP aggregate the local information of tokens. We argue that linear attention can achieve results comparable to vanilla attention with proper design and is more efficient for high-resolution image generation (e.g., accelerating by 1.7x x at 4K). Additionally, the indirect benefit of Mix-FFN is that we do not need position encoding (NoPE). For the first time, we removed the positional embedding in DiT and find no quality loss.
Efficient Linear DiT: We introduce
Decoder-only Small LLM as Text Encoder: Section 2.3, we utilize the latest Large Language Model (LLM), Gemma, as our text encoder to enhance the understanding and reasoning capabilities regarding user prompts. Although texl-to-image generation models have advanced significantly over the years, most existing models still rely CLIP or T5 for text encoding, which often lack robust text comprehension and instruction-follone followinn abilities. Decoder-only LLMs, such as Gemma, ex- hibit strong text understanding and reasoning capabilities, demonstrating an ability follow human instructions effectively. In this work, we first address the training instability issues that arise from di- rectly adopting an LLM as text encoder. Secondly, we design complex human instructions (CHI) to
2

---

Published as a conference paper at ICLR 2025
leverage the LLM's powerful instruction in-context learning, and reasoning capabilities to improve image-text alignment.
Efficient Training and Inference Strategy: In Section 3.1, propose set of automatic labelling and training strategies to improve the consistency between text and images. First, for each image, we utilize multiple VLMss generate re-captions. Although the capabilities of these VLMs vary, their complementary strengths improve the diversity of the captions. In addition, we propose a clipscore- based training strategy (Section 3.2), where we dynamically select captions with high clip scores for the multiple captions corresponding to an image based on probability. Experiments show that this approach improve training convergence and text-image alignment. Furthermore, We propose Flow-DPM-Solf Moolerl that reduces the inference sampling steps from 28-50 to 14-20 steps compared to the widely used Noo-Fuler-Solvee while achieving better results.
In conclusion, our Sana-0.6B achieves throughput that is over 100x faster than the current state-of- the-art method (FLUX) for 4K image generation (Figure 2), and 40x faster for 1K resolution (Fig- ure 4), while delivering competitive results across many benchmarks. In addition, Sana-0.6B and deploy it on an edge device, as detailed in Section 4. takes only 0.37s to generate 1024x 1024 resolution image on customer-grade 4090 GPU, providing powerful foundation model for real-time image generation. We hope that our model can be efficiently utilized by all industry professionals and everyday users, offering them significant business value.
quantize
METHODS
2.1 DEEP COMPRESSION AUTOENCODER
2.1.1 PRELIMINARY
To mitigate the excessive training and inference costs associated with directly running diffusion models in pixel space, Rombach et al. (2022) proposed latent diffusion models that operate a compressed latent space produced by pre-trained autoencoders. The most commonly used autoen- coders in previous latent diffusion works (Peebles & Xie, 2023; Bao er al., 2022; Cai et al., 2024; Esser al., 2024; Dai et al., 2023; Chen et al., 2024b;a) feature down-sampling factor of F 8, mapping images from pixel space #x to latent space IR # 4 xC where C' represents the number of latent channels. In DiT-based methods (Peebles & Xie, 2023), the number of tokens pro- cessed by the diffusion models is also influenced by another hyper-paramettr, P. known as patch size. The latent features grouped into patches of size P P. resulting in H W tokens. A typical patch size in previous works
PF x PF
In summary, previous latent diffusion models (LDM), c.g. PixArt (Chen et al., 2024b), SD3 (Esser et al., 2024) and Flux (Labs, 2024), usually employ AE-F8C4P2 or AE-F8C16P2, where the AE compresses images by 8x and DiT compresses by 2x. In our Sana, aggressively scale the compression factor to 32x and propose several techniques to maintain the quality.
2.1.2 AUTOENCODER DESIGN PHILOSOPHY
Unlike the previous AE-F8, aim to increase the compression ratio more aggressively. The mo- tivation is that high-resolution images naturally contain more redundant information. Moreover, efficient training and inference of high-resolution images (e.g., 4K) necessitate a high compression ratio for the autoencoder. Table illustrates that on MJHQ-30K, although previous methods (e.g., SDv1.5) have attempted use AE-F32C64, the quality remains significantly inferior to that of AE- F8C4. Our AE-F32C32 effectively bridges this quality gap, achieving reconstruction capabilities comparable to SDXL's A&C4 We believe that the minor difference in AE will not become bottleneck for DiT's capability.
Moreover, instead of increasing the patch size P. we argue that the autoencoders should take full responsibility for compression, allowing the latent diffusion models to focus solely denoising. Therefore, we develop an AE with samn-iampling factor of F = 32, Channel C = 32, and run diffusion models in its latent space with patch size of 1 AE--F32C32P)). This design reduces the number tokens by 4x, significantly improving training and inference speed while lowering GPU memory requirements.
3

---

Published as a conference paper at ICLR 2025
2.1.3 ABLATION OF AUTOENCODER DESIGNS
From the perspective of model structure, we implement several adjustments accelerate conver- gence. Specifically, We replace the vanilla self attention mechanism with linear attention blocks to improve the efficiency of high-resolution generation. Additionally, from a training standpoint, we propose mtati-stage training strategy to improve training stability, which involving finetune our AE-F32C32 on 1024 x 1024 images to achieve better reconstruction results on high-resolution data.
Can we compress tokens in DiT using larger patch size? We compare AE-F8C16P4, AE-F16C32P2 and AE-F32C32PI. These three settings compress a 1024x1024 image into the same number of token numbers 32 x 32. As shown in Figure 3(a), although AE-F8C16 ex- hibits the best reconstruction ability (rFID: (CC16=<FF66633223322222
Table 1: Reconstruction capability of different Autoencoders.
Autoencoder
rFID
PSNR
SSIM 1 LPIPS
F8C4 (SDXL)
0.31
31.41
0.88
0.04
F32C64 (SD)
0.82
27.17
0.79
F32C32 (Ours)
0.34
29.29
0.84
0.05
we empirically
find that the generation results of F32C32 are superior (FID: F12033PP<FF1C3322<<T8811844 This indicates that allowing the autoencoder to focus soiely on high-ratio compression and the dif- fusion model concentrate on denoising the optimal choice.
F8C16P4 (/FID-0.09)
=1 F16C32P2
F32C32P1 (/FID+0.66
F32016PP (/FID=0.85)
F32C84P1 (FIDD-031)
F32C3PP1 (FID--048)
(FIDD-0116
26.2
26
100
24
23.8
50
15
18
5k
10k
17k
35k
5k
10k
17k
35k
5k
10k
17k
39k
5k
10k
17k
30k
FID comparison (sleps)
CLIP soore comparison (sleps)
FID comparison (steps)
CLIP score comparison (steps)
(a) Ablation where to compress: AE/ DIT
(b) Ablation on the effect #channels AE-F32.
Figure 3: Ablation study on our deep compression autoencoder (AE).
We explore various channel configurations and finally choose C=32 as our optimal setting. As shown in Figure 3(b), fewer channels converge more quickly, but the reconstruction quality is worse. We observe that after 35K training steps, the convergence speeds of C=16 and C=32 are similar; however, C=32 yields better reconstruction metrics, resulting in better FID and CLIP scores. Although C=64 offers superior reconstruction, its following DiT's training convergence speed is significantly slower than that of C=32.
Different Channels in AE-F32:
2.2 EFFICIENT LINEAR DIT DESIGN
Model Performance Comparison
The self-attention used by DiT has com- putational complexity of O(?2), resulting in low computational efficiency when processing high-resolution images and incurring signifi- cant overhead. To address this issue, we first proposed linear DiT, which completely replaces the original self-attention with linear attention, achieving higher computational efficiency in high-resolution generation without compromis- ing performance. In addition, employ Mix- FFN to replace the original MLP-FFN, incor- porating 3x3 depth-wise convolution better aggregate token information. These micro de- signs are inspired by Cai et al. (2023); Xie et al. (2021), but we keep DiT's macro architecture design to maintain simplicity and scalability.
FLLIK-Dev
FLUX Fohn-Gchnell
GenEval Overall Results
Sana-1 1,8 (Our))
Sana-0.00 0.08 (Dun))
8D3-
Params
0,0t
40x acceleration
PlayGronnd12.5
128
LUMINA-Nor Next
14 Throughput (samples/second)
Figure 4: Comparison between Sana and of the-art diffusion models under 1024x 1024 reso- lution. All models are tested on an A100 GPU. Sana provides 0.64 GenEval overall performance with only 590M model parameters.
/nttps://huggt waw.nnggaacccooo stabi tityat /sdxl vae 2nttps:/ thub.
http:///gtthuubccco/o////ttet/te wwwp/////////////e/oooooooo attettt--datttt Ffuston
4

---

Published as a conference paper at ICLR 2025
Cemple indeectre ofter Froeatt
Linear Attention
Mix-FFN
nes plocs
Small LLM
Toxe
cale
ConvLayer
Txx
Euf de ae chli
thone 14012
Time
MatVul
noor aiga thar sags
User Forrst Exbrrunn cal
Pos
MarMMl
SILU
SANA
Linear Attn
Cross Atnn
Mix FFN
ReLU
nxd
ReLU
x Convlayer
Deep Compression AutoE ncoder (32x)
Linear
Linear
1x ConvLayer
Linear DIT Blocks)
Cost:
(a). Architecture overview of our Sana.
(b). Linear DiT Module.
Figure 5: Overview of Sana: Fig. (a) describes the high-level training pipeline, containing our 32x deep compression Autoencoder, Linear DiT, and complex human instruction. Note that Positional embedding is not required in framework. Fig. (b) describes the detailed design of the Linear Attention and Mix-FFN in Linear DIT.
Linear Attention block. An illustration of utilized linear attention module is provided in Fig- ure 5. To reduce computational complexity, we replace the traditional softmax attention with ReL.U linear attention (Katharopoulos et al., 2020). While ReLU linear attention and other variants (Cai et al., 2023; Wang et al., 2020; Shen et al., 2021; Bolya et al., 2022) have primarily been explored in high-resolution dense prediction tasks, our work represents an early exploration demonstrate the effectiveness of linear attention in image generation.
The computational benefits of our approach are evident in the implementation. As shown in Eq. 1, instead of computing attention for each query, we compute shared terms (Σ ReLU(K,) e iedd and (>ny1 ReLU(K,))) e gaxt only once. These shared terms can then be reused for each query, leading to a linear computational complexity of O(N) in both memory and computation.
ReLU(Q. RLUUKK,
(EX RoLUCK,) VV))
O, - j=1 EF (ELUUQ,,ReLLCA))
ReLU(QU(E)) (e)L(
Mix-FFN block. As discussed in EfficientViT (Cai al., 2023), linear attention models benefit from reduced computational complexity and lower latency compared to softmax attention. However, the absence of non-linear similarity function may lead to sub-optimal performance. We observe sim- ilar conclusion in image generation, where linear attention models suffer from much slower conver- gence despite eventually achieving comparable performance. To further improve training efficiency, replace the original MLP-FE MLP-FFN with -FN. The Mix-FFN consists of an inverted residual block, a 3x3 depth-wise convolution, and a Gated Linear Unit (GLU) (Dauphin et al., 2017). The depth-wise convolution enhances the model's 's ability capture local information, compensating for the weaker local information-captuee capturing ability of ReLU linear attention. Performance ablations for the model design space are shown in Table
DIT without Positional Encoding (NoPE). We are surprised that we can remove Positional Embed- ding without any loss in performance. Some earlier theoretical and practical works have mentioned that introducing 3x3 convolution with zero padding can implicitly incorporate the position infor- mation (Islam al., 2020; Xie et al., 2021). contrast to previous DiT-based methods that mostly use absolute PF, learnable PF, and RoPF, we propose NoPE, the first design that entirely omits po- sitional embedding in DIT. Recent cutting-edge research the L.M field (Kazemnejad al., 2024; Haviv et al., 2022) has also indicated that NoPE may offer better length generalization ability.
Triton Acceleration Training/Inference To further accelerate linear attention, we use Tri- ton (Tillet et al., 2019) to fuse kernels for both the forward and backward passes of the linear atten- tion blocks to speed up training and inference. By fusing all element-wise operations aperations-includingg activation functions, precision conversions, padding operations, and divisions iivisions-into matrix multi- plications, we reduce the overhead associated with data transfer. We attach more details and benefits coming from Triton to the appendix.
5

---

Published as a conference paper at ICLR 2025
2.3 TEXT ENCODER DESIGN
Why Replace T5 to Decoder-only LLM as Text Encoder? The most advanced LLMs nowadays are decoder-only GPT architectures that are trained on a larger scale of data. Compared to T5 (a method proposed in 2019), decoder-only LLMs possess powerful reasoning capabilities. They can follow complex human instructions by using Chain-of-Thought (CoT) (Wei et al., 2022) and In- context-learning (ICL) (Brown, 2020). In addition, some small LLMs, such Gemma-2 (Team et al., 2024a), can rival the performance of large LLMs while being very efficient. Therefore, we choose adopt Gemma-2 our text encoder.
As shown in Table. 9, Compared to T5-XXL, Gemma-2-22 2-2B has an inference speed that is 6x faster, while the performance of Gemma-2B is comparable to T5-XXL in terms of Clip Score, FID, GenEval, and HPSv2 (Wu et al., 2023).
Stabilize Training using LLM as encoder: We extract the last layer of features of the Gemma- 2 decoder as text embedding. We empirically find that directly following the approach of using T5 text embedding key, value, and image tokens (as the query) for cross-attention training results in extreme instability, with training loss frequently becoming NaN.
We find that the variance of T5's text embedding is several orders of magnitude smaller than that of the decoder-only LLMs 1arrmma-1-2B. Gemma-2-22- Qwen-2-0.5B), indicating that there are many large absolute values in the text embedding output. To address this issue, we add a normalization layer (i.e., RMSNorm) after the decoder-only text encoder, which normalizes the variance of the text embeddings to 1.0. In addition, we discover useful trick that further accelerates model convergence by initializing small learnable scale factor (e.g., 0.01) and multiplying it by the text embedding. The results are shown Figure 6.
Table 2: Ablation study of whether using Complex Human Instruction (CHI).
scalle:1.0 1.0
scale=0.01
26
27.5
25
Prompt
Train Step
GenEval
18
22.5
User
45.5
CHI User
52K
47.7 (+2.2)
10
20
16K
34K
52K
User
140K
52.8
FID comparison (stepas
CUP score comparison (steps)
CHI User
5K(tinetune)
54.8 (+2.0)
Figure 6: Ablation study of whether using text embedding normalization and small scale factor.
Complex Human Instruction Improves Text- Image Alignment: As mentioned above, Gemma has better instmaction-following capa- bilities than T5. We can further leverage this ca- pability strengthen text embedding. Gemma is a chat model, and although it possesses strong capabilities, it be somehow unpre- dictable, thus we need to add instructions to help it focus on extracting and enhancing the prompt itself. LiDiT (Ma et al., 2024) is the first combine simple human instruction with user prompts. Here, we further expand it by using in-context learning of LLM design a complex human instruction (CHI). As shown in Table 2, incorporating CHI during whether from scratch or through fine-tuning- prove the image-text alignment capability.
User prompt A wio Complex Human Instruction
Complex Human Instruction
further im-
Details Miss ing
Artifacts x
R Realistic v
Rich Details
Figure 7:
Generations w/ w/o Complex- Hluman-Instruche (CHI). Without CHI, a simple prompt may lead to inferior generations, includ- ing artifacts and less-detailed results.
Additionally, shown in Figure 7, we find that when given a short prompt such as "a cat", CHI helps the model generate more stable content. This is evident in the fact that models without CHI often output content unrelated the prompt.

---

Published as a conference paper at ICLR 2025
3 EFFICIENT TRAINING/INFERENCE
3.1 DATA CURATION AND BLENDING
Multi-Caption Auto-labelling Pipeline: For each image, whether or not it contains an origi- nal prompt, we will use four VLMs to label it: VILA-3BB13B (Lin et al., 2024), Inter 8B/26B (Chen et al., 2024d). Multiple VLMs can make the caption more accurate and more diverse.
ALIP-Score-base Caption Sampler: One problem multi-captioning presents selecting the cor- responding one for an image during training. The naive approach randomly selects a caption, which may select low-quality text and affect model performance.
We propose clip score-based sampler, the motivation is to sample high-quality text with greater probability. We first calculate the clip score G, for all captions corresponding to an image, and then, when sampling, sample according to the probability based the clip score. Here, introduce
exp(a./)) an additional hyper-parameter, temperature into the probability formulation P(e) Zسz. expley/r)
The temperature can be used to adjust the sampling intensity. If the temperature is near 0, only the text with the highest clip score is sampled. The results in Table 4 show that variations in captions have minimal impact on image quality (FID) while improving semantic alignment during training.
Cascade Resolution Training: Benefiting from using AE-F32C32PI, we skip the 256px pre- training and start pre-training directly at resolution of 512px, gradually fine-tuning the model to 1024px, 2K and 4K resolution. We believe that the traditional practice of pre-training at 256px too cost-effective, as images at 256 resolution lose too much detailed information, resulting in slower learning for the model in terms of image-text alignment.
Flow-Euler
Flow-DPM-Sovver
Table 3: Comparison of different training sched- ules on 256x 256 resolution.
30
27
Schedule
FID |
CLIP
Iterations
25
26
DDPM
19.5
24.6
120K
20
25
Flow Matching
16.9
25.7
120K
15
24
Comparison of image-text pair sam- pling strategies during training.
Table 4:
10
14
20
28
50
10
14
20
28
50
FID comparison (sampling steps)
CLIP score comparison (sampling steps)
Prompt Strategy
FID +
CLIP
Iterations
Figure 8: Impact of sampling steps on FID and CLIP-Score: A Comparison between Flow- DPM-Solver and Flow-Euler.
Single
6.13
27.10
65K
Multi-random Multi-clipscore
6.15
27.13
65K
6.12
27.26
65K
3.2 FLOW-BASED TRAINING/ / INFERENCE
Flow-based Training: We analyze the superior performance of Rectified Flow from SD3 (Esser et al., 2024) and find that, unlike DDPM (Ho al., 2020), which rely noise prediction, both I-Rectified Flow (RF) (Lipman et al., 2022) and EDM (Karras et al., 2022) use data or velocity pre- diction, resulting in faster convergence and improved performance. Specifically, all these methods ag Xg + o2 -e, where x, represents the image data, denotes random noise, and a and o, are hyper-parameters of the diffusion process. In DDPM train- ing, the objective is noise prediction, defined as en (x,,,)) t)
follow a common diffusion formulation:
However, both EDM and RF follow different approach: EDM aims for data prediction with the objective x (x,, t) Xo. while RF uses velocity prediction with the objective no (x,,7 t) to data or velocity prediction is critical near = T. where noise prediction can lead to instability, while data velocity prediction provides more precise and stable estimates. As noted by Balaji et al. (2022), attention activation near t = T is stronger, further emphasizing the importance of accurate predictions at this critical point. This shift effectively reduces cumulative errors during sampling, resulting in faster convergence and improved performance. Further details can be found in Appendix B.
Xg. This transition from noise prediction
7

---

Published as a conference paper at ICLR 2025
Flow-based Inference: In our work, modify the original DPM-Solver++ (Lu et al., 2022b) adapting the Rectified Flow formulation, named Flow-DPM-Solver. The key adjustments involve substituting the scaling factor a, with 1 22 where S, remains unchanged but time-steps are defined over the range [o. 1] instead of [1, 1000), with a time-step shift applied to achieve a lower signal-noise ratio, following SD3 (Esser al., 2024). Additionally, our model predicts the veloeity field, which differs from the data prediction in the original D Specifically, data is derived from the relation: data + 20 = IT oT uo (IT, (TT,tT), where Uy (-) is the velocity predicted by the model.
As a result, in Figure 8, our Sow-DDMM-Sofe converges at 14~20 steps with better performance, while the Flow-Fulle sampler needs 28~50 steps for convergence with a worse result.
4 ON-DEVICE DEPLOYMENT
To enhance edge deployment, we quantize our model with 8-bit integers. Specifically, we adopt per- token symmetric INT8 quantization for activation and per-channel symmetric INT8 quantization for weights. Moreover, to preserve high semantic similarity to the 16-bit variant while incurring minimal runtime overhead, we retain normalization layers, linear attention, and key-value projection layers within the cross-attention block full precision.
Table 5: On-device Deployment: our inference engine with W8A8 quantization realized 2.4x speedup when generating 1024px images on the laptop GPU. The performance of Sana is assessed with the CLIP-Score MJHQ-30K (Li et al., 2024a) and the Image-Rewand Reward (Xu et al., 2024) on its
first IK images.
Methods
Latency (s)
CLIP-Score
Rewae-Reward? f
Sana (FP166)
0.88
28.5
1.03
W8A8 Quantization
0.37
28.3
0.96
We implement our W8A8 GEMM kernel in CUDA C++ and employ kemnll fusion techniques to mitigate the overhead associated with unnecessary activation loads and stores, thereby enhancing overall performance. Specifically, we integrate the ReLU(K)TV product of linear attention (Equa- tion 1) with the QKV -projecct layer; we also fuse the Gated Linear Unit (GLU) with the quanti- zation kernel in Mix-FFN, and combine other element wise operations. Additionally, we adjust the activation layout to avoid any transpose operations in GEMM and Conv kernels.
Table 5 shows the speed comparison before and after deployment optimization a laptop GPU. For generating a 1024px image, our optimized implementation achieves 2.4 speedup, taking only 0.37 seconds, while maintaining almost lossless image quality.
EXPERIMENTS
Model Details. Table describes the details of the network architecture. Our Sana-0.6B only contains 590M parameters, and the number of layers and channels is almost identical to those of the original DIT-XL and PixArt-E, Our S.na-1.6B increases the parameters to 1.6B, with 20 layers and 2240 channels per layer, and increases the channels to 5600 in FFN. We believe that keeping the model layers between 20 and 30 strikes a good balance between efficiency and quality.
Evaluation Details. We use five mainstream evaluation metrics to evaluate the performance of our Sana, namely FID, Clip Score, GenEval (Ghosh et al., 2024), DPG- (Hu et al., 2024), and ImageReward (Xu et al., 2024), comparing it with SOTA methods. FID and Clip Score are evaluated on the MJHQ-30K (Li et al., 2024a) dataset, which contains 30K images from Midjourney. GenEval and DPG-Bench both focus on measuring text-image alignment, with 533 and 1,065 test prompts, respectively. ImageReward assesses human preference performance and includes 100 prompts.
Table 6: Architecture details of the proposed Sana.
Model
Width
Depth
FFN
#Heads
#Param (M)
Sana-0.6B
1152
28
2880
36
590
Sana-1.65
2240
20
5600
70
1604
8

---

Published as a conference paper at ICLR 2025
Table 7: Comprehensive comparison of our method with SOTA approaches in efficiency and performance. The speed is tested on one A100 GPU with FP16 Precision. Throughput: Measured with batch=16. Latency: Measured with batch=1 and sampling step=20. We highlight the best, second best, and third best entries.
Methods
Throughput (samples/s)
Lateney (8)
Params (B)
Speedup
FID
CLIP
GenEval
DPG 1
512 x512 resolution
PixArt-a (Chen et al., 2024b)
1.5
1.2
0.6
1.0x
6.14
27.55
0.48
71.6
PixArt-S (Chen al., 2024a)
1.5
1.2
1.0x
6.34
27.62
79.5
Sama-0.6B
6.7
0.8
0.6
5.0x
5.67
27.92
0.64
84.3
Sana-166
3.8
0.6
1.6
2.5x
5.16
28.19
0.66
85.5
1024 x 1024 resolution
LUMINA-Next (Zhuo et al., 2024)
0.12
9.1
2.0
2.8x
7.58
26.84
0.46
74.6
SDXL (Podell
2023)
3.5x
6.63
74.7
PlayGroundv2.5 (Li al. 2024a)
0.21
5.3
2.6
4,9x
6.09
29.13
0.56
75.5
Hunyuan- DIT (Li et al., 2024c)
0.05
18.2
1.5
1.2x
6.54
28.19
0.63
78.9
PixArt-E (Chen al., 2024a)
0.4
2.7
0.6
9.3x
6.15
28.26
0.54
80.5
DALLES (OpenAl, 2023)
0.67
83.5
SD3-medium (Esser al., 2024)
0.28
4.4
2.0
6.5x
11.92
27.83
84.1
FL.UX (1 (Labs, 2024)
0.04
23.0
12.0
1.0%
10.15
27.47
0.67
84.0
FLUX-schnell (Labs, 2024)
0.5
2.1
12.0
11.6x
7.94
28.14
0.71
84.8
Sana-0.6B
1.7
0.9
39.5x
5.81
28.36
83.6
Sana-166
1.0
1.2
1.6
23.3x
5.76
28.67
84.8
Table 8: Performance of Sana block design space. The speed is tested on one A100 GPU with FP16 Precision with 1024 image size. MACs: acltulacc operations for single forward pass. TP (Throughput): Measured with batch=16. Latency: Measured with batch=1.
(T)
TP (/s)
Latency (ms)
FullAnt & FFN
FSC4P2
6.48
0.49
2250
+ LinearAttn
FXC4P2
4,30
0.52
1931
+ MixFFN
F8C4P2
4.19
0.46
2425
Kernel Fusion
F8C4P2
4.19
0.53
2139
LincarAttn & MixFFN
F32C32P1
1.08
1.75
826
+Kernel Fusion
F32C32P1
1.08
2.06
748
5.1 PERFORMANCE COMPARISON AND ANALYSIS
We compare Sana with the most advanced text-to-image diffusion models in Table 7. For 512 x 512 resolution, Sama-0.6 demonstrates a throughput that is 5x faster than PixArt- which has a similar model size, and significantly outperforms it in FID, Clip Score, GenEval, and DPG-Bench. For 1024 x 1024 resolution, Sana is considerably stronger than most models with <3B parameters and excels in inference latency. Our models achieve competitive performance even when compared to the most advanced large model FLUX-dev. For instance, while the accuracy on DPG-Bench is equivalent and slightly lower GenEval, Sana-0.6B's throughput is 39x faster, and Sana-1.6B is 23x faster.
In Table 8, we analyze the efficiency of replacing the original DiT's modules with the corresponding linear DiT's modules under the 1024 x 1024 resolution setting. We observe that using AE-F8C4P2, replacing the original full attention with linear attention can reduce latency from 2250ms m but the generation results are worse. Replacing the original FFN with our Mix-FFN compensates for the performance loss, although it sacrifices some efficiency. With Triton kernel fusion, our linear DIT can ultimately be slightly faster than the original DiT at the 1024px scale and faster at higher resolution. Moreover, when upgrading from AE-F8C4P2 to AE-F32C32P1, the MACs can be further reduced by 4x, and throughput can also be improved by 4x. Triton kernel fusion can bring 100 speed acceleration.
The left side of Figure 9 compares the generation results of Sana, Flux-dev, SD3, and PixArt-E. In the first row text rendering, PixArt-E lacks text rendering capability, while Sana can render text accurately. In the second row, the quality of the images generated by Sana and FLUX is comparable, while SD3's text understanding is inaccurate. The right side of Figure shows that Sana can be successfully deployed laptop locally. A Demo video is provided in the appendix.

---

Published as a conference paper at ICLR 2025
Table Comparison of different Text-Eneoders. All models are tested with an A GPU with FP16 precision after 50K training steps. Gemma-2B models achieve better performance than T5- large at similar speed and comparable performance the larger, much slower T5-XXL
Text-Encoder
#Params (M)
Latency (s)
FID
CLIP f
GenEval
HPSv2 1
T5-XXL
4762
1.61
6.1
27.1
0.51
27.95
T5-Large
341
0.17
6.1
26.2
0.39
27.57
Gemma2-2B
2614
0.28
6.0
26.9
0.45
27.92
Demmma-2B-l
2506
0.21
5.9
26.8
0.47
27.78
Gemmal-2B4.4
2614
0.28
6.1
26.9
0.50
27.92
Sana-1.8 (1.2s)
FLUX-dev (23s)
(4.4s)
(2.7s)
barta
FAST
Fast
fast
Cymppae cal with raco aign Ihai saya "Fast"
SAHA
Sana locally deployable laptop
atficctc Shiba while orveticad straght girtt Haater), while ahorta, and shott whill stoows.
detaileed
Figure 9: Left: Visualization comparison of Sama-1.66 vs FLUX-dev, SD3 and PixArt-E. The speed is tested A100 GPU with FP16 precision. Right: Quantize Sana-1.6B is deployable on a GPU laptop generating an 1Kx x 1K image within seconds.
RELATED WORK
We put a relatively brief overview related work in the main text, with a more comprehensive version available in the supplementary material. In terms of generative model architecture, Diffusion Transformer (Peebles & Xie, 2022) and DiT-based text-to-image extensions (Chen et al., 2024b; Esser et al., 2024; Labs, 2024) have made significant progress over the past year. Regarding text encoder, the earliest work (Rombach et al., 2022) uses CLIP, while subsequent works (Saharia et al., 2022; Chen al., 2024b;a) adopt XS-XXL. There also efforts that combine T5 and CLIP (Balaji et al., 2022; Esser et al., 2024). For high-resolution generation, PixArt-S (Chen et al., 2024a) the first model capable of directly generating images at 4K resolution. Additionally, GigaGAN (Kang et al., 2023) can generate 4K images using super-resolution model. In the context of on-device deployment, Zhao et al. (2023); Li et al. (2024b) have explored the deployment of diffusion models on mobile devices.
7 CONCLUSION
This paper builds new efficient text-to-image pipeline named Sana. We have made improvements in the following dimensions: we propose a deep compression autoencoder, widely use linear atten- tion to replace self-attention in DIT, utilize decoder-only L.M as text encoder with complex human instruction, establish an automatic image caption pipeline, and propose flow-based DPM-Solver to accelerate sampling. Sana can generate images at maximum resolution of 4096 4096, delivering throughput more than x higher than the SOTA methods while maintaining competitive genera- tion results.
In the future, we will consider building an efficient video generation pipeline based on Sana. A potential limitation of this work is that it cannot fully guarantee the safety and controllability of the generated image content. Additionally, challenges remain in certain complex cases, such as text rendering and the generation of faces and hands.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements. We would like to thank Shuchen Xue from UCAS, Cheng Lu from OpenAI, Jincheng Yu from HKUST, and Chongjian Ge from HKU for discussions and data collection.
REFERENCES
Yogesh Balaji. Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint urXiv:2211.01324 2022.
Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu. All are worth words: vit backbone for score-based diffusion models. In NeurIPS 2022 Workshop on Score-Basc Methods, 2022.
Daniel Bolya, Cheng-Yau Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman. Hydra attention: Efficient attention with many heads. In European Conference on Computer Vision, pp. 35-49. Springer, 2022.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005 2020.
Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han. Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17302-17313, 2023.
Han Cai, Muyang Li. Qinsheng Zhang, Ming- Yu Liu, and Song Han. Condition-awand neural net- work for controlled image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7194-7203, 2024.
Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13166-13175. 2020.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.0440 2024a.
Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang. James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International Conference on Learning Representations, 2024b.
Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep compression autoencoder for efficient high-resolution diffusion models. arXiv preprint arXiv:2410.1073: 2024c.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24199 2024d.
Xiaoliang Dai, Ji Hou, Chih- Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pp. 933-941. PMLR, 2017.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focusee framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024.
11

---

Published as a conference paper at ICLR 2025
Cong Guo, Yuxian Qiu, Jingwen Leng, Xiaotian Gao, Chen Zhang, Yunxin Liu, Fan Yang, Yuhao Zhu, and Minyi Guo. Squant: On-the-fly data-free quantization via diagonal hessian approxima- tion. ArXiv, abs/2202.07471, 2022.
Adi Haviv, Oni Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. arXiv preprint arXiv:2203.16634, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with I1m for enhanced semantic alignment. arXiv preprini 25X0v:2403.05 2024.
Md Amirul Islam, Sen Jia, and Neil DB Bruce. How much position information do convolutional neural networks encode? arXiv preprini arXiv:2001.0822 2020.
Minguk Kang. Jun-Jay Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124-10134, 2023.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577, 2022.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. 5156-5165, PMLR, 2020.
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024.
Black Forest Labs. Flux, 2024. URL https: /ttpps:///llaeee http///kkkkorrttab..se..e..
Daiging Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.1 17245, 2024a.
Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang. Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 17535-17545, October 2023.
Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. Advances in Neural Information Processing Systems, 36, 2024b.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. [BRECQ}: Pushing the limit of post-training quantization by block reconstruction. In International Conference Learning Representations, 2021. URL https: htttps://ppeneeview /oppnrevie......ee net/ forum? (tt.==owwvwddddo
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolltiion diffusion transformer with fine-grained chinese understanding. arXiv preprint arXh:22405.08748, 2024c.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi. and Song Han. Vila: On pre- training for visual language models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. (6689-26699)) 2024.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamme Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint 2rXiv:2210.02777, 2022.
12

---

Published as a conference paper at ICLR 2025
Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving eext-t--maae alignment with deep-fusion large language models. arXiv preprint arXtv:2409.10693, 2024a.
Mushui Liu, Yuhang Ma, Yang Zhen, Jun Dan. Yunlong Yu, Zeng Zhao, Zhipeng Hu, Bai Liu, and Changjie Fan. Llm4gen: Leveraging semantic representation of 1lms for text-to-image generation. arXiv preprint arXiv:2407.007377 2024b.
Cheng Lu. Research on reversible generative models and their efficient algorithms, 2023. URL pdf.
https:://ucchee
githhbi io/ files/ wwwes/chenglllleeeeeeeeeeee cwwnghuedeheeee
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 1155775-5787 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:221..01995, 2022b.
Bingqi Ma, Zhuofan Zong, Guanglu Song, Hongsheng Li, and Yu Liu. Exploring the role of large language models in prompt encoding for diffusion models. arXiv preprint arXiv:2406. 1131, 2024.
OpenAl. Dalle-3, 2023. URL httpt::t////ooeteot com/ fomm/dal dall-e-31
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748. 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Pablo Pernias, Dominic Rampas, Mats L. Richter, Christopher Pal, and Mare Aubreville. Wuer- stchen: An efficient architecture for large-scale text-to-image diffusion models, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXivv2307.01952, 2023.
Jingjing Ren, Wenbo Li, Haoyu Chen, Renjing Pei, Bin Shao, Yong Guo, Long Peng, Fenglong Song, and Lei Zhu. Ultraixel: Advancing tetra-high-reolutin image synthesis to new peaks. arXiv preprint arXiv:2407.000 020000 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image syuthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-366 36494, 2022.
Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on appli- cations of computer vision, pp. 3531-3539, 2021.
Gemma Team. Gemma. 2024. doi: 10.347 10.34770KKAGGLL///SSS hRL kaggle. com/m/ 3301.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv:2488.001 18, 2024a.
13

---

Published as a conference paper at ICLR 2025
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Fer- ret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Char- line Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchi- son, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antomia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette- Choo, Danila Sinopalnikov, David Wein- berger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczakk-Puuiffee Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan. Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kar- tikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moyni- han, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalic Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Osear Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culli- ton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Amold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ron- strom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Hli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dra- gan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Fara- bet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at practical size, 2024b. URL https: : /atxxvxxvvv////////////////aaa org/ abs / 2408 00118.
Harleen
Yao Teng, Yue Wu, Han Shi, Xuefei Ning, Guohao Dai, Yu Wang, Zhenguo Li, and Xihui Liu. Dim: Diffusion mamba for efficient high-resolution image synthesis. arXiv preprint 2:X.i:2405.14222 2024.
Yuchuan Tian, Zhijun Tu, Hanting Chen, Jie Hu, Chao Xu, and Yunhe Wang. U-dits: Downsample tokens in u u-shaped diffusion transformers. arXiv preprint wwXiv:2405.02730. 2024.
Philippe Tillet, Hsiang-Tsuue Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10-19, 2019.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.cm 04768. 2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis, 2023. URL https: / a arxiv. .rrg/ aass/ 6.366 09341.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077-12090, 2021.
14

---

Published as a conference paper at ICLR 2025
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Jing Nathan Yan, Jiatao Gu, and Alexander M Rush. Diffusion models without attention. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8239-8249, 2024.
Wenjun Zeng. Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, Olivia Sturman, and Oscar Wahltinez. Shieldgemma: Generative content moderation based on gemma, 2024. URL attps:://rxtiee org/ wtg/abs////ooooooorot.go//....ts
Yang Zhao, Yanwu Xu, Zhisheng Xiao, and Tingbo Hou. Mobilediffusion: Subsecond text-to-image generation on mobile devices. arXiv preprint arXiv:2311. 2023.
Lianghui Zhu, Zilong Huang, Bencheng Liao, Jun Hao Liew, Hanshu Yan, Jiashi Feng, and Xing- gang Wang. Dig: Scalable and efficient diffusion models with gated linear attention. arXiv preprint arXiv:2405. 18428, 2024.
Le Zhuo, Ruoyi Du Han Xiao, Yangguang Li, Dongyang Liu, Rongje Huang, Wenze Liu, Lirui Zhao, Fu- Yun Wang. Zhanyu Ma, et al. Lumina-next next: Making lumina-(2x stronger and faster with next-dit. arXiv preprint arXiv:2406. /8583, 2024.
15

---

Published as a conference paper at ICLR 2025
A FULL RELATED WORK
Efficient Diffusion Transformers. The introduction of Diffusion Transformers (DiT) (Peebles & Xie, 2023) marked a significant shift in image generation models, replacing the traditional U-Net ar- chitecture with transformer-basee approach. This innovation paved the way for more efficient and scalable diffusion models. Building upon DIT, PixArt-a (Chen et al., 2024b) extended the concept to text-to-image generation, demonstrating the versatility of transformer-based diffusion models. Sta- ble Diffusion 3 (SD3) (Esser et al., 2024) further advanced the field by proposing the Multi-modal Diffusion Transformer (MM-DiT), which effectively integrates text and image modalities. More recently, Flux (Labs, 2024) showcased the potential of DiT architectures in high-resolution image generation by scaling up 12B parameters. In addition, earlier works like CAN (Cai et al., 2024) and DiG (Zhu et al., 2024) explored linear attention mechanisms in class-condition image genera- tion. Several works are also related to modifying the model configuration, e.g., diffusion without attention (Yan et al., 2024; Teng et al., 2024) and cascade model structures (Pernias al., 2023; Ren et al., 2024; Tian et al., 2024).
Text Encoders in Image Generation. The evolution of text encoders in image generation models has significantly impacted the field's progress. Initially, Latent Diffusion Models (LDM) (Rom- bach et al., 2022) adopted OpenAl's CLIP the text encoder, leveraging its pre-trained visual- linguistic representations. A paradigm shift occurred with the introduction of Imagen (Saharia et al., 2022), which employed the T5-XXL. language model its text encoder, demonstrating superior text understanding and generation capabilities. Subsequently, eDiff-I (Balaji et al., 2022) proposed hybrid approach, ensemble T5-XXL, and CLIP encoders combine their respective strengths in language comprehension and visual-textual alignment. Recent advancements (Ma et al., 2024; Liu al., 2024b; Hu et al., 2024; Liu et al., 2024a), such as Playground v3, have explored the use of decoder-only Large Language Models (LLMs) as text encoders, potentially offering more nuanced text understanding and generation. This trend towards more sophisticated text encoders reflects the ongoing pursuit of improved text-to-image alignment and generation quality in the field.
On Device Deployment. Several studies have explored post-training quantization (PTQ) techniques to optimize diffusion model inference for edge devices. Research in this area has focused on cal- ibration objectives and data acquisition methods. BRECQ (Li et al., 2021) incorporates Fisher in- formation into the objective function. ZeroQ (Cai et al., 2020) uses distillation to generate proxy input images for PTQ. SQuant (Guo et al., 2022) employs random samples with objectives based on Hessian spectrum sensitivity. Recent work such Q-Diffusion (Li et al., 2023) has achieved high-quality generation using only 4-bit weights. In our work, we choose W8A8 to reduce peak memory usage.
B MORE IMPLEMENTATION DETAILS
Rectified-Flow vs. DDPM. In our theoretical analysis, we investigate the reasons behind the fast convergence of flow-matching methods, demonstrating that both 1st flow-matching and EDM mod- els rely on similar formulations. Unlike DDPMs, which use noise prediction, flow-matching and EDM focus on data or velocity prediction, resulting in improved performance and faster conver- gence. This shift from noise prediction to data prediction is particularly critical at T, where noise prediction tends be unstable and leads to cumulative errors. noted by Balaji et al. (2022), attention activation near f = T' grows stronger, highlighting the necessity of accurate predictions at this key moment in the sampling process.
As discussed in Lu (2023), the behavior of diffusion models near ł = T reveals that when t RJ T. the data distribution resembles noise, and noise prediction approaches randomness. The challenge arises because the errors made t = T propagate through all subsequent sampling steps, making crucial for the sampler to be particularly precise near this time step. Based on Tweedie's formula, the gradient of the log density at time Vx, log (x(x), is approximated by:
X₁ agEqor xx,) [Xo]
Vx, x, log qe (x:)) =
(2)
of
16

---

Published as a conference paper at ICLR 2025
When t = T, Xy and x, become conditionally independent, leading to que (xe(Xx)
xx) = (x(x))
Consequently, the noise prediction model's ' optimal solution becomes:
co(x,, o, Vx. log (x,) X2 go (xo) (Xo)
(3)
Since (xa) [xo] is independent of X,, the noise prediction model simplifies to linear function of X,. However, as discussed Section 5.2.1, this additional linearity can result more accumulated emross during sampling, explaining why the original DPM-Solver struggles with guided sampling in such cases.
(xa)
To address this issue and improve stability, DPM-Solver (Lu et al., 2022a) proposes modifying the noise prediction model to more stable parameterization. By subtracting all linear terms inspired by equation 3, the remaining term is proportional to (x)())) [xo]] corresoondigg to the data prediction model. Specifically, when x T. the data prediction model approximates a constant:
Xy (xe, Xr + V x. log (x:)
(oo(x)) [x]].
(4)
a2
Thus, for T, the data prediction model becomes approximately constant, and the discretiza- tion error for integrating this constant is significantly smaller than for the linear noise prediction model. This insight guides our development of an improved Flow-DPM-Solver based on DPM- Solver++ (Lu et al., 2022b), which adapts velocity prediction model Sana data prediction one, enhancing performance for guided sampling.
Flow-based DPM-Solver Algorithm. We present the rectified flow-based DPM-Solver sampling process in Algorithm 1. This modified algorithm incorporates several key changes: ppper-parameter and time-step transformations, as well as model output transformations. These adjustments are highlighted in blue to differentiate them from the original solver.
In addition improvements in FID and CLIP-Score, which are shown in Figure 8 of the main paper, our Flow-DPM-Son Solver- also demonstrates superior convergence speed and stability compared the Flow-Euler sampler. As illustrated in Figure 10, Now-DPM-Solver retains the strengths of the original DPM-Solver, converging in only 10-20 steps to produce stable, high-quality images. By comparison, the Flow-Euler sampler typically requires 30-50 steps to reach a stable result.
Algorithm
(Modified from DPM-Solver++)
Require: initial value zr, time steps {t.) 1M data prediction model zy, velocity prediction model vs, time- step shift factor s
1: Denote h; = Xx; At for = 1, 1....,M M 5.04,
2: ãz. 14(x-1))-n
= =1 0z, Z2, oc, (o tx)
b Hyper paraerrp and Time-step transformation
3: ze xo(āc,,te)
Model output transformation
4: Eto t- FT. Initialize empty buffer Q
5: Quutier t- ==tootttt))
6: Tt, + Tel $io ou, (e-hi hi - rrs (ito
7: Quatter +- 2a(Fr,, (1)
8: for to M do
Ti b
10:
D.
1 2
(Tt; 1: (1111) 1)- 34 zo(l 2-4)2
Ds; + dc, :u.-1 au; (6 - 1) D.
11:
5:,-1
12:
if < M then
Quatier (o
14: end if
15:
end for
16:
return Se
Multi-Caption Auto-labeling Pipeline. In Figure 12, we present the results of our CLIe-Score- based multi-caption auto-labeling pipeline, where each image is paired with its original prompt and captions generated by different powerful VLMs. These captions complement each other, enhanc- ing semantic alignment through their variations.
17

---

Published as a conference paper at ICLR 2025
Flow-Euler
Flow-DPM-Solvee
steps
steps
steps
14 sleps
20 steps
Figure 10: Visual comparison of Flow-Fuler Sampler with 50 steps and Flow-DPM-Sollver with 5/8/14/20 steps.
43 d/
5b
40
44
42
44 43
63
45 43
45
6.0
060 10250
1846
46
45.
44
41
42
08:
4b
Original images
DC-AE-F32C32
SDXL-VAE-FBCA
Figure 11: Comparison between the original images, our DC-AE-F32C32 (Chen et al., 2024c) and SDXL's VAE-F8C4.
18

---

Published as a conference paper at ICLR 2025
HAPPY VALENTINE
Caption_origine (ClipScore: 25.67)
view the written HAPPY VALENTINE" tart chocolate cake, black wood background
Caption_VILA VILA-13B (ClipScore: 26.33)
The image captures a delightful scene of Valentine's Day celebration. At the center of the image is a round chocolate cake, rich and inviting. The cake is adomed with red heart on top, symbolizing love and affection. The words "Happy Valentine's are provides a
eleganily written red icing rustic charm to the satting. Scattered around the cake are red heart-shaped confett, further onhanding the Valentine's theme. In the background, there's a hint of bouquet of red roses, classic symbol of love and romence. The roses are slightly blurred, drawing focus
cake, adding the festive almusphere.
cake is placed a wooden table,
foreground. Overall, the image beautifuly encapsulates the spirit Valentine's Day, with focus love, affection, and celebration.
Caption _aption_VILA-3B
VILA-3B (ClipScore: 25.46)
image captures delightful scane Valentine's Day celebration. surface adomed with a gloesy chocolate ganache. The cake further embelliheed with a scattering of chocolate shavings and e heart-shaped topper, adding to festive appeal. The cake placed wooden surface, which provides rustic contrast the rich, dark color of the chocolate. The background is dark, alloving the cake stand out prominently. Adding the Valentine's es Day theme, red heart- shaped sprinkles are scatlered around the cake, their vibrant color popping against the dark backdrop. The cake is positioned slighlly off-center, drawing the viewers attention oelebration, with the cake as the star of the show. The image does not contain any discernile text. The reistive positions of the objects and their delailed descriptions provide a comprehensive underslanding of the image content.
center the frame a round chocolate cake,
The overall compostioon of the image suggests carefully arranged
IntiionnnternVL222228
(ClipScore: 25.57)
The image feutures a buauuifully decorated chocolate cake placed on a dark woaden surface. The cake is richly covered in layer of dark chocolate ganache. giving smooth and glossy appearance. The top of the cake adomed with piped chocolate swiris remance.
heart shaped decorations, adding louch of elegance VALENTINE" are written in bold, rad kangg creating a striking contrast against the dark chocolate background. Surrounding the eake, there are additional heart- shaped decorations red, some of which placed on the wooden surface, while others are overall setting is warm inviling, indulgent nature of the chocolate cake. The use of red hearts and the "HAPPY VALENTINE" message clearly indicate that this cake is intended for Valentine's Day celebration
center the cake, words "HAPPY
scatlered around
ruslic wooden background complements rich,
Caption Intern (ClipScore: 26.37)
The image features beautifully decorated chocolate cake with a heartfelt Valentine's Day message. The cake is round and sits on dark, rustic wooden surface. The cake covered in a rich, dark chocolate ganache, and is adorned with generous amount of chocolate shavings crushed nuls, giving luxurious and indulgent appeerence.
the cake, worde "HAPPY VALENTINE" are written in bold, red icing. creating a striking contrast against the dark chocolate. The icing is piped in a slighlly uneven, handwrilten style, adding personal touch the message. Surrounding the cake are several red heart-shaped decorations, some which are whole to the romantic and festive theme of the cake. The cake is placed on a dark, round cake board, which complements the rustic wooden background. The overall presentation elegant feetive, making it perfect for a Valentine's Day oslebration
are broken smaller pieces, scattered around the cake. These hearts add
Figure 12: Illustration of re-caption of an image with multiple VLMs.
Discussion: CLIP Text Encoder's Input Token Length limitation. Sana primarily uses the CLIP model to score the caption as the selective strategy during training. The 77-token limitation of the original CLIP model may seem restrictive. However, after extensive observation, we found that the most valuable content typically appears in the first half of the caption, such "The image captures Scontent of images..." that is. 77-token length is typically sufficient to encapsulate essential se- mantic information for purposes. Consequently, we believe that the CLIP-Score based 77
19

---

Published as a conference paper at ICLR 2025
tokens still offers valuable insights and relevance for assessing caption quality in our context. In the future, we will attempt to ane-tune a larger context-length CLIP model or use more advanced VLM to score the text-image pairs.
Triton Acceleration Training/Inference Detail. This section describes how to accelerate the train- ing inference with kernel fusion using Triton. Specifically, for the forward pass, the ReLU activa- tions are fused to the end of QKV projection, the precision conversions and padding operations are fused the start of KV and QKV multiplications, and the divisions are fused to the end of QKV multiplication. For the backward pass, the divisions are fused to the end of the output projection, and the precision conversions and ReLU activations are fused to the end of KV and QKV multiplications.
2K/4K Fine-tuning, We reintroduce Positional Encoding (PE) to improve performance during the fine-tuning of 2K models on top of IK models. For fine-tuning 4K models based on 2K models, we apply the same PE interpolation strategy used in Chen et al. (2024a). The training process with the addition of positional encoding (PE) converges remarkably quickly, typically within just 10K iterations.
Positional Embedding. As previously discussed, positional embeddings can be effectively re- implemented in just a few training steps. Specifically, our experiments demonstrate that the model rapidly adapts to the absolute positional embedding, utilizing the sin-cos formulation, through a brief fine-tuning process. This adaptation occurs within merely 10K training steps, using total batch size of 1024.
Definition of Gemma2-2B-IT. Gemma2 (Team et al., 2024b) is family of lightweight, state-of-the- art open models from Google, built from the same research and technology used to create the Gemini models. The specific Gemma use in Sana is the Gemma-2-2B-1 IT (Team, 2024) version. "IT"" here represents the model is Instruct-Tuune Tuned for better prompt following and chat ability, experiments show that instruction-followin tuned versions superior non-Instruction-1 Tunne models, as shown in Table 9.
C MORE RESULTS
Comparison Between Autoencoders Figure 11 illustrates the visual differences between the orig- inal images and the reconstructions generated by two distinct models: our DC-AE-F32C32 (Chen et al., 2024c) and SDXL's VAE-P8C4, Both models deliver reconstructions nearly indistinguishable from the original images, showcasing their powerful encoding and decoding capabilities.
Ablation on Sana Blocks. Table 13 describes how different block designs affect performance. Directly switching from DIT's self-attention to linear attention will result in FID and Clip Score performance loss, but adding Mix-FFN can compensate for the performance loss. Adding triton kernel fusion can speed up training/inference without negatively impacting performance.
Compare SANA'S CHI with LiDiT's SHI. LiDIT (Ma et al., 2024) is the first work that attempts to promote the reasoning ability of LLM through relatively simple human instruction Additionally, LiDiT is necessary to fine-tune the last few layers of the LLM model, which increases the training cost and pipeline complexity. In contrast, our CHI designs more complex prompt templates and utilizes in-context-leaming techniques, which can fully stimulate LLM's high-order reasoning ability, extract better text embedding, and thus improve semantic alignment capabilities.
Complex Human Instruction Analysis. To observe the effectiveness of CHI, we input the user prompt with/without CHI into Gemma- We believe strong positive correlation ex- ists between LLM output and text embedding quality. As shown in Figure 13, without CHI, although Gemma-2 can understand the mean- ing of the input, the output is conversational and does not focus understanding the user prompt itself. After adding CHI, Gemma-2's ss output is better focused on understanding and enhanc- ing the details of the user prompt. Besides, as illustrated in Table 10, fine-tune both CHI and SHI
Table 10: Comparison of CHI vs. SHI perfor- mance on GenEval metric during fine-tuning.
Training Step
CHI
SHI
+2K steps
0.636
0.617
+5K steps
0.642
0.626
20

---

Published as a conference paper at ICLR 2025
settings from pre-trained model without human instructions. After 2K and 5K fine-tuning steps, CHI consistently shows better GenEval performance.
Detailed Results on DPG-Bench, GenEval and ImageReward. As extension of Table. in the main paper, we show all the metric details of GenEval, DPG-Bench, and ImageReward for reference in Table 11 and Table 12 respectively.
Finding: Zero-shot Language Transfer Ability. As shown in Figure 14, we were surprised to find that by using Gemma-2 the text encoder and Chinese/Emoji expressions text prompts; our Sana can also understand and generate corresponding images. Note that we filter out all prompts other than English during training, so Gemma-2 brings the zero-shot generation capability of Chinese/B- moji.
w/o complex human Instruction
w/ complex human Instruction
Gemma Input
Gemma Input
A man is walking
<Complex Human Instruction man is
walking
Gemma Output
Gemma Output
That's great start! To make this interesting, tell more the man walk. For example:
man wom leather jacket walks briskly down a cobolestone street, his head downcast, his silhouette framed by the setting sun. He wears a faded blue scarf leather briefcase. cobolestones are worn smooth by time and weather, and the lined quaint, brightly painted buildings.
* ""Where walking?" bustling city quiet
forest path?
* ""What canying?" brisfcase? basket? dog?
* "WWat his mood?" Happy and carefrec? Stressed and hurrid?? Lost thought?
"WWhat the weather like?" Sunny and wem?? Rainy cold? Snowy crisp?
The more details you give me, the more vivid and engaaging the story be!
X
Complex Human Instruction
Given user promot, generate 'Enhenced prompt" provides detailed descriptions image generation. Evaluate the level detail the user prompt:
the is simple, focus adding specifics about colors, shapes, textures, and relationships create vivid and concrete scenes. - the prompt is already datailed, refine and enhance the existing details slightly without overcomplicating.
Here are examples of how to transform refine prompts:
Prompt: cat sleeping t> Enhanced: small, fluffy white
up round shape, sleeping peacefully on
warm sunny windowsill, surounded by pots of blcoming red flowers.
featuring glowing lamps, diverse crowd of people in colorful clothing, and double-decker bus passing by lowering glass skyscrapers.
Prompt: busy city street Enhanced: bustling city
Please generate only the enhanced description for the prompt below and avoid including any additional commeniary or evalustions:
User Prompt:
Figure 13: Illustration of Gemma-2's s output with/without complex human instruction, and the full prompt of our complex human instruction.
Detailed Speed Comparison of Text Encoder. In Table 14, we compare the latency and parameters for various T5 models alongside the Gemma models. Notably, the Gemma-2B model exhibits similar latency to Ta-lerge while significantly increasing the model size. This enhancement in model size a key factor achieving improved capabilities with greater efficiency.
Detailed Speed Comparison of Diffusion Model. In Table 15, we compare the throughput and latency of mainstream DiT-based text-to-image method and our model in detail and test them at resolutions of 512, 1024, 2048, and 4096, respectively. Our Sana is far ahead of other methods at different resolutions. As the resolution increases, the efficiency advantage our Sana becomes more significant.
More Visualization Images. As shown in Figure 15, we can see that 4K images can directly generate details than 1k images. In Figure 16, show more images generated by model with various prompts. We also provide video mp4 demo the supplementary material (zip file) to show that Sana is deployed a laptop.
21

---

Published as a conference paper at ICLR 2025
Table 11: Comparison of SOTA methods on GenEval with details. The table includes different metrics such overall performance, single object, two objects, counting, colors, position, and color attribution.
Objects Single
Model
Params (B)
Overall
Counting
Colors
Position
Color Attribution
Two
512 x512 resolution
PixArt-a PixArt-E
0.6
0.48
0.98
0.50
0.44
0.80
0.08
0.07
0.52
0.98
0.59
0.80
0.10
0.15
Sana-0.6B (Ours)
0.64
0.99
0.71
0.63
0.91
0.16
0.42
Sana-L.6B (Ours)
0.6
0,66
0.99
0.79
0.63
0.88
0.18
0.47
1024 x1024 resolution
L.MINA NextNA-Next&ee (Zhuo et al,, 2024)
0.46
0.92
11.46
01.48
0.70
0.09
0.13
SDXL (Podell al., 2023)
2.6
0.55
0.98
0.74
0.39
0.85
0.15
0.23
PlayGrundy2.55 (1.
2024a)
0.56
0.84
Hunyuan-DiT (Li al., 2024c)
1.5
0.63
0.97
0.77
0.71
0.88
0.13
0.30
DALLE3 (OpenAl, 2023)
0.67
0.96
0.87
0.47
0.83
0.43
0.45
SID3-medium (Fsser al., 2024)
FLUX-dev (Labs, 2024)
12.0
0.67
0.99
0.81
0.79
0.74
0,20
0.47
FLUX schnell (Labs, 2024)
12.0
0.71
0.99
0.92
0.73
0.78
0.28
0.54
Sana-0.6B (Ours)
0.64
0.99
0.76
0.88
0.18
0.39
Sana-1.6B (Ours)
1.6
0.66
0.99
0.77
0.62
0.88
0.21
0.47
Table 12: Comparison of SOTA methods on DPG-Bench and ImageReward with details. The table includes different metries such as overall performance, entity, attribute, relation, and other categories.
Purams (R)
Glohal
Emity
Autrhutee
ImageReward
512 x512 resolution
PixArt-a (Chen et al., 2024h)
0.6
71.6
81.7
80.1
80.4
81.7
76.5
0.92
PixArt-E (Chen et al., 2024a)
0.6
79.5
87.5
87.1
86.5
84.0
86.1
0.97
Sana-0.6B (Ours)
0.6
84.3
82.6
90.0
88.6
90.1
91.9
0.93
Sana-1.6 (Ours)
0.6
85,5
90.3
91.2
89.0
88.9
92.0
1.04
resolution
LUMINA-Nest (Zhao al., 2024)
2.0
74,6
82.8
888.7
86.4
80.5
81.8
SDXL (Podell al., 2023)
74.7
83.3
82.4
80.9
86.8
80.4
0.69
PlayGroundv2.5 2.5 (Li at, 2024a)
2.6
75.5
83.1
82.6
81.2
84,1
83.5
1.09
Hunyuan-DiT (Li al. 2024c)
78.9
84.6
80.6
88.0
74.4
86.4
PixArt-E (Chen et al., 2024a)
0.6
80.5
86.9
82.9
88.0 9
86.6
87.7
0.87
(OpenAl, 2023)
SD3-medium (Esser et 2024)
2.0
84.1
87.9
91.0
88.5 8
80.7
88.7
0.86
dev (Lahs, 2024)
FLUX schnell (Labs, 2024)
12.0
84.8
91.2
91.3
89.7
86.5
87.0
0.91
Sana-0.6B (Ours)
83.0
89.5
90.1
Sama-1.6B (Ours)
1.6
84,8
86.0
91.5
88.9
91.9
90.7
0.99
Wearing flynng on
Figure 14: Visualization of zero-shot language transfer ability. Our Sana only has English prompts during training but can understand Chinese/Emoji during inference. This benefits from the general- ization brought by the powerful pre-training of Gemma-2.
22

---

Published as a conference paper at ICLR 2025
Table 13: Performance of Sana block design space. We train all the models with the same training setting with 52K iterations.
Blocks
AE
Res.
FID
CLIP t
FutlAttn & FFN
F8C4P2
256
18.7
24.9
Linear
F8C4P2
256
21.5
23.3
+ MixFFN
F8C4P2
256
18.9
24.8
Kernel Fusion
F8C4P2
256
18.8
24.8
Minnar++GUMBBonO22
F32C32P1
512
6.4
27.4
Kernel Fusion
F32C32P1
512
27.4
Table 14: Comparison of various T5 models and Gemma models based speed and parameters.
The sequence length (Seq Len) is the number of text tokens.
Text Encoder
Batch Size
Seq Len
Latency
Params (M)
TS-XXL
4762
TS-XL
0.5
1224
Tarle
0.2
341
300
TS-base
0.1
110
T5-small
35
Gemma-2b
2506
Gemma-2-26
2614
D DISCUSSION OF POTENTIAL MISUSE OF SANA
Misusing generative Al models to generate NSFW content is challenging issue for the community. To enhance safety, we have equipped SANA together with safety check model (e.g. ShieldGemma- 2B (Zeng et al., 2024)). Specifically, the user prompt will first be sent to the safety check model to determine whether contains (nFW safe for work) content. If the user prompt does not contain NSFW, it will continue be sent to Sana to generate image. If the user prompt contains NSFW content, the request will be rejected. After extensive testing, we found that ShieldGemma perfectly filter out NSFW prompts entered by users under strict thresholds and our pipeline will not create harmful Al-generated content.
Table 15: Comparison of throughput and latency under different resolutions. All models tested on an A100 GPU with FP16 precision.
Methods
Speedup
Throughput(ss)
Latency(ms)
Methods
Speedup
Throughput(/s)
Latency(ms)
512x512 Resolution
1024x 1024 Resolution
SD3
7.6x
1.14
1.4
SD3
7.0x
0.28
FLUX-schnell
10.5x
1.58
0.7
FLUX-schnel
12.5x
0.50
2.1
dev
7.9
FLUX
PixArt-Y
10.3x
1.54
1.2
PixArt-Y:
10.0x
0.40
2.7
Hin
1.3x
0.20
5.1
HonyuanDiT
1.2x
Sana-0.6B
44.5x
6.67
0.8
Sama-0.6B
43.0x
1.72
0.9
Sana-1.6B
25.6%
3.84
0.6
Sama-1.6B
25.2x
2048x2048 Resolation
4096 4096 Resolution
22
SD3
FUUX-schnell
11.2x
0.09
10,5
FLUX-schnell
13.0x
0.013
76
FLUX-dev
1.0x
0.008
117
FLUX-dev
1.xx
0.001
PixArt- Σ
7.5%
0.06
18.1
PixArt-E
5.0x
0.005
186
HinyuanDiT
1.2x
0.01
96
HunyuanDiT
1.0x
0.001
861
Sana-0.6B
0.43
2.5
Sana-0.6B
Sana-1.6B
31.2x
0.25
4.1
Sama-1.6B
66.0x
0,066
5.9
23

---

Published as a conference paper at ICLR 2025
4096x4096
1024x1024
Figure 15: Comparison of 4K and IK resolution images. We can see that the 4K image contains more details.
24

---

Published as a conference paper at ICLR 2025
Res:3168x485
pecTar NoTe looking thrugh
ther Arma Rasumoolaay
Cereste comers
U'ART DOREET DR 1OO an
Astronaut ostar muled detelled,
carcuar
Help
Ond SOUR FOR ORIGWWG
Vellow brock wall
Arror balloon the shape of heart
A porra Fumaa growing colotut Fowees from herhak Hypernolctio
Figure 16: More samples generated from Sana.
25


Reviews:
[{"review_id": "sShz8YYxql", "reviewer": "Reviewer_9t32", "summary": "The paper presents Sana, a cutting-edge text-to-image synthesis framework that can generate high-resolution images up to 4096×4096. Here are the key highlights:\nUnlike conventional autoencoders that compress images by a factor of 8×, Sana's autoencoder achieves a remarkable 32× compression, drastically reducing the number of latent tokens.\n\nSana innovates by replacing the standard attention mechanisms in the Diffusion Transformer (DiT) with linear attention. This change enhances efficiency at high resolutions without sacrificing image quality.\n\nDecoder-Only Text Encoder: Instead of relying on T5, Sana uses a modern, decoder-only small language model (LLM) as its text encoder. This model, combined with sophisticated human instructions and in-context learning, significantly improves text-image alignment.\n\nSana introduces Flow-DPM-Solver to cut down on the number of sampling steps and employs efficient caption labeling and selection techniques to speed up training convergence.\n\nThese advancements allow Sana to produce high-quality images at a significantly faster pace than existing models.", "questions": "1. The Gemma2-2B-IT model in Table 9 needs to be explained, as not all readers have a solid background in LLMs.\n\n2. Further comparisons between Gemma2 and T5-XXL are needed, such as showing HPSv2 metrics on complex prompt benchmarks. The current FID scores in Table 9 are insufficient to highlight the true advantages of Gemma2.\n\n3. Why does SANA perform significantly better than LUMINA-Next, even though both use the Gemma-2B model?\n\n4. I believe some relevant works should be cited or compared, such as high-resolution generation work: UltraPixel: Advancing Ultra-High-Resolution Image Synthesis to New Peaks (NeurIPS 2024) and efficient generation work: Stable Cascade.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This work demonstrates its originality through several innovative contributions, including the Deep Compression Autoencoder, Linear DiT, and impressive 4K generation ability. These innovations enhance the quality and efficiency of high-resolution image generation while reducing computational requirements. The paper is well-written and clearly structured, with detailed experiments and results that validate the effectiveness of the proposed methods. Additionally, the significance of the work lies in its practical applications and the removal of limitations from prior models, making high-resolution image synthesis more accessible and scalable.\nOverall, I find this work to be a breakthrough for diffusion text-to-image models.", "weaknesses": "1. The Gemma2-2B-IT model in Table 9 needs to be explained, as not all readers have a solid background in LLMs.\n\n2. Further comparisons between Gemma2 and T5-XXL are needed, such as showing HPSv2 metrics on complex prompt benchmarks. The current FID scores in Table 9 are insufficient to highlight the true advantages of Gemma2.\n\n3. Why does SANA perform significantly better than LUMINA-Next, even though both use the Gemma-2B model?\n\n4. I believe some relevant works should be cited or compared, such as high-resolution generation work: UltraPixel: Advancing Ultra-High-Resolution Image Synthesis to New Peaks (NeurIPS 2024) and efficient generation work: Stable Cascade.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pky2j6JqOS", "reviewer": "Reviewer_mBM3", "summary": "The paper \"SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers\" introduces SANA, a high-efficiency text-to-image generation framework capable of producing high-resolution images up to 4096×4096 pixels. SANA employs multiple innovations, including a deep compression autoencoder (compressing images 32× for latency reduction), linearized Diffusion Transformer (DiT) attention to improve high-resolution efficiency, and the use of a decoder-only LLM as a text encoder with \"Complex Human Instruction\" for enhanced image-text alignment. The model also implements a Flow-DPM-Solver to reduce sampling steps, achieving competitive quality at 100× faster speeds compared to state-of-the-art diffusion models. SANA's improvements are demonstrated on high-resolution benchmarks and support deployment on a laptop GPU.", "questions": "1. SANA uses the Gemma-2 LLM with Complex Human Instruction (CHI) to improve text-image alignment. While similar works like Playground v3 uses Llama3-8B, this paper only compared T5 with Gemma, do we know if larger LLMs can further improve the generation quality?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovative Efficiency Strategies: SANA’s integration of linear attention, a high-compression autoencoder, and a unique sampling solver enables fast high-resolution generation, which holds practical value for many applications.\n2. Effective Use of Large Language Model (LLM) for Text Encoding: SANA’s implementation of a decoder-only LLM with CHI to refine prompts improves text-to-image alignment, enhancing quality without incurring high latency.", "weaknesses": "Limited Ablation in Model Design: While SANA combines existing methods effectively, many techniques—such as linear attention, high-compression auto-encoders, and Flow-based solvers—are iterative upon recent advancements in diffusion transformers. Explicit comparisons with recent models like PixArt-Σ or Playground v3, which also incorporate high-efficiency strategies or decoder-only LLMs, with component by component comparisons would better attribute SANA’s unique contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J23nfJ5GJR", "reviewer": "Reviewer_S4E6", "summary": "This paper proposes a text-to-image framework that can efficiently generate images up to 4096 x 4096 resolution. Comparing to existing works, the paper proposes four designs to facilitate fast speed, high quality text-to-image generation: (1) An auto encoder that down samples to input image by 32 times. (2) Replacing the quadratic self attention with linear attention. (3) Applies decoder-only LLM as text encoder. (4) Adopts efficient caption labelling and Flow-DPM-Solver for training and inference. Experimental results show that the proposed model achieves better or comparable results in terms of image quality, text image alignment comparing to existing methods, while having much higher throughput and lower latency", "questions": "Please see weaknesses section", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well motivated, as developing method that facilitates high quality, high resolution text-to-image generation in resource limited scenarios is of great application value.\n2. The paper addresses the problem from several aspects, including auto encoder design, light weight self attention module, sampling method, and better caption labeling procedure. \n3. The experimental results on resolutions of 512 x 512, 1024 x 1024 show that the proposed indeed achieve better or comparable performance comparing to previous state of the art methods, while having much higher throughput and lower latency. \n4. The ablation experiments of different design components are sufficient and convincing.", "weaknesses": "1. The design of linear attention block needs to be explained in more detail. For example, the original self attention helps modelling long range dependency between tokens, here in the linear attention block, how is the dependency / contextual relation between tokens modeled through the new operations ?\n2. The paper claims that the proposed method is able to generate images up to 4096 x 4096 resolution, however, in terms of quality comparison with existing methods, there are only ones with resolution of 512, 1024. For 4096 resolution, only the speed comparison is reported (Table 14), it would be more convincing to include FID and other image-text alignment metrics comparison for 4096 resolution as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vpI9LzxZR2", "reviewer": "Reviewer_Dipw", "summary": "The paper introduces Sana, a novel framework for efficient text-to-image generation, capable of producing 4K images while maintaining strong text-image alignment at remarkable speed. Sana’s core innovations include: a deep compression autoencoder, a Linear DiT architecture to enhance efficiency and a LLM-based text encoder. The model achieves competitive results compared to much larger models like Flux-12B, boasting significantly lower computational requirements.", "questions": "I do not have any further question", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Efficiency is a really big selling point. With a very high compression rate of deep autoencoder and linear DiT, SANA can generate high-quality images even on a laptop is very impressive. \n\n2. The fact that using a small but good LLM can outperform a big but bad text encoder is another interesting point. This boosts text-image alignment a lot when used with their proposed complex human instruction (CHI) pipeline.\n\n3. They provide extensive ablation experiments on each components and measure their impact on efficiency and perform of the model, which is very helpful for future work to build upon.", "weaknesses": "1. Although the results are impressive, the contribution of this paper on the technical side is somewhat limited. Their three main components have been already explored in the literature, deep autoencoder in [1], linear DiT in [2] and using LLM as text encoder in [3, 4]. So combining them is not really a significant novel idea.\n\n2. In section 2.3, the author do not fully explain their CHI pipeline, is it same as [5] or completely different. If it is similar then this even reduces the novelty of the work further.\n\nReference:\n\n[1] Chen et. al. Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models. 2024\n\n[2] Liu et. al. LinFusion: 1 GPU, 1 Minute, 16K Image. 2024\n\n[3] Hu et.al. ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment. 2024\n\n[4] Liu et.al. LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation. 2024\n\n[5] Ma et. al. Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models. 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)