SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer
Authors: Enze Xie
Junsong Chen
Yuyang Zhao
Jincheng YU
Ligeng Zhu
Yujun Lin
Zhekai Zhang
Muyang Li
Junyu Chen
Han Cai
Bingchen Liu
Daquan Zhou
Song Han
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Model
Inference Scaling
Linear Attention
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer

Abstract: This paper presents SANA-1.5, a linear Diffusion Transformer for efficient scaling in text-to-image generation. Building upon SANA-1.0, we introduce three key innovations: (1) Efficient Training Scaling: A depth-growth paradigm that enables scaling from 1.6B to 4.8B parameters with significantly reduced computational resources, combined with a memory-efficient 8-bit optimizer.   (2) Model Depth Pruning: A block importance analysis technique for efficient model compression to arbitrary sizes with minimal quality loss.  (3) Inference-time Scaling: A repeated sampling strategy that trades computation for model capacity, enabling smaller models to match larger model quality at inference time. Through these strategies, SANA-1.5 achieves a text-image alignment score of 0.72 on GenEval, which can be further improved to 0.80 through inference scaling, establishing a new SoTA on GenEval benchmark. These innovations enable efficient model scaling across different compute budgets while maintaining high quality, making high-quality image generation more accessible.

SANA 1.5: Efficient Scaling of Traening-Time and inference-Time Compute in Linear Diffusion Transformer
Enze Xie
Junsong Chen" Yuyang Zhao Jincheng Yut Ligeng Zhu Yujun Lin Zhekai Zhang?
Muyang Li Junyu Chen Han Cai
Bingchen Liu" Daquan Zhou Song Han 12
gitnbb.on//Vlabs/Sana
Abstract
instance, recent industry models have grown from PixArt's 0.6B parameters (Chen et al., 2024b) to 24B in Playground v3 (Liu et al., 2024a), resulting prohibitive training and inference costs for most practitioners.
This paper presents SANA-1.5, a linear Diffusion Transformer for efficient scaling in text-to-image generation. Building upon SANA-1.0, we intro- duce three key innovations: (1) Efficient Training Scaling: A depth-growen paradigm that enables scaling from 1.6B to 4.8B parameters with signif- icantly reduced computational resources, com- bined with a memory-efficient 8-bit optimizer. (2) Model Depth Pruning: A block importance analysis technique for efficient model compres- sion arbitrary sizes with minimal quality loss. (3) Inference-time Scaling: A repeated sampling strategy that trades computation for model ca- pacity, enabling smaller models to match larger model quality at inference time. Through these strategies, SANA-1.5 achieves a text-image align- ment score of 0.72 on GenEval, which can be further improved to 0.80 through inference scal- ing, establishing new SoTA on GenEval bench- mark. These innovations enable efficient model scaling across different compute budgets while maintaining high quality, making high-quality im- age generation more accessible. Our code and pre-trained models will be released.
In contrast, SANA-1.0 (Xie al., 2024) introduced effi- cient linear diffusion transformer that achieved competitive performance while significantly reducing computational re- quirements. Building upon this foundation, this work ex- plores two fundamental questions: i) how is the scalability of linear diffusion transformer; ii) how can we scale up large linear DiT and reduce the training cost?
This paper presents SANA-1.5, which introduces three key innovations for efficient model scaling in both training and inference time. First, we propose an efficient model growth strategy that enables scaling SANA from 1.6B (20 blocks) to 4.8B parameters (60 blocks) while reusing the knowledge learned in the smaller model. Unlike traditional scaling approaches that train large models from scratch, our method initializes additional blocks strategically, allowing the large model to retain the prior knowledge of the small model. This approach reduces training time by 60% compared to training from scratch, as shown in Figure 2.
Second, we introduce a model depth pruning technique that enables efficient model compression. By analyzing block importance through input-output similarity patterns in dif- fusion transformers, we prune less important blocks and quickly recover the model quality through fine-tuning (e.g.. 5 minutes on single GPU). Our grow-then-pe prune approach effectively compresses the 60-block model to various con- figurations (40/30/20 blocks) while maintaining competitive quality, providing an efficient path for flexible model de- ployment across different compute budgets.
1. Introduction
Text-to-image diffusion models have demonstrated remark- able progress in the past year, with a clear trend towards larger model sizes. Although scaling up the size of the model has proven effective in improving the quality of gen- eration, it comes with substantial computational costs. For
Third, we propose an inference-ting time scaling strategy for SANA, which enables smaller models to match larger model quality through compute rather than parameter scaling. By generating multiple samples and leveraging a VLM-based selection mechanism, our approach improves the GenEval score from 0.81 to 0.96. This improvement follows similar log-linear scaling pattern observed in LLMs (Brown et al.,
Equal contribution 'NVIDIA *MIT "Tsinghua University "Playground 'Peking University. Correspondence to: Enze Xie <enzex @ nvidia.com>. Song Han <songhan@ mit.edu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SANA 1.5: Efficient Scaling of Training-Time Time and Inference-Tinee Compute in Linear Diffusion Transformer
8 B-bit CAME Optimizer
Modell GenEval 0.96
User Prompt: photo of stop sign and toaster.
Linear Dit Blook
Linear DIT
Unner DIT
Linear DIT
Unpatchity
Prediction
Latrnns
Block
Blook
Infarance Scaling
trained B
New
Forward
Model Growth.
Puredto Morke eMl
STOP
4.88 Model Gentivll 0.87
Line 311 daokk
Liuw 3i1 dacck
Liu 3IT Black
TYww 311 Blick
TYe 317 Blick
Towrr 317 Blick
TaE 317 Banck
GPU
Moddl Grouth
Verifier IRQUP,, Vang
LD Brch
Pre-trained Blockax(11 M)
Mobilc
samppaa
8, M.M.M Me namton oi Mock
1.08 Model GenEval 0.66
Rtocke
(b) Model Depth Pruning.
(c) Inference Scaling.
(d) Performance Boost.
Figure 1. The overall framework SA5A-1.5. (a) Model Growth: We initialize the large model with pre-trained small model, and train the large model with 8-bit CAME, which largely accelerates the training convergence and reduces VRAM requirements. (b) Model Pruning: After training the large model, smaller models of different sizes are pruned and mnnet for different situations. (c-d) Inference Scaling: We repeat generating many samples with SANA and use VLM verifier select the best-of-N samples, which largely boosts the quality.
2024), demonstrating that computational resources can be effectively traded for model capacity, challenging the con- ventional wisdom that larger models are always necessary for better quality.
Our extensive experiments demonstrate that SANA-1.5 achieves 2.5x faster training convergence than the tra- ditional approach (i.e., scale up and train from scratch). Through our training scaling strategy, we improve the GenEval score from 0.66 0.81, which can be further boosted to 0.96 with inference scaling, establishing new satte-of-the-ard art on the GenEval benchmark. More impor- tantly, our findings reveal a fundamental insight: efficient scaling can be achieved through better optimization trajec- tories rather than simply increasing model capacity. By leveraging knowledge from smaller models and carefully designing the growth-pruning process, we show that the path better quailty does not always require larger models.
rThese three technical contributions model growth, model depth pruning and inference scaling- form coherent frame- work for efficient model scaling. The model growth strategy first explores a larger optimization space, discovering bet- ter feature representations. The model depth pruning then identifies and preserves these essential features, enabling efficient deployment. Meanwhile, inference-tine scaling provides a complementary perspective. When model capac- ity is constrained, we can utilize extra inference-time com- putational resources to achieve similar even better results than larger models. Together, these approaches demonstrate that thoughtful optimization strategies can outperform sim- ple parameter scaling, providing multiple paths to achieve high quality under different resource constraints.
In summary. 1.5 introduces model scaling in text-to-image generation. Rather than following the conventional paradigm "bigger is better", we demonstrate that the growth and pruning of strategic models, combined with the inference-time scaling, can achieve com- parable or better results with significantly reduced training resources. This approach not only advances the theoretical understanding of model scaling but also makes high-quality text-to tot-to--mmae generation more accessible to the broader re- search community and practical applications.
perspective
To enable efficient training and fine-tuning large models, optimizer CAME-Sbir by extending CAME (Luo et al., 2023) with block-wiss 8-bit quantization (Dettmers et al., 2021). CAME-8bit reduces memory usage by ~8x x compared to AdamW. W- 32bit (Loshchilov, 2017) while maintaining training stability. This optimization proves effective not only in pre-training but particularly valuable for single-GPU fine-tuning sce- narios, enabling researchers to fine-tune SANA-4.8B consumer GPUs like RTX 4090, making large model fine- tuning more accessible to the open-source community.
implement memory-
2. Methods
2.1. Overview
Increasingly larger models have dominated text-to-image generation, but SANA-1.5 introduces different paradigm
2

---

SANA 1.5: Efficient Scaling of Training- Training-Time and Inference- Timee Compute in Linear Diffusion Transformer
0.70 Genével
layers, with special handling of key components. Formally, for i-th layer index:
SANA-10088 GenEval
0.6
if < N if N
0.5
0,
((
GenEvaal
0.4
saving 60% training cast
where (0, o²) is the normal distribution with standard deviation as
0.3
(2) Cyclic Replication Init, which repeats the pre-trained
SANA-4.03 4.03 from Scretch
SANA-4.88 8B Model Growth
layers periodically. For i-th layer in the expanded model:
0.1
20k
40k
100k
120k
140k
150k
Training Steps
02
P mod N
Figure 2. Training efficiency comparison of different initial- ization strategies. Training curves on GenEval benchmark for SANA-1.5 4.8B using model growth strategy vs training from scratch. Our model growth approach achieves the same perfor- mance (0.70 GenEval) with 60% fewer training steps, significantly improving training efficiency.
(3) Block Replication Init, which extends each pre-trained layer into consecutive layers. Given expansion ratio r M/N, for pre-trained i-th layer, it initializes r consecutive layers in the expanded model:
Oxts= = 0, for j E r0,...,--11,7 E [0, - 1].
that achieves efficient scaling through three complementary strategies. Rather than training large models from scratch, we first expand a base model with N transformer layers to N + MM layers (where N = 20, M = 40 in our exper- iments) while preserving its learned knowledge. During inference, we employ two complementary approaches for efficient deployment: (1) model depth pruning mechanism that identifies and preserves essential transformer blocks, abling flexible model configurations with small fine-tuning cost, and (2) an inference scaling strategy that trades com- putation for model capacity through repeat sampling and VLM-guided selection. Mcanwhile, our memory memory-efficient efffccient CAME-&bii optimizer makes it possible to fine-tune billion- scale models on a single consumer GPU. Figure illustrates how these components work together to achieve efficient scaling in different computational budgets.
where represents the expansion ratio (e.g., r = 3 when expanding from 20 to 60 layers), 0; denotes the parame- ters of layer in the expanded model, 6.te represents the parameters from the pre-trained model.
Stability Enhancement To ensure training stability across all initialization strategies, we incorporate layer normal- ization for query and key components in both linear self- attention and cross-attention modules. This normalization technique is crucial as it: (I) stabilizes the attention com- putation during the early stages of training, (2) prevents potential gradient instability when integrating new layers, and (3) enables rapid adaptation while maintaining model quality.
Identity Mapping Initialization We initialize the weights of specific components zero in new layers, particularly the output projections of self-attention, cross-attention, and the final point-wise convolution in MLP blocks, following (Chen et al., 2015). This cctoiinitialization ensures that new transformer blocks initially behave as identity functions, providing two key benefits: (1) exact preservation of the pre-trained model's behavior at the start of training, and (2) stable optimization path from known good solution.
2.2. Efficient Model Growth
Rather than training large models from scratch, we propose an efficient model growth strategy that expands pre-trained DIT with N layers to N + M layers while preserving its learned knowledge. We explore three initialization strategies to ensure effective knowledge transfer during model expansion. Figure 1l in the appendix illustrates the three strategies.
Design Choice Among these strategies, we adopt the partial preservation initialization approach for its simplicity and stability. This choice creates natural division of labor: the pre-trained N layers maintain their feature extraction capabilities while the randomly initialized M layers, starting from identity mappings, gradually learn to refine these representations. Empirically, this approach provides the most stable training dynamics compared to
Initialization Strategies Let 0, E Rd denote the parame- ters of layer in the expanded model and 0.TC E R represent the parameters of layer of the pre-trained model, where d is the parameter dimension of each layer. We investigate three approaches for parameter initialization:
(1) Partial Preservation Init, where we preserve the first N pre-trained layers and randomly initialize the additional M
3

---

SANA 1.5: Efficient Scaling of Training-Ting Time and Inference- Compute in Linear Diffusion Transformer
eyclie and block expansion strategies. Considering the block importance (see analysis in Section 3.3), we drop the last two blocks in the pre-trained model to enhance the leamning of newly added blocks.
where & is the of quantized layers, ny is the parameter count of layer 1, and 24 represents the maximum bytes saved per parameter. In practice, the actual memory savings are slightly lower due to several factors: (1) small layers (<16K parameters) remain in 32-bit precision, (2) second-order statistics are kept in 32-bit, and (3) additional overhead from quantization metadata. Nevertheless, this approximation provides good estimate of the memory efficiency gained through our hybrid quantization strategy.
2.3. Memory-Efficient CAME-Sbit Optimizer
Building upon CAME (Luo et al., 2023) and AdamW- 8bit (Dettmers et al., 2021), we propose CAME-Sbit for efficient large-scale model training. CAME reduces mem- ory usage by half compared to AdamW through matrix factorization of second-order moments, making it particu- larly efficient for large linear and convolutional layers. We further extend CAME with block-wise 8-bit quantization for first-order moments, while preserving 32-bit precision for critical statistics to maintain optimization stability. This hy- brid approach reduces the optimizer's 's memory footprint to approximately 1/8 of AdamW, enabling billion-scale model training on consumer GPUs without compromising conver- gence properties.
2.4. Model Depth Pruning
To address the challenge of balancing effectiveness and effi- ciency in large models, we introduce model depth pruning approach that efficiently compresses large models into vari- ous smaller configurations while maintaining comparable quality. Inspired by Minitron (Sreenivas et al., 2024), a transformer compression technique for LLMs, we analyze block importance through input-output similarity patterns:
X BI: 1 E Ex, X2,1 1
(3)
Block-wise Quantization Strategy We adopt selective quantization approach where only large matrices (> 16K pa- rameters) in linear and I IxI I convolution layers are quantized, as these layers dominate the optimizer's ' memory footprint. For each block of size 2048, we compute independent scal- ing factors to preserve local statistical properties. Given tensor block x _º representing the first-order momentum values, the quantization function ((x) an 8-bit integer:
where X denotes the input of the i-th transformer block. We average the block importance across diffusion time- steps and our calibration dataset, which contains 100 diverse prompts. As shown in Figure 5c, the block importance is higher in head and tail blocks, and we conjecture that the head blocks change the latent distribution to diffusion distribution and the tail blocks change it back. The middle blocks commonly have higher similarity between input and output features, demonstrating the gradual refinement of the generated results. We prune the transformer blocks based on the importance of the sorted block. As illustrated in Figure 4, pruning the blocks will gradually impair the high-frequency details. Therefore, after pruning, we further fine-tune the model to compensate for the information loss. Specifically, we use the same training loss as the large model to supervise the pruned models. Adapting the pruned model to complete information surprisingly easy. With only 100 fine-tune steps, the pruned 1.6B model can achieve comparable quality with the full 4.8B model and outperform the SANA-1.0 1.6B model (Table 3).
maps each value to
x - min(r) max min(x) x 255
((x) round
where min(x) and max(x) are the minimum and maxi- mum values in the block respectively, and round(-) maps to the nearest integer. This linear quantization preserves the relative magnitude of values within each block while compressing the storage to 8 bits per value.
Hybrid Precision Design To maintain optimization sta- bility, we keep second-order statistics in bit precision, as these are critical for proper gradient scaling. Benefit- ing from CAME's matrix factorization, these statistics are already memory-efficient for linear layer with dm in- put dimensions and dout output dimensions, the storage of second-order moments is reduced from Oddn x dous) to Oddn making their precision less critical for overall memory consumption. This hybrid approach reduces mem- ory usage while preserving CAME's convergence properties. Memory reduction can be formulated as:
2.5. Inference-Timee Scaling
With sufficient training, SANA-1.5 gains stronger genera- tion abilities after efficient model growth. Inspired by the recent success of inference-time scaling in large language models (LLMs) (Brown et al., 2024), we are interested in inference-time scaling to push the generation upper bound.
Scaling Denoising Steps v.s. Scaling Samplings For SANA and many other diffusion models, a natural option to scale up the inference-time computation to increase the
Maaeed
Σ 1EC
(n4 x 24) bytes,
(2)
4

---

SANA 1.5: Efficient Scaling of Training-Time Time and inference-Ting Compute in Linear Diffusion Transformer
Artffctss cannot be self-correcting when scaling denoising steps.
Prompt: photo of an apple above tv
VLN
VLN
xN
VIN
VLN
be0
t=20
(=50
t-0
t-20
Minor Improvements increasing from 50
(a). Scale Denoising Steps (X)
(b). Scale Sampling Noises (V)
(c). Best-of-N Jdged by VLM Specialist
Figure 3. Comparing Scaling Between Denoising Steps and Samples with VLM Judgment Visualization. (a) Scaling denoising steps show only minor improvements and often fail to self-correct artifacts, making a poor option for scaling up. (b) In contrast, scaling sampling noise proves effective, with VLM specialists helping to verify and select images that match the prompts. (c) VLM evaluates and ranks the hest images in a tournament format.
number of denoising steps. However, using more denoising steps is not ideal for scaling for two reasons. First, addi- tional denoising steps cannot self-correct errors. Figure ((a) illustrates this with a sample, where objects misplaced at an early stage remained unchanged in subsequent steps. Sec- ond, the generation quality quickly reaches a plateau, As shown in Figure 3, SANA produces visually pleasing results with just 20 steps, showing no significant visual improve- ment even increase x steps.
ing the best-matching images from multiple options, both models exhibited a strong bias toward the first-presented options, regardless of image ordering or shuffling.
Instead of applying existing models or APIs, we trained specialized NVILA-2B (Liu et al., 2024b) to score the images, which named Sa VILA:: Sana generates then VILA picks. We created a 2M prompt-matching dataset with prompts generated in the GenEval style. We excluded the prompts that are already existed in GenEval evalset and generated the images using using Flux-Schnell (Labs, 2024) to avoid overfitting. We then formatted these as multimodal conversations, as shown below
In contrast, scaling the number of sampling candidates is a more promising direction. As presented in Figure 3(b), small model SANA (1.6B) can also generate correct re- sults for difficult test prompts when given multiple attempts, much like sloppy/scattered student who can draw re- quested but sometimes makes mistakes during execution. With enough opportunities to try, it can still provide a sat- isfactory answer. Therefore, we choose to generate more images and introduce a "patient teacher" to score the results, which will be expanded in the following.
User: You are an AI assistant specializing in image analysis and ranking. Your task is to analyze and comрare image based on hew well they match the given prompt. <image> The given prompt is: <prompt>, Please consider the prompt and the image to make a decision and response directly with 'yes' or 'no'
Visual Language Model (VLM) as the Judge To find images that best match given prompt, model that under- stands both text and images is needed. While popular mod- els like CLIP (Radford et al., 2021) and SigLIP (Zhai et al., 2023) offer multi-modal capabilities, their small context win- dows (77 tokens for CLIP and 66 tokens for SigLIP) limit their effectiveness. This limitation poses a problem since SANA usually takes long, detailed descriptions as inputs. To address this, we explored Visual Language Models to evalu- ate prompt-matching for generated images. We tested com- mercial multi-modal APIs, specifically GPT-4o (OpenAI, 2023b) and Gemini- 1.5-pro (Team, 2024), but encountered two significant issues. First, when evaluating single images against prompts, both APIs lacked consistency in their rat- ings across different runs. Second, when tasked with select-
SaVILA: 'yos' / 'no'.
The fine-tuned SaVIIA can effectively assess how well images match their prompts and robustly filters out prompt- mismatching images. During inference, compare two images in each round until the top-N candidates are deter- mined
When SaVILA responses one 'yes' and one 'no', we pick the image with 'yes';
When SaVILA responses both 'yes' or 'no', we pick the image with higher confidence (logprob);
5

---

SANA 1.5: Efficient Scaling of Training-Timg and Inference- Compute in Linear Diffusion Transformer
Such tournament-style comparison robustly filters out prompt-mismatching images and consistently boosts the GenEval scores, as illustrated in Figure 3(c) and Table. 2. More details of inference time scaling are attached in Ap- pendix B.
Model Depth Pruning
2.3 Gentvar0a77
Prune 2.48 GerSutto 607
Prunets 146 Gerb
Prine lnll 3h-1 Cenborloaae
Prateto alwer Gardaal
Powe Telee Centaal 8222
3. Experiments
Prompe: Actsneupp photo of Jerson subjectis gny diuza undernssth....
3.1. Experimental Setup
Model Architecture. Our final model (SANA-4.8B) scales to 60 layers while maintaining the same channel dimension (2240 per layer) and FFN dimension (5600) as SANA-1.6B The architecture, training data, and other hyperparameters remain consistent with SANA-1.6B (Xie et al., 2024).
Figure 4. Visual comparison of SANA-1.5 models with differ- ent pruned configurations. Our adaptive depth pruning enables efficient compression to various model sizes (from 1.6B to 4.8B), While aggressive pruning may slightly affect fine-grained details, the semantic content is well preserved, and the overall image qual- ity can be easily recovered after brief fine-tuning (100 steps on I GPU), demonstrating the effectiveness of our pruning strategy.
Training Details. We conduct distributed training using Py PyTorch DDP across 64 NVIDIA A100 GPUs on DGX nodes. Our training pipeline follows two-phase strategy: first pre-train the model with learning rate of le-4, followed by supervised fine-tuning (SFT) with a reduced learning rate of 2e-5. The global batch size is dynamically adjusted between 1024 and 4096 throughout the training process. Following common practice in large language model training. we initially pre-train on large-scale dataset before performing SFT on a high-quality dataset.
to 0.81. Notably, SANA-4.8B demonstrates 0,81 GenEval score, outperforning Playground v3's 0.76, but with 5.5 times lower latency than FLUX-dev (23.0s). Our model also maintains 6.5 times higher throughput than these larger models (compared to FLUX-dev's 0.04 samples/s), making it more practical for real-world applications. The speed is tested on one A100 GPU with FP16 Precision.
Evaluation Protocol. We adopt multiple evaluation metrics, including FID, CLIP Score, GenEval (Ghosh et al., 2024), and DPG Bench (Hu et al., 2024b), comparing it with SOTA methods. FID and Clip Score are evaluated on the MJHQ- 30K (Li et al., 2024a) dataset, which contains 30K images from Midjourney. GenEval and DPG-Bench both focus on measuring text-image alignment, with 553 and 1,065 test prompts, respectively. We particularly emphasize the GenEval as it better reflects text-image alignment and shows more room for improvement than other metrics.
Model Pruning We compare among difference sizes of SANA-1.5 and SANA-1.0 models in Figure 4 and Table 3. For a fair comparison with SANA-1.0 1 1.6B, the SAN 4.8B model here is trained without supervised fine-tuning from high-quality data. All results are evaluated on im- ages of size 512x512. With a small computational cost, the pruned and fine-tuned model outperforms the model trained from scratch (0.672 v.s. 0.664), which an efficient approach to obtaining models of various sizes.
3.2. Main Results
Model Growth We compare SANA-4.8B with the most advanced text-to-image generation methods in Table 1. The scaling from SANA-1.6B to 4.8B Pre (Pre-trained) brings substantial improvements: 0.06 absolute gains in GenEval (from 0.66 to 0.72), 0.34 reduction FID (from 5.76 to 5.42), and 0.2 improvement in DPG score (from 84.8 to 85.0). Compared to state-of-the-art methods, our 4.8B model achieves comparable better results than much larger models like Playground v3 (24B) and FLUX (12B) while using only a fraction of their parameters. We fur- ther perform post-training on the 4.8B model using a high- quality dataset, with details provided in Sec. 3.3. To dif- ferentiate between the models, we refer to the pre-trained model Pre and the post-trained model as Ours. After post- training, SANA-4.8B achieves significant performance im- provement on the GenEval benchmark, increasing from 0.72
Inference Scaling We incorporate inference scaling with the SANA-1.5 4.8B v2 model and compare it against other large image generation models on the GenEval benchmark (Table 2). By selecting samples from 2048 generated images, the inference-scalede model outperforms naive generation by 15% in overall accuracy from 0.81 to 0.96, with particularly significant improvements in the "Position" (from 0.59 to 0.96), and "Color Attribution" (from 0.65 to 0.87). Furthermore, equipped with inference scaling, 4.8B model outperforms Playground v3 (24B) by 20% in overall accuracy (0.76 vs 0.96). These results demonstrate that trading inference efficiency can enhance model gener- ation quality and accuracy, even with much smaller base model size (4.8B vs 24B).
6

---

SANA 1,5: Efficient Scaling of Training-Time Time and Inference- Compute in Linear Diffusion Transformer
Table Comprehensive comparison of our method with SOTA approaches efficiency and performance. The speed tested on one A100 GPU with BF16 Precision. Throughput: Measured with batch=10. Latency: Measured with batch=l and sampling step=20. We highlight best and second best entries.
Throughput (samples/s)
Latency
Params (B)
Methods
FID L
CLIP
GenEval
DPG f
(8)
LUMINA-Next (Zhuo al., 2024)
0.12
9.1
2.0
7.58
26.84
0,46
74.6
SDXL (Podell et al., 2023)
0.15
2.6
6.63
29.03
0.55
74.7
Playground v2.5 (Li et al., 2024a)
0.21
5.3
2.6
6.09
29.13
0.56
75.5
Hunyuan-DiT (Li et al., 2024d)
0.05
18.2
1.5
6,54
28.19
0.63
78,9
PixArt-E (Chen et al., 2024b)
0,4
2.7
0.6
6,15
28.26
0.54
80.5
DALLE 3 (OpenAI, 2023a)
0.67
83.5
SD3-medium (Esser et al., 2024)
0.28
4.4
2.0
11.92
27.83
0.62
84.1
FLUX-dev (Labs, 2024)
0.04
23.0
12.0
10.15
27.47
0.67
84.0
FLUX-schnell (Labs, 2024)
0.5
2.1
12.0
7.94
28.14
0.71
84.8
Playground v3 (Liu et al., 2024a)
0.06
15.0
24
0.76
87.0
SANA-1.0 0.6B (Xie et al., 2024)
1.7
0.9
0.6
5.81
28.36
0.64
83.6
SANA-1.0 1. 1.6B (Xic et al., 2024)
1.0
1.2
1.6
5.76
28.67
0.66
84.8
SANA-1.5 4.8B Pre
0.26
4.2
4.8
5.42
29.16
0.72
85.0
SANA-1.5 4.8B Ours
0.26
4.2
4.8
5.99
29.23
0.81
84.7
Table Detailed GenEval evaluation benchmark. SANA-1.5 Inference Scaling with 2048 samples achieves absolute SoTA compared open-source and commercial methods. We used the numbers from Playground v3 (Liu al., 2024a) for the baseline methods.
Method
Overall
Single
Two
Counting
Colors
Position
Color Attribution
SDXL (Podell et al., 2023)
0.55
0.98
0.74
0.39
0.85
0.15
0.23
DALLE 3 (OpenAI, 2023a)
0.67
0.96
0.87
0.47
0.83
0.43
0.45
Flux-dev (Labs, 2024)
0.68
0.99
0.85
0.74
0.79
0.21
0.48
SD3 (Esser et al., 2024)
0.74
0.99
0.94
0.72
0.89
0.33
0.60
Playground v3 (Liu et al., 2024a)
0.76
0.99
0.95
0.72
0.82
0.50
0.54
SD1.5 (Rombach et al., 2022a)
0.42
0.98
0.39
0.31
0.72
0.04
0.06
+ Inference Scaling
0.87
1.00
0.97
0.93
0.96
0.75
0.62
SANA-1.5 4.8B Pre
0.72
0.99
0.85
0.77
0.87
0.34
0.54
SANA-1.5 4.8B Ours
0.81
0.99
0.93
0.86
0.84
0.59
0.65
+ Inference Scaling
0.96
1.00
1.00
0.97
0.94
0.96
0.87
Table Evaluation of pruned SANA models. "3.2B" and "1.6B" denote the model directly pruned from SANA-1.5 4.8B, and "+FT" denotes efficiently fine-tuning the pruned model.
Table 4. Model performance across different seales. Models are first pre-trained and then fine-tuned (SFT) high-quality data.
Params. (B)
Stage
Train Steps
GenEval 1
Method
4.8B
3.28
+FT
1.6B
+FT
SANA-LO 1.6B
pre-train
200K
0.64
GenEval
0.693
0.673
0.684
0.571
0.672
0.664
SFT
10K
0.68 (+4%)
pre-train
200K
0.66
1.6
3.3. Analysis
SFT
~
0.69 (+3%)
pre-train
100K
0.69
Comparison of Different Optimizers We compare CAME-Sbit with AdamW-Sbit and their 32-bit counterparts in Figure 6 for SANA-1.61 training. The 8-bit optimizers [AdamW-8he) Came-8bit) achieve comparable convergence to their 32-bit counterparts while significantly reducing GPU memory usage. Specifically, CAME-Sbi reduces memory consumption by 25% compared to AdamW (43GB vs 57GB)
4.8
SFT
10K
0.72 (+3%)
with no degradation in training convergence speed. Note that CAME-Sbi reduces optimizer state memory usage pro- portionally to model size, yielding greater memory savings for larger models.

---

SANA 1.5: Efficient Scaling of Training-Time and Inference-Tinee Compute in Linear Diffusion Transformer
Block Importance Score
10
Block Importance Score
10
Black Importance Score
10
10-3
10-3
10"
Low block importance, potential for pruning
10
10
10
20
10
10
10-4
10"5
10
10
10
10
15
20
45 50
10
10 15 20 25
35 40 45 50 55
Block Index
Block Index
Block Index
(a) SANA 1.6B trained from scratch.
(b) SANA-4.8B trained from scratch.
(c) SANA 4.8B with model growth.
Figure 5. Analysis of block importance (BI) across different models: (a) SANA-LO 1.6B, (b) SANA 4.8B trained from scratch, and (c)
final SANA-1.5 4.8B with initialization.
Partil on
CANNE SOUB
CAASS- Ball C3:8
Block Rupiicaioo
GAME 4011
CAME
0.90
Cyddic lisph
0.86
072
0000
0.2
5464-1300 33x
0.32
1000
2200
3000
4000
1000 0.44
10
201
Traning Stape
Figure 6. Training loss curves for different optimizers on 1.6B parameter diffusion model. The CAME-Sbit reduces memory consumption by 25% compared to AdamW while maintaining the convergence speed.
Figure 7. Comparison of different initial- ization strategies. Partial Preservation Init shows stable training behavior while Cyclic and Block Replication strategies suffer from training instability (NaN losses).
Figure 8. Inference-time Scaling Results. Scaling up inference time compute consis- tently yields better GenEval scores, and helps the small model to achieve compara- ble or better performance with larger ones.
Comparison of Different Initialization Strategies We compare the three types of initialization strategies in Fig- ure 7. Partial Preservation Init shows stable training be- havior while Cyclic and Block Replication strategies suffer from training instability (NaN losses). Such observation also supported by the block importance analysis in Fig- ure 5. The feature distribution of the 4.8B model different from the 1.6B model due to the model capacity, and thus, replication of the block weight increases the difficulty of convergence to the final distribution.
Block Importance Instructing Model Depth Pruning As shown in Figure 5c, blocks in the middle to the end have low importance scores, especially when compared with the model trained from scratch (Figure 5b), This indicates potential for model size reduction. Based on this observa- tion, we prune the blocks in SANA-1.5 4.8B according to their sorted importance scores. In Figure 4, pruning the blocks (gradually reducing from 60 to 20 blocks) impairs high-frequency information. The lack of details degrades the accuracy of GenEval benchmark to 0.571. However, the image layout and semantic information are well preserved. Therefore, high high-frequency information can quickly recovered with 100 steps of fine-tuning on a single GPU.
Block Importance Instructing Model Growth The anal- ysis of block importance instructs both our initialization strategy and pruning approach. The block importance of the pre-trained SANA-1 O model is shown in Figure 5a, where more information resides the head and tail blocks. Dur- ing model scaling, we initially attempted to append new blocks directly after all the pre-trained blocks. However, we observed that the newly added blocks failed learn effec- tive information and became stuck in local minima. The primary reason is that the well-learned pre-trained features dominate the feature representation through skip connec- tions. Therefore, we remove the last two blocks, which are more relevant,,,,,, before adding new blocks. This process effectively facilitates learning in the later blocks.
High-quality Data Fine-tuning While extensive pre- training on large-scale datasets leads to quality saturation,
fine-tuning on a curated dataset (3M samples from 50M pre- training data) significantly and efficiently improves model capabilities of different model sizes. Specifically, by fine- tuning image-text pairs with CLIP score 25, our 4.8B model achieves 3% improvement in GenEval score com- pared to the pre-trained model, as shown in Table 4.
Furthermore, we created new small-scale SFT data set, consisting of 144,291 images generated from 18,240
8

---

SANA 1.5: Efficient Scaling of Training-Tine Time and Inference- Compute in Linear Diffusion Transformer
prompts in the GenEval style. Each prompt generates 7.91 "correct" images on average. We limit each prompt to gen- erate at most 10 correct images to prevent overfitting, The correct image is filtered by the GenEval toolkit. Note that we excluded the prompts already existing in GenEval evalset and generated images using Flux-Schnell to avoid overlit- ting. As shown in Table 2, this new SFT data improves the GenEval result of SANA-I.5-4.888 from 0.72 (v1) to 0.81 (v2). Figure 9 presents comprehensive comparison be- tween SANA-1.5-4.88 and current state-of-the-art methods across multiple challenging scenarios. The results demon- strate DANA-I.5-4.88 4.'s's consistent superiority in generated image quality, as evidenced by both quantitative metrics and visual assessment.
The only limitation is increased computational cost: sam- pling N images requires N x 49, 140G FLOPPs for SANA generation and 2/N x 4, 518G FLOPs for SaVILA judgment and comparison. We leave the efficiency for future work.
4. Related Work
We put a relatively brief overview of related work in the main text, with a more comprehensive version in the ap- pendix. Text-to-image generation has evolved rapidly, from Stable Diffusion (Rombach et al., 2022b) to more recent architectures like DiT (Peebles & Xie, 2022) and its vari- ants (Chen et al., 2024c; Labs, 2024; Esser et al., 2024). Efficiency-foaue works like SnapGen (Hu et al., 2024a), Arx-r (Chen et al., 2024c) and SANA (Xie et al., 2024) have significantly reduced training and inference costs. Au- toregressive models (Tang et al., 2024; Tian et al., 2024; Sun et al., 2024) are also developed rapidly and achieve comparable quality as diffusion models. Research in lan- guage (Kaplan et al., 2020) and vision domains (Li et al., 2024b; Liang et al., 2024) both revealed power-law rela- tionships. In the image generation field, (Xu et al., 2023; 2024) has explored RLHF align model with human pref- erences. More recent concurrent work (Singhal et al., 2025; Ma et al., 2025) have explored various strategies to improve generation quality without increasing model size.
SDS Medium
FLUX-dev
FLUU--chhell SANA 1.0 1.SB 1.5 4.8B
Reget
prigeuu Aghr, minorr rnflection, detal, angle lans
Bogh
5. Conclusion
atooy-ooot glant gou--tte
This paper presents a comprehensive approach to efficient model scaling, addressing both training and inference com- pute challenges. For training efficiency, we propose memory-efficient optimizer CAME-Sbit and a stable model growth strategy. For inference scaling and acceleration, we introduce repeat sampling and depth pruning techniques. These approaches collectively enable significant quality im- provements under limited computing budgets, making large- scale generative models more accessible. This work con- tributes to democratizing large-scale AI research by making it more accessible to researchers with limited resources.
Onler photo of coug) mank of ofrmssppennt glans, maoniigg on mh wiweoodd Visemin Madherta Pavt
Alsmect moneey with a tepton, in plonl ArT style
Figure 9. Comparison of SANA-1.5 against SoTTA methods. We evaluate SANA-1.5 alongside contemporary approaches using identical text prompts, including challenging cases with noisy inputs and long-form text. For detailed assessment, please zoom into the comparative samples.
Impact Statement
Misusing generative AI models to generate NSFW con- tent is challenging issue for the community. To enhance safety, we have equipped SANA together with safety check model (Zeng et al., 2024a). Specifically, the user prompt will first be sent to the safety check model to determine whether it contains NSFW safe for work) content. While SANA- 1.5 demonstrates efficient model scaling, challenges remain in complex generation tasks, particularly text rendering and human details. Our work makes large-scale model training more accessible, while encouraging responsible develop- ment and deployment to prevent misuse.
Inference Time Scaling Figure 8 demonstrates the bene- fits of scaling up inference-time computation. First, SANA's accuracy on GenEval consistently improves with more sam- plings. Second, inference-time scaling enables smaller SANA models to match or even surpass the accuracy of larger ones (1.6B + scaling is better than 4.8B). This re- veals the potential of scaling up inference and allows SANA to push toward new state-of of-the art results. As shown in Table 2, our best SANA model with inference scaling out- performs all previous commercial and community models.

---

SANA 1.5: Efficient Scaling of Training-Time and Inferenee-Timee Compute in Linear Diffusion Transformer
References
Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference Machine Learning, 2024.
Alabdulmohsin, I. M., Neyshabur, B., and Zhai, X. Revisit- ing neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300- 22312, 2022.
Ghosh, D., Hajishirzi, H.,, and Schmidt, L. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024.
Brown, B.. Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint urXiv:2407.11877 2024.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. NeurlPS, 2014.
Cai, H., Gan, C., Wang, T., Zhang, Z., and Han, S. Once- for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv: 1908. 097919 2019.
Han, J., Liu, J., Jiang, Y., Yan, B., Zhang, Y., Yuan, Z., Peng, B., and Liu, X. Infinity: Scaling bitwise autoregres- sive modeling for high-resolution image synthesis. arXiv preprint arXiv:2412.04433 2024.
Cai, H., Li, J., Hu, M., Gan, C., and Han, S. Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction. Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pp. 17302- 17313. 2023.
Han, S., Mao, H., and Dally, W. J. Deep Compression: Com- pressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. In ICLR, 2016.
Chen, J., Cai, H., Chen, J., Xie, E., Yang, S., Tang, H., Li, M., Lu, Y., and Han, S. Deep compression autoen- coder for efficient high-resolutio diffusion models. arXiv preprint ar arXiv:2410. 10733, 2024a.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6855 2020.
Chen, J., Ge, C., Xie, E., Wu, Y., Yao, L., Ren, X., Wang, 7., Luo, P., Lu, H., and Li, 7.. Pixart-o: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.0460 04692, 2024b.
Hu, D., Chen, J., Huang, X., Coskun, H., Sahni, A., Gupta, A., Goyal, A., Lahiri, D., Singh, R., Idelbayev, Y., et al. Snapgen: Taming high-resolution text-to-image models for mobile devices with efficient architectures and train- ing. arXiv preprint 2rXi: 2024a.
Chen, J., Jincheng. Y., Chongjian, G., Yao, L., Xie, E., Wang, 7.,, Kwok, J., Luo, P., Lu, H., and Li, 7. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International Conference on Learning Representations, 2024c.
Hu, X., Wang, R., Fang, Y., Fu, B., Cheng. P., and Yu, G. Ella: Equip diffusion models with Ilm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024b.
Chen, L.,, Davis, J. Q., Hanin, B., Bailis, P., Stoica, I., Za- haria, M., and Zou, J. Are more IIm calls all you need? towards scaling laws of compound inference systems. arXiv preprint arXiv:2403.02419, 2024d.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Chen, T., Goodfellow, I., and Shlens, J. Net2net: Accel- erating learning via knowledge transfer. arXiv preprint arXiv:1511.0564/ 2015.
Labs, B. F. Flux, 2024. blackforestlabbe ai/.
URL https://
Li, D., Kamko, A., Akhgari, E., Sabet, A., Xu, L.,, and Doshi, S. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint 2402 2024a.
Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L. 8- bit optimizers via bise quantization. arXiv preprint 10.00:2110.02861, 2021.
Dubey. A.. Jauhri, A.. Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llaaa 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
Li, H., Zou, Y., Wang. Y., Majumder, O., Xie, Y., Manmatha, R., Swaminathan, A., Tu, Z., Ermon, S., and Soatto, S. On the scalability of diffusion- text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9400- 9409, 2024b.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al.
10

---

SANA 1.5: Efficient Scaling of Training- Time and Inference-Timg Compute in Linear Diffusion Transformer
Li, M., Lin, J., Ding, Y., Liu, Z., Zhu, J.-Y., and Han, S. Gan compression: Efficient architectures for interactive conditional gans. In CVPR, 2020.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Li, Y.. Wang, H., Jin, Q. Hu, J., Chemerys, P. Fu, Y., Wang, Y., Tulyakov, S., and Ren, J. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. Advances in Neural Information Processing Systems, 36, 2024c.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
Li, Z.,, Zhang, J., Lin, Q., Xiong, J., Long, Y., Deng, X., Zhang, Liu, X., Huang, M., Xiao, Z., al. Hunyuan- dit: A powerful multi-resolution diffusion transformer with tineigrained chinese understanding. arXiv preprint wxXiv:2405.08744 2024d.
Radford, A., Kim, J. W., Hallacy, C.,, Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger. G., and Sutskever, I. Learning transferable visual models from natural language supervision, 2021. org///g///aa....oooo aab.//133, 00020.
URL
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022a.
Liang, Z., He, H., Yang, C., and Dai, B. Scaling laws for diffusion transformers. arXiv preprint 1rXiv22400.0884, 2024.
Liu, B., Akhgari, E., Visheratin, A., Kamko, A., Xu, I.., Shrirao, S., Souza, J., Doshi, S., and Li, D. Playground v3: Improving text-to-image aligmment with deep-fusion large language models. arXiv preprint arXiv:2409. 10695, 2024a.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022b.
Liu, Z., Zhu, L., Shi, B., Zhang, Z., Lou, Y., Yang. S., Xi, H., Cao, S., Gu, Y., Li, D., et al. Nvila: Efficient frontier vi- sual language models. arXiv preprint urXiv:2412.04468. 2024b.
Singhal, R., Horvitz, Z., Teehan, R., Ren, M., Yu, Z., McK- eown, K., and Ranganath, R. A general framework for inference-tine scaling and steering of diffusion models. arXiv preprint arXiv:2501.0688 2025.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint 1rXiv: 2017.
Sreenivas. S. T., Muralidharan, S.. Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Lim pruning and distillation in practice: The minitron approach. arXiv preprint arXiv:2408. 0e..22408.19996, 2024.
Luo, Y., Ren, X., Zheng, Z., Jiang, Z.,, Jiang, X., and You, Y. Came: Confidence-guided adaptive memory efficient optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 4442 4453, 2023.
Sun, P., Jiang, Y., Chen, S., Zhang, S., Peng, B.. Luo, P., and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 2406222062.2222 2024.
Ma, N., Tong, S., Jia, H., Hu, H., Su, Y.-C., Zhang, M., Yang, X., Li, Y., Jaakkola, T., Jia, X., et al. Inference-time scaling for diffusion models beyond scaling denoising steps. arXiv preprint 3xXi::5501.09732 2025.
Tang, H., Wu, Y., Yang. S., Xie, E., Chen, J., Chen, J., Zhang, Z., Cai, H., Lu, Y., and Han, S. Hart: Efficient visual generation with hybrid autoregressive transformer. arXiv preprint 24Xiv:2410.100 10812, 2024.
Ma, X., Fang, G., and Wang, X. LIm-pruner: On the struc- tural pruning of large language models. NeurlPs, 2023.
OpenAI. Dalle-3, 2023a. URL hwwph:tpt///ooee dall-e-3.
com/
Team, G. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.0555 055303 2024. URL https://
OpenAI. Gpt-4 technical report. arXiv preprint 0rXiv:2303.08774 2023b. URL attps://arxin
arxiv. abs /403. .0..00....00 05530.
brg/abs/23033 08774.
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang, L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. arXiv preprint urXiv:2212.09745 2022.

---

SANA 1.5: Efficient Scaling of Training-Time Time and Inference- Time Compute in Linear Diffusion Transformer
Wang. X., Wei, J., Schuurmans, D., Le, Q. Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language models. arXiv preprint arXiv:2203.
Zheng, C., Liu, Z., Xie, E., Li, Z., and Li, Y. Progressive- hint prompting improves reasoning in large language models. arXiv preprint arXiv:2304.09797 2023. 2023.
2022.
Zhuo, L., Du, R., Xiao, H., Li, Y., Liu, D., Huang, R., Liu, W. Zhao, L., Wang, F.-Y. Ma, Z., al. Lumina- next: Making lumina-12x stronger and faster with next-dit. arXiv preprint arXiv:2406. 18583, 2024.
Wang, X., Zhang, X., Luo, Z., Sun, Q. Cui, Y., Wang, J.. Zhang, F., Wang. Y., Li, Z., Yu, Q., et al. Emu3: Next-token prediction is all you need, arXiv preprint arXiv:2409.1886 2024.
Wu, Y. Sun, Z., Li, S., Welleck, S., and Yang, Y. Infer- ence scaling laws: An empirical analysis of compute- optimal inference for moblem-solving with language models. arXiv preprini TrXiv:2408.00724 2024.
Xie, E., Chen, J., Chen, J., Cai, H., Tang, H., Lin, Y., Zhang, Z., Li, M., Zhu, L.,, Lu, Y., et al. Sana: Efficient high- resolution image symthesis with linear diffusion trans- formers. arXiv preprint arXiv:2410. 0tti::2411.000000 10629, 2024.
Xu, J., Liu, X., Wu, Y., Tong, Y. Li, Q.. Ding, M., Tang, J., and Dong, Y. Imagereward: learning and evaluating human preferences for text-to-image generation. In Pro- ceedings of the 37th International Conference on Neural Information Processing Systems, pp. 15903-15935, 2023.
Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang. J., and Dong, Y. Imagereward: Learning and evaluating hu- preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Zeng, W., Liu, Y., Mullins, R., Peran, L., Fernandez, J., Harkous, H., Narasimhan, K., Proud, D., Kumar, P., Rad- harapu, B., Sturman, O., and Wahltinez, O. Shieldgemma: Generative ai content moderation based on gemma, 2024a. URL www.s:ttp//......... / arxiv.ai orgg abs/ / 2407.2172. 21772.
Zeng. W., Liu, Y. Mullins, R., Peran, L., Fernandez, J., Harkous, H., Narasimhan, K., Proud, D., Kumar, P., Radharapu, B., Sturman, O., and Wahltinez, O. Shicld- gemma: Generative content moderation based on gemma, 2024b. URL htttps://arxiii org/ /
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. arXiv preprint arXiv:2106.04560, 2021.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training, 2023. URL https: /ttp///aavvvv/vaa//aa/eeeeeawavl org/ wtgg/bb/////oo /20..3333...3333...
Zhang, B. and Sennrich, R. Root mean square layer nor- malization. Advances in Neural information Processing Systems, 32, 2019.
Zhao, Y., Xu, Y., Xiao, Z., and Hou, T. Mobilediffusion: Subsecond toxt-to-image generation on mobile devices. arXiv preprint arXiv:2311.16500 2023.
12

---

SANA 1.5: Efficient Scaling of Training-Time and InferenceTTimg Compute in Linear Diffusion Transformer
A. Full Related Work
Text to Image Generation Text-to-image generation has undergone rapid evolution in model architectures and efficiency. The field gained momentum with Stable Diffusion (Rombach et al., 2022b), and later witesssdd a pivotal shift towards Diffusion Transformers (DiT) (Peebles & Xie, 2022) architectures. PixArt-a (Chen al., 2024c) demonstrated competitive quality while significantly reducing training costs to just 10.8% of Stable Diffusion v1.5's requirements (Rombach et al., 2022b). Recent large-scale models like FLUX (Labs, 2024) and Stable Diffusion 3 (Esser et al., 2024) have pushed the boundaries in compositional generation capabilities, while Playground v3 (Liu et al., 2024a) achieved state-ofthhe-rt image quality through full integration with Large Language Models (LLMs) (Dubey et al., 2024). Pix (Chen et al., 2024b) further enabled direct 4K resolution image generation with a compact 0.6B parameter model. In parallel, efficiency- -occued innovations like SANA (Xie et al., 2024) introduced breakthrough capabilities in high-resolution synthesis through deep compression autoencoding (Chen al., 2024a) and linear attention mechanisms, making deployment possible even on laptop GPUs. These developments showcase the field's progression toward both more powerful and more accessible text-to-imme to--maaee generation.
Diffusion Model Pruning Neural network pruning (Han et al., 2016) is an effective technique for improving the efficiency of neural models, particularly for deployment on resource-constrained devices. By removing redundant weights, it reduces both model size and computational complexity. In LLMs, researchers have successfully applied pruning to shrink models for various applications (Sreenivas et al., 2024; Ma et al., 2023). For generative models, (Li et al., 2020) employ neural architecture search (Cai et al., 2019) to prune GAN channels (Goodfellow et al., 2014). SnapFusion (Li et al., 2024c) extends this to diffusion models, using elastic depth (Cai et al., 2019) to prune UNet blocks (Ho et al., 2020), managing to deploy Stable Diffusion on mobile phones. Similarly, MobileDiffusion (Zhao et al., 2023) shrinks UNet depth and distills the model for single-step inference. Our approach targets the recent DiT architecture (Peebles & Xie, 2023). We instead use a heuristic method to identify and prune less important blocks directly, avoiding the overhead of search.
Training Scaling in LLM and DiT Training scaling laws have been extensively studied in both language (Kaplan et al., 2020; Alabdulmohsin et al., 2022) and vision (Li et al., 2024b; Zhai et al., 2021; Liang et al., 2024) domains. For language models, research has revealed power-law relationships between model accuracy and factors like model size, dataset size, and compute (Kaplan et al., 2020). These scaling patterns have been consistently observed across several orders of magnitude. Recently, similar scaling properties have been discovered in diffussion DIT's pre-training loss follows pawer-law relationships with computational resources (Liang et al., 2024). Furthermore, extensive experiments on scaling both denoising backbones and training sets reveal that increasing transformer blocks is more parameter-efficient than increasing channel numbers for improving text-image alignment. The quality and diversity of the training set prove more crucial than mere dataset size (Li et al., 2024b). These findings provide valuable insights for determining optimal model architectures and data requirements in both domains.
text-to-image generation. Studies show that
Inference Scaling Law Recent studies have revealed significant insights into inference scaling laws for large language models. The pioncering work 'Large Language Monkeys" (Brown et al., 2024) discovered that coverage (the fraction of problems solved) scales with the number of samples following a log-lineat relationship. Building upon this, self-consistency approaches demonstrated that sampling multiple reasoning paths and selecting the most consistent answer can substantially improve model accuracy (Wang et al., 2022). This was further enhanced by progressive-hint prompting techniques (Zheng et al., 2023), achieving significant gains various reasoning benchmarks. Recent theoretical work (Wu et al., 2024) shows that smaller models paired with advanced inference algorithms can outperform larger models under the same computation budget. However, studies on compound inference systems (Chen et al., 2024d) reveal that increasing L.LM calls shows non-monotonic behavior, performing better on "easy" queries but worse on "hard" ones. These findings collectively demonstrate the importance of optimizing inference strategies rather than simply scaling up model size or increasing the sampling budget. Concurrent works (Singhal al., 2025; Ma al., 2025) have also independently explored and validated the effectiveness of inference scaling in diffusion models.
B. Inference-Tine Scaling Details
Dataset, To finetune NVILA (Liu et al., 2024b) for SANA inference-time scaling, we generated 2.5M images and evaluated their alignment with the given prompts using the GenEval tooliit. We utilize 15,654 unique prompts and generate 160 images per prompt using Flux-Schnell Our prompts are constructed in style similar to GenEval, incorporating 80 object classes
13

---

SANA 1.5: Efficient Scaling of Training-Time and Inference- Compute in Linear Diffusion Transformer
Linear DIT Block
Linear DIT Block
Linear DIT Block
Linear DiT Block
Linear DIT Block
Linear DiT Block
Linear DIT Block
Linear DIT Block
Linear DIT Block
Linear DiT Block
Linear DiT Block
Linear DiT Block
Linear DiT Block
Linear DiT Block
Linear DIT Block
Linear DIT Block
Linear DiT Block
Linear DiT Block
Linear Block
Linear DiT Block
Linear DiT Block
Linear Block
Linear Block
Linear DIT Block
Weihh
Random-ine Weight
Linear DIT Block
Linear DIT Block
Linear DIT Block
Linear DiT Block
Linear DIT Block
Linear DiT Block
(a) Partial Preservation Init
(b) Cyclic Replication Init
(c) Block Replication Init
Figure 11. Illustration of initialization strategies. (a) Partial Preservation Init, which preserves the pre-trained layers and randomly initialize the new layers. (b) Cyclic Replication Init, which repeats the pre-trained layers periodically, (c) Block Replication Init, which extends each pre-trained layer into consecutive layers.
from Mask2Former along with attributes such as color, spatial location, quantity, and various relationships. Importantly, our prompts do not overlap with the GenEval set. evaluate the generated images, we leverage the GenEval toolkit. Finally, we train NVILA using the prompts, images, and their corresponding labels (yes no).
Training Setup. We follow the setting in NVILA with learning 2 x 10 Adam optimizer, cosine scheduler with warmup ratio 0.03 and batch size of 8 per device and train the 2M dataset with one epoch.
C. More Implementation Details
Scale
Scale
nxd
Attention with QK Norm As shown Figure 10, we intro- duce RMS normalization (Zhang & Sennrich, 2019) Query and Key in both linear attention's self-attention block and vanilla cross-attention module to stabilize the training of large diffusion models. Similar to the findings in (Esser et al., 2024), we observe that the attention logits in Rel.U-hased linear atten- tion (Cai et al., 2023) also grow uncontrollably and frequently exceed the numerical range of FP16 precision (6.5e5), which leads training instability (NaN). By incorporating QK nor- malization, effectively address this issue in large linear transformers. Notably, although 1. 1.6B was not initially trained with QK normalization, we also find that it quickly adapts to these additional normalization layers within just 1K fine-tuning step (Esser et al., 2024). This modification, combined with bf16 mixed precision and our proposed CAME-Sbir optimizer (Section 2.3), enables effi- cient scaling of linear transformer models while maintaining training stability.
MatMul
MatMul
nxx
dx
MatMul
Softmax
dxn
nxd
nxd
x
ReLU
ReLUU
RMSNoor
RMSNor
AMSNorm
AMSNorm
Linear
Linear
Linear
Linear
Linear
Linear
pretrained SANA-1.0
Text Embedding
(a). Linear self attention (b). Vanilla attention
Figure 10. Architecture design of linear self-attention and cross-attention blocks in SANA. Both attention blocks incor- porate RMSNorm on query and key for training large model more stable, where linear self-attention is used for content en- coding and vanilla cross-attention for text condition injection.
Multilingual Auto-labeling Pipeline In Figure 13, we present the results of our multilingual multi-caption auto-labeling pipeline. For each image, we use GPT-4 to translate small-scale data, only 100K English prompts, into: pure Chinese, English-Chine Chinese mixed, and emoji-enriched text. This approach enables to build comprehensive multilingual training
14

---

SANA 1.5: Efficient Scaling of Training- Training-Time and Inference-Tine Compute in Linear Diffusion Transformer
STNG 4.8 hoin Scathh SPN4--486 Node Orowh
CLI
DRC
S0306-4.6 4.06 from Scorch SAU---L0 Nodel Crowc
SAW 4.6 hro Searth
kodel Growd
602
654
D00
12Ck
100k
-04
40c
120k
140k
L00k
Training Steps
Tratning Stops
(a). FID Score
(b). CLIP Score
(c). DPG Score
Figure 12. Performance comparison of model growth and training from scratch across FID, CLIP score, and DPGBench metries. Our model growth strategy demonstrates superior performance over training from scratch, achieving either better results within the training duration or equivalent performance with approximately 60% less training time.
dataset that captures diverse ways of describing the same visual content. More results are shown in Figure 14. As result, fine-tune SANA with only few iterations~~10K), then it demonstrates more stable and accurate outputs for Chinese text and emoji expressions.
Comparison of Different Initialization Strategies We illustrate the three types of initialization strategies in Figure 11. Partial Preservation Init preserves the pre-trained layers and randomly initialize the new layers (Figure 1a()).. Cyclic Replication Init repeats the pre-trained layers periodically (Figure 11b)).. Block Replication Init extends each pre-trained layer into consecutive layers (Figure I1(c).. Among these strategies, adopt the partial preservation initialization approach for its simplicity and stability. Empirically, this approach provides the most stable training dynamics compared to cyclic and block expansion strategies, as shown in Figure 7.
D. More Results
Model Growth Results As shown in Figure 12, our model growth strategy consistently outperforms training from scratch across FID, CLIP score, and DPG benchmarks. Specifically, our approach achieves better quality within the same training duration reaches equivalent quality with an approximately 60% reduction in training time compared to training from scratch.
Comparison between different pruned model sizes As shown in Figure 16, compare different sizes of SANA-1.5 and LAN models. Starting from SANA-1.5 4.8B model (GenEval score 0.693), pruned variants maintain strong accuracy with 3.2B (0.684) and 1.6B (0.672) parameter counts, consistently outperforming SANA-1.0 1.6B (0.665). This flexible pruning approach allows us to obtain models of any desired size while preserving quality. In particular, larger models demonstrate superior capabilities in image details, pixel quality, and semantic alignment.
More Visualization Images In Figure 18, show more images generated by our model with various prompts. SANA demonstrates comprehensive generation capabilities across multiple aspects, including high-fidelity detail rendering, accurate semantic understanding, and reliable text generation. The samples showcase the model's versatility in handling diverse scenarios, from intricate textures and complex compositions to accurate text rendering and faithful prompt interpretation. These results highlight the robust image quality of the model in both artistic and practical generation tasks.
More Inference-Time Scaling Examples We provide additional inference-time scaling examples in Figure 17. During the tournament, VLM judges and filters prompt-mismatching images. We highlight winners with bold green lines and include the winning rationale. Images with incorrect object counts (c.g., the 4th image in Figure 17b) lose the comparison and are filtered by VLM. When two images match the prompt with similar quality, VLM fairly judges that "Both images match the prompt" shown in Figure 17a and selects one based on preference. Therefore, 1A inference scaling effectively filter those "bad" generations and improves GenEval scores.
Prompt Rewrite Enhancement As discussed in (Wang et al., 2024: Han et al., 2024), at inference time, we employ GPT-4o to rewrite user prompts by adding more details, which leads to richer and more detailed visualization results. This
15

---

SANA 1,5: Efficient Scaling of Training- Training-Time and Inference-Tinee Time Compute in Linear Diffusion Transformer
GPT-4
User
You will be provided with an image description. You have three separate goals. The first one is to translate it into Chinese. Try use simple words that native Chinese would say.
The second one to replace nouns verbs in the original description with emojis possible, You should English. replaced word Only verbs and nouns are replaced.
ensure non-emoji
appear together replacement.
The third one is to create description of mixture of English, and Chinese words and emojis. You cen pick the Chinese or English version of description, then replace only verbs and nouns with the other language or emojis possible. The replaced word should not sppear together with replacement. Only verbs and nouns replaced.
You will output JSON object containing the following information:
"Chinese": "estrings", Chinese description "Mix": "<stringa" // mixture description "Emoji": "<string>", emoji description
output anything else.
Original: Illustration of an office worker dressed beach wear and surfing pants, reclining on an office chair, looking at his big desktop setup his office desk, relaxed and happy body posture, with straw hat, office environment, illustrated style children's book, Illustration
T カ E. $8.99 d 2 30. #U #
Chinese: - II
Mix: A I A S सt beach wear HI surfing pants, # office chair F. * t x desktop
setup #b office desk, X6 62 happy 89 15. $2 # beach straw hat, office environment, 18 children's
book (E E9
illustration of X dressed relaxed and happy body posture, with book, Illustration
Emoji:
and M, reclining fh, looking at his setup his office environment, illustrated in the
children's
Figure 13. Illustration of our multi-lingual prompt translation pipeline. We leverage GPT-4 to translate 100k English prompts into four formats: (I) Pure English (2) Pure Chinese (3) English- Ceglish-Chhieee mixture (4) Emoji-mixed prompts. Example shows single English prompt translated into these parallel versions, demonstrating how we construct our multi- multi-lingual training data.
Original: cute black swan surrounded by colorful flowers, depicted in the style of an oil painting with vibrant colors and textured brush strokes. The background is filled with various blossoms of different sizes and hues, creating lively atmosphere that complements the adorable swan's is exp ressioon.. Chinese: R 2
H BE ff 72 (a
tp
R.
Ad =
Mix:
i BAR oil painting
#
#
3
$ Fi iो / T
lively L.
#E
2 f expression,
depicted in the style of an oil painting with vibrant colors and textured brush strokes. The background is filled with various of different sizes and hues, creating lively atmosphere that complements the adorable io expression.
Emoji:
A cute surrounded by
Figure 14, More illustration multi-lingual dataset.
16

---

SANA 1.5: Efficient Scaling of Training-Time
Time and Inference- Time Compute in Linear Diffusion Transformer
without multi- sft
with multi-lingua
without multi-lingual sft
with muitii-inggullst sft
Engligh/Emoji teaching & catch W
Engligh/Chinese/Emmg) Mix: Wearing flying on the ML with in the
Figure 15. SANA's multi-lingual capabilities unlocked through efficient fine-tuning. Comparing image generation quality be- tween baseline model (left, English-only training) and our model (right, fine-tuned with 100k multi-lingual samples) on mixed En- glish/Chiness/emooj prompts.
demonstrates the importance of prompt engineering in maximizing model capabilities. The comparisons are shown in Figure 19.
E. Discussion of Potential Misuse of SANA-1.5
Misusing generative Al models to generate NSFW content is a challenging issue for the community. To enhance safety, we have equipped SANA-1.5 together with a safety check model (e.g.. ShieldGemma-2B (Zeng et al., 2024b)). Specifically, user prompt will first be sent the safety check model to determine whether it contains NSFW(not safe for work) content. If the user prompt does not contain NSFW, will continue be sent to SANA-1.5 generate an image. If the user prompt contains NSFW content, the request will be rejected. After extensive testing, found that ShieldGemma perfectly filter out NSFW prompts entered by users under strict thresholds, and our pipeline will not create harmful Al-generated content.
17

---

SANA 1,5: Efficient Scaling of Training- Training-Time and Inference- Time Compute in Linear Diffusion Transformer
SANA 1.5 (4.8B) GenEval 0. 693
SANA 1.5 (3.2B)
SANA 1.5 (1.6B)
SANA 1.0 (1.6B)
Prompt
GenEval 0.684
CenEval 0.672
GenEval 0.664
An entique botenical Illustration drawn with fine lines and touch of watercotour whimsy...
Pirate ship trapped in cosmic mactstrom nebula, rendered cosmic beach whirlpool engine...
Color photo of a corgi made of transparent glass, standing on the riverside Yosemite National Park...
Figure 16. Comparison among different sizes of SANA 1.5 and SANA 1.0. With model scaling and pruning. SANA 1.5 achieves better performance than SANA 1.0 of the same size, while maintaining flexibility in model capacity selection. Larger models demonstrate enhanced capabilities in detail rendering, image quality, and semantic alignment.
18

---

SANA 1,5: Efficient Scaling of Training- Training-Time and Inference-Tinee Compute in Linear Diffusion Transformer
VIM Lddg
VCM 100e
VIM Ludg
(a) Prompt: photo of purple suitcase and orange pizza
jkdge
juege
(b) Prompt: photo of orange tennis racket and yellow sports ball
Figure 17. Visualization of SANA-115 inference-time scaling, During the tournament, VLM judges and filters nompt-mismatching images.
19

---

SANA 1.5: Efficient Scaling of Training- Time and Inference-Timee Compute in Linear Diffusion Transformer
AGI IS COMING
A-bund vil congor wed
pertcctaa setween Soc di diking sad scht
for cigh paprte suon
SANA
TO THE MOON
SANA 1,5 1.5
And wnllect
Al
pranuee
now spocti tn siele shaue
bndicaas
case slc iraliticioo van't suss Io endeggiia to
the
2025
NVDA1000
1.90 1500"" chaggn cnooreti
witte ferte
shiing vidematn
The cap mnsss
sir neree The conoopatiio mentin cacasi with and dight oct 1e the S0 grg vrt tutes pase enach v/0 wr cesld
Pagiaair suut ahotng tii te citee pol glocc de uiuu plids Eunc turtine
tedur dls de beder in des
Figure 18. High-resolution image generation examples from SANA 1.5, showcasing its capabilities in the accurate prompt following, spatial reasoning, rendering, and aestheties different styles and aspect ratios.
20

---

SANA 1,5: Efficient Scaling of Training-Time Time and Inference- Time Compute in Linear Diffusion Transformer
Original (Left): photo of two sheep.
Rewrite (Right): pestoral scene captured vivid photographic detail. Two woolly ovine graze peacefully lush, verdant their rleecy gleaming the golden afternoon sunliht. The sheep head as If pausing to regard the camera with mild curiosity. Their large, soulful eyes convey sense of gentle tranquility, timeless serenity in simple rhythma
alight tilt its
patchwork emerald sage interspersed with delicate wildflowers in shades of lavender and buttercup yellow. In the distance, the silhouettes of rolling hillsides recede into a hazy azure horizon, creating a bucolic, pestoral tableau. This photographic portrait captures the inherent dignity and peaceful grace of these woolly companions.
grassy upon which they
Original (Left):
bench.
Rewrite (Right):
tranquil garden vignette by rustic wooden bench weathered by elements. bench's 's sturdy stretch in gently curving lines, their surface smooth by passage countless visitors seeking respite. Verdant vines flowering plants spill
edges, softening bench's rigid Dappled sunlight filters through the canopy overhead, casting warm glow and creating contemplation. simple, elegant design bench serves an understated yet focal paint, beckoning pause, and immerse themselves in the calming natural ambiance.
bursts of pastel petals. trailing inviting atmosphere for quiet
Figure 19. Visual comparison of image generation results before and after prompt enhancement. For each example, the left shows the result from the original simple prompt, while the right demonstrates the output with enhanced prompt, showing improved visual quality and richer details.
21


Reviews:
[{"review_id": "QV9f5W0UpI", "reviewer": "Reviewer_nKKe", "summary": "The paper introduces SANA-1.5, an efficient linear Diffusion Transformer for text-to-image generation. \n\nKey contributions include: \n\n(1) A depth-growth paradigm that scales models from 1.6B to 4.8B parameters, reducing training costs by 60%; \n\n(2) A technique called model depth pruning via block importance analysis for flexible compression; \n\n(3) An inference-time scaling strategy using VLM selection.\n\nThe experimental results show the proposed model's superior performance.\n\n## update after rebuttal\nThank the authors for their rebuttal. I have no more concerns and would like to keep my score (4: Accept).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### We sincerely appreciate your insightful feedback, thoughtful comments and kind words. We have carefully addressed all questions to the best of our ability and hope the revisions meet your expectations.\n\n### Q1: Will VLM-based Verifier be the Best and any Other Choices?\n\n**A1:** We choose VLM-based verifier mainly because the advantages in **context length and flexible number of images**. In the L252-256 of our submission, we discussed that \n> While popular models like CLIP (Radford et al., 2021) and SigLIP (Zhai et al., 2023) offer multi-modal capabilities, their small context windows (77 tokens for CLIP and 66 for SigLIP) limit their effectiveness. This limitation poses a problem since SANA requires long, detailed descriptions.\n\nWe want the verifier to be more general and can be scaled up to long descriptions and multi-images thus choose VLM as the backbone. The performance shows ideal scaling curve in our Figure 8.\n\n---\n\n### Q2: More Quantitative of VLM-based Inference Time Scaling\n\n**A2**: While we have presented the quantitative results in Table 2 and Figure 8, we further **applied the inference-time scaling to SD1.5 to validate** the generalization capability. \n\nWe have iterate a version of VLM-verifier with more data and better training setting and with the new VLM-verifier, we achieved a remarkable improvement in **SD1.5's GenEval score from 0.42 to 0.87**. When combined with our more advanced SANA-1.5 4.8B model (0.72 GenEval), this approach reaches **0.95** GenEval score, setting a new state-of-the-art on GenEval.\n\n| Method                  | **Overall** | Single | Two  | Counting | Colors | Position | Color Attribution |\n|-------------------------|---------|--------|------|----------|--------|----------|-------------------|\n| SD1.5              | **0.42**    | 0.98   | 0.39 | 0.31     | 0.72   | 0.04     | 0.06              |\n| SD1.5 + Inference Scaling     | **0.87**    | 1.00   | 0.97 | 0.93     | 0.96   | 0.75     | 0.62              |\n| SANA-1.5 4.8B        | **0.72**    | 0.99   | 0.85 | 0.77     | 0.87   | 0.34     | 0.54              |\n| SANA-1.5 4.8B + Inference-Time Scaling     |   **0.95**        | 1.00 | 1.00 | 0.95 | 0.96 | 0.91 | 0.85\n\n---\n\n### Q3: Efficiency-Cost Tradeoff of VLM verifier\n**A3:** This is a good question and we list the FLOPs of each model \n\n* SANA 0.6B model: N x 261G*20(steps)=N x 5,220G\n* SANA 1.6B model: N x 704G*20=N x 14,040G\n* SANA 4.8B model: N x 2457G*20=N x 49,140G\n* NVILA-verfier-2B: (N-1) x 4,518G\n\nwhere $N$ is the numer of generated images. We notice that SANA-0.6B with 16 generation scaling is better than SANA-4.8B without verification in both performance and FLOPS.\n\n| Method+Samples          | FLOPs    | GenEval |\n|-------------------------|----------|---------|\n| SANA-0.6B + 16 images   | 151,290G | 0.75    |\n| SANA-4.8B + 4 images    | 196,560G | 0.72    |\n\n---\n\n### Q4: Writing Suggestion and Visualization\n**A4:** Thanks to the reviewer's carefullness, we will revise accordingly:\n\n* We will change the title to from \"Inference Scaling Law\" to \"Inference Time Scaling\". \n* We will polish the tables and figures using consistent style for better presentation. \n* We will add a visualization figure comparing with other SoTA methods with the same prompts, including some real-world noisy or long text prompts. We prepare a preview for all reveiwers here: https://www.filemail.com/d/xgaxjwwlpdjttkx\n\n---\n\n### Q5: Extension to Video/3D\n**A5:** This is an exciting direction we're exploring. Specifically, we can directly add an additional temporal attention layer in each block and further fine-tune with video data, while preserving the linear attention block and cross attention blocks. In this way, we can better utilize our pre-trained text-to-image model and try to make the training of text-to-video model more efficient. We'll try our best to give more answers in our future study in these fields.", "claims_and_evidence": "Most claims made in the paper are generally supported by clear and convincing observations and experiments. \n\nThe authors have presented sufficient qualitative and quantitative results to support their main contributions.\n\nOne concern is that more results and experiments are required. to verify the effectiveness of the proposed inference-time scaling strategy.", "methods_and_evaluation_criteria": "The proposed methods are well-designed and appropriate for address text-to-image generation.\n\nIt provided some insight to push the boundaries of this fields.", "theoretical_claims": "This paper doesn't contain any complex proofs and the math notations are correct so far.", "experimental_designs_or_analyses": "The experimental designs are generally sound and align well with its claims.\nEvaluations on benchmarks such as GenEval, FID, and CLIP score support the results, with practical metrics like memory usage and efficiency included.\nIt would be better if the authors can provide some qualitative comparison with other models based on the same text prompts.\nMoreover, the the VLM-based inference-scaling method need to be further verified.", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\n- The overall improvement is intuitive, which also maintains the elegance of the framework.\n- The writing is easy-to-following. The figures and tables are all good. \n- It is good to see that the overall image quality of pruned models can be easily recovered after brief fine-tuning (Figure 4).\n\nWeaknesses:\n- Please see other sections for weaknesses.", "other_comments_or_suggestions": "- It would be better if you can align the styles of different tables and figures for a more cohesive and polished presentation.\n- Using \"Inference Scaling Law\" in Line 370 is a little bit overclaimed.\n- Incorporating human evaluations based on some real-world noisy or long text prompts would be better.", "questions_for_authors": "- Could you provide more quantitative results to show the claims of Figure 3? Why the adopted VLM-based method is best and in what scenarios(or let's say, situations) is it the best scheme?\n- Generating multiple images per prompt increases inference costs. Have you quantified the additional computational overhead compared to the efficiency gains achieved by using a smaller model? Is there a clear analysis of this trade-off?\n- Have you considered extending the whole paradigm to other fields like text-to-video or text-to-3D generation? Are there any further different aspects that need to be further dealed with?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YEOfoHBVAP", "reviewer": "Reviewer_fWyY", "summary": "The paper introduces SANA-1.5, which incorporates a series of techniques to efficiently scale up the SANA-1.0 linear-attention diffusion Transformer. Specifically, it proposes a depth-growth paradigm that includes partial-layer preservation initialization and a memory-efficient CAME-8bit optimizer. Additionally, the paper presents a grow-then-prune-and-tune approach to reduce the number of layers and an inference-time scaling strategy that leverages VLM models. The proposed paradigm demonstrates a highly efficient scaling path, significantly reducing training compute while maintaining image generation quality.\n\n**Update after rebuttal**: Based on the reviews, rebuttal, and discussions, I am keeping my final rating as an Accept. In addition to the strengths highlighted in my initial review, the additional information and experiments provided in the rebuttal further solidify the contributions of this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### We sincerely appreciate your insightful feedback, thoughtful comments and kind words. We have carefully addressed all questions to the best of our ability and hope the revisions meet your expectations.\n\n### Q1: Growth-then-Prune vs Train-from-Scratch\n**A1:** Direct training of 30/40/60-layer models is prohibitively expensive and we are unable to finish training due to limited rebuttal time window (**typically requiring more than two weeks**). However, we have conducted a rigorous comparison between growth-then-prune and train-from-scratch approaches using our 1.6B-parameter (20-layer) models as a representative case study.\n\nThe results, shown in the table below, demonstrate that even with identical training configurations, the 1.6B model derived from our 4.8B growth-then-prune pipeline consistently outperforms its directly-trained counterpart across multiple metrics, e.g., FID (**28.67→29.08**), CLIP score (**28.67→29.08**), and GenEval (**0.66→0.67**). Notably, the SANA-1.0 1.6B is fully trained, and further training will not improve the metrics.\n\nWe believe these benefits would scale consistently to larger configurations (30/40/60 layers), as the fundamental advantages of our approach - superior initialization and selective compression - remain architecture-agnostic. The computational savings (60% training cost reduction) further demonstrate our method's practical superiority for scalable model training.\n\n\n| Method        | FID  | CLIP  | GenEval | DPG  |\n|---------------|------|-------|---------|------|\n| SANA-1.0 1.6B | 5.76 | 28.67 | 0.66    | 84.8 |\n| SANA-1.5 1.6B | 5.70 | 29.08 | 0.67    | 84.8 |\n\n---\n\n### Q2: Stability Quantification for Different Initalizating Strategies\n**A2:** As demonstrated in Figure 7 and the accompanying table, the initialization strategy critically impacts training stability. Our experiments show that suboptimal initialization frequently causes training instability, manifesting as: (1) numerical instability (NaN values) during optimization, and (2) failure to produce meaningful quantitative results.\n\n\n| Method                    | 3k Iterations Training Loss |\n|---------------------------|-----------------------------|\n| Cyclic Replication Init   | NaN                         | \n| Block Replication Init    | NaN                         | \n| Partial Preservation Init | 0.835                       | \n\nThe identity mapping initialization we employ represents the current standard practice for model growth scenarios, as evidenced by[1-3]. This initialization scheme effectively addresses the vanishing/exploding gradient problems that commonly plague direct training of large architectures, while maintaining the model's representational capacity at the beginning of the growth process.\n\n\n[1]Han S, Mao H, Dally W J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding[J]. arXiv preprint arXiv:1510.00149, 2015. \n\n[2]Chen, T., Goodfellow, I., and Shlens, J. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641, 2015.\n\n[3]Karras T, Aila T, Laine S, et al. Progressive growing of gans for improved quality, stability, and variation[J]. arXiv preprint arXiv:1710.10196, 2017.", "claims_and_evidence": "The paper’s central claim regarding the high training and inference costs of existing visual generation models is both critical and practical, particularly as large-scale models become essential for high-quality generation yet remain inaccessible to many practitioners due to computational constraints. This work is well-motivated, and the proposed paradigm offers a comprehensive and feasible solution well-supported by empirical results. It addresses key research questions, such as \"How is the scalability of linear diffusion Transformer?\" and \"How can we scale up large linear DiT and reduce the training cost?\", and presents a practical and inspiring approach with implications for future research and applications.", "methods_and_evaluation_criteria": "The proposed techniques, partial-layer preservation initialization, the memory-efficient CAME-8bit optimizer, the grow-then-prune-and-tune approach, and inference-time scaling, are all well-motivated and highly effective. Together, they form a comprehensive framework for efficiently scaling up the model.\n\nThe evaluation primarily consists of detailed analyses and ablations on various design choices, including different initialization strategies, comparisons between the CAME-8bit optimizer and CAME, AdamW, and AdamW-8bit, as well as the impact of high-quality data fine-tuning, model block importance, and the grow-prune-tune approach on computational cost and output quality. This thorough and systematic analysis makes the findings compelling. Additionally, the paper includes comparisons with existing methods on widely applied benchmarks such as GenEval, DPG, and MJHQ, clearly demonstrating the effectiveness of the proposed techniques.", "theoretical_claims": "Apart from the proposed techniques discussed earlier, which are well-supported by experiments, other key claims and design choices, such as the block-wise quantization strategy and the hybrid precision design of the CAME-8bit optimizer, as well as the distinction between inference scaling and training scaling, are highly reasonable. I did not identify any critical correctness issues in the claims presented in this paper, at least within the reasonable scope of this paper.", "experimental_designs_or_analyses": "Please see my comment in Methods And Evaluation Criteria above. Overall, I am very satisfied with the thorough experiments on each argument and design choice made in this paper.", "relation_to_broader_scientific_literature": "This work builds upon existing literature and effectively integrates a diverse range of techniques into a comprehensive system. It extends prior works such as SANA-1.0 (Xie et al., 2024), DiT (Peebles & Xie, 2022), CAME (Luo et al., 2023), and block-wise 8-bit quantization (Dettmers et al., 2021), as well as various techniques detailed in Appendix A, including diffusion model pruning and training and inference scaling. The paper precisely cites and references relevant works. Overall, this work presents a novel and well-grounded approach.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "N/A.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "Only two non-critical questions:\n1. How is the grow-then-prune approach compared to directly training a 40/30/20 blocks model with the same initialization methods?\n2. Quantitative experiments on stability enhancement and identity mapping initialization could further support the claims in those sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KY5DTdaVsk", "reviewer": "Reviewer_mxDw", "summary": "This paper proposes three important components: 1. efficient model scaling with depth, 2. efficient model depth pruning, and 3. inference time model performance scaling with VLM. Those three components show that an efficient training strategy can enable a smaller model to achieve performance comparable with the larger models for linear DIT.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### We sincerely appreciate your insightful feedback, thoughtful comments and kind words. We have carefully addressed all questions to the best of our ability and hope the revisions meet your expectations.\n\n### Q1: Efficiency-Cost Tradeoff of VLM verifier\n**A1:** This is a good question and we list the FLOPs of each model \n\n* SANA 0.6B model: N x 261G*20(steps)=N x 5,220G\n* SANA 1.6B model: N x 704G*20=N x 14,040G\n* SANA 4.8B model: N x 2457G*20=N x 49,140G\n* NVILA-verfier-2B: (N-1) x 4,518G\n\nwhere $N$ is the numer of generated images. We notice that SANA-0.6B with 32 generation scaling is better than SANA-4.8B without verification in both performance and FLOPS.\n\n| Method+Samples          | FLOPs    | GenEval |\n|-------------------------|----------|---------|\n| SANA-0.6B + 16 images   | 151,290G | 0.75    |\n| SANA-4.8B + 4 images    | 196,560G | 0.72    |\n\n---\n\n### Q2: Add more good references\n**A2:** Thanks for noticing the reference we should have. We will add SnapGen[1] (please correct if you mean other projects) to our efficiency related work.\n\n[1] Hu D, Chen J, Huang X, et al. SnapGen: Taming High-Resolution Text-to-Image Models for Mobile Devices with Efficient Architectures and Training[J]. arXiv preprint arXiv:2412.09619, 2024.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes, they all look good to me.", "experimental_designs_or_analyses": "The experimental designs are sufficient.", "relation_to_broader_scientific_literature": "Previous works have demonstrated that linear transformer can be applied well on the T2I generation, but this paper summarizes previous works on scaling efficient linear DiT.", "essential_references_not_discussed": "Recently, there is one work demonstrated that around 300M on-device model can also achieve 0.55 GenEval score on the device. This might also be a good work to refer.", "other_strengths_and_weaknesses": "Strengths:\n\nThis paper proposes how to scale efficient training and inference for linear DiT and make it comparable with the much larger models. The experiments efficiently show that the proposed method is effective.\n\nWeakness:\nIt might not be a weakness, but this paper proposes those methods without novelty but works pretty well.", "other_comments_or_suggestions": "NAN", "questions_for_authors": "For the inference-time scaling, how is the speed compared with the larger models with the same quality.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8xGELFrO8e", "reviewer": "Reviewer_Z15v", "summary": "This paper introduces SANA 1.5, aims at scaling up the linear diffusion transformers from the training and inference perspectives. Based on a pre-trained linear diffusion model. This paper proposes three techniques, Efficient Training Scaling, Model Depth Pruning, and Inference-time Scaling to investigate the scaling question. Each technique is simple and effective. Besides, to further reduce the GPU memory cost during the large model training, SANA 1.5 proposes a memory-efficient optimizer CAME-8bit, and demonstrates its validity. Experimental results show SANA 1.5 is able to surpass some larger models on different metrics.\n\n## update after rebuttal\nI appreciate that the authors take time to explain the reason of choosing a deeper rather than a wider model, the experimental results of a from scratch training 20 layers model to address my concerns. And the discussion regarding whether removing the last two layers is instructive. The rebuttal has fully addressed my concerns. I will improve my score and support its acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### We sincerely appreciate your insightful feedback, thoughtful comments and kind words. We have carefully addressed all questions to the best of our ability and hope the revisions meet your expectations.\n\n### Q1: Scaling Depth v.s. Scaling FFN / channel dimensions\n**A1:** While conventional approaches typically scale both depth and width, **SANA-1.5 focuses exclusively on depth scaling to fully utilize existing learned representations.** Recent literature [1] supports this observation, demonstrating that for a fixed parameter budget, deeper models outperform wider architectures [2]. Our experiments in Figure 2 confirm this advantage, showing 60% faster convergence compared to training from scratch.\n\n[1] Wu C, Gan Y, Ge Y, et al. Llama pro: Progressive llama with block expansion[J]. arXiv preprint arXiv:2401.02415, 2024.\n\n[2] Petty J, van Steenkiste S, Dasgupta I, et al. The impact of depth on compositional generalization in transformer language models[J]. arXiv preprint arXiv:2310.19956, 2023.\n\n---\n\n### Q2: 3.2B model's GenEval Results if Trained from Scratch\n**A2:** Direct training of 30/40/60-layer models is prohibitively expensive and we are unable to finish training due to limited rebuttal time window (**typically requiring more than two weeks**). However, we have conducted a rigorous comparison between growth-then-prune and train-from-scratch approaches using our 1.6B-parameter (20-layer) models as a representative case study.\n\nThe results, shown in the table below, demonstrate that even with identical training configurations, the 1.6B model derived from our 4.8B growth-then-prune pipeline consistently outperforms its directly-trained counterpart across multiple metrics, e.g., FID (**28.67→29.08**), CLIP score (**28.67→29.08**), and GenEval (**0.66→0.67**). Notably, the SANA-1.0 1.6B is fully trained, and further training will not improve the metrics.\n\nWe believe these benefits would scale consistently to larger configurations (30/40/60 layers), as the fundamental advantages of our approach - superior initialization and selective compression - remain architecture-agnostic. The computational savings (60% training cost reduction) further demonstrate our method's practical superiority for scalable model training.\n\n| Method        | FID  | CLIP  | GenEval | DPG  |\n|---------------|------|-------|---------|------|\n| SANA-1.0 1.6B | 5.76 | 28.67 | 0.66    | 84.8 |\n| SANA-1.5 1.6B | 5.70 | 29.08 | 0.67    | 84.8 |\n\n---\n\n### Q3: Is the Model of Figure.5 (b) and \\(c\\) Trained to Convergent?\n**A3:** As shown in Figure 2 and Appendix Figure 11, the training curves for GenEval, FID, CLIP-Score, and DPG-Bench all **confirm stable model convergence** throughout training. The block importance score variations originate from our layer-specific initialization strategies (Section 3.2), not from convergence problems.\n\n---\n\n### Q4: As Written in Line315, the Author Removes the Last Two Blocks. What if Do not Removing These Two Blocks and Adding New M Layers Before These Layers?\n**A4:** That is a good discussion. We share the insights behind our design: considering that new blocks use **identical mapping initialization**, adding new M blocks after 20 blocks (1.6B) model should be same as adding new M blocks between 18 blocks and the last two blocks. However, when adding new layers without removing last two blocks, they will tend to keep original representation, and hardly to learn from scaling. By removing the last two blocks, the balance of distribution is broken and it will encourage the new layers to learn from scaling. \n \nOur block importance analysis in the table below reveals that initializing from all 20 blocks results in near-zero importance scores. Following our importance metric definition - where higher values indicate more learned knowledge - these results empirically validate our claim (Lines 155-158): **removing and adding will encourage the newly added layers to easily learn new knowledge when scaling model size**. \n\n|Average Block Importance | 1-10 Blocks | 11-20 Blocks | 21-30 Blocks | 31-40 Blocks | 41-50 Blocks | 51-60 Blocks |\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- | \n| Initialize from 20 Blocks |6e-3 | 7e-4 | 2e-7 | 2e-7 | 1e-7 | 1e-7 |  \n| Initialize from 18 Blocks | 4e-3 | 5e-4 | 3e-5 | 2.5e-5 | 2e-5 | 2e-5|", "claims_and_evidence": "The paper claims that both training and inference have scaling properties, but it only show the results of one model scale (1.8B) directly trained from the original 1.6B model. And this paper does not give the log-linear scaling curve between the sampling time (or other metric) v.s. the GenEval (or other metric).", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A, no proofs in this paper.", "experimental_designs_or_analyses": "1. For the model configuration, the paper uses the same channel dimension and FFN dimension for different model size (Line 247-Line251), which is not a common choice. \nThe author claims that directly train the larger models with this configuration from scratch is worse than that using the paper proposed techniques. Thus, is it possible that, using a more common model configuration, the model training from scratch is comparable or even better than the SANA 1.5?\n2. In table 3, what is the GenEval result of directly train the model with 3.2B parameter using the technique of Efficient Model Growth?\n3. As written in Line315, the author removes the last two blocks. What if do not removing these two blocks and adding new M layers before these layers?", "relation_to_broader_scientific_literature": "This paper is related to the recently popular test time scaling and accelerated diffusion model.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strength:\n1. The paper is well written and easy to follow.\n2. The problems and contents studied in this paper are of practical application value.\n3. The performance is good.\nWeakness: Besides the contents in Claims And Evidence and Experimental Designs Or Analyses\n1. In figure5(c), it seems that the importance score of newly added layers in SANA 4.8B is significant lower than the original layers in SANA 1.6B. Besides, in figure5(b), the importance scale distribution of SANA-4.8 B is  very different with the SANA 1.6B in figure5(a). A natural question is: Does the model of figure5 b and c trained to convergent?", "other_comments_or_suggestions": "No", "questions_for_authors": "See Other Strengths And Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: SANA-1.5 introduces an efficient linear DiT for text-to-image generation, incorporating techniques such as efficient training scaling, depth pruning, and inference scaling. The paper presents promising generation results with significantly reduced training costs. The authors actively engaged in rebuttal and discussion with reviewers. All reviewers unanimously voted for acceptance. The paper is recommended for acceptance, and the authors are encouraged to include the reviewer-author discussion in the final version.
Decision: Accept (poster)