SPRINT: Sparse-Dense Residual Fusion for Efficient Diffusion Transformers
Authors: Dogyun Park
Moayed Haji-Ali
Yanyu Li
Willi Menapace
Sergey Tulyakov
Hyunwoo J. Kim
Aliaksandr Siarohin
Anil Kag
Venue: ICLR 2026 Poster (2026)
Keywords: diffusion models
generative models
flow matching
efficient training
image generation
Primary area: generative models
TL;DR: Efficient diffusion transformer training

Abstract: Diffusion Transformers (DiTs) deliver state-of-the-art generative performance but their quadratic training cost with sequence length makes large-scale pretraining prohibitively expensive. Token dropping can reduce training cost, yet naïve strategies degrade representations, and existing methods are either parameter-heavy or fail at high drop ratios. We present SPRINT (Sparse--Dense Residual Fusion for Efficient Diffusion Transformers), a simple method that enables aggressive token dropping (up to 75%) while preserving quality. SPRINT leverages the complementary roles of shallow and deep layers: early layers process all tokens to capture local detail, deeper layers operate on a sparse subset to cut computation, and their outputs are fused through residual connections. Training follows a two-stage schedule: long masked pre-training for efficiency followed by short full-token fine-tuning to close the train--inference gap. On ImageNet-1K 256^2, SPRINT achieves 9.8x training savings with comparable FID/FDD, and at inference, its Path-Drop Guidance (PDG) nearly halves FLOPs while improving quality. These results establish SPRINT as a simple, effective, and general solution for efficient DiT training.

Published as a conference paper at ICLR 2026
SPRINT: SPARSE-DENSE RESIDUAL FUSION FOR EFFICIENT DIFFUSION TRANSFORMERS
Dogyun Park 1.2+
Moayed Haji-Ali'
Yanyu Li'
Willi Menapace'
Sergey Tulyakov
Hyunwoo J. Kim
Aliaksandr Siarohin' Anil Kag*
'Snap Inc. Korea University RKAIST
Project page: https///snnp-rssearhhggiiibbbbb
ABSTRACT
Diffusion Transformers (DiTs) deliver state-of-the-art generative performance but their quadratic training cost with sequence length makes large-scale pretraining pro- hibitively expensive. Token dropping can reduce training cost, yet naïve strategies degrade representations, and existing methods either parameter-heave high drop ratios. We present SPRINT (Sparse-Denn) Dense Residual Fusion for Efficient Diffusion Transformers), simple method that enables aggressive token dropping (up to 75%) while preserving quality. SPRINT leverages the complementary roles of shallow and deep layers: early layers process all tokens capture local detail, deeper layers operate on a sparse subset to cut computation, and their outputs are fused through residual connections. Training follows two-stage schedule: long masked pre-training for efficiency followed by short full-token fine-tuning to close the train-inference gap. On ImageNet- 2562, SPRINT achieves 9.8x training savings with comparable FID/FDD, and at inference, its Path-Drop Guidance (PDG) nearly halves FLOPs while improving quality. These results establish SPRINT as simple, effective, and general solution for efficient DiT training.
fail
INTRODUCTION
OF
DIT M
Progrestvve Twining
DOSOBO TuBe
100
MicroDil
TREAD
MiskDIT
OIT BACK
FDD
60
DIT
9.% Faster
MDTV2
SIT
SPRINT T (Ours)
Faster
60
SPRINT-REP. (Curs)
Sedia
50
100
150
200
250
300
350
400
Training GFL.P%4 (x10")
Vanilla DIT
SPRINT (Ours)
(c) Training FLOPs
Figure 1: dense residual fusion improves the efficiency of diffusion transformer training. SPRINT decouples the computationally heavy middle blocks of DiT into a sparse-deep path and dense-shallow residual path. Notably, SPRINT achieves up to 5.6x and 9.8x lower training cost compared to vanilla models, while improving generation quality.
Diffusion Transformers (DiTs) (Peebles & Xie, 2023; Esser et al., 2024b) have emerged as powerful class of generative models (OpenAI, 2024; Labs, 2024a). Yet their training cost scales quadratically with sequence length, making large-scale pretraining prohibitively expensive in compute and memory. natural way to reduce training cost to shorten sequences by dropping tokens during training. However, naïve token dropping (Sehwag al., 2025) degrades representations and leads to poor generalization when models are evaluated with full-token inputs at inference.
Work done during internship Snap Inc.

---

Published as a conference paper at ICLR 2026
Another direction is to guide DiTs with external supervision. For instance, REPA (Yu et al., 2024) aligns intermediate DiT features with DINOv2, accelerating convergence. However, such auxiliary losses can harm long- long-term performance destabilize training (Wang et al., 2025), since pre- mre-trained vision features are not naturally aligned with diffusion's iterative denoising. Recent work (Zheng et al., 2024; Gao et al., 2023) has explored more advanced token-dropping strategies. While promising, these methods either add substantial parameters (Sehwag et al., 2025) or only support moderate drop ratios (Krause et al., 2025; Zheng et al., 2024), and break down under aggressive settings (e.g.. 75%).
In this work, we present a training algorithm that enables high-ratio token dropping while preserving robust, semantically meaningful representations that transfer effectively to full-token fine-tuning, Our design philosophy is to train DiTss efficiently with minimal architectural changes, achieving perfor- mance on par with-or better than-strong baselines. The core idea is to exploit the complementary roles of shallow and deep layers in neural networks: shallow layers capture fine-grained local details, while deeper layers model global semantics. However, standard DiT training, deeper layers often waste computation redundant local details that contribute little to modeling global semantics, due to the homogeneous architecture of DiTs. This redundancy significantly slows training convergence and reduces efficiency. We demonstrate that reformulating the architecture and coupling it with principled token- boken-dropping strategy resolves this issue.
Our Solution. We introduce Sparse-Dente Residual Fusion for Efficient Diffiusion Transformers (SPRINT), simple strategy that enables aggressive token dropping while preserving representation quality. Specifically, we partition the DiT into three components: encoder, middle blocks, and decoder. The encoder processes all tokens to encode local information, producing dense shallow features. Before the middle blocks, we drop most tokens (typically 75%), forcing deeper layers to focus on sparse global context with far lower compute, making sparse deep features. Simple residual fusion mechanism then combines dense shallow features with sparse deep features, while dummy masking tokens ensure dimensional alignment, and the fused representation is passed to the decoder.
Training proceeds in two stages. First, we perform long pre-training with 75% token dropping. yielding large compute savings. Then, a short fine-tuning stage restores full-token processing in the middle blocks, allowing them to adapt to dense inputs and closing the train-inference gap. Training uses the standard diffusion loss, and the DiT block design remains unchanged, making SPRINT easy to integrate into existing codebases.
Notably, the dual-path structure of SPRINT (dense shallow and sparse deep) enables surprisingly efficient guidance sampling strategy, which we denote as Path- Drop Guidance (PDG). Standard classifier-fre guidance requires two full forward passes of the model to compute conditional and unconditional estimates, thereby doubling inference cost. In contrast, under our framework, we can efficiently obtain the unconditional estimate by entirely bypassing the middle blocks and using only the dense shallow path. We demonstrate that PDG reduces the cost of guidance sampling by nearly 50% while improving generation quality.
Contributions. Our work makes the following key contributions:
We propose Sparse-Dent Residual Fusion (SPRINT), which fuses dense shallow and sparse deep features for efficient DiT training, supporting up to 75% token dropping and yielding large efficiency gains over prior methods (Tab. I, Fig. 3c).
We demonstrate faster convergence and improved efficiency on modern DiTs. On ImageNet- 1K 2562 class-conditional generation, SPRINT reduces training GFLOPs by 9.8x compared to standard SiT training while achieving similar or better quality (Fig. 1c. Tab. 3).
SPRINT provides new insights into DiT representations: our dense-shallow features are more noise-invariant and semantically expressive (Fig. 6); achieve higher CKNNA scores than vanilla DiT (Fig. 3b); and shallow versus deep paths specialize in local versus global semantics (Fig. 4).
We introduce Path-Drop Guidance (PDG), replacement for classifier-free guidance (CFG) that computes the unconditional pass using only dense shallow features. PDG nearly halves inference FLOPs while surpassing CFG in generation quality (Fig. 2, Tab. 3).
We show that SPRINT is simple, architecture agnostic, and complementary to alignment- based methods. It applies seamlessly across architectures (SiT, UVIT), latent spaces (SD, FLUX VAE), and resolutions (256, 512), and provides further gains when combined with REPA (Yu et al., 2024).
2

---

Published as a conference paper at ICLR 2026
REPA + CTG
Oun + CFG
P
REPA - CRG
Our- CPG
Ours+
REPA + CFG
Ours CPG
Oure PBG
FDO
FDU-756
POU-57.1
FD00-7688
E100-75.8
1130-57.1
F000-571
0.475
1FL0P5-0.274
T.70P
TEL0P-
TFLOP-00473
TFLOP
TFLOP
Figure 2: SPRINT improves visual quality over baseline with only 57% of inference FLOPs. We present samples from two SiT-XXL/2REPX models after 1M training iterations, where SPRINT is applied to one of the models. For our approach, we further incorporate the proposed Path-Drov Guidance (PDG), yielding improved FDD scores and higher visual quality compared vanilla REPA.
RELATED WORK
Accelerating DiT training via representation alignment. Several works accelerate DiT convergence by aligning internal features with pre-trained vision transformers. REPA (Yu et al., 2024) aligns intermediate DiT activations with DINOv2 features, while Lee et al. (2025) extend this text-image models via contrastive loss. Wang & He (2025) instead propose a dispersive loss that spreads features without external alignment. However, HASTE (Wang et al., 2025) shows that alignment signals can conflict with diffusion objectives and destabilize training. These objectives are complementary to our token droppigg scheme and can be combined to further boost performance (Tab. 2).
Efficient DiT training with token dropping. Another direction reduces training cost by shortening sequences. Progressive training (Podell al., 2024; Esser et al., 2024b) first pre-trains at 128x 128 before fine-tuning 256. MDTv2 (Gao al., 2023) restructures DiT into encoder- processing masked tokens with skip connections and optimizing both reconstruction and diffusion losses. MaskDiT (Zheng et al., 2024) drops random patches, replaces them with mask tokens, and trains an auxiliary decoder, which adds inference cost. MicroDiT (Sehwag et al., 2025) adds match-mixer for high masking ratios; and TREAD (Krause al., 2025) bypasses subsets of tokens through inner layers to optimize full denoising loss. These approaches work at moderate drop ratios (< 50%) but degrade at aggressive settings (e.g., 75%) and are difficult to pair with alignment losses. In contrast, our approach remains alignment-frienlly and robust even under high drop rates.
3 SPRINT: SPARSE-DENSH RESIDUAL FUSION FOR EFFICIENT DIFFUSION TRANSFORMERS
3.1 PRELIMINARIES
Diffusion and flow-based generative models. Diffusion and flow-based models (Ho et al., 2020; Song et al., 2020; Lipman et al., 2023; Liu et al., 2023) learn a continuous transformation between simple reference distribution m (e.g., Gaussian noise) and target data distribution To- Given Xg ~ To and X1 ~ Ti, the transformation evolves over t E [o, 1] by the ODE
dx dt
v uxx,, l),
where x, interpolates between xo and x1, and Rd [0, 1] - Rd is the velocity field. We use = O, and adopt linear schedule (Ma et al., 2024): ar - oe =t. A neural network v4 (e.g., DiT) learns v by minimizing
X₁ ~ N Vaaxx of with ag 0 = 1,
min E Exo. [xl 1) - ds (x,, ] 8
(2)
3

---

Published as a conference paper at ICLR 2026
90-
0.23
5AAM 199M
STAO e rea
SFTNT 437 584
80
SPAMIT 1795
CKNNA Value
0.21
70-
Grodient Norm
0.20
0.38
40-
26.1
0.36
SPENT DESS
SPANTISS (5.22))
0.35
-1
100
200
300
400
100
203
300
400
100
200
300
400
Training Iterations (x1K)
Training Iterations (x1K)
Training Iterations (x1K)
(a) 6 gradient norm of fo
(h) CKNNA(fo, DINOv2)
(C) FID-10K
Figure 3: Training behavior of diffusion transformers. We empirically analyze the training dynamics of SiT-B/2 and its SPRINT variants under different token-drop ratios. (a) We measure the l2 gradient norm of fe. showing that SPRINT enables the encoder to receive stronger gradient signals from the loss. (b) SPRINT variants achieve higher and carlier CKNNA scores than SiT, indicating SPRINT learns more semantic, noise-robust representations. (c) SPRINT converges substantially faster and to lower FID than SiT, with the gap further widening at higher drop ratios (up to 75%), highlighting both the effectiveness and efficiency of our framework.
Token dropping in diffusion transformers. Given noisy image x,, DiT divides it into overlapping p x p patches, producing tokens X₃ E Bxx where N HWW is the embedding g²
dimension, and H x W the image resolution. Since the attention cost in DiTs scales quadratically with N. dropping tokens reduces training cost. For a drop ratio т, we remove [r-]] tokens and process only the remaining N [rN] with DiT blocks. Although described for 2D images, this naturally extends other modalities such as video.
3.2 BOTTLENECK IN STANDARD DIT TRAINING
Standard Diffusion Transformers (DiTs) use a homogeneous architecture where every layer, from shallow to deep, processes the full set of dense tokens. This is inefficient: in deeper layers, token representations become redundant as features shift from local, high-frequency patterns to global, low-frequency semantics (Hoover et al., 2019; Voita al., 2019). Inference-time pruning and merging methods (Rao et al., 2021; Chang et al., 2023; Bolya & Hoffman, 2023) further show that large fractions of tokens can be removed in later layers with minimal effect on output quality. Training deep layers on all tokens thus wastes compute, spending large portion of the FLOP budget fine-grained details that contribute little to modeling global structure.
We address this by introducing architectural specialization: 1. Early layers process dense tokens to robustly capture local evidence under noisy input and build a rich foundation of features. 2. Deeper layers operate on a sparse subset of tokens to efficiently model global semantic relationships without redundant computation. 3. Final layers reintroduce all tokens for dense prediction. Based on these principles, reformulate the DiT architecture with dense-sparss fusion mechanism.
3.3 DENNS RESIDUAL FUSION
We propose Sparse Dense Residual Fusion for Efficient Diffusion Transformers (SPRINT), which decouples dense local details from sparse global semantics, improving efficiency by accelerating convergence and reducing compute. An overview is shown in Fig. 1. We begin with standard DiT, divided into encoder fe (first two blocks), middle blocks go, and decoder hg (final two blocks), and reformulate the computation flow as:
1. Encoder fo processes all noisy tokens to produce feature map that retains fine-grained local noise information across all spatial locations.
2. Dense shallow path creates residual connection that directly forwards the dense feature map from fo to the fusion block, preserving local, high-freqqeen detail.
3. Sparse deep path drops a large fraction of tokens (e.g., 75%) before go. forcing the deep layers to operate sparse subset, yielding sparse global context.
4. Fusion and decoder integrate dense local information from the shallow path with sparse global context from the deep path to predict all tokens.
4

---

Published as a conference paper at ICLR 2026
Formally, given imput tokens X, E IR BxxxC first compute dense features fr = fx()).. A fraction r of tokens (the drop ratio) removed form E, which is processed by the middle blocks: drop to the original sequence length gi drop go (f(())) To fuse the dense and sparse paths, we restore gt by padding the dropped positions with fixed [MASK] token (denoted M), yielding g! E BxXc We concatenate and gt along the channel dimension, project back to the original size, and feed the fused representation to the decoder hg. This enables the decoder to combine local details from the encoder with sparse global semantics from the middle blocks for full-token prediction. The entire model is trained end-to-end by minimizing the flow matching loss in Eq. (refer to Alg. 1).
Improving training efficiency with minimal modification. SPRINT improves training through two key mechanisms. First, it reduces per-iteration compute cost by restricting the expensive middle blocks go sparse token set, while the dense shallow path preserves fine-grained information. Unlike prior methods, it remains stable even under aggressive drop ratios (Fig. 3c) where others fail. Second, it accelerates iteration-wise convergence by enhancing contextual and relation learning: the decoder ha must predict all tokens despite most deep-path inputs being [MASK] tokens. This forces encoder (fo) and middle blocks (go) learn robust, context-aware features, as reflected in faster FID improvement (Fig. 3c), stronger gradient flow (Fig. 3a), and richer representations (CKNNA in Fig. 3b). These gains come with minimal architectural change: the standard DiT blocks remain intact, making SPRINT easy to integrate into existing codebases. Analysis details are in Appendix A.
Dense-shallow vs. sparse-deep features. The ablation Fig. highlights their complementary roles. The dense-shallow puth preserves local textures (e.g., feathers, skin patterns) but fails to form coherent global structure. The sparse-deey path captures global shapes (c.g., bird outline, shark body) but introduces severe texture artifacts. Fusing both yields high-quality outputs with realistic global semantics and fine local detail, showing that dense-sine shallow features encode local evidence while sparse-degs features capture global semantics.
Both path
sparse path
wro dense path
Both path
w/o sparse path
w/o dense path
Figure 4: Roles of dense-shallow and sparse-deep features. Dense-shallle features preserve local textures but lose global structure, while sparse-deep features capture global shapes but distort local details. Fusing both recovers high-quality outputs with coherent semantics and fine detail.
Fine-tuning with full tokens. After efficient sparse pre-training, we transition the middle blocks to operate on the full token set for a brief fine-tuning stage, addressing the potential train- inti-infeeence gap as demonstrated in prior works (Zhang al., 2024; Schwag et al., 2025; Krause et al., 2025) (refer to Alg. 2). Since pre-training typically dominates with MM-MM iterations, this fine-tuning phase is short (e.g., 100K -20K iterations), yet sufficient for the deeper layers adapt the full data distribution, ensuring high inference quality while retaining most of the pre-training speedup.
3.4 EFFICIENT PATH-DROP GUIDANCE (PDG)
SPRINT's dual-path design also enables efficient guidance during inference. Standard Classifier- Guidance (CFG) doubles sampling cost by requiring two forward passes per step: one conditional ve(x,, c) and one unconditional ve(x,, 0). Auto Guidance (Karras et al., 2024a) shows that the unconditional pass can be replaced by a weaker network. The SPRINT architecture inherently contains natural weaker network: the dense shallow path that bypasses the deep middle blocks. We therefore introduce Path-Drop Guidance (PDG): For the conditional estimate, we perform a full forward pass. For the unconditional estimate, bypass go entirely, replacing it with [MASK]
5

---

Published as a conference paper at ICLR 2026
Table 1: Training efficiency on ImageNet 256 Iteration- wise results of different poken-dropping methods with same 75% dropping rate. We report total training TFLOPs (using Deepspeed library) and performance with/without classifier-free guidance, along with SPRINT's relative gains over SiT (Gain A). All methods use 50 sampling steps with ODE sampler.
TFLOPs (x 10")
w/o CFG
CPG
1.4)
Method
AE
FDD
FID
IS
Pre.
Rec.
FDD J
FID.
18
Pre. 1
Rec.
400K training iterations
Improved SIT-XIJ2
SD
24.4
351.1
12.8
97.4
0.66
0.65
185.0
3.09
211.6
0.81
0.55
Progressive Training
SD
16.8
365.5
12.7
96.2
0.67
0.63
215.6
3.47
206.4
0.83
0.53
MDDTv
21.2
558.5
21.1
68.9
0.61
0.63
366.5
5.61
176.3
0.76
0.54
MicroDiT
SD
20.8
349.9
11.5
99.9
0.67
0.64
178.1
3.16
213.7
0.82
0.54
-Tread
SD
19.7
461.1
16.3
89.9
0.63
0.64
264.3
4.07
201.2
0.80
0.54
SPRRINT (Ours)
SD
18.7
262.6
9,30
118.5
0.65
136.5
2.56
247.1
0.82
0.56
Gain &
x1.32
+88.5
+24.1
+48.5
+0.53
+35.5
1M training iterations
Improved SiT-XL/2
SD
61.2
290.0
10.9
113.4
0.66
0.67
146.0
2.36
243.7
0.80
0.58
Progressive Training
SD
25.8
359.4
12.3
102.2
0.67
0.65
188.1
2.95
222.22
0.82
0.55
MDTv2
SD
39.2
522.7
18.8
77.2
0.61
0.64
326.7
4.68
183.1
0.77
0.55
MicroDiT
SD
37.5
293.4
10.9
113.8
0.68
0.65
147.6
2.53
241.4
0.82
0.55
TTead
SD
34.5
372.6
12.3
112.1
0.66
0.66
197.7
2.82
242.9
0.80
0.57
SPRINT (Ours)
SD
31,5
248.8
9.15
129.5
0.67
0.67
126.1
2.29
268.3
0.81
0.59
Gain A
x1.94
+41,2
+1.75
+16.1
+14.9
+0.07
24.6
400K training iterations
Improved S/T-X1/2
Flux
24.6
358.9
14.8
84.4
0.64
0.63
178.7
3.95
210.77
0.83
0.50
Progressive Training
Flux
17.0
375.3
13.5
89.2
0.66
0.63
186.3
4.02
205.4
0.84
0.49
MicroDiT
Flux
20.9
420.9
17.8
76.8
0.61
0.64
212.9
4.45
196.2
0.81
0.51
Tread
Flux
19.8
470.1
19.9
72.2
0.60
0.63
255.0
5.18
187.5
0.79
0.50
SPRINT (Ours)
Flux
18.8
268.4
11.4
101.9
0.66
0.63
135.4
3.77
239.8
0.83
0.51
Gain A
x1.31
+90.2
+3.4
+17.5
+43.3
+0.18
+29.1
tokens. Formally, the conditional and unconditional velocities are:
v(xe; c) he (Fusion (ge (fa (xe, c)), fo(xx,c)), c)),
cxee, 0) = heg (Fusion [M, fe (x, 0)), 0),
(4)
where M denotes the [MASK] token tensor. This provides high-quality generation while nearly halving FLOPs and latency per step, since the expensive middle blocks executed only once.
3.5 STRUCTURED GROUP-WISE TOKEN SUBSAMPLING
The effectiveness token dropping depends not just on how many tokens are removed, but on which are kept. Uniform random sampling risks leaving large contiguous holes in the feature map. To avoid this, we propose a structured group-wise subsampling strategy that guarantees local coverage while maintaining global irregularity. Specifically, we partition tokens into small, non-overlapping groups in their native topology (e.g., 2D for images). For images, we divide the (H/p) x (W/p) grid into n groups, At each training iteration, we randomly select & tokens per group, giving drop ratio 1 k/n². We use = 2, k = 1, corresponding to a 75% drop ratio. This ensures that every local patch represented while preventing the model from overfitting to fixed sampling patterns.
EXPERIMENT
4.1 EXPERIMENTAL DETAILS
Training details. Our framework follows the setups of DiT (Peebles & Xie, 2023) and SiT (Ma al., 2024). Unless stated otherwise, most of the experiments are trained on ImageNet-IK I at 256 x 256 resolution using pretrained VAEs from Stable Diffusson (Rombach et al., 2022) and Flux (Labs, 2024b), both with 8x downsampling but encoding into and 16 channels, respectively. Unless stated otherwise, models are pre-trained with 75% token drop ratio using our structured group-wise subsampling. We adopt the SiT architecture, where each block contains a self-attention and feed-forward layer, and apply standard improvements: RMS Normalization for queries and keys (Touvron et al., 2023a;b,, 2D RoPE for positional embeddings (Wang et al., 2024), and lognormal timestep sampling (Esser et al., 2024a). Experiments focus SiT-B/2 and SiT-XL/2. Additional hyperparameters and training details are provided in Appendix C. Pre-training and fine-tunine fine-tuning algorithm is provided in Alg. and respectively.

---

Published as a conference paper at ICLR 2026
Table Compatibility with other architectures. We apply SPRINT REPA and U-ViT on the SD autoencoder, reporting performance at 400K iterations with/without dussifier-free guidance and SPRINT's relative gains over SiT (Gain A). All metries use ODE sampling steps.
Method
w/o CFG (w
1.0)
w CFG (ur 1.4)
FDD
FID
18t
Pre.
Rec.
FDD 4
FID
1S
Pre.
Rec.
Improved SIT-XL/2REN
279.6
10.0
114.0
0.67
0.66
146.6
2.42
237.1
0.81
0.57
-PRINT (Ours)
234.5
8.68
129.6
0.67
125.1
2.38
259.8
0.59
Gain A
+45.1
+1.32
+15.6
+21.5
+0.04
+22.7
Improved L-VI2-XL/2
335.1
12.1
98.6
0.67
0.64
193.7
3.36
200,3
0.80
0.56
SPRINT (Ours)
Gain A
+63.4
+2.9
+15.8
+30.1
+0.39
+36.4
100K
400K
100K
400K
r=0
ST-UU/2 ST-XU2 Ours
Ours
SIT-U/2- ST-XU/2 REPA
fo
Bo
Figure 5: SPRINT improves visual scaling. Qual- itative comparison of images generated without classifier-free guidance at 400K iterations using two SiT-XLL/REPAA models, with SPRINT applied to the model in the upper row.
Figure 6: SPRINT improves feature seman- tics. We visualize the PCA features of fo and ge from two SiT-XL/2 models at 400K itera- tions, with SPRINT applied to the model in the upper row.
Evaluation details. We evaluate generation quality using standard metrics: FDD (Fréchet Distance
on DINOv2 (Oquab et al., 2023) features), FID (Fréchet Inception Distance (Heusel al., 2017)),
Inception Score (IS) (Salimans et al., 2016), Precision, and Recall (Kynkäänniemi et al., 2019). Among these, FDD has been shown to be more reliable for diffusion models (Stein et al., 2023).
To assess training and inference efficiency, computed with the Speed library. We provide details in Appendix C, D. Inference algorithm is provided in Alg.3. 3.
report total training FLOPs and inference FLOPs
4.2
COMPARISON
We compare against following methods to demonstrate the effectiveness and efficiency our method:
1. Dense SOTA: We use SIT (Ma al., 2024) models our primary baseline. This represents state-of-the-art model trained with full, dense tokens, providing direct measure of the trade-off between performance and our efficiency gains.
2. Sparse SOTA: We compare against recent methods that leverage token-dropping strategy to accelerate training, e.g., MicroDiT (Sehwag et al., 2025) and Tread (Krause et al., 2025).
3. Alternative methods: We also compare against progressive training, a popular strategy where model pretrained on a lower resolution and finetuned on the full resolution.
more detailed description of each baseline provided in Appendix We adopt the same archi- tectural improvements for all models and report the performance during training in Tab. 1. SPRINT demonstrates superior performance and efficiency across all settings. At just 400K training iterations with the SD-V VAE, our model significantly outperforms SiT-XL/2 baseline (e.g., a +88.5 improve- ment in FDD) while using 1.32x fewer FLOPs. As training progresses to 1M iterations, SPRINT consistently improves over the baseline while becoming even more efficient, achieving 1.95 computational speedup. This result highlights SPRINT's dual acceleration: achieving higher sample quality in the same training iterations with lower computational budget. In contrast, competing token-on token-dropping dropping methods fail to match the performance of SiT-XL/2, even at a higher computational cost than our method. These trends hold when using classifier-fre guidance.

---

Published as a conference paper at ICLR 2026
Table Comprehensive comparison on ImageNet 256 256 class-conditioned generation with classifier-free guidance. / indicate whether lower or higher values are better, respectively. denotes training with batch size 1024, our reproduction with architectural improvements, and use of guidance scheduling. Metries are evaluated with 250 sampling steps using the SDE sampler. TFLOPs are measured with the DeepSpeed library (refer to Appendix D for details.)
Training TELOPs (x10%)
Inference TFLOPs !
Method
Epochs
#Params.
FDD 4
FID
Pre.
Rec.
ADM (Dhariwal Nichol, 2021)
400
G73M
3,94
0.82
CDM (Ho 2022)
2160
4.88
LDM-4 (Rombach et 2022)
200
400M
3.60
0.87
0.48
U-VIT-" (Bao al., 2023)
240
501M
2.29
0.82
0.57
DFT-XI. DiT-XI.(Peehless & Xie, 2023)
1400
675M
427.7
0.475
79.5
2.27
0.83
0.57
FiTv2-XL (Wang et al, 2024)
400
671M
80.5
2.26
0.81
0.59
MDTV2-XI. (Gao et at. 2023)
1080
742M
258.3
0.521
77.3
1.86
0.81
0.60
MDTv2-XL, (Gao et al., 2023)
1080
742M
258.3
0.521
75.2
1.58
0.79
0.65
MaskDiT (Zheng al., 2024)
1600
730M
268.0
0,513
82.4
2.28
0.80
Tread (Krause al,, 2025)
740
675M
146.0
0..75
2.09
0.81
0.62
SiT-XL (Ma 2024)
1400
675M
427.7
0.475
78.5
2.06
0.82
0.59
STT-XL
400
675M
122.2
0,474
79.5
2.04
0.82
0.60
+ *SPRINT
200
677M
43.7
0.477
79.0
2.01
0.82
0.60
SPRINT
400
677M
65.1
0.477
75.4
1.96
0.80
0.61
SPRINTPIG
400
677M
0.274
58.4
1.62
0.80
SPPIITT
PDG
400
677M
65.1
0,263
54.9
1,55
0.80
SiT-XLupPA (Ya ul., 2024)
800
675M
248.6
0.475
72.5
1.80
0.81
0.61
SiT-XL KERA
200
675M
62.1
0,474
78.8
1,93
0.81
0.60
+ -SPRINT
200
677M
44.3
0.477
75.6
1.87
0.81
0.61
.SPRINTPGG
200
677M
44.3
0,274
57.1
1.61
0.80
-SPRINTpUG
400
677M
66.7
0.274
54.7
1.59
0.80
0.64
:SPRINTT
400
677M
66.7
0.263
49.6
1.49
0.81
Generalization to other diffusion architectures. To demonstrate that our method is a general training strategy and not limited to specific DiT architecture, we apply SPRINT two other prominent models: REPA (Yu et al., 2024) and U-ViT (Bao al., 2023). We integrate our dense-sparse fusion mechanism into their respective backbones and report the results after 400K training iterations in Tab. 2. The results show that SPRINT provides significant improvements in all cases. When applied to REPA, SPRINT improves the FDD by +45.1 and FID by +1.32 (w/o CFG). Similarly, for V-V observe +63.4 improvement in FDD and +2.9 improvement in FID. These experiments confirm that SPRINT is broadly applicable and effective method for accelerating the training.
Visual analysis. In Fig. 5, we show that SPRINT not only accelerates convergence quantitatively but also enhances the visual progression. At just 100K iterations, SPRINT produces coherent global structures (e.g., the shape of car) along with fine details, whereas REPA lags behind. Furthermore, in Fig. 6, we analyze the PCA of features from fe and ge. demonstrating that SPRINT learns more noise-invariant and semantically vivid representations than the SiT model across diffusion timesteps.
4.3 COMPARISON WITH STATE-OF-THE- ART MODELS
Tab. compares SPRINT against recent state-of-the-ard diffusion transformers. Our improved SiT closely matches the original SiT performance after 400 epochs (78.5 vs. 79.5 FDD). In contrast, SiT trained with SPRINT achieves comparable performance 79.0 FDD only 200 epochs. At 400 epochs, SPRINT outperforms the improved SiT baseline by 4.4 FDD (from 79.5 to 75.4) and 0.08 FID while using just 53% of the training FLOPs. This shows that SPRINT both accelerates convergence and substantially reduces training cost. At inference, Path-Drop Guidance (PDG) further boosts efficiency: with only 57% of the inference cost, SPRINT improves performance by 21.1 FDD (from 79.5 to 58.4) over the improved SIT.
Similar trends hold when combined with REPA. SPRINT reduces FDD from 78.8 to 75.6 using only 71% of the training FLOPs. With PDG sampling at 400 epochs, it surpasses the official REPA model trained for 800 epochs by 17.8 FDD and 0.21 FID, while using only 27% of the training FLOPs. Overall, SPRINT consistently improves generation quality while drastically lowering both training and inference cost, outperforming strong baselines and aligneen-aaugment
models. Moreover, incorporating the recent guidance schedule (Kynkäänniemi et al., 2024) further boosts performance.
8

---

Published as a conference paper at ICLR 2026
Table 4: Effect of token-drop strategies on FID.
Table 5: Effect of dense-
Table 6: Effect of fa, go, he on compute and performance.
sparse residuals on FID.
Strategy
FID +
Dense
Sparse
FID +
FLOPs
fa
9o
he
/ /iter
FID
Random
30.1
X
/
85.1
Structured (Ours)
27.5
/
x
81.4
2
8
2
7,47G
27.5
/
/
27.5
6
3
9.33G
29.1
5
2
5
13.1G
49.2
Table 7: Effect of dense residuals and drop ratio on FID.
Table Effect of fo and he depth FID.
Figure 7: Effect of guidance scale on SiT and our SPRINT.
205
Method
Dense residual
FID
fo
96
h9
FID
79.7
SiT-B/2 X
55.6
1
61.5
170
54.1
2
8
27.5
V
25%
43.2
SPRINT
V
50%
32.3
3
44.4
150
/
75%
27,5
4
81.5
140
87.5%
50.2
2.5
Guidancc scalt
4.4 ANALYSIS
We mostly use SiT-B/2 configuration at 400K training iterations (detailed in Tab. 9) in following analysis unless stated otherwise.
Sparse-dense residual fusion (Tab, 5). To evaluate the importance of each path in sparse-dense residual fusion, we perform an ablation by disabling each of the two parallel paths during training. Removing the dense shallow path causes a sharp performance drop, with FID rising from 27.5 to 85.1, underscoring
role accurate velocity prediction. Conversely, removing the sparse deep path reduces the model to a standard dense DiT with only four layers, which also degrades performance due limited capacity. These results confirm that the parallel sparse dense design critical for maintaining high performance under token dropping.
Token sampling strategy (Tab. 4). compare our structured group-wise sampling strategy with standard uniform random sampling. At the same 75% drop ratio, structured sampling improves FID from 30.1 1 to 27.5, demonstrating that preserving local coverage crucial for effective sparse training.
Effect of ga depth (Tab. 6). We study the trade-off between performance and computation function of middle block depth. The default configuration yields the best FID (27.3) with the lowest cost (7.47G). Shifting layers from the middle block to the encoder and decoder (e.g., 3-6-3 5-2-5) increases cost without benefit, and FID degrades to 29.1 and 49.2, respectively. Thus, the default configuration strikes the best balance between efficiency and performance.
Effect of So and hg depth (Tab. 8). We find that allocating at least two blocks to both So (dense shallow path) and he (sparse deep path) is critical for high performance. Reducing either single block already degrades results (FID 61.5 and 44.4). Moreover, entirely removing either block collapses performance (FID > 79): this supports our encoder (didse)-middle (dparse)-decoder (dense) design. The encoder must first operate on dense tokens to transform noisy inputs into noise- invariant features, after which the middle blocks can safely work on sparse tokens, and the decoder applied after residual fusion. This is necessary for accurate prediction under high drop-ratio training.
Drop ratio (Tab. 7). As the drop ratio increases from 0 to 75%, model performance steadily improves, with FID decreasing from 54.1 to 27.5. This trend indicates that higher sparsity in SPRINT promotes complementary interactions between the encoder and middle blocks, leading to more robust and efficient representations. However, at extreme drop ratio of 87.5%, FID rises to 50.2, suggesting that excessive sparsity limites the model's representational capacity.
Path-drop guidance (Fig. 7). We compare FDD across guidance scales for CFG (SIT-XL/2), CFG (SPRINT), and PDG (SPRINT). PDG consistently outperforms both CFG baselines, achieving a lower (better) peak FDD. Moreover, it delivers these gains nearly half the inference cost, since the unconditional estimate bypasses the middle blocks. These results show that PDG provides a superior trade-off, generating higher-quality samples while substantially reducing computational cost.

---

Published as a conference paper at ICLR 2026
Fincture
Ficcune
CRG
rogime
PDG
3.00
2.50
2.00
1.75
gap
20
50
100
150
200
OK 30K 100K OK 20K 200K
Full-oken floetuning steps (=K)
Figure 8: Visual progression over fine-tuning steps. Before fine-tuiing (OK), SPRINT already produces class- aligned samples but exhibits slight artifacts in fine details (e.g., the turtle's eye, the ram's 's leg). After a short 20K- step fine-tuning, SPRINT largely recovers these details and overall visual quality.
Figure 9: FID over fine-tuning steps for CFG and PDG sampling. Just 20K fine-tuning steps recover over 94% of the 200K performance, indicating that rela- tively short fine-tuning stage is sufficient close the train-inferene gap.
Training at higher resolution (Appendix F.1, Fig. 33). We also evaluate our model against baselines at 5122 resolution with XI. config. Results are provided in Tab. 10 and show that SPRINT achieves 1.96 FID compared to 2.63 of SiT baseline with only 50% of training compute (184.8 vs. 366.6).
Lower sampling steps (Appendix F.2), SPRINT remains competitive few-step inference, consis- tently surpassing SiT-XL/2 Tab. 11. At steps, reduces FID from 7.37 to 6.29 and FDD from 205.2 to 174.5, highlighting the representational strength of our method.
4.5 BENEFITS OF FINE-TUNING
Here, we analyze the train-inference gap of SPRINT after the pre-training stage and the effect of the subsequent fine-tuning. Specifically, we perform qualitative and quantitative ablations over the number of fine-tuning steps after 2M pre-training iterations, reported in Fig. and Fig. 9, respectively.
In Fig. 8, we observe that, before fine-tuning, SPRINT already produces class-aligned samples with globally coherent structure, but tends to miss some freeufryyy details (c.g., the turtle's 's eye in second row, the ram's leg in third row), which is expected given that most tokens are dropped during pre-training. The role of the fine-tuning stage is therefore to recover these local details. Notably, after only 20K fine-tuning steps, SPRINT largely restores these details and improves overall visual quality. This observation is consistent with the quantitative trends in Fig. 9. For both CFG and PDG sampling. FID improvements beyond 50K fine-tuning iterations are marginal and eventually plateau. In particular, after just 20K steps, SPRINT recovers over 94% of the FID improvement achieved at 200K fine-tuning steps. This indicates that the majority of the train-inference gap closes very early-within 20K-50K iterations, corresponding only 2.5% of the pre-training steps. This further confirms that SPRINT learns the necessary representations for high-quality generation during sparse pre-training, and that these representations transfer effectively the full-token regime.
Overall, these results show that SPRINT is not overly sensitive to the precise length of the fine-tuning stage: relatively short full-token fine-tuning is sufficient recover the high-frequency details missing from sparse pre-training.
5 CONCLUSION
We introduced SPRINT. simple and aechtecture-agnostie training framework for DiTs that com- bines dense- dense-shallow and sparse-dest features through residual fusion. By exploiting the complemen- tary strengths of shallow and deep layers, it enables aggressive token dropping (up to 75%) while preserving representation quality, and two-stage schedule with masked pre-training and short full- token fine-tuning closes the train-inference gap. Experiments on ImageNet- IK show that SPRINT reduces training cost by up to 9.8x while matching or surpassing the quality of strong baselines. SPRINT also enables Path-Drop Guidance. simple replacement for CFG that halves inference cost while improving sample quality. Thus, SPRINT a simple, effective, and general approach for efficient DiT training, applicable across architectures, resolutions, and alignment methods.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY
We have made every effort to ensure the reproducibility of our results. Detailed hyper-parameteny training schedules, and architectural configurations are provided in the Appendix, including model definitions, pre-training and fine-tuning iterations, number of sampling steps at inference, and compute resources. Our framework follows the well-established setups of DiT (Peebles Xie, 2023) and SiT (Ma et al., 2024), which are widely adopted in diffusion research. Although our training code cannot be released at submission time, the use of these standardized setups, along with the provided experimental details, should allow independent reproduction of our results.
REFERENCES
Fan Bao, Shen Nic, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Conference on Computer Vision and Pattern Recognition, CVPR, 2023.
Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In Conference on Computer Vision and Puttern Recognition, CVPR, 2023.
Shuning Chang, Pichao Wang, Ming Lin, Fan Wang, David Junhao Zhang, Rong Jin, and Mike Zheng Shou. Making vision transformers efficient from a token sparsification view. In Conference on Computer Vision and Pattern Recognition, CVPR, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, NeurIPS, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In arXiv preprint arXiv:2010. 11929, 2020.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning, ICML, 2024a.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high- image synthesis. In ICMI., 2024b. URI. https::tpt rew.eeeieww met/forunt d=FFnÜhsQJ55..
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion transformer a strong image synthesizer. arXiv preprint arXiv 14389, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a time-scale update rule converge a local nash equilibrium. Advances in Neural Information Processing Systems, NeurIPS, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In arXiv preprint 2rXiv:2207.125940 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, NeurIPS, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 2022.
Benjamin Hoover, Hendrik Strobelt, and Sebastian Gehrmann. exbert: A visual analysis tool to explore learned representations in transformers models. In arXiv preprint arXiv: 2019.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. In International Conference Machine Learning, ICMI., 2024.

---

Published as a conference paper at ICLR 2026
Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with bad version of itself. In Advances in Neural Information Processing Systems, NeurIPS, 2024a.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Conference on Computer Vision and Pattern Recognition, CVPR. 2024b.
Yujin Kim, Dogyun Park, Dohee Kim, and Suhyun Kim. Naturalinversion: Data-free image synthesis improving real-world consistency. In Proceedings of the AAAI conference on artificial intelligence, 2022.
Juyeon Ko, Inho Kong, Dogyun Park, and Hyunwoo Kim. Stochastic conditional diffusion models for robust semantic image synthesis. In arXiv preprint arXiv:2402.16500 2024.
Felix Krause, Timy Phan, Ming Gui, Stefan Andreas Baumann, Vincent Tao Hu, and Björn Ommer. Tread: Token routing for efficient architecture-agnosii diffusion training, 2025.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, NeurlPS, 2019.
Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. In arXiv preprint arXhv:2404.07724, 2024.
Black Forest Labs. Flux: A generative model by black forest labs. https://github.co labs/ flux, 2024a. Accessed: 2025-05-14.
Black Forest Labs. Flux. https: //ggggithhuub/g
com/black bock forest- Labettlabb//ffaa 20244 2024b.
Jaa- Ycon Lee, Byunghee Cha, Jeongsol Kim, and Jong Chul Ye. Aligning text to image in diffusion models is easier than you think, 2025. URL https: / wogg httg//ab//// 2503. 08250.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In International Conference on Learning Representations, ICLR, 2023.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. In International Conference on Learning Representations, ICLR, 2023.
Nanye Ma. Mark Goldstein, Michael S Albergo, Nicholas M Bofli, Eric Vanden-Eijnnenn, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, ECCV, 2024.
OpenAI. Video generation models as world simulators: Introducing sora. https::/t/ppptt com/index: wndex/// video generation conneratioomoddlss--eee aasss-aswoons as-world----oooooo-ooo------oo-s www.wwwiiolaaeeeeeeeeeeeeeoeeeae simulators/, 2024. Ac-
cessed: 2025-05-14.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafranicc, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinoy2: Learning robust visual features without supervision. In arXiv preprint arXiv:2304.07199. 2023.
Dogyun Park and Suhyun Kim. Probabilistic precision and recall towards reliable evaluation of generative models. In Proceedings of the IEEE/CVF international conference on computer vision, ICCV, 2023.
Dogyun Park, Sihyeon Kim, Sojin Lee, and Hyunwoo J Kim. Ddmi: Domain-agnostic latent diffusion models for synthesizing high-quality implicit neural representations. arXiv preprint arXiv:2401.12517, 2024a.
Dogyun Park, Sojin Lee, Sihyeon Kim, Taehoon Lee, Youngjoon Hong. and Hyunwoo Kim. Constant acceleration flow. In Advances in Neural Information Processing Systems, NeurIPS, 2024b.
12

---

Published as a conference paper at ICLR 2026
Dogyun Park, Taehoon Lee, Minseok Joo, and Hyumwoo J Kim. Blockwise flow matching: Improving flow matching models for efficient high-quality generation. In arXiv preprint arXiv:2510.21167, 2025.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Conference on Computer Vision and Pattern Recognition, CVPR, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024. URL https: fictps:/epperree/ http//ppeeeeeeeeeeeeeeeeeeeeeeee net forum? d=di 522R8xgf.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems, NeurIPS, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Conference on Computer Vision and Pattern Recognition, CVPR, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, NeurlPS, 2016.
Vikash Sehwag. Xianghao Kong, Jingtao Li, Michael Spranger, and Lingjuan Lyu. Stretching each dollar: Diffusion training from scratch on a micro-budget. In Conference on Computer Vision and Pattern Recognition, CVPR, 2025.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In arXiv preprint wrXiv:2010002502. 2020.
George Stein, Jesse Cresswell, Rasa Hosseinzadch, Yi Sui, Brendan Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L Caterini, Eric Taylor, and Gabriel Saiza-Gaaem. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. In Advances in Neural Information Processing Systems, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anm Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. In arXiv preprint arXiv:2302. 139V:, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. In arXiv preprinl arXiv:2307.0922 2023b.
Elena Voita, Rico Sennrich, and Ivan Titov. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In Empirical Methods in Natural Language Processing, EMNLP, 2019.
Runqian Wang and Kaiming He. Diffuse and disperse: Image generation with representation regularization, 2025. URL htt.t:tp///// http:///axwvvooro..o/ee a2tr/500//25066 0..566..0 09027.
ZiDong Wang, Zeyu Lu, Di Huang, Cai Zhou, Wanli Ouyang, al. Fitv2: Scalable and improved flexible vision transformer for diffusion model. In arXiv preprint arXiv:2410.13925, 2024.
Ziqiao Wang, Wangbo Zhao, Yuhao Zhou, Zekai Li, Zhiyuan Liang, Mingjia Shi, Xuanlei Zhao, Pengfei Zhou, Kaipeng Zhang, Zhangyang Wang, Kai Wang, and Yang You. Repa works until doesn't t: Etrly-stopped, holistic alignment supercharges diffusion training, 2025. htttt:: //arxiv. www.v..or/aaaeee abs/ / 2505. 6792.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is casier than you think. In arXiv preprint 1rXiv:2410.069..0 2024.
13

---

Published as a conference paper at ICLR 2026
Yuzhe Zhang, Jiawei Zhang, Hao Li, Zhouxia Wang, Luwei Hou, Dongqing Zou, and Liheng Bian. Diffusion- based blind text image super-resolution. In Conference on Computer Vision and Pattern Recognition, CVPR, 2024.
Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers. In Transactions on Machine Learning Research, TMLR, 2024.
14

---

Published as a conference paper at ICLR 2026
A ANALYSIS DETAILS
Training behavior (Fig. 3). We provide the implementation details used to measure the training behavior shown in Fig. 3. We adopt the SiT-B/2 configuration from the SiT paper (Ma al., 2024), which consists of 2 encoder blocks, 8 middle blocks, and 2 decoder blocks. In Fig. 3a, we plot the l2 gradient norm of the encoder Ja with respect to the flow-matching loss L. i.e., , across pretraining iterations. This analysis highlights the improved gradient flow within the encoder blocks. Compared to the SiT baseline, SPRINT exhibits consistently stronger gradient propagation the encoder as sparsity increases, leading to more effective parameter updates and faster sonvergence-reflettee in both higher CKNNA scores and lower FID values.
In Fig. 3b, report the Centered Kerne Nearest-Neighbor Alignment (CKNNA) (Huh al., 2024) score, a relaxed variant of Centered Kernel Alignment (CKA). CKNNA is commonly used to assess the semantic alignment (Yu et al., 2024) between diffusion models and large-scale self-supervised visual encoders such as DINOv2. Intuitively, given a noisy input X,, CKNNA quantifies how well the intermediate features of diffusion model capture inise-invariant semantics by comparing them with DINOv2 features extracted from the corresponding clean image Xo. Higher CKNNA scores indicate more semantically meaningful and noise- representations that align more closely with the features of the visual encoder. We follow the definition and implementation provided in the original work (Huh et al., 2024). Specifically, we compute the CKNNA score between the output of the encoder fe on noisy inputs and the output of DINOv2 on clean inputs Xo- We randomly sample 10K images from the ImageNet-1 validation set and report results with 10.
Finally, in Fig. 3c, we report FID values computed with 10K generated images. Consistent with previous findings (Yu ct al., 2024), we observe a strong negative correlation between the CKNNA values of intermediate diffusson features and FID scores. This suggests that higher alignment between diffusion features and high-quality visual representations leads to better generation quality.
Roles of dense-shallow and sparse-deep features (Fig. 4). In Fig. 4, we analyze the contribution of each path in SPRINT. To generate samples using only single path, replace the feature representation of one path with that of the other. In other words, we duplicate the features from one path and concatenate the original and duplicated features before feeding them into the decoder.
PCA visualization of diffusion features (Fig. 6). In Fig. 6, we perform principal component analysis (PCA) of the intermediate features to better understand what the model has learned. PCA identifies the principal axes that capture the greatest variance in the feature space and is widely used to analyze representations learned by neural networks (Kim et al., 2022; Oquab et al., 2023; Park al., 2024a; Ko al., 2024). We compute PCA across patch embeddings and visualize the first three principal components as RGB channels. Specifically, we examine the outputs of the encoder So and the middle blocks go at different timesteps to observe how the feature representations evolve throughout the diffusion process. Additional PCA visualizations are provided in Fig. 34.
B SPRINT WITH DIFFERENT DIFFUSION TRANSFORMERS
We provide details of the different diffusion transformers used the main paper and describe how SPRINT is implemented top of them.
SIT (Ma et al., 2024). We closely follow the architecture of SiT. The SIT model is structurally analogous to a Vision Transformer (ViT) (Dosovitskiy et al., 2020), consisting of a sequence of identical transformer blocks that process a patchified 1D token sequence. SiT adapts this for the diffusion task by incorporating timestep and class conditioning, which is injected into each block via AdaIN-zero layers. Because the architecture is simple, homogeneous stack of blocks, it is straightforward to decouple into our encoder, middle, and decoder blocks when applying SPRINT.
REPA (Yu et al., 2024). Representation Alignment reguairegslariies a DiT by aligning hidden states with clean image features from pre-trained DINOv2 model. The architecture largely follows SiT, with the key modification being a projection layer inserted at the Sth transformer block to perform the alignment. To integrate SPRINT with REPA, place this projection layer the corresponding
15

---

Published as a conference paper at ICLR 2026
Table 9: Hyperparameters used for SPRINT.
STT-B+SPRINT
SIT-XL+SPRINT (Tab. 1,3)
SIT REA SSPRINT STASPIINT
U-II-XILAPPRINT
(Fig. Tab,
(Tab
(Tab. 10)
(Tab.
Architectune
Target latent
32 32
32 x
32
64 64
32 x 2
Patch size
2
2
2
Layers
28
28
Num. Jo Layers
2 24
2
2
Num. de Layers
24
24
Num. Layers
2 1152
2
Hidden dims
384
1152
1 1152
1152
Nom. heads
16
16
Pretraining config.
Optimizer
Adam 0,0001
Adam
Adamt W 0.0001
Adam 0.0001
Adam 0,0001
Leaning rate
0,0001
Batch size
256
256
256
256
256
Visual Encoder
DINOv2-B
Token drop ratio
75%
75%
75%
75%
75% 10%
PDG drop ratio
10%
10%
10%
Finetuning confgg
Training iterations
100K
100K
200K
100K
Warmup iterations
5K
5K
5K
5K
Optimizer
Adam W 0.0002
Adam
AdamW
AdamW
Learning rate
0.00015
Batch size
512/1024
512/1024
1024
512
Token drop ratio
75%
75%
75%
75%
75%
PDG drop ratio
10%
10%
10%
10%
10%
Evaluation config.
Sampler
ODE
ODE/SDE 50/250
ODE/SDE 50/250
SDE
ODE
Sampling steps
250
50
location within our sparse middle block, go- A key consideration is that the hidden states in go operate on sparse token set of length N', while the target DINOv2 features have full sequence length of N. To resolve this, we simply apply the same token-dropping mask to the DINOv2 feature sequence, ensuring one-to-one correspondence for the alignment loss. Since DINOv2 also uses standard transformer architecture with positional encodings, aligning the corresponding tokens straightforward.
U-VIT (Bao et al., 2023). U-VIT extends the Vision Transformer with U-Net (Ho al., 2020)- style architecture. Similar to U-Net, it stacks transformer blocks with long skip-connections between encoder and decoder stages, directly passing features from encoder to decoder. To apply SPRINT, we first conceptually decompose the U-ViT into our standard fe. 9a. and hg sections while preserving all original skip-connections. We then introduce our dense residual path between fo and ha and apply token dropping to the middle section, fo. The U-Net skip-connections remain compatible with this design. The long-range skips between the encoder and decoder are unaffected. The shorter skip-connections within the sparse middle section naturally operate the reduced set of tokens. This allows SPRINT to be integrated cleanly without disrupting the U-ViT's core component.
C IMPLEMENTATION DETAILS AND HYPERPARAMETERS
C.I TRAINING DETAILS
We follow the model configuration of the original SiT implementation (Ma et al., 2024), with the only modification being a single linear projection layer for sparse sparse-dense dense residual fusion. This adds only marginal number of parameters, approximately 0.3% of the original model size. We use pre-computed latent vectors from raw images via Stable Diffusion (Rombach et al., 2022) and Flux (Labs, 2024b) VAEs, and, following common practice, do not apply any data augmentation. For pretraining, we train SPRINT with batch size of 256, learning rate of le-4, a fixed drop ratio of 75%, and an EMA decay rate of 0.9999, following standard configuration (Ma et al., 2024; Yu et al., 2024; Park et al., 2024b; 2025). After pre- pre-training, short fine-tuning stage for 100K iterations. We increase the batch size and the learning rate, following standard practice (Zheng al., 2024; Krause al., 2025). We found that applying linear learning
switch the middle blocks operate
the full token set for
16

---

Published as a conference paper at ICLR 2026
Algorithm 1 SPRINT Pre-training
Require: Input Xo, Drop ratio r, Path-drop prob p, encoder So, middle blocks go, decoder he
condition c
1: while not converged do
2: Sample t [0, 1] and e ~~ I))
3:
X4 4 (1 x) te
X, BBNNC
4:
D f E BRNNX
5:
tarop + Drop(fe,r) r)
edrop
E R (x(I--)NXX
6:
B:
drop (f, c)
Br
drop
E B
7:
g pud BndiMMMass((.
g
R RxNxC
8:
ge + [MASK] with probability
D Path-drop learning
9:
h + Fusion(fe, (fe, g(
Sparse- dense residual fusion -, TR. BxXC
V. ho(h,, c.)
11:
List < 11: v2
12: Update using VaLvel
13:
end while
14:
return fo.90, 9o, he
Algorithm SPRINT Fine-tuning
Require: Input Xo, Path-drop prob p, encoder So, middle blocks go, decoder ho, condition c.
1: while not converged do
2: Sample [0, 1] and ~~ 10,,
3: X₁ f- (1- 11-t)xo tte
X, BxNCC
4:
f - co(x,.c)
b f E R BxXXC gi R BxNXC
5:
g: c)
6:
g: + IMASK] with probability p
D Path-drop learning
7:
h, Fusion(fe,, ge)
Sparse-dense residual fusion >> >v, R BBBxxx XXC
8:
Vi + ho(h, c)
9:
Lvel + Iive V2
10: Update 0 using V VoCvel
end while
12:
return fo. go, ho
rate warm-up from 2e-6 to 2e-4 over the first 5K iterations stabilizes the training. During the warm-up stage, use an EMA decay rate of 0.999, which is restored to 0.9999 afterward. For both training phases, we introduce a path-drop learning strategy to maximize the effectiveness of our path-drop guidance, in addition to the standard class-condition dropping. Specifically, following the practice in CFG training, we randomly drop the features of the sparse-deep path with a probability of 10% and replace the dropped features with mask tokens. This random dropping is performed independently of the condition dropping in CFG. To accelerate training, we adopt mixed-precision (bf16) training and apply gradient norm clipping at 1.0 during both pretraining and finetuning. Detailed hyperparameters are summarized in Table 9, All experiments are conducted on 8 NVIDIA A100 80GB GPUs.
C.2 EVALUATION DETAILS
Metrics. We evaluate generation performance using several standard metrics: FDD (Stein et al., 2023) (Fréchet Distance on DINOv2), FID (Heusel et al., 2017) (Fréchet Inception Distance), IS (Salimans et al., 2016) (Inception Score), and Precision/Recall (Kynkäännicmi et al., 2019; Park Kim, 2023). Unless otherwise specified, follow the evaluation protocol of (Dhariwal & Nichol, 2021) and report results using 50K generated samples.
FID is the most widely used metric, measuring the feature distance between the distributions of real and generated images. It relies on the Inception- V3 network and assumes both feature distributions follow multivariate Gaussian distributions. IS also uses the Inception-V V3 network but instead evaluates
17

---

Published as a conference paper at ICLR 2026
Algorithm 3 SPRINT Inference
Require: encoder fa. middle blocks go, decoder he, condition guidance scale w,, sampling steps
N, sampler
1: (N( I)
2: for = to do
3:
f
4: if Path-drop guidance then
5: v(x,, 0) ho (Fuusionn (xx, 0)),
Path-drop guidance
6: else
7:
v(x,, 0) he (Ffuusion(po(/o(xx,,) (xx 0)),
Classifier-free guidance
8:
end if
9:
ú(x,, + v(x,, 0) w- (e(xx, c) v(x,, 9))
10: x S S(x,, c))
end for
12:
refurn Xo
the quality and diversity of generated images by computing the KL--divergence between the marginal label distribution and the conditional label distribution predicted from logits.
FDD adopts the same formulation as FID but replaces Inception features with DINOv2 features, which provide stronger semantic alignment and robustness noise. Notably, FDD has been shown to be more reliable for diffusion models (Stein et al., 2023; Karras et al., 2024b).
Finally, Precision (Kynkäänniemi et al., 2019) measures the fraction of generated images that realistic, while Recall measures the fraction of the training data manifold covered by generated samples.
Guidance scale. We use the following formulation for guidance sampling (Ho & Salimans, 2022):
o(x,, c) = v xx, 0) + w- (v(x,, c) vxx,,00 0)),
(5)
where w denotes the guidance scale. standard Classifier Free Guidance (CFG), the unconditional velocity v(x,, 0) is computed using the full model path with null condition. In contrast, our Path-Drop Guidance (PDG) replaces the unconditional branch with weaker network, as defined in Eq. 4.
For the results in Tables 1 and 2, we consistently use a CFG scale of 1.4 with the ODE sampler across all methods.
For Table 3, we adopt the SDE sampler (Ma et al., 2024) to compare baselines. Under this setting, CFG scale of 1.35 achieve the best FID and 2.0 achieve the best FDD. For our PDG sampling, the optimal scales are 1.35 for FID and 1.9 for FDD.
For model in Table 10, CFG and PDG.
the scale of 1.35 and 1.8 for FID and FDD, respectively, for both
D COMPUTATION ANALYSIS
We use the SiT-XL//2 configuration for evaluating computational analysis below.
FLOPs. To estimate the total training FLOPs, we measure the forward pass FLOPs over 100 iterations with batch size of 256, average the results, and multiply by the total number of training iterations. For inference FLOPSs we sum the forward-pass FLOPs across all sampling timesteps using a batch size of 32 and report the average over both timesteps and batch size. This procedure provides consistent and reproducible measure of computational cost across methods. Note that report floating- point operations (FLOPs), not multiply-accumulal operations (MACs), where one MAC corresponds to approximately two FLOPs.
Training speed. Here, we compare the actual run-time performance of each method on Stable Diffusion VAE latents. For all token-dropping methods, use fixed drop rate of 75%. At
18

---

Published as a conference paper at ICLR 2026
Table 10: Comprehensive performance comparison on ImageNet 512 512 class-conditioned generation with classifier-free guidance. + indicate whether lower or higher values are better, respectively. All metrics are evaluated with 250 sampling steps using the SDE sampler. Training and inference TFLOPs are measured with the DeepSpeed library.
Training TELOPs (x109)
Inference TELOPs L
Method
Epochs
#Paramss
FDD
FID
Pre.
Rec.
ADM (Dhariwal & Nichol, 2021)
400
2.85
0.84
0.53
Simple diffiusion (U-Net)
800
4.28
Simple diffusion (U-VIT-LE
800
4.53
MaskDiT (Zheng ct al., 2024)
800
730M
327.2
1,029
2.50
0.83
0.56
DIT-XI. (Pechles Xie, 2023)
600
675M
366.6
0.952
3.04
0.84
0.54
SIT-XL (Ma al., 2024)
600
675M
366.6
0.952
2.62
0.84
0.57
SiT-XI.
SPRINT
400
677M
184.8
0.954
53.6
2.23
0.83
0.57
SPRINTPDG
400
677M
184.8
0.471
46.9
1.96
0.83
0.58
the ImageNet resolution of 2562, SPRINT achieves a pretraining speed of 5.2 iters/sec, which more than 2 faster than the SiT baseline (2.5 iters/sec) and clearly outperforms other token- dropping baselines, including MaskDiT (4.57 iters/sec), MicroDiT (3.9 iters/sec), and Tread (4.7 iters/sec). At the higher ImageNet resolution of 5122, SPRINT maintains its advantage, achieving 2.01 iters/sec-ove 2.5% x faster than the SiT baseline (0.79 iters/sec) and again surpassing MaskDiT (1.77 iters/sec), MicroDiT (1.54 iters/sec), and Tread (1.79 iters/sec). This acceleration results in substantial reductions in wall-clock training time and GPU consumption, making large-scale diffusion model training significantly more practical and resource-efficiet.
VRAM memory consumption. In addition to reducing computational cost, SPRINT significantly lowers GPU memory requirements during training. For example, when training with a batch size of 32 and image resolution 256 on single GPU, SPRINT requires only 19.6 GB of memory, compared to 29.6 GB for the baseline SiT-XL/2 model. At resolution 512 our SPRINT requires 37.9 GB, whereas the baseline SiT-XL/2 model requires 77.7 GB. This represents 33.8% reduction in memory usage at 256² and 51.2% reduction at 5122. Such efficiency enables training with larger batch sizes or higher resolutions on the same hardware, making our method more accessible for researchers with limited GPU resources. Importantly, this reduction comes without sacrificing performance, underscoring the practicality of SPRINT resource-constrann constraind environments.
E BASELINES
E. BASELINE DETAILS ON TABLE 1
For fair system-level comparison in Tab. 1, apply the same pretraining and finetuning strategies, along with identical transformer block configurations, a fixed drop ratio of 75%, and consistent evaluation hyperparameters, across all baselines.
Progressive training. We adopt the same network architecture for progressive training. The model is first pretrained 128 x 128 images and then finetuned 256 x 256 images, with positional embeddings resized using bilinear interpolation during the resolution transition. This approach is slightly more efficient than SPRINT in terms of computational vs. 31.5 GFLOPs (x 10") at IM training iterations. However, despite the efficiency advantage, progressive training lags behind SPRINT performance and even fails match the baseline SiT results, underscoring its limited effectiveness.
per iteration, achieving 25.8 8
MicroDiT (Schwag et al., 2025). MicroDiT introduces deferred masking, where token dropping is applied only after several additional patch-mixing blocks. These modules allow local patch tokens to fuse information, enriching their semantic content. Following the original protocol, we modify the Si2 model by inserting patch-mixing modules composed of six transformer blocks. As shown in Tab. 1, this modification substantially increases computational cost and the number of parameters. Nevertheless, despite the additional overhead, MicroDiT underperforms relative to
19

---

Published as a conference paper at ICLR 2026
SPRINT, highlighting that the deferred masking strategy and additional compute does not translate into superior efficiency or accuracy.
Tread (Krause et al., 2025). Tread introduces a token-routing strategy which randomly dropped tokens early layers are routed directly to deeper layers. While this resembles SPRINT in that tokens bypass the middle layers, the two approaches differ fundamentally. In Tread, only the dropped tokens are bypassed, forcing the middle block to encode local noise information in order to estimate velocity. In contrast, SPRINT employs a full dense residual path that delivers complete local noise information to the decoder, freeing the middle block focus modeling global contextual information. This design choice makes SPRINT highly effective under aggressive dropping ratios (75%), whereas Tread fails under the same setting. We follow the implementation details provided in the original Tread paper.
E.2 MORE DISCUSSION ON OTHER BASELINES
MaskDiT (Zheng et al., 2024). MaskDiT introduces an additional reconstruction task for masked tokens alongside the diffusion objective, encouraging the model recover missing information and thereby improve contextual understanding. While this approach provides some efficiency gains, it requires an extra decoder module, increasing the model size from 675M 730M and adding computational overhead. Moreover, its effectiveness is limited to moderate dropping ratios (e.g., 50%). As shown in Tab. 3, these limitations restrict its overall efficiency compared to our framework. Specifically, MaskDiT requires 1600 training epochs to reach 65.4 FDD and 2.28 FID, whereas SPRINT surpasses this in just 200 epochs with 61.8 8 FDD and 2.01 FID. This underscores the superior effectiveness and efficiency of SPRINT over MaskDiT.
MDT (Gao et al., 2023). The Masked Diffusion Transformer (MDT) also aims to improve the contextual understanding of diffusion models through token dropping. They designed masked diffusion transformer with encoder-decoder split of the diffusion transformer, where the encoder processes masked tokens and forwards them to the decoder along with remaining tokens through additional side-inte inde-itterpoltoor model. It adds additional long shorcut connections between encoder blocks along with long full token input to all decoder blocks. MDT optimizes the reconstruction loss on masked tokens along with diffusion loss. The added complexity in the training and architectural changes is aimed for better generative performance. Similar to MaskDiT, this work also operates only with moderate token dropping ratios (e.g., (30%, 50%)). MDT does not work well with high token dropping ratio such as 75%.
F ADDITIONAL QUANTITATIVE RESULTS
F.1 IMAGENET 512X512 EXPERIMENT
In the main text, we have already demonstrate that SPRINT outperforms many existing training methods and state-of-the-art models at 256 class conditional image generation. In this experiment, we train our models to generation images at 5122 resolution.
Tab. 10 compares our method with strong basclines on ImageNet-IK class-conditional generation at 5122. We pre-train SPRINT for 1.8M iterations and finetune for 200K iterations (refer to Table 9). SPRINT achieves comparable or better generation quality while using substantially fewer training TFLOPs (x 10%): only 184.8 at 400 epochs, versus 366.6 for SIT-XL at 600 epochs. This demonstrates much faster convergence, reaching better FID with nearly 2x lower training cost. At inference, Path- Drop Guidance provides further benefits, nearly halving inference TFLOPs (0.471 vs. 0.952) while improving FDD. Overall, SPRINT consistently demonstrates efficiency compared to the baselines at 5122. by combining lower training and inference costs. Refer to Fig. 33 for qualitative results.
F.2 PERFORMANCE WITH FEW-STEP GENERATION
Tab. 11 compares SLT-XL/2 and SiT-XL/2 SPRINT across lower inference steps (NFEs), essential setting for achieving efficient and practical image generation. In real-world scenarios, reducing the number of function evaluations (NFEs) directly translates to faster sampling and lower inference
20

---

Published as a conference paper at ICLR 2026
Table 11: Performance of SIT-XL/2 and SPRINT across NFEs. Results are reported IM training iterations using the ODE sampler with 50K generated samples.
Method
NFE
FDD
FID 4
Pre.
Rec. 1
SIT-XL/2
200
132.3
2.18
249.9
0.81
0.59
+ SPRINT (Ours)
200
120.4
2.08
272.2
0.81
0.60
Gain A
+11.9
+0.1
+22.3
SIT-XL/2
133.1
2.19
249.6
0.81
0.59
+ SPRINT (Ours)
150
121.1
2.09
271.5
0.81
0.59
Gain A
+12.0
+0.1
+21.9
SIT-X1/2
100
134.7
2.22
248.4
0.81
0.58
+ SPRINT (Ours)
100
122.2
2.10
271.0
0.81
0.59
Gain A
+12.4
+0.12
+22.6
SIT-XL/2
50
140.6
2.34
244.0
0.80
0.58
SPRINT (Ours)
50
126.5
2.19
267.7
0.81
0.59
Gain A
+14.1
+0.15
+23.7
STT-XL/2
156.1
2.91
234.4
0.80
0.57
+ SPRINT (Ours)
25
138.2
2.59
256.3
0.80
0.58
Gain A
+17.9
+0.32
+21.9
SIT-XL/2
10
222.4
7.37
187.3
0.74
0.54
+ SPRINT (Ours)
10
191.7
6.29
211.3
0.74
0.54
Gain A
+30.7
+1.08
+24.0
cost, often at the expense of generation quality. While both models perform similarly at large NFEs (200), SPRINT consistently outperforms the baseline as the number of steps decreases. At 50 steps. SPRINT improves FID from 2.34 to 2.19 and IS from 244.0 to 267.7, and at only 10 steps it achieves much larger gain, reducing FID from 7.37 to 6.29 and improving IS from 187.3 to 211.3. These results highlight that SPRINT is more competitive under low-step inference. This demonstrates the strong representational power fused dense-shallow and sparse-deep features.
21

---

Published as a conference paper at ICLR 2026
G
ADDITIONAL QUALITATIVE RESULTS
G.1 VISUAL COMPARISON ON IMAGENET 256 x 256
SIT-XL/2-REPA SPRINT
SIT-XU/2+REPA
SIT-XL/2+REPA * SPRRINT
NONY
SET-XL/2+REPA
SIT-XL/2+REPA SIT-XL/2+REPA
+ SPRINT
Figure 10: SPRINT improves visual quality over baseline with only 57% of inference FLOPs (additional examples). We present samples from two SiT-XL/2 + REPA models after IM training iterations, where SPRINT is applied to one of the models. For our approach, we further incorporate the proposed Path-Drop Guidance (PDG), yielding higher visual quality compared to the REPA.
22

---

Published as a conference paper at ICLR 2026
G.2 UNSELECTED GENERATED RESULTS BY SPRINT ON IMAGENET 256 X 256
Figure 11: Unselected generation results of SIT-XL/2 SPRINT We use classifier-free
guidance with w 4.0. Class label "park bench" (706)
Figure 12: Unselected generation results of SiT-XL/2 SPRINT We use our path-drop
guidance with w 4.0. Class label "park bench" (706)
23

---

Published as a conference paper at ICLR 2026
Figure 13:
Unselected generation results of SiT-XL/2 SPRINT We use classifier-fre
guidance with = 4.0. Class label "hammerhead, hammerhead shark" (4)
Figure 14: Unselected generation results of SiT-XL/2 + SPRINT We use our path-drop
guidance with 4.0. Class label "hammerhead, hammerhead shark" (4)
24

---

Published as a conference paper at ICLR 2026
Figure 15: Unselected generation results of SiT-XL/2 SPRINT We use classifier-fre
guidance with = 4.0. Class label "magpic" (18)
Figure 16: Unselected generation results of SiT-XL/2 + SPRINT We use our path-drop
guidance with 4.0. Class label "magpie" (18)
25

---

Published as a conference paper at ICLR 2026
Figure 17: Unselected generation results of SiT-XL/2 + SPRINT We use classifier-fres
guidance with = 4.0. Class label "bullfrog, Rana catesbeiana" (30)
Figure 18: Unselected generation results of SiT-XL/2 SPRINT We use our path-drop
guidance with 4.0. Class label "bullfrog, Rana catesbeiana" (30)
26

---

Published as a conference paper at ICLR 2026
Figure 19: Unselected generation results of SiT-XL/2 SPRINT We use classifier-free
guidance with = 4.0. Class label "tusker" (101)
Figure 20: Unselected generation results of SiT-XL/2 + SPRINT We use our path-drop
guidance with 4.0. Class label "tusker"
27

---

Published as a conference paper at ICLR 2026
Figure 21: Unselected generation results of SiT-XL/2 SPRINT We use classifier-free guidance with = 4.0. Class label "beagle" (162)
Figure 22: Unselected generation results of SiT-XL/2 + SPRINT We use our path-drop
guidance with 4.0. Class label "beagle" (162)
28

---

Published as a conference paper at ICLR 2026
P
Figure 23: Unselected generation results of SiT-XL/2 + SPRINT We use classifier-fre
guidance with = 4.0. Class label "coffeepot" (505)
Figure 24: Unselected generation results of SiT-XL/2 + SPRINT We use our path-drop
guidance with 4.0. Class label "coffeepot" (505)
29

---

Published as a conference paper at ICLR 2026
JUBOS
Figure 25: Unselected generation results of SiT-XL/2 + SPRINT We use classifier-fres
guidance with = 4.0. Class label "computer keyboard, keypad" (508)
#
le
R
Figure 26: Unselected generation results of SiT-XL/2 + SPRINT We use our path-drop
guidance with 4.0. Class label "computer keyboard, keypad" (508)
30

---

Published as a conference paper at ICLR 2026
Figure 27: Unselected generation results of SiT-XL/2 + SPRINT We use classifier-fres free guidance with = 4.0. Class label "convertible" (511)
Figure 28: Unselected generation results of SiT-XL/2 + SPRINT We use our path-drop
guidance with 4.0. Class label "convertible" (511)
31

---

Published as a conference paper at ICLR 2026
Figure 29: Unselected generation results of SiT-XL/2 + SPRINT We use classifier-fre
guidance with 4.0. Class label "cornet, horn, trumpet, trump" (513)
Figure 30: Unselected generation results of SiT-XL/2 + SPRINT We use our path-drop
guidance with 4.0. Class label "cornet, horn, trumpet, trump" (513)
32

---

Published as a conference paper at ICLR 2026
TE
Det
Figure 31: Unselected generation results of SiT-XL/2 + SPRINT We use classifier-fres guidance with = 4.0. Class label "cowboy hat, ten-gallon hat" (515)
Ldd
HOL
Figure 32: Unselected generation results of SiT-XL/2 + SPRINT We use our path-drop
guidance with 4.0. Class label "cowboy hat, ten cen-gallon hat" (515)
33

---

Published as a conference paper at ICLR 2026
G.3
GENERATED RESULTS BY SPRINT ON IMAGENET 512 X 512
Figure 33: Generation results of SiT-XL/2 SPRINT We use our path-drop guidance with w
= 3.0.
34

---

Published as a conference paper at ICLR 2026
G.4 ADDITIONAL FEATURE PCA VISUALIZATION
In the main text (Figure 6), we analyzed PCA visualizations of features from So and showing that SPRINT learns more noise-invariant and semantically vivid representations than the SiT baseline across diffusion timesteps. Figure 34 presents additional examples of these dense-shallow and sparse-deep features learned by SPRINT, contrasted with those from standard SiT-XL/2 model trained with full tokens.
t=1
t 0
Xg
SIT-XL/2 + Ours
fg
Be
SIT-XL/2
go
SIT-XL/2 + Ours
fg
go
fg
SIT-XL/2
ge
Figure 34: SPRINT improves feature semantics (additional examples). We visualize PCA features of Ja and go from two SiT-XL/2 models at 400K iterations. The top rows show the model trained with SPRINT, while the bottom rows show the baseline. Compared to the baseline, features from SPRINT exhibit clearer semantic structure across both images.
35

---

Published as a conference paper at ICLR 2026
H LIMITATION AND FUTURE WORK
Our study is limited by the available computational resources, which prevented us from conducting experiments on large-scale text-t-immae video diffusion models. Exploring the scalability of SPRINT in such settings remains an important direction. In partieular, the quadratic complexity of transformers becomes increasingly prohibitive model size and input resolution grow. Since SPRINT is specifically designed to reduce redundant computation in deeper layers, we expect it to be especially beneficial for large-scale architectures where efficiency bottlenecks are most severe. Thus, extending SPRINT to other modalities such video, 3D, multi-modal generative models an exciting direction. These domains pose even greater computational and memory challenges, particularly in video, where the temporal dimension compounds complexity, making our sparse-dense residual fusion especially relevant for future research.
Another promising avenue is the integration of SPRINT with recent advances in efficient attention mechanisms and scalable training strategies. Such combinations could amplify the benefits of our approach, further reducing training and inference costs while maintaining improving performance.
36


Reviews:
[{"review_id": "cS1ix4cUbd", "reviewer": "Reviewer_pKvE", "summary": "This paper proposes SPRINT, a method to accelerate Diffusion Transformer (DiT) training through sparse-dense residual fusion. SPRINT processes all tokens in shallow layers for local details, drops 75% of tokens in deep layers for global semantics, and fuses outputs via residual connections. Training uses sparse pre-training followed by brief full-token fine-tuning. On ImageNet-1K 256×256, SPRINT achieves 9.8× training speedup with comparable quality.", "questions": "(1)Why is there no comparative validation using features from different layers for unconditional guidance?\n(2)After pre-training with 75% drop ratio, if full-token inference is performed directly (without fine-tuning), how much would the performance degrade?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1)SPRINT achieves 9.8× training speedup on ImageNet-1K while maintaining comparable quality. The method adds only 0.3% parameters and preserves standard DiT blocks, making it easy to integrate. Strong generalization across architectures (SiT, U-ViT, REPA) demonstrates practical value.\n(2)Path-Drop Guidance (PDG) halves inference FLOPs while improving quality. Comprehensive experiments reveal complementary roles of sparse-deep and dense-shallow features, providing valuable insights into DiT representation mechanisms and explaining why the sparse-dense fusion design is effective.", "weaknesses": "(1)The paper claims that two-stage training can \"close the train-inference gap,\" but does not quantify how large this gap actually is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oyE12fcBVZ", "reviewer": "Reviewer_wnyK", "summary": "This paper proposes SPRINT, a training method for Diffusion Transformers (DiTs) that aims to reduce training costs through aggressive token dropping (up to 75%). The core idea is to partition the DiT into encoder-middle-decoder components, where the encoder processes all tokens, the middle blocks operate on sparse tokens, and outputs are fused through residual connections. The authors claim 9.8x training savings with comparable quality on ImageNet-1K $256^2$.", "questions": "1. Can you provide a direct, controlled comparison with MDTv2 using identical experimental settings?\n2. What theoretical or empirical evidence supports the advantages of the residual fusion over MDTv2's approach?\n3. Why are the FLOPs for MDTv2 missing in Table 3? It has open-sourced official code, you should check it and measure the costs.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Practical Problem: The paper addresses the important issue of quadratic training costs in DiTs, which is highly relevant for the community.\n2. Strong Empirical Results: The reported 9.8x training speedup with maintained quality is impressive if valid.\n3. Architecture Agnostic: The method appears to work across different architectures (SiT, UViT) and can be combined with other techniques like REPA.\n4. Comprehensive Experiments: The paper includes extensive ablations and analysis across multiple settings.", "weaknesses": "Major Concerns\n\n1. Limited Technical Novelty.\nThe core contribution appears to be a modification of MDTv2, essentially replacing the side-interpolator with simple residual connections. The encoder-middle-decoder architecture is questonable, and the paper fails to provide compelling theoretical or empirical justification for why this specific design should outperform existing methods like MDTv2.\n\n2. Insufficient Comparison with Prior Work.\nThe paper does not adequately explain why SPRINT should be superior to MDTv2. The fundamental question remains unanswered: what specific advantages does replacing MDTv2's side-interpolator with residual connections provide? The paper lacks rigorous analysis of this key design choice.\n\n3. Questionable Performance Claims.\nFrom Table 3, SPRINT appears to underperform compared to MDTv2 (in terms of FID). This contradicts the paper's claims of superiority and raises questions about the experimental setup and evaluation fairness.\n\n4. Lack of Theoretical Foundation.\nThe paper provides insufficient theoretical justification for why the proposed encoder-middle-decoder architecture can support such high drop rates (75%). The explanation about shallow vs. deep layer specialization is intuitive but lacks rigorous analysis or proof.\n\n5. Missing Critical Analysis.\nThe paper doesn't adequately address:\n- Why simple residual fusion should be better than more sophisticated fusion mechanisms\n- Why SPRINT can tolerate 75% drop rate\n- How the method compares to MDTv2 in controlled settings with identical experimental conditions\n\nMinor Issues\n- Writing Quality: Some sections lack clarity, particularly the technical description of the fusion mechanism.\n- Experimental Setup: More details needed on fair comparison protocols with baseline methods.\n- Ablation Studies: While extensive, the ablations don't address the core question of why (if really so) this approach works better than MDTv2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GZJlzbS89Q", "reviewer": "Reviewer_orVZ", "summary": "This paper introduces SPRINT, a novel framework for efficient training of Diffusion Transformers (DiTs) by leveraging sparse-dense residual fusion. It enables aggressive token dropping (up to 75%) while preserving representation quality, significantly reducing training costs (up to 9.8×) and inference FLOPs. SPRINT trains DiTs in two stages: sparse pre-training and short full-token fine-tuning to bridge the train-inference gap. It also introduces Path-Drop Guidance (PDG), a more efficient alternative to classifier-free guidance, further improving generation quality and efficiency. The method is simple, architecture-agnostic, and applicable across various resolutions and models.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Good performance\n+ The proposed Dense shallow path  and sparse deep path can effectively accelerate the training speed.", "weaknesses": "1. More discussion on Path-Drop Guidance should be included in the Introduction. Currently, the manuscript treats it as merely a supplementary design.\n\n2. The font size in the tables should be consistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3nr1zw3Jnl", "reviewer": "Reviewer_VDGb", "summary": "This paper addresses the prohibitive quadratic training cost of Diffusion Transformers (DiTs). The authors propose SPRINT (Sparse-Dense Residual Fusion), a novel training and architecture strategy that enables aggressive token dropping (up to 75%) while preserving quality. SPRINT divides the DiT into three parts: a dense shallow encoder (fθ) that processes all tokens to capture local detail, a sparse deep middle block (gθ) that processes a subset of tokens to model global semantics, and a decoder (hθ) that fuses features from both paths via a residual connection. The method uses a two-stage training schedule (long sparse pre-training followed by short full-token fine-tuning) and a structured group-wise token sampling strategy to ensure local coverage. Additionally, the paper introduces Path-Drop Guidance (PDG), an efficient substitute for Classifier-Free Guidance (CFG) that nearly halves inference FLOPs by bypassing the expensive middle blocks during the unconditional pass. Experiments on ImageNet show SPRINT achieves significant training speedups (e.g., 9.8x fewer TFLOPs to reach comparable quality to SiT-XL) and improved inference efficiency with PDG.", "questions": "1.The performance of PDG is excellent. Does the SPRINT training enable PDG? In other words, could PDG (i.e., using only the first few layers fθ for the unconditional pass) be applied as an inference-time optimization to a standard, densely trained SiT model, or is the sparse-dense training process necessary for fθ to become an effective unconditional predictor?\n2. Could you provide experimental results on text-to-image models (such as PixArt, Flux), similar to the fine-tuning experiments on FLUX, to more convincingly demonstrate the effectiveness of your proposed method?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Significant Training Efficiency: The primary strength is the massive reduction in training computation. The paper demonstrates that SPRINT can reach a quality level comparable to the 1400-epoch SiT-XL baseline in only 200 epochs, translating to a 9.8x reduction in TFLOPs. This is a highly practical and valuable contribution. \n 2. Novel and Effective Inference Acceleration (PDG): The proposed Path-Drop Guidance (PDG) is an excellent finding. It cleverly repurposes the SPRINT architecture to create an efficient alternative to CFG, using the shallow path as a \"\"weaker network\"\". This method nearly halves inference FLOPs while also improving generation quality, as shown in Tables 1, 2, 3, and 10, and Figure 7. \n 3. Simplicity and Generality: The SPRINT framework is conceptually simple and not tied to a specific architecture. The authors demonstrate its broad applicability by successfully integrating it with SiT, U-ViT, and the alignment-based REPA, showing consistent and significant improvements in all cases. \n 4. Strong Empirical Validation: The paper is supported by extensive experiments and insightful ablations. The validation of the sparse-dense design (Table 4), the structured sampling strategy (Table 5), and the optimal layer allocations (Tables 6, 8) builds strong confidence in the method's design. The analysis of feature specialization (Fig. 4) provides good intuition for why the method works.", "weaknesses": "1. Limited Ablation on Fine-tuning Stage: The paper adopts a fixed two-stage schedule: long sparse pre-training followed by a 100K-iteration full-token fine-tuning stage. While effective, the sensitivity to the duration of this fine-tuning is not analyzed. It is unclear how much fine-tuning is necessary to close the train-inference gap or if more fine-tuning would yield further gains. An ablation on this hyperparameter would strengthen the paper. \n 2. Conceptual Justification for PDG: The paper empirically shows that PDG works well by using the shallow path fθ as the unconditional estimate, inspired by Auto Guidance. This is effective, but a deeper conceptual analysis of why fθ is a suitable unconditional predictor would be beneficial. Does the SPRINT training force fθ to learn an \"\"average\"\" or \"\"blurry\"\" representation that mimics a true unconditional pass? A brief discussion would be insightful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)