SparseDiT: Token Sparsification for Efficient Diffusion Transformer
Authors: Shuning Chang
Pichao WANG
Jiasheng Tang
Fan Wang
Yi Yang
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Transformer
Sparse Tokens
Efficient DiT
Primary area: deep_learning

Abstract: Diffusion Transformers (DiT) are renowned for their impressive generative performance; however, they are significantly constrained by considerable computational costs due to the quadratic complexity in self-attention and the extensive sampling steps required. While advancements have been made in expediting the sampling process, the underlying architectural inefficiencies within DiT remain underexplored. We introduce SparseDiT, a novel framework that implements token sparsification across spatial and temporal dimensions to enhance computational efficiency while preserving generative quality. Spatially, SparseDiT employs a tri-segment architecture that allocates token density based on feature requirements at each layer: Poolingformer in the bottom layers for efficient global feature extraction, Sparse-Dense Token Modules (SDTM) in the middle layers to balance global context with local detail, and dense tokens in the top layers to refine high-frequency details. Temporally, SparseDiT dynamically modulates token density across denoising stages, progressively increasing token count as finer details emerge in later timesteps. This synergy between SparseDiT’s spatially adaptive architecture and its temporal pruning strategy enables a unified framework that balances efficiency and fidelity throughout the generation process. Our experiments demonstrate SparseDiT’s effectiveness, achieving a 55\% reduction in FLOPs and a 175\% improvement in inference speed on DiT-XL with similar FID score on 512$\times$512 ImageNet, a 56\% reduction in FLOPs across video generation datasets, and a 69\% improvement in inference speed on PixArt-$\alpha$ on text-to-image generation task with a 0.24 FID score decrease. SparseDiT provides a scalable solution for high-quality diffusion-based generation compatible with sampling optimization techniques.  Code is available at https://github.com/changsn/SparseDiT.

SparseDiT: Token Sparsification for Efficient Diffusion Transformer
Shuning Chang 23 Pichao Wang2*
Jiasheng Tang? Fan Wang? Yi Yang'
'Zhejiang University
Damo Academy, Alibaba Group "Hupan Lab shuning. can@alibaba com
Abstract
Diffusion Transformers (DiT) are renowned for their impressive generative perfor- mance; however, they are significantly constrained by considerable computational costs due to the quadratic complexity in self-attention and extensive sampling steps required. While advancements have been made in expediting the sampling pro- cess, the underlying architectural inefficiencies within DiT remain underexplored. We introduce SparseDiT, novel framework that implements token sparsification across spatial and temporal dimensions to enhance computational efficiency while preserving generative quality. Spatially, SparseDiT employs tri-segment archi- tecture that allocates token density based on feature requirements at each layer: Poolingformer in the bottom layers for efficient global feature extraction, Sparse- Dense Token Modules (SDTM) in the middle layers to balance global context with local detail, and dense tokens the top layers to refine high-frequency details. Tem- porally, SparseDiT dynamically modulates token density across denoising stages, progressively increasing token count finer details emerge in later timesteps. This synergy between SparseDiT's spatially adaptive architecture and its temporal pruning strategy enables a unified framework that balances efficiency and fidelity throughout the generation process. Our experiments demonstrate SparseDiT's effectiveness, achieving 55% reduction FLOPs and a 175% improvement inference speed on DiT-XL with similar FID score on 512 x12 ImageNet, 56% reduction FLOPs across video generation datasets, and a 69% improvement inference speed on PixArt-a on cext-to-image generation task with 0.24 FID score decrease. SparseDiT provides a scalable solution for high-quality diffusion-bar based generation compatible with sampling optimization techniques. Code is available at https://giithhn com/ wwwngshnnnnanaeee Sthangsn/SparseDi77
Introduction
43 3 have emerged as powerful tools in visual generation, producing photorealistic images and videos by gradually refining structured content from noise. Leveraging the scalability of Transformers. Diffusion Transformer (DiT) models [41] extend these capabilities to complex and detailed tasks [3] 62 and form the backbone of advanced generation frameworks such as Sora (6). Yet, despite their impressive generative capabilities, DiT models face significant computational limitations that restrict their broader applicability, especially in scenarios where efficiency is paramount.
Diffusson models 18
12
The computational burden of DiT models arises mainly from the need for numerous sampling steps the denoising process and the quadratic complexity of Transformer structures with respect to number of tokens. Although recent research has focused reducing this burden by accelerating the sampling process through techniques such as ODE solvers [51 30], flow-based methods [26] 28).
Work done at Alibaba Group, and now affiliated with Amazon.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Sansinn Dapa
Layer
Layer
Layer
Layer 26
Layers
(a)
(b)
Figure 1: (a) The attention maps in different self-attention layers. Zoom-in for better visibility. (b) The normalized variances of attention maps in DiT-XL. Different curve lines represent different sampling steps.
these approaches often overlook the core architectural inefficiencies of DiT itself. Unlike U-Net-based architectures 44 which mitigate complexity through a contracting exxpaniiie structure, DìT's reliance on token-level self-attention incurs high costs that scale with model size and token count. This bottleneck highlights a critical need for DiT-specific innovations that manage token density intelligently, balancing efficiency and quality without sacrificing generation fidelity.
and knowledge distillation 53 43
35 63
To address this challenge, we examine the distribution of attention across DiT's layers and sampling steps, seeking to uncover structural patterns that could guide more efficient token management. As illustrated in Figure Ta| attention maps reveal that tokens capture different levels of granularity across layers: in the bottom layers, such as layer O, attention maps approximately appear uniform distribution, indicating focus on broad, global features akin to global pooling. Meanwhile, middle layers alternate in their attention, with certain layers (e.g.. layer and layer 26) capturing local details, while others (e.g., layer 22) emphasize global structure. This observation quantified in Figure lb] where the variance in attention scores highlights DiT's consistent pattern of alternation between global and local feature extraction across layers. addition to spatial analysis, Figure [b] also reveals insights across sampling steps. We observe that, while the pattern of alternation between global and local focus remains stable, attention variance increases as the denoising process advances, indicating growing emphasis on local information at lower-noise stages. This analysis yields three core insights: (I) the initial self-attention layers exhibit minimal variance, showing low discrepancy in feature extraction across tokens; (2) DiT's architecture inherently alternates between global and local feature focus across layers, pattern consistent across all sampling steps; and (3) as denoising progresses, the model increasingly prioritizes local details, dynamically adapting to heightened demands for detail at later stages.
Building on these insights, we propose SparseDiT, which reimagines token density management as dynamic process adapting across both spatial layers and temporal stages. Spatially, shallow layers capture smooth, global features, making complex self-attention less efficient. Subsequent layers alternate between high-frequency local details and www-frquency global information, where sparse tokens efficiently capture global features, and dense tokens refine local details. Temporally, denois- ing advances, the need for local detail increases, motivating a dynamic token adjustment strategy. This dual-layered adaptation integrates layer-wise token within each sampling step and a tmesttp-wisee pruning strategy that adjusts token density dynami- cally. Together, these spatial and temporal adaptations allow SparseDiT to achieve computational efficiency without sacrificing high-fidelity detail, as reflected in SparseDiT's architectural design and pruning strategy, tailored to meet the unique spatial and temporal demands of the generation process.
balancing efficiency and detail
SparseDiT's architecture employs three-segment design that aligns token density with the features each layer captures. In the bottom layers, we replace self self-attention with a poolingformer structure to capture broad global features through average pooling, reducing computation while preserving essential structure. In the middle layers, SparseDiT introduces Sparse- Dense Token Modules (SDTM), blending sparse tokens for global structure with dense tokens to refine local details. This design enables SparseDiT to retain efficiency while maintaining stability and
Dia Architecture:

---

SDTM
SDTM
Sparte
Denee Tokers
Bottom
Top
Middle
Architecture of SparseDiT.
Denoising Process
Data
Nolse
7maz
Skep
57/4
(b) Timestep-win wise pruning rate strategy.
Figure 2: Architecture of SparseDiT and timestep-wist pruning rate strategy. The architecture consists of three segments: bottom, middle, and top. The bottom segment includes poolingformers (PF). The middle segment comprises multiple sparse-dense token modules (SDTMs), where sparse token generation transformers ("Sparse Gen") and dense token recovery transformers ("Dense Rec") alternate to balance global and detailed information processing. The top segment contains standard transformers ("Tr") for final processing. During the denoising process, we apply varying pruning rates at different stages (depicted four stages).
detail. In the top layers, the model processes all tokens densely, focusing on high-frequency details for refined output quality.
Timestep-War Wise Pruning Rate Strategy: Complementing spatial adjustments, SparseDiT's timestep- wise pruning rate strategy adapts token density across denoising stages. In early stages, where broad, low-frequency structures dominate, SparseDiT applies a high pruning rate, conserving resources. As the process progresses and intricate details emerge, the pruning rate decreases, allowing for gradual increase in token density. This adjustment aligns computational effort with each stage's complexity, dynamically balancing efficiency and detail optimize resources for high-fidelity output.
Our empirical results substantiate the effectiveness SparseDiT's integrated approach. SparseDiT reduces the FLOPs of DiT-XL by 55% and improves inference speed by 175% on 512x512 Ima- geNet [11] images, with only a increase 0.09 in FID score [17.. On the Latte-XL dataset [34), Spa achieves 56% reduction in FLOPs across standard video generation datasets, including FaceForensics 45), SkyTimee Tiymeelaeee 660. UCF101 (54] and Taichi-HD [50]. Additionally, on the
more challenging text-to-image generation task, achieve 69% improvement in inference speed on PixArt-a with 0.24 FID score reduction. These results demonstrate SparseDiT's capability to provide efficient, high-quality architecture for liffusion-based generation, compatible with further sampling optimization techniques for enhanced efficiency.
Related works
Efficient sampling process. This direction focuses on optimizing sampling steps in diffusion models. Some approaches [51] I 31 design new solvers to achieve faster sampling with fewer
58 21 closely related to diffusion models, achieving by distilling pre-trained diffusion models or training from scratch. These models learn one-step mapping between noise and data, and all the points the sampling trajectory map to the same initial data point. Flow-based [26] 28
steps. Consistency models (CMs) 53 52
[3 771) approaches straighten the transport trajectories among different distributions and thereby realize efficient inference. Knowledge distillation are applied by some methods 5
72
47
32
35 63 to reduce the sampling steps and match the output of the
combined conditional and unconditional models. Parallel sampling techniques 49 70 56] employ
Picard- iteration or Fourier neural operators for solving SDE/ODE..
Efficient model structure. Another avenue focuses on improving inference time within a single model evaluation. Some methods attempt to compress the size of diffusion models via pruning [14]

---

(a) Standard DiT-XL..
(b) Modifying first two attention in DiT-XL.
Figure 3: Comparison of images generated by the standard DiT-XL model and a modified DiT-XL model where the first two attention maps are replaced with full-one matrices without fine-tuning. Both sets of images exhibit similar visual quality.
or quantization 25 48 Spectral Diffusion 61 boosts structure design by incorporating frequency
dynamics and priors. OMS- DPM [27] proposes model schedule that selects small models or large models specific sampling steps to balance generation quality and inference speed. Deepeache [33. TokenCache [29]. and Fit (66] reuse or share similar features across sampling steps. The early stopping mechanism in diffusion is explored in 24
55
Most of the above methods are designed for general diffusion models. Recently, the Diffusion Transformer (DiT) has emerged as a more potential backbone, surpassing the previously dominant U-Net architectures. Unlike U-Net-based models, DīT's inefficiency stems largely from the self- attention mechanism. Limited research has specifically addressed the efficiency of DiT-based models. Pu et al. [42 identifies the redundancies in the self attention and adopts timestep dynamic mediator tokens to compress attention complexity, achieving some performances but limited FLOPs reduction, as this approach primarily reduces the number of keys and values rather than the overall tokens count.
In Vision transformers for classification task, numerous methods (59] 15 7 73 36
67
successfully removed redundant tokens to improve performance-effiiente trade-offs. ToMeSD S
the first to explore token reduction in diffusion models. However, its performance on DiT was U-Net can be regarded sparse network due to its contracting
suboptimal, as shown in [37 68) and expansive structure. EDT 10] applies a novel architecture similar to U-Net, while they still have 1.4 FID score gap compared to its baseline model MDTv2 [16] via 2,000K iterations. These findings indicate that directly transferring token reduction techniques from classification tasks or reusing U-Net structures may be insufficient for DIT. Instead, specialized token reduction strategy needed. DyDiT (68) and DiffCR (64] design dynamic networks to compress DiT from multiple dimensions, such as token, layer, attention head, channel.
3 Method
We first introduce SparseDiT architecture in Section 3.1 followed by our timestep-wise pruning rate strategy in Section 3.2
Finally, Section 3.3 details the initialization and fine-tuning of our network.
3.1 SparseDiT
Overview architecture. The SparseDiT architecture is illustrated in Figure 2a where the trans- former layers of DiT based model are divided into three groups: bottom, middle, and top. The bottom layers leverage poolingformers to efficiently capture global features. The middle layers contain multiple sparse-dense token modules (SDTM), which decouple the representation process into global structure capture and local details enhancement using sparse and dense tokens, respectively. The top layers retain the standard transformers, processing dense tokens generate the final predictions at each sampling step. The primary computational savings are achieved through sparse tokens, hence the majority of transformer layers are located in the middle section maximize efficiency.
Poolingformer in the bottom layers. As shown in the first subfigure Figure Ta attention scores in the bottom layers exhibit nearly uniform distribution, with each token evenly extracting global features, akin to a global average pooling operator. To investigate further, we conducted toy experiment where the attention maps in the first two transformer layers were replaced with

---

matrices filled with constant value (e.g., 1), without any fine-tuning. Figure 3 shows that, using the same initial noise and random seed, the images generated by the original DiT-XL and the modified DiT-XI. are nearly identical, suggesting that complex self-attention calculations offer limited additional information. Given that self-attention in the bottom layers captures global features, we further question whether sparse tokens can be used in these layers. However, experiments (in Table o demonstrated that applying sparse tokens in the bottom layers results in unstable training, highlighting the necessity of retaining complete tokens in these layers.
Based on the above analysis, we adopt poolingformers to replace the original transformers. In poolingformer, we remove queries and keys as attention maps are not computed. Instead, we perform global average pooling on the value V RNC and integrate it into input tokens X, which can be represented as
X
(I)
where V is the mean along the dimension of i.., 8 N EV E RI and then repeating N i.e,, times to shape N x C. The parameters from adaL. block are omitted in all the equations for brevity. Our poolingformer can be viewed as a special case of the model in 655] behaving identically when the pooling kernel size is equal to the input size.
Sparse-dense token module. We present the sparse-dense token module (SDTM) to generate sparse and dense tokens processed by the corresponding transformer layers. The high-level idea is to decouple global structure extraction and local detail extraction. Sparse tokens capture global structure information and reduce computational cost, while dense tokens enhance local detail and stabilize training. Sparse and dense tokens are converted to each other within SDTM.
Initially, SDTM introduces set of sparse tokens X, E IR RRC where M is the number of sparse tokens. typically M < N. These sparse tokens can be initialized by adaptively pooling the dense tokens X. We define pruning rate r = - M/N to represent the sparsity degree. To store the structure information, we first reshape the dense tokens X R into the latent shape. For instance, if the input is an image, we reshape it into shape H x W x C, then pool it across the spatial dimensions to shape H' x W" C, where H' x W M. The intentions of spatial pooling initialization are two-fold fold. First, the initial sparse tokens can be distributed uniformly in space and the representation of each sparse token associated with a specific spatial location, which is beneficial for downstream tasks such image editing (20). Second, it can prevent the semantic tokens from collapsing to one point in the following layers. Sparse token interact with full-size dense tokens via an attention layer to integrate the global information:
X, = X,+ + MHA(X,, X),),
where the triplet input of MHA A are queries, keys, and values in turn.
The generated sparse tokens are fed into subsequent transformer layers, referred to as sparse trans- formers. Since M < N, our SDTM can substantially reduce computational cost.
Following this, restore dense tokens from sparse tokens. First, reshape the sparse tokens X, E R MxC XC into strueture shape H' x W' x C and upsample it to the same shape as input dense tokens X #x We introduce two linear layers to combine upsampling sparse tokens with dense tokens, which is represented as:
X mergeed U (SSamlle(X.) W1 + X W2.
where Wi, We E CCC are the weights of two linear layers. Then, to further incorporate sparse tokens, we utilize an attention layer to perform a reverse operation of the generation of sparse token to produce the restored dense tokens, which is written as:
X Xmerged + M MHA(X,mm A(eAAeeeeedd ged. X,. X²).
At the end of SDTM, several transformer layers, termed dense transformers, process full-size dense tokens to enhance local details.
We cascade multiple SDTMs in our network. By repeating sparse and dense tokens, the network effectively preserves both structural and detailed information, achieving substantial reductions in computational cost while maintaining high-quality generation outputs.
5

---

Table 1: Comparison of performance between SparseDiT and DiT for class-conditional image generation task on ImageNet.
Resolution
Model
FLOPs (G)
Throughput (img/s)
FID 4
sFID
18
Precision
Recall
DiT-B
23.01
7.84
9.07
5.44
121.07
0.74
0.54
256
Ours (r E (0.61.0.8))
14,34 (-38%)
13.2 (+68%)
8.23
6.20
134,30
0.74
0.54
DIT-XI
2.27
4.60
278, 24
0.57
256 256
Ours ) Ours (r E (0.61, 0.86)
88.91 (-25%)
2.13 (+35R)
2.23
4.62
278.91
0.84
0.58
68.05 (-43%)
2.95 (+87%)
2.38
4.82
276.39
0.82
0.58
DIT-X1.
0.249
512 512
Ours (r (..6))
286 (-46%))
0.609 (+145%)
2.96
5,00
242.4
0.84
0.54
0.96))
(-55%%)
(+175%)
236.56
(a) Images generated by DIT.
(b) Images generated by SparseDiT (-43% FLOPs),
Figure 4: Generating images from DiT-XL and SparseDiT-XL with the same random seed.
3.2 Timestep-wise pruning rate
We have observed that the tokens display varying denoising behavior over sampling steps. They generate the low-freque www-freqenccy global structure information in the early denoising stage and the high frequency details the late denoising stage. The token count requirement is progressively increase along sampling steps. We exploit this by dynamically adjusting the pruning rate r across sampling steps, increasing tokens as sampling progresses.
Given 7 sampling steps, we propose sample-specific approach to dynamically adjust the pruning rate r. thereby controlling the number of sparse tokens across the sampling steps. We define range for r, such that E [Tmin: I- Observing that generation quality is highly relative with the later denoising stages (in Section 4.4), we hold r constant at Tain for the first T/4 sampling steps. Subsequently, we adjust the pruning rate r linearly based on the current sampling step f.. The specific formula for provided as follows:
Tmins 1T
4 < T/4,
r
1-T 37
T/4 <4<T.
(5)
mmin
mam:
During training, we sample step 1; and compute the corresponding according Eq. 5 However, the input tokens should the same to train the model in batch, and the randomness of the sampling from 7 should also be maintained. To solve this contradiction, we modify the linear function of Eq. when 7'4 < & < I" to piecewise function. Normally, the model is trained by multiple GPUs. We request sampled specific piece in each GPU. Therefore, in each iteration, it can achieve both uniformly random sampling and batch training.
3.3 Initialization and fine-tuning
Our method fine-tunes pre-trained DiT models to improve efficiency. The processes generating sparse tokens and restoring dense tokens utilize off-the-shed transformers, avoiding the need for additional networks. During fine-tuning, the parameters of transformers DiT-based model are loaded into the corresponding transformers in our SparseDiT, with two exceptions: (1) queries and keys are absent in Poolingformerss hence related parameters in pre-trained model are not required; (2) the weights W, and W2 in Eq. are initialized by full zeros and identity matrix, receptively.

---

Table 2: FVD scores of Sp in class-conditional video generation task on four mainstream video generation datasets. We do not find the official reference sets of these dataset when we compute FVD scores, hence we build the reference sets by ourselves and use the official checkpoint to re-compute the FVD scores of Latte- XL in this table.
Model
FLOPs(G)
Throughput (clipls)
FaceForensic
Sky Timelapse
UCF101
Taichi-HD
Latte-XI.
1894
0.108
24.10
40.78
284.54
89.63
Ours (r E (0.80,0993) 0.93)
828 (-56%)
0.228 (+111%)
24.51
39.02
288.90
84.13
4 Experiments
SparseDiT is applied in three representative DiT-based models, DiT Latte [34), and PixArt- [8) for class condiiional image generation, class-conditional video generation, and text-to-image generation, respectively.
Fine-tuning overhead. All training settings and hyperparameters follow their respective papers. Fine-tuning requires approximately 6% of the time needed for training from scratch, e.g., 400K iterations for DiT-XL fine-tuning.
4.1 Class-conditional image generation
Experimental setting. We conduct our experiments on ImageNet- 1k [I]] at resolutions of 256 2556 and 512x x 512, following the protocol established in DiT. For DiT-XL, the model consists of 2, 24, and 2 transformers in the bottom, middle, and top segments, respectively. The middle segment includes 4 sparse-dense token modules, which comprise sparse token generation transformer, 3 sparse transformers, dense token recovery transformer, and 1 dense transformer. For DiT-B, the bottom, middle, and top segments contain 1, 10, and transformer layers, respectively. The middle segment consists of 2 sparse-dense token modules, including I sparse token generation transformer, 2 sparse transformers, 1 dense token recovery transformer, and I dense transformer. As poolingformers and sparse transformers can interfere with the function of position embedding, we reintroduce sine-cosine position embedding at each stage of dense token generation. For DiT-XL, we use the checkpoint from the official DiT repository, while for DĪT-B, we utilize the checkpoint provided by [40]. Following prior works, we sample 50,000 images to compute the Fréchet Inception Distance (FID) [7 using
the ADM TensorFlow evaluation suite [2], along with the Inception Score (IS) [66. sFID 38 and
Precision-Recal metrics (23] Classifier-free guidance [9 (CFG) is set to 1.5 for evaluation and 4.0
for visualization. Throughput is evaluated with batch size of 128 on an Nvidia A100 GPU.
Main results. The results of our approach on the class-conditional image generation task using ImageNet are presented in Table [ We evaluate our method on two model sizes, DiT-B and DiT-XL, and at two resolutions. 256 256 and 512 512. pruming rates E [0.61, 0.86). SparseDiT-XL achieves a 43% reduction in FLOPs and an 87% improvement in inference speed, with only 0.11 increase in FID score. This indicates significant redundancy within DiT, as using only about 25% tokens in specific layers maintains similar performance levels. By increasing the token count, we observe slight improvements over baseline performance with 145% improvement in throughput. The impact of our approach is even more pronounced at a resolution of 512 x 512. With a higher pruning rate than the 256 256 resolution, our method yields a superior performance-effien -ffeniienefc trade-off. By pruning over 90% of the tokens, we achieve 55% reduction in FLOPs and 175% increase in throughput, with only a 0.09 increase FID score. When reducing the pruning rate further, our results slightly surpass the baseline, delivering 145% throughput improvement. ToMeSD [5] is the first approach to reduce tokens. However, it applics token merging and recovery at every layer in a generic manner, resulting in significant performance drop. For example, with a 0. merging rate on DIT-XL, ToMeSD achieves an FID score of 14.74, which is substantially worse than our result. A detailed comparison between other methods and our method is provided in Appendix A.
From 256 x 256 to 512 x 512, FLOPs increase by only 4.4 times, whereas the speed declines by a factor of 6.3, demonstrating that the primary speed bottleneck in DiT architecture is the number of tokens. Our method effectively reduces token count, resulting in significant gains, particularly for high-resolution content generation. For example, we achieve 55% reduction in FLOPPs at 512 x 512 while enhancing speed by 175%. Compared to other methods, our real-world speed improvements far exceed those achieved with similar reductions in FLOPs.
7

---

Table 3: Text-to-image generation SAM dataset. E [0.61, 0.86).
Model
FLOPs (G)
Throughput(img/s)
FID 1
PixArt-o
148.73
0,414 0.701 (+69%)
4.53
Ours
91.62 (-38%)
4.29
Visualization results. To further validate the effectiveness of our method, we visualize some samples generated from SparseDiT-XL at 256 256 resolution with a pruning rate r E [0.61, 0.86] (.43% FLOPs) and compare them with DiT-XL in Figure Each cell in the same position corresponds to images generated from the same random seed and class. Since our model is fine-tuned from the pre- vre-trained DiT, the overall styles and structures of the two subfigures are similar. The images generated by SparseDiT still retain rich high-frequency details. Furthermore, SparseDiT images demonstrate more precise structure, as evidenced by the accurate depiction of the "golden retriever's" nose and the "macaw's" eyes, which appear misplacement or missing in the images generated by DiT-XL. We provide samples at a resolution of 512 x 512 in Appendix.
4.2 Class-conditional video generation
Experimental settings. We conduct experiments at resolution of 256 x 256 on four public datasets: FaceForensics [45]. Timelapee [60]. UCF101 (54), and Taichi-HD [50]. Latte comprises two distinet types of transformer blocks: spatial transformers, which foeus capturing spatial information, and temporal transformers, which capture temporal information. To accommodate this separation of spatial and temporal feature extraction, we adjust our model's schedule within the sparse-dense token module. Specifically, we employ two transformers to prune spatial tokens and temporal tokens separately. The sparse tokens are then processed by two sparse transformers, and finally two transformers are used to recover the temporal and spatial tokens. Dense transformers are discarded in this setup. All other model configurations original training settings and hyperparameters used for Latte. To evaluate performance, we sample 2,048 videos, each consisting of 16 frames. and measure the Fréchet Video Distance (FVD) (57). Throughput is measured using a batch size of 2 clips on an Nvidia A100 GPU.
the same as in DIT, and we follow the
Main results. Table presents the main results of our approach. Leveraging the additional temporal dimension in video data allows for higher pruning rate. Our method achieves 56% reduction in FLOPs while maintaining a competitive FVD score compared to the baseline, demonstrating its effectiveness in video generation tasks.
4.3 Text-to-image generation
Experiment setting. We further assess the effectiveness of our method on text-to-image generation, which presents greater challenge compared to class-conditional image generation. We adopt PixArt-a g a text-to-image generation model built based on DiT as the base model. Our model initialized using the official Aix-rt-a checkpoint pre-trained on SAM dataset [22) containing 10M images. We further fine-tune it with our method on a subset SAM dataset including 1M images. In PixArt-o: the transformer architecture includes two types of attention layers: self-attention layer and a cross-attention layer, which integrates textual information. We apply our method to the self-attention layers to either reduce or recover tokens.
For evaluation, we randomly select text prompts from SAM dataset and adopt 100-step IDDPM solver (39 to sample 30,000 images, with the FID score serving as the evaluation metric. Throughput evaluated with batch size of 128 on an Nvidia A100 GPU.
Main results. As shown Table 3] our SparseDiT achieves an FID score comparable to the original PixArT-a with significantly accelerating the generation, showing that our method is effective for text-to-image generation task.
Visualization results. We visualize some samples of our method and compare them with original PixArt-a in Figure [ The classifier- -freffeerrfeeeeeeeeeee guidance is set to 4.5. The results demonstrate that our method effectively maintains both image quality and semantic fidelity.
4.4 Ablation study
All the following ablation experiments of SparseDiT-XL are conducted on ImageNet at 256 256 resolution.

---

(a) Images generated by PixArt-n
(h) Images generated by SparseDiT
Figure Generating images from PixArt-a (a) and SparseDiT (b). The captions in the same positions are identical. They are provided in Appendix A.3.
Table 4: Performance evaluation on different numbers of SDTMs. "NAN" indicates loss nan.
Table 5: Combination our method with efficient
samplers. The numbers represent FID scores.
No. SDTMs
1
3
4
Model
250-DDPM
25-DDIM
5-RFlow
FID FLOPs
NAN
3.86
2.51
2.38
DiT-XL
2.27
2.89
43.40
68.74
67.00
69.97
68.05
Ours
2.38
3.31
43.69
The number SDTMs. The alternation between sparse and dense tokens is a key factor con- tributing to the success of our method. Sparse tokens capture global structures, while dense tokens capture detailed information. In SparseDiT, we adopt four SDTMs by default. Table shows the effect of using different numbers of SDTMs. To isolate the impact of SDTM count, we maintain the same FLOPs across models by adjusting the number sparse and dense transformers while varying only the number of SDTMs. In configurations with only one SDTM, the model is modified into U-shaped architecture. The results indicate that fewer SDTMs reduce interactions between global and local information, resulting in weaker performance. Furthermore, reducing SparseDiT to a U-Net structure compromises training stability, leading to a collapse. We do not experiment with more than four SDTMs, increasing SDTM count beyond four offers diminishing returns in FLOP reduction.
Combination with efficient samplers. Our SparseDiT is model compression method that can be seamlessly integrated with efficient samplers, such as DDIM [5]] and Rectified Flow (RFlow) [28] Note that the RFlow variant of DiT was trained by us 2,000K iterations. As shown in Table when our method is combined with the 25-step DDIM and 5-step Rectified Flow samplers, it achieves approximately 18 and 93.4x improvements in inference speed, respectively. compared to the standard 250-step DDPM. Notably, our method does not introduce a significant performance gap relative to the baseline, demonstrating that can be effectively combined with efficient samplers.
Table 6 we examine the impact varying the number of poolingformers in the model. We keep the total number of transformers in the bottom and top segments constant across configurations. When using three poolingformers, performance declines substantially due to the global average pooling operator's 's limited ability adaptively capture information. For configurations with one or two poolingformet the models achieve same results. The poolingformer more efficient and consume fewer parameters than the standard transformer; therefore, we default to using poolingformers. Reducing the poolingformer count to zero results in training instability, suggesting that maintaining full-size tokens in the initial layers is crucial. This finding aligns with the observations in Figure [b] in DiT, the first two attention maps exhibit uniform distribution, allowing us replace them with poolingformers, but further replacement disrupts information flow and reduces performance.
The number of poolingformers.
The effectiveness of timestep- wise pruning rate. To evaluate the effectiveness of the timestep-wise pruning rate strategy, we conduct ablation experiments, as shown Table 7 In these experiments, we maintain the same number of FLOPs while varying the number of tokens along timesteps, either using a constant token count or dynamically adjusting the token count. The results demonstrate that dynamically tuning the pruning rate significantly improves performance. Additionally, Table 7 highlights an observation: the performance of the dynamic pruning strategy is particularly relative
Table 6: Performance evaluation on different numbers of poolingformers. "NAN" indicates loss nan during training.
Table 7: Comparison of FID scores with and without timestep-wise pruning rate strategy.
No. of Poolingformers No. of trans in top seg.
0
FLOPS (G)
No. of Tokens
Dymaic
FID 1
4
3
8 x
2.48
FID FLOPs
NAN
2.38
2.38
2.56
69.56
68. 80
68.05
67.29
4 4 1 x
6
V
2.50
2.63

---

with the token count of the later stages of denoising, For instance, the configuration with token counts ranging from 4 to 8 x only shows a 0.02 FID score increase compared to the configuration with constant 8 x 8 token count.
5 Conclusion
The primary limitation of our method arises from the manually pre-defined its struc- ture, containing the number of layers in each module and the number of sparse tokens.
Limitations.
In this paper, we introduce SparseDiT, a novel approach to enhancing the efficiency of DiT by leveraging token sparsity. By incorporating SDTM and timestep-wise pruning rate in strategically layered manner, SparseDiT significantly reduces computational complexity while preserving high- generation quality. Experimental results show substantial FLOP reductions with minimal performance degradation, demonstrating SparseDiT's effectiveness across multiple DiT- based models and datasets. This work marks a step forward in developing scalable and efficient diffusion models for high- resolution content generation, expanding the practical applications of DiT-based architectures.
References
[1] F. Bao, C. Li, J. Zhu, and B. Zhang. Analytic-DPM: analytic estimate of the optimal reverse variance in diffusion probabilistic models. International Conference on Learning Representations, 2022.
[2] F. Bao, S. Nie, Xue, Y. Cao, C. Li, Su, and 7hu. are worth words: A backbone for diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 22669-22679, 2023.
[3] A. Blattmann, T. Dockhorn S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, al. Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv preprint ar 1522217 2023.
[4] D. Bolya, C.-Y. Fu, X. Dai, P. Zhang, C. Feichtenhafer, and J. Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023.
[5] D. Bolya and J. Hoffman. Token merging for fast stable diffusion. CVPR Workshop on Efficient Deep Learning for Computer Vision, 2023.
[6] T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr,, J. Taylor, T. Luhman, E, Luhman, et al. Video generation models world simulators, 2024.
S. Chang, P. Wang, M. Lin, F. Wang, D. J. Zhang, R. Jin, and M. Shou. Making vision transformers efficient from a token sparsification view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6195-6205, 2023.
[8] J. Chen, J. Yu, C. Ge, L. Yao, E. Xie, Y. Wu, Z. Wang, J. Kwok, P. Luo, H. Lu, and 7. L... Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023.
[9] T. Chen, Y. Cheng, Z. Gan, L. Yuan, L. Zhang, and Z. Wang. Chasing sparsity in vision transformers; An end-to-end exploration. Advances in Neural Information Processing Systems, 34:19974- 19988, 2021.
[10] X. Chen, N. Liu, Y. Zhu, F. Feng. and Tang. Edt: efficient diffusion transformer framework inspired by human-like sketching. arXiv preprint arXiv:2410.23788 2024.
[11] Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248- 448-255. Icce, 2009.
[12] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
[13] P. Esser, Kulal, Blattmann, Entezari, Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
[14] G. Fang, X. Ma. and X. Wang. Structural pruning for diffusion models. In Advances in Neural Information Processing Systems, 2023.
10

---

[15] M. Fayyaz, S. A. Koohpayegani, F. R. Jafari, S. Sengupta, H. R. Joze, E. Sommerlade, H. Pirsiavash, and J. Gall. Adaptive token sampling for efficient vision transformers. In European Conference Computer Vision, pages 396-414, Springer, 2022.
[16] S. Gao, P. Zhou, M.-M. Cheng, and S. Yan. Mdtv2: Masked diffusion transformer is strong image synthesizer. arXiv preprint arXiv:2303. 14389, 2023.
[17] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[18] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[19] J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12590 2022.
[20] Y. Huang, Huang, Y. Liu, M. Yan, J. Lv, Liu, Xiong, H. Zhang, Chen, and Cao. Diffusion model-based image editing: A survey. arXiv preprint arXiv:2402. 17525, 2024.
[21] D. Kim, C.-H. Lai, W.-H. Liao, N. Murata, Takida, T. Uesaka, Y. He, Y. Mitsufuji, and S. Ermon. Consistency trajectory models: Learning probability flow ODE trajectory of diffusion. In The Twelfth International Conference on Learning Representations, 2024.
[22] A. Kirillov, Mintun, N. Ravi, H. Mao, Rolland, L. Gustafson, Xiao, S. Whitehead, A. C. Berg. W.-Y. Lo, et al. Segment anything. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[23] T. Kynkäänniemi, Karras, S. Laine, J. Lehtinen, and Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
[24] L. Li, H. Li, Zheng, Wu, Xiao, R. Wang, Zheng, X. Pan, Chao, and Ji. Autodiffusion: Training-fres optimization time steps and architectures for automated diffusion model accelcration. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7105-7114, 14, 2023.
[25] X. Li, Y. Liu, L. Lian, H. Yang. Z. Dong, D. Kang, S. Zhang, and Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17535- 1155511755,220 2023.
[26] Y. Lipman, R. Chen, H. Ben-Hamm, M. Nickel, and Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
[27] E. X. Ning, Lin, H. Yang, and Y. Wang. OMS-DPM, Optimizing the model schedule for diffusion probabilistic models. In Proceedings of the 40th International Conference on Machine Learning, pages 21915-21936. PMLR, 2023.
[28] X. Liu, C. Gong, and Q. Liu. Flow straight fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03000 2022.
[29] Lou, Luo, Y. Liu, Li, X. Ding, W. Hu, Cao, Y. Li, and C. Ma. Token caching for diffusion transformer acceleration. arXiv preprint arXiv:2409.1833 18523, 2024.
[30] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[31] C. Lu, Y. Zhou, F. Bao, Chen, C. Li, and J. Zhu. Dpm-solver++: Fast solver for guided sampling of diffussinn probabilistic models. arXiv preprint 1xX0v22211.01095, 2022.
[32] S. Luo, Y. Tan, L. Huang. J. Li, and H. Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023.
[33] X. Ma. G. Fang, and X. Wang. Deepcache: Accelerating diffuusin models for free. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 15762-15772, 2024.
[34] X. Ma, Y. Wang, G. Jia, X. Chen, 7. Liu, Y.-. Li, C. Chen, and Y. Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401 2401.03048, 2024.
[35] C. Meng, R. Rombach, R. Gao, D. Kingma, S. Ermon, J. Ho, and T. Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297 14306, 2023.

---

[36] L. Meng, H. Li, B.-C. Chen, S. Lan, Z. Wu, Y.-G. Jiang, and S.-N. Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 23309-12318, 2022.
[37] T. Moon, M. Choi, E. Yun, J. Yoon, G. Lee, and J. Lee. Early exiting for accelerated inference in diffusion models. In ICMI. 2023 Workshop Structured Probabilistic Inference {(&} Generative Modeling, 2023.
[38] C. Nash, J. Menick, S. Dieleman, and P. W. Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2103.cm (00000 2021.
[39] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International conference machine learning, pages 8162 8171. PMLR, 2021.
[40] 7. Pan, B. Zhuang, D.-A. Huang, W. Nie, Z. Yu, C. Xiao, J. Cai, and A. Anandkumar. T-stitch; Accelcrating sampling in pre-trained diffusion models with trajectory stitching. arXiv preprint 14111, 2024.
[41] W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195 4205, 2023.
[42] Y. Pu, Z. Xia, J. Guo, D. Han, Q. Li, D. Li, Y. Yuan, J. Li, Y. Han, S. Song, et al. Efficient diffusion transformer with step-wise dynamic attention mediators. arXiv preprint S.Ki::2408.05/111 2024.
[43] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 10684-10695, 2022.
[44] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-asiisted Intervention-MICCA.. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[45] A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner. Faceforensics: A large-scale video dataset for forgery detection human faces. arXiv preprint arXiv:180.0091 2018.
[46] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. Advancex in neural information processing systems, 29, 2016.
[47] T. Salimans and J. Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512. 2022.
[48] Y. Shang. Z. Yuan, B. Xie, B. Wu, and Y. Yan. Post-training quantization on diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1972-1981, 2023.
[49] A. Shih, S. Belkhale, S. Ermon, D. Sadigh, and N. Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[50] A. Siarohin, S. Lathuilière, S. Tulyakov, F. Ricci, and N. Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019.
[51] Song, C. Meng, and Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
[52] Y. Song and P. Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310.144 14189, 2023.
[53] Y. Song, Dhariwal, M. Chen, and I. Sutskever. Consistency models. arXiv preprint arX1v:2303.01469. 2023.
[54] K. Soomro, A. R. Zamir, and M. Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2(11):1-7. 2012.
[55] S. Tang, Y. Wang, C. Ding, Y. Liang, Y. Li, and D. Xu. Deediff: Dynamic uncertainty aware early exiting for accelerating diffusion model generation. arXiv preprint arXiv:2309. 17074, 2023.
[56] Z. Tang, J. Tang, H. Luo, Wang, and T.-H. Chang. Accelerating parallel sampling of diffusion models. In Forty-first International Conference on Machine Learning, 2024.
[57] T. Unterthiner, S. Van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint 1112:11812.01117,. 2018
12

---

[58] F.-Y. Wang. Z. Huang, A. W. Bergman. D. Shen, P. Gao, M. Lingelbach, K. Sun, W. Bian, G. Song, Y. Liu, et al. Phased consistency model. arXiv preprint arXiv:2405. (0t4:2405..8407,,, 2024.
[59] Y. Wang, R. Huang. S. Song. 7. Huang, and G. Huang. Not all images are worth 16x 16 words: Dynamic transformers for efficient image recognition. Advances in neural information processing systems, 34:11960-
11973, 2021.
[60] W. Xiong, W. Luo, L. Ma, W. Liu, and J. Luo. Learning generate time-lapse videos using multi-stage dynamic generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2364 2373, 2018.
[61] X. Yang. D. Zhou, Feng, and X. Wang- Diffusion probabilistic model made slim. In Proceedings of the IEEE/CVF Conference on computer vision and paltern recognition, pages 22552-22562, 2023.
[62] Z. Yang. J. Teng, W. Zheng, Ding, S. Huang. Xu, Y. Yang. Hong, X. Zhang, G. Feng, et Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint 2rXiv:2408.06007 06602. 2024.
[63] T. Yin. M. Gharbi, R. Zhang. E. Shechtman, F. Durand, W. T. Freeman, and T. Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6613 6623, 2024.
[64] H. You, C. Barnes, Y. Zhou, Y. Kang. Z. Du, W. Zhou. L. Zhang, Y. Nitzan, X. Liu, Z. Lin, E. Shechtman, S. Amirghodsi, and Y. C. Lin. Layer-and timestep-ade -adaptite differentiable token compression ratios for efficient diffusion transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[65] Yu. M. Luo, Zhou, Si, Zhou, Wang. J. Feng, and S. Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10819 19-9999 2022.
[66] Z. Yuan, H. Zhang. L. Pu, X. Ning. L. Zhang. T. Zhao, S. Yan, G. Dai, and Y. Wang. Ditfastattn: Attention compression for diffusion transformer models. Advances in Neural Information Processing Systems, 37:1 196-1219, 2025.
[67] W. Zhao, Y. Han, J. Tang, Z. Li, Y. Song, K. Wang, Z. Wang, and Y. You. A stitch in time saves nine: Small vim is a precise guidance for accelerating large vims. In Proceedings the Computer Vision and Pattern Recognition Conference, pages 19814 10884-19824, 2025.
[68] W. Zhao, Y. Han, J. Tang, K. Wang, Y. Song. G. Huang, F. Wang, and Y. You. Dynamic diffusion transformer. arXiv preprint arXh::2410.03456 2024.
[69] W. Zhao, J. Tang, Y. Han, Y. Song, K. Wang, G. Huang, Wang, and Y. You. Dynamic tuning towards parameter and inference efficiency for vit adaptation. Advances in Neural Information Processing Systems, 37:1 11177765-111766... 2024.
[70] H. Zheng, W. Nie, A. Vahdat, Arizzadenesheli, and A. Anandkumar. Fast sampling diffusion models via operator learning. International conference on machine learning, pages 42390-42402. PMLR, 2023.
[71] Y. 7hu, X. Liu, and Q. Liu. Slimflow: Training smaller one-step diffusion models with rectified flow. In Europea Conference on Computer Vision, pages 342-359. Springer, 2025.
[72] Y. Zhu, W. Zhao, A. Li, Y. Tang, J. Zhou, and Lu. Flowie: Efficient image enhancement via rectified flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13-22, 2024.
[73] Z. Zong, K. Li, G. Song, Wang, Y. Qiao, B. Leng, and Y. Liu. Self-slimmed vision transformer. In European Conference on Computer Vision, pages 432-448. Springer, 2022.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All the methods and experiments are served for claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
We discuss them in Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
All assumptions are provided.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the key information is provided to reproduce the results. The codes will be released soon. All the experiment settings are the same as the original models.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may he tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code will be released soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
ionpoedSSSmmissioooooocoyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https c/nips.cc/publi///guu
Codeeuumiiii wettsbbmmssiiioi ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
All the experiment settings are the same the original models.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: This paper does not include statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All key details are provided in the Experiment section and the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have checked it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential societal impacts in the text.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: We discuss safeguards in the Appendix.
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the original paper that produced the code package and datasets and their licenses.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pataasess.paperswtt paperswi thcode. cow/coo////ooooo /atasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: his paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.M policy http::///earrppscccc///ffferrrrp for what should or should not be described.
20

---

A Appendix
A.1 Comparison with other methods
Comparison with ToMeSD, Recent studies 37 68 have evaluated ToMeSD S on DiT models and reported significant performance drops, which much lower than those achieved by our method. Our approach fundamentally different from ToMeSD. While ToMeSD reduces and recovers tokens generically at every layer, our method specifically analyzes the local-global relationships unique to DiT models. Based on these insights, we strategically reduce and recover tokens. We have evaluated ToMeSD DiT and compared it with our method, shown in Table
Comparison with DyDIT. DyDIT (68) represents the current state-of-the-at approach for enhanc- ing the efficiency of Diffusion Transformers (DiTs). In contrast to our method, which emphasizes token reduction, DiDIT implements pruning across tokens, attention heads, and channels. A compar- ative analysis of our method against DyDiT, as detailed in Table Saolee/using ImageNet 512x512 images, illustrates a substantial advantage in computational efficiency for our approach. Specifically, our method achieves 46% reduction in FLOPs, markedly surpassing DyDiT's 29% reduction, coupled with 145% increase inference speed, significantly outperforming the 31% improvement achieved by DyDiT. The only trade-off is minor increase in the Fréchet Inception Distance (FID), where our method's FID is 0.08 higher than that of DyDiT. Nevertheless, this difference in FID is nearly imperceptible human observers in practical applications.
Comparison with TokenCache. We compare our method with TokenCache (29 in Table 9 We adopt the standard DiT baseline 2.27. while TokenCache applies baseline 2.24. TokenCache achieves a 32% speed gain with a slight FID increase. Our approach attains an 87% speed gain with a smaller FID increase, demonstrating superior efficiency and performance.
Comparison with Ditfastattn. We compare our method with Ditfastattn [66] in Table D We adopt the standard DiT baseline 3.04, while Ditfastattn applies a baseline 3.. Ditfastattn shows substantial FID increase for speed gain, while our method results in better speed and less FID.
A.2 Training form scratch
We train our SparseDiT from scratch, reaching 400K training iterations. We compare our result with original DiT paper in Table 4. The results (cfg=1.0) are in Table T0
A.3 Captions in Figure 5
Column 1: The image depicts beach scene with a large body of water, such as a lake or ocean, and sandy shoreline. The beach is filled with people, including group of people swimming the water.
The image depicts a beautiful outdoor dining area with a large number tables and chairs arranged in row, overlooking a picturesque lake. The tables covered with white tablecloths, and there are several umbrellas providing shade for the guests. The scene is set in a lush green field, with large building in the background, possibly hotel a restaurant. The tables are arranged in way that allows for an unobstructed view of the lake, creating a serene and relaxing atmosphere for the diners. The image has stylish and elegant feel, with the attention detail in the table arrangement and the choice of location contributing to a memorable dining experience.
Column 2:
Column 3: The image depicts busy city street at night, with group of people standing outside restaurant and a bar. The scene is set in European city, and the atmosphere is lively and bustling.
Column 4: The image features a vintage-style train, parked on track surrounded by trees and grass. The train appears be an old-fashioned steam engine, which is a type locomotive powered by steam. The train is positioned in a park-like setting, with a tree-lined path nearby. The scene is set in a sunny day, creating a pleasant atmosphere. The image has a nostalgic and historical feel, evoking a sense of the past and the charm of old-time trains.
21

---

Table 8: Comparison of our method with ToMeSD on DiT models.
Model
Method
FID
Speed-up
ToMeSD
29.24
+20%
DIT-B
Ours
8.23
+68%
ToMeSD
14.74
+66%
DiT-XL
Ours
2.38
+87%
Table 9: Comparison of our method with DyDiT on DiT models.
Model
Resolution
Method
FLOPs
FID
IS
Speed-up
DyDiT Ditfastattn
(-29%)
2.88(-0.16)
+31% %
DIT-XI
512 x 512
4.52(+1.36 1,36)
180.34
+98%
Ours
286 (-46%)
2.96(-0.08)
242.4
+145%
TokenCache
72.25(-39%)
2.37 (+0.13)
262.00
+32%
DiT-XL
Ours
68.05(-43%)
2.38 (+0.11)
276.39
+87%
A.4
Additional visualization
We provide additional visualizations at resolution of 512 512 on SparseDiT-XL from Figure 6 Figure 17 The class labels, including "arctic wolf", "volcano", "cliff drop-off", "balloon", "sulphur- crested cockatoo", "lion". "otter", "coral reef" "macaw", "red panda". ""husky,, and "panda", correspond to the same cases presented in DiT. Readers compare results with those of DiT. Our samples demonstrate comparable image quality and fidelity. The classifier free guidance scale is set to 4.0, and all samples shown here uncurated.
22

---

Table 10: Comparison of our method with ToMeSD on DiT models.
Method
FID
Flops
Interations
DiT-XL
19.47
118.64 68.05
400K
Ours (from scratch)
15.11
400K
Figure 6: Uncurated 512 x 512 SparseDiT-XL samples. Classifier-free guidance scale = 4.0 Class label "aretic wolf" (270)
Figure 7: Uncurated 512 x 512 SparseDiT-XL samples. Classifier-free guidance scale = 4.0 Class label "volcano" (980)
23

---

Figure 8: Uncurated 512 x 512 SparseDDT--LL samples. Classifier-free guidance scale = 4.0 Class label = "husky" (250)
Figure 9: Uncurated 512 x 512 SparseDiT-XL samples. Classifier free guidance scale = 4.0 Class label "sulphur-crested cockatoo" (89)
24

---

Figure 10: Uncurated 512 512 SparseDiT-XX samples. Classifier-free guidance scale = 4.0
Class label = "cliff drop-off" (972)
Figure 11: Uncurated 512 x 512 SparseDiT-XL samples. Classifier free guidance scale = 4.0 Class label "balloon" (417)
25

---

Figure 12: Uncurated 512 512 SparseDiT-XLL samples. Classifier-free guidance scale = 4.0 Class label = "lion" (291)
C.L.
Figure 13: Uncurated 512 512 SparseDiT-XL samples. Classifier free guidance scale = 4.0 Class label "otter" (360)
26

---

Figure 14: Uncurated 512 512 SparseDiT-XL samples. Classifier-free guidance scale = 4.0 Class label "red panda" (387)
Figure 15: Uncurated 512 512 SparseDiT-XL samples. Classifier freee guidance scale = 4.0 Class label "panda" (388)
27

---

Figure 16: Uncurated 512 512 SparseDiT-XL. samples. Classifier-fret guidance scale = 4.0 Class label "coral reef" (973)
Figure 17: Uncurated 512 512 SparseDiT-XL samples. Classifier free guidance scale = 4.0 Class label "macaw" (88)
28


Reviews:
[{"review_id": "aIsy4i8BM8", "reviewer": "Reviewer_gHMa", "summary": "The paper introduces SparseDiT, a novel framework for enhancing the computational efficiency of Diffusion Transformers (DiT) through token sparsification across spatial and temporal dimensions. The authors employ a tri-segment architecture (Poolingformer, Sparse-Dense Token Modules, and dense transformers) to dynamically adjust token density, achieving significant reductions in FLOPs (up to 55%) and improvements in inference speed (up to 175%) while maintaining generative quality, as demonstrated on tasks like image and video generation.", "questions": "Please see the weakness.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. Clear motivation. The token sparsification strategy is well-motivated by empirical observations of attention patterns in DiT layers.\n\n2. The paper demonstrates impressive performance gains across multiple benchmarks (e.g., ImageNet, video datasets, and text-to-image generation) with minimal degradation in output quality (e.g., FID).\n\n3. Overall, this paper is easy to follow and is also written well.\n\n\n## Weaknesses\n\n1. My main concern is the practical value of this technique. Specifically, today's text to image/video generation models are pretrained on large-scale datasets with many parameters. Thus, finetuning makes it rather challenging as it may change the underlying distribution of their initial generation space. For this reason, training-free methods seem to be more likely preferred by the community.\n\n2. Lack deeper analysis of sparse/dense tokens. For example, in L301-302, the paper claims that \"Sparse tokens capture global structures, while dense tokens capture detailed information.\", however, it would be more convincing if the authors can provide some visualizations or comparisons, e.g. more sparse tokens help to capture global or more dense tokens help to generate detailed information.", "was_revised": true, "final_justification": "This work proposed to speedup DiT with further finetuning. The results are quite positive. However, there are still concerns on hurting initial pretraining distribution, which could hinder the performance of the pretrained model.", "rebuttal": "# To Reviewer gHMa\nThank you for recognizing the strengths in our paper, including the well-motivated token sparsification strategy and our impressive performance across multiple benchmarks. We're glad the paper was clear and well-written. We are eager to address your queries and improve our work based on your valuable feedback.\n## **Q1: The practical value of this technique.**\nTraining-free methods indeed garner significant interest in the community. These methods often target redundant sampling steps in diffusion models. For instance, advanced sampling strategies like DDIM and DPM can achieve similar results in fewer steps compared to DDPM, while DeepCache shares features between adjacent steps. Our approach, however, emphasizes a different dimension—simplifying the model structure, which typically necessitates additional fine-tuning. Specifically, we prune the number of tokens in a single sampling step, and thereby cannot circumvent finetuning. As diffusion models evolve, the number of sampling steps decreases, reducing the scope for optimization within sampling steps. We think that the significance of efficient model structures will increase.\n\nFine-tuning is widely recognized as an essential method for achieving good performance-efficiency diffusion models. Many common methods often require additional fine-tuning. For example, standard RF [1], consistency models [2], and other sampling step distillation [3] techniques all necessitate this process. Leading open-source T2I model Flux [4], and T2V model HuanyuanVideo [5] both employ additional distillation for efficiency gains. Consequently, we argue that minimal fine-tuning for enhancing efficiency is justifiable in practical applications.\n \nMoreover, the overhead associated with fine-tuning in our approach is reasonable. We require only 6% of the training iterations. For instance, DiT-XL/2 necessitates just 400K iterations for fine-tuning (about 1.5 dayes on 8 A100 GPUs), compared to 7,000K iterations for full training, achieving comparable efficacy to recent works [6, 7, 8].\n\n\n## **Q2: Lack deeper analysis of sparse/dense tokens.**\nThank you for pointing out this crucial aspect, which warrants further exploration.\n\nThe differentiation between global and local tokens arises from observations documented in Figure 1, where some tokens correspond to larger areas of images, while others focus on smaller regions. Our method harnesses and amplifies this phenomenon by directing certain layers to generate global tokens, while others preserve their original representation. To substantiate the effectiveness of our approach, we extract the attention map with dimensions N$\\times$ N, where N is the number of tokens of an H$\\times$W image. By reshaping the attention map to $N\\times H \\times W$, each $H\\times W$ heatmap is the receptive field of the corresponding token. It is evident that global token heatmaps emphasize larger regions, presenting more precise and meaningful semantic information, whereas local token heatmaps concentrate on smaller areas. I apologize for OpenReview's current constraints with image uploads, preventing us from displaying these visualizations. Instead, I provide quantitative results, calculating normalized variances of heatmaps using the method from Figure 1b, detailed in the table below:\n\n| TimeStep | Global Token | Local Token |\n| :---: | :---: | :---: |\n| 10 | 0.29 | 1.00 |\n| 500 | 0.18 | 0.82 |\n| 900 | 0.00 | 0.68 |\n\nNote: Values are normalized to a 0-1 scale, aligned with Figure 1b. Higher variance indicates a local property, while a lower variance suggests a global property. This table demonstrates that global and local tokens effectively capture global and local information, respectively. We intend to include visualization results in the next version of our paper.\n\n\n### **Reference**\n[1] Liu, X., Gong, C., & Liu, Q. (2022). Flow straight and fast: Learning to generate and transfer data with rectified flow.\n\n[2] Song, Y., Dhariwal, P., Chen, M., & Sutskever, I. (2023). Consistency models.\n\n[3] Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., & Park, T. (2024). One-step diffusion with distribution matching distillation. computer vision and pattern recognition (pp. 6613-6623).\n\n[4] FLUX.1: Redefining Text-to-Image AI with Superior Visual Fidelity.\n\n[5] Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., ... & Zhong, C. (2024). Hunyuanvideo: A systematic framework for large video generative models.\n\n[6] You, H., Barnes, C., Zhou, Y., Kang, Y., Du, Z., Zhou, W., ... & Lin, Y. C. (2025). Layer-and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers. Computer Vision and Pattern Recognition Conference (pp. 18072-18082).\n\n[7] Zhao, W., Han, Y., Tang, J., Wang, K., Song, Y., Huang, G., ... & You, Y. (2024). Dynamic diffusion transformer. ICLR.\n\n[8] Yuan, Z., Zhang, H., Pu, L., Ning, X., Zhang, L., Zhao, T., ... & Wang, Y. (2024). Ditfastattn: Attention compression for diffusion transformer models. Advances in Neural Information Processing Systems, 37, 1196-1219."}, {"review_id": "I9jY0cbdjF", "reviewer": "Reviewer_kysk", "summary": "In this paper, the authors aim to improve the DiT method by pruning its architecture. They propose a new architecture called SparseDiT, which achieves performance comparable to the original DiT while using significantly fewer FLOPs after fine-tuning pretrained DiT models. The main approach is to: (1) apply average pooling to extract sparse tokens, (2) alternate self-attention on the sparse tokens with cross-attention between the sparse and dense tokens, and (3) apply a few dense attention layers at the end. In a fine-tuning setting, the authors show that their proposed method can achieve performance comparable to DiT while significantly reducing FLOPs.", "questions": "1. As raised in the weaknesses section, how does the model perform when trained from scratch? Is its effectiveness limited to a fine-tuning context, or can it achieve strong performance without relying on weights from a pre-trained dense model?\n\n2. Could the authors provide more insight into the selection of the model's hyperparameters?\n\n(a) Is there a principled method for determining the number of layers for each module (e.g., PoolingFormer, SDTM)? How sensitive is the model's performance to this specific configuration?\n\n(b) Similarly, what is the rationale behind the design of the pruning schedule across different timesteps (t)? How sensitive is the final performance to changes in this schedule?\n\n3. The model's non-uniform architecture raises questions about potential optimization challenges. Conventional transformers benefit from well-studied optimization strategies due to their uniform layer structure. Does the heterogeneity of SparseDiT introduce any new difficulties for training stability? For instance, when training from scratch, do the different modules require distinct learning rates or optimizers to converge effectively?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. In Figures 1 and 3, the authors show that the early-layer attention maps in DiT are highly uniform. This is an insightful observation that provides a clear motivation for using average pooling.\n\n2. The authors validate the proposed SparseDiT across various benchmarks, including image and video generation, demonstrating improved efficiency in all cases. The experiments are solid.\n\n3. The performance gains appear consistent across different model sizes, showing strong potential for scalability.\n\nWeaknesses\n\n1. My major concern is with the experimental setup. The authors only report performance after fine-tuning a pre-trained model and do not include results from training SparseDiT from scratch. The claims would be much stronger if the proposed architecture also performs comparably to the original DiT when pre-trained. Otherwise, the method should be compared to knowledge distillation techniques, which also create smaller, more efficient models from a pre-trained teacher.\n\n2. The architectural design is non-uniform and involves many hyperparameters, including:\n\n(1) The number of layers for the PoolingFormer, the sparse-dense modules, and the final dense block.\n\n(2) The specific pruning ratio schedule used across different timesteps (t).\n\nI am concerned that without a principled way to determine these hyperparameters, the method's broader applicability may be limited.", "was_revised": true, "final_justification": "I thank the authors for their rebuttal. The newly presented results on pretraining the proposed architecture are impressive and compelling. This evidence significantly strengthens the paper's contribution, demonstrating that the proposed architecture has the potential to be highly impactful for the design of future diffusion-based models.\n\nGiven these strong results, which solidify the paper's contribution to the field, I now recommend acceptance.", "rebuttal": "# To Reviewer kysk\nThank you for your valuable feedback and for highlighting the strengths of our work. We are delighted that you found our observations regarding early-layer attention maps in DiT insightful and supportive of the use of average pooling. We appreciate your recognition of the comprehensive validation of SparseDiT across various benchmarks and your acknowledgment of the experiments as solid. Furthermore, your observation of consistent performance gains across different model sizes indicates the scalability potential of our approach.\n\nWe are committed to addressing each of your inquiries and suggestions thoroughly in the upcoming revisions. Your feedback is instrumental in enhancing the quality of our research.\n## **Q1: Training from scratch.**\nThank you for your insightful observation. Our method indeed can be trained from scratch; however, leveraging a well-pretrained model offers significant advantages, aligning with our research goal of enhancing the efficiency of well-trained diffusion models by pruning tokens. Unlike specialized architectural design methods such as Pixart-$\\alpha$ [1], our aim is not to develop a new model architecture. We just argue that our findings can inspire future designs. We opted for fine-tuning rather than training from scratch for two primary reasons:\n\n1. Computational Cost: Training diffusion models from scratch, such as our baseline model, DiT-XL/2, requires 7,000K iterations, approximately two months on A100 GPUs. While our training speed surpasses the original DiT, it still demands about a month of processing, which is not feasible for us. Especially considering more complex T2I models, their computational cost will be more tremendous.\n\n2. Practical Application: From a practical standpoint, an algorithm that requires training from scratch holds less value compared to one that can be fine-tuned. Fine-tuning techniques are increasingly common and effective. Leading open-source foundation generation models such as WanX2.1, HunyuanVideo, and Flux are built on traditional DiT architectures and derive their efficient variants through fine-tuning. Comparable works [2, 3, 4] also employ fine-tuning or training-free strategies without training from scratch.\n\nI understand the importance of assessing our model's capability and the dependency of pre-trained weights. We have trained our model from scratch, although due to time constraints, only reaching 400K training iterations during the rebuttal period. A complete training cycle like DiT requires 7,000K steps, roughly equivalent to one month of computing. We compare our result with original DiT paper in Table 4. The results (cfg=1.0) are as follows:\n| Model | FID | FLOPs | Iterations |\n| :---: | :---: | :---: |  :---: | \n| DiT | 19.47 | 118.64 | 400K |\n| Ours (from scratch) | 15.11 | 68.05 | 400K |\n\nThese results indicate our method can be trained from scratch, achieving strong performance without dependence on pre-trained dense model weights. Nevertheless, we believe our method’s fine-tuning capability possesses more practical significance.\n\nRegarding comparisons with distillation techniques, diffusion models commonly use timestep distillation, with some methods claiming single-step inference [5, 6, 7]. Unlike these, our approach focuses on model pruning to reduce computational costs in one step. Our method and distillation techniques improve diffusion model efficiency along orthogonal dimensions and should be considered complementary rather than directly comparable. For a one-step distillation method, they can reduce the timestep from 100 to 1, which saves 99\\% FLOPs, whereas achieving a 99\\% reduction in FLOPs through model structure compression alone is nearly impossible. To our knowledge, related works also do not compare their results with distillation methods.\n\n## **Q2: The selection of the model's hyperparameters.**\n1. We have conducted ablation studies, as shown in Tables 4 and 6, to demonstrate the selection of several key hyperparameters. Regarding the number of PoolingFormer layers, Figure 1 indicates that the attention maps from the first two transformer layers display a uniform distribution. Consequently, one or two PoolingFormer layers suffice, as corroborated by Table 4. In the SDTM architecture, one transformer is designated for generating sparse tokens, another for recovering dense tokens, and one is dense transformer. These three transformers remain fixed. We provide an additional ablation study in the below table to verify it.\nThe number of sparse transformers can be adjusted, with more sparse transformers yielding more efficient models. Performance remains stable as long as the number of sparse tokens does not exceed three. The final dense block should contain no more than three transformers to ensure efficiency, with additional transformers having no significant impact on performance.\n| No. of sparse transformer | FID | FLOPs |\n| :---: | :---: | :---: | \n| 3 | 2.38 | 68.05G | \n| 4 | 2.76 | 69.41G | \n| 7 | 3.89 | 65.67G | \n2. The rationale for the pruning schedule across different timesteps is detailed in lines 56-58. In Figure 1 b, we find that attention variance increases as the denoising process advances, indicating a growing emphasis on local information at lower-noise stages. As denoising progresses, the model increasingly prioritizes local details, dynamically adapting to heightened demands for detail at later stages. Therefore, we adopt larger pruning rate at early stages while smaller pruning rate at later stages. We discuss how to set the pruning rates in Table 7. Our method is not sensitive for the pruning rates beside the pruning rate of the last stage. As said in line 333-336, the performance of the dynamic pruning strategy is particularly relative with the token count of the later stages of denoising. For instance, the configuration with token counts ranging from $4\\times 4$ to $8\\times 8$ only shows a 0.02 FID score increase compared to the configuration with a constant $8\\times 8$ token count. In the practical application, we recommend a $10\\times 10$ token count in the final stage for optimal performance, and a $4\\times 4$ token count in the first stage for efficiency. The intermediate stages can be set to $6\\times 6$ and $8\\times 8$, with the method displaying robustness to these settings.\n\nI will reorganize the content above and include it in our supplementary material to clearly assist readers in selecting the hyperparameters.\n\n## **Q3: Training stability.**\nDespite our model's non-uniform architecture, it demonstrates remarkable training stability. PoolingFormer structures are inherently stable, as evidenced by [8]. Our SDTM architecture alternates between sparse and dense structures, with dense transformers ensuring stability. The final top block comprises standard transformers, which naturally exhibit stability. Throughout extensive experiments, we have encountered minimal issues concerning training stability. The sole instance of instability arises when no PoolingFormer layers are used, causing a sudden shift to sparse tokens in the initial layer. In training DiT-XL/2 from scratch, we employed a singular global learning rate, achieving rapid convergence. The training loss decreased from 0.0308 at 100 iterations to 0.0120 at 10,000 iterations, affirming the model's stability.\n### **Reference**\n[1] Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., ... & Li, Z. (2023). Pixart-$\\alpha $: Fast training of diffusion transformer for photorealistic text-to-image synthesis.\n\n[2] You, H., Barnes, C., Zhou, Y., Kang, Y., Du, Z., Zhou, W., ... & Lin, Y. C. (2025). Layer-and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers.\n\n[3] Zhao, W., Han, Y., Tang, J., Wang, K., Song, Y., Huang, G., ... & You, Y. (2024). Dynamic diffusion transformer.\n\n[4] Bolya, D., & Hoffman, J. (2023). Token merging for fast stable diffusion. \n\n[5] Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., & Park, T. (2024). One-step diffusion with distribution matching distillation.\n\n[6] Song, Y., Dhariwal, P., Chen, M., & Sutskever, I. (2023). Consistency models.\n\n[7] Xie, S., Xiao, Z., Kingma, D., Hou, T., Wu, Y. N., Murphy, K. P., ... & Gao, R. (2024). Em distillation for one-step diffusion models.\n\n[8] W. Yu, M. Luo, P. Zhou, C. Si, Y. Zhou, X. Wang, J. Feng, and S. Yan. Metaformer is actually what you need for vision."}, {"review_id": "mA4wEf1GHQ", "reviewer": "Reviewer_pfYM", "summary": "The paper introduces SparseDiT, a novel framework designed to tackle the significant computational inefficiency of Diffusion Transformers. The core contribution lies in a unified spatio-temporal token sparsification strategy, motivated by the different attentions shown in different layers and denoising steps. Spatially, it utilizes a tri-segment architecture: a lightweight Poolingformer captures global features in early layers, an innovative Sparse-Dense Token Module (SDTM) alternates between sparse and dense tokens to efficiently process mid-level features, and standard Transformers refine details in final layers. Temporally, it employs a dynamic pruning strategy, progressively increasing the number of active tokens throughout the denoising process to match the evolving demand for detail, thereby optimizing computation across both space and time.", "questions": "1.\tTemporal sparsification design: Could you provide more justification—either theoretical or empirical—for the choice of T/4 as the divide point of the denoising stage? Why was the specific linear interpolation chosen, and have you considered or tested alternative scheduling strategies?\n2.\tLimited baseline comparisons: The main paper includes only ablation-type comparisons (with/without SparseDiT), which makes it hard to gauge the method’s performance relative to prior work. Could you include more detailed quantitative and qualitative comparisons with relevant baselines such as ToMeSD, DyDiT, and other token sparsification or merging techniques on more complex  datasets?\n3.\tRelated work inclusion: Several optimization strategies mentioned in the Related Work section (e.g., adaptive token routing, dynamic masking, or hybrid dense-sparse attention methods) are not covered in the experiments. What is the rationale for excluding these, and how do you expect SparseDiT would compare to them?", "limitations": "A key limitation of this work also stated in limitation part is its heavily manual design choices across both spatial and temporal sparsification modules. Many core architectural decisions—such as the pruning schedule over diffusion steps, the sparse-dense transition structure,—are introduced without theoretical grounding or extensive empirical search, making them appear heuristic rather than principled. This raises concerns about the generality and robustness of the design. Another notable limitation is the insufficient engagement with prior work. While some related methods are briefly mentioned in the Related Work section, the paper does not clearly acknowledge how existing techniques (e.g., token merging from ToMeSD) have inspired or relate to the proposed method. A deeper and more explicit comparison, both conceptual and empirical, would significantly strengthen the work’s positioning and clarify its contribution boundaries.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper derives its core contribution from a careful empirical and conceptual analysis of attention distributions across Transformer layers in DiT, revealing meaningful structural patterns that inform the proposed sparsification mechanism. This insight leads to a well-designed spatial token compression strategy, which is both intuitive and effective, and clearly contributes to improving computational efficiency. The implementation is elegant and shows strong alignment between observation and design. However, the proposed temporal sparsification—adjusting the pruning rate based on diffusion timesteps—feels more arbitrary and lacks sufficient theoretical or empirical grounding. It appears less principled compared to the layer-wise design. A more critical issue lies in the experimental evaluation: the main text only includes with/without SparseDiT comparisons, while the appendix contains limited and coarse comparisons with ToMeSD and DyDiT. Many related methods mentioned in the Related Work section—such as other token pruning, merging, or flow strategies—are not included in the empirical evaluation at all. This makes it difficult to assess the actual advantage and novelty of SparseDiT in the broader context of efficient models.", "was_revised": true, "final_justification": "The rebuttal addresses several of my concerns, particularly by providing more comprehensive baseline comparisons and additional supporting results. However, the core design choices, especially the temporal sparsification schedule, remain primarily empirically driven with limited theoretical grounding. While the paper is technically sound and demonstrates clear practical value, the lack of stronger theoretical support for these mechanisms prevents me from giving a higher score. Given its solid engineering contributions and the potential to inspire follow-up work, I find a borderline accept to be the most appropriate rating. I appreciate the authors’ efforts in the rebuttal and look forward to seeing further refinement and clearer justification of these design choices in future work.", "rebuttal": "# To Reviewer pfYM\nThank you for your thoughtful and encouraging feedback. We are delighted that you found value in our empirical and conceptual analysis.\n## **Q1: Temporal sparsification design.**\nThank you for this insightful question. We will address it from three perspectives:\n1. **Theoretical support to timestep-wise pruning rate.** The rationale behind the pruning schedule across different timesteps is detailed in lines 56-58 of our paper. Figure 1b shows that attention variance increases as the denoising process progresses, highlighting a growing emphasis on local information at lower-noise stages. As the denoising proceeds, the model increasingly focuses on local details, dynamically adapting to heightened demand for detail in the later stages. Consequently, we employ larger pruning rates ($r$) in the early stages and smaller rates in the later stages.\n\n2. **The choice of T/4.** This decision is primarily informed by our experimental findings. We discuss pruning rates in Table 7, and lines 333-334 highlight that the performance of the dynamic pruning strategy is particularly impacted by the token count in the later stages of denoising. Our experiments reveal that performance correlates most strongly with the last T/4 timesteps. Choosing T/3 would entail higher computational costs for tiny gains, and choosing 5/T would result in a noticeable performance decline. T/4 provides the optimal performance-efficiency trade-off.\n\n3. **Linear interpolation chosen.** Our ablation study, referenced in lines 328-336, provides insights into this decision. Experiments indicate that the performance of the dynamic pruning strategy is particularly sensitive to the token count in the later stages of denoising. Our method is less sensitive to $r$ during early stages. Thus, more complex interpolation and scheduling strategies did not result in significant improvements. We opted for simple linear interpolation to maintain a gradual increase in $r$.\n\nWe acknowledge that our paper may not fully convey the rationale behind various choices for our timestep-wise pruning rate strategy, and we will enrich the relevant sections in future revisions.\n\n## **Q2: Limited baseline comparisons.**\nThank you for pointing out the need for more comprehensive baseline comparisons. We have focused on comparing our method with ToMeSD and DyDiT because they provide a standardized baseline using DiT. Additionally, DyDiT is recognized as a state-of-the-art technique in this domain. Other methods utilize different base models, and even when employing DiT, they may not adhere to the standard and optimal results from the DiT paper (e.g., FID score 2.27 on $256\\times 256$ ImageNet and FID score 3.04 on $512\\times 512$ ImageNet), which complicates achieving a perfectly fair comparison. Below, I outline comparisons with nearly all relevant baselines:\n\n1. **Compare with TokenCache [1].**\n| Method |Base Model|FID Score|Speed Up| \n|:---:|:---:|:---:|:---:| \n| TokenCache | DiT-XL/2 | 2.37 (+0.13) | x 1.32 | \n|  Ours | DiT-XL/2 | 2.38 **(+0.11)** | **x 1.87** | \n\nWe adopt the standard DiT baseline 2.27, while TokenCache applies a baseline 2.24. TokenCache achieves a 32% speed gain with a slight FID increase. Our approach attains an 87% speed gain with a smaller FID increase, demonstrating superior efficiency and performance.\n\n2. **Compare with Ditfastattn [2].**\n| Method | Base Model | FID Score | IS | Speed Up | \n| :---:| :---: | :---:|:---:| :---:| \n| Ditfastattn | DiT-XL/2 (512x512) | 4.52 (+1.36) |  180.34  | x 1.98 | \n|  Ours | DiT-XL/2 (512\\x512) | 2.96 **(-0.08)** |   **242.4**  | **x 2.45** | \n\nWe adopt the standard DiT baseline 3.04, while TokenCache applies a baseline 3.16. Ditfastattn shows a substantial FID increase for speed gain, while our method results in better speed and less FID.\n\n3. **Compare with DiffCR [3].**\n| Method | Base Model | FID Score | Speed Up | \n|:---:| :---:|:---:|:---:| \n| DiffCR | PixArt-sigma | 10.68 (**-1.27**) | x 1.26 | \n|  Ours | PixArt-alpha | 4.29 (-0.24) | **x 1.69** | \n\nWhile DiffCR achieves a greater performance improvement (1.27 vs. 0.24), our results are based on a more challenging baseline of 4.53 compared to their 11.95. The marginal effect implies that enhancing performance from 4.53 is more challenging than from 11.95. Additionally, our method provides a significantly higher speed improvement than theirs.\n\n4. **Compare with ToMeSD and DyDiT.**\n\n**(1) Quantitative comparisons**\n\nWe include quantitative comparisons on class-conditional image generation on ImageNet in Appendix. ToMeSD, being a workshop paper, does not demonstrate high effectiveness, whereas DyDiT delivers substantial results. Therefore, I mainly compare our results with DyDiT. For the text-to-image task, we both employ PixArt-$alpha$ as our base model. DyDiT uses a smaller dataset, COCO, while we utilize a more complex dataset, SAM. The results are:\n\n| Method | Base Model | FID Score ($\\Delta$) | Speed Up | \n|:---:|:---:|:---:|:---:| \n| DyDiT | PixArt-alpha | 19.75 (-0.13) | x 1.32 | \n|  Ours | PixArt-alpha | 4.29 (**-0.24**) | **x 1.69** | \n\nOur SparseDiT demonstrates better performance improvement and speed up in the text-to-image task.\n\n**(1) Quantitative comparisons**\n\nDue to OpenReview's limitations on image uploads, we assess visual quality using ChatGPT's evaluation comparing our SparseDiT with ToMeSD and DyDiT:\n\n**ToMeSD (Model A) v.s.  SparseDiT (Model B)**\n\nChatGPT assessment:\n\nModel B is a far more powerful, capable, and useful generative model than Model A.\n\nProfile of Model B (Bottom Set):\n\nTechnically Superior: It consistently produces high-resolution, sharp, and richly detailed images.\n\nAesthetically Driven: It demonstrates a sophisticated understanding of composition, lighting, and color theory to create beautiful, engaging results.\n\nIdealized Generation: It excels at capturing the core essence of a concept (e.g., the eruption of a geyser, the majesty of a mountain) and presenting it in its most ideal and visually striking form.\n\nProfile of Model A (Top Set):\n\nResembles a \"Raw Database\": Its output mirrors the varied quality and style of a large, unfiltered image dataset scraped from the web, containing both good and bad examples.\n\nOccasional Serendipity: Due to its less guided nature, it can sometimes produce unique and narratively rich images (like the recumbent parrot) that a more polished, aesthetically-driven model might not.\nFor any practical application requiring high-quality, visually appealing, and reliable image generation, Model B is the clear and definitive winner.\n\n**DyDiT (Model A) v.s.  SparseDiT (Model B)**\n\nBoth our Appendix and DyDiT's provide numerous visualizations, with our images presented at a resolution of \n$512\\times 512$ in Figures 4-15 and DyDiT's at $256\\times 256$ in Figures 9-22. Despite the difference in resolution, the comparison remains valid as both utilize the same training dataset. While both sets of results exhibit high generation quality, SparseDiT images display a more precise structure in certain cases compared to DyDiT.\n\nChatGPT assessment:\n\nModel A (left) is a model that strives for ultimate aesthetics. It will create the most stunning, idealized, and dramatic images, sometimes at the cost of coherence. Its \"ceiling\" is very high, capable of producing breathtaking results. However, its \"floor\" is low; the artifact in the parrot image proves that it can fail in certain cases and cannot be fully trusted.\n\nModel B (right) is a model that prioritizes realism and robustness. Its output is more like a faithful reproduction of high-quality photographs from the real world. Its aesthetic \"ceiling\" might not be as stunning as Model A's best shots, but its \"floor\" is very high—it rarely, if ever, produces obvious errors, making its output highly reliable and consistent.\n\n## **Q3 Related work inclusion**\nSeveral optimization strategies mentioned in our related work. We have listed some comparisons in Q2. Some methods, like flow-based methods and timestep distillation methods focus on reducing or optimizing timesteps to enhance efficiency, whereas our method centers on pruning model structure and reducing FLOPs within a single timestep. Our method and their techniques improve diffusion model efficiency along orthogonal dimensions and should be considered complementary rather than directly comparable. In Table 5, we have shown that our method can be integrated with advanced sampling methods utilizing fewer timesteps. Additionally, we present another experiment to showcase its combination with DeepCache:\n\n| Method | interval | FID Score | Speed Up|\n|:---:|:---:| :---:|:---:| \n|DiT-XL | 0 | 2.27 |x 1.0| \n|DiT-XL + DeepCache |2|2.48|x 1.93|\n|DiT-XL + DeepCache + SparseDiT |2|2.78| x 3.52| \n\nIf there are specific papers or methods you believe warrant further comparison, I would appreciate you point out them and we will do further comparisons.\n\n## **Limitations**\n(1) For hyperparameter selection details, please refer to my response to Reviewer kysk Q2.\n\n(2) I provide an analysis of ToMeSD in our Appendix. Unlike ToMeSD, which directly adapts motivation and techniques from vision transformers, our approach involves a detailed observation of DiT's behavior in generation tasks and a comprehensive redesign of all modules accordingly.\n### **Reference**\n[1] Lou, J., Luo, W., Liu, Y., Li, B., Ding, X., Hu, W., ... & Ma, C. (2024). Token Caching for Diffusion Transformer Acceleration. arXiv preprint arXiv:2409.18523.\n\n[2] Yuan, Z., Zhang, H., Pu, L., Ning, X., Zhang, L., Zhao, T., ... & Wang, Y. (2024). Ditfastattn: Attention compression for diffusion transformer models. NeurIPS, 37, 1196-1219.\n\n[3] You, H., Barnes, C., Zhou, Y., Kang, Y., Du, Z., Zhou, W., ... & Lin, Y. C. (2025). Layer-and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers. CVPR (pp. 18072-18082).\n\n[4] G. Fang, X. Ma, and X. Wang. Structural pruning for diffusion models. NeurIPS, 2023."}, {"review_id": "NODvSNYGkZ", "reviewer": "Reviewer_rhnM", "summary": "The paper proposes SparseDiT, a new architecture that avoids redundant computations in diffusion transformers, leading to increased throughput at minimal or no quality cost. Compared to the normal DiT, the authors introduce two components. First, poolingformer, which simply removes the computation from queries and keys, and simply average pools and repeats the values of the first layer's attention to save the quadratic attention computation (in the first few transformer blocks). And second, sparse-dense token modules (SDTM), which cross-attend to a pooled version of the dense tokens, perform attention with sparse tokens, and then upsample the tokens back to dense tokens. The authors propose a timestep-wise pruning strategy for the number of tokens in the SDTM modules to achieve a balance between performance and speed. The authors show the superiority of their method on class-conditional and text-conditional image generation, as well as video generation.", "questions": "I believe the proposed method has clear potential. However, to consider increasing my final rating, I would appreciate clarification on the following key points (which were already mentioned above):\n\n1. Clarification on baselines and direct empirical comparison to other test-time caching or attention compression methods.\n2. Direct comparison of FLOPs, parameters, and throughput for a DiT block versus SDTM, to isolate the efficiency gains due to token sparsity.\n3. Isolate the gain from the proposed architectural components compared to continued training.", "limitations": "The limitations section mainly consists of a single sentence that describes the main limitation of requiring manual analysis of the structure design. This neither discusses the limitation nor provides any explanation or solution to this problem. Hence, I would consider this as insufficient.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is well-organized and well-written, with a clear common thread. A key strength lies in the comprehensive experimental evaluation, which spans class-conditional and text-conditional image generation as well as video synthesis, demonstrating the broad applicability. The proposed modules are simple, yet effective in increasing the throughput of diffusion transformers. The spatially-adaptive design based on their empirical analysis of attention is interesting. Additionally, the timestep-aware pruning strategy is both intuitive and effective. Overall, I think it is a solid and simple method to improve efficiency.\n\n**Weaknesses**\n\n- Lack of baselines: While the paper provides many internal comparisons, it lacks comparisons to other strong baseline methods. In the supplementary material, the authors compare to DyDiT and ToMeSD, however, there exist other test-time caching methods, some of which are mentioned in the related works section, but not empirically compared to (i.e., [1, 2, 3]). I would consider direct performance comparisons to be crucial since some of these methods avoid computing tokens entirely and do not require retraining the base model at all. The authors also do not compare their method to attention compression methods (e.g., [4]).\n- Potential mischaracterization of flow-based methods: The paper describes flow-based approaches (e.g., Rectified Flow) as methods that “straighten transport trajectories” (l.99), which may oversimplify their role. In practice, Rectified Flow provides a more general framework that modifies aspects like the interpolant and training target [5], rather than explicitly straightening trajectories. While trajectory straightening can be achieved through techniques like ReFlow [2], which trains on image-noise pairs from a pre-trained flow model, this is not a property of Rectified Flow per se (albeit found in some papers empirically, e.g. [5, 7]). This conceptual ambiguity also makes the comparison between DDIM and RFlow in Table 5 somewhat unclear. Since DDIM is closely related to Euler discretization in Rectified Flow, a more representative comparison of samplers would involve solvers such as DPM [8] or DPM++ [9].\n- Lack of analysis protocol: The proposed tri-segment design and pruning strategy requires prior analysis on a trained model. Hence, the method can be considered as a post-training strategy. However, the analysis on which the authors ground their method doesn't establish a clear and systematic protocol for selecting key hyperparameters (e.g. number of SDTM modules). While the justification for using poolingformer layers is reasonable, it remains unclear how the authors infer the distinction between local and global attention solely from the normalized attention variances across layers. A more systematic protocol and detailed analysis would definitely strengthen the paper.\n- Missing isolated analysis of SDTM vs DiT block: The SDTM module introduces additional parameters and FLOPs through the two cross-attention operations and the two linear mixing matrices $W_1, W_2 \\in \\mathbb{R}^{C \\times C}$. I would highly appreciate a direct comparison of throughput, FLOPs, and parameter counts between a standard DiT block and SDTM. This would clarify whether the efficiency gains are truly due to token sparsity.\n- Confound metrics from finetuning: The improved FID scores reported e.g., in table 1, could also stem from continued training of the DiT rather from the proposed architectural changes. A comparison to the DiT-XL and PixArt-$\\alpha$ baselines with continued training would help isolate the effect of the proposed method.\n\n**General Questions and Clarifications**\n- Usually DiT models contain high norm tokens. Did you observe similar patterns in your analysis?\n- What kind of token upsampling do you use for your method? How do you handle different sizes for floating point numbers of $r$?\n- l266ff: You mention that SparseSiT gives a more precise structure. Is this consistent behaviour or are these just artefacts from the initial noise? For example, sometimes one seed might work better for your method than for the baseline. Can you provide any support for this claim or an intuitive explanation?\n\n**Minors**\n- writing (L96ff and L193ff)\n- improved readability of the settings for DiT-B and DiT-XL in experimental section (e.g. in form of a table)\n- label of table 5 and table 7 seem to be switched\n- lack of other metrics for the T2I setting (e.g. CLIP score, etc)\n\n\n**References**\n- [1] Wimbauer, F., Wu, B., Schoenfeld, E., Dai, X., Hou, J., He, Z., ... & Wang, J. (2024). Cache me if you can: Accelerating diffusion models through block caching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6211-6220).\n- [2] Ma, X., Fang, G., & Wang, X. (2024). Deepcache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 15762-15772).\n- [3] Lou, J., Luo, W., Liu, Y., Li, B., Ding, X., Hu, W., ... & Ma, C. (2024). Token Caching for Diffusion Transformer Acceleration. arXiv preprint arXiv:2409.18523.\n- [4] Yuan, Z., Zhang, H., Pu, L., Ning, X., Zhang, L., Zhao, T., ... & Wang, Y. (2024). Ditfastattn: Attention compression for diffusion transformer models. Advances in Neural Information Processing Systems, 37, 1196-1219.\n- [5] Ma, N., Goldstein, M., Albergo, M. S., Boffi, N. M., Vanden-Eijnden, E., & Xie, S. (2024, September). Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision (pp. 23-40). Cham: Springer Nature Switzerland.\n- [6] Liu, X., Gong, C., & Liu, Q. (2023, January). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. In The Eleventh International Conference on Learning Representations (ICLR).\n- [7] Schusterbauer, J., Gui, M., Ma, P., Stracke, N., Baumann, S. A., Hu, V. T., & Ommer, B. (2024, September). FMBoost: Boosting Latent Diffusion with Flow Matching. In European Conference on Computer Vision (pp. 338-355). Cham: Springer Nature Switzerland.\n- [8] Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., & Zhu, J. (2022). Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35, 5775-5787.\n- [9] Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., & Zhu, J. (2025). Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. Machine Intelligence Research, 1-22.", "was_revised": true, "final_justification": "The authors clarified all my remaining questions in the rebuttal.", "rebuttal": "# To Reviewer rhnM\nWe are pleased that you found our paper well-organized and appreciated the comprehensive experimental evaluation across various generation tasks. Your recognition of the effectiveness of our proposed modules is greatly appreciated. We are committed to addressing any remaining issues and questions you have.\n## **Weakness 1: Lack of baselines**\nThank you for emphasizing the importance of comparing our method with strong baseline techniques. \n1. **Compare with TokenCache [3].**\n| Method | Base Model | FID Score | Speed Up | \n| :---:|:---:| :---:|  :---:| \n|TokenCache|DiT-XL/2 | 2.37 (+0.13) | x 1.32 | \n|Ours| DiT-XL/2 |2.38 **(+0.11)**|**x 1.87**| \n\nWe adopt the standard DiT baseline 2.27, while TokenCache applies a baseline 2.24. TokenCache achieves a 32% speed gain with a slight FID increase. Our approach attains an 87% speed gain with a smaller FID increase, demonstrating superior efficiency and performance.\n\n2. **Compare with Ditfastattn [4].**\n| Method | Base Model | FID Score | IS | Speed Up | \n| :---:| :---: | :---:|:---:| :---:| \n| Ditfastattn | DiT-XL/2 (512x512) | 4.52 (+1.36) |  180.34  | x 1.98| \n|  Ours | DiT-XL/2 (512\\x512) | 2.96 **(-0.08)** |**242.4**  | **x 2.45** | \n\nWe adopt the standard DIT baseline 3.04, while TokenCache applies a baseline 3.16. Ditfastattn shows a substantial FID increase for speed gain, while our method results in better speed and less FID.\n\n3. **Compare with DeepCache [2] and BlcokCaching [1]**\n\nDeepCache and BlockCaching share similar features across sampling steps, relying on the redundancy present in multiple steps. In contrast, our approach targets reducing redundancy within the DiT structure at a single sampling step. Our method and their techniques improve diffusion model efficiency along orthogonal dimensions and should be considered complementary rather than directly comparable. For a 1000-step DiT model, DeepCache can achieve substantial reductions in FLOPs—up to 90%—by leveraging massive redundant features across steps. However, obtaining a 90% reduction through model structure compression alone is impractical. Conversely, in a single-step DiT model, our approach can achieve a 50% reduction in FLOPs, while DeepCache and BlockCaching cannot reduce FLOPs, as there are no features to share across steps. Therefore, our method complements DeepCache and BlockCaching by optimizing different aspects of the model's efficiency. Other state-of-the-art methods like DyDiT, DiffCR, and Ditfastattn similarly do not compare against DeepCache and BlockCaching directly.\n\nInstead of comparing solely, we have integrated our approach with DeepCache, demonstrating enhanced performance as follows:\n| Method | interval | FID Score | Speed Up|\n|:---:|:---:| :---:|:---:| \n|DiT-XL | 0 | 2.27 |x 1.0| \n|DiT-XL + DeepCache |2|2.48|x 1.93|\n|DiT-XL + DeepCache + SparseDiT |2|2.78| x 3.52| \n\nThese results show the effectiveness of combining SparseDiT with DeepCache.\n\n## **Weakness 2: Potential mischaracterization of flow-based methods.**\nThank you for identifying conceptual inaccuracies in our paper. We agree with your points and will revise the flow-based approach descriptions, particularly those related to Rectified Flow, to better reflect their roles and frameworks. We experiment 10-step DPM in Table 5. DiT FID: 12.01; Our FID: 12.30.\n\n## **Weakness 3: Lack of analysis protocol.**\n(1) SDTM Module Configuration: The number of SDTM modules is empirically determined based on the baseline model's layers. Typically comprising six layers with three sparse transformers, which can achieve a good performance-efficiency trade-off, a DiT-XL houses 28 layers, necessitating four SDTM modules in SparseDiT-XL. Please refer to my response to Reviewer kysk Q2 for more details.\n\n(2) We calculate attention variance to assess the uniformity of attention value distribution. A variance close to zero indicates that attention values are nearly identical, allowing self-attention to function as a global pooling mechanism. Conversely, high variance suggests that each token focuses on a limited set of other tokens. Unlike understanding tasks, such as classification, where global tokens might exist, generation tasks involve attention patterns that tend to capture local information, visible as diagonal lines on attention maps (approximate full-rank matrix). High variance indicates that all tokens are necessary for feature integration. Therefore, a higher variance reflects a greater need for localized information capture. In understanding tasks, high variance can still signify global information capture, but this scenario does not apply to generation tasks.\n\n## **Weakness 4: Missing isolated analysis of SDTM vs DiT block.**\nIt seems there may be a misunderstanding regarding our method. We do not incorporate additional cross-attention layers; rather, the cross-attention layers in SDTM are derived from the existing self-attention layers. The total number of attention layers in SparseDiT, encompassing both self-attention and cross-attention, remains identical to that of DiT. The parameters of the cross-attention layers match those of the initial self-attention layers, hence no additional parameters are introduced via the cross-attention layers. We acknowledge that two additional linear layers, $W_1$ and $W_2$, introduce extra parameters. However, our poolingformers counterbalance this by utilizing parameter-free full-one matrices to replace self-attention layers, thereby reducing overall parameters. Consequently, the parameter count of SparseDiT remains nearly unchanged. Notably, one SDTM comprises six transformer layers. Below, we provide a detailed parameters and FLOPs for these six transformer layers ($r\\in$ [0.61, 0.86]):\n\n| Block Name | Parameters | FLOPs| \n| :---:| :---:| :---:|  \n| Sparse Token Generation Transformer (Cross-Attention) | 23.905M | 1.539G | \n|  Sparse Transformer (Self-Attention) | 23.905M | 1.023G |\n|  Sparse Transformer (Self-Attention) | 23.905M | 1.023G |\n|  Sparse Transformer (Self-Attention) | 23.905M | 1.023G |\n|  Dense Token Recovery Transformer (Cross-Attention) | 26.562M | 4.258G |\n|  Dense Transformer (Self-Attention) | 23.905M | 4.087G |\n\nEach SDTM consists of six transformers, and should be compared with six DiT transformer blocks in terms of parameters, FLOPs, and throughput:\n| Block Name | Parameters | FLOPs | Throughput|\n|:---:| :---: |:---:|:---:|  \n| SDTM | 0.146G | 12.967G | 12.1 |\n| Six DiT Transformers | 0.143G | 24.522G | 7.33 |\n\nOwing to the efficiency of poolingformers, the difference in parameters between SparseDiT and DiT is minimal (680M vs. 680M). The efficiency gains are truly from token sparsity.\n\n## **Weakness 5: Confound metrics from finetuning.**\n| Model | Continued Iteration | FID |\n| :---:| :---: | :---:| \n| DiT-XL/2 | 400K |  2.15 |\n|  PixArt-$alpha$ | 400K | 4.36 |\n\nWe acknowledge that continued training can enhance the performance of base models, as demonstrated in the table above. However, the primary objective of our method is to achieve a superior performance-efficiency trade-off, rather than improving performance. Additionally, the benefits gained from continued training apply equally to all post-training methods, such as DyDiT, DiffCR [10], TokenCache [3], and Ditfastattn [4]. Consequently, the comparative advantages of our method remain consistent.\n\n## **Q1: High norm tokens**\nWe did not specifically observe the norm of tokens in our analysis. Our approach focuses on reducing the number of tokens without directly discarding them. Tokens are merged via pooling layers, which represents a form of soft pruning, allowing both high and low norm tokens to be retained without information loss.\n\n## **Q2: Upsampling and floating point numbers of $r$**\nOur method utilizes trilinear upsampling, implemented through torch.nn.Upsample. The computation of $r$ is detailed in line 166, where it is adjusted by altering the number of sparse tokens, set in configurations like $4\\times 4$ or $8\\times 8$. Notably, $r$ remains a discrete value rather than a continuous value.\n\n## **Q3: l266ff**\nWe have observed that our method generates more stable and precise images across different seeds. The seed used in Figure 4 is the default seed 1, chosen to facilitate easy reproduction by readers. We plan to provide additional visualizations in the next version to further support this observation.\n\n## **Minors**\nWe appreciate your detailed suggestions regarding the minors. We will modify these problems in the next version.\n\n## **Limitation**\nDue to constraints on text space, I have not included a detailed discussion of the solution. Regarding the number of PoolingFormer layers, Figure 1 illustrates that the attention maps from the first two transformer layers exhibit a uniform distribution. Therefore, one or two PoolingFormer layers are adequate, as confirmed by Table 4. In the SDTM architecture, there are three fixed transformers: one for generating sparse tokens, one for recovering dense tokens, and one dense transformer. The number of sparse transformers can be varied, with more sparse transformers enhancing model efficiency. Performance remains consistent as long as the number of sparse tokens does not exceed three. Concerning the pruning rate $r$, our method is not particularly sensitive except for the rate in the final stage. As mentioned in lines 333-336, the effectiveness of the dynamic pruning strategy is closely related to the token count in the later phases of denoising. For practical applications, we suggest using a $10\\times 10$ token count in the final stage for optimal performance and a $4\\times 4$ token count in the initial stage for efficiency.\n\n### **Reference**\n[10] You, H., Barnes, C., Zhou, Y., Kang, Y., Du, Z., Zhou, W., ... & Lin, Y. C. (2025). Layer-and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers. CVPR (pp. 18072-18082)."}]

Author rebuttal:

Meta-review: This paper presents a novel framework for improving the computational efficiency of diffusion Transformers through token sparsification. Reviewers acknowledged the contribution and strong performance of the proposed method, while initially raising some concerns, such as its unclear experiment setups and incomplete baseline comparisons.

After the rebuttal, the authors addressed most of the concerns, and all reviewers agreed to accept this paper. AC read all the reviews, author rebuttals, and the paper, and believes this is a strong paper and recommends acceptance.
Decision: Accept (poster)