PT-T2I/V: An Efficient Proxy-Tokenized Diffusion Transformer for Text-to-Image/Video-Task
Authors: Jing Wang
Ao Ma
Jiasong Feng
Dawei Leng
Yuhui Yin
Xiaodan Liang
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion Transformer
Image Generation
Video Generation
Primary area: generative models

Abstract: The global self-attention mechanism in diffusion transformers involves redundant computation due to the sparse and redundant nature of visual information, and the attention map of tokens within a spatial window shows significant similarity. To address this redundancy, we propose the Proxy-Tokenized Diffusion Transformer (PT-DiT), which employs sparse representative token attention (where the number of representative tokens is much smaller than the total number of tokens) to efficiently model global visual information. Specifically, within each transformer block, we compute an averaging token from each spatial-temporal window to serve as a proxy token for that region. The global semantics are captured through the self-attention of these proxy tokens and then injected into all latent tokens via cross-attention. Simultaneously, we introduce window and shift window attention to address the limitations in detail modeling caused by the sparse attention mechanism. Building on the well-designed PT-DiT, we further develop the PT-T2I/V family, which includes a variety of models for T2I, T2V, and T2MV tasks. Experimental results show that PT-DiT achieves competitive performance while reducing computational complexity in image and video generation tasks (e.g., a reduction 59\% compared to DiT and a reduction 34\% compared to PixArt-$\alpha$). The visual exhibition of and code are available at https://360cvgroup.github.io/Qihoo-T2X/.

Published as a conference paper at ICLR 2025
PT-T2I/V: AN EFFICIENT PROXY-TOKENIZED DIF- FUSION TRANSFORMER FOR TASK
Jing Wang+.2; Ao Ma Jiasong Feng"*, Dawei Leng" Yuhui Yin', Xiaodan Liang2.3.4f
1 Shenzhen Campus of Sun Yat-Sen University, 2360 Al Research, *Pngg Cheng Laboratory, *Gunggoong Key Laboratory of Big Data Analysis and Processing,
wangj9777maii sysu. edu. cn {maao, Fengjiasong lengdawei] 03300.cm on xdl xdliangg288ggaiii coom
ABSTRACT
The global self-attention mechanism in diffusion transformers involves redun- dant computation due to the sparse and redundant nature visual informa- tion, and the attention map of tokens within spatial window shows signifi- cant similarity. To address this redundancy, we propose the Proxy-Tokenized Diffusion Transformer (PT-DiT), which employs sparse representative token at- tention (where the number of representative tokens is much smaller than the to- tal number of tokens) to efficiently model global visual information. Specifi- cally, within each transformer block, we compute averaging token from each spatial-temporal window to serve as a proxy token for that region. The global semantics are captured through the self-attention of these proxy tokens and then injected into all latent tokens via cross-attention. Simultaneously, we introduce window and shift window attention to address the limitations in detail modeling caused by the sparse attention mechanism. Building on the well-designed PT- DIT. further develop the PT-T2I/V family, which includes variety of models for T21, T2V, and T2MV tasks. Experimental results show that PT-DiT achieves competitive performance while reducing computational complexity in image and video generation tasks (e.g., a reduction 59% compared to DiT and a reduction 34% compared PixArt-a). The visual exhibition and code are available htttp:://660vvgree github.
INTRODUCTION
Recent advancements in core diffusion models, including Sora (OpenAl 2024). Kling (Huawei 2024). Stable Diffusion 3 (Stability Al 2024). PixArt-a/21) /// (Chen et al. 2023
2024a b). Vidu
(Shengshu AI 2024). Lumina-T2X (Gao et al. 2024). Flux (BlackForestlabs Al 2024). and VigVi Yang al. 2002)), have led to significant achievements in the creation of photo-realistic image and video. Transtormer-bass
models such as Sora and Vidu have demonstrated the ability to generate high-quality samples at arbitrary resolutions. These models also adhere strongly to scal- ing laws, achieving superior performance as parameter sizes increase. Additionally, Lumina-Te T2x has shown uniformity in performing various generation tasks, further validating the potential of the transformer- basedd architectures in diffusion models.
However, the quadratic complexity of global self-attention concerning sequence length increases the computational cost of the Diffusion Transformer, leading practical challenges such as longer generation times and higher training costs. This issue also hinders the application of DiT to high-
quality video generation. For example, while 3D attention- based approaches(Xu et al.] 2024 Yang
ct al. 2024: Lab & etc. 2024 Gao et al. 2024) have demonstrated superiority over 2D spatial at- Tention combined with II) temporal attention counterparts
al. 2024 Ma et al. 2024b
Bar-Tal et al. 2024: Blattmann et al. 2023 Lu et al. (003). the extensive computational demands limit their scalability for higher-resolution and longer video generation. Current studies (Han al.)
2023 Koner et al. 2024, Yu et al.] 2024) in visual understanding and recognition have highlighted that global attention mechanisms often exhibit redundancy due to the sparse and repetitive nature of visual information. Specifically, by visualizing the attention map, we observe that the atten- tion of tokens within the same window similar for spatially distant tokens, while differing for
"Equal Contribution. 'Corresponding Authors

---

Published as a conference paper at ICLR 2025
A lugh pil Jant De betwart Sage
atribed vit Mie
baibeckgree and sapissad crlor, cmadiig
stcoplinn
bodlgronnd secatts
Two papet thollce shoe cipeppee alarge spantlly Bylng cover cit.
WiE coradaddaie outrads sutrunalll
seal plangroois be ond. Steson pervidin wanst aallight
A socare of thep laving taved aereddiee cased
Figure 1: The samples from PT-T2I showcase high fidelity and aesthetic qualities, demonstrating strong consistency with given textual descriptions.
spatially neighboring tokens, as illustrated in Fig. 3 This observation indicates that the dense long-range attention, which triggers significant computational overhead, is redundant, Thus, reduc- ing this redundancy is believed to enhance the efficiency of Diffusion Transformers in generating higher-resolution images and longer videos.
In this paper, we propose the Proxy-Tokenized Diffusson Transformer (PT-DiT) and further present the PT-T2I/V series, which includes both Text-to-Image, Text-to-Video, and Text-to-Multi View generation models. To address the redundancy of visual information, PT-DiT employs proxy- tokenized attention instead of a global attention mechanism to reduce the computational complexity of visual token interaction. Specifically, we first recover the spatial and temporal relationships of the token sequence through a reshaping operation. Given the similarity of visual information within localized spatial regions and temporal frames, calculate temporal window as a representative token, forming a set of proxy tokens. The interaction and broadcasting of visual global information then achieved through self-attention among proxy kens and cross-attention between proxy tokens and all latent tokens. Additionally, to enhance the texture modeling capabilities, we introduce window attention and incorporate shift-window atten- tion, similar to Swin Transformer (Liu et al. 2021) to avoid lattice artifacts as shown in Fig. g
averaging token from each spatial-
With the dell-designed proxy-tokenized attention, PT-DiT can be adapted both image and video generation tasks without structural adjustments. For image generation, as shown in Fig. 2 compared PixArt-a (Chen et al. (023), our method achieves an approximate 33% reduction in computa- tional complexity GFLOPs under the same parameter scale. For video generation, in contrast to 2D spatial and iD temporal attention, which has limited spatial-temporal modeling, and 3D full- attention, which suffers from high computational complexity, PT-DiT can efficiently and compre- hensively extracts 3D information, benefiting from proxy token interaction mechanisms.
2

---

Published as a conference paper at ICLR 2025
Experimental results demonstrate that our method achieves competitive performance with significant efficiency. As shown in Fig.
8000
8536.
Complesll (FFLOPS
5425
33.6%
PT-T21 can generate high-quality and high- fidelity images while closely adhering the provided text instructions. Mcanwhile, for
32.5%
2000 4000
142.8
366.1
5664.9
the image generation task, PT-DiT's computa- tional complexity is 51% of DiT and 66% of PixArt-a for the same parameter size. For the video generation task, despite having 3 million more parameters than EnimAnimate V4, the PT- DiT/H's computational complexity is only 82% of EasyAnimate (Xu et al. 2024) and 77% of CogVideo2 (Yang et al. 2024) for the same pa-
101.7
2141.2
Pb
1423.9
PT-DIT/L-0GB
Resolution 256
512
1024
2048
Figure 2: Comparison of complexity between
PixArt-a and PT-DiT/L at various resolutions.
rameter size. Overall, using the standard 3D VAE settings (8x spatial downsampling rate and 4x temporal downsampling rate), experimental tests indicate that we can train the PT-DiT/XL (1.1B) model for images at resolution of 2048 2048 or for video at resolution of 512 x 512 288 on the 64GB Ascend 9 910B (Huawei 2024).
In summary, the unique contributions of this paper be summarized follows:
We analyze redundant computations in self-attention within Diffusion Transformers, caused by visual sparsity and redundancy. We find that redundancy primarily arises in at- tention interactions within the same spatial window and between distant tokens, while mod- eling relationships between spatially adjacent tokens crucial and should be preserved.
We introduce a proxy token mechanism that leverages spatial priors for local token fusion generate proxy tokens, and uses Proxy Token Attention and Visual Cross-Attention to efficiently establish and propagate global associations, while retaining all latent tokens to preserve detailed texture. Additionally, window and shifted window attention are employed to enhance the modeling of spatial proximity.
Through extensive experimentation, we demonstrate that our efficient mrxxytooeen-base diffusion transformer model achieves competitive performance with state-of-the-art T21 and T2V models. Our approach lays the foundation for T2I and T2V models across diverse scenarios, offering significant computational efficiency advantages. We will open-source both our models and code support the advancement of efficient diffusion transformers.
RELATED WORK
Image Generation with Diffusion Transformer.
Please refer to Appendix, A.I
Video Generation with Diffusion Transformer. Building on the advancements in image gen- eration with Diffusion Transformers, recent work (Xu et al. 2024 Yang et al., 2024 Lab & etc. 2024 Zheng et al., 2024 Ma et al. 2024b Lu et al. 2023) has been devoted to extending the DiT structure to video generation. Easy Animaee V2 (Xu et al. (024), Open-Sora (Zheng et al. 2024) is based on PixArt-a (Chen et al., 2023), incorporating temporal ID attention and utilizing a 3D VAE generate additional frames. Cog VideoX (Yang et (222). Open-Sora-Plas (Lab & etc. 2024).
Lumina-T2M, T2X (Gao et al., 2024) and EasyAniman V4 (Xu et al. (0244) points out the shortcomings of temporal 1D attention and employs an expert transformer with 3D attention. While 3D attention effectively manages significant motion between adjacent frames, it also incurs a substantial com- putational cost. To overcome this challenge, we propose PT-DiT, which introduces an innovative token compression strategy. This strategy compresses not only in spatial dimensions but also across frames, enabling 3D attention with significantly reduced computational overhead.
Efficient Diffusion Transformer. As the application of transformers becomes more mature, there are some solutions proposed in different fields targeting the attention computation problem (Han) jet al. 2023, Dubey et al. 2024, Shi et al. 2023a Bolya et al. 2022 Jiang et al. 2022). Llama3 (Dubey et al. 2024) adopts the K V-Cache (Pope et al. 2023) to reduce the number of redundant cal- culations. AgentAttention (Han et al., 2023) employs the mediator token mechanisms to compress the interaction scale between Query and Key, yielding promising results in fundamental visual tasks.
3

---

Published as a conference paper at ICLR 2025
ToMe (Bolya et al. 2022). TRIPS (Jiang al., 2022 and CrossGET Shi et al. 2023a) propose token merging strategies to reduce the number of tokens involved in global self attenttttnionn thereby im- proving the efficiency of image, language, and multimodal understanding models. Similarly, DAM et al. 2024) and ToMeSD (Bolya & Hoffman, 2023) apply mediator token and token merging to Diffusion Transformer save computational costs, thereby reducing training overhead and inference time. However, token merging often leads to the loss of detailed information, which is particularly problematic for generative tasks that require strict detail preservation. Additionally, the compression of interaction scales between Query and Key in the mediator token mechanism can lead loss of important spatial relationships between neighboring tokens, which is erucial as illustrated in Sec. 3 To address these challenges, we propose a proxy token mechanism that establishes global associations through attention on limited number of proxy tokens, guided by visual-spatial priors. At the same time, all tokens are retained to prevent the loss of detailed information, and the global associations in the proxy tokens are propagated to all detailed tokens. Furthermore, the modeling of spatially neighboring tokens is supported by window attention strategies.
METHOD
3.1
REDUNDANCY ANALYSIS
h X
h
X
W
spatial window
Index
MXW W
Assamble
Taken index
Attention map
Latent code
Similar
Different
redundant
Attention map of all latent token.
Attention maps of tokens in a window.
The attention map of self-attention in PixArt-a: at 512 resolution. We assemble the attention map for 16 tokens within a x 4 spatial window. The vertical axis represents different tokens within the window, and the horizontal axis represents their correlation with all latent tokens. It is evident that the attention of different tokens in the same window is almost identical for spatially distant tokens, whereas there noticeable variation for spatially neighboring tokens.
Figure 3:
Due to the sparsity and redundancy of visual information, global attention mechanisms in existing DiTs exhibit significant redundancy and computational complexity, particularly when processing high-resolution images and longer vidcos. We analyze this computational redundancy by visualiz- ing the at a resolution of 512> x 512, as shown on the left in Fig. 3 The attention map for latent codes within a spatial window then assembled, as depicted on the right side of Fig. 3 (where the vertical axis represents different tokens in a window, and the horizontal axis represents the correlation with all latent tokens). It is evident that the attention maps for different tokens within the same window are nearly uniform for spatially distant tokens (i.c., at the same horizontal position, the vertical values are almost identical). Moreover, window tokens exhibit varying attention to spatially neighboring to- kens. This suggests that computing attention for all latent tokens is redundant, while attention for spatially neighboring tokens is critical. Consequently, we propose a sparse attention strategy that samples limited proxy tokens from each window perform self-attention, thereby reducing redundancy and decreasing complexity. Additionally, the association between spatially neighboring tokens is established through window attention. Further details are elaborated in Sec. 3.2
maps. Specifically, we examine the attention map of self-attention in PixArt-a
3.2 ARCHITECTURE OF PT-DIT
As shown in Fig. our proposed Proxy-Tokenized Diffusion Transformer (PT-DiT) introduces the proxy-tokenized mechanism to reduce the number of tokens involved in computing global self-attention, thereby efficiently establishing global visual associations. Specifically, the latent code z E Fx x is passed through path embedding to obtain the latent code sequence Ra RND. Subsequentl, we add 3D positional encoding 2, and feed it into the well-designed
4

---

Published as a conference paper at ICLR 2025
Timestep
tth Proxy Token Block
Scale
A serene scene surrounded lush green forest
Feed Forward Network
MLP
Scale, Shift
Image or Video
GIIM
T5
Visual Cross-A ttention
K
Text Cross Attentonn
Text F Feature
TCM
Scale
Scale
Proxy Token A ttention
Zero Linegr
Shift Window A Attention
3D VAE
Scale, Shift
Scale, Shift
Noise,
Averaging
Noised Latent
Scale
Patch
Window ttention
Embedding
Latent tokens
Scale, Shift
The scope of window attention
+ 3D PE
Figure 4: The overall architecture of PT-DÎT. The image or video undergoes processing through a 3D VAE, followed by noise addition, patch embedding, and positional encoding to generate latent tokens. We replace global attention with proxy-tokenized attention to establish contextual associa- tions and employ visual cross-attention to propagate this information to all tokens, thereby reducing computational redundancy. Moreover, texture detail modeling is enhanced through window attention and shifted window attention.
Proxy-Tokenized Blocks (PT-Block). Compared the vanilla diffusion transformer block, the PT- Block introduces Global Information Interaction Module (GIIM) and a Texture Complement Mod- ule (TCM). The GIIM facilitates efficient interaction among all latent codes using sparse proxy- tokenized mechanisms, while the TCM further refines local detail through window attention and window attention. Below, we describe the GIIM and TCM detail.
3.2.1 GLOBAL INFORMATION INTERACTION MODULE
Given series of latent tokens, we first sample a set of proxy tokens based on their spatial and tem- poral priors. Each proxy token represents a localization within the image video and interacts with proxy tokens in other regions to establish global relationships. Then, the information contained in proxy tokens is propagated to latent tokens, enabling efficient global visual information interaction.
Specifically, we reshape the latent code sequence 2, E NxD to E [BdxxxxxD where f. h and denotes the frame, height, and width of video or image (f 1) in the latent space after patch embedding, thereby recovering its temporal and spatial connections. The set of proxy tokens
Dx 1 P. IR. Pr Ph Pa is calculated from each window of size Pt Ph X Pw using the averaging operation. The parameters Pf. Ph. and per indicate the compression ratios for frame, height, and width, respectively. Each proxy token represents p, X Pı P. tokens, modeling global information with the other proxy token through self-attention. Subsequently, cross-attention is performed to propagate the global visual information into all latent tokens 2,, where the 2, serves as the Query and the proxy tokens Pa serve as the Key and Value. The above process is mathematically expressed follows:
24 CS( SA SAAeSAAAAAiiiigg.....
where Averaging(-) refers to the averaging operation applied to tokens within the same window to extract proxy tokens, and CS(:,) .) and SA(-) represent the cross-attention and self-attentio opera- tions, respectively. Besides, we introduce a linear layer with zero initialization to enhance training stability. This approach allows the PTT-Block to achieve efficient global information modeling and
5

---

Published as a conference paper at ICLR 2025
avoids the computational overhead caused by redundant computations in self-attention. We will analyze the computational complexity advantages of GIIM further in Sec. 3.3
3.2.2 TEXTURE COMPLEMENT MODULE
Due to the characteristics of the sparse proxy tokens interactions, the model's capacity to capture detailed textures is limited, making it challenging meet the high-quality demands of generation tasks. To solve this problem, we introduce localized window attention, as proposed in Liu et al. (202I), which models texture information by computing attention only within local spatial window of the image. This mechanism is incorporated into the Texture Complement Module (TCM) due to its effectiveness in detail modeling and computational efficiency. Specifically, the latent tokens 2, fxhx reshaped to 2, E PPPx xps Xpp Xp where fxhxw denotes the number of window PxXpt Xp in image. Window self-attention is computed along the second dimension. To further enhance the model. shidtw attention is integrated into TCM, which applies spatial tramslation to the window divisions, enabling connections between neighboring tokens across different windows and mitigating the "grid" phenomenon caused by localized window attention. The formula for this process is as follows:
E, WSA(2.) 2a:
2s:
Zw = SWSA($)) + E.:
where SWSA(-) and WSA(-) denote shift-window attention and window attention respectively. Both window attention and shift-window attention introduce visual prior to DiT, which aids in the construction of texture details and advances the training of visual generators. Moreover, the increase computation is minimal due to the limited number of tokens each window, We will analyze this in detail in Sec. 3.3 Then, Zm is reshaped to Zw NxD and fed into Textual and MLP, similar to DIT.
3.2.3 COMPRESSION RATIOS
For the image generation task, we first determine the compression ratio at a resolution of 256. At this resolution, after applying the VAE (8x down-sampling) and patch embedding (2x down- sampling), the image is reduced to only 16 x 16 tokens. With a compression ratio of (1, 4, 4), the number of windows in the space becomes 4 x 4, and the number of proxy tokens is 16. With this configuration, the limited number of windows and the large coverage area of each window make it challenging for single proxy token effectively represent the complex information within the window for global information modeling. This leads to anomalies in the image layout and perfor- mance degradation, as shown in the Table. 2(d) Setting the compression ratio to (1, 2, 2) maintains reasonable number of windows while preserving the necessary semantic richness within each win- dow. Meanwhile, maintaining the same number of windows across different resolutions benefits the model's 's training process from low-to-higg resolutions, as it ensures a consistent semantic hierarchy across resolutions. Therefore, the compression ratios (py, Ph. Pw)) is set to (1, 2, 2), (1, 4, 4), (1, 8, 8), and (1, 16. 16) at 256, 512, 1024, and 2048 resolution respectively. It is worth noting that when the input is an image, f and py will be set to 1.
For the video generation task, compression consistent. Owing to token compression in the frame, height and width dimensions, PT-DiT can effectively train a generator for longer videos.
Py 4 across different resolution to maintain the temporal
3.3 COMPLEXITY ANALYSIS
With small number of representative token attention, PT-DiT reduces the computational redun- dancy of the original full token self-attention. The advantages of our method in terms of computa- tional complexity are further analyzed theoretically in the following.
The computational complexity of self-attention is 2N D, computed as follows:
Softmax(2(0) (N)T (VD)2
complerity
N²D
N'D
+O(N2),
(3)
Softmex())())

---

Published as a conference paper at ICLR 2025
where N denotes the length of latent tokens and D represents feature dimension. Similarly, the computational complexity of GIIM and TCM is computed as follow:
complerity = 2
N²
D+
N PSPADW
P/PAPo (pyPhPw)22
(4)
SA in GITM
CS in GIIM
WSA and SWSA in TCM
PprPAP"
D.D.
(PpPhPm)
PsPsPu
Obviously, due to the proxy-tokenized strategy, our method provides significant advantages, espe- cially with larger compression ratios (py, Phe Pw)) and longer sequence lengths (N). When (p/, Ph Pw) are (1, 2. 2), (1, 4, 4). (1, 8, 8), and (1, 16, 16) and the image resolution are 256 (N 256), 512 (N 1024). 1024 (NV 4096), and 2048 (N 16348). method accounts for only 34.3%, 9.7%, 4.7%, and 2.3% of the total self-attention. addition, PT-DiT offers even greater benefits for video generation tasks with longer sequence lengths. Experimental analysis is available in Sec. 4.4
4 EXPERIMENT
4.1 EXPERIMENTAL SETUP
Training Setting. Due to limitations in computational resources, only trained PT-T21 and PT- T2V based on PT-DiT/XL I.IB. Following previous methods (Xu et al. 20244) Yang et al. 2024] Chen et 2023), utilize the T5 large language model the encoder and train PT-T2I using a low- to high-res resolutio strategy divided into three stages. Detailed seper-parameter settings and the model configurations for various PT-DiT scales are provided in Appendix, A.2
Ablation Study. We conduct ablation experiments using class-conditional version of PT-DiT/S- Class (32M) on the ImageNet (Deng et al., 2009) benchmark at 256 resolution. The AdamW opti- mizer is utilized with constant learning rate of Te-4. We train the models for 400,000 iterations with a batch size of 256, while maintaining an exponential moving average (EMA) of the model weights. During inference, set the denoising step as 50 and use flassfier-ffree guidance (cfg=6.0).
4.2 QUALITATIVE ANALYSIS
PEP
PixArt-a
Flux. 1-dee
redhead woman in a nocklace, posing bofore à comic poster clad wall.
In video game, in suit and tie standing in the rain, holding a gun.
Futuristic city with green- white tree- likelike building by river boats floating.
large, vibrant painting of mountain with rod and blue hua.
Figure Qualitative comparison of Text-to-Image generation models.

---

Published as a conference paper at ICLR 2025
PFTTV
Ens/Anlnnteva
CocVideoX
The bustling city zight scene, tall buildings, the camera shills from lell side the frame right.
PET2
EageAninsztev
CogVidex
Blonde with sunflowers, smiling sunflower field under blue
Figure 6: Qualitative comparison of Text-to-Videe generation models.
Text-eo-Image. We provide qualitative comparison of PT-T2I with existing state-of-the-art Text- to-Image models (c.g.. PixArt-a and Flux) at resolution of 1024, as shown in Fig. PT-T2I ex- hibits competitive performance, generating photo-realistic images that align well with the provided text prompts. Additional samples generated by PT-T2I can be found in Anonymous Repository
Text-fo- Video. We also compare PT-T2V with the recently released open- source Text-to- Video models (i.e.. EasyAnimate V4 and CogVideoX) at resolution of 512, achieving comparable results, as depicted in Fig. 6 More video samples are available in the Anonymous Repository
Text-to-MV. Please refer to Appendix, A.5
4.3 QUANTITATIVE ANALYSIS
MS-COCO. We conduct experiments quantitatively evaluate PT-T2I using zero-shot FID-30K on the MS-COCO (Lin et al.) 2014) 256 x 256 validation dataset, as shown in Table I(a) Due to the distribution gap between our collected data and MS-COCO, there a resulting decrease in FID (Heusel al. 2017) metrics. Nevertheless, PT-T2I achieves competitive score of 15.70.
MSR-VTT and UCF-101. We evaluate PT-T2V on two standard video generation benchmarks, MSS-VVTT Xu al. 2016) and UCF-101 (Soomro et al. 2012), at resolution of 256. As shown in Table I(b) PT-T2V achieves state-of-the- tfe-----thhrtttt results among DiT-based approaches and demon- strates competitive performance compared to U-Net-based approaches. Notably, since Cog VideoX,
8

---

Published as a conference paper at ICLR 2025
EasyAnimate and PT-T2V all utilize T5 the text encoder, this creates gap in CLIPSIM com- pared to methods that employ the CLIP as the text encoder, such as AnimateDiff, DynamiCrafter, PixelDance, and FancyVideo.
Table 1: The quantitative evaluation of the Text-to-Image (a) and Text-to-Video (b) tasks.
(b) Quantitative evaluation on the UCF-101 (Soomro al. 2012 and (a) Quantitative evaluation the MS- M---MSRRVIT (Xu al. 2016). The best and second performing metries highlighted in bold and underline respectively.
COCO FID-30K scores (zero-shot).
Method
FID-30K4
Are Data UCE-101
MSR-VTT
Method
DALL-E Ramesh et
10.39
SD Romhach
2122
8.73
Imagen Sahama ei al 2122
7.27
AnimateDift Guo et al.
2023
U-NettOM 584.85 37.01
61.24
628.57
0.2881
RAPHAEL Exie
DynamiCrafter (Xin al
U-NetIOM 404.50
0.2659
Kolors [Team RUze)
23.15
PixelDance Zeng ml. 2024
ULNNet 242.82
42.10
49.36
181.00
0.3125
PixArt-O Chei el
FancyVidoe Fong
U-Net 10M 412.64
43.66
47.01
333.52
0.3076
Flox. (- PorckPorssliiss Al
2024
22.76
Videx 2B Yang et
2024
DTT 35M 680.11 13.44
62.57
418 14
0.2318
PT:T21
15.70
EasyAnimate
DIT 12M 694,80 44.09
92.35
568.99
0.2285
PT-12V
DIT 10M 384.03 35.19
$1.95
375.23
0.2349
2000
65590
261454.3
2446
22.8%
PT FT-DITTH-1.88
Exso Aimaatt 1.
27.2
2008
Compaaiis (CILORN)
201 764.7
66,4
70.7
500 1000 2000
66244-
30264.2
120514.0
80.8
64.2
65.5
55,7
P201.7
52.8
Cogy MecX-1.77
BasyAntmane 4-1.38
49
48.2
PT-DITAE-LLO
44.6
43.4
43.9
PT-DITXL-LL1BB
41.6
Resolution 256
512
1024
Frames
48
144
192
240
Figure Comparison of video generation models terms of GFLOPs and GPU memory usage.
4.4 ALGORITHMIC EFFICIENCY COMPARISON
As discussed in Sec. 3.3 method effectively reduces complexity. In this section, further analyze the computationa advantages of PT-DiT in T2I and T2V tasks. For the T21 task, the results reported in Fig TO with related discussions referenced in Appendix. A.4
In the video generation task, we assess our model based from two aspects: computational complexity and GPU memory consumption, illustrated in Fig. We conduct experiments using two scales of PT-DiT (i.e., PT-DiT/H (1.8B) for consistent scale comparison and our utilized PT-DiT/XL (I.1B) for training PT-T2V) and select the latest open-source T2V model (i.e., Cog X2 (actual test at 1.7B) and Easy Animate V4 (1.5B)) as the comparison methods. The left side of Fig. 7 displays the GFL.Oss calculated at different resolutions, with the latent code set to a time dimension of 48. It is obvious that, despite having the largest number of parameters, PT-DiT/H exhibits the lowest computational complexity. Meanwhile, the computational complexity PT-T2V is only 50% that of Cog VideoX and Easy Animatee, On the right side of Fig, 7 we further compare the GPU memory usage during training with Easy Animate V4 at resolution of 512, across different frame counts. Since the T2V version of Easy Animate V4 employs HunyuanDiT with full 3D attention, its memory consumption increases dramatically with the number of video frames. In contrast, PT-DiT, which also utilizes 3D spatial-temporal modeling, experiences only slight in- crease memory consumption due to its well-designed proxy- rroxy-tokenized attention mechanism. The above experiments demonstrate the potential of PT-DiT for generating longer and higher-resolution videos.
PT-DiT/XL employed by
Table 2: Ablation study PT-DiT/S-Class. Models are trained for 400k iterations.
(a) Major component. (b) Proxy token extraction. (c) Global information injection. (d) Compressed ratio.
Structure
FID-5Ok]
Method
FID-50k]
Method
FID-50k]
Ratio
FID-50kJ
w/o GIIM 23.71
Average
19.30
Cross-Attention
19.30
1, 2
19.30
w/o SWA 23.59
Top-Left
20.84
Interpolate
21.82
1,4,4
21.24
w/o TCM
69.07
Random
21.00
Linear
20.24
1, 8,
20.43

---

Published as a conference paper at ICLR 2025
4.5 ABLATION STUDY
Major Component. We conduct quantitative experiments to assess the effectiveness of the GIIM and TCM proposed in this paper. The absence of either GIIM or TCM results in substantial perfor- mance loss (i.e., 19.30 - 23.71 or 19.30 - 69.07). Specifically, without TCM, the model struggles to capture fine details, making it challenging meet generation tasks that demand high-quality detail, leading significant decline in perfor- mance. Additionally, we investigated the role shift-window attention through both qualitative evaluation at resolution of 512 and quantitative analyses resolution of 256, illustrated Fig. 8 and Table 2(a) respectively. As antici- pated, there is a noticeable decrease (i.e., 19.30 - 23.59) in FID without shift-window attention, accompanied by pronounced "grid" phenomena.
Artiloot
Wilhaut Shift- Atlention
With Shalh winzoww Allettion
Figure 8: Ablation shift-window attention.
Proxy Token Extraction. As illustrated in Table 2(b) we explore three methods for obtaining the proxy token: the top-left token, a randomly selected token, and averaging the in-window tokens. A performance gap exists between the Top-Left (20.84) Random (21.00) selections and the av- eraging manner (19.30). We believe this gap arises because the random and top-left tokens fail to adequately represent the overall characteristies of the region, compromising the effectiveness of proxy-tokenized attention and leading to performance loss. We use averaging as the default setting.
Global Information Injection. Due to the misalignment between the number of proxy tokens and latent tokens, we investigate three schemes for injecting global information into latent tokens: Cross- Attention, Interpolation, and Linear projection, shown in Table [(c) Among these, interpolation involves applying spatially bilinear interpolation to the proxy tokens, while linear projection aligns proxy tokens with latent tokens through linear layer. Since each latent code can leverage global information from the entire set of proxy tokens, Cross-attes attsentttnnionn achieves performance advantage with an FID of 19.30 compared to Linear projection 20.24 and Interpolation 21.82.
Compressed Ratio. As reported in Table 2(d) we examine the impact of compression ratio on performance at resolution of 256. It is evident that when the compression ratio high, the repre- sentative token fails to adequately capture the features of the region for effective global modeling, leading to a noticeable decline performance (i.e., from 19.30 to 21.24 at (1, 4, 4)).
5 CONCLUSION
Given the sparsity and redundancy of visual information, this paper proposes PT-DiT, which lever- ages the proxy-tokenized attention mechanism to mitigate the computational redundancy of self- attention in diffusion transformers. A series of representative tokens are calculated based on tem- poral and spatial priors, with global interactions between them. Additionally, window attention and shifted window attention are introduced to refine the modeling of local details. Our proposed rep- resentative token mechanism particularly effective for video tasks with redundant information, enabling 3D spatio-temporal modeling while avoiding an explosion in computational complexity. Experiments demonstrates that PT-DiT achieves competitive performance while delivering signifi- cant efficiency. We further develope the PT-T2X series based on PT-DiT, including models like T2I, T2V, and T2MV. We hope PT-DiT and PT-T2I/V can provide new insights and references for the field of diffusion transformers.
ACKNOWLEDGEMENT
This work was supported by National Key Research and Development Program China (202 National Natural Science Foundation of China (NSFC) under Grants 0o.62232482 and 62466226,,, Nansha Key R&D Program under Grant N0.2022Z0014, and Gen- eral Embodied Al Center of Sun Yat-sen University.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings the IEEE/CVF international conference on computer vision, pp. 1728-1738, 2021.
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22669-22679 2023.
Omer Bar- Bar-Tal, Hila Chefer, Omer Tov, Charles Hermmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprini 1224::000:::211 2024.
BlackForestlabs AL Flux. https:://blackfooesssstst http://laabbaa/tttt---eeeee flux, 2024. Accessed:
2024-09-03.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint
2023.
Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 4599-4603, 2023.
Daniel Bolya, Cheng Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.0944 2022.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, al. Pixart-a: Fast training of diffusion transformer for photore- alistic text-to-image synthesis. arXiv preprint 10Xiv:2310000426 2023.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint urXiv:2403.0400% 2024a.
Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixart-6: Fast and controllable image generation with latent consistency models. arXiv preprint arXiv:2401.05252. 2024b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. Ieee, 2009.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
Jiasong Feng, Ao Ma, Jing Wang, Bo Cheng, Xiaodan Liang, Dawei Leng, and Yuhui Yin. Fan- cyvideo: Towards dynamic and consistent video generation via cross-frame textual guidance. arXiv preprint arXiv:2408. 08189, 2024.
Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, al. Lumina-12x: Transforming text into any modality, resolution, and duration via flow-base large diffusion transformers. arXiv preprint arXiv:2405.05945 2024.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang. Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. arXiv preprint arXiv:2307.04775. 2023.
Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Shiji Song, and Gao Huang. Agent attention: On the integration of softmax and linear attention. arXiv preprint arXiv:23 /2,08774 2023.
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1): (1000,81 2022.

---

Published as a conference paper at ICLR 2025
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Huawei. Mindspeed-mm.
wwwiskuu. 2024. Accessed: 2024-10-
16.
Chaoya Jiang, Haiyang Xu, Chenliang Li, Ming Yan, Wei Ye, Shikun Zhang, Bin Bi, and Songfang Huang. Trips: Efficient aaiionaandlanngugee pre-training with text-relevant image patch selection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4084-4096, 2022.
Rajat Koner, Gagan Jain, Prateek Jain, Volker Tresp, and Sujoy Paul. Lookupvit: Compressing visual information to a limited number of tokens. arXiv preprint arXiv:2407.1275. 2024.
PKU- Yuan Lab and Tuzhan Al etc. Open-sora-plan, April 2024. URL
5281/zenoda 10948109
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, Dayou Chen, Jiajun He, Jiahao Li, Wenyue Li, Chen Zhang, Rongwei Quan, Jianxiang Lu, Jiabin Huang, Xiaoyan Yuan, Xiaoxiao Zheng, Yixuan Li, Jihong Zhang, Chao Zhang, Meng Chen, Jie Liu, Zheng Fang, Weiyan Wang, Jinbao Xue, Yangyu Tao, Jianchen Zhu, Kai Liu, Sihuan Lin, Yifu Sun, Yun Li, Dongdong Wang, Mingtao Chen, Zhichao Hu, Xiao Xiao, Yan Chen, Yuhong Liu, Wei Liu, Di Wang, Yong Yang, Jie Jiang, and Qinglin Lu. Hunyuan-dit: powerful multi-resolution diffusion transformer with fine-grained chinese understanding, 2024a.
Zhimin Li, Jianwei Zhang. Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint arXiv:2405.08748, 2024b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland. September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer. 2014.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.027447 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding. Vdt: General-purpose video diffusion transformers via mask modeling. arXiv preprint arXiv:2305.13311 2023.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden- Vanden-Eijnden, and Sain- ing Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.0874h1 2024a.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03044 2024b.
OpenAL. Sora. /ttep:://ppeaa..co 2024. Accessed: 2024-09-03.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4205, 2023.
Reiner Pope. Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606-624, 2023.
12

---

Published as a conference paper at ICLR 2025
Yifan Pu, Zhuofan Xia, Jiayi Guo, Dongchen Han, Qixiu Li, Duo Li, Yuhui Yuan, Ji Li, Yizeng Han, Shiji Song, et al. Efficient diffusion transformer with step-wise dynamic attention mediators. arXiv preprint arXiv:2408.0577 2024.
Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9914-9925, 2024.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint (xXi::2204.06222 1(2):3 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-36499 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Muliis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278- 2022.
Shengshu AL Vidu. https: //ww:///wwwwww wwtp::ividuuu... www.. 2024. Accessed: 2024-09-03.
Dachuan Shi, Chaofan Tao, Anyi Rao, Zhendong Yang, Chun Yuan, and Jiaqi Wang. Crossget: Cross-guided ensemble of tokens for accelerating vision-language transformers. arXiv preprint arXiv:2305. 17455, 2023a.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long. Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023b.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2(11))1-7, 2012.
Stability AI. Stablediffusion3. 2024. Accessed: 2024-09-03.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568:127063 2024.
Kolors Team. Kolors: Effective training diffusion model for photorealistic text-to-image synthe-
sis. arXiv preprint, 2024.
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint 1rXiv:2310.12190, 2023.
Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang. Easyanimate: nigh-performance long video generation method based on transformer architecture. arXiv preprint arXiv:2405.188 2024.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: large video description dataset for bridging video and language. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 9288-5296, 2016.
Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXtv:2408.06077 2024.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. arXiv preprint arXiv:2406.07550 2024.
Yan Zeng. Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8850-8860, 2024.
Biao Zhang and Rico Sennrich, Root mean square layer normalization. Advances in Neural Infor- mation Processing Systems, 32, 2019.
Zangwei Zheng, Xiangyu Peng. Tianji Yang. Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024. [RL /ttp::://qqthas cww///ncaaatecchh/eeeeeaaaaa
Le Zhuo, Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongjie Huang, Wenze Liu, Lirui Zhao, Yun Wang. Zhanyu Ma, et al. Lumina-next: Making lumina-12x stronger and faster with next-dit. arXiv preprint arXiv:2406.18583 2024.
Qi Zuo, Xiaodong Gu, Lingteng Qiu, Yuan Dong, Zhengyi Zhao, Weihao Yuan, Rui Peng. Siyu Zhu, Zilong Dong, Liefeng Bo, et al. Videomy: Consistent multi-view generation based on large video generative model. arXiv preprint 220.:0000..10000, 2024.
14

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 RELATED WORK
Image Generation with Diffusion Transformer. Recent studies (Peebles & Xie 2023 Ma al.
2024a; Li et al. 2024b/a) a) have demonstrated the potential of using the Vision Transformer (ViT)
(Han et al. [022 as an alternative backbone for image generation. U-VIT (Bao
2023) encodes
the condition as tokens and incorporates skip connections inspired by U-Net, achieving excellent performance. DiT (Peebles & Xie 2023) introduces AdaLN integrate conditions and analyzes
the scalability, complexity, and performance of VIT in comparison to U-Net. SiT (Ma et al., 2024a)
adds an interpolant framework to DiT, achieving even better scores on ImageNet (Deng et al. 2009).
PixArt-a (Chen al.) (023) integrates cross-altention modules into DiT to inject text conditions and optimize the class-conditional branch. Flag-Di7 (Gao et al. 2024) and Next-DiT (Zhuo et al. 2024)
employ advanced techniques like RoPE (Su al. 2024), RMSNorm (Zhang & Sennrich] 2019),
and flow matching (Lipman et al. 2022) to enhance stability, and they use zero-initialized attention to incorporate complex text instructions. Although the effectiveness of transformers in diffusion models has been validated, the substantial computational and spatial complexities of these models still need to be addressed. DAM (Pu al. 2024) has drawn attention the redundancy within DiT and uses the mediator tokens to directly proxy the query and key in the attention operation, breaking down the process into two distinct attention calculations. We propose PT-DiT, method that employs a more gentle strategy for compressing hidden states. This approach not only reduces redundancy but also guarantees minimal loss of information during the compression process, thereby substantially decreasing the computational complexity.
A.2 TRAINING DETAIL AND MODEL CONFIGURATION
We collect total of 50M data points for the training set, including 32M images with aesthetic score 5.5 or higher from Laion (Schuhmann et al. 2022) and 18M high-resolution, high-quality datasets that constructed. During the high-resolution training phase, exclusively use 18M high-quality data. We train PT-T2V by progressing through three stages starting from stage 1 of PT-T21, with detailed hyper-parameters shown in Table The WebVid 10M (Bain al. 2021) dataset employed as the 256-resolution video training data. Additionally, we collect 3M high- resolution, high-quality video samples from the Internet to train the high-resolution video generator. The training objective for PT-T2I/V is v-prediction, with an extracted text token length of 120. During the inference phase, the denoising steps are set 50, and the scale of classifier-free guidance is set to 6.0. The specific parameter configurations for various scales of PT-DiT are presented in Table 4
Table 3: The training setups of PT-T2I and PT-T2V
Text-to-Image
Text-to- Video
Resolution
Data
Learning Rate
Butch Size lerattin Resolution Frume
Data
Learning Rate Batch Size lteration
256
50M
2e-5
10240
100k
18M HQ
1024
18M HQ
2e-5
512
50k
512 # 96
3M HQ
2e.5
256
50k
Table 4: The model configurations for various scales of PT-DiT.
Model
Layers
Hidden Dim
Head Number
Param, (M)
PT--IITS--Can
10
288
6
PTTD Class
12
576
8
154
PT-DIT/B
12
640
10
144
PT-DITVL
28
864
12
605
PT.DIT/XI.
16
PT-DIT/H
30
1440
20
1795
A.3 PERFORMANCE AND EFFICIENCY COMPARISON
15

---

Published as a conference paper at ICLR 2025
FID-50K
To fairly compare the performance of PT-DiT with DAM and DiT, we conduct the widely used class-to-image experiments
70
DIFS
ImageNet 256.
.2.54PID
Since DAM is trained on the SiT codebase, we utilize PT-DiTYS-Clas
66
PI-DIIS-CClss
align with DiT-S, SiT-
S. and DAM-S, using two different codebases (DiT-based and SiT-based) with the same param- eters, training data, training steps (400k), and ex- perimental configurations. The detailed model
60
SIF-S
37% reduction
55
settings are provided in Table. 4 and the re- sults are shown the Fig. 9 with the data re- ported without the use of Classifier-free guidance (CFG).
PT-DIT/S- Class (SIT cocb))
DAMS
GFLOPs
50
6.5
7.5
Figure 9: Performance and efficiency compari- son ImageNet at resolution of 256.
Compared to DiT and SIT, PT-DiT achieves com- petitive performance and a significant efficiency advantage, thanks to the well-designed proxy to- ken mechanism and texture refinement module. For instance, PT-DiT/S TT-Dit/S-Class achieves the FID of 65,86 (1 2.54 FID) and 55.05 (L 3.56 FID), outperforming DĪT-S (68 40 FID) and SiT (68.61 FID). Meanwhile, the computational complexity is reduced by 37% (3.8 6.1 GFLOPs). These results demonstrate that our method can maintain competitive performance while benefiting from reduced computational complexity. This advantage is primarily due the window-attention. which aligns with visual priors and models the spatial neighboring token, as well as the efficiency gains from the proxy token mechanism. Furthermore, when compared DAM, PT-DiT achieves similar per- formance while offering significant reduction in computational complexity (i.e., 3.8 GFLOPs for PT-DiT vs. 5.8 GFLOPs for DAM).
A.4 ALGORITHMIC EFFICIENCY COMPARISON
conduct comparisons at different parameter scales. With equivalent parameter counts, we compared Lumina-Next (1.7B) to our PT-DiT/H (1.8B), DiT/B (0.13B) and DAM/B (0.13B) our PT-DiT/B-Clan 14.) illustrated in Fig. 10 As shown on the left side of Fig. TO the GFLOPs of PT-DiT/H are significantly lower than Lumina-Next across multiple scales. Specifically, resolutions of 512 and 2048, PT-DiT/H achieves complexity reduction of respectively 82.0% and 82.5%. Similarly, the right side of Fig. 10 indicates that PT-DiT/B- BlT/ssCllas requires 59.5% less computation than DiT/B resolution of 1024. Com-
In the image generation task, similar to Fig.
pared to DAM/B, which has an attention computation complexity of O(n), our method exhibits comparable level of computation complexity across all resolutions.
8030
L.umnee Next-11.1 1.7B
PT-D/T/H-1.88
96069.1
00
106.
650.5
8100
40.0
Complestyy 6G (GILLOPS
59.3%
82.0%
82.5%
80
68.63
360.
Compbsity
200 4000
1513.9
269.6
302
1089.1
24027
200
18.3
D77/--0130
DAM& 0.13
4236.9
16828.3
Resolution 256
1024
2048
Resolution
256
Figure 10: Comparison of image generation models in terms of GFLOPs.
A.5 PT-T2MV
Me. We further explore the effectiveness of PT-DiT on Texl-to-MultiVies (T2MV) tasks. The trained PT-T2MV is capable of generating 512 512 x 24 images from various viewpoints based on the provided text instruction, showcasing strong spatial consistency, as illustrated in Fig. II The detailed experimental and training setups are as follow.
Basic setting. Multi images of 3D objects can be interpreted as videos of static objects. We uti- lize subset of approximately 40k samples from G-Objaverse (Qiu al. 2024), following VideoMV
16

---

Published as a conference paper at ICLR 2025
[uo et al. 2024), which is rendered video data to train PT-T2MV model. Each object is ren- dered with a uniformly distributed azimuth from 0% to 360 and an elevation ranging from 5 to 30" resulting in 512 512 x 24 video.
Training setting. Following previous works Zuo et al. 2024 Shi et al. 2023b), we only accept text instruction input generate the Multi Multi-View images of 3D object without additional reference images and camera parameters. The PT-T2MV is trained from stage 2 of the PT-T2I, with a bacthsize of 128 and 20k iterations. The other hyperparameters and experimental settings are the same in PT-T2I.
Los duck aodd wia orage best and gremn
Poolitd Nize rotl eved with pélie landle
Pis weeartooce mocel zfe bart ard mive-rollath withwwhhee me descarda peeeoelltg light feom
Fix Nithi d.Focklyss yelan fux darattz:
Figure 11: Samples by PT-T2MV. It is important to note that PT-T2MV does not accept any image inputs camera parameters and relies solely on text prompts.
17


Reviews:
[{"review_id": "w0JpU2AZfe", "reviewer": "Reviewer_vCtb", "summary": "This paper proposes Proxy-Tokenized Diffusion Transformer to reduce the computational complexity. By utilizing sparse representative token attention, window and shift window attention, the proposed method effectively reduces redundancy in global self-attention computation and optimizes the modeling process of visual information. Also, the Qihoo-T2X family includes multiple models for text to image, text to video, and text to multi view tasks. However, it is difficult to fully demonstrate the effectiveness of the method at various resolutions in the experimental section.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method significantly reduces computational complexity and has certain competitiveness in various tasks, such as T2I, T2V,and T2MV.", "weaknesses": "1)\tInsufficient experiments and lack of comparative experiments at high resolution.\n2)     How is the value of compression ratio determined, and is it related to both semantic information and resolution size?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DlItfKzvaE", "reviewer": "Reviewer_h1hT", "summary": "This paper introduces a proxy-tokenized mechanism for transformer-based diffusion models (e.g., DiT), utilizing sparse representative tokens to alleviate redundant computations arising from inherent visual redundancy. Specifically, the authors observe that tokens representing spatially proximate regions tend to exhibit high similarity, while tokens corresponding to spatially distant regions contribute less to meaningful computation in the attention map. Leveraging this insight, the authors propose the use of a proxy token for each window by averaging the tokens within it and subsequently employing self-attention and cross-attention operations involving both proxy tokens and original tokens to ensure efficient interaction across all tokens. Furthermore, the proposed approach integrates a Swin-based architecture to implement efficient attention mechanisms in accordance with these observations. The method achieves competitive performance compared to state-of-the-art approaches across three tasks—text-to-image, text-to-video, and text-to-multiview—at varying resolutions, underscoring its effectiveness and significance within the field.", "questions": "- Although the authors provide extensive comparisons between resolution and computational metrics (e.g., memory usage, GFLOPs), I believe that additional comparisons between performance and computational cost relative to other approaches in text-to-image and text-to-multiview tasks are necessary. Assessing the trade-off between performance and computational efficiency, rather than focusing solely on resolution, would offer a more comprehensive evaluation of the proposed method's effectiveness. Since identical resolutions do not inherently guarantee equivalent performance, it is conceivable that approximate methods may result in some performance degradation.\n\n- Several prior works, such as ToMe, ToMeSD, TRIPS, and CrossGET, have also addressed token merging. The absence of a detailed discussion or comparison with these existing studies undermines the claimed novelty of Qihoo-T2X. I suggest that the authors more explicitly delineate the unique contributions and advantages of their approach in relation to these established methods.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-crafted, providing a clear exposition of the proposed methodology.\n- The approach is conceptually straightforward and highly intuitive.\n- The motivation and objectives are innovative, focusing on a token-efficient mechanism for DiT-based generative models, which is particularly significant given the paucity of existing approaches for DiT-based models.", "weaknesses": "- The lack of ablation studies assessing performance metrics, such as FID with respect to GFLOPs, diminishes the practical relevance of the proposed methodology.\n- The discussion and comparison do not sufficiently address related works on token merging (e.g., ToMe [1], ToMeSD [2], CrossGET [3], and TRIPS [4]), which limits the comprehensiveness of the literature review.\n- The proposed method integrates a Swin-based architecture within the DiT-based model, employing a straightforward average operation to represent tokens. From the perspective of novelty, this contribution appears to be incremental.\n\n\n\n[1] Token Merging: Your ViT But Faster, ICLR 2023\n\n[2] Token Merging for Fast Stable Diffusion, CVPR Workshop 2023 \n\n[3] CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers, ICML 2024\n\n[4] TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection, EMNLP 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TzCUrEQTzc", "reviewer": "Reviewer_g2Xu", "summary": "The paper introduces an efficient Proxy-Tokenized Diffusion Transformer (PT-DiT) for accelerating self-attention mechanism, addressing the redundancy in global self-attention mechanisms of diffusion transformers. It develops Qihoo-T2X family for text-to-any-task applications. PT-DiT uses sparse representative token attention, computing an averaging token from each spatial-temporal window as a proxy, reducing computational complexity. It includes window and shift window attention to enhance detail modeling. The Qihoo-T2X family, which includes models for T2I, T2V, and T2MV tasks, shows competitive performance with reduced computational complexity in image and video generation tasks.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The PT-DiT addresses the redundancy in global self-attention by using sparse representative token attention, reducing computational complexity.\n\n2. The introduction of window and shift window attention mechanisms helps to capture fine details and textures, improving the quality of generated images and videos.\n\n3. The method is designed to be adaptable to both image and video generation tasks without structural adjustments, demonstrating competitive performance compared to original models and great flexibility in application\n\n4. This method is simple and effective.\n\n5. This paper is well written.", "weaknesses": "- Insufficent performance comparison to peer methods such as DAM [1] and AgentAttention [2]. While PT-DiT significantly outperforms baselines, its performance relative to peer methods like DAM and AgentAttention is not clearly superior (see Figure 8). Need more comparison.\n- **Need further discussion on the relation and differences to previous methods.**\n    - For example, DAM **also finds significant attention map redundancy** by analyzing the Divergence Score and proposes a set of Attention Mediator Tokens. These tokens can also **be seen as a kind of extractor** for extracting information from the original latent codes (Queries and Keys).\n    - From this perspective, **based on DAM, PT-DiT only points out that redundancy in attention maps remains due to spatial reasons**, and the proposed average operation and sparse representative token attention may be implicitly included by the interaction between the original Queries/Keys and the Attention Mediator Tokens in DAM.\n    - Additionally, PT-DiT doesn't analyze the significance of the observed phenomenon across different denoising timesteps, as DAM does.\n- From the ablation study in Table 2(a), the results show that without TCM (an auxiliary module), the performance drops significantly, whereas without GIIM, which is the core component proposed in the paper, the performance decline is much less. Does this imply that auxiliary module TCM is more important than core module GIIM?\n- A statistical comparison between Qihoo-T2V with and without PT-DiT is necessary for clearer insights into the model's impact.\n\n[1] Pu Y, Xia Z, Guo J, et al. Efficient diffusion transformer with step-wise dynamic attention mediators[J]. arXiv preprint arXiv:2408.05710, 2024.\n\n[2] Han D, Ye T, Han Y, et al. Agent attention: On the integration of softmax and linear attention[J]. arXiv preprint arXiv:2312.08874, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CQVhwMx8Qn", "reviewer": "Reviewer_wrGZ", "summary": "This paper introduces the Qihoo-T2X series, featuring the Proxy-Tokenized Diffusion Transformer (PT-DiT) for efficient text-to-video generation. PT-DiT uses proxy-tokenized attention to reduce computational redundancy by selecting representative tokens within spatial-temporal windows, enhancing global information capture without the high complexity of full self-attention. This design integrates global semantics efficiently through sparse proxy tokens and refines texture details using window and shifted-window attention, similar to the Swin Transformer. The Qihoo-T2X family demonstrates competitive performance with reduced complexity compared to models like DiT and PixArt-α.", "questions": "About method section: \n-  l201-203, f, h, w are not the actual frame, height, and width of the video/image, but the latent, right? Also, for images, how does the 3D VAE, with temporal downampling layers, handle them? \n- the texts from l242 to l244 are confusing, $(p_f \\cdot p_f, p_h)$ should be ($p_f, p_h, p_w$)? so we have $p_h = p_w$ = <some value depends on resolution> and $p_f$ = 1 for image and $p_f$ = 4 for videos, right?\n- Details for the window attention and shifting window attention are missing. Only Figure 4 shows some fundamental ideas of them but how do they work exactly? How does the shifting window work?\n\nAbout T2MV: \n- Why Text-to-MV task is interesting? At the training level, there is no difference from this task to Text-to-video unless the training data; while for real practice, it's hard to control the generated content from only texts without any reference image. \n- It's hard to evaluate the performance of T2V-MV. Can the T2V-MV work for out-of-distribution prompts? And how about the visual and quantitative evaluations with other models, given that the task setting is following previous approaches? \n\nOther questions:\n- The page limit is 10, no need to move related work to the appendix. \n- Can we use a VAE with a higher compression ratio, or a large patch embedding to achieve a similar level of efficiency? In principle, with the proposed design, more details could be restored compared with directly using a higher compression ratio (but with vanilla attention). It would also be great to show that.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. \n- The visualizations are promising. \n- Compared with SOTA methods, the model can achieve a similar level of performance with less training data. Meanwhile, the inference cost is much lower.", "weaknesses": "- The method section is not clear enough. The mechanisms of window attention and shifting window attention are still not clear.  \n- The task of T2MV is a little strange. What would be an application scenario for that task?  \n- The \"text-to-any-task\" is overclaimed. The focused three tasks are essentially video generation (with a single frame or multi-view video).  \n\nSee the next question sections for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6XJikXFGyi", "reviewer": "Reviewer_q44Z", "summary": "This paper first highlights the computational redundancy issue of global self-attention in diffusion transformers, stemming from sparse visual information, through an analysis of the attention map. To tackle this problem, the authors propose the Proxy-tokenized Diffusion Transformer (PT-DiT), which substitutes the global self-attention of the vanilla DiT with Global Information Interaction Module (GIIM) and Texture Complement Module (TCM), enabling efficient modeling of both global and local visual information. In experiments, PT-DiT demonstrates theoretical and experimental advantages in computational complexity and shows competitive performance across various tasks, including T2I, T2V, and T2MV, underscoring its generalization capabilities.", "questions": "-What accounts for PT-DiT's 80% improvement over Lumina in Figure 8 (left), especially since this enhancement is significantly greater than the improvements observed for other models?\n\n-How about the training time and inference time of PT-DiT?\n\n-Does the author have any plans to open source the Qihoo-T2X series of models and code?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "-The motivation for reducing the training and inference costs of DiT by minimizing redundant computation in global self-attention is clear.\n\n-The GIIM and TCM design in PT-DiT is promising and straightforward to implement.\n\n-Qualitative and quantitative experiments on a variety of tasks (e.g., Text-to-Image, Text-to-Video, and Text-to-MultiView) are comprehensive and convincing.", "weaknesses": "-This article lacks results on ImageNet for a fair comparison with existing state-of-the-art diffusion transformer models. Previous studies, such as DiT, MDT, VAR, and SiT, have all been evaluated on ImageNet using metrics like FID and IS. To fully establish its effectiveness, PT-DiT should also be assessed under similar conditions. This is essential for demonstrating its competitive performance despite reduced computational complexity and should be included in evaluations.\n\n-How is the hyper-parameter compression ratio determined in this paper? Why is the compressed ratios set to (1, 2, 2) at a resolution of 256, and will a ratio of (1, 16, 16) result in excessive information loss at a resolution of 1024? The authors should provide further clarification on the rationale and motivations behind the chosen compressed ratios settings in their methodology.\n\n-The extraction of proxy tokens through averaging is straightforward, but how representative are these proxy tokens when obtained in this manner? Could proxy tokens also be extracted using clustering or similarity-based methods for potentially improved representativeness?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)