PT-T2I/V: An Efficient Proxy-Tokenized Diffusion Transformer for Text-to-Image/Video-Task

Authors: Jing Wang, Ao Ma, Jiasong Feng, Dawei Leng, Yuhui Yin, Xiaodan Liang
Primary area: generative models
Keywords: Diffusion Transformer, Image Generation, Video Generation

Abstract:
The global self-attention mechanism in diffusion transformers involves redundant computation due to the sparse and redundant nature of visual information, and the attention map of tokens within a spatial window shows significant similarity. To address this redundancy, we propose the Proxy-Tokenized Diffusion Transformer (PT-DiT), which employs sparse representative token attention (where the number of representative tokens is much smaller than the total number of tokens) to efficiently model global visual information. Specifically, within each transformer block, we compute an averaging token from each spatial-temporal window to serve as a proxy token for that region. The global semantics are captured through the self-attention of these proxy tokens and then injected into all latent tokens via cross-attention. Simultaneously, we introduce window and shift window attention to address the limitations in detail modeling caused by the sparse attention mechanism. Building on the well-designed PT-DiT, we further develop the PT-T2I/V family, which includes a variety of models for T2I, T2V, and T2MV tasks. Experimental results show that PT-DiT achieves competitive performance while reducing computational complexity in image and video generation tasks (e.g., a reduction 59\% compared to DiT and a reduction 34\% compared to PixArt-$\alpha$). The visual exhibition of and code are available at https://360cvgroup.github.io/Qihoo-T2X/.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730194892123

thread_type:
official_review

<thread_object>
creator: reviewer_q44z

<content>
summary:
This paper first highlights the computational redundancy issue of global self-attention in diffusion transformers, stemming from sparse visual information, through an analysis of the attention map. To tackle this problem, the authors propose the Proxy-tokenized Diffusion Transformer (PT-DiT), which substitutes the global self-attention of the vanilla DiT with Global Information Interaction Module (GIIM) and Texture Complement Module (TCM), enabling efficient modeling of both global and local visual information. In experiments, PT-DiT demonstrates theoretical and experimental advantages in computational complexity and shows competitive performance across various tasks, including T2I, T2V, and T2MV, underscoring its generalization capabilities.

strengths:
-The motivation for reducing the training and inference costs of DiT by minimizing redundant computation in global self-attention is clear.

-The GIIM and TCM design in PT-DiT is promising and straightforward to implement.

-Qualitative and quantitative experiments on a variety of tasks (e.g., Text-to-Image, Text-to-Video, and Text-to-MultiView) are comprehensive and convincing.

weaknesses:
-This article lacks results on ImageNet for a fair comparison with existing state-of-the-art diffusion transformer models. Previous studies, such as DiT, MDT, VAR, and SiT, have all been evaluated on ImageNet using metrics like FID and IS. To fully establish its effectiveness, PT-DiT should also be assessed under similar conditions. This is essential for demonstrating its competitive performance despite reduced computational complexity and should be included in evaluations.

-How is the hyper-parameter compression ratio determined in this paper? Why is the compressed ratios set to (1, 2, 2) at a resolution of 256, and will a ratio of (1, 16, 16) result in excessive information loss at a resolution of 1024? The authors should provide further clarification on the rationale and motivations behind the chosen compressed ratios settings in their methodology.

-The extraction of proxy tokens through averaging is straightforward, but how representative are these proxy tokens when obtained in this manner? Could proxy tokens also be extracted using clustering or similarity-based methods for potentially improved representativeness?

questions:
-What accounts for PT-DiT's 80% improvement over Lumina in Figure 8 (left), especially since this enhancement is significantly greater than the improvements observed for other models?

-How about the training time and inference time of PT-DiT?

-Does the author have any plans to open source the Qihoo-T2X series of models and code?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730663947977

thread_type:
official_review

<thread_object>
creator: reviewer_wrgz

<content>
summary:
This paper introduces the Qihoo-T2X series, featuring the Proxy-Tokenized Diffusion Transformer (PT-DiT) for efficient text-to-video generation. PT-DiT uses proxy-tokenized attention to reduce computational redundancy by selecting representative tokens within spatial-temporal windows, enhancing global information capture without the high complexity of full self-attention. This design integrates global semantics efficiently through sparse proxy tokens and refines texture details using window and shifted-window attention, similar to the Swin Transformer. The Qihoo-T2X family demonstrates competitive performance with reduced complexity compared to models like DiT and PixArt-α.

strengths:
- The paper is well-written and easy to follow. 
- The visualizations are promising. 
- Compared with SOTA methods, the model can achieve a similar level of performance with less training data. Meanwhile, the inference cost is much lower.

weaknesses:
- The method section is not clear enough. The mechanisms of window attention and shifting window attention are still not clear.  
- The task of T2MV is a little strange. What would be an application scenario for that task?  
- The "text-to-any-task" is overclaimed. The focused three tasks are essentially video generation (with a single frame or multi-view video).  

See the next question sections for more details.

questions:
About method section: 
-  l201-203, f, h, w are not the actual frame, height, and width of the video/image, but the latent, right? Also, for images, how does the 3D VAE, with temporal downampling layers, handle them? 
- the texts from l242 to l244 are confusing, $(p_f \cdot p_f, p_h)$ should be ($p_f, p_h, p_w$)? so we have $p_h = p_w$ = <some value depends on resolution> and $p_f$ = 1 for image and $p_f$ = 4 for videos, right?
- Details for the window attention and shifting window attention are missing. Only Figure 4 shows some fundamental ideas of them but how do they work exactly? How does the shifting window work?

About T2MV: 
- Why Text-to-MV task is interesting? At the training level, there is no difference from this task to Text-to-video unless the training data; while for real practice, it's hard to control the generated content from only texts without any reference image. 
- It's hard to evaluate the performance of T2V-MV. Can the T2V-MV work for out-of-distribution prompts? And how about the visual and quantitative evaluations with other models, given that the task setting is following previous approaches? 

Other questions:
- The page limit is 10, no need to move related work to the appendix. 
- Can we use a VAE with a higher compression ratio, or a large patch embedding to achieve a similar level of efficiency? In principle, with the proposed design, more details could be restored compared with directly using a higher compression ratio (but with vanilla attention). It would also be great to show that.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704842429

thread_type:
official_review

<thread_object>
creator: reviewer_g2xu

<content>
summary:
The paper introduces an efficient Proxy-Tokenized Diffusion Transformer (PT-DiT) for accelerating self-attention mechanism, addressing the redundancy in global self-attention mechanisms of diffusion transformers. It develops Qihoo-T2X family for text-to-any-task applications. PT-DiT uses sparse representative token attention, computing an averaging token from each spatial-temporal window as a proxy, reducing computational complexity. It includes window and shift window attention to enhance detail modeling. The Qihoo-T2X family, which includes models for T2I, T2V, and T2MV tasks, shows competitive performance with reduced computational complexity in image and video generation tasks.

strengths:
1. The PT-DiT addresses the redundancy in global self-attention by using sparse representative token attention, reducing computational complexity.

2. The introduction of window and shift window attention mechanisms helps to capture fine details and textures, improving the quality of generated images and videos.

3. The method is designed to be adaptable to both image and video generation tasks without structural adjustments, demonstrating competitive performance compared to original models and great flexibility in application

4. This method is simple and effective.

5. This paper is well written.

weaknesses:
- Insufficent performance comparison to peer methods such as DAM [1] and AgentAttention [2]. While PT-DiT significantly outperforms baselines, its performance relative to peer methods like DAM and AgentAttention is not clearly superior (see Figure 8). Need more comparison.
- **Need further discussion on the relation and differences to previous methods.**
    - For example, DAM **also finds significant attention map redundancy** by analyzing the Divergence Score and proposes a set of Attention Mediator Tokens. These tokens can also **be seen as a kind of extractor** for extracting information from the original latent codes (Queries and Keys).
    - From this perspective, **based on DAM, PT-DiT only points out that redundancy in attention maps remains due to spatial reasons**, and the proposed average operation and sparse representative token attention may be implicitly included by the interaction between the original Queries/Keys and the Attention Mediator Tokens in DAM.
    - Additionally, PT-DiT doesn't analyze the significance of the observed phenomenon across different denoising timesteps, as DAM does.
- From the ablation study in Table 2(a), the results show that without TCM (an auxiliary module), the performance drops significantly, whereas without GIIM, which is the core component proposed in the paper, the performance decline is much less. Does this imply that auxiliary module TCM is more important than core module GIIM?
- A statistical comparison between Qihoo-T2V with and without PT-DiT is necessary for clearer insights into the model's impact.

[1] Pu Y, Xia Z, Guo J, et al. Efficient diffusion transformer with step-wise dynamic attention mediators[J]. arXiv preprint arXiv:2408.05710, 2024.

[2] Han D, Ye T, Han Y, et al. Agent attention: On the integration of softmax and linear attention[J]. arXiv preprint arXiv:2312.08874, 2023.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705597204

thread_type:
official_review

<thread_object>
creator: reviewer_h1ht

<content>
summary:
This paper introduces a proxy-tokenized mechanism for transformer-based diffusion models (e.g., DiT), utilizing sparse representative tokens to alleviate redundant computations arising from inherent visual redundancy. Specifically, the authors observe that tokens representing spatially proximate regions tend to exhibit high similarity, while tokens corresponding to spatially distant regions contribute less to meaningful computation in the attention map. Leveraging this insight, the authors propose the use of a proxy token for each window by averaging the tokens within it and subsequently employing self-attention and cross-attention operations involving both proxy tokens and original tokens to ensure efficient interaction across all tokens. Furthermore, the proposed approach integrates a Swin-based architecture to implement efficient attention mechanisms in accordance with these observations. The method achieves competitive performance compared to state-of-the-art approaches across three tasks—text-to-image, text-to-video, and text-to-multiview—at varying resolutions, underscoring its effectiveness and significance within the field.

strengths:
- The paper is well-crafted, providing a clear exposition of the proposed methodology.
- The approach is conceptually straightforward and highly intuitive.
- The motivation and objectives are innovative, focusing on a token-efficient mechanism for DiT-based generative models, which is particularly significant given the paucity of existing approaches for DiT-based models.

weaknesses:
- The lack of ablation studies assessing performance metrics, such as FID with respect to GFLOPs, diminishes the practical relevance of the proposed methodology.
- The discussion and comparison do not sufficiently address related works on token merging (e.g., ToMe [1], ToMeSD [2], CrossGET [3], and TRIPS [4]), which limits the comprehensiveness of the literature review.
- The proposed method integrates a Swin-based architecture within the DiT-based model, employing a straightforward average operation to represent tokens. From the perspective of novelty, this contribution appears to be incremental.



[1] Token Merging: Your ViT But Faster, ICLR 2023

[2] Token Merging for Fast Stable Diffusion, CVPR Workshop 2023 

[3] CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers, ICML 2024

[4] TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection, EMNLP 2022

questions:
- Although the authors provide extensive comparisons between resolution and computational metrics (e.g., memory usage, GFLOPs), I believe that additional comparisons between performance and computational cost relative to other approaches in text-to-image and text-to-multiview tasks are necessary. Assessing the trade-off between performance and computational efficiency, rather than focusing solely on resolution, would offer a more comprehensive evaluation of the proposed method's effectiveness. Since identical resolutions do not inherently guarantee equivalent performance, it is conceivable that approximate methods may result in some performance degradation.

- Several prior works, such as ToMe, ToMeSD, TRIPS, and CrossGET, have also addressed token merging. The absence of a detailed discussion or comparison with these existing studies undermines the claimed novelty of Qihoo-T2X. I suggest that the authors more explicitly delineate the unique contributions and advantages of their approach in relation to these established methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731066864460

thread_type:
official_review

<thread_object>
creator: reviewer_vctb

<content>
summary:
This paper proposes Proxy-Tokenized Diffusion Transformer to reduce the computational complexity. By utilizing sparse representative token attention, window and shift window attention, the proposed method effectively reduces redundancy in global self-attention computation and optimizes the modeling process of visual information. Also, the Qihoo-T2X family includes multiple models for text to image, text to video, and text to multi view tasks. However, it is difficult to fully demonstrate the effectiveness of the method at various resolutions in the experimental section.

strengths:
The proposed method significantly reduces computational complexity and has certain competitiveness in various tasks, such as T2I, T2V,and T2MV.

weaknesses:
1)	Insufficient experiments and lack of comparative experiments at high resolution.
2)     How is the value of compression ratio determined, and is it related to both semantic information and resolution size?

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732631025140

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comment by Authors

comment:
Dear reviewers,

We sincerely thank the reviewers for their thoughtful and thorough review, as well as for their praise regarding the clarity of our writing (**h1Ht**, **g2Xu**, **wrGZ**), the motivation behind our study (**h1Ht**, **g2Xu**, **q44Z**), the innovativeness of the proposed methodology (**h1Ht**, **g2Xu**, **q44Z**), and the diversity of our experiments and analyses (**vCtb**, **g2Xu**, **wrGZ**, **q44Z**).
_____
We would also like to express our sincere gratitude for taking the time to review our paper and for offering constructive and valuable feedback to help improve it. We have updated the manuscript to address the points you raised, with the revised sections highlighted in red. Below is a summary of the major revisions:

1.	Add a brief discussion of existing methods in related work.

2.	Modify name of the T2I and T2V models and title of the paper.

3.	Add a description of this paper's contribution to introduction.

4.	Add explanations about window-attention and shift window attention in Sec. 3.2.2 (Texture Complement Module).

5.	Add the process of determining the compression ratio in Sec. 3.2.3 (Compression Ratios).

6.	Correct data in Fig. 8 (now Fig. 10).

7.	Add a comprehensive integrated comparison of performance and efficiency in Appendix. 2.

8.	Fix some writing errors and revise the paper layout.

Authors of Submission715
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) The paper proposes the Proxy-Tokenized Diffusion Transformer to reduce computational complexity by using sparse representative token attention and window-based attention. The Qihoo-T2X family includes models for text-to-image, text-to-video, and text-to-multi-view tasks.

(b) The strengths of the paper are: The paper effectively reduces training and inference costs of DiT by minimizing redundant computation through the promising GIIM and TCM designs, demonstrating strong performance across tasks (Text-to-Image, Text-to-Video, Text-to-MultiView) with comprehensive experiments, clear visualizations, and lower data and computational requirements compared to SOTA methods.

(c) The weaknesses of the paper are: unclear method part, overclaimed "text-to-any-task", lack of ImageNet comparisons, hyper-parameter ablation, the mechanism of proxy tokens, lack of related works on token merging and lack of comparative experiments at high resolution.

(d) The most important reasons for acceptance are that this work introduces the sparse representative token attention to model global visual information efficiently, whilst makes solid contributions by introducing a variety of models for a series of  T2I, T2V, and T2MV tasks.