Qihoo-T2X: An Efficient Proxy-Tokenized Diffusion Transformer for Text-to-Any-Task

## Abstract

Abstract The global self-attention mechanism in diffusion transformers involves redundant computation due to the sparse and redundant nature of visual information, and the attention map of tokens within a spatial window shows significant similarity.
To address this redundancy, we propose the P roxy- T okenized D iffusion T ransformer ( PT-DiT ), which employs sparse representative token attention (where the number of representative tokens is much smaller than the total number of tokens) to model global visual information efficiently.
Specifically, within each transformer block, we compute an averaging token from each spatial-temporal window to serve as a proxy token for that region.
The global semantics are captured through the self-attention of these proxy tokens and then injected into all latent tokens via cross-attention.
Simultaneously, we introduce window and shift window attention to address the limitations in detail modeling caused by the sparse attention mechanism.
Building on the well-designed PT-DiT, we further develop the Qihoo-T2X family, which includes a variety of models for T2I, T2V, and T2MV tasks.
Experimental results show that PT-DiT achieves competitive performance while reducing the computational complexity in both image and video generation tasks (e.g., a 49% reduction compared to DiT and a 34% reduction compared to PixArt- α 𝛼 \alpha italic_α ). The visual exhibition of Qihoo-T2X is available at https://360cvgroup.github.io/Qihoo-T2X/ .

## 1 Introduction

Figure: Figure 1: The samples from Qihoo-T2I showcase high fidelity and aesthetic qualities, demonstrating a strong consistency with given textual descriptions.
Refer to caption: x1.png

Recent advancements in core diffusion models, including Sora (OpenAI, ), Kling (Kuaishou, ), Stable Diffusion 3 (Stability AI, ), PixArt-$\alpha/\Sigma/\delta$ (Chen et al., ; ; ), Vidu (Shengshu AI, ), Lumina-T2X (Gao et al., ), Flux (BlackForestlabs AI, ), and CogVideoX (Yang et al., ), have led to significant achievements in the creation of photo-realistic image and video. Transformer-based models such as Sora and Vidu have demonstrated the ability to generate high-quality samples at arbitrary resolutions. These models also adhere strongly to scaling laws, achieving superior performance as parameter sizes increase. Additionally, Lumina-T2X has shown uniformity in performing various generation tasks, further validating the potential of the transformer-based architectures in diffusion models.

However, the quadratic complexity of global self-attention concerning sequence length increases the computational cost of the Diffusion Transformer, leading to practical challenges such as longer generation times and higher training costs.
This issue also hinders the application of DiT to high-quality video generation. For example, while 3D attention-based approaches(Xu et al., ; Yang et al., ; Lab & etc., ; Gao et al., ) have demonstrated superiority over 2D spatial attention combined with 1D temporal attention counterparts(Zheng et al., ; Ma et al., ; Bar-Tal et al., ; Blattmann et al., ; Lu et al., ), the extensive computational demands limit their scalability for higher-resolution and longer video generation.
Current studies (Han et al., ; Koner et al., ; Yu et al., ) in visual understanding and recognition have highlighted that global attention mechanisms often exhibit redundancy due to the sparse and repetitive nature of visual information. Specifically, by visualizing the attention map, we observe that the attention of tokens within the same window is similar for spatially distant tokens, while differing for spatially neighboring tokens, as illustrated in Fig. 3.
This observation indicates that the dense long-range attention, which triggers significant computational overhead, is redundant.
Thus, reducing this redundancy is believed to enhance the efficiency of Diffusion Transformers in generating higher-resolution images and longer videos.

In this paper, we propose the Proxy-Tokenized Diffusion Transformer (PT-DiT) and further present the Qihoo-T2X series, which includes both Text-to-Image, Text-to-Video, and Text-to-MultiView generation models.
To address the redundancy of visual information, PT-DiT employs proxy-tokenized attention instead of a global attention mechanism to reduce the computational complexity of visual token interaction.
Specifically, we first recover the spatial and temporal relationships of the token sequence through a reshaping operation.
Given the similarity of visual information within localized spatial regions and temporal frames, we calculate an averaging token from each spatial-temporal window as a representative token, forming a set of proxy tokens.
The interaction and broadcasting of visual global information are then achieved through self-attention among proxy tokens and cross-attention between proxy tokens and all latent tokens.
Additionally, to enhance the texture modeling capabilities, we introduce window attention and incorporate shift-window attention, similar to Swin Transformer (Liu et al., ), to avoid lattice artifacts as shown in Fig. 10.

With the well-designed proxy-tokenized attention, PT-DiT can be adapted to both image and video generation tasks without structural adjustments. For image generation, as shown in Fig. 2, compared to PixArt-$\alpha$ (Chen et al., ), our method achieves an approximate 33% reduction in computational complexity GFLOPs under the same parameter scale. For video generation, in contrast to 2D spatial and 1D temporal attention, which has limited spatial-temporal modeling, and 3D full-attention, which suffers from high computational complexity, PT-DiT can efficiently and comprehensively extracts 3D information, benefiting from proxy token interaction mechanisms.

Figure: Figure 2: Comparison of complexity between PixArt-$\alpha$ and PT-DiT/L at various resolutions.
Refer to caption: x2.png

Experimental results demonstrate that our method achieves competitive performance with significant efficiency. As shown in Fig. 1, Qihoo-T2I can generate high-quality and high-fidelity images while closely adhering to the provided text instructions. Meanwhile, for the image generation task, PT-DiT’s computational complexity is 51% of DiT and 66% of PixArt-$\alpha$ for the same parameter size. For the video generation task, despite having 3 million more parameters than EasyAnimateV4, the PT-DiT/H’s computational complexity is only 82% of EasyAnimateV4 (Xu et al., ) and 77% of CogVideoX (Yang et al., ) for the same parameter size. Overall, using the standard 3D VAE settings ($8\times$ spatial downsampling rate and $4\times$ temporal downsampling rate), experimental tests indicate that we can train the PT-DiT/XL (1.1B) model for images at a resolution of $2048\times 2048$ or for video at a resolution of $512\times 512\times 288$ on the 64GB Ascend 910B.

## 2 Related work

##### Image Generation with Diffusion Transformer.

Recent studies (Peebles & Xie, ; Ma et al., ; Li et al., ; ) have demonstrated the potential of using the Vision Transformer (ViT) (Han et al., ) as an alternative backbone for image generation.
U-ViT (Bao et al., ) encodes the condition as tokens and incorporates skip connections inspired by U-Net, achieving excellent performance.
DiT (Peebles & Xie, ) introduces AdaLN to integrate conditions and analyzes the scalability, complexity, and performance of ViT in comparison to U-Net. SiT (Ma et al., ) adds an interpolant framework to DiT, achieving even better scores on ImageNet (Deng et al., ). PixArt-$\alpha$ (Chen et al., ) integrates cross-attention modules into DiT to inject text conditions and optimize the class-conditional branch. Flag-DiT (Gao et al., ) and Next-DiT (Zhuo et al., ) employ advanced techniques like RoPE (Su et al., ), RMSNorm (Zhang & Sennrich, ), and flow matching (Lipman et al., ) to enhance stability, and they use zero-initialized attention to incorporate complex text instructions. Although the effectiveness of transformers in diffusion models has been validated, the substantial computational and spatial complexities of these models still need to be addressed. DAM (Pu et al., ) has drawn attention to the redundancy within DiT and uses the mediator tokens to directly proxy the query and key in the attention operation, breaking down the process into two distinct attention calculations. We propose PT-DiT, a method that employs a more gentle strategy for compressing hidden states. This approach not only reduces redundancy but also guarantees a minimal loss of information during the compression process, thereby substantially decreasing the computational complexity.

##### Video Generation with Diffusion Transformer.

Building on the advancements in image generation with Diffusion Transformers, recent work (Xu et al., ; Yang et al., ; Lab & etc., ; Zheng et al., ; Ma et al., ; Lu et al., ) has been devoted to extending the DiT structure to video generation. EasyAnimateV2 (Xu et al., ), Open-Sora (Zheng et al., ) is based on PixArt-$\alpha$ (Chen et al., ), incorporating temporal 1D attention and utilizing a 3D VAE to generate additional frames. CogVideoX (Yang et al., ), Open-Sora-Plan (Lab & etc., ), Lumina-T2X (Gao et al., ) and EasyAnimateV4 (Xu et al., ) points out the shortcomings of temporal 1D attention and employs an expert transformer with 3D attention. While 3D attention effectively manages significant motion between adjacent frames, it also incurs a substantial computational cost. To overcome this challenge, we propose PT-DiT, which introduces an innovative token compression strategy. This strategy compresses not only in spatial dimensions but also across frames, enabling 3D attention with significantly reduced computational overhead.

##### Efficient Diffusion Transformer.

As the application of transformers becomes more mature, there are some solutions proposed in different fields targeting the attention computation problem (Han et al., ; Dubey et al., ). Llama3 (Dubey et al., ) adopts the KV-Cache (Pope et al., ) to reduce the number of redundant calculations. AgentAttention (Han et al., ) employs the mediator token mechanisms to compress the interaction scale between Query and Key, yielding promising results in fundamental visual tasks. In the field of image and video generation, Diffusion Transformer also requires strategies to save on computational costs to generate images or videos with higher resolutions. DAM (Pu et al., ), inspired by AgentAttention, applies the concept of the mediator token to the Diffusion Transformer, reducing the computational complexity. We introduce a new proxy attention mechanism that ensures the model adequately learns from each patch before compression. Furthermore, this computational saving is particularly outstanding in the generation of text-to-video content, where it exhibits enhanced performance.

## 3 Method

### 3.1 Redundancy Analysis

Figure: Figure 3: The attention map of self-attention in PixArt-$\alpha$ at 512 resolution. We assemble the attention map for 16 tokens within a $4\times 4$ spatial window. The vertical axis represents different tokens within the window, and the horizontal axis represents their correlation with all latent tokens. It is evident that the attention of different tokens in the same window is almost identical for spatially distant tokens, whereas there is noticeable variation for spatially neighboring tokens.
Refer to caption: x3.png

Due to the sparsity and redundancy of visual information, global attention mechanisms in existing DiTs exhibit significant redundancy and computational complexity, particularly when processing high-resolution images and longer videos.
We analyze this computational redundancy by visualizing the self-attention maps.
Specifically, we examine the attention map of self-attention in PixArt-$\alpha$ at a resolution of $512\times 512$, as shown on the left in Fig. 3.
The attention map for latent codes within a spatial window is then assembled, as depicted on the right side of Fig. 3 (where the vertical axis represents different tokens in a window, and the horizontal axis represents the correlation with all latent tokens).
It is evident that the attention maps for different tokens within the same window are nearly uniform for spatially distant tokens (i.e., at the same horizontal position, the vertical values are almost identical).
Moreover, window tokens exhibit varying attention to spatially neighboring tokens.
This suggests that computing attention for all latent tokens is redundant, while attention for spatially neighboring tokens is critical.
Consequently, we propose a sparse attention strategy that samples limited proxy tokens from each window to perform self-attention, thereby reducing redundancy and decreasing complexity. Additionally, the association between spatially neighboring tokens is established through window attention.
Further details are elaborated in Sec. 3.2.

### 3.2 Architecture of PT-DiT

Figure: Figure 4: The overall architecture of PT-DiT. The image or video undergoes processing through a 3D VAE, followed by noise addition, patch embedding, and positional encoding to generate latent tokens. We replace global attention with proxy-tokenized attention to establish contextual associations and employ visual cross-attention to propagate this information to all tokens, thereby reducing computational redundancy. Moreover, texture detail modeling is enhanced through window attention and shifted window attention.
Refer to caption: x4.png

As shown in Fig. 4, our proposed Proxy-Tokenized Diffusion Transformer (PT-DiT) introduces the proxy-tokenized mechanism to reduce the number of tokens involved in computing global self-attention, thereby efficiently establishing global visual associations.
Specifically, the latent code $z\in\mathbb{R}^{C\times F\times H\times W}$ is passed through path embedding to obtain the latent code sequence $z_{s}\in\mathbb{R}^{N\times D}$. Subsequently, we add 3D positional encoding to $z_{s}$ and feed it into the well-designed Proxy-Tokenized Blocks (PT-Block).
Compared to the vanilla diffusion transformer block, the PT-Block introduces a Global Information Interaction Module (GIIM) and a Texture Complement Module (TCM). The GIIM facilitates efficient interaction among all latent codes using sparse proxy-tokenized mechanisms, while the TCM further refines local detail through window attention and shift-window attention. Below, we describe the GIIM and TCM in detail.

#### 3.2.1 Global Information Interaction Module

Given a series of latent tokens, we first sample a set of proxy tokens based on their spatial and temporal priors. Each proxy token represents a localization within the image or video and interacts with proxy tokens in other regions to establish global relationships. Then, the information contained in proxy tokens is propagated to latent tokens, enabling efficient global visual information interaction.

Specifically, we reshape the latent code sequence $z_{s}\in\mathbb{R}^{N\times D}$ to $z_{s}\in\mathbb{R}^{f\times h\times w\times D}$, where $f$, $h$ and
$w$ denotes the frame, height, and width of video or image ($f=1$) in the latent space after patch embedding, thereby recovering its temporal and spatial connections. The set of proxy tokens $P_{a}\in\mathbb{R}^{D\times\tfrac{f}{p_{t}}\times\tfrac{h}{p_{h}}\times\tfrac{
w}{p_{w}}}$ is calculated from each window of size $p_{t}\times p_{h}\times p_{w}$ using the averaging operation. The parameters $p_{f}$, $p_{h}$, and $p_{w}$ indicate the compression ratios for frame, height, and width, respectively.
Each proxy token represents $p_{t}\times p_{h}\times p_{w}$ tokens, modeling global information with the other proxy token through self-attention. Subsequently, cross-attention is performed to propagate the global visual information into all latent tokens $z_{s}$, where the $z_{s}$ serves as the Query and the proxy tokens $P_{a}$ serve as the Key and Value. The above process is mathematically expressed as follows:

$$ $z_{s}=\mathrm{CS}(z_{s},\mathrm{SA}(\mathrm{Averaging}(z_{s})),$ (1) $$

where $\mathrm{Averaging}(\cdot)$ refers to the averaging operation applied to tokens within the same window to extract proxy tokens, and $\mathrm{CS}(\cdot,\cdot)$ and $\mathrm{SA}(\cdot)$ represent the cross-attention and self-attention operations, respectively. Besides, we introduce a linear layer with zero initialization to enhance training stability.
This approach allows the PT-Block to achieve efficient global information modeling and avoids the computational overhead caused by redundant computations in self-attention. We will analyze the computational complexity advantages of GIIM further in Sec. 3.3.

#### 3.2.2 Texture Complement Module

Due to the characteristics of the sparse proxy tokens interactions, the model’s capacity to capture detailed textures is limited, making it challenging to meet the high-quality demands of generation tasks.
To solve this problem, we introduce localized window attention in the Texture Complement Module (TCM). Specifically, the latent tokens $z_{s}$ are reshaped to $z_{s}\in\mathbb{R}^{\tfrac{f\times h\times w}{p_{t}\times p_{h}\times p_{w}}
\times(p_{t}\times p_{h}\times p_{w})\times D}$ and self-attention is computed along the second dimension. Additionally, shift-window attention is integrated into TCM to avoid the “grid” phenomenon caused by localized window attention. The formula for this process is as follows:

$$ $\displaystyle\hat{z_{s}}=\mathrm{WSA}(z_{s})+z_{s},$ (2) $\displaystyle z_{w}=\mathrm{SWSA}(\hat{z_{s}})+\hat{z_{s}},$ $$

where $\mathrm{SWSA}(\cdot)$ and $\mathrm{WSA}(\cdot)$ denote shift-window attention and window attention respectively. Both window attention and shift-window attention introduce a visual prior to DiT, which aids in the construction of texture details and advances the training of visual generators.
Moreover, the increase in computation is minimal due to the limited number of tokens in each window. We will analyze this in detail in Sec. 3.3.
Then, $z_{w}$ is reshaped to $z_{w}\in\mathbb{R}^{N\times D}$ and fed into Textual Cross-Attention and MLP, similar to DiT.

#### 3.2.3 Compression Ratios

For the image generation task, we find that maintaining the same number of windows across different resolutions is crucial to ensure consistent semantic hierarchies, which aids in training process from low-to-high resolutions.
At the same time, the number of windows should be sufficient to prevent semantic richness within a window from causing a single token to inadequately represent the local area. Therefore, we set compression ratios ($p_{w}$. $p_{f}$, $p_{h}$) to (1, 2, 2), (1, 4, 4), (1, 8, 8), and (1, 16, 16) at 256, 512, 1024, and 2048 resolution respectively.
It is worth noting that when the input is an image, $f$ and $p_{f}$ will be set to $1$. For the video generation task, we set $p_{f}$ = 4 across different resolution to maintain the temporal compression consistent. Owing to token compression in the frame, height and width dimensions, PT-DiT can effectively train a generator for longer videos.

### 3.3 Complexity Analysis

With a small number of representative token attention, PT-DiT reduces the computational redundancy of the original full token self-attention. The advantages of our method in terms of computational complexity are further analyzed theoretically in the following.

The computational complexity of self-attention is $2N^{2}D$, computed as follows:

$$ $\displaystyle z$ $\displaystyle=\mathrm{Softmax}(z^{(q)}z^{(k)\top}/\sqrt{D})z^{(v)},$ (3) $\displaystyle complexity$ $\displaystyle=\underbrace{N^{2}D}_{z^{(q)}z^{(k)\top}:\mathbb{R}^{N\times D} \cdot\mathbb{R}^{D\times N}}+\underbrace{N^{2}D}_{Softmax(\cdot)z^{(v)}: \mathbb{R}^{N\times N}\cdot\mathbb{R}^{N\times D}}+\mathcal{O}(N^{2}),$ $$

where $N$ denotes the length of latent tokens and $D$ represents feature dimension.
Similarly, the computational complexity of GIIM and TCM is computed as follow:

$$ $\displaystyle complexity$ $\displaystyle=\underbrace{2\tfrac{N^{2}}{(p_{f}p_{h}p_{w})^{2}}D}_{\mathrm{SA \;in\;GIIM}}+\underbrace{2\tfrac{N^{2}}{p_{f}p_{h}p_{w}}D}_{\mathrm{CS\;in\; GIIM}}+\underbrace{4\tfrac{N}{p_{f}p_{h}p_{w}}(p_{f}p_{h}p_{w})^{2}D}_{\mathrm {WSA\;and\;SWSA\;in\;TCM}}$ (4) $\displaystyle=2(\tfrac{1}{(p_{f}p_{h}p_{w})^{2}}+\tfrac{1}{p_{f}p_{h}p_{w}}+ \tfrac{2p_{f}p_{h}p_{w}}{N})N^{2}D.$ $$

Obviously, due to the proxy-tokenized strategy, our method provides significant advantages, especially with larger compression ratios ($p_{f}$, $p_{h}$, $p_{w}$) and longer sequence lengths ($N$). When ($p_{f}$, $p_{h}$, $p_{w}$) are (1, 2, 2), (1, 4, 4), (1, 8, 8), and (1, 16, 16) and the image resolution are 256 ($N=256$), 512 ($N=1024$), 1024 ($N=4096$), and 2048 ($N=16348$), our method accounts for only 34.3%, 9.7%, 4.7%, and 2.3% of the total self-attention. In addition, PT-DiT offers even greater benefits for video generation tasks with longer sequence lengths. Experimental analysis is available in Sec. 4.4.

## 4 Experiment

### 4.1 Experimental Setup

Training Setting.
Due to limitations in computational resources, we only trained Qihoo-T2I and Qihoo-T2V based on PT-DiT/XL 1.1B.
Following previous methods (Xu et al., ; Yang et al., ; Chen et al., ), we utilize the T5 large language model as the text encoder and train Qihoo-T2I using a low-to-high resolution strategy divided into three stages. Detailed hyper-parameter settings and the model configurations for various scales of PT-DiT are provided in Appendix. A.1.

Ablation Study.
We conduct ablation experiments using a class-conditional version of PT-DiT/S-Class (32M) on the ImageNet (Deng et al., ) benchmark at 256 resolution. The AdamW optimizer is utilized with a constant learning rate of 1e-4. We train the models for 400,000 iterations with a batch size of 256, while maintaining an exponential moving average (EMA) of the model weights. During the inference, we set denoising step is 50 and use classifier-free guidance (cfg=6.0).

### 4.2 Qualitative analysis

Figure: Figure 5: Qualitative comparison of Text-to-Image generation models.
Refer to caption: x5.png

Text-to-Image.
We provide a qualitative comparison of Qihoo-T2I with existing state-of-the-art Text-to-Image models (e.g., PixArt-$\alpha$ and Flux), as shown in Fig. 5. Qihoo-T2I exhibits competitive performance, generating photo-realistic images that align well with the provided text prompts. Additional samples generated by Qihoo-T2I can be found in the [Project Homepage](https://360cvgroup.github.io/Qihoo-T2X/).

Text-to-Video.
We also compare Qihoo-T2V with the recently released open-source Text-to-Video models (i.e., EasyAnimateV4 and CogVideoX) at a resolution of 512, achieving comparable results, as depicted in Fig. 6. More video samples are available in the [Project Homepage](https://360cvgroup.github.io/Qihoo-T2X/).

Figure: Figure 6: Qualitative comparison of Text-to-Video generation models.
Refer to caption: x6.png

Text-to-MV.
We further explore the effectiveness of PT-DiT on Text-to-MultiView (T2MV) tasks.
The trained Qihoo-T2MV is capable of generating $512\times 512\times 24$ images from various viewpoints based on the provided text instruction, showcasing strong spatial consistency, as illustrated in Fig. 7.
For detailed experimental and training setups, please refer to Appendix. A.2.

Figure: Figure 7: Samples by Qihoo-T2MV. It is important to note that Qihoo-T2MV does not accept any image inputs or camera parameters and relies solely on text prompts.
Refer to caption: x7.png

### 4.3 Quantitative analysis

MS-COCO.
We conduct experiments to quantitatively evaluate Qihoo-T2I using zero-shot FID-30K on the MS-COCO (Lin et al., ) $256\times 256$ validation dataset, as shown in Table 8(a). Due to the distribution gap between our collected data and MS-COCO, there is a resulting decrease in FID (Heusel et al., ) metrics. Nevertheless, Qihoo-T2I achieves a competitive score of 15.70.

MSR-VTT and UCF-101.
We evaluate Qihoo-T2V on two standard video generation benchmarks, MSR-VTT (Xu et al., ) and UCF-101 (Soomro et al., ), at a resolution of 256. As shown in Table 8(b), Qihoo-T2V achieves state-of-the-art results among DiT-based approaches and demonstrates competitive performance compared to U-Net-based approaches. Notably, since CogVideoX, EasyAnimateV4, and Qihoo-T2V all utilize T5 as the text encoder, this creates a gap in CLIPSIM compared to methods that employ the CLIP as the text encoder, such as AnimateDiff, DynamiCrafter, PixelDance, and FancyVideo.

**Table 1: The quantitative evaluation of the Text-to-Image (a) and Text-to-Video (b) tasks.**
| Method | FID-30k$\downarrow$ |
| --- | --- |
| DALL-E 2 (Ramesh et al., ) | 10.39 |
| SD (Rombach et al., ) | 8.73 |
| Imagen (Saharia et al., ) | 7.27 |
| RAPHAEL (Xue et al., ) | 6.61 |
| Kolors (Team, ) | 23.15 |
| PixArt-$\alpha$ (Chen et al., ) | 10.65 |
| Flux.1-dev(BlackForestlabs AI, ) | 22.76 |
| Qihoo-T2I | 15.70 |

Figure: Figure 8: Comparison of image generation models in terms of GFLOPs.
Refer to caption: x8.png

Figure: Figure 9: Comparison of video generation models in terms of GFLOPs and GPU memory usage.
Refer to caption: x9.png

### 4.4 Algorithmic Efficiency Comparison

As discussed in Sec. 3.3, our method effectively reduces complexity. In this section, we further analyze the computational advantages of PT-DiT in T2I and T2V tasks.

In the image generation task, similar to Fig. 2, we conduct comparisons at different parameter scales. With equivalent parameter counts, we compared Lumina-Next (1.7B) to our PT-DiT/H (1.8B), DiT/B (0.13B) and DAM/B (0.13B) to our PT-DiT/B (0.14B), as illustrated in Fig. 8. As shown on the left side of Fig. 8, the GFLOPs of PT-DiT/H are significantly lower than Lumina-Next across multiple scales. Specifically, at resolutions of 512 and 2048, PT-DiT/H achieves complexity reduction of respectively 82.0% and 82.5%. Similarly, the right side of Fig. 8 indicates that PT-DiT/B requires 48.6% less computation than DiT/B at a resolution of 1024. Compared to DAM/B, which has an attention computation complexity of $O(n)$, our method exhibits a comparable level of computation complexity across all resolutions.

In the video generation task, we assess our model based from two aspects: computational complexity and GPU memory consumption, as illustrated in Fig. 9. We conduct experiments using two scales of PT-DiT (i.e., PT-DiT/H (1.8B) for a consistent scale comparison and our utilized PT-DiT/XL (1.1B) for training Qihoo-T2V) and select the latest open-source T2V model (i.e., CogVideoX-2B (actual test at 1.7B) and EasyAnimateV4 (1.5B)) as the comparison methods. The left side of Fig. 9 displays the GFLOPs calculated at different resolutions, with the latent code set to a time dimension of 48.
It is obvious that, despite having the largest number of parameters, PT-DiT/H exhibits the lowest computational complexity.
Meanwhile, the computational complexity of PT-DiT/XL employed by Qihoo-T2V is only 50% that of CogVideoX and EasyAnimateV4.
On the right side of Fig. 9, we further compare the GPU memory usage during training with EasyAnimateV4 at a resolution of 512, across different frame counts.
Since the T2V version of EasyAnimateV4 employs HunyuanDiT with full 3D attention, its memory consumption increases dramatically with the number of video frames. In contrast, PT-DiT, which also utilizes 3D spatial-temporal modeling, experiences only a slight increase in memory consumption due to its well-designed proxy-tokenized attention mechanism.
The above experiments demonstrate the potential of PT-DiT for generating longer and higher-resolution videos.

**Table 2: Ablation study on PT-DiT/S-Class. Models are trained for 400k iterations.**
| Structure | FID-50k$\downarrow$ |
| --- | --- |
| w/o GIIM | 23.71 |
| w/o SWA | 23.59 |
| w/o TCM | 69.07 |

### 4.5 Ablation Study

Major Component.
We conduct quantitative experiments to assess the effectiveness of the GIIM and TCM proposed in this paper. The absence of either GIIM or TCM results in a substantial performance loss (i.e., 19.30 $\rightarrow$ 23.71 or 19.30 $\rightarrow$ 69.07). Specifically, without TCM, the model struggles

Figure: Figure 10: Ablation on shift-window attention.
Refer to caption: x10.png

to capture fine details, making it challenging to meet generation tasks that demand high-quality detail, leading to a significant decline in performance. Additionally, we investigated the role of shift-window attention through both qualitative evaluation at a resolution of 512 and quantitative analyses at a resolution of 256, as illustrated in Fig. 10 and Table 10(a) respectively. As anticipated, there is a noticeable decrease (i.e., 19.30 $\rightarrow$ 23.59) in FID without shift-window attention, accompanied by pronounced “grid” phenomena.

Proxy Token Extraction.
As illustrated in Table 10(b), we explore three methods for obtaining the proxy token: the top-left token, a randomly selected token, and averaging the in-window tokens. A performance gap exists between the Top-Left (20.84) or Random (21.00) selections and the averaging manner (19.30). We believe this gap arises because the random and top-left tokens fail to adequately represent the overall characteristics of the region, compromising the effectiveness of proxy-tokenized attention and leading to performance loss. We use averaging as the default setting.

Global Information Injection.
Due to the misalignment between the number of proxy tokens and latent tokens, we investigate three schemes for injecting global information into latent tokens: Cross-Attention, Interpolation, and Linear projection, as shown in Table 10(c).
Among these, interpolation involves applying spatially bilinear interpolation to the proxy tokens, while linear projection aligns proxy tokens with latent tokens through a linear layer.
Since each latent code can leverage global information from the entire set of proxy tokens, Cross-attention achieves a performance advantage with an FID of 19.30 compared to Linear projection at 20.24 and Interpolation at 21.82.

Compressed Ratio.
As reported in Table 10(d), we examine the impact of compression ratio on performance at a resolution of 256. It is evident that when the compression ratio is high, the representative token fails to adequately capture the features of the region for effective global modeling, leading to a noticeable decline in performance (i.e., from 19.30 to 21.24 at (1, 4, 4)).

## 5 Conclusion

Given the sparsity and redundancy of visual information, this paper proposes PT-DiT, which leverages the proxy-tokenized attention mechanism to mitigate the computational redundancy of self-attention in diffusion transformers. A series of representative tokens are calculated based on temporal and spatial priors, with global interactions between them. Additionally, window attention and shifted window attention are introduced to refine the modeling of local details. Our proposed representative token mechanism is particularly effective for video tasks with redundant information, enabling 3D spatio-temporal modeling while avoiding an explosion in computational complexity. Experiments demonstrates that PT-DiT achieves competitive performance while delivering significant efficiency.
We further develope the Qihoo-T2X series based on PT-DiT, including models like T2I, T2V, and T2MV. We hope PT-DiT and Qihoo-T2X can provide new insights and references for the field of diffusion transformers.

## Appendix A Appendix

### A.1 Training detail and model configuration

We collect a total of 50M data points for the training set, including 32M images with an aesthetic score of 5.5 or higher from Laion (Schuhmann et al., ) and 18M high-resolution, high-quality datasets that we constructed. During the high-resolution training phase, we exclusively use 18M high-quality data.
We train Qihoo-T2V by progressing through three stages starting from stage 1 of Qihoo-T2I, with detailed hyper-parameters shown in Table 3. The WebVid 10M (Bain et al., ) dataset is employed as the 256-resolution video training data. Additionally, we collect 3M high-resolution, high-quality video samples from the Internet to train the high-resolution video generator. The training objective for Qihoo-T2X is v-prediction, with an extracted text token length of 120. During the inference phase, the denoising steps are set to 50, and the scale of classifier-free guidance is set to 6.0. The specific parameter configurations for various scales of PT-DiT are presented in Table LABEL:model_config.

**Table 3: The training setups of Qihoo-T2I and Qihoo-T2V**
| Text-to-Image | Text-to-Video |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Resolution | Data | Learning Rate | Batch Size | Iteration | Resolution # Frame | Data | Learning Rate | Batch Size | Iteration |
| 256 | 50M | 2e-5 | 10240 | 100k | - | - | - | - | - |
| 512 | 18M HQ | 2e-5 | 768 | 50k | 256 # 96 | 10M | 2e-5 | 512 | 100k |
| 1024 | 18M HQ | 2e-5 | 512 | 50k | 512 # 96 | 3M HQ | 2e-5 | 256 | 50k |

**Table 4: The model configurations for various scales of PT-DiT.**
| Model | Layers | Hidden Dim | Head Number | Param. (M) |
| --- | --- | --- | --- | --- |
| PT-DiT/S-Class | 10 | 288 | 6 | 32 |
| PT-DiT/B | 12 | 640 | 10 | 144 |
| PT-DiT/L | 28 | 864 | 12 | 605 |
| PT-DiT/XL | 28 | 1152 | 16 | 1142 |
| PT-DiT/H | 30 | 1440 | 20 | 1795 |

### A.2 Training detail about Qihoo-T2MV

Basic setting. Multi-view images of 3D objects can be interpreted as videos of static objects. We utilize a subset of approximately 40k samples from G-Objaverse (Qiu et al., ), following the VideoMV (Zuo et al., ), which is rendered as video data to train our Qihoo-T2MV model. Each object is rendered with a uniformly distributed azimuth from 0°  to 360°  and an elevation ranging from 5°  to 30° , resulting in a $512\times 512\times 24$ video.

Training setting. Following previous works (Zuo et al., ; Shi et al., ), we only accept text instruction as input to generate the Multi-View images of 3D object without additional reference images and camera parameters. The QIhoo-T2MV is trained from stage 2 of the Qihoo-T2I, with a bacthsize of 128 and 20k iterations. The other hyperparameters and experimental settings are the same as QIhoo-T2I.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness.
soundness: 3
strengths: The proposed method significantly reduces computational complexity and has certain competitiveness in various tasks, such as T2I, T2V,and T2MV.
summary: This paper proposes Proxy-Tokenized Diffusion Transformer to reduce the computational complexity. By utilizing sparse representative token attention, window and shift window attention, the proposed method effectively reduces redundancy in global self-attention computation and optimizes the modeling process of visual information. Also, the Qihoo-T2X family includes multiple models for text to image, text to video, and text to multi view tasks. However, it is difficult to fully demonstrate the effectiveness of the method at various resolutions in the experimental section.
weaknesses: 1)	Insufficient experiments and lack of comparative experiments at high resolution.
2)     How is the value of compression ratio determined, and is it related to both semantic information and resolution size?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Although the authors provide extensive comparisons between resolution and computational metrics (e.g., memory usage, GFLOPs), I believe that additional comparisons between performance and computational cost relative to other approaches in text-to-image and text-to-multiview tasks are necessary. Assessing the trade-off between performance and computational efficiency, rather than focusing solely on resolution, would offer a more comprehensive evaluation of the proposed method's effectiveness. Since identical resolutions do not inherently guarantee equivalent performance, it is conceivable that approximate methods may result in some performance degradation.

- Several prior works, such as ToMe, ToMeSD, TRIPS, and CrossGET, have also addressed token merging. The absence of a detailed discussion or comparison with these existing studies undermines the claimed novelty of Qihoo-T2X. I suggest that the authors more explicitly delineate the unique contributions and advantages of their approach in relation to these established methods.
soundness: 2
strengths: - The paper is well-crafted, providing a clear exposition of the proposed methodology.
- The approach is conceptually straightforward and highly intuitive.
- The motivation and objectives are innovative, focusing on a token-efficient mechanism for DiT-based generative models, which is particularly significant given the paucity of existing approaches for DiT-based models.
summary: This paper introduces a proxy-tokenized mechanism for transformer-based diffusion models (e.g., DiT), utilizing sparse representative tokens to alleviate redundant computations arising from inherent visual redundancy. Specifically, the authors observe that tokens representing spatially proximate regions tend to exhibit high similarity, while tokens corresponding to spatially distant regions contribute less to meaningful computation in the attention map. Leveraging this insight, the authors propose the use of a proxy token for each window by averaging the tokens within it and subsequently employing self-attention and cross-attention operations involving both proxy tokens and original tokens to ensure efficient interaction across all tokens. Furthermore, the proposed approach integrates a Swin-based architecture to implement efficient attention mechanisms in accordance with these observations. The method achieves competitive performance compared to state-of-the-art approaches across three tasks—text-to-image, text-to-video, and text-to-multiview—at varying resolutions, underscoring its effectiveness and significance within the field.
weaknesses: - The lack of ablation studies assessing performance metrics, such as FID with respect to GFLOPs, diminishes the practical relevance of the proposed methodology.
- The discussion and comparison do not sufficiently address related works on token merging (e.g., ToMe [1], ToMeSD [2], CrossGET [3], and TRIPS [4]), which limits the comprehensiveness of the literature review.
- The proposed method integrates a Swin-based architecture within the DiT-based model, employing a straightforward average operation to represent tokens. From the perspective of novelty, this contribution appears to be incremental.



[1] Token Merging: Your ViT But Faster, ICLR 2023

[2] Token Merging for Fast Stable Diffusion, CVPR Workshop 2023 

[3] CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers, ICML 2024

[4] TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection, EMNLP 2022

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: See weaknesses.
soundness: 3
strengths: 1. The PT-DiT addresses the redundancy in global self-attention by using sparse representative token attention, reducing computational complexity.

2. The introduction of window and shift window attention mechanisms helps to capture fine details and textures, improving the quality of generated images and videos.

3. The method is designed to be adaptable to both image and video generation tasks without structural adjustments, demonstrating competitive performance compared to original models and great flexibility in application

4. This method is simple and effective.

5. This paper is well written.
summary: The paper introduces an efficient Proxy-Tokenized Diffusion Transformer (PT-DiT) for accelerating self-attention mechanism, addressing the redundancy in global self-attention mechanisms of diffusion transformers. It develops Qihoo-T2X family for text-to-any-task applications. PT-DiT uses sparse representative token attention, computing an averaging token from each spatial-temporal window as a proxy, reducing computational complexity. It includes window and shift window attention to enhance detail modeling. The Qihoo-T2X family, which includes models for T2I, T2V, and T2MV tasks, shows competitive performance with reduced computational complexity in image and video generation tasks.
weaknesses: - Insufficent performance comparison to peer methods such as DAM [1] and AgentAttention [2]. While PT-DiT significantly outperforms baselines, its performance relative to peer methods like DAM and AgentAttention is not clearly superior (see Figure 8). Need more comparison.
- **Need further discussion on the relation and differences to previous methods.**
    - For example, DAM **also finds significant attention map redundancy** by analyzing the Divergence Score and proposes a set of Attention Mediator Tokens. These tokens can also **be seen as a kind of extractor** for extracting information from the original latent codes (Queries and Keys).
    - From this perspective, **based on DAM, PT-DiT only points out that redundancy in attention maps remains due to spatial reasons**, and the proposed average operation and sparse representative token attention may be implicitly included by the interaction between the original Queries/Keys and the Attention Mediator Tokens in DAM.
    - Additionally, PT-DiT doesn't analyze the significance of the observed phenomenon across different denoising timesteps, as DAM does.
- From the ablation study in Table 2(a), the results show that without TCM (an auxiliary module), the performance drops significantly, whereas without GIIM, which is the core component proposed in the paper, the performance decline is much less. Does this imply that auxiliary module TCM is more important than core module GIIM?
- A statistical comparison between Qihoo-T2V with and without PT-DiT is necessary for clearer insights into the model's impact.

[1] Pu Y, Xia Z, Guo J, et al. Efficient diffusion transformer with step-wise dynamic attention mediators[J]. arXiv preprint arXiv:2408.05710, 2024.

[2] Han D, Ye T, Han Y, et al. Agent attention: On the integration of softmax and linear attention[J]. arXiv preprint arXiv:2312.08874, 2023.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: About method section: 
-  l201-203, f, h, w are not the actual frame, height, and width of the video/image, but the latent, right? Also, for images, how does the 3D VAE, with temporal downampling layers, handle them? 
- the texts from l242 to l244 are confusing, $(p_f \cdot p_f, p_h)$ should be ($p_f, p_h, p_w$)? so we have $p_h = p_w$ = <some value depends on resolution> and $p_f$ = 1 for image and $p_f$ = 4 for videos, right?
- Details for the window attention and shifting window attention are missing. Only Figure 4 shows some fundamental ideas of them but how do they work exactly? How does the shifting window work?

About T2MV: 
- Why Text-to-MV task is interesting? At the training level, there is no difference from this task to Text-to-video unless the training data; while for real practice, it's hard to control the generated content from only texts without any reference image. 
- It's hard to evaluate the performance of T2V-MV. Can the T2V-MV work for out-of-distribution prompts? And how about the visual and quantitative evaluations with other models, given that the task setting is following previous approaches? 

Other questions:
- The page limit is 10, no need to move related work to the appendix. 
- Can we use a VAE with a higher compression ratio, or a large patch embedding to achieve a similar level of efficiency? In principle, with the proposed design, more details could be restored compared with directly using a higher compression ratio (but with vanilla attention). It would also be great to show that.
soundness: 3
strengths: - The paper is well-written and easy to follow. 
- The visualizations are promising. 
- Compared with SOTA methods, the model can achieve a similar level of performance with less training data. Meanwhile, the inference cost is much lower.
summary: This paper introduces the Qihoo-T2X series, featuring the Proxy-Tokenized Diffusion Transformer (PT-DiT) for efficient text-to-video generation. PT-DiT uses proxy-tokenized attention to reduce computational redundancy by selecting representative tokens within spatial-temporal windows, enhancing global information capture without the high complexity of full self-attention. This design integrates global semantics efficiently through sparse proxy tokens and refines texture details using window and shifted-window attention, similar to the Swin Transformer. The Qihoo-T2X family demonstrates competitive performance with reduced complexity compared to models like DiT and PixArt-α.
weaknesses: - The method section is not clear enough. The mechanisms of window attention and shifting window attention are still not clear.  
- The task of T2MV is a little strange. What would be an application scenario for that task?  
- The "text-to-any-task" is overclaimed. The focused three tasks are essentially video generation (with a single frame or multi-view video).  

See the next question sections for more details.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 3
questions: -What accounts for PT-DiT's 80% improvement over Lumina in Figure 8 (left), especially since this enhancement is significantly greater than the improvements observed for other models?

-How about the training time and inference time of PT-DiT?

-Does the author have any plans to open source the Qihoo-T2X series of models and code?
soundness: 3
strengths: -The motivation for reducing the training and inference costs of DiT by minimizing redundant computation in global self-attention is clear.

-The GIIM and TCM design in PT-DiT is promising and straightforward to implement.

-Qualitative and quantitative experiments on a variety of tasks (e.g., Text-to-Image, Text-to-Video, and Text-to-MultiView) are comprehensive and convincing.
summary: This paper first highlights the computational redundancy issue of global self-attention in diffusion transformers, stemming from sparse visual information, through an analysis of the attention map. To tackle this problem, the authors propose the Proxy-tokenized Diffusion Transformer (PT-DiT), which substitutes the global self-attention of the vanilla DiT with Global Information Interaction Module (GIIM) and Texture Complement Module (TCM), enabling efficient modeling of both global and local visual information. In experiments, PT-DiT demonstrates theoretical and experimental advantages in computational complexity and shows competitive performance across various tasks, including T2I, T2V, and T2MV, underscoring its generalization capabilities.
weaknesses: -This article lacks results on ImageNet for a fair comparison with existing state-of-the-art diffusion transformer models. Previous studies, such as DiT, MDT, VAR, and SiT, have all been evaluated on ImageNet using metrics like FID and IS. To fully establish its effectiveness, PT-DiT should also be assessed under similar conditions. This is essential for demonstrating its competitive performance despite reduced computational complexity and should be included in evaluations.

-How is the hyper-parameter compression ratio determined in this paper? Why is the compressed ratios set to (1, 2, 2) at a resolution of 256, and will a ratio of (1, 16, 16) result in excessive information loss at a resolution of 1024? The authors should provide further clarification on the rationale and motivations behind the chosen compressed ratios settings in their methodology.

-The extraction of proxy tokens through averaging is straightforward, but how representative are these proxy tokens when obtained in this manner? Could proxy tokens also be extracted using clustering or similarity-based methods for potentially improved representativeness?

Meta-review: (a) The paper proposes the Proxy-Tokenized Diffusion Transformer to reduce computational complexity by using sparse representative token attention and window-based attention. The Qihoo-T2X family includes models for text-to-image, text-to-video, and text-to-multi-view tasks.

(b) The strengths of the paper are: The paper effectively reduces training and inference costs of DiT by minimizing redundant computation through the promising GIIM and TCM designs, demonstrating strong performance across tasks (Text-to-Image, Text-to-Video, Text-to-MultiView) with comprehensive experiments, clear visualizations, and lower data and computational requirements compared to SOTA methods.

(c) The weaknesses of the paper are: unclear method part, overclaimed "text-to-any-task", lack of ImageNet comparisons, hyper-parameter ablation, the mechanism of proxy tokens, lack of related works on token merging and lack of comparative experiments at high resolution.

(d) The most important reasons for acceptance are that this work introduces the sparse representative token attention to model global visual information efficiently, whilst makes solid contributions by introducing a variety of models for a series of  T2I, T2V, and T2MV tasks.

Decision: Accept (Poster)