Title: CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Transformer

High-Resolution Image Generation

Efficient Attention
Abstract: Diffusion Transformers (DiT) have become a leading architecture in image generation. However, the quadratic complexity of attention mechanisms, which are responsible for modeling token-wise relationships, results in significant latency when generating high-resolution images. To address this issue, we aim at a linear attention mechanism in this paper that reduces the complexity of pre-trained DiTs to linear. We begin our exploration with a comprehensive summary of existing efficient attention mechanisms and identify four key factors crucial for successful linearization of pre-trained DiTs: locality, formulation consistency, high-rank attention maps, and feature integrity. Based on these insights, we introduce a convolution-like local attention strategy termed CLEAR, which limits feature interactions to a local window around each query token, and thus achieves linear complexity. Our experiments indicate that, by fine-tuning the attention layer on merely 10K self-generated samples for 10K iterations, we can effectively transfer knowledge from a pre-trained DiT to a student model with linear complexity, yielding results comparable to the teacher model. Simultaneously, it reduces attention computations by 99.5% and accelerates generation by 6.3 times for generating 8K-resolution images. Furthermore, we investigate favorable properties in the distilled attention layers, such as zero-shot generalization across various models and plugins, and improved support for multi-GPU parallel inference. Models and codes will be available.

CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up

Abstract

Diffusion Transformers (DiT) have become a leading architecture in image generation. However, the quadratic complexity of attention mechanisms, which are responsible for modeling token-wise relationships, results in significant latency when generating high-resolution images. To address this issue, we aim for a linear attention mechanism in this paper that reduces the complexity of pre-trained DiTs to linear. We begin our exploration with a comprehensive summary of existing efficient attention mechanisms and identify four key factors crucial for the successful linearization of pre-trained DiTs: locality, formulation consistency, high-rank attention maps, and feature integrity. Based on these insights, we introduce a convolution-like local attention strategy termed CLEAR, which limits feature interactions to a local window around each query token, and thus achieves linear complexity. Our experiments indicate that by fine-tuning the attention layer on merely 10K self-generated samples for 10K iterations, we can effectively transfer knowledge from a pre-trained DiT to a student model with linear complexity, yielding results comparable to those of the teacher model. Simultaneously, it reduces attention computations by 99.5% and accelerates generation by 6.3 times for generating 8K-resolution images. Furthermore, we investigate favorable properties in the distilled attention layers, such as zero-shot generalization across various models and plugins, as well as improved support for multi-GPU parallel inference. Models and codes are available here.

1 Introduction

Diffusion models [42, 14, 50, 26] have gained widespread attention in text-to-image generation, proving to be highly effective for producing high-quality and diverse images from textual prompts [10, 64]. Traditionally, architectures based on UNet [51, 50] have dominated this field due to their robust generative capabilities. In recent years, Diffusion Transformers (DiTs) [44, 1, 6, 17, 36, 18, 5] have emerged as a promising alternative, achieving leading performance in this field. Unlike the UNet-based architectures, DiTs leverage the attention mechanism [56] to model intricate token-wise relationships with remarkable flexibility, enabling them to capture nuanced dependencies across all tokens in images and texts, and thus produce visually rich and coherent outputs.

Despite their impressive performance, the attention layers-which model intricate pairwise token relationships with quadratic complexity-can introduce substantial latency in high-resolution image generation. As shown in Fig. 2, FLUX.1-dev [34], a state-of-the-art text-to-image DiT, requires over 30 minutes to generate 8K-resolution images with 20 denoising steps, even with hardware-aware optimizations like FlashAttention [12, 11].

Focusing on these drawbacks, we are curious about one question: Is it possible to convert a pre-trained DiT to achieve linear complexity? The answer is not straightforward, in fact, as it remains unclear whether existing efficient attention mechanisms-despite their recent widespread exploration [13, 31, 65, 49, 58, 52, 3, 68, 8, 67, 71, 28, 21]-can be effectively applied to pre-trained DiTs.

To answer this question, we initiate our exploration with a summary of previous methods dedicated to efficient attention, categorizing them into three main strategies: formulation variation, key-value compression, and key-value sampling. We then experiment with fine-tuning the model by replacing the original attention layers with these efficient alternatives. Results indicate that while formulation variation strategies have proven effective in attention-based UNets [39] and DiTs trained from scratch [62], they do not yield similar success with pre-trained DiTs. Keyvalue compression often leads to distorted details, and key-value sampling highlights the necessity of local tokens for each query to generate visually coherent results.

Building on these observations, we figure out four elements crucial for for linearizing pretrained DiTs, including locality, formulation

6.3

×

Faster

99.5%

Saved

Scale)

Original FLUX-1.dev

Scale)

20

*

OriginalFLUX-1.dev

10

Ours (r=8)

Ours (r=16)

18

Ours (r=8)

Ours (r=16)

Ours (r=32)

607)

16

Ours (r=32)

(Log

8

14

GFLOPS/Layer

./Image

6

12

10

4

u

8

S

2

6

1k2

2k2

4k²

8k2

1k²

2k2

4k²

8k2

NumberofPixels

consistency, high-rank attention maps, and feature integrity. Satisfying all these criteria, we present a convolution-like linearization strategy termed CLEAR , where each query interacts only with tokens within a predefined distance r . Since the number of key-value tokens interacting with each query is fixed, the resulting DiT achieves linear complexity with respect to image resolution.

To our surprise, such a concise design yields results comparable to original FLUX.1-dev after a knowledge distillation process [25] with merely 10K fine-tuning iterations on 10K self-generated samples. As shown in Fig. 1, CLEAR exhibits satisfactory cross-resolution generalizability, a property also reflected in UNet-based diffusion models [2, 15, 23, 29]. For ultra-high-resolution generation like 8K, it reduces attention computations by 99.5% and accelerates the original DiT by 6.3 times, as shown in Fig. 2. The distilled local attention layers are also compatible with different variants of the teacher model, e.g. , FLUX.1-dev and FLUX.1-schnell, and various pre-trained plugins like ControlNet [69] without requiring any adaptation.

As the token interactions are performed locally, it is convenient for CLEAR to support multi-GPU parallel inference. We further develop a patch-parallel paradigm that minimizes communication overhead. Our contribution can be summarized as follows:

We provide a taxonomic overview of recent efficient attention mechanisms and identify four elements essential for linearizing pre-trained DiTs.

Based on them, we propose a convolution-like local attention mechanism termed CLEAR as an alternative to default attention, which is the first linearization strategy tailored for pre-trained DiT to the best of our knowledge.

We delve into multiple satisfactory properties of CLEAR through experiments, including its comparable performance with the original DiT, linear complexity, cross-resolution generalizability, cross-model/plugin generalizability, support for parallel inference, etc.

2 Efficient Attention: A Taxonomic Overview

The attention mechanism [56] is known for its flexibility in modeling token-wise relationships. It takes a query matrix Q ∈ R n × c , a key matrix K ∈ R m × c , and a value matrix V ∈ R m × c ′ as input and produces an output matrix O ∈ R n × c ′ via:

where n and m are the numbers of query and key-values tokens respectively, and c and c ′ are the feature dimensions for query-key and value tokens. In line with standard design conventions, we assume c = c ′ throughout this paper, and in the case of self-attention, Q , K , and V come from the same feature maps with m = n .

As shown in Eq. 1, self-attention involves constructing n × n attention maps to model pair-wise token-to-token relationships, which results in both time and memory complexity. To address this issue, numerous studies focus on developing efficient attention mechanisms. In this section, we summarize recent work in this area and assess its applicability to DiT linearization. Specifically, we categorize existing approaches into three main categories: formulation variation, key-value compression, and key-value sampling.

2.1 Formulation Variation

Revisiting Eq. 1, if the softmax operation is omitted, we can first compute K ⊤ V , yielding a c × c matrix with linear time in relation to n . In this way, a series of linear attention mechanisms apply kernel functions f ( · ) and g ( · ) to Q and K respectively to mimic the effect of softmax :

such as Mamba2 [13], Gated Linear Attention [65], and Generalized Linear Attention [39]. Another mainstream of methods try to replace the softmax operation with efficient alternatives, like sigmoid [49], relu 2 [28, 67], and Nystrom-based approximation [63].

2.2 Key-Value Compression

In the default setting of self-attention, the numbers of query and key-value tokens are consistent, i.e. , m = n , and the shape of the attention map would be n × n . It is thus promising to compress key-value tokens so that m can be smaller than n to reduce the complexity. Following this routine, PixArtSigma [5] compress KV tokens locally with a downsampling Conv2d operator. Agent Attention [21] first conducts attention with downsampled Q and full-sized K and V to select agent KV tokens for compression. Then, original Q would interact with these compressed tokens. Similarly, Slot Attention [71] adopts learnable slots to obtain agent KV. Linformer [58] introduces learnable maps to obtain compressed tokens from the original ones.

2.3 Key-Value Sampling

Efficient attention based on key-value sampling is based on the assumption that not all key-value tokens are important for a query and the attention matrix is highly sparse. Comparing with key-value compression, it prunes key-value tokens for each query instead of producing new tokens. For instance, Strided Attention [7] samples one key-value token at a regular interval. Routing Attention [52] samples key-value tokens based on grouping. Swin Transformer [40] divides feature maps into nonoverlapping local windows and performs attention independently for each window. Neighborhood

Add

Noise

Query

Original Image

Attention Maps w.r.t. Query

Full Attention Maps

𝑑

!"

∗

= 32

𝑑

!"

∗

= 16

𝑑

!"

∗

= 8

min

𝑑

!"

∗

= 2

Attention [22] selects key-value tokens within a local window around each query. BigBird [68] uses a selection strategy combining neighborhood attention and random attention, and LongFormer [3] combines neighborhood attention with some global tokens that are visible to all tokens.

3 Methods

3.1 What are Crucial for Linearizing DiTs?

Building on the overview of recent efficient attention mechanisms in Sec. 2, we explore a key question here: What specific features are essential for successfully linearizing pre-trained DiTs ? We thus try substituting all the attention layers in FLUX.1-dev with various efficient alternatives and fine-tuning parameters in these layers. The preliminary text-to-image results are shown in Fig. 3, through which we figure out four key elements: locality, formulation consistency, high-rank attention maps, and feature integrity. According to these perspectives, we summarize some previous efficient attention methods in Tab. 1.

Locality indicates that key-value tokens fallen in the neighborhood of a query are included for attention. From Fig. 3, we observe that many methods equipped with this feature yield at least plausible results, like PixArt-Sigma, Swin Transformer, and Neighborhood Atten-

tion. Particularly, comparing the results of Neighborhood Attention and Strided Attention, we find that incorporating local key-value tokens diminishes a lot of distorted patterns.

The reason for these phenomena is that pre-trained DiTs, such as FLUX, rely heavily on local features to manage token relationships. To validate this, we visualize attention maps in Fig. 4 and observe that the most significant attention scores fall in the local area around each query.

In Fig. 5, we provide further evidence to illustrate the importance of local features, that perturbing remote features would not damage the quality of FLUX.1-dev much. Specifically, FLUX.1-dev relies on rotary position embedding [55] to perceive spatial relationships and is sensitive to the relative distance ( d ( x ) ij , d ( y ) ij ) on the two axes of a 2D feature map, where indices i and j denote query and key token indices respectively. We perturb remote features by clipping the relative distances for rotary position embedding to a maximum value r when they exceed this threshold,

i.e. , d ( ∗ ) ′ ij = d ( ∗ ) ij . clip( -r, r ) . As shown in Fig. 5(left), the results are reasonable for a 64 × 64 feature map when r is as small as 8 . Conversely, if we perturb local features by setting their minimum absolute distances to r , even with r as small as 2 , the result still collapses as shown in Fig. 5(right)-emphasizing the importance of locality.

Formulation Consistency denotes that the efficient attention still applies the softmax -based formulation of the scaled dot-product attention. LinFusion [39] has shown that linear attention approaches like linear attention achieve promising results in attention-based UNets. However, we find that it is not the case for pre-trained DiTs, as shown in Fig. 3. We speculate that it is due to attention layers being the only modules for token interactions in DiTs, unlike the case in U-Nets. Substituting all of them would have a substantial impact on the final outputs. Other formulations like Sigmoid Attention fail to converge within a limited number of iterations, unable to mitigate the divergence between the original and modified formulations. It is thus beneficial to maintain consistency with the original attention function.

High-Rank Attention Maps means that attention maps calculated by efficient attention alternatives should be sufficient to capture the intricate token-wise relationships. As visualized in Fig. 4, extensive attention scores are concentrated along the diagonal, indicating that the attention maps do not exhibit the low-rank property assumed by many prior works. That is why methods like linear attention and Swin Transformer largely produce blocky patterns.

Feature Integrity implies that raw query, key, and value features are more favorable than the compressed ones. Although PixArt-Sigma has demonstrated that applying KV compression on deep layers would not hurt the performance much, this approach is not suitable for completely linearizing pre-trained DiTs. As shown in Fig. 3, methods based on KV compression, such as PixArt-Sigma and Agent Attention, tend to produce distorted textures compared to the results from Swin Transformer and Neighborhood Attention, which highlights the necessity to preserve the integrity of the raw query, key, and value tokens.

3.2 Conv-Like Linearization

Given the above analysis of the crucial factors for linearizing DiTs, Neighborhood Attention is the only scheme satisfying all 4 constraints. Motivated by this, we propose CLEAR, a conv-like linearization strategy tailored for pre-trained DiTs. Specifically, given that state-of-the-art textto-image DiTs, like FLUX and StableDiffusion 3 series, typically adopt text-image joint selfattention, in CLEAR, for each text query, it still gathers features from all text and image keyvalue tokens; while for each image query, it interacts with all text tokens and local key-value tokens fallen in a local neighborhood around it. Since the number

Attention

…

Attention

Attention

Attention

Text Query

Image Query

𝑦

Text

Features

Key-Value for

Queries

x

respectively

Module

Trainable

Text Key-Value

of text tokens and the local neighborhood size remains constant as resolution increases, the overall complexity scales linearly with the number of image tokens.

Unlike Neighborhood Attention and standard 2D convolution, which use a square sliding window, CLEAR employs circular windows, selecting key-value tokens within a predefined Euclidean radius r for each query. Compared with corresponding square windows, the computation overhead introduced by this design is ∼ π 4 times. Formally, the attention mask M is constructed as follows:

where n text denotes the number of text tokens. Fig. 6 illustrates this paradigm.

3.3 Training and Optimization

Although each query only has access to tokens within a local window, stacking multiple Transformer blocks enables each token to gradually capture holistic information-similar to the way convolutional neural networks operate. To promote functional consistency between models before and after finetuning, we employ a knowledge distillation objective during the fine-tuning process. Specifically, the conventional flow matching loss [17, 38] is included:

where z 0 is denotes the feature of an image x encoded with a pre-trained VAE encoder E ( · ) while z t is its noisy version at the t -th timestep, y is the text condition, and ϵ θ ( · ) is the DiT backbone for denoising with parameters θ . Beyond that, we encourage consistency between the linearized student model and the original teacher model, in terms of predictions and attention outputs:

where θ org denotes parameters of the original teacher DiT, L is the number of attention layers applying the loss term, and the superscript ( l ) indicates the layer index. The training objectives can be written as:

where α and β are hyper-parameters controlling the weights of the corresponding loss terms. Only parameters in the attention layers are trainable.

For the training data, we find that training on samples generated by the original DiT model yields significantly better results than training on a real image dataset, even when the real dataset contains much more higher-quality data. Please refer to Sec. 4.3 for more discussions.

3.4 Multi-GPU Parallel Inference

Since attention is confined to a local window around each query, CLEAR offers greater efficiency for multi-GPU patch-wise parallel inference compared to the full attention in the original DiTs, which is particularly valuable for generating ultra-high-resolution images. Specifically, each GPU is responsible for processing an image patch, and the GPU communication is only required in the boundary areas. In other

𝑁 = 2

𝑁 = 4

𝑁 = 8

𝑁 = 1

𝑂

!"#!

= softmax

𝑄

!"#!

𝐾

$

𝑐

𝑉

𝑂

!"#!

≈

1

𝑁 1

%&'

(

softmax

𝑄

!"#!

𝐾

%

$

𝑐

𝑉

%

words, if we divide a H × W feature map into N patches along the vertical dimension, with each GPU handling a H N × W patch, the communication cost for image tokens between each adjacent GPUs is O ( r × W ) in CLEAR comparing with O ( H × W ) in the original DiT.

Nevertheless, since each text token requires information from all image tokens, the exact attention computation in CLEAR still necessitates synchronization of all key-value tokens specially for text tokens, which compromises its potential in this regard. Fortunately, as shown in Fig. 7, we empirically find that without any training or adaptation, the original attention computation for text tokens can be effectively approximated by a patch-wise average while not hurting the performance too much, i.e. ,

where p is the patch/GPU index. Consequently, we only need to aggregate attention outputs for text tokens, resulting in a constant communication cost and eliminating the need to transmit all tokens.

Moreover, our pipeline is orthogonal to existing strategies for patch parallelism such as Distrifusion [35], which applies asynchronous computation and communication by using staled feature maps. Building CLEAR on top of these optimizations would lead to even greater acceleration.

4 Experiments

4.1 Settings and Implementation Details

In this paper, we primarily conduct experiments with the FLUX model series due to its state-of-the-art performance in text-to-image generation. Studies on more DiTs can be found in the appendix. We replace all the attention layers in FLUX-1.dev with the proposed CLEAR and experiment with three various window sizes with r = 8 , r = 16 , and r = 32 . Leveraging FlexAttention in PyTorch [43], CLEAR, as a sparse attention mechanism, can be efficiently implemented on GPUs with low-level optimizations. We fine-tune parameters in attention layers on 10K samples with 1024 × 1024 resolution generated by FLUX-1.dev itself for 10K iterations under a total batch size 32 using the loss function defined in Eq. 6. L attn is applied on single_transformer_blocks of FLUX, whose layer indices are 20 ∼ 57 . Following previous works on architectural distillation for diffusion models [32, 39], both hyper-parameters α and β are set as 0 . 5 . Other hyper-parameters, including schedulers, optimizers, etc , follow the default settings provided by Diffusers [57]. The training is conducted on 4 H100 GPUs supported by DeepSpeed ZeRO-2 [48], which takes ∼ 1 day to finish. Unless otherwise specified, all inference is conducted on a single H100 GPU.

Following previous works [35, 39], we quantitatively study the proposed method on the validation set of COCO2014 [37] and randomly sample 5,000 images along with their prompts for evaluation. Since CLEAR aims to linearize a pre-trained DiT, we also benchmark our method against the results by the original DiT using consistent random seeds. Following conventions [53, 33, 60, 66], we consider FID [24], LPIPS [70], CLIP image similarity [47], and DINO image similarity [4] in this setting as metrics. For settings requiring pixel-wise alignment like image upsampling and ControlNet [69], we additionally incorporate PSNR [27] and multi-scale SSIM [59] for reference. While comparing with real images in COCO, we only include FID and LPIPS for distributional distances. Furthermore,

CLEAR

(

𝑟 = 8

)

CLEAR

(

𝑟 = 16

)

CLEAR

(

𝑟 = 32

)

Original

FLUX-1.dev

Victorian woman in

elaborate dress, grand

ballroom setting, rich

colors, elegant and

historical

golden hour beach with

crashing waves, warm

sunlight, peaceful and

idyllic setting

bustling city street

market, colorful

stalls, people

shopping, lively

atmosphere

steampunk airship over

city, gears and cogs,

Victorian architecture,

smokey atmosphere

colorful coral reef

underwater, exotic

fish, clear blue water,

vibrant marine life

portrait of elderly man

with weathered face,

strong character,

natural lighting, black

and white

CLIP text similarity [47], Inception Score (IS) [54], and the number of floating point operations (FLOPS) are adopted to reflect textual alignment, general image quality, and computational burden, respectively. Text prompts for qualitative examples are generated by GPT-4o.

4.2 Main Comparisons

We aim to linearize a pre-trained DiT in this paper, and the linearized model is expected to perform comparably to the original one. As illustrated in Sec. 3.1, most efficient attention algorithms result in suboptimal performance for the target problem, as quantitatively supported by the evaluation in Tab. 2. In contrast, the proposed convolution-like linearization strategy achieves comparable or even superior performance to the original FLUX-1.dev while requiring less computation, underscoring its potential for effectively linearizing pre-trained DiTs. Please refer to the appendix for an analysis of training dynamics and convergence.

With the knowledge distillation loss terms defined in Eq. 5, the differences between the outputs of the linearized models and the original model are further minimized. More evaluations with the GenEval benchmark [19] and GPT scores can be found in the appendix. Qualitatively, as shown in Fig. 8, the linearized models produced by CLEAR yield comparable results.

For efficiency, CLEAR significantly reduces FLOPS compared to the original FLUX. However, due to the complexity of kernel implementation, as shown in Fig. 2, its practical speedup does not fully match the theoretical gains. Nevertheless, at 1K resolution, CLEAR with r = 8 and r = 16 still outperforms the original FLUX, with acceleration gains increasing as the resolution scales up.

4.3 Empirical Studies

In this part, we examine several noteworthy properties of CLEAR, including resolution extrapolation, zero-shot generalization across different models and plugins, multi-GPU parallel inference, and the effects of various training data.

Resolution Extrapolation. One of the key advantages of a linearized diffusion model is its ability to efficiently generate ultra-high-resolution images [39]. However, many previous studies have revealed that it is challenging for diffusion models to generate images beyond their native resolution during training. [15, 23, 2, 61, 16]. They thus apply a practical solution for generating high-resolution

1024×1024

4096×4096

4096×4096

1024×1024

1024×1024

4096×4096

CLEAR (

𝑟 = 16

)

Strength = 0.5

Strength = 0.7

CLEAR (

𝑟 = 16

)

FLUX.1-schnell

1024×1024

1024×1024

CLEAR (

𝑟 = 8

)

CLEAR (

𝑟 = 32

)

CLEAR (

𝑟 = 16

) w. ControlNet

G.T.

Cond.

a man in armor with a beard

ancient warrior holding torch in dark cave, firelight shadows, intense and adventurous vibe

cyclist on a mountain road at sunrise, misty hills in background, dynamic and adventurous

images in a coarse-to-fine manner and devise adaptive strategies for components such as position embeddings and attention scales. The proposed CLEAR, on the other hand, makes architectural modifications to a pre-trained diffusion backbone, making it seamlessly applicable to them.

In this paper, we adopt SDEdit [41], a simple yet effective baseline adapting an image to a larger scale, for generating high-resolution images. In addition, we also enlarge the NTK factor of rotary position embeddings from 1 to 10 following [45], balance the entropy shift of attention using a logscale attention factor [30], and disable the resolution-aware dynamic shifting [17] in the denoising scheduler. By adjusting the editing strength in SDEdit, as shown in Fig. 9(left), we can effectively control the trade-off between fine details and content preservation. In the appendix, we also try building CLEAR on top of various methods for resolution extrapolation.

Quantitatively, we measure the dependency between results by CLEAR with those by the original FLUX-1.dev. As shown in Tab. 3, we achieve MS-SSIM scores as high as 0.9, showcasing the effectiveness of the linearized model with CLEAR as an efficient alternative to the original FLUX.

Circular Neighborhood vs Square Neighborhood vs Standard Convolution. Compared with Neighborhood Attention [22], a noticeable difference in CLEAR is its circular neighborhood instead of a square. Computationally, the FLOPS of a circular neighborhood with radius r is ∼ π 4 × that of a square neighborhood with a side length 2 r -1 . We empirically find that they achieve comparable performance, as shown in Tab. 5. We also experiment with the standard convolution and find that it fails to generate visually plausible images, which highlights the importance of formulation consistency mentioned in Sec. 3.1.

Compatibility with DiT Plugins. It is favorable that substituting the original attention layers with linearized ones would not impact the functionality of plugins trained for the original DiT. As shown in Fig. 9(right), CLEAR exhibits this property by supporting the pre-trained ControlNet [69] using grayscale images as a condition for FLUX-1.dev. Quantitative results can be found in Tab. 4, including grayscale images, tiled images, and blur images as conditions, respectively.

Multi-GPU Parallel Inference. A linear complexity DiT can inherently support patch-wise multiGPU parallel inference. In practice, for text-image joint attention used in modern DiT architectures [17, 34], we have to figure out a communication-efficient solution to handle text tokens, which requires gathering information from all key-value tokens. In Eq. 7, we propose an approximation

based on patch-wise averaging and validate its effectiveness quantitatively in Tab. 6. Results indicate a high correlation in semantics before and after this approximation, demonstrating its practical effectiveness.

For CLEAR, since there are only token interactions in the boundary areas of the patch handled by each GPU and the approximation of feature aggregation for text tokens defined in Eq. 7, we achieve satisfactory efficiency on multi-GPU parallel inference. Furthermore, we can apply the asynchronous communication strategy in Distrifusion [35] to achieve even more significant acceleration. As shown in Tab. 7, the acceleration becomes more significant with the increase of image resolution, while the original DiT encounters an out-of-memory (OOM) error due to the necessity of caching all key-value tokens.

5 Conclusions

In this paper, we present CLEAR, a convolution-like local attention strategy that effectively linearizes the attention mechanism in pre-trained Diffusion Transformers (DiTs), making them significantly more efficient for high-resolution image generation. Specifically, we identified four key elements-locality, formulation consistency, high-rank attention maps, and feature integrity-that are essential for successful linearization in the context of pre-trained DiTs. CLEAR leverages these principles by restricting attention to a circular local field around each query, achieving linear complexity while retaining high-quality results comparable to the original model. Our experiments demonstrate that fine-tuning on merely 10K self-generated samples allows for efficient knowledge transfer to a student model, leading to a 99.5% reduction in attention computations and a 6 . 3 × acceleration in 8K-resolution image generation. Moreover, CLEAR, once trained, supports zero-shot generalization across different models and plugins and improves multi-GPU parallel inference capabilities, offering broader applicability and scalability.

Acknowledgment

This project is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: MOE-T2EP20122-0006).

References

Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 22669-22679, 2023.

Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. In International Conference on Machine Learning , pages 1737-1752. PMLR, 2023.

Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV) , 2021.

Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixartσ : Weak-to-strong training of diffusion transformer for 4k text-to-image generation, 2024.

Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixartα : Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426 , 2023.

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019.

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794 , 2020.

Djork-Arné Clevert. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 , 2015.

Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 , 2023.

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems , 35:16344-16359, 2022.

Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060 , 2024.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780-8794, 2021.

Ruoyi Du, Dongliang Chang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. Demofusion: Democratising high-resolution image generation with no $$$. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6159-6168, 2024.

Ruoyi Du, Dongyang Liu, Le Zhuo, Qin Qi, Hongsheng Li, Zhanyu Ma, and Peng Gao. I-max: Maximize the resolution potential of pre-trained rectified flow transformers with projected flow. arXiv preprint arXiv:2410.07536 , 2024.

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning , 2024.

Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.05945 , 2024.

Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems , 36:52132-52152, 2023.

Dongchen Han, Ziyi Wang, Zhuofan Xia, Yizeng Han, Yifan Pu, Chunjiang Ge, Jun Song, Shiji Song, Bo Zheng, and Gao Huang. Demystify mamba in vision: A linear attention perspective. arXiv preprint arXiv:2405.16605 , 2024.

Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Siyuan Pan, Pengfei Wan, Shiji Song, and Gao Huang. Agent attention: On the integration of softmax and linear attention. In European Conference on Computer Vision , pages 124-140. Springer, 2025.

Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Yingqing He, Shaoshu Yang, Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang, Ran He, Qifeng Chen, and Ying Shan. Scalecrafter: Tuning-free higher-resolution visual generation with diffusion models. In The Twelfth International Conference on Learning Representations , 2024.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition , pages 2366-2369. IEEE, 2010.

Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In International conference on machine learning , pages 9099-9117. PMLR, 2022.

Linjiang Huang, Rongyao Fang, Aiping Zhang, Guanglu Song, Si Liu, Yu Liu, and Hongsheng Li. Fouriscale: A frequency perspective on training-free high-resolution image synthesis. arXiv preprint arXiv:2403.12963 , 2024.

Zhiyu Jin, Xuli Shen, Bin Li, and Xiangyang Xue. Training-free diffusion model adaptation for variable-sized text-to-image synthesis. Advances in Neural Information Processing Systems , 36:70847-70860, 2023.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning , pages 5156-5165. PMLR, 2020.

Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi. Bk-sdm: Architecturally compressed stable diffusion for efficient text-to-image generation. In Workshop on Efficient Systems for Foundation Models@ ICML2023 , 2023.

Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multiconcept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1931-1941, 2023.

Black Forest Labs. Flux: Official inference repository for flux.1 models, 2024. Accessed: 2024-11-12.

Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7183-7193, 2024.

Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint arXiv:2405.08748 , 2024.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 , 2022.

Songhua Liu, Weihao Yu, Zhenxiong Tan, and Xinchao Wang. Linfusion: 1 gpu, 1 minute, 16k image. arXiv preprint arXiv:2409.02097 , 2024.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 , 2021.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

Bowen Peng and Jeffrey Quesnelle. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation, 2023.

Hao Peng, Jungo Kasai, Nikolaos Pappas, Dani Yogatama, Zhaofeng Wu, Lingpeng Kong, Roy Schwartz, and Noah A Smith. Abc: Attention with bounded-memory control. arXiv preprint arXiv:2110.02488 , 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1-16. IEEE, 2020.

Jason Ramapuram, Federico Danieli, Eeshan Dhekane, Floris Weers, Dan Busbridge, Pierre Ablin, Tatiana Likhomanenko, Jagrit Digani, Zijin Gu, Amitis Shidani, et al. Theory, analysis, and best practices for sigmoid self-attention. arXiv preprint arXiv:2409.04431 , 2024.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 , pages 234-241. Springer, 2015.

Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics , 9:53-68, 2021.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22500-22510, 2023.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems , 29, 2016.

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 568:127063, 2024.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/ diffusers , 2022.

Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 , 2020.

Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 , volume 2, pages 1398-1402. Ieee, 2003.

Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302.13848 , 2023.

Haoning Wu, Shaocheng Shen, Qiang Hu, Xiaoyun Zhang, Ya Zhang, and Yanfeng Wang. Megafusion: Extend diffusion models towards higher-resolution image generation without further tuning. arXiv preprint arXiv:2408.11001 , 2024.

Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Yujun Lin, Zhekai Zhang, Muyang Li, Yao Lu, and Song Han. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410.10629 , 2024.

Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 14138-14148, 2021.

Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys , 56(4):1-39, 2023.

Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635 , 2023.

Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. 2023.

Haoran You, Yichao Fu, Zheng Wang, Amir Yazdanbakhsh, et al. When linear attention meets autoregressive decoding: Towards more effective and efficient linearized large language models. arXiv preprint arXiv:2406.07368 , 2024.

Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems , 33:17283-17297, 2020.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3836-3847, 2023.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Yu Zhang, Songlin Yang, Ruijie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, et al. Gated slot attention for efficient linear-time sequence modeling. arXiv preprint arXiv:2409.07146 , 2024.

A Details of Efficient Attention Alternatives

The vanilla scaled dot-product attention, although effective and flexible, introduces quadratic computational complexity. Many works have focused on its efficient alternatives. In Sec. 2, we provide a taxonomic overview of recent works and will supplement more details regarding the specific formulations and implementations here.

Linear Attention avoids the softmax operation in the vanilla attention, supporting computing K ⊤ V first with the associative property of matrix-wise multiplication, and thus achieves linear complexity. Before that, non-negative kernel functions f ( · ) and g ( · ) are applied on Q and K respectively such that the similarity between each query-key pair is non-negative. Furthermore, the similarity score between each query-key pair is normalized by the sum of similarity scores of between this query and all key tokens separately, to mimic the functionalities of softmax . Following [31, 20, 39], we implement f ( · ) and g ( · ) by the elu function [9]. Formally, the operation for the i -th query can be written as 2 :

Sigmoid Attention replaces the softmax with the formulation of sigmoid , which removes the need to compute the softmax normalization, and thus achieves acceleration:

where b is a hyper-parameter. In this paper, we follow the official implementation of FlashSigmoid with hardware-aware optimization 3 when applying Sigmoid Attention to DiTs.

PixArt-Sigma achieves acceleration by spatially down-sampling the key-value token maps [5]. Following the official implementation 4 , we use learnable group-wise Conv4 × 4 kernels with stride = 4 and initialize the weights to 1 16 so that it is equivalent to an average pooling operation at the beginning. Formally, it can be written as:

Although it has been demonstrated that such a strategy can work well at relatively deep layers of DiTs, the results are still unsatisfactory for a completely linearized DiT.

Agent Attention performs efficient attention operations via agent tokens A from a down-sampled query token map [21]:

The derived agent tokens A are then used as value tokens:

Such operations can be viewed as an adaptive token down-sampling strategy.

Slot Attention implemented in this paper is adapted from [71, 46], which contain s key-value memory slots derived by adaptively aggregating key-value tokens:

where P ∈ R s × c is learnable and introduced for modeling the writing intensity of each input token to each memory slot. These slots are then used as alternatives to original key-value tokens for attention computation:

This strategy presents a different fashion for adaptive key-value compression.

Strided Attention samples tokens at a regular interval [7]. As a sparse attention strategy, the attention mask of the l -th layer with a down-sampling ratio of r × r can be constructed in the following way:

where r x = l % r and r y = l//r ensure that each token has a chance to be sampled as key-value tokens.

Swin Transformer adopts a sliding window partition strategy [40], where attention interactions are independently conducted for each window. Formally, the attention map can be constructed via:

We set the window size to 16 and apply a shift of 8 for windows in layers with odd indices, following the approach described in the original manuscript. The rank of the image-to-image attention mask corresponds to the number of windows, which poses challenges in achieving the high-rank requirement introduced in the main manuscript needed for linearizing DiTs. In other words, all tokens within the same window share the same set of key and value tokens. This results in many duplicate rows in M , significantly reducing its rank.

Neighborhood Attention performs like convolution: for each query, it only samples key-value tokens within a neighborhood sliding window:

Comparing with Swin Transformer, each query token in neighborhood attention has a distinct key and value token set, and each row in M is linearly independent of each other, which ensures the high-rank property. The formulation used in CLEAR is developed based on such Neighborhood Attention. The difference is in the functions of checking whether a key-value token is in the neighborhood of a given query token, as shown in Eq. 3.

For CLEAR, we provide a further illustration of the r = 4 case in Fig. 10 for better clarity.

B Training Dynamics

We supplement the curves of training losses of various efficient attention alternatives in Fig. 11. The conclusion is consistent with the main manuscript, that strategies fulfilling the requirements of locality, formulation consistency, high-rank attention map, and feature integrity yield the most satisfactory training convergence.

C Raw Data for Efficiency Comparisons

We supplement raw data for Fig. 2 on efficiency comparisons in Tab. 8 for better clarity.

Query

Key / Value

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

32

25

18

13

10

9

10

13

18

20

13

8

5

4

5

8

13

17

10

5

2

1

2

5

10

16

9

4

1

0

1

4

9

17

10

5

2

1

2

5

10

20

13

8

5

4

5

8

13

35

18

13

10

9

10

13

18

32

25

20

17

16

17

20

D Circular Neighborhood or Square Neighborhood

In fact, circular or square windows do not fundamentally affect the performance and the linearization properties. Our main conclusion of the manuscript is the four key factors for successful linearization of pre-trained DiTs. The standard neighborhood attention, which follows these four principles, is definitely effective, and so is the introduced method based on circular windows. They yield

1024×1024

4096×4096

w. SDEdit

4096×4096

comparable performance as mentioned in Tab. 5. Given that the FLOPS can be reduced largely while preserving the performance, which is a free-lunch benefit, we apply the latter form. As shown in Tab. 11, it is indeed more efficient in terms of wall-clock time, especially at higher resolutions.

E Results on More DiTs

We additionally deploy our method on DiT models other than FLUX used in the main manuscript to demonstrate the universality of the proposed CLEAR. Here, we consider StableDiffusion3.5-Large 5 [17] (SD3.5-L), another state-of-theart text-to-image generation DiT. We use the default setting of r =

16 , which yields the best trade-off between quality and efficiency according to our experiments. Results on the COCO2014 validation dataset are shown in Tab. 16.

We also supplement more qualitative comparisons with results by the original FLUX-1.dev and SD3.5-L in Fig. 15. Results indicate an overall comparable performance. Due to the absence of explicit long-distance token interactions, our method may underperform in capturing overall structural properties, such as potential symmetry. Additionally involving more or less global tokens, such as down-sampled tokens as employed in PixArt-Sigma [5], could potentially mitigate this issue. However, as the primary objective of this paper is to highlight the significance of locality as a simple yet effective baseline, we leave detailed design explorations to future work.

F GPT Evaluation

Following prior work [5, 16], we use GPT as a human-like evaluator to assess the quality of generated images across three aspects: aesthetics, prompt alignment, and overall quality. Here, we consider three candidates: the original FLUX-1.dev, its linearized version with CLEAR ( r = 16 ), and SANA [62], a linear-complexity text-to-image DiT trained from scratch. We further instruct GPT-4o to compare CLEAR separately against the original FLUX-1.dev and SANA, reporting the win rates. Results across various resolutions are presented in Tab. 10, further demonstrating the effectiveness and superior performance of the proposed convolution-like linearization approach.

G More High-Resolution Results

In the main manuscript, we build our CLEAR on top of SDEdit [41], a simple yet effective strategy for image generation given a conditional image, for coarse-to-fine high-resolution generation. We demonstrate here that our method is also compatible with a variety of pipelines dedicated to resolution extrapolation. As shown in Fig. 12, we deploy CLEAR on I-Max [16], a concurrent work for training-free high-resolution generation with pre-trained DiTs, and observe that it may yield a more optimal balance between preserving low-resolution content and capturing high-resolution details. For instance, as shown in Fig. 12, I-Max preserves the textures of the dresses from the low-resolution result with minimal variation while effectively enhancing clear high-resolution details.

In fact, it requires two steps to achieve efficient high-resolution generation. The first is to adapt the original DiT to make it effective at higher scales. The second is to make it more efficient. CLEAR mainly addresses the second step. For the first step, the techniques are mainly adapted from previous works, e.g. , SDEdit and NTK rotary embedding. That is why we do not include so many details about them.

Nevertheless, we fully agree with the reviewer that it would be helpful to supplement the related details. We include some key information here and will further elaborate on it in the revision.

SDEdit is a simple yet effective training-free method for image editing based solely on a pre-trained text-to-image diffusion model, which can be used for high-resolution generation in a coarse-to-fine manner. Specifically, we first generate an image at the native resolution scale of the diffusion model. Then, we resize it to a larger size and add a certain noise to it. A noise scale of 0.7 is adopted empirically in our experiments. Starting from this point, the model conducts the remaining denoising steps. In this way, the original image structures are preserved and low-level details are refined.

NTK rotary embedding applies a scaling factor s to the rotary base b used for rotary positional embedding, i.e. , b ′ = bs , where s = √ L cur L native and L cur is the current sequence length while L native is the native length seen in the training time.

Resolution-aware dynamic shifting applies a factor s to the projection function of the current denoising time step t for the flow matching scheduler: t ′ = st 1+( s -1) t , where s is positively related to the image resolution. When this is enabled, at a high resolution, the projection function would appear too 'skew', i.e. , the number of denoising steps allocated to the stage of high noise level is insufficient.

Regarding the three factors, we conduct the GPT evaluation following the same protocol in Tab. 10, and the results are shown in Tab. 13, highlighting their effectiveness.

a photo of a bench

CLEAR (r=8) 4.38s

a photo of a horse and a train

CLEAR (r=8) 4.38s

H Cross-DiT Generalization

We empirically find that the trained CLEAR layers for one DiT are also applicable for others within the same series without any further adaptation efforts. For example, as shown in Fig. 9(middle), the CLEAR layers trained on FLUX-1.dev can be directly applied to inference on FLUX-1.schnell, a timestep-distilled DiT supporting 4-step inference, yielding results similar to those of the original FLUX-1.schnell. Such zero-shot generalization is quantitatively evaluated in Tab. 9.

I Hybrid Attention Extension

In practice, we observe that a small r , e.g. , r = 8 , tends to produce results with inconsistent layouts and multi-objects, as shown in Fig. 13. As an extension, we propose a hybrid structure that incorporates coarse tokens as additional key-value tokens to enhance holistic interactions. Similar to [5], we conduct 8 × downsampling to the key-value maps to derive these global tokens. As shown in Fig. 13 and Tab. 12, we find that these issues can be effectively resolved with minimal additional computational overhead, underscoring the practical benefits. However, as this paper primarily emphasizes the importance of local tokens, we do not adopt this strategy by default.

J Studies on Loss Weights

In this part, we study the individual contributions of the two additional loss terms in Eq. 6 by varying the weights α and β . In general, both loss terms contribute to the consistency between the distilled and original models. Nevertheless, we find that they are complementary:

α controls the weight of L pred , regulating the consistency between the final output results, which, according to our experiments shown in the grid search in Tab. 14, appears to be relatively more effective than L attn . For example, the cases of α > 0 , β = 0 generally yield better FiD scores than those of α = 0 , β > 0 .

β controls the weight of L attn , regulating the consistency between the intermediate attention results, which, according to our experiments, serves as an auxiliary measure to facilitate the training convergence. To verify this, we present the values of the flow matching loss L fm after moving average at various steps in the training progress in Fig. 14. Results indicate that L attn can facilitate training by offering informative intermediate supervision signals beyond the relatively late supervision provided by L pred at the final outputs.

10

100

1K

2K

10K

Training Steps

0.350

0.375

0.400

0.425

0.450

0.475

0.500

Flow Matching Loss

w/o

attn

w

attn

Overall, applying both terms achieves the best performance, and the performance is insensitive to the specific values of the hyper-parameters according to the grid search. There can be some subtle tradeoffs between the FID and CLIP-T metrics, where the default values achieve a good balance.

K Generalizability across Different r

Qualitatively, we find that the model trained with a smaller window radius r can be used for a larger one. However, the opposite case does not work. We speculate that it is because the information provided for the model is complete for inference with additional clues when r is larger, while the information would be insufficient if r becomes smaller. We supplement the quantitative results of GPT evaluation in Tab. 15.

L Limitation

One limitation of our approach is that the practical acceleration achieved by CLEAR does not fully meet the theoretical expectations indicated by FLOPS. It becomes less significant at relatively low resolutions and can even be slower than the original DiT when the resolution is below 1024 × 1024 . This drawback arises partially because hardware optimization for sparse attention is inherently more challenging than the optimizations achieved by FlashAttention for full attention computation. Addressing this limitation may require developing fused CUDA operators specifically optimized for the specific sparse pattern of CLEAR, which is a valuable direction for future works.

FLUX-1.dev

StableDiffusion3.5-Large

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction, we have clarified the focused problem and our contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in the appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We mention our experiment settings in Sec. 4 and include the detailed configurations in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We attach the source codes to reproduce the main experimental results as supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We mention our experiment settings in Sec. 4 and include the detailed configurations in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Following the evaluation convention for text-to-image generation, only average results are reported.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We mention the related information in Sec. 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the guidelines and confirm that the conducted research follows them.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the impacts in the appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: The applied safeguards are discussed in the appendix.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The license and terms of use of the used assets are respected and mentioned in the appendix.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims at a linear attention mechanism that reduces the complexity of pre-trained DiTs to linear. This paper discusses 4 specific features that are essential for successfully linearizing pre-trained DiTs: 1) Locality, 2) Formulation Consistency, 3) High-Rank Attention Maps, and 4) Feature Integrity. Based on the 4 feature, this paper proposes CLEAR, which based on MM-DiT architectures: for each text query, it still gathers features from all text and image key-value tokens; while for each image query, it interacts with all text tokens and local key-value tokens fallen in a local neighborhood around it. Unlike Neighborhood Attention and standard 2D convolution, which use a square sliding window, CLEAR employs circular windows.\n\nThe student model is trained by fine-tuning the attention layer on 10K self-generated samples for 10K iterations. Experiments show that the distilled local attention layers are also compatible with different variants of the teacher model, e.g., FLUX.1-dev and FLUX.1-schnell, and various pre-trained plugins like ControlNet without requiring any adaptation. Another side benefits of locality is multi-GPU patch-wise parallel inference.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n\n1. The proposed method significantly reduces the required generation computation and time for large images.\n2. The analysis of the 4 features are nice, except for locality.\n\n# Weakness:\n\n1. The analysis of the importance of locality is questionable. Perturbing an image of \"A small blue plane sitting on top of a field\" may not get the conclusion that local features are more important, because this specific prompt and image itself does not have long dependency. An good example to demonstrate locality should be the one in Figure 4. This image shows strong long dependency. If locality is also more important here, we can thus conclude. In particular, in Figure 4, local attention seems not to make any better sense than the global attention.\n2. The improvement compared with Neighborhood Attention seems questionable. Although the GFLOPS decrease by 12.5%, according to Figure 2, there is only a very limited improvement in real wall-time speed.\n3. The proposed method seems underperform Swin Transformer given the same GFLOPS.\n\nQUESTIONS:\n1. Compared with Neighborhood Attention, what is the wall-time speed improvement?\n2., Except for the GFLOPS, is there any other advantages over Neighborhood Attention? Can Neighborhood Attention also be used for DiT Plugins, and Multi-GPU Parallel Inference?\n3. In Table 2, why increasing radius significantly improves the FiD against original FLUX-1.dev, while has no almost no impact against real images? What are the vital changes between them?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a convolution-like local attention mechanism called CLEAR, designed to linearize existing DiT models. By restricting each query token to interact only with tokens within its local neighborhood during attention computation, CLEAR significantly accelerates high-resolution image generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The paper thoroughly analyzes various efficient attention mechanisms and conducts comprehensive evaluations and validations on image generation performance at a resolution of 1024×1024.\n\n2. While maintaining generation quality, CLEAR achieves a 99.5% reduction in attention computation and a 6.3× speed-up, demonstrating particularly significant advantages in 8K image generation tasks.\n\nWeaknesses\n\n1. In Table 2, are the different efficient attention methods also evaluated based on distilled models from FLUX-1.dev with their attention layers replaced accordingly? Additionally, since both Swin attention and CLEAR adopt local attention, differing mainly in window shape, is it theoretically possible to adjust Swin’s window size or shape to approximate CLEAR’s performance?\n\n2. Is the model’s ability to generate 8K-resolution images primarily distilled from FLUX-1.dev, or is it achieved through additional data-specific fine-tuning? Moreover, how does the method address the common issue of repeated content or artifacts in high-resolution image generation?\n\n3. The ablation study appears insufficient, as the individual contributions of the two additional loss terms introduced (prediction loss and attention loss) are not separately verified. Also, does each choice of attention radius 𝑟 require training a separate model, or can one model generalize across different domain sizes?\n\nQUESTIONS:\nMy major concerns are the weakness. I would like the authors to directly address these concerns in their response. My final rating is subject to change based on the quality and clarity of the authors’ feedback.\n\nLIMITATIONS:\nThe authors are encouraged to answer the questions and address the weakness above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates efficient attention mechanisms for Diffusion Transformers (DiTs), with a focus on adapting them for high-resolution image generation. The authors begin with a systematic analysis of existing efficient attention variants, identifying locality as the most promising design principle for this context. Building on this insight, they propose a circular local attention mechanism that uses a fixed-radius mask to approximate convolution-like inductive bias while retaining transformer flexibility. The method enables a linear-time attention variant and is further enhanced via knowledge distillation, achieving strong efficiency while minimizing performance degradation compared to the teacher model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper provides a detailed discussion of existing efficient attention mechanisms and analyzes their suitability in diffusion models. This comparative analysis offers valuable insight into the design space of efficient attention for image generation and strengthens the motivation for the proposed method.\n\n- The experimental section is comprehensive and well-structured, with comparisons across multiple baselines and resolutions. The reported results are convincing and support the core claims.\n\n- The paper includes thorough implementation details, making the proposed method highly reproducible and potentially practical for real-world deployment.\n\nWeaknesses:\n\n- The proposed circular local attention mask offers limited novelty compared to existing square-shaped attention masks. Empirically, the observed improvements are marginal, which raises questions about the necessity of the circular design.\n\n- While the paper discusses extending the method to ultra-high-resolution image generation (e.g., 2K and 4K) using SDEdit, NTK rotary embeddings, and disabling resolution-aware dynamic shifting, the technical details of this pipeline are insufficiently explained. Moreover, the paper lacks ablation studies isolating the impact of these components. Given that CLEAR's claimed speedup at ultra-high resolutions is one of its major selling points, the underexplored treatment of this aspect undermines the strength of the argument.\n\nQUESTIONS:\n1. Could the authors further elaborate on the advantages of the proposed circular attention mask?\nWhile the design introduces a novel geometric locality, its empirical benefits over conventional square masks (e.g., those used in Neighborhood Attention) appear marginal based on current results. A clearer justification—whether theoretical (e.g., isotropy, boundary continuity) or task-specific—would help clarify the necessity and value of this design.\n\n2. Can the authors provide more implementation details and design considerations regarding the application of their method to ultra-high-resolution generation?\nThe paper mentions several techniques (e.g., SDEdit, NTK rotary embedding, disabling resolution-aware dynamic shifting) used to enable high-resolution generation, but these are not described in sufficient depth. A more detailed technical discussion would improve the clarity and reproducibility of the proposed pipeline.\n\n3. Could the authors include ablation studies specifically focused on ultra-high-resolution settings?\nSince CLEAR’s improved efficiency at 2K–4K resolutions is one of its key claims, it would be valuable to see component-wise evaluations (e.g., NTK rotary embedding, disabling resolution-aware dynamic shifting) at these resolutions to understand which factors contribute most to quality and speed in this context.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-28"}

comments: SUMMARY:
In the paper, the authors identify four key factors for linearization of pre-trained DiTs: locality, formulation consistency, high-rank attention maps, and feature integrity. And they introduce a convolution-like local attention strategy, which limits feature interactions to a local window around each query token, and achieves linear complexity. In experiments, it reduces attention computations by 99.5% and accelerates generation by 6.3 times for generating 8K-resolution images.

STRENGTHS AND WEAKNESSES:
Strengths：
1. The authors identify four key factors crucial for successful linearization of pre-trained DiTs: locality, formulation consistency, high-rank attention maps, and feature integrity.
2. They introduce a convolution-like local attention strategy, which limits feature interactions to a local window around each query token.

Weaknesses：
1. The convolution-like local attention strategy employs circular windows instead of a square that seems not indispensable. Although the FLOPS of a circular neighborhood is ∼ π/4× that of a square neighborhood, but more data operations are required, and they achieve comparable performance.
2. Among the four key elements, the high-rank attention maps and feature integrity seem not notable in the convolution-like local attention strategy.
3. The experiment on relation of window size and performance is not presented.

QUESTIONS:
1. The meaning of 'convolution-like' is not clear. What parts are convolution-like?
2. What size of the window is optimal? Isn't a larger window size always better?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-29