CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up
Authors: Songhua Liu
Zhenxiong Tan
Xinchao Wang
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Transformer
High-Resolution Image Generation
Efficient Attention
Primary area: deep_learning
TL;DR: We propose a convolution-like linearization strategy that accelerates pre-trained diffusion transformers for ultra-resolution image generation.

Abstract: Diffusion Transformers (DiT) have become a leading architecture in image generation. However, the quadratic complexity of attention mechanisms, which are responsible for modeling token-wise relationships, results in significant latency when generating high-resolution images. To address this issue, we aim at a linear attention mechanism in this paper that reduces the complexity of pre-trained DiTs to linear. We begin our exploration with a comprehensive summary of existing efficient attention mechanisms and identify four key factors crucial for successful linearization of pre-trained DiTs: locality, formulation consistency, high-rank attention maps, and feature integrity. Based on these insights, we introduce a convolution-like local attention strategy termed CLEAR, which limits feature interactions to a local window around each query token, and thus achieves linear complexity. Our experiments indicate that, by fine-tuning the attention layer on merely 10K self-generated samples for 10K iterations, we can effectively transfer knowledge from a pre-trained DiT to a student model with linear complexity, yielding results comparable to the teacher model. Simultaneously, it reduces attention computations by 99.5% and accelerates generation by 6.3 times for generating 8K-resolution images. Furthermore, we investigate favorable properties in the distilled attention layers, such as zero-shot generalization across various models and plugins, and improved support for multi-GPU parallel inference. Models and codes will be available.

CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up
Songhua Liut.,, Zhenxiong Tan', and Xinchao Wang School of Artificial Intelligence, Shanghai Jiao Tong University, National University of Singapore liusonghuadsj tu. edu. cn, zhenxiongêu. nus edu, xinchao@nus. edu.sg
2340-4695
Kww www.
C2022 2072
2272 u01
Hello CLEAR
224-2045
4028-2244
ALAS-2HE
J036-244
Figure 1: Ultra--reoluuton results generated by the linearized FLUX. 1 1edev model with our approach CLEAR. Resolution is marked the top-right corner of each result in the format of width x height. Corresponding prompts can found the appendix.
Abstract
Diffusion Transformers (DiT) have become a leading architecture image gen- eration. However, the quadratic complexity of attention mechanisms, which are responsible for modeling tosen-wise relationships, results in significant latency when generating high-resolution images. To address this issue, aim for a linear attention mechanism in this paper that reduces the complexity of pre-trained DiTs linear. We begin exploration with comprehensive summary of existing efficient attention mechanisms and identify four key factors crucial for the success- ful linearization of pre-trained DiTs: locality. formulation consistency, high-rank attention maps, and feature integrity. Based on these insights, we introduce a convolution-liks local attention strategy termed CLEAR, which limits feature in- teractions to a local window around each query token, and thus achieves linear complexity. Our experiments indicate that by fine-tuning the attention layer on merely 10K self- self-generated samples for 10K iterations, we can effectively transfer knowledge from a pre-trained DiT a student model with linear complexity, yield- ing results comparable to those of the teacher model. Simultaneously. it reduces attention computations by 99.5% and accelerates generation by 6.3 times for gener- ating 8K-resolution images. Furthermore, we investigate favorable properties in the distilled attention layers, such as zero-shot generalization across various models and plugins, well as improved support for multi-GP Multi-GGUU parallel inference. Models and codes are available here
"Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
have gained widespread attention in text-to-image generation. proving to be highly effective for producing high- and diverse images from textual prompts [TO] 64]. Traditionally, architectures based on UNet [TT [5) have dominated this field due to their robust generative capabilities. In recent years, Diffusion Transformers (DiTs)
Diffusion models 42
50 26
17
36
18 S
have emerged as a promising alternative, achieving leading performance this field. Unlike the UNet-based architectures, DiT's leverage the attention mechanism [56] to model intricate token-wise relationships with remarkable flexibility, enabling them to capture nuanced dependencies across all tokens in images and texts, and thus produce visually rich and coherent outputs.
Despite their impressive performance, the attention layers-which model intricate pairwise token relationships with quadratic complexity- can introduce substantial latency in high-resolution image generation. As shown in Fig. 2 FLUX.I-dev [34, a state-of-the- tfr text-to-image DiT, requires over 30 minutes to generate 8K-resolution images with 20 denoising steps, even with hardware-aware optimizations like FlashAttention [12
Focusing on these drawbacks, we are curious about one question: Is it possible to convert a pre-trained DiT achieve linear complexity? ? The answer is not straightforward, in fact, as it remains unclear titaanismsdessppii their recent widespread exploration 113 whether existing efficient attention can be effectively applied to pre-trained DiTs.
65
49
58
To answer this question, we initiate our explo- ration with summary of previous methods ded- icated to efficient attention, categorizing them into three main strategies: formulation varia- tion, key-value compression, and key-value sam- pling. We then experiment with fine-tuning the model by replacing the original attention layers with these efficient alternatives. Results indicate that while formulation variation strategies have proven effective in attention-based UNets [39] and DiTs trained from scratch [62), they do not yield similar success with pre-trained DiTs. Key- value compression often leads to distorted de- tails, and key-value sampling highlights the ne- cessity of local tokens for each query to generate visually coherent results.
Sec. /Inage (Log Scala)
10-
trrs (n-66)
GF OPP/ Layer (Log Scale
18
Gara (n-16)
Eurs (r-S2)
16
-6-
Oara (P-S23
12-
10-
312
20
4.7
3k0 21¹
8c7
Number of Pixels
Number of Pixels
Figure 2: Comparison of speed and GFLOPS be-
tween the proposed linearized DiT and the original FLUX. Speed is evaluated by performing 20 denoising steps on a single H100 GPU. FLOPS is calculated with the approximation: x Σ M x e, where e is the feature dimension and M denotes the attention masks. log, is applied on both verti- cal axes for better visualization. The raw data are supplemented in the appendix.
Building on these observations, we figure out four elements crucial for for linearizing pre- trained DiT', including locality, formulation consistency, high-rank attention maps, and feature integrity. Satisfying all these criteria, we present a convolution-lice lincarization strategy termed CLEAR, where each query interacts only with tokens within a predefined distance r. Since the number of key-value tokens interacting with each query is fixed, the resulting DiT achieves linear complexity with respect to image resolution.
To our surprise, such concise design yields results comparable original FLUX.1-dev after knowledge distillation process (55 with merely 10K fine-tuning iterations on 10K generaedd samples. As shown in Fig. [ CLEAR exhibits satisfactory cross-resolution generalizability. property also reflected in UNetdbased diffusion models 22 15 23 29). For hitrrsiigh-rssltution generation like 8K, it reduces attention computations by 99.5% and accelerates the original DiT by 6.3 times, as shown in Fig. 2 The distilled local attention layers are also compatible with different variants of the teacher model, e.g., FLUX. 1ed and FLUX, I -schnell, and various pre-trained plugins like ControlNet [69 without requiring any adaptation.
As the token interactions are performed locally, it is convenient for CLEAR to support multi-GPU parallel inference. We further develop a patch-parallel paradigm that minimizes communication overhead. Our contribution can be summarized as follows:
We provide a taxonomic overview of recent efficient attention mechanisms and identify four elements essential for lincarizing pre-trained DiTs.

---

Based on them, we propose convolution- like local attention mechanism termed CLEAR as an alternative to default attention, which the first linearization strategy tailored for pre-trained DiT to the best of our knowledge.
We delve into multiple satisfactory properties of CLEAR through experiments, including comparable performance with the original DiT, linear complexity, cross-resolution generalizability, cross-model/plugin generalizability. support for parallel inference, etc.
2 Efficient Attention: A Taxonomic Overview
The attention mechanism [56 is known for its flexibility in modeling token-wise relationships. takes a query matrix Q E R a key matrix K E Ikm and a value matrix V E RRmx'' as input and produces an output matrix O E Raxe' via:
QK
0 softmax
V.
(I)
where n and m are the numbers of query and key-values tokens respectively, and c and e' the feature dimensions for query-key and value tokens. In line with standard design conventions, we assume c' throughout this paper, and in the case of self-attention, Q, K, and V come from the same feature maps with m = n..
As shown in Eq. [ self-attention involves constructing token-to-token relationships, which results in both time and memory complexity. To address this issue, numerous studies focus developing efficient attention mechanisms. In this section, we summarize recent work in this area and assess its applicability to DiT linearization. Specifically, we categorize existing approaches into three main categories: formulation variation, key-value compression, and key-value sampling.
attention maps to model pair-wise
2.1 Formulation Variation
Revisiting Eq. 1 if the sof tmax operation is omitted, we can first compute K" KVV, yielding a matrix with linear time in relation to In this way, a series of linear attention mechanisms apply kernel functions f() and g(-) Q and K respectively to mimic the effect of softmax:
O == (k)))
(2)
such as Mamba2 13
Gated Linear Attention [65], and Generalized Linear Attention [39]. An-
other mainstream of methods try replace the sof tmax operation with efficient alternatives, like s sigmoid rehu 28 67]. and Nystrom-base approximation 63
2.2 Key-Value Compression
In the default setting of self-attention, the numbers of query and key-value tokens are consistent, i.e. m = n. and the shape of the attention map would be n x n. is thus promising to compress key-value tokens that m can be smaller than n reduce the complexity. Following this routine, PixArt- Sigma (5] compress KV tokens locally with a downsampling Conv2d operator. Agent Attention [2] first conducts attention with downsampled Q and full-sized K and to select agent KV tokens for compression. Then, original Q would interact with these compressed tokens. Similarly, Slot Attention [71] adopts learnable slots obtain agent KV. Linformer (58) introduces learnable maps to obtain compressed tokens from the original ones.
2.3 Key- Value Sampling
Efficient attention based on key-value sampling is based on the assumption that not all key-value tokens are important for query and the attention matrix is highly sparse. Comparing with key-value compression, it prunes key-value tokens for each query instead of producing new tokens. For instance, Strided Attention 7 samples one key-value token at a regular interval. Routing Attention [52) samples key-value tokens based on grouping. Swin Transformer 40 divides feature maps into non- overlapping local windows and performs attention independently for each window. Neighborhood

---

Agent Attention Figure 3: Preliminary results of various efficient attention methods on FLUX-I.dev. The prompt small blue plane sitting on top of field".
Linear Attention
Signoid Actention
Downsonple w. PixArt-Signa
Strided Actention
Swin Transforner
NeLghborhood Altention
FLUX-1.ddv
Pertum
Parturt Lecsl Fecturee
Querr
Add
man(KG
min(H5]
Figure 5: We try perturbing remote and local fea- tures respectively through clipping the relative distances required for rotary position embedding. Perturbing remote features has no obvious impact image quality, whereas altering local ones re- sults in significant distortion. The text prompt and the original result consistent with Fig.
Oriptinot Inage Figure 4: Visualization of attention maps by vari- heads. Attention in pre-trained DiT's is largely conducted in local fashion.
FULl Altentton Nops
Attontton Mops
k.r.C. Query
Attention 22 selects key-value tokens within a local window around each query. BigBird 68 uses selection strategy combining neighborhood attention and random attention, and LongFormer [ combines neighborhood attention with some global tokens that are visible to all tokens.
Methods
3.1
What are Crucial for Linearizing DiTs?
Building on the overview of recent efficient attention mechanisms in Sec. we explore a key question here: What specific peaciuieeeauuees features are essential for successfully linearizing pre-trained DiTs? We thus try substituting all the attention layers FLUX.1-der with various efficient alternatives and fine-tuning parameters in these layers. The preliminary text-to-image results are shown in Fig. through which we figure out four key elements: locality, formulation consistency, high-rank attention maps, and feature integrity. According to these perspectives, we summarize some previous efficient attention methods in Tab. I
Locality indicates that key-value tokens fallen in the neighbor- hood of a query are included for attention. From Fig. 3 we observe that many meth- ods equipped with this feature yield at least plausible results, like PixArt- Swin Trans- former, and Neighborhood Atten- tion. Particularly, comparing the results of Neighborhood Attention and Strided Attention, we find that incorporating local key- value tokens diminishes lot of distorted patterns.
Formulation Consistency
High-Rank Attention Maps
Feature Integrity
Method
Locality
Lincar Attention
[55
Sigmoid Attention
Yes
No
Yes
PixArt-Sigma
1
Yes
Yes
Yes
No
Agent Atteation
Maybe
Strided Attention
?
No
Yes
Yes
Yes
Swin Transformr 10
Yes
Yos
No
Yos
Neighborhood
Table I: Summary of existing efficient attention mechanisms
based on the four factors erucial for linearizing DiTs.
The reason for these phenomena is that pre- pre-trained DiTs, such FLUX, rely heavily on local features to manage token relationships. To validate this, we visualize attention maps in Fig. and observe that the most significant attention scores fall in the local area around each query.
In Fig. 5 we provide further evidence to illustrate the importance of local features. that perturbing remote features would not damage the quality of FLUX.1-de much. Specifically, FLUX. 1-dev relies on rotary position embedding [55] to perceive spatial relationships and is sensitive to the
relative distance (d(a) (x)) on the two axes of 2D feature map, where indices and j denote query and key token indices respectively. We perturb remote features by clipping the relative
distances for rotary position embedding maximum value when they exceed this threshold,

---

Le., d.j"ccip(-rrr)) As shown in Fig. 5 leleft, the results are reasonable for a 64 x 64 r is as small as 8. Conversety, if we perturb local features by setting their
feature map wh minimum absolute distances r, even with r as small as the result still collapses as shown in Fig. Srighh)-emphasizinn the importance of locality.
Formulation Consistency denotes that the efficient attention still applies the softmax-base formu- lation of the scaled dot- dot-product attention. LinFusion [9 has shown that linear attention approaches like linear attention achieve promising results in attention-basee UNets. However,
find that it is not the case for pre-trained DiT',, as shown in Fig. 3 We speculate that it is due to attention layers being the only modules for token interactions in DiTs, unlike the case in U-Nets. Substituting all of them would have a substantial impact on the final outputs. Other formulations like Sigmoid Attention fail to converge within limited number of iterations, unable mitigate the divergence between the original and modified formulations. It is thus beneficial to maintain consistency with the original attention function.
High-Rank Attention Maps means that attention maps calculated by efficient attention alternatives should be sufficient to capture the intricate token-wiss relationships. As visualized in Fig. extensive attention scores are concentrated along the diagonal, indicating that the attention maps do not exhibit the low-rank property assumed by many prior works. That is why methods like linear attention and Swin Transformer largely produce blocky patterns.
Feature Integrity implies that raw query, key, and value features are more favorable than the compressed ones. Although PixArt-Sigma has demonstrated that applying KV compression on deep layers would not hurt the performance much, this approach is not suitable for completely linearizing pre-trained DiTs. As shown in Fig. 3 methods based on KV compression, such as PixArt-Sigma and Agent Attention, tend produce distorted textures compared to the results from Swin Transformer and Neighborhood Attention, which highlights the necessity to preserve the integrity of the raw query, key, and value tokens.
3.2 Conv-Like Lincarization
Text Key- dey-Value
Given the above analysis of the crucial factors for linearizing DiT', Neighborhood Attention is the only scheme satisfying all 4 constraints. Motivated by this, we propose CLEAR, conv-like lincarization strategy tailored for pre-trained DiTs. Specifically, given that state-of-the-art text- to-image DiTs, like FLUX and StableDiffusion 3 series, typi- cally adopt text-image joint self- attention, in CLEAR, for each text query, it still gathers fea- tures from all text and image key- value tokens; while for each im- age query, it interacts with all text tokens and local key-value tokens fallen in local neighbor- hood around it. Since the number of text tokens and the local neighborhood size remains constant as resolution increases, the overall complexity scales linearly with the number of image tokens.
Text Auoor Aba
Image Query
Teat
Feotures
Attention
Attention
Attention
Key Walue for Quertier
Frozon
Tritinatle
2 Figuré 6: Illustration the proposed convolation-like tion strategy for pre-trained DiTs. In each text-image joint atten- tion module, text queries aggregate information from all text and image tokens, while each image token gathers information only from tokens within local circular window.
Pre-Trgined DiT
lineariza-
Unlike Neighborhood Attention and standard 2D convolution, which use a square sliding window, CLEAR employs circular windows, selecting key-value tokens within predefined Euclidean radius for each query. Compared with corresponding square windows, the computation overhead introduced times. Formally, the attention mask M is constructed as follows:
by this design is
[1, 1, if S Fleent or j S Preast or ((2)2 + d < 2
Mi =000
(3)
0, otherwise,
5

---

Method/Settnng
FID
LPIPS
Against Original
CLIP-I
DINO (T)
FID
Against Real
LPIPS
CLIPT (1)
18
GELOPS (L)
Original de
34,93
0.81
31.06
38.25
260.9
Signoid Atlention
49
447.3 80
0.91
41.34
0.25
457.69
0.84
17.53
1.15
260.9
Linear Attention 133
6y
65
5T
524 54
(1.85
51.37
2.17
325.58
0.87
19.16
2.91
174.0
PixAn-Simga
30.64
86.43
71.45
31.12
32.14
67.7
Agent Altention
69.85
0.65
78.
56.09
54.31
30.38
21.03
80.5
Strided Artention
24.88
0.61
85.50
70.72
35.27
0.89
30.62
32.05
67.7
Swin Transformer
20
18.00
0.65
85.72
73.43
32.20
0.87
30.64
34,68
67.7
CLEAR (r 8)
15.53
0.64
86,47
74.36
32.06
0.83
30.69
34.47
63.5
distill
13:07
0.62
88.56
77.66
33.06
0.82
30.82
35,92
63.5
80.6
w. disnil
13.72
0.58
88.83
77.30
33.63
0.88
30.65
37.84
80.6
CLEAR - 32)
11.07
0.52
89.92
81.20
33.47
0.82
30.96
$7.80
154.1
Table 2: Quantitative results of the original CLEAR proposed this paper with various r on 5,000 images from the COCO2014 validation
previous efficient attention methods, and
dataset a resolution of 1024 x 1024.
where Ntet denotes the number of text tokens. Fig. 6 iillutrates this paradigm.
3.3 Training and Optimization
Although each query only has access to tokens within a local window, stacking multiple Transformer blocks enables each token to gradually capture holistic information- similar the way convolutional neural networks operate. To promote functional consistency between models before and after fine- tuning, we employ knowledge distillation objective during the fine-tuning process. Specifically, the conventional flow matching loss 17 38 is included:
fim II(e =0) - c0 (t, t, v.2.
where 2g is denotes the feature of an image x encoded with a pre-trained VAE encoder E(-) while 2, is its noisy version at the 1-th timestep, y is the text condition, and co(-) is the DiT backbone for denoising with parameters 0 Beyond that, we encourage consistency between the linearized student model and the original teacher model, in terms of predictions and attention outputs:
Cpred ||Eg(2e, y) - EBarg (zc, t, 21
1 Lattn =
(5)
11 (zt, 1.)) - Corg (2,1,8)11
where Oorg denotes parameters of the original teacher DiT, L is the number of attention layers applying the loss term, and the superscript indicates the layer index. The training objectives can be written as:
min (t),
aCpred + 3Lottn];
(6)
where a and are hyper-parameters controlling the weights of the corresponding loss terms. Only parameters in the attention layers are trainable.
For the training data, we find that training on samples generated by the original DiT model yields significantly better results than training on real image dataset, even when the real dataset contains much more higher-quality data. Please refer to Sec. for more discussions.
3.4 Multi-GPU Parallel Inference
Oen potra
Since attention is confined to local window around each query, CLEAR offers greater ef- ficiency for multi-GPU patch-wise parallel in- ference compared to the full attention in the original DiTs, which is particularly valuable for generating altra-high-resolution images. Specif- ically, each GPU is responsible for processing an image patch, and the GPU communication only required in the boundary areas. In other
HELLO
hello
futtle
HELLO WORLDD
R-
Figure 7: To enhance multi-GPU parallel inference, each text query aggregates only the key-value to- kens from the patch managed by its assigned GPU, then averages the attention results across all GPUs, which also generates high-quality images.

---

Setting
PSNR (*) SSIM (*) FID LPIPS (1) CLIP-I()) DINO CLIP-T (1) (1) GFLOPS
-1024
1024
2048 x 2048--
FLUX-Ldey
31.11
24.53
3507.9
CLEAR (r 8)
27.57
0.91
13.55
0.12
98.97
98.37
31.09
25.05
246.2
CLEAR 16)
27.60
0.92
13.43
0.12
98.97
98.34
31.08
25,46
352.6
CLEAR (r- 32)
28.95
0.94
10.87
0.10
99.23
98.82
31.09
25,48
724,3
-2048
2048
4096
4096-
FLUX- 1.dev
31.29
24.36
53604.4
CLEAR (r 8)
26.19
0.87
20.87
0.22
98.02
96.56
31.16
25.87
979.3
CLEAR (r = 16)
26.98
0.88
16.20
0.19
98.48
97 64
31.25
25.13
1433.2
CLEAR 32)
27.70
0.90
13.56
0.17
98.72
98.21
31.20
24.81
3141.7
Table 3:
Quantitative results of the original FLUX-1.dey and our CLEAR with various r on 1,000
images from the COCO2014 validation dataset at resolutions of 2048 2048 and 4096 4096.
words, if we divide a H x W feature map into N patches along the vertical dimension, with each GPU handling a H. GPUs is O(r x W) in CLEAR comparing with O(H x W) in the original DiT.
N x W patch, the communication cost for image tokens between each adjacent
Nevertheless, since each text token requires information from all image tokens, the exact attention computation in CLEAR still necessitates synchronization of all key-value tokens specially for text tokens, which compromises its potential in this regard. Fortunately, as shown in Fig- 7 we empirically find that without any training or adaptation, the original attention computation for text tokens can be effectively approximated by a patch-wise average while not hurting the performance too much, i.e.,
(Qeeat K,
Otenn RV
softmax
Vp
N
where is the patch/GPU index. Consequently, we only need to aggregate attention outputs for text tokens, resulting in constant communication cost and eliminating the need to transmit all tokens.
Moreover, our pipeline is orthogonal to existing strategies for patch parallelism such as Distrifu- sion 33), which applies asynchronous computation and communication by using staled feature maps. Building CLEAR on top of these optimizations would lead to even greater accelcration.
Experiments
4.1 Settings and Implementation Details
In this paper, we primarily conduct experiments with the FLUX model series due to its state-of-the-art performance in text-to-image generation. Studies on more DiTs can be found in the appendix. We replace all the attention layers in FLUX-1.dev with the proposed CLEAR and experiment with three various window sizes with r 8, r = 16, and CLEAR, as a sparse attention mechanism, can be efficiently implemented on GPUs with low-level optimizations. We fine-tune parameters in attention layers on 10K samples with 1024 x 1024 resolution generated by FLUX-1.dev itself for 10K iterations under total batch size 32 using the loss function defined in Eq. Lattn is applied on single transformer www.oorrrrrrrrrrreeeeeeeeeeeeeee of FLUX, whose layer indices are 20 57. Following previous works on architectural distillation for diffusion models [32 39), both hyper-parameters a and are set as 0.5. Other hyper-parameters. including schedulers, optimizers, etc, follow the default settings provided by Diffusers [57]. The training is
32, Leveraging FlexAttention in PyTorch [43),
conducted on H100 GPUs supported by DeepSpeed ZcRO-2 [48), which takes ~ 1 day to finish. Unless otherwise specified, all inference is conducted on a single H100 GPU.
Following previous works 35 39 we quantitatively study the proposed method on the validation set of COC02014 37 and randomly sample 5,000 images along with their prompts for evaluation. Since CLEAR aims to linearize pre-trained DIT, we also benchmark our method against the results by the original DiT using consistent random seeds. Following conventions [53 33
60
66], we consider
FID 24 LPIPS 77], CLIP image similarity [47) and DINO image similarity 4 this setting as
metrics. For settings requiring pixel-wise alignment like image upsampling and ControlNet (69, we additionally incorporate PSNR 27 and multi-scale SSIM [59] for reference. While comparing with real images in COCO, we only include FID and LPIPS for distributional distances. Furthermore,
7

---

CLEAN O -0100
C2
FLUX-1 2.26
Alaswiree EONSS, pravt
stearaakk cirolip 2HEr Nicheria Cesh-becbee,
colertii cordl resf fist clea bluc rater, vibian
butsting city avted
elderly west vaned Eeen.
golden hour beach with son Light posschul and
Inatorioot
condighers
and white
Figure 8: Qualitative results by the linearized FLUX-1.dev with CLEAR and the original model.
CLIP text similarity [47]. Inception Score (IS) [54] and the number of floating point operations (FLOPS) are adopted to reflect textual alignment, general image quality, and computational burden, respectively. Text prompts for qualitative examples are generated by GPT-4o.
4.2 Main Comparisons
We aim to lincarize a pre-trained DiT in this paper. and the lincarized model expected to perform comparably to the original one. As illustrated in Sec. Sec.3.1l most efficient attention algorithms result in suboptimal performance for the target problem, as quantitatively supported by the evaluation in Tab. 2 In contrast, the proposed convolution-like lincarization strategy achieves comparable even superior performance to the original FLUX-1 1.dev while requiring less computation, underscoring its potential for effectively linearizing pre-trdi DiTs. Please refer the appendix for analysis of training dynamics and convergence.
With the knowledge distillation loss terms defined in Eq. 5 the differences between the outputs of the linearized models and the original model are further minimized. More evaluations with the Gvalval benchmark [19] and GPT scores can be found in the appendix. Qualitatively, as shown in Fig. 8 the lincarized models produced by CLEAR yield comparable results.
For efficiency, CLEAR significantly reduces FLOPS compared to the original FLUX. However, due the complexity of kernel implementation, shown in Fig. 2 its practical speedup does fully match the theoretical gains. Nevertheless, at IK resolution, CLEAR with r 8 and = 16 still outperforms the original FLUX, with acceleration gains increasing as the resolution scales up.
4.3 Empirical Studies
In this part, examine several noteworthy properties of CLEAR, including resolution extrapolation, zero-shot generalization across different models and plugins, multi-GPo parallel inference, and the effects of various training data.
Resolution Extrapolation. One of the key advantages of linearized diffusion model its ability to efficiently generate ultra-high-resolution images (39]. However, many previous studies have revealed that it is challenging for diffusion models to generate images beyond their native resolution during training. [15] 23 61 [6]. They thus apply practical solution for generating high-resolution

---

sille
X01-1104
(6-)
Stoughh
Strcighh 07
schnetl
CLEAR (--2)
(r- 153
CLEAR G2-
(r-16) n. ControlNes
Figure 9: Qualitative examples of using CLEAR with SDEdit [41 for high-resolution generation
(left), FLUX-1 I sccnnel in a zero-shot manner (middle), and ControlNet [69 (right). G.T. and Cond.
denote ground-truth and condition images, separately.
Condition
Setting
PSNR (1) SSIM (1) FID ()) LPIPS (1) (L) DINO FIID ()) LPIPS ())
Against Original
Against GT
CLIP-T (f) IS (*) RMSE ())
FLUX-
40.25
0.32
30.16
22.22
0,039
Gray
CLEAR (r-
25.98
0.93
26.14
0.19
93.39
94.24
43,82
0.31
29,90
21.29
0.036
CLEAR 16)
28.24
0.95
16.86
0.13
96.00
96.73
40.45
0.31
30.19
22.34
0,040
CLEAR 32)
30.59
0.97
11.57
0.09
97.33
98.12
40.21
0.31
30.21
21.94
0,042
Tile
CLEAR = 16)
FLUUX-1.de dev
30.12
0.97
9.1
0.13
99.25
99.04
39.73
38.20
0.34
30.11
21.77
54
0.019
0.021
Blur
CLEAR 16)
FLUX-1.ddv
28.92
0.96
10.56
0.13
99/02
98.67
39.66
58.72
0.31
30,20
21.67
21.42
0.033
0.028
Table 4: Quantitative zero-shot generalization results of the proposed CLEAR to a pre-trained ControlNet with gray, tiled, and blur image conditions on 1,000 images from the COC02014 validation dataset. RMSE here denotes the Root Mean Squared Error between the extracted conditions and the input conditions.
Against Original
Against Real
Setting
CLIP-T (*) IS (1) GFLOPS (1)
FID (L) LPIPS (J) CLIP-I (1) DINO (f) FID (L) LPIPS (J)
CLEAR (r= 16)
13.72
0.58
88.53
77.30
33,63
0.88
30.65
37.84
80.6
Square Neighhorhood
13.77
0.58
88.47
76.53
33.16
0.88
M4.69
37.96
92.1
Convolution 210.2
0.73
57.86
21.20
190.04
0.89
23.26
6.63
78.9
Table 5: Comparisons of circular neighborhood, square neighborhood, and standard attention.
images in a coarse-to-fine manner and devise adaptive strategies for components such as position embeddings and attention scales. 'he proposed CLEAR, on the other hand, makes architectural
modifications to a pre-trained diffusion backbone, making it seamlessly applicable to them.
In this paper, we adopt SDEdit [41). a simple yet effective baseline adapting an image to a larger scale, for generating high-resolution images. In addition, we also enlarge the NTK factor of rotary position embeddings from 1 to 10 following (45). balance the entropy shift of attention using a log- scale attention factor 130, and disable the resolution-awarc dynamic shifting [17 in the denoising scheduler. By adjusting the editing strength in SDEdit, as shown in Fig. [left), we can effectively
control the trade-off between fine details and content preservation. In the appendix, we also try building CLEAR on top of various methods for resolution extrapolation.
Quantitatively, measure the dependency between results by CLEAR with those by the original FLUX-1.dev. As shown in Tab. 3] we achieve MS-SSIM scores as high as 0.9, showcasing the effectiveness of the lincarized model with CLEAR as an efficient alternative the original FLUX.
Circular Neighhorhood vs Square Neighborhood vs Standard Convolution. Compared with Neighborhood Attention 2 noticeable difference in CLEAR is its circular neighborhood instead of a square. Computationally. the FLOPS of a circular neighborhood with radius is ~ x that of a square neighborhood with side length 2r- - 1. We empirically find that they achieve comparable performance, as shown in Tab. We also experiment with the standard convolution and find that it fails to generate visually plausible images, which highlights the importance of formulation consistency mentioned in Sec. 3.1
Compatibility with DIT Plugins. It is favorable that substituting the original attention layers with linearized ones would not impact the functionality of plugins trained for the original DiT. As shown in Fig. D right), CLEAR exhibits this property by supporting the pre-trained ControlNet 669 using grayscale images as a condition for FLUX-1.des Quantitative results can be found in Tab, including grayscale images, tiled images, and blur images as conditions, respectively.
Multi-GPU Parallel Inference. A linear complexity DiT can inherently support patch-wise multi- GPU parallel inference. In practice, for text-image joint attention used in modern DiT architec- tures 17 34,, we have to figure out a communication-effiiii requires gathering information from all key-value tokens. In Eq. 7 we propose an approximation
solution to handle text tokens, which
9

---

Against Original
Against Real
Setting
CLIP-T (t) IS (t)
FID (.) LPIPS (1) (L) DINO (1)
FID (3) LPIPS (0)
CLEAR (r 16)
33.63
30.65
37.84
11.55
0.51
90.46
80.89
33.74
0.81
31.21
39.26
N
4
12.78
0.54
89.74
79.99
33.07
0.81
31.27
40.01
14.21
0.57
88.92
78.65
32.26
0.80
31.22
39.34
Table 6: Results of patch- wise multi-GPU parallel inference with various numbers of patches using the approximation in Eq.
GPUs
Synchronous
Asynchronous
FLUX-Ldev
CLEAR (r 16)
CLEAR (r = 8)
FLUX-L.dev
CLEAR = 16)
CLEAR (r 8)
-1024 1024-
11.13
11.40
11.00
2
7.98
8.52x131
x.
7.64.14
8.10
7.50
2.93
6.01 NA
5,38
5.64x1m
5.67-am NA
5.11
8
4.84x230
4.37
4.49.148
3,90.
-2048 x 2048-
52.25
42.98
39.18
30.96
26.26x144
23.96
30.17
25.41x119
23.01
4
18. 18.94
15.64.228
13.86
18.58
15.12x184
-3.4-2.0
8
12.97
9.72.cm
8.40).ss
12.57
9.30
8.04
4096 4096-
372.43
207.83
173.53
2
200.16.x.ss
115.02
96.65
OOM
112.34xias
91.84xx.9
k05.59
59.65
49.70.2.4
OOM
57.42x3.2
48.57x339
59.18
32.33
26.88xx46
OOM
31.23
26.26x661
Table 7: Efficiency of multi-GPU parallel inference measured by sec./50 denoising steps on HGX H100 8-GPU server. We adapt Distrifusion (35] to FLUX-1.dew here for asynchronous communication. The ratios of acceleration are highlighted with red. Results of CLEAR with r 16 at the 1024 x 1024 resolution are not available (NA) because the patch size processed by each GPU smaller than the boundary size. OOM denotes encountering out-of-memory error.
based on patch-wise averaging and validate its effectiveness quantitatively in Tab. 6 Results indicate high correlation in semantics before and after this approximation, demonstrating its practical effectiveness.
For CLEAR, since there are only token interactions in the boundary areas of the patch handled by each GPU and the approximation of feature aggregation for text tokens defined in Eq. we achieve satisfactory efficiency multi-GPU parallel inference. Furthermore, we apply the asynchronous communication strategy in Distrifusion (35) to achieve even more significant acceleration. As shown in Tab. 7 the acceleration becomes more significant with the increase of image resolution, while the original DiT encounters an out-of-memory (OOM) error due to the necessity of caching all key-value tokens.
5 Conclusions
In this paper, we present CLEAR, a convolution-like local attention strategy that effectively lin- earizes the attention mechanism in pre-trained Diffusion Transformers (DiT's, making them sig- nificantly more efficient for high-resolution image generation. Specifically, we identified four key clements-localin formulation consistency, high-rank attention maps, and feature integrity-thal are essential for successful linearization in the context of pre-trained DiTs. CLEAR leverages these prin- ciples by restricting attention to circular local field around each query, achieving linear complexity while retaining tigh-quality results comparable to the original model. Our experiments demonstrate that
on merely 10K self- self-generated samples allows for efficient knowledge transfer to a student model, leading to a 99.5% reduction in attention computations and a 6.3x acceleration in 8K-resolution image generation. Moreover, CLEAR, once trained, supports zero-shot generalization across different models and plugins and improves multi-GPU parallel inference capabilities, offering broader applicability and scalability.
10

---

Acknowledgment
This project is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: MOE-T22P20122-0006)
References
[1] Fan Bao, Shen Nic, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 22669-22611 2023.
[2] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. In International Conference on Machine Learning, pages 1737-1752. PMLR, 2023.
[3] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprini 25X1::2004.05/50, 2020.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[5] Junsong Chen, Chongjian Ge, Enze Xie. Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to-strong training of diffusion transformer for 4k text-to-image generation, 2024.
[6] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao. Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, al. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:23 2023.
[7] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10500 10509, 2019.
[8] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreca Gane, Tamas Sarios, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
[9] Djork-Amé Clevert. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07288 2015.
[10] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffussonn models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[11] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0869), 2023.
[12] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:1634441066 16359, 2022.
[13] Tri Dao and Albert Gu. Transformers ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060) 2024.
[14] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[15) Ruoyi Du, Dongliang Chang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. Demofusion: Democratising high-resolution image generation with no $$S. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6168, 2024.
[16] Ruoyi Du, Dongyang Liu, Le Zhuo, Qin Qi, Hongsheng Li, Zhanyu Ma, and Peng Gao. I-max: Maximize the resolution potential pre-trained rectified flow transformers with projected flow. arXiv preprint
2024.

---

[17] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high- high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
[18] Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-12x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.05945. 2024.
[19] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An abject-focusee framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132-521522 2023.
[20] Dongchen Han, Ziyi Wang. Zhuofan Xia, Yizeng Han, Yifan Pu, Chunjiang Ge, Jun Song, Shiji Song, Bo Zheng, and Gao Huang. Demystify mamba in vision: A linear attention perspective. arXiv preprint 2rXiv:2405.16600 2024.
[21] Dongchen Han, Tianzhu Ye. Yizeng Han, Zhuofan Xia, Siyuan Pan, Pengfei Wan, Shiji Song, and Gao Huang. Agent attention: On the integration of softmax and linear attention. In European Conference on Computer Vision, pages 124-140. Springer, 2025.
[22] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[23] Yingqing He, Shaoshu Yang, Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang, Ran He, Qifeng Chen, and Ying Shan. Scalecrafter: Tuning-free higher-resolution visual generation with diffusion models. In The Twelfth International Conference on Learning Representations, 2024.
[24] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[25] Geoffrey Hinton. Distilling the knowledge in neural network. arXiv preprint arXiv: 1r3iv:5503.02531. 2015.
[26] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
[27] Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pages 2366-2369. IEFE, 2010.
[28] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In International conference on machine learning, pages 9099-911 17. PMLR, 2022.
[29] Linjiang Huang, Rongyao Fang, Aiping Zhang, Guanglu Song, Si Liu, Yu Liu, and Hongsheng
Li. Fouriscale: A frequency perspective on training-free free high-resolution image synthesis. arXiv preprint arXiv:2403. /2963, 2024.
[30] Zhiyu Jin, Xuli Shen, Bin Li, and Xiangyang Xue. Training-free free diffusion model adaptation for variable-sized text-to-image synthesis. Advances in Neural Information Processing Systems, 36.70847-70860, 2023.
[31] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156-5165. PMLR, 2020.
[32] Bo- Bo-Kyeong Kim, Hyoung- Kyu Song. Thibault Castells, and Shinkook Choi. Bk-sdm: Archi- tecturally compressed stable diffusion for efficient text-to-image generation. In Workshop on Efficient Systems for Foundation Models@ ICML2023, 2023.
[33] Nupur Kumari, Bingliang Zhang, Richard Zhang. Eli Shechtman, and Jun- Yan Zhu. Multi- concept customization of -on-tto-mmage diffusion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1931-1941, 2023.
12

---

[34] Black Forest Labs. Flux: Official inference repository for flux.1 models, 2024. Accessed:
2024-11-12.
[35] Muyang Li. Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high rishuuosol diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 7183-7193, 2024.
[36] Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Ying- fang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint WxXiv:2405.08745, 2024.
[37] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[38] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.027447 2022.
[39] Songhua Liu, Weihao Yu, Zhenxiong Tan, and Xinchao Wang. Linfusion: 1 gpu, minute, 16k image. arXiv preprint arXiv:2409. 02097, 2024.
[40] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[41] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2 01073, 2021.
[42] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[43] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[44] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[45] Bowen Peng and Jeffrey Quesnelle. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation, 2023.
[46] Hao Peng, Jungo Kasai, Nikolaos Pappas, Dani Yogatama, Zhaofeng Wu, Lingpeng Kong, Roy Schwartz, and Noah A Smith. Abc: Attention with bounded-me memory control. arXiv preprint 1r.iv:2100.02488. 2021.
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages
PMLR, 2021.
[48] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE, 2020.
[49] Jason Ramapuram, Federico Danieli, Eeshan Dhekane, Floris Weers, Dan Busbridge, Pierre Ablin, Tatiana Likhomanenko, Jagrit Digani, Zijin Gu, Amitis Shidani, et al. Theory, analysis, and best practices for sigmoid self-attention. arXiv preprint arXiv:2409.0 0443044999440....000000111111114 2024.
13

---

[50] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[51] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation.
Medical image computing and computer-asiistdd intervention-MICCAL 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part 18, pages 234-241. Springer, 2015.
[52] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53-68, 2021.
[53] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-201 22510, 2023.
[54] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, Advances in neural information processing systems, 29, 2016.
[55] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputinng 568: 127063, 2024.
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[57] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. http::/gggtuuuu..o//oogeeeeee www.oogggiggaae.oooooooooooooo.o diffusers 2022.
[58] Sinong Wang. Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0477 2020.
[59] Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for im- age quality assessment. In The Serety-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pages 1398-1402. Icce, 2003.
[60] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302.133.. 2023.
[61] Haoning Wu, Shaocheng Shen, Qiang Hu, Xiaoyun Zhang, Ya Zhang, and Yanfeng Wang. Megafusion: Extend diffusion models towards higher-resolution image generation without further tuming, arXiv preprint arXiv:2408. 11001, 2024.
[62] Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Yujun Lin, Zhekai Zhang, Muyang Li, Yao Lu, and Song Han. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410. 10629, 2024.
[63] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström- based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14138-14144 2021.
[64] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
[65] Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-cfficieet training. arXiv preprint arXiv:2312.06635, 2023.
14

---

[66] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. 2023.
[67] Haoran You, Yichao Fu, Zheng Wang, Amir Yazdanbakhsh, et When linear attention meets autoregressive decoding: Towards more effective and efficient linearized large language models. arXiv preprint arXiv:2406.07366, 2024.
[68] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santi- ago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-17297, 2020.
[69] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3841 2023.
[70] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018
[71] Yu Zhang, Songlin Yang, Ruijie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, et al. Gated slot attention for efficient linear-time sequence modeling. arXiv preprint arXiv:2409.077 146, 2024.
15

---

A Details of Efficient Attention Alternatives
The vanilla scaled dot-product attention, although effective and flexible, introduces quadratic putational complexity. Many works have focused on its efficient alternatives. In Sec. 2, we provide a taxonomic overview of recent works and will supplement more details regarding the specific formulations and implementations here.
Linear Attention avoids the sof tmax operation in the vanilla attention, supporting computing K" KVV first with the associative property of matrix-wise wise multiplication, and thus achieves linear complexity, Before that, non-negative kernel functions f(-) and g()) are applied on and K respectively such that the similarity between each query-key pair is non-negative. Furthermore, the similarity score between each query-key pair is normalized by the sum of similarity scores of between this query and all key tokens separately, to mimic the functionalities of soft Following [31] (0) (3), we implement f(-) and g(-) by the elu function written as
[ Formally, the operation for the i-th query can be
(clu(Qi) + ()(cl(( +1) V. O (elu(Q.) 1) Ej (elu(K)) +1
(8)
Sigmoid Attention replaces the sof tmax with the formulation of sigmoid, which removes the need to compute the softmax tmax normalization, and thus achieves acceleration:
QK T
O sigmoid(
+ b)V,
Ve
where b is hyper-parameter. In this paper, we follow the official implementation of fashSigmoia with hardware-awar optimization when applying Sigmoid Attention to DiT'.
PixArt-Sigma achieves acceleration by spatially down- down-sampling the key-value token maps [5]. Following the official implementatiorf) we use Iearnable group-wise Conv4 x kernels with stride = 4 and initialize the weights to Tद that it is equivalent to average pooling operation at the beginning. Formally, it can be written as:
1
O softmax( QConva (K) Conv.. Vc
(10)
Although has been demonstrated that such a strategy can work well relatively deep layers of DiTs, the results are still unsatisfactory for a completely linearized DIT.
Agent Attention performs efficient attention operations via agent tokens A from down-sampled query token map 21:
Down (ONn(QKKTTT JV. Ve
= softmax
(11)
The derived agent tokens A are then used as value tokens:
QDown (Q) )A.
= softmax
(12)
Ve
Such operations can be viewed as an adaptive token down-sampling strategy.
Slot Attention implemented in this paper is adapted from 771 41,, which contain key-value memory slots derived by adaptively aggregating key-value tokens:
PXT ),
PXT
K softmax(
softmaxx
V
(13)
VC
VC
where P e xR is learnable and introduced for modeling the writing intensity of each input token to each memory slot. These slots are then used as alternatives original key-value tokens for attention computation:
QK VC
O = softmax
)İ.
(14)
16

---

Setting
Running
Time (Sec. / 20 Steps)
TELOPS/ Luyer
1024
1024
2018
4096
8192
1024
4006 4096
FLUX-I I. dev
4.81
20,90
148.97
1842.48
0.26
3.51
$3.60
847.73
CLEAR fr
4.38
15.67
69.41
293.50
0.25
3.92
CLEAR 16)
4.56
17.19
83.13
360.83
0.09
0.35
1.43
5.79
CLEAR
5.45
19.95
109.57
496622
0.72
3,14
13.09
Table 8: Raw data for Fig.
on efficiency comparisons.
This strategy presents a different fashion for adaptive key-value compression.
Strided Attention samples tokens at regular interval [7]. As sparse attention strategy, the attention mask of the (-th layer with a sampligg ratio x r can be constructed in the following way:
1, if Ntext j S Ntert
M
-
(r = Tx and di %r ry);
d00%
(15)
0.
otherwise,
where Tz 1%r and Ty 1//r ensure that each token has chance to be sampled as key-value tokens.
Swin Transformer adopts sliding window partition strategy 44 where attention interactions are independently conducted for each window. Formally, the attention map can be constructed via:
1, if S Atxt j S Attezt tokens and j in the same window; 0, otherwise.
Mij
(16)
We set the window size to 16 and apply shift of 8 for windows in layers with odd indices, following the approach described the original manuscript. The rank of the image-to-image attention mask corresponds the number of windows, which poses challenges in achieving the high-rank requirement introduced in the main manuscript needed for linearizing DiTs. other words, all tokens within the same window share the same set of key and value tokens. This results in many duplicate in M, significantly reducing its rank.
Neighborhood Attention performs like convolution: for each query, it only samples key-value tokens within a neighborhood sliding window:
if Mext of S lttart
or (dl <r and (3)) < <r); otherwise.
(17)
Comparing with Swin Transformer, each query token in neighborhood attention has distinct key and value token set, and each row M linearly independent each other, which ensures the high-rank property. The formulation used in CLEAR is developed based such Neighborhood Attention. The difference is in the functions checking whether key-value token is in the neighborhood of a given query token, shown in Eq. 3
For CLEAR, we provide further illustration of the case Fig. for better clarity.
B Training Dynamies
We supplement the curves of training losses of various efficient attention alternatives in Fig. [1 The conclusion is consistent with the main manuscript, that strategies fulfilling the requirements of locality, formulation consistency, high-rank attention map, and feature integrity yield the most satisfactory training convergence.
C Raw Data for Efficiency Comparisons
We supplement raw data for Fig. 2 efficiency comparisons in Tab. 8 for better clarity.
17

---

Query
Key/ Value
Pixel Center
0
1
2
3
4
5
6
7
32
25
20
17
16
17
20
32
25
18
13
10
9
10
13
18
20
13
8
5
4
5
8
13
10
5
2
1
5
10
16
9
4
1
0
1
4
9
17
10
5
2
1
2
5
10
20
13
8
5
4
5
8
13
35
18
13
10
9
10
13
18
Numbers: (xq- + yx) (x) (4,4), 4, threshold r²
16
Figure 10: The neighborhood region for an exemplar query in CLEAR when r 4.
Signoid Attention
Linear Attention
3.0
PixArt-Sigma Agent Attention
Flow Matching Loss
Strided Attention
2.5
Swin Trrmsformer
8)
2.0
Ours (r=
1.5
1.0
0.5
0.0
10
10a
102
10a
10a
Number of Training Steps (Log Scale)
Figure 11: Training dynamics of various efficient attention aliernatives FLUX-1.dev.
Against Original
Aguinst Real LPIPS (
Setting
LPIPS ( CLIP-I
||
CLIP-T(())
IS (1)
FLUX-1-dv
||
29.19
31.53
36.41
CLEAR (r 8)
13.62
0.62
88.91
78.
33.51
0.81
31.35
38.42
CLEAR
12.51
81.32
34.43
31.38
39.66
CLEAR (r 32)
12.43
0.57
90.70
82.61
33.57
0.83
31.48
39.68
Table 9: Quantitative zero-shot generalization results to FT. using CLEAR layers trained on FLUXX-1.deY
Circular Neighborhood or Square Neighborhood
In fact, circular or square windows do not fundamentally affect the performance and the linearization properties. Our main conclusion of the manuscript is the four key factors for successful linearization of pre-trained DiTs. The standard neighborhood attention, which follows these four principles, is definitely effective, and so is the introduced method based circular windows. They yicld
18

---

Aesthetic
Prompt Aligument
Overall
Win Rate vs Other
Setting
dLUX-I.dev
91.52
87.46
88.
0.57
CLEAR (r 16)
89.62
90.22
90.09
92.13
91.94
92.29
88.52
88.71
88.81
SANA
82.42
86.86
87.63
84.68
90.68
90.57
84.32
86.96
86.71
0.83
0.57
0.62
Table 10: GPT scores compared with the original FLUX-1.de model and SANA, a recent linear- complexity DiT trained from scratch.
1024x1024 by FLUX-1.dev
4096x4096
4096x4096
W. SDEdit
I-Mox
Figure 12: The linearized DiTs by CLEAR are compatible with various pipelines dedicated to high-resolution inference. The prompt is shown in Fig. 17
Setting
One Obj.
Two Obj
Count
Color
Prs.
Color Attr.
Aoc.
Img
Text Acc.
Overall
FLUX- 1.dev
98.44
81,82
72.50
77.93
21.75
41.75
64.10
79.57
0.657
CLEAR (r 16)
99.06
83.84
69.69
79.79
23.25
48.75
66.00
81.19
0.674
CLEAR
99.38
78.03
48.75
76.60
16.50
41.00
58.52
76.13
0.600
Coarse Tokens
98.77
84.85
67.81
79.52
21.00
52.00
66.00
81.74
0.673
Table 12: Studies on the GenEval benchmark.
comparable performance as mentioned Tab. Given that the FLOPS can be reduced largely while preserving the performance, which is free-lunch benefit, apply the latter form. As shown in Tab. II it is indeed more efficient in terms of wall-clock time, especially at higher resolutions.
E Results on More DiTs
We additionally deploy our method on DiT models other than FLUX used in the main manuscript to demonstrate the
Time (s) 20 Steps
1K 1K
2K x
4K 4K
8K 8K
Square (r 8)
4.63
16.40
72.70
310.50
universality of the proposed CLEAR. Here, StableDiffusion3.--Larggg 17 (SD3.5-L), another state-of-the- art text-to-image generation DIT. We use the default setting of 16, which yields the best trade-off between quality and efficiency according to our experiments. Results the COCO2014 validation dataset are shown in Tab. T6
Circular (r = 8)
4.38
15.67
69.41
293.50
consider
Square (r 16)
4.84
18.77
87.09
382.07
Circular (r = 16)
4.56
17.19
83.13
360.83
Table 11: Comparisons on the running time required by circular and square attention windows at various resolution scales.
We also supplement more qualitative comparisons with results by the original FLUX-1.des and SD3.5-L in Fig. [5] Results indicate an overall comparable performance. Due to the absence of explicit long-distance token interactions, our method may underperform capturing overall structural properties, such as potential symmetry. Additionally involving more less global tokens, such as down-sampled tokens as employed in PixArt-Sigma (5), could potentially mitigate this issue. However, the primary objective of this paper is highlight the significance of locality simple yet effective baseline, we leave detailed design explorations to future work.
19

---

2K xK
4k x4K
Method
Aesthetic
Prompt Align.
Overall
Win Rate
Aesthetic
Prompt Align.
Overall
Win Rate
SDEdii
86.32
89.08
87.22
82.36
w/ NTK
87.73
91.32
87.10
0.75
86,54
91.35
85,93
0.78
Dynamic Shifting
88.44
91.68
86.59
0.70
84.02
91.40
84.29
0.73
Ours
90.22
92.29
88.81
Table 13: Quantitative comparison at 2K and 4K resolutions.
F
GPT Evaluation
Following prior work S 16,, we use GPT human-like evaluator to assess the quality of generated
images across three aspects: aesthetics, prompt alignment, and overall quality. Here, we consider three candidates: the original FLUX-1 ,dev, its linearized version with CLEAR (r 16), and SANA [62), linear- linear-complexity text-to-image DiT trained from scratch. We further instruct GPT-4o to compare CLEAR separately against the original FLUX-1 SIXX-.dde d.dev and SANA, reporting the win rates. Results across various resolutions are presented in Tab. TO further demonstrating the effectiveness and superior performance of proposed convolution-like linearization approach.
G More High-Resolution Results
In the main manuscript, we build our CLEAR on top of SDEdit 41
simple yet effective strategy for image generation given conditional image, for coarse-to-fine high-resolution generation. We demonstrate here that our method also compatible with variety of pipelines dedicated to resolution extrapolation. As shown in Fig. 12
we deploy CLEAR on I-Max [16]. concurrent work for training-free free high-resolution generation with pre-trained DiTs, and observe that it may yield a more optimal balance between preserving low-resolution content and capturing high-resolution details. For instance, as shown in Fig. 12 I-Max preserves the textures of the dresses from the low-resolution result with minimal variation while effectively enhancing clear high-resolution details.
In fact, it requires two steps achieve efficient high-resolution generation. The first is adapt the original DiT to make it effective at higher scales. The second is to make it more efficient. CLEAR mainly addresses the second step. For the first step, the techniques are mainly adapted from previous works, e.g., SDEdit and NTK rotary embedding. That is why we do not include many details about them.
Nevertheless, we fully agree with the reviewer that it would be helpful to supplement the related details. We include some key information here and will further elaborate on it in the revision.
SDEdit is simple yet effective training-free method for image editing based solely pre-trained text-to-image diffusion model, which can be used for high-resolution generation a coarse-to- fine manner. Specifically, we first generate an image at the native resolution scale of the diffusion model. Then, we resize it to larger size and add certain noise to it. A noise scale of 0.7 is adopted empirically in our experiments. Starting from this point, the model conducts the remaining denoising steps. In this way, the original image structures are preserved and low-level details are refned.
NTK rotary embedding applies scaling factor the rotary base used for rotary positional embedding, i.e., 6' bs, where s - Lhar and Lear is the current sequence length while Lnative is the native length seen in the training time.
Resolution-aware dynamic shifting applies a factor to the projection function of the current denoising time step for the flow matching scheduler: related to the image resolution. When this is enabled, a high resolution, the projection function would appear too "skew", i.e., the number of denoising steps allocated to the stage of high noise level insufficient.
I+(.-1)7+ where is positively
Regarding the three factors, we conduct the GPT evaluation following the same protocol Tab. and the results are shown in Tab. B3 highlighting their effectiveness.
20

---

a photo of a bench
a photo
horse and a train
CLEAR (r=8) 4.38s
+ Coarse 4.5 Tokens (+0. 38s)
CLEAR (r=8)
+ Coarse 4.766 765
4.38s
Tokens (+0. 38s)
Figure 13: Potential issues of small r. Adding down-sampled coarse key-value tokens can effectively address the problems.
FID (Against Original) CLIP-T
a 0
a: 0.05
= 0.5
@ =
14.27 / 30.90
13.98 / 30.77
13.78 30.66
13.70 30.56
8 = 0.05
13.90 / 30.81
13.88 30.81
13.81 / 30.72
13.68 / 30.64
0.5
13.83 / 30.68
13.82 / 30.68
13.72 / 30.65
13.47 30.59
8 =
13.77 30.65
13.69 / 30.66
13.45 / 30.62
13.44 30.58
Table 14: Grid search on and / for FID and CLIP-T metrics.
H Cross-DiT Generalization
We empirically find that the trained CLEAR layers for one DiT are also applicable for others within the same series without any further adaptation efforts. For example, shown in Fig. 9 middle), the CLEAR layers trained on FLUX-1.de can be directly applied to inference on FLUX-1 -.schel,, timestep-distilled DiT supporting 4-step inference, yielding results similar to those of the original FLUX-1.schnelI. Such zero-shot generalization is quantitatively evaluated in Tab. 9
I Hybrid Attention Extension
In practice, observe that small т. e.g., layouts and multi-objects, as shown in Fig. 13 As an extension, we propose hybrid structure that incorporates coarse tokens additional key-value tokens enhance holistic interactions. Similar to [5]. we conduct 8x downsampling to the key-value maps to derive these global tokens. As shown in Fig. 13 and Tab. [2] we find that these issues can be effectively resolved with minimal additional computational overhead, underscoring the practical benefits. However, as this paper primarily emphasizes the importance of local tokens, do not adopt this strategy by default.
tends produce results with inconsistent
J Studies on Loss Weights
In this part, we study the individual contributions of the two additional loss terms in Eq. by varying the weights a and B. In general, both loss terms contribute the consistency between the distilled and original models. Nevertheless, we find that they are complementary:
controls the weight of Lpred, regulating the consistency between the final output results, which, according to our experiments shown the grid search in Tab. [4] appears to be relatively more effective than Lattn. For example, the cases of 0, B = generally yield better FiD scores than those of a 0, > 0.
3 controls the weight of Lattn, regulating the consistency between the intermediate attention results, which, according to our experiments, serves as an auxiliary measure to facilitate the training convergence. To verify this, we present the values of the flow matching loss L after moving average at various steps in the training progress in Fig. 14 Results indicate that Luttn can facilitate training by offering informative intermediate supervision signals beyond the relatively late supervision provided by Lpred at the final outputs.
21

---

w/o
0.500
Cate
Flow Matching Loss
0.475
0.450
0,400
0.375
0.350
100
2K
Training Steps
Figure 14: Effectiveness of Cuttn on accelerating the training convergence.
Aesthetic
Prompt Alignment
Overall
Train r 8, Eval r =
89.19
88,38
87.96
Train r 8, Eval 16
89.49
88.83
88.31
Train r 16, Eval
16
89.62
92.13 79.08
88.52
Train r = 16, Eval r = 8
86.88
82.95
Table 15: Evaluation under different train and evaluation window radius (r).
Method/Setting
FID
LPIPS
Against Original
CLIP-I()
DINO (f)
FID
Against Real
LPIPS
CLIP-T (1)
IS (T)
GELOPS (1)
SD3.5-I.
34,10
0.83
31.40
36.06
206.5
W. CLEAR (r 16)
11.21
0.57
90.9
81.47
36.98
0.83
31.23
36.28
63.8
Table 16: Quantitative results of the original SD3-Large and its linearized version by CLEAR proposed this paper on 5,000 images from the COCO2014 validation dataset at a resolution of 1024 1024.
Overall, applying both terms achieves the best performance, and the performance is insensitive to the specific values of the hyper- ppprrpparaeeters according to the grid search. There can some subtle tradeoffs between the FID and CLIP-T metrics, where the default values achieve good balance.
K Generalizability across Different
Qualitatively, we find that the model trained with a smaller window radius r can be used for larger one. However, the opposite case does not work. We speculate that it is because the information provided for the model is complete for inference with additional clues when r is larger, while the information would be insufficient if becomes smaller. We supplement the quantitative results of GPT evaluation Tab.
L Limitation
One limitation our approach is that the practical acceleration achieved by CLEAR does fully meet the theoretical expectations indicated by FLOPS. It becomes less significant at relatively low resolutions and can even be slower than the original DiT when the resolution is below 1024 x 1024. This drawback arises partially because hardware optimization for sparse attention is inherently more challenging than the optimizations achieved by FlashAttention for full attention computation. Addressing this limitation may require developing fused CUDA operators specifically optimized for the specific sparse pattern of CLEAR, which is valuable direction for future works.
22

---

FLUX- dev
Stable ffusion? 5-arge
Figure 15: Qualitative comparisons on FLUX-I.dev (top) and L (bottom). The left subplots are results by the original models while the right ones are by the CLEAR linearized models. Prompts are listed in Fig. 19
23

---

Figure 16: More 4K examples by the CLEAR linearized FLUX-1.dev. Prompts are listed in Fig. 19
24

---

Fig. 1, according
top-left corner, top to battos, laft to right
1f 1, alxo uxed in Fig- and Fig. 8 flerce
sky betveen dragons povezful vizards. Ohe side the scens shave sizards casting spalla, thair staffs gloving with ragical anargy while on the other, dragons with scalas of fire and lightsing breaths torrents of flans. The sky is torn with stores of nagic, and belos, nedieval kingdex vatches is are the akies fury
fantasy
taking place
"ftturiatic cityacape, tovering akyscrapers, asos lights, apeeding cars, holographie advertiaenanta cybargunk, sltra-raaliatie, high rasolution. cinenttic lighting, highly datailed, ultra 8b, 8k, reflections glass, vibrant
nighttise,
nisty ataosphere",
//
13 tiger is kizaing rabbict,
11
engravinge, glase stains, sarn lighting, rich woed dask, soft shadoxs, high datail on pan hottle, ultra- realiatic teztures, vintage and rafined, cals and artistic feel, clase-up, high resclction, deup blue and golden accents',
pen
reflectione,
bbaautifll Chinere conan in tanfu, currounded by blooning peocies, floving silk rober, elegant and athereal. sott lighting, pastel colore, highly datailed fabric textures, delicate hair ornanants, paeny patals in the air, gracaful
traditional hairpin",
'charring coustryeide cottage, early norning smilight, nist the air, lush garden, rustic and cozy, lvy-eavared valls, sooden fasce, high datail, alsra-ranliatie, peaceful atnosphere, blooning flevere,
quiet cereze',
'faturistic racing car, sleck design, neen underglov, high speed action, dust trail, dynanic motion blur, cinenstic lighting, high resolution, Altra-realietis ultra WD, 88 dark background, neen lighta, flying, intenss colors, reflections
surface',
"pajestic Chinere dragon, siirling clouds, vater and ink effect, poverful precence, dynanic and drazatic, sonochrozatic ink cast, wirling nation, high detail on dragon scales, whirluind of eplachee,
cloude,
"traditional Chinese night narket, red laatorus, crouded stalls, vibrant atmosphere, warn and lively, golden lighting, reslistic and buetling, intricate narket detaile, traditionsl enacks, morchante robes, lanterns 10, 'onchanted forest, gloving plante, tovoring anciest trees, zystical girl, zagical aura, fastasy style, vibrant colore, ethereal lighting, boken eftect, ultra-detatled, painterly, ultra HD,, 8K, gloving
anixated croud background",
Fig. (Appendix)
gloving
sparkling particles",
Figure 17: GPT-generated prompts used in the main manuscript.
25

---

1/ 11
'portrait of oideriy fenale artist scart, coft studio lighting, high detail thoughtful, sax and viee, subtle background, rich textures, peasetul and inriting, close-up', // 12
and colerful uitra realietic, vara lighting, creative and
gentle
'faturistic soldier, rebotic lighting, dynamie pose, ultra-realistie, background, gloving HBD, inteuse expression, battle scars on armor',
respon,
highly digital HUD, HD,
// 13
'elle CLEAR' textured lettering, gubtle paint splaches",
vatercelor- style
gradients of
purple,
19
"bidden paradise with peach blessoxs, floving river, distant rouaraies, Quaiet cottages, dreamlike vibrant colers, river, villagore traditional attire, sunlight filtering through petals",
idyllic landecape, blossoming peach
15
'Parisia: street night, iconic street lights, cobblestone path, Vie of Eiffel Tover, vibrant reflections
city atzosphere, realiseeiii details, cizeratic lighting, 16
bistoric architecture, ronantic aabiance, scene, high resolution",
"astronaut zeeting alies creatures, coanic background, colorful nebula, stars in backgreund, high peaceful, otherveridly creatures,
lighting, sci-fi setting, eitra realistic, adventure epace, detailed environzent",
spacesuit,
// 17
"ructic kooden cabia interior, cozy and vare, fireplace gloving, 20oden beans, vintage furniture, iitra-realiatic detaila,
light from vindex, blaskets and cushions, peaceful asbiance, high resolution, natural voed grain visible", 18
earthy
cosy
Chinese ink landacape painting, nisty nountains, sinding rivers, sncient pise trees, traditienal brashstrokes, somechronatic, etheresl thatched pavilion,
painting,
axong
nountains,
gradation ink, natural flo2",
// 19
"phoenix rising frea flares, vibrant feathers, traditional Chinese aythological style, vivid and dranatic lighting,
aajestic, dynanic
dotaile, golden plusage, traditional patterns on vings, sense rabirth",
radiant
// 20
*city street reflecting day, vat pavement vith reflactions, people under unbrellas, soft city puddles, detailed raindropa, rltra-realistic details, vibrant deep colors, high contrast, peaceful anbiagce, soft abadova, atrest lights gloving",
cosy
atzosphere,
/1 21
forest, paths, serese, samn lighting, natural greens, intricate soodwork, rustic textures, banbea shadovs on ground, calligraphy brushez, traditional acrolla, acholars in robes",
Chirese acadezy, surrounded by
study
and
//22 22
*1950a Aserican
aigas, nostalgie ataosphere, sazz lighting, retro decor, viatage aenu, chrose accenta, classic style, cozy inviting, high deteil, ultra-rellittee,,
leather
checkerboard
/1 23
'ascient 1ibrary,
shelves filled eld detailed carvinga,
lighting, maxaive wonden tables vintage glabes, sarn light filtaring through vindous, realistic, intricate detaila on book apines, neataigic atzosphere, high resolution, serese and historical feel',
Fig.
"A cat holding eign that says hallo world"
Figure 18: GPT-generated prompts used in the main manuscript. (Cont.)
26

---

Fig. 15, top hotton,
right
//
it colorful serrounding ocean reflecting vibrant colors, ultra-detailed, photorealiatic, altra HD, 88, surreal and artistic componition, bald contrants, intricate reflections",
polar sitting
icebarg, holding
//2
green valley aurrouaded tovering cliffa, vinding reflecting the sky, fluffy whise clauds casting shadovs, grazing deer in the distance, ultra-detailed, photorealistic ultra 8D, 8k, natural vibrancy, peaceful vildernass atnosphere, intricate vatar and vegatation testures"
"peccaful Chinese lake acene, traditional pagoda on anall ialand, water reflecting atructure, distant nisty aountains, pisk lotus flavera floating, vars norning light, ultra- datailed, photoraalistic, ultra ID, 88, atnosphare, traditional anathaticn, vibrast
//
toul in darss forest at night, glaving yallos ayex, dark and systerioux atmosphare",
//
glowing jellyfish floating zid-air, young girl reaching touch one, shalvex filled anciant tores, soft anbiant lighting, eltra-datailed. photorealiatie, eltra HD, 8k, vhissical and nagical atmosphere, intricate textures",
library
akyacrapara, bustling nighttine setting, bolographic adverticemonte, rain- rain-roaked roads, uitra detailed, cineratic lighting, ultra nD, 8K, vivid colore, dranatic atmosphere, intricate reflections, dystopian ribe", /1
*ayberpank sityисаря, gloving
lights,
flying
futuristic tending garden glosing delicately handling the plants, eaterfall of stare in the background, ultra-detatled. pottorealiatic, eltra HD, 8k, atherma! lighting, blanding nature and tachnology".
fiovere, zetallic
forsations, gloving plante, aysterione alien figures, science fictien style, uitra derailed, cineratic lighting, uitra BB BK, vibrant colors, surreal anbtance, dranatic shadous, expansive vistas",
landscape, multiple
sky, strange
//9
'astuan
red, orange, yollor leaves, covered in fallen foliage, sunlight casting vara hees, dietant hills, ultra detailed, photoroaliable, eltra HD, 8k, rich colors, peacetul stnosphere, intricate datails of laaves and bark".
golden
path
'ondless lavonder fields at sunset, soft purple haes biending with golden say, a seall zustic farnhouse the distance, rolling hills on the horizoz, ultra detailed, photorealietie, uitra ID stmospherie dapth",
delicate lavender flovers,
'cozy rursl zitchen, rooden cabisets, frech bread on the couster, sunlight streaning through lace cursaine. ceranic jars and freeh herbe, rastic chara, varn soses, disra-datailes phesorealtstse, peaceful ataosphere",
pportrait of young girl with freckles, natural outdoor setting, senlight filtering through leaver soft focus background, vibrant hair azd rivid coler, Attra-dotailed photorealistie, uitra delicate facial
innocent
Figure 19; GPT-generated prompts used in the appendix.
27

---

Fig. 16, according to the top left corner, fron top to botton, from left to right
#/1
vineyard
curmer, , rove grapevizer heavy ripe truit, rustic farmhosse photorealietic, uitra 80,
distance. soft hills and clear sky, vare golden light, R, istricate grape and leaf textures, serene countryeide atmospacre",
1/2
texple Mill, forest, stone lanterns lining the path, seft goldon hour light, uitra detailed, plororeslistic, uitza IID, 88, traditional Chinese arcaitecture, peacetul arbiance, intricate carvings and orzase designs",
golden
surrounded
green
bbstling fich narket at sunrise, vibrant colore of fresh resfood, fichermen unloading crates, intricate details of fish scales and ice, anbient light, bustling atmospāere, ultra detraedetaleed, photoresiietic,
stmospheric realics, sharp textures, lively dynsmice",
//
"nsjestic dragon flying orer a nedioral castle, fiery sunset, rolling hills, dranatic clouds, fantasy style, sitra detailed, painterly sesthetic, uitra HD, 88, para bues, gloving eabers, intricate
golden lighting',
'fsturistic laboratory interior, gloving screens, robotic arns, holographic dieplaye clock design. science fiction style, uitra-detailed, ulira HD, SE, cold lighting, netallic textures, high-tech detailed equipaens",
la sezene Chizese garden, a curved atoze bridge over a lotas-filled pond, elegant pavilioas with ornate designs, veeping willow trees, hei fith avinsing, gentle saalight, ultra-derasled photorealiatic, ultra harmenious colozs",
traditional landacape design. tranquil atsosphere, vibrast yet
7
'asciant Greek texple hilltop, aurrounded leah gardens, golden hour, marble colurne, painterly style, dltra-dotailed, lighting, screac atzosphere, hictorical accuracy",
intricate carvings, mythological
8K,
//
t chessboard floating is cosxic vold, pieces made of planets and stars, hsean hand reaching detailed, photorealistie, vivid lighting, surceal thought-provoking atmospheze",
cocnic
design.
//
" vintage granophone is the niddle of luah rainforeat, vines vrapping around the horn, maii aninals like parrota nankeya curiously gethered,
visibly floating detailed, photoreslierie, ultra HD, SE, vibrant colers, nagical shisicaa atnesphere, rich testurez", // 10
cleck's geare, lush greenery aurrounding scane, ultra-datuiled, photorealiatie, ultra HD, 8X, tineleas surraal atmcsphere, intricate zechanical details, draxatic lighting",
giant eabedded
vaterfalls
/1 11
jagged eliffa, vivid orange purple sky, seabirds flying above, pools with reflections, dltra-detalled, photorealiatic, ultra HD, 88, dynasic sotias, tranquil yes drazatic stnosphere, intricate rack and vater testures"
rocky
crashing
// gigantic hourglass baried desert, golder sard alouly flaving batueen the chanbers, group of esplorers clinbing the hourglass, a atora brewing in the background, ultra-detailed, photorealietie, eltra HD, 8k, drazatic lighting, surreal and adventurous aubiance",
13
porrreit
vith long white beard, searing craditional zobes, holding wooden staff, nountain landscape in the background, soft diffused light, ultra-dstatled, photorealistie. ultra HD, 8k, deep vrinklas, zerane aspraazion, systical and tineless atnasphers'
Figure 20: GPT-generated prompts used in the appendix. (Cont.)
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
In the abstract and introduction, we have clarified the focused problem and our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We mention our experiment settings in Sec. and include the detailed configu- rations in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

Answer: [Yes]
Justification: We attach the source codes to reproduce the main experimental results as supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https /mips.ccc/pullc//uuui guwde//ddee/ee/ Cot//ooooeeesiseeeeeee hwtpbbmmsssiooooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We mention our experiment settings in Sec. and include the detailed configu- rations in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Following the evaluation convention for text-to-image generation, only average results are reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
31

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mention the related information in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp::/neuurip:: http://wwr.p...cooooo.//ccccaaaa
Answer: [Yes]
Justification: We have reviewed the guidelines and confirm that the conducted research follows them.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the impacts in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
32

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The applied safeguards are discussed in the appendix.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The license and terms of use the used assets are respected and mentioned in the appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
33

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
34

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
35


Reviews:
[{"review_id": "XrwCMJDccv", "reviewer": "Reviewer_SrwR", "summary": "In the paper, the authors identify four key factors for linearization of pre-trained DiTs: locality, formulation consistency, high-rank attention maps, and feature integrity. And they introduce a convolution-like local attention strategy, which limits feature interactions to a local window around each query token, and achieves linear complexity. In experiments, it reduces attention computations by 99.5% and accelerates generation by 6.3 times for generating 8K-resolution images.", "questions": "1. The meaning of 'convolution-like' is not clear. What parts are convolution-like?\n2. What size of the window is optimal? Isn't a larger window size always better?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths：\n1. The authors identify four key factors crucial for successful linearization of pre-trained DiTs: locality, formulation consistency, high-rank attention maps, and feature integrity.\n2. They introduce a convolution-like local attention strategy, which limits feature interactions to a local window around each query token.\n\nWeaknesses：\n1. The convolution-like local attention strategy employs circular windows instead of a square that seems not indispensable. Although the FLOPS of a circular neighborhood is ∼ π/4× that of a square neighborhood, but more data operations are required, and they achieve comparable performance.\n2. Among the four key elements, the high-rank attention maps and feature integrity seem not notable in the convolution-like local attention strategy.\n3. The experiment on relation of window size and performance is not presented.", "was_revised": true, "final_justification": "I appreciate the authors’ detailed response, which has clarified part of my concerns. However, I remain some cautious about the novelty and effectiveness. I have decided to increase my score to Borderline accept.", "rebuttal": "We would like to express our sincere gratitude to Reviewer SrwR for the constructive comments and are happy that the reviewer finds the four key factors for successful linearization of pre-trained DiTs and the introduced convolution-like local attention strategy effective. We would like to address the concerns and questions reflected in the review below.\n\n> 1. **(W1: Circular window vs square window)** The convolution-like local attention strategy employs circular windows instead of a square that seems not indispensable. Although the FLOPS of a circular neighborhood is ∼ π/4× that of a square neighborhood, but more data operations are required, and they achieve comparable performance.\n\nThanks for the valuable point. Indeed, circular or square windows do not fundamentally affect the performance and the linearization properties shown in the manuscript. **The main conclusion of the manuscript is on the four key factors for successful linearization of pre-trained DiTs**, listed in Tab. 1. The standard neighborhood attention, which follows these four principles, is definitely effective, and so is the introduced method based on circular windows. Indeed, they yield comparable performance as mentioned in Lines 321~325. Given that **the FLOPS can be reduced largely while preserving the performance, which is a free-lunch benefit**, we apply the latter form. As shown in the following study, it is indeed **more efficient in terms of wall-clock time**, especially at higher resolutions.\n\n| Time (s) / 20 Steps | 1K*1K | 2K*2K | 4K*4K | 8K*8K  |\n| :-----------------: | :---: | :---: | :---: | :----: |\n|    Square (𝑟=8)     | 4.63  | 16.40 | 72.70 | 310.50 |\n|   Circular (𝑟=8)    | 4.38  | 15.67 | 69.41 | 293.50 |\n|    Square (𝑟=16)    | 4.84  | 18.77 | 87.09 | 382.07 |\n|   Circular (𝑟=16)   | 4.56  | 17.19 | 83.13 | 360.83 |\n\nMeanwhile, we find that circular windows can potentially be **more robust to image rotation**, given that circles are rational invariance, which is not the case for square windows. To validate this property, we send original images to the network and get the outputs. Then, we rotate the original images by 60 degrees, send them to the network, and rotate the outputs back. The average SSIM between the two outputs are shown below.\n\n|      | Square | Circular |\n| :--: | :----: | :------: |\n| SSIM |  0.70  |   0.75   |\n\n> 2. **(W2: The high-rank attention maps and feature integrity)** Among the four key elements, the high-rank attention maps and feature integrity seem not notable in the convolution-like local attention strategy.\n\nThanks for the comment. In fact, the proposed convolution-like local attention strategy are adhere to these two principles:\n\n* On the one hand, each query token has a different set of key and value tokens, which makes the overall attention masks exhibit a high-rank pattern. \n* On the other hand, the proposed method does not compress or recompute key and value features in any forms, which ensures their integrity.\n\nThese two factors contribute significantly to the image quality, as shown in Fig. 3:\n\n* The Swin Transformer's strategy, which lacks high-rank attention maps, results in \"grid\" artifacts. \n* If we zoom the images, we can find that the PixArt-Sigma's strategy, which lacks feature integrity, yields distorted local details.  \n\nAs a result, they fail to produce high-quality images. We further introduce the following GPT and GenEval scores to verify the conclusions:\n\n|                    | Aesthetic | Prompt Alignment |  Overall  | Win Rate vs Other |  GenEval  |\n| :----------------: | :-------: | :--------------: | :-------: | :---------------: | :-------: |\n|     Swin Attn.     |   51.33   |      68.77       |   53.26   |       0.95        |   0.483   |\n| PixArt-Sigma Attn. |   70.28   |      79.20       |   71.32   |       0.97        |   0.428   |\n|    CLEAR Attn.     | **89.62** |    **92.13**     | **88.52** |         -         | **0.674** |\n\n> 3. **(W3: Performance vs window size)** The experiment on relation of window size and performance is not presented.\n\nThanks for the valuable point, indicating an important study. The related experiments can be found in Tab. 2, Tab. 3, and Fig. 8, with various $r$. We will further highlight them in the revision.\n\n> 4. **(Q1: Definition of \"convolution-like\")** The meaning of 'convolution-like' is not clear. What parts are convolution-like?\n\nThanks for the good question. For each query token, only several local key and value tokens are taken for attention interaction, which is similar to the behavior of a convolution with a fixed-size kernel. We provide an illustration of this paradigm in Fig. 6.\n\n> 5. **(Q2: Optimal window size)** What size of the window is optimal? Isn't a larger window size always better?\n\nThanks for the good questions. The reviewer's insight that a larger window size yields better performance is correct. However, it requires more computational overhead, as mentioned in Fig. 2 and Tab. 2. We empirically find that $r=16$ yields the best trade-off between performance and efficiency and thus adopt this setup by default. We will enhance the clarity of this part in the revision.\n\nWe would like to thank Reviewer SrwR again for the valuable feedback. Hope our responses alleviate the reviewer's concerns, and we are happy to answer additional questions if there are."}, {"review_id": "3HXWhY8JGi", "reviewer": "Reviewer_JWRk", "summary": "This paper investigates efficient attention mechanisms for Diffusion Transformers (DiTs), with a focus on adapting them for high-resolution image generation. The authors begin with a systematic analysis of existing efficient attention variants, identifying locality as the most promising design principle for this context. Building on this insight, they propose a circular local attention mechanism that uses a fixed-radius mask to approximate convolution-like inductive bias while retaining transformer flexibility. The method enables a linear-time attention variant and is further enhanced via knowledge distillation, achieving strong efficiency while minimizing performance degradation compared to the teacher model.", "questions": "1. Could the authors further elaborate on the advantages of the proposed circular attention mask?\nWhile the design introduces a novel geometric locality, its empirical benefits over conventional square masks (e.g., those used in Neighborhood Attention) appear marginal based on current results. A clearer justification—whether theoretical (e.g., isotropy, boundary continuity) or task-specific—would help clarify the necessity and value of this design.\n\n2. Can the authors provide more implementation details and design considerations regarding the application of their method to ultra-high-resolution generation?\nThe paper mentions several techniques (e.g., SDEdit, NTK rotary embedding, disabling resolution-aware dynamic shifting) used to enable high-resolution generation, but these are not described in sufficient depth. A more detailed technical discussion would improve the clarity and reproducibility of the proposed pipeline.\n\n3. Could the authors include ablation studies specifically focused on ultra-high-resolution settings?\nSince CLEAR’s improved efficiency at 2K–4K resolutions is one of its key claims, it would be valuable to see component-wise evaluations (e.g., NTK rotary embedding, disabling resolution-aware dynamic shifting) at these resolutions to understand which factors contribute most to quality and speed in this context.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The paper provides a detailed discussion of existing efficient attention mechanisms and analyzes their suitability in diffusion models. This comparative analysis offers valuable insight into the design space of efficient attention for image generation and strengthens the motivation for the proposed method.\n\n- The experimental section is comprehensive and well-structured, with comparisons across multiple baselines and resolutions. The reported results are convincing and support the core claims.\n\n- The paper includes thorough implementation details, making the proposed method highly reproducible and potentially practical for real-world deployment.\n\nWeaknesses:\n\n- The proposed circular local attention mask offers limited novelty compared to existing square-shaped attention masks. Empirically, the observed improvements are marginal, which raises questions about the necessity of the circular design.\n\n- While the paper discusses extending the method to ultra-high-resolution image generation (e.g., 2K and 4K) using SDEdit, NTK rotary embeddings, and disabling resolution-aware dynamic shifting, the technical details of this pipeline are insufficiently explained. Moreover, the paper lacks ablation studies isolating the impact of these components. Given that CLEAR's claimed speedup at ultra-high resolutions is one of its major selling points, the underexplored treatment of this aspect undermines the strength of the argument.", "was_revised": true, "final_justification": "The authors have provided a clear and convincing explanation of the experimental methodology, which addressed my earlier concerns. The additional results and clarifications strengthened the credibility of the work. I also appreciate the authors' perspective in analyzing the problem and the insights presented in the paper. Based on these considerations, I have decided to raise my score to an accept.", "rebuttal": "We sincerely thank Reviewer JWRk for the valuable feedback on the manuscript and are very excited that the reviewer finds the comparative analysis valuable, the experiments comprehensive and convincing, and the method highly reproducible and potentially practical. The questions are addressed below.\n\n> 1. **(W1: Circular window vs square window)** The proposed circular local attention mask offers limited novelty compared to existing square-shaped attention masks. Empirically, the observed improvements are marginal, which raises questions about the necessity of the circular design.\n> 2. **(Q1: Circular window vs square window)** Could the authors further elaborate on the advantages of the proposed circular attention mask? While the design introduces a novel geometric locality, its empirical benefits over conventional square masks (e.g., those used in Neighborhood Attention) appear marginal based on current results. A clearer justification—whether theoretical (e.g., isotropy, boundary continuity) or task-specific—would help clarify the necessity and value of this design.\n\nThanks for the good points. We would like to clarify that the main contribution of the manuscript is to identify the four key factors for successful linearization of pre-trained DiTs, listed in Tab. 1, instead of introducing a brand-new attention strategy. **Local attention based on circular windows is one feasible strategy following the four factors, and it reduces the computational overhead of square windows without hurting the performance, which is like a free lunch.** That is the reason we adopt this form.\n\nMoreover, we indeed find that circular windows can offer an additional advantage: **they are potentially more robust to image rotation**, given that circles are rational invariance, which is not the case for square windows. To validate this property, we send original images to the network and get the outputs. Then, we rotate the original images by 60 degrees, send them to the network, and rotate the outputs back. The average SSIM between the two outputs are shown below.\n\n|      | Square | Circular |\n| :--: | :----: | :------: |\n| SSIM |  0.70  |   0.75   |\n\n> 3. **(W2: Technical details of high-resolution generation)** While the paper discusses extending the method to ultra-high-resolution image generation (e.g., 2K and 4K) using SDEdit, NTK rotary embeddings, and disabling resolution-aware dynamic shifting, the technical details of this pipeline are insufficiently explained. Moreover, the paper lacks ablation studies isolating the impact of these components. Given that CLEAR's claimed speedup at ultra-high resolutions is one of its major selling points, the underexplored treatment of this aspect undermines the strength of the argument.\n>\n> 4. **(Q2: Technical details of high-resolution generation)** Can the authors provide more implementation details and design considerations regarding the application of their method to ultra-high-resolution generation? The paper mentions several techniques (e.g., SDEdit, NTK rotary embedding, disabling resolution-aware dynamic shifting) used to enable high-resolution generation, but these are not described in sufficient depth. A more detailed technical discussion would improve the clarity and reproducibility of the proposed pipeline.\n\nThanks for the insightful comments. In fact, it requires two steps to achieve efficient high-resolution generation. The first one is to adapt the original DiT to make it effective at higher scales. The second is to make it more efficient. **CLEAR mainly addresses the second step.** For the first step, the techniques are mainly adapted from previous works, *e.g.,* SDEdit and NTK rotary embedding. That is why we do not include so many details about them.\n\nNevertheless, we fully agree with the reviewer that it would be helpful to supplement the related details. We include some key information here and will further elaborate on it in the revision.\n\n1. **SDEdit** is a simple yet effective training-free method for image editing based solely on a pre-trained text-to-image diffusion model, which can be used for high-resolution generation in a coarse-to-fine manner. Specifically, we first generate an image at the native resolution scale of the diffusion model. Then, we resize it to a larger size and add a certain noise to it. A noise scale of 0.7 is adopted empirically in our experiments. Starting from this point, the model conducts the remaining denoising steps. In this way, the original image structures are preserved and low-level details are refined.\n2. **NTK rotary embedding** applies a scaling factor $s$ to the rotary base $b$ used for rotary positional embedding, *i.e.*, $b'=bs$, where $s=\\sqrt{\\frac{L_{cur}}{L_{native}}}$, and $L_{cur}$ is the current sequence length while $L_{native}$ is the native length seen during the training time.\n3. **Resolution-aware dynamic shifting** applies a factor $s$ to the projection function of the current denoising time step $t$ for the flow matching scheduler: $t'=\\frac{st}{1+(s-1)t}$, where $s$ is positively related to the image resolution. When this is enabled, at a high resolution, the projection function would appear too \"skew\", *i.e.*, the number of denoising steps allocated to the stage of high noise level is insufficient.\n\n> 5. **(Q3: Ablation studies on high-resolution generation settings)** Could the authors include ablation studies specifically focused on ultra-high-resolution settings? Since CLEAR’s improved efficiency at 2K–4K resolutions is one of its key claims, it would be valuable to see component-wise evaluations (e.g., NTK rotary embedding, disabling resolution-aware dynamic shifting) at these resolutions to understand which factors contribute most to quality and speed in this context.\n\nThanks for the good questions. Regarding the three factors demonstrated in our response to the previous question, we conduct the following GPT evaluation.\n\n|                    | Aesthetic (2K*2K) | Prompt Alignment (2K*2K) | Overall (2K*2K) | Win Rate vs Other (2K*2K) | Aesthetic (4K*4K) | Prompt Alignment (4K*4K) | Overall (4K*4K) | Win Rate vs Other (4K*4K) |\n| :----------------: | :---------------: | :----------------------: | :-------------: | :-----------------------: | :---------------: | :----------------------: | :-------------: | :-----------------------: |\n|     w/o SDEdit     |       86.32       |          89.08           |      85.37      |           0.87            |       84.15       |          87.22           |      82.36      |           0.92            |\n|      w/o NTK       |       87.73       |          91.32           |      87.10      |           0.75            |       86.54       |          91.35           |      85.93      |           0.78            |\n| w Dynamic Shifting |       88.44       |          91.68           |      86.59      |           0.70            |       84.02       |          91.40           |      84.29      |           0.73            |\n|        Ours        |       90.22       |          91.94           |      88.71      |             -             |       90.09       |          92.29           |      88.81      |             -             |\n\nThrough the results, we can observe that SDEdit mainly helps the robustness of the overall contents, and NTK rotary embedding disabling dynamic shifting mainly benefits details and aesthetics, especially for 4K images. We will include qualitative comparisons in the revision."}, {"review_id": "rhtuhrZbqS", "reviewer": "Reviewer_BEVg", "summary": "This paper proposes a convolution-like local attention mechanism called CLEAR, designed to linearize existing DiT models. By restricting each query token to interact only with tokens within its local neighborhood during attention computation, CLEAR significantly accelerates high-resolution image generation.", "questions": "My major concerns are the weakness. I would like the authors to directly address these concerns in their response. My final rating is subject to change based on the quality and clarity of the authors’ feedback.", "limitations": "The authors are encouraged to answer the questions and address the weakness above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The paper thoroughly analyzes various efficient attention mechanisms and conducts comprehensive evaluations and validations on image generation performance at a resolution of 1024×1024.\n\n2. While maintaining generation quality, CLEAR achieves a 99.5% reduction in attention computation and a 6.3× speed-up, demonstrating particularly significant advantages in 8K image generation tasks.\n\nWeaknesses\n\n1. In Table 2, are the different efficient attention methods also evaluated based on distilled models from FLUX-1.dev with their attention layers replaced accordingly? Additionally, since both Swin attention and CLEAR adopt local attention, differing mainly in window shape, is it theoretically possible to adjust Swin’s window size or shape to approximate CLEAR’s performance?\n\n2. Is the model’s ability to generate 8K-resolution images primarily distilled from FLUX-1.dev, or is it achieved through additional data-specific fine-tuning? Moreover, how does the method address the common issue of repeated content or artifacts in high-resolution image generation?\n\n3. The ablation study appears insufficient, as the individual contributions of the two additional loss terms introduced (prediction loss and attention loss) are not separately verified. Also, does each choice of attention radius 𝑟 require training a separate model, or can one model generalize across different domain sizes?", "was_revised": false, "final_justification": "", "rebuttal": "We deeply thank Reviewer BEVg for the valuable comments and are glad that the reviewer finds our analysis thorough, evaluations comprehensive, and methods effective. We would like to address the concerns as below.\n\n> 1. **(W1A: Whether all the methods are evaluated on distilled models from FLUX)** In Table 2, are the different efficient attention methods also evaluated based on distilled models from FLUX-1.dev with their attention layers replaced accordingly?\n\nThanks for the good question. Yes. All the attention methods are implemented and compared on FLUX-1.dev. We will clarify this in the revision.\n\n> 2. **(W1B: Approximation based on Swin Transformer)** Additionally, since both Swin attention and CLEAR adopt local attention, differing mainly in window shape, is it theoretically possible to adjust Swin’s window size or shape to approximate CLEAR’s performance?\n\nThanks for the insightful question. In our initial exploration, we have indeed tried to adjust the window size of Swin Transformer. **Unfortunately, we find that it still results in the \"grid\" artifact, similar to Fig. 3**. We speculate that it is because of the non-overlapped window partition for each attention layer.\n\nNevertheless, we indeed find an approximated form between CLEAR and Swin Transformer. **The key point here is to apply overlapped window partition instead.** Specifically, CLEAR can be viewed as a form of local window attention with stride=1. We show in the following study that stride=2 achieves comparable performance.\n\n|          | Aesthetic | Prompt Alignment |  Overall  | Win Rate vs Other |  GenEval  |\n| :------: | :-------: | :--------------: | :-------: | :---------------: | :-------: |\n| Stride=2 |   89.41   |      91.19       |   88.37   |       0.54        |   0.665   |\n| Stride=1 | **89.62** |    **92.13**     | **88.52** |         -         | **0.674** |\n\nThis property could be useful for hardware optimization. Thanks again to the reviewer for introducing a promising direction.\n\n> 3. **(W2A: The source of the high-resolution capacity)** Is the model’s ability to generate 8K-resolution images primarily distilled from FLUX-1.dev, or is it achieved through additional data-specific fine-tuning?\n\nThanks for the valuable question. **The model’s ability to generate 8K-resolution images comes from the ability of training-free resolution extrapolation of FLUX.** In fact, it requires two steps to achieve efficient high-resolution generation. The first one is to adapt the original DiT to make it effective at higher scales. The second is to make it more efficient. **CLEAR mainly addresses the second step.**\n\n> 4. **(W2B: The common issue of repeated content or artifacts)** Moreover, how does the method address the common issue of repeated content or artifacts in high-resolution image generation?\n\nThanks for the insightful question. In fact, as shown in Line 311, we apply the SDEdit algorithm for high-resolution generation. Specifically, we first generate an image at the native resolution scale of the diffusion model. Then, we resize it to a larger size and add a certain noise to it. A noise scale of 0.7 is adopted empirically in our experiments. Starting from this point, the model conducts the remaining denoising steps. In this way, the original image structures are preserved and low-level details are refined. In other words, the issue of repeated content is suppressed by skipping the initial denoising steps corresponding to the overall image layouts. We include the following study to verify its effectiveness.\n\n|            | Aesthetic (2K*2K) | Prompt Alignment (2K*2K) | Overall (2K*2K) | Win Rate vs Other (2K*2K) | Aesthetic (4K*4K) | Prompt Alignment (4K*4K) | Overall (4K*4K) | Win Rate vs Other (4K*4K) |\n| :--------: | :---------------: | :----------------------: | :-------------: | :-----------------------: | :---------------: | :----------------------: | :-------------: | :-----------------------: |\n| w/o SDEdit |       86.32       |          89.08           |      85.37      |           0.87            |       84.15       |          87.22           |      82.36      |           0.92            |\n|    Ours    |       90.22       |          91.94           |      88.71      |             -             |       90.09       |          92.29           |      88.81      |             -             |\n\nMoreover, as shown in Appendix E and Fig. 13, it is convenient to build CLEAR upon more advanced methods of training-free high-resolution generation, since our method is orthogonal to them. We will enhance the clarity of this part in the revision.\n\n> 5. **(W3A: Ablation studies on the individual contributions of the two additional loss terms)** The ablation study appears insufficient, as the individual contributions of the two additional loss terms introduced (prediction loss and attention loss) are not separately verified.\n\nThanks for pointing this out. **In fact, as shown in Lines 254~255, the two loss terms and the corresponding loss weights adopt the default setups in [30] and [37], which also focus on architectural distillation of diffusion models.** Nevertheless, we are willing to add the corresponding studies below.\n\n| FID (Against Original) / CLIP-T |      α=0      |    α=0.05     |     α=0.5     |      α=5      |\n| :-----------------------------: | :-----------: | :-----------: | :-----------: | :-----------: |\n|               β=0               | 14.27 / 30.90 | 13.98 / 30.77 | 13.78 / 30.66 | 13.70 / 30.56 |\n|             β=0.05              | 13.90 / 30.81 | 13.88 / 30.81 | 13.81 / 30.72 | 13.68 / 30.64 |\n|              β=0.5              | 13.83 / 30.68 | 13.82 / 30.68 | 13.72 / 30.65 | 13.47 / 30.59 |\n|               β=5               | 13.77 / 30.65 | 13.69 / 30.66 | 13.45 / 30.62 | 13.44 / 30.58 |\n\nWe find that the performance is overall insensitive to the hyper parameters. Too small values may result in insufficient effects of these loss terms, while too large values would not benefit performance, either. \n\n> 6. **(W3B: Generalizability across different 𝑟)** Also, does each choice of attention radius 𝑟 require training a separate model, or can one model generalize across different domain sizes?\n\nThanks for pointing out an interesting experiment. **Qualitatively, we find that the model trained with a smaller 𝑟 can be used for larger 𝑟. However, the opposite case does not work.** We speculate that it is because the information provided for the model is complete for inference with additional clues when 𝑟 is larger, while the information would be insufficient if 𝑟 becomes smaller.\n\nWe supplement the quantitative results of GPT evaluation below.\n\n|                       | Aesthetic | Prompt Alignment | Overall |\n| :-------------------: | :-------: | :--------------: | :-----: |\n|  Train 𝑟=8, Eval 𝑟=8  |   89.19   |      88.38       |  87.96  |\n| Train 𝑟=8, Eval 𝑟=16  |   89.49   |      88.83       |  88.31  |\n| Train 𝑟=16, Eval 𝑟=16 |   89.62   |      92.13       |  88.52  |\n| Train 𝑟=16, Eval 𝑟=8  |   86.88   |      79.08       |  82.95  |"}, {"review_id": "9ilp2TyJLc", "reviewer": "Reviewer_a9dc", "summary": "This paper aims at a linear attention mechanism that reduces the complexity of pre-trained DiTs to linear. This paper discusses 4 specific features that are essential for successfully linearizing pre-trained DiTs: 1) Locality, 2) Formulation Consistency, 3) High-Rank Attention Maps, and 4) Feature Integrity. Based on the 4 feature, this paper proposes CLEAR, which based on MM-DiT architectures: for each text query, it still gathers features from all text and image key-value tokens; while for each image query, it interacts with all text tokens and local key-value tokens fallen in a local neighborhood around it. Unlike Neighborhood Attention and standard 2D convolution, which use a square sliding window, CLEAR employs circular windows.\n\nThe student model is trained by fine-tuning the attention layer on 10K self-generated samples for 10K iterations. Experiments show that the distilled local attention layers are also compatible with different variants of the teacher model, e.g., FLUX.1-dev and FLUX.1-schnell, and various pre-trained plugins like ControlNet without requiring any adaptation. Another side benefits of locality is multi-GPU patch-wise parallel inference.", "questions": "1. Compared with Neighborhood Attention, what is the wall-time speed improvement?\n2., Except for the GFLOPS, is there any other advantages over Neighborhood Attention? Can Neighborhood Attention also be used for DiT Plugins, and Multi-GPU Parallel Inference?\n3. In Table 2, why increasing radius significantly improves the FiD against original FLUX-1.dev, while has no almost no impact against real images? What are the vital changes between them?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 1, "originality": 2, "strengths_and_weaknesses": "# Strengths:\n\n1. The proposed method significantly reduces the required generation computation and time for large images.\n2. The analysis of the 4 features are nice, except for locality.\n\n# Weakness:\n\n1. The analysis of the importance of locality is questionable. Perturbing an image of \"A small blue plane sitting on top of a field\" may not get the conclusion that local features are more important, because this specific prompt and image itself does not have long dependency. An good example to demonstrate locality should be the one in Figure 4. This image shows strong long dependency. If locality is also more important here, we can thus conclude. In particular, in Figure 4, local attention seems not to make any better sense than the global attention.\n2. The improvement compared with Neighborhood Attention seems questionable. Although the GFLOPS decrease by 12.5%, according to Figure 2, there is only a very limited improvement in real wall-time speed.\n3. The proposed method seems underperform Swin Transformer given the same GFLOPS.", "was_revised": true, "final_justification": "The rebuttal has resolved most of my concerns. I have increased my rating accordingly.", "rebuttal": "We appreciate Reviewer a9dc's thoughtful comments and are glad that the efficiency of our work for large images and the analysis of the 4 features, except for locality, are recognized. We would like to address the concerns as below.\n\n> 1. **(W1: Local attention may not be effective in dealing with long dependencies in images)** The analysis of the importance of locality is questionable. Perturbing an image of \"A small blue plane sitting on top of a field\" may not get the conclusion that local features are more important, because this specific prompt and image itself does not have long dependency. An good example to demonstrate locality should be the one in Figure 4. This image shows strong long dependency. If locality is also more important here, we can thus conclude. In particular, in Figure 4, local attention seems not to make any better sense than the global attention.\n\nThanks for the insightful review. We would like to rephrase the reviewer's concern as \"local attention may not be effective in dealing with long dependencies in images\". Regarding this, we have the following opinions:\n\n* **No matter whether there is long dependency or not, we find that locality is a necessary condition.** For the case of no long dependency, we provide examples in Figs. 3 and 5. For the case of long dependency, we supplement the GPT and GenEval scores below, which focus on evaluating the capability of handling the holistic layouts in generated images.\n\n  |               | Aesthetic | Prompt Alignment |  Overall  | Win Rate vs Other |  GenEval  |\n  | :-----------: | :-------: | :--------------: | :-------: | :---------------: | :-------: |\n  | Strided Attn. |   67.20   |      78.85       |   68.81   |       0.98        |   0.433   |\n  |  CLEAR Attn.  | **89.62** |    **92.13**     | **88.52** |         -         | **0.674** |\n\n​\tThe results suggest the importance of locality in the case of long dependency.\n\n* Since images are locally continuous by nature, local attention is crucial to preserve this property. For long dependency, the overall receptive field of the network can be enlarged with a multi-layer structure of the DiT, a feature similar to CNN.\n\n* We fully agree with the reviewer that using an image with a long dependency is helpful for the conclusion. We will use another sample to reflect this in the revision.\n\n> 2. **(W2: Circular window vs square window)** The improvement compared with Neighborhood Attention seems questionable. Although the GFLOPS decrease by 12.5%, according to Figure 2, there is only a very limited improvement in real wall-time speed.\n\nThanks for the good point. As mentioned in Lines 321~325, the standard Neighborhood Attention and the proposed method with circular windows yield comparable performance. **Given that the FLOPS can be reduced largely, we apply the latter form.** Furthermore, as shown in the following study, **it is indeed more efficient in terms of wall-clock time**, especially at higher resolutions.\n\n| Time (s) / 20 Steps | 1K*1K | 2K*2K | 4K*4K | 8K*8K  |\n| :-----------------: | :---: | :---: | :---: | :----: |\n|    Square (𝑟=8)     | 4.63  | 16.40 | 72.70 | 310.50 |\n|   Circular (𝑟=8)    | 4.38  | 15.67 | 69.41 | 293.50 |\n|    Square (𝑟=16)    | 4.84  | 18.77 | 87.09 | 382.07 |\n|   Circular (𝑟=16)   | 4.56  | 17.19 | 83.13 | 360.83 |\n\n> 3. **(W3: Improvement over Swin Transformer)** The proposed method seems underperform Swin Transformer given the same GFLOPS.\n\nThanks for the comment. In fact, as mentioned in Fig. 3, **the Swin Transformer's strategy, which lacks high-rank attention maps, results in \"grid\" artifacts and fails to generate visually coherent results**. Also, as shown in Tab. 2, our method outperforms Swin Transformer for most metrics with less GFLOPS. We further validate the superiority of the proposed solution over Swin Transformer via the following study on the GPT scores and GenEval benchmark:\n\n|             | Aesthetic | Prompt Alignment |  Overall  | Win Rate vs Other |  GenEval  |\n| :---------: | :-------: | :--------------: | :-------: | :---------------: | :-------: |\n| Swin Attn.  |   51.33   |      68.77       |   53.26   |       0.95        |   0.483   |\n| CLEAR Attn. | **89.62** |    **92.13**     | **88.52** |         -         | **0.674** |\n\n> 4. **(Q1: Advantage to neighborhood attention)** Compared with Neighborhood Attention, what is the wall-time speed improvement? 2., Except for the GFLOPS, is there any other advantages over Neighborhood Attention? Can Neighborhood Attention also be used for DiT Plugins, and Multi-GPU Parallel Inference?\n\nThanks for the good questions. **We kindly refer the reviewer to our response to W1 for the wall-time speed.** Indeed, Neighborhood Attention in principle equips with the features explored in the manuscripts. We would like to emphasize that the main contribution of the manuscript is to identify the four key factors for successful linearization of pre-trained DiTs, listed in Tab. 1, instead of introducing a brand-new attention strategy. **The initial motivation of circular windows is that they can reduce the computational overhead without hurting the performance, which is a free-launch benefit.**\n\nMoreover, we indeed find that circular windows can offer an additional advantage: **they are potentially more robust to image rotation**, given that circles are rational invariance, which is not the case for square windows. To validate this property, we send original images to the network and get the outputs. Then, we rotate the original images by 60 degrees, send them to the network, and rotate the outputs back. The average SSIM between the two outputs are shown below.\n\n|      | Square | Circular |\n| :--: | :----: | :------: |\n| SSIM |  0.70  |   0.75   |\n\n> 5. **(Q2: FID against real images in Table 2)** In Table 2, why increasing radius significantly improves the FiD against original FLUX-1.dev, while has no almost no impact against real images? What are the vital changes between them?\n\nThanks for the insightful question. The reason behind this is that the model is distilled from the original FLUX. Increasing the radius can result in a smaller architectural gap with the original model, which is reflected in the FiD scores against original FLUX-1.dev. For the scores against real images, as shown in Tab. 2, **the FLUX-1.dev itself has a relatively large FiD**, indicating that images generated by the teacher model do not strictly follow the distribution of COCO images, which also affects the distilled models.\n\nWe would like to thank Reviewer a9dc again for the in-depth reviews. We would definitely love to further interact with the reviewer if there are any further questions."}]

Author rebuttal:

Meta-review: The paper proposes CLEAR, a convolution-like local attention for linearizing pre-trained DiTs, enabling efficient high-resolution image generation with 99.5% reduced attention computations and 6.3x speedup at 8K. All reviewers (SrwR, JWRk, BEVg, a9dc) reached consensus on acceptance post-rebuttal, with numerous addressed concerns on locality analysis, high-resolution details, loss contribution and robustness, comparison to neighborhood attention etc. Overall, this work confirms attention transfer also works well across architectures for generation tasks, and adds value to the community. Please incorporate promised revisions in the camera-ready version.
Decision: Accept (poster)