LEDiT:  Your Length-Extrapolatable Diffusion Transformer without Positional Encoding
Authors: Shen Zhang
Siyuan Liang
Yaning Tan
Zhaowei Chen
Linze Li
Ge Wu
Yuhao Chen
Shuheng Li
Zhenyu Zhao
Caihua Chen
Jiajun Liang
Yao Tang
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Transformer
Length Extrapolation
Primary area: applications

Abstract: Diffusion transformers (DiTs) struggle to generate images at resolutions higher than their training resolutions. 
The primary obstacle is that the explicit positional encodings (PE), such as RoPE, need extrapolating to unseen positions which degrades performance when the inference resolution differs from training.  In this paper, We propose a Length-Extrapolatable Diffusion Transformer (LEDiT) to overcome this limitation. LEDiT needs no explicit PEs, thereby avoiding PE extrapolation. The key innovation of LEDiT lies in the use of causal attention. We demonstrate that causal attention can implicitly encode global positional information and show that such information facilitates extrapolation.  We further introduce a locality enhancement module, which captures fine-grained local information to complement the global coarse-grained position information encoded by causal attention. Experimental results on both conditional and text-to-image generation tasks demonstrate that LEDiT supports up to 4× resolution scaling (e.g., from 256$\times$256 to 512$\times$512), achieving better image quality compared to the state-of-the-art 
 length extrapolation methods. We believe that LEDiT marks a departure from the standard RoPE-based methods and offers a promising insight into length extrapolation. Project page: https://shenzhang2145.github.io/ledit/

LEDiT: Your Length-Extrapooattbbl Diffusion Transformer without Positional Encoding
Shen Zhang'
Siyuan Liang'
Yaning Tan?
Zhaowei Chen' Linze Li'
Ge Wu Yao Tang"
Yuhao Chen
Shuheng Li
Zhenyu Zhao'
Jiajun Liang Nanjjngg University Nankai University
Caihua Chen
JoO Technology
shen wwwzgaaainnooooo com
tracyliang18@gen
yao. tang@jiiov.ccm
Abstract
Diffusion transformers (DiTs) struggle generate images resolutions higher than their training resolutions. The primary obstacle is that the explicit positional encod- ings (PE), such as RoPE, need extrapolating to unseen positions which degrades performance when the inference resolution differs from training. In this paper, We propose
Diffusion Transformer (LEDiT) to overcome limitation. LEDiT needs explicit PEs, thereby avoiding PE extrapolation. The key innovation LEDiT lies in the use of causal attention. We demonstrate that causal attention can implicitly encode global positional information and show that such information facilitates extrapolation. We further introduce a locality enhance- ment module, which captures fine-grained local information complement the global coarse-grained position information encoded by causal attention. Experi- mental results both conditional and text-to-image generation tasks demonstrate that LEDiT supports up to 4x resolution scaling (e.g.. from 256x256 to 512x512), achieving better image quality compared the state-of-the-art length extrapolation methods. We believe that LEDiT marks a departure from the standard RoPE-based methods and offers a promising insight into length extrapolation. Project page: https://shenzhhngg 145.214555512 155.iithubio//edil
Figure 1: Selected restutton samples (5122, 512x256, 256x512, 3842 2562. 1282) from LEDiT-XL/2 trained on ImageNet 256 resolution. LEDiT can generate high-quality images beyond the limitations of training resolution.
"Correspondngg author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Training Resolution
Beyond Training Resolution 512x512
256x256
DiT-Sin/Cos
FiTy2-VisionNTK FiTv2-VisionYaRN
DiT-RIFLEx
LEDIT
Figure Diffussion Transformer performs well at the training resolution. However, when extrapolated to higher resolutions, DiT [3T], FiT [28 43 and RIFLEx [50] suffer notable quality degradation. In contrast, our LEDiT can generate reasonable and realistic higher-resolution images with fine- graiind details. The class label is 270 (white wolf).
1 Introduction
Diffusion models have emerged as powerful foundation technique in vision generation tasks. The architecture of diffusion models has progressed from U-Net [36 35 to vaasformer-based
designs MM Diffusion Transformers (DiTs) have become state-of-the-art generators [I]
4 48 44
Despite their success, DiTs face critical limitations when generating images at resolutions beyond those encountered during training 28 42 As shown in Figure 2 DiTs trained on ImageNet ( x56x256 resolution produce high-quality samples at this scale, but struggle to generalize to higher resolutions such as 512x512. Due to the expensive quadratic cost of self-attention and the scarcity of large-scale, high-resolution datasets, models are typically trained at relatively small resolutions. In practice, many real-world applications, such as high-definition film and computer graphics, require higher-resolution images, presenting a significant length extrapolation challenge for current DiTs.
highlight the importance of positional encoding (PE) in length extrapolation. Rotary Positional Embeddings (RoPE) (39] and its variants, such as NTK-
Many studies 33 39
32
23
28
have been developed improve the extrapolation ability of language transformers. In the vision domain, Flexible Vision Transformers (FiT) [28] 43) integrate RoPE into DiTs support variable input resolutions. RIFLEx [50] reduced the intrinsic frequency of RoPE to alleviate extrapolation issues. Despite these advances, performance still degrades notably beyond the training resolution (see Figure 2 Since diffusion models are not trained for such out-of-range positions, this leads to a distribution shift in positional indices and results in out-of- distribution issues [ On the other hand, Recent studies [I5] 23 challenge the need of explicit PE, showing that large language models (LLMs) without PE (NoPE) perform well in in-distribution settings and even outperform explicit PEs length extrapolation. The advantage NoPE lies in avoiding PE extrapolation, which reduces performance when inference resolution differs from training. LookHere 12 removes positional encoding and carefully designs various combinations causal masks, achieving notable success in image recognition. However, directly applying LookHere to image generation tasks results in severe object duplication (see Figure [5) and fails to yield effective length extrapolation. Therefore, it remains unclear whether DiTs can similarly benefit from NoPE for resolution extrapolation. We ask: Can DiTs leverage NoPE to train at low resolutions and generalize to higher resolutions?
scaling 2 and YaRN 32
In this paper, We propose explicit PE and can generate high-quality images at arbitrary resolutions. A key architectural modification the adoption of causal attention. We demonstrate that causal attention can implicitly encode positional information. Specifically, we provide both theoretical and empirical evidence that token variance decreases when position increases, providing an implicit ordering (see Section (.22). We reveal that this implicit position ordering yields better extrapolation abilities than explicit PEs. Furthermore, we introduce a negligible-cost multi-dilation convolution as a locality enhancement module to improve local fine-grained details, complementing the global coarse-grained information captured by causal attention.
Extrapolatable Diffusion Transformer (LEDiT), which removes
We conduct extensive experiments on both conditional and text-to ext-to-imaae generation tasks to validate the effectiveness of LEDiT. Notably, LEDiT supports up to 4x inference resolution scaling while maintaining structural fidelity and fine-grained details, outperforming ttaee-of-the-art extrapolation methods. Moreover, LEDIT can generate images with arbitrary aspect ratios (e.g., 512x384 or 512x256) without any multi-aspect-rant training techniques. We also show that fine-tuning LEDiT

---

from a pretrained DiT for only 100K steps yields strong extrapolation performance, highlighting its potential for efficient integration into existing powerful DiTs. We hope our findings provide valuable insights for future research on transformer length extrapolation.
2 Related Work
Diffusion Transformers. Building the success of DiT [ subsequent works such as PixArt- Alpha A and PixArt-Sigma (3] further extend diffusion transformers for higher-quality image generation. Stable Diffusion 3 11] and Flux [25]
substantially improve the performance of diffusion transformers by scaling up parameters. Sana 46 focused on fast generation through deep compres-
sion autoencoding and linear attention Despite these advances, most DiT's struggle when inference resolution differs from training, motivating our exploration of length extrapolation.
Length Extrapolation in Language. Since the introduction of the transformer [41. length extrapolation has remained a significant challenge, with positional encoding playing a critical 32 23 29]. Absolute Positional Encoding (APE) [4]] struggles to handle longer sequences. To address this, ALiBi [33] modifies attention biases to facilitate length extrapolation. Rotary Position Embedding (RoPE) [39] and its extrapolation refinements, including NTK-aware scaling 2 and YaRN 32 further improve length generalization. Adaptive embedding schemes like Data-Adaptive Positional Encoding (DAPE) [5]) and Contextual Positional Encoding (CoPH) [13] have also been explored. In contrast to explicit PEs, NoPE [5] 23 demonstrates that language models can implicitly encode positional information. We independently observe similar phenomenon in diffusion models. Importantly, compared to (5). our analysis provides theoretical proof under more relaxed assumptions, requiring only finite mean and variance and imposing weaker constraints on weight matrices. Furthermore, we demonstrate that this implicit positional information benefits length extrapolation in diffusion models, which the main focus of this paper.
role [3 39
Length Extrapolation in Diffusion. Length extrapolation has been extensively studied in diffusion U-Net architectures [16] 44 10 20 14], but remains largely unexplored in DīTs. RoPE-Mixed (17) employs mnation-based embeddings for variable image sizes. FiT
28
43 adopts RoPF, NTK-Awar
and YaRN in 21) variants for resolution extrapolation. RIFLEx
(50 analyzes the role of different
frequency components in RoPE and found that reducing the intrinsic frequency can boost length extrapolation. LookHere (12] carefully designs various combinations of causal masks provide directional inductive biases. It conducts experiments to demonstrate the extrapolation capabilities in classification tasks. However, directly adapting LookHere to image generation tasks results in severe object duplication (see Figure 15). Therefore, comprehensive strategy for high-resolution extrapolation in diffusion transformers remains elusive. In this work, we address this gap by enabling DiTs to generate high-fidelity images at arbitrary resolutions. There are some conceptual similarities between LEDiT and LookHere. Both LEDiT and LookHere explore causal attention to enhance length extrapolation. But there are key differences. LEDiT provides a theoretical framework that explains why causal attention is capable of encoding positional information and enabling length extrapolation. Moreover, LEDIT combines simple causal attention with multi-dilation convolutions, effectively mitigating object duplication (see Figure 15) and achieving better extrapolation performance.
3 Method
We first introduce some preliminaries about DiT's and causal attention. DiTs is primarily built upon the ViT [ Each DiT block contains multi-head self-attention (MSA), followed by adaptive layer normalization (AdaLN) and feed-forward network (MLP), Residual connections are applied by scaling ag and ay. Given an input x #xx the computation of DiT block is as follows:
2g = Flatten + Epon
2 = (SAA(adaLN(zx-1114 a1
(2)
22 MLPPadal.N(E/, c)) + a/21-
Causal attention only allows the given position in sequence to attend to the previous positions, not to future positions. The causal attention map is:
QKK Vite
A = softmax
+ M1)
(4)
3

---

Lx
Lx
MLP
ML.P
MLP
Layer Norm
Layer Norm
Layer Norm
Multi-Head (Causal) Attention
Multi-Head Attention
Multi-Head
Attention
Layer Norm
Layer Norm
RoPE
Layer Norm
Sinusoidal PF
Conv
Input Tokens
Inpat Tokems
Input Tokens
(a) DiT-Sin/Cos PE
(b) DiT-RoPE
LEDiT
Figure 3: Comparison between DiT-Sin/Cos PE, DřT-RoPE, and our LEDiT. We omit AdaLN for the sake of simplicity. DiT-Sin/Cos PE the vanilla DiT [3I, which incorporates Sinusoidal PE into the transformer. DiT-RoPE introduces rotary position encoding by rotating the query and key in each transformer block. In contrast, our LEDiT model does require explicit position encoding. The main difference lies in the incorporation of causal attention and convolution after patchification.
where Q E Raxde and K E Raxd are query and key, de is the dimension, and M E R is mask matrix with definition follows:
if S i, if j i.
Mij
(5)
This ensures attention scores for future tokens are nearly zero after softmax, enforcing strict causality.
3.1 LEDiT Block
The overall architecture of LEDiT is illustrated in Figure 3c Our LEDiT does not need explicit
PEs. The main modifications include the use of causal attention and a negligible-cost multi-dilation convolution. We design LEDiT blocks alternate between causal attention and self-attention The first LEDiT block uses self-attention, formulated as:
2 MSA(adaLN(zz-nnttc))) + a
(6)
Zq MM + a/2'
(7)
The subsequent LEDiT block uses causal attention, which can be written as:
2/41 MCA(adal c)) + ae+1
(8)
Zf+1 MLPP(aaLL(((441 d/412/41
(9)
where MCA represents multi-head causal attention. We explore more LEDiT designs in Table ld
3.2
Why Causal Attention
Explicit PEs are widely used in transformers, but their performance degrades when extrapolating to resolutions larger than those seen during training, as shown in Figure To address this limitation. we attempt to remove explicit PEs to avoid position encoding extrapolation. Prior work [23] suggests that causal attention enables better length extrapolation without PEs LLM. Motivated by this, we introduce causal attention to diffusion models and further demonstrate that causal attention (i) implicitly encodes positional information to tokens, and (ii) that such implicit positional encodings facilitate length extrapolation.

---

T L-15
442, L-3
T-714,1-4
290
75
190
50 189 150 200
250
50
100 15t
300
253
50
109
156 Pestbun
20
150
203
408 538 998 Podlium
1080
209 a6 e0d 901 1093
Prodtin
Packion
Pasiiin
Figure 4: Var(ga) distributions across various timestep (T) and DiT layers (L). Left: variance distribution the training resolution (256x256). Right: variance distribution beyond the training resolution (512x512). Best viewed when zoomed in.
Causal attention implicitly encodes positional information. we formally establish that, under spe- cific assumptions, the variance of causal attention output encodes positional information. Specifically, we prove the following theorem:
Theorem 3.1. For a Transformer architecture with Causal Attention, assume that the value V is i.i.d. with mean by and variance oy. Then, the variance of the causal attention output Yς at position and dimention is given by:
Var(Yu) = i+1
Ov +
uy.
(10)
i(i +
When i is large, we can approximate
4-1 a(++1)
F+I leading to the reasonable approximation
Var(Yn)
(11)
i+1'
where the constant C. = 207 + p-:
Please refer to Appendix A for the complete proof. This theorem reveals that, if the conditions are met, the variance inversely proportional to the position a rate of FFI' We further conduct experiments to verify whether applying causal attention in DiT can assign different variances to different positions.
Sir/Des PR
RePF
Casai
5124572 250x255
We train a DīT-XL/2 that replaces all self-attention with causal attention and use it to verify the theorem. Given an input sequence E RRnxde, causal attention takes = and outputs y (y1:..., Un) RRxx We approximate Var(ya) using the variance of yı- As shown in the left figures of Figure 4 Var(ya) inversely proportional the position i across various timestep and layers. This indicates the existence of causal attention in DiT that meets the conditions of the theorem. Intuitively, a smaller Var(y,) indicates that its elements are more concentrated. During training, the neural network can learn to leverage this concentration to determine token position, thereby implicitly encoding positional ordering. We also observe variance distribution in the later denoising stage differs from the theorem. To evaluate the impact of this phenomenon, we conduct experiments with switching from causal attention to self-attention at different timesteps. Our findings show that positional information is primarily acquired in the early denoising stage, and the causal attention with variance deviation the later denoising stage has minor effects on extrapolation. See Section Coction[ooor for the discussion.
Figure 5: Explicit PEs degrade in ex- trapolation, while causal attention con- sistently outputs coherent coarse-grained structures. The class label 17 (jay).
The implicit positional information facillitaess length extrapolation. As shown in the right figures of Figure 4 when extrapolating to higher resolutions, the variance remains inversely proportional to the position, consistent with the in-distribution variance distribution. This preservation of implicit positional ordering enables the model to generalize across larger resolutions, as illustrated in Figure 5 When scaling the resolution by 4x, models with explicit positional encodings exhibit severe structural degradation, while models with causal attention continue to generate structurally coherent objects. In addition to providing implicit positional information to tokens, causal attention also acts as a learnable, global receptive-field mechanism, which differs from static positional encodings. It can make predictions based on previous tokens and learn the dependencies between tokens from large-scale data, which may also enhance the model's ability extrapolate to higher resolution.
5

---

(a) One- Dimension
(b) Mask Lower-right
(c) Unmask Neighborhood
(d) Mask Lower-right Comer
Figure 6: Comparison of causal attention scan variants. We use variant (d) as our default,
Causal scan variants. We introduce four causal attention scan variants, as depicted in Figure (a) represents the traditional 1D scan used in PixeICNN 44), where each position attends only to preceding tokens in flattened sequence. To leverage the spatial characteristics of images, we also
consider scanning along both the height and width dimensions and propose (b) We ablate the
performance of these variants in Table lb Variant is set as our default.
3.3 Locality Enhancement
Although causal attention provides tokens with global implicit positional ordering, when is large, the variance between adjacent tokens be- comes indistinguishable (see Figure prevent- ing accurate position information and leading to blurry images, see Figures 5 and T8a To distinguish the relative relationships between neighborhood tokens, we need enhance the local perception abilities of the neural network. lo-
Input Token
Input Token
Specifically, introduce convolution cality enhancement module. Previous work [55] replaced the qkv-prop projecrerttrr with convolution or integrated convolution into the MLP 47 46
LEDIT
LEDiT Block
x1
LEDiT Block
in each transformer block, which significantly increased the model's computational cost. We find that adding a convolution after patchifica- tion is sufficient, while only increasing ignor- able overhead. This can written by slightly modifying Equation (
Noise and
Noise and Σ
Noise and Σ
Training Pipeline
Inference Pipeline
Figure 7: LEDiT pipeline.
Platten(Cs,,....1 (Pach(faa))
(12)
where Cs.p.s.d a denotes convolution filter with kernel size k, padding stride and dilation d. Zero padding is applied, which enables convolution to leak local positional information 47).
Multi-dilation training strategy. Although the generated higher-resolution images are visually compelling, they often encounter duplicated object artifacts due to the fixed receptive fields of convolutional kernels [16]. To mitigate this problem, we adopt a multi muult--diattion training strategy, wherein dilation and padding are randomly adjusted during training (see Figure 77. For standard convolution filter C3,1,1,1, we set probability to expand both its dilation rate and padding size to 2, transforming it into C3.2,12- During inference, we empirically find that fixing dilation and padding as is sufficient. This strategy trains shared-pare parameter convolutions with varying receptive fields and empirically improves extrapolation abilities.
Experiments
4.1 Experiment Settings
Model Architecture. For conditional generation on ImageNet [6], we use patch size p 2 and follow DiT-XL (31) set the same layers, hidden size, and attention heads for the XLarge model,
denoted by LEDiT-XI/2. For text-to-image generation the hidden dimension as 768 and the model depth as 24, following the design REPA 48 denoted
COCO (26), We use MMDiT [1] and set
LEMMDIT. We use the CLIP (34 text encoder to compute text captions.

---

Table I: Ablations using LEDiT-XI/2 on 256x 256 ImageNet. We report FID and scores. For each ablation, we load the pretrained DiT weights and fine-tune LEDiT-XI/2 for 100K iterations. Default settings are marked in gray See Figure 18 for visualization.
(a) Components ablations. Causal attention and convolution are effec-
(b) Casual scan variants.
(c) Multi-dilation strategy.
2D casual scan variants out-
LEDiT benefits from multi- dilation strategy.
tive in length extrapolation.
perform ID variants.
case
FID
1St
scan
FID
IS
case
FID
IS
NoPE
378.95
3.79
(a)
62.49
78.25
w/o multi-dila
39.20 127.8 84
Cau.
286.01
6.96
(b)
89.77
50.10
w/ multi-dila
35,86
139.91
+ Con.
130.91
28.66
(c)
43.17
116.03
Cau. + Con.
35.86
139.91
35.86
139.91
(d) Block design. The alternating order works better than the sequen- tial order.
(c) Multi-dilation probabil-
(f) Dilation rate. (2,3) means randomly selecting 2 or as the rate during training.
ity. LEDiT with a small prob- ability works better.
order
FID4
1St
prob
FID
1St
dilation FID. 1ST
CAL2 + SALz
36.81
139.88
0
39.20
127.84
39.20 127 .84
SALa + CAre
48.65
103.14
0.1
35.86
139.91
2 35,86 139.91
(CA.SA)z
36.05
143.26
0.2
37.99
135.85
(2.3)
37.24
136.23
(SA,CA)zz
35.86
139.91
0.5
37.56
133.07
Training Details. The experiments are trained on ImageNet 6 with 256x256 and 512x512 resolutions, and COCO [26] with x56x25 resolution. On ImageNet, We (i) train the randomly initialized LEDiT for 400K steps or (ii) fine-tune LEDiT for 100K steps. We set the batch size as 256. On COCO, We follow REPA [8 and train LEMMDIT for 200K steps with batch size of 192. We use 8x NVIDIA V100 GPUs as default training hardware.
Evaluation Metrics. We primarily use Fréchet Incep- tion Distance (FID) [18), the standard metric for evaluat- ing generative models. We additionally report Inception Score 337. sFID [30,, and Precision/Recall 24 as sec- ondary metrics. Without further elaboration, ImageNet, we generate 50K samples using 250 DDPM sampling steps with classifier-free guidance (CFG) scale of 1.5, On COCO, we generate 40,504 images (one per caption) using 50 ODE sampling steps with CFG=2.0. For fair comparison, all values reported in this paper are obtained by exporting samples and using ADM's TensorFlow eval- uation suite 7
Figure 8: Visualization of the ablation study. The first row illustrates the ab- lations of the components proposed in this paper, while the second row displays the ablations of the causal sean vari- ants. The models are trained on 256x 256 ImageNet and generate images with 512x512 resolution. See Figure 18 for more visualization.
Evaluation Resolution. Compared to previous work (28) 431, this paper tests at more extreme resolutions. When trained on ImageNet 256 256, we compare the extrap- olation performance of LEDiT with other methods [31] 50) at 384 (2.25x), 448 x 448 (about 3x), and 512x512 (4x) resolutions. When trained on Ima- geNet 512, compare LEDiT with other methods
28 43
at 768 x 768 (2.25x), 896x896 (about 3x), and 1024x 1024 (4x) resolutions. Additionally, we assess performance at different aspect ratios, specifically 512x384 (3:2) and 384x512 (2:3). On COCO, extrapolation is evaluated at 512x512 (4x). All token lengths are much longer than those seen during training. Following the widely adopted practice in transformers, we apply attention scaling [22 for length extrapolation.
4.2 LEDIT Ablations
In this section, we ablate LEDIT design settings on 256x256 ImageNet. We use LEDIT-XII/2 to ensure that our method works at scale. We evaluate performance by loading DiT pretrained weights
7

---

Table 2: Comparison of state-of-the-art extrapolation methods and LEDiT trained 256x256 ImageNet at various resolutions beyond the training image size. We set CFG=1.5. indicates training from seratch. indicates additional architecture refinement.
584
448x.448
S12
Madell
FInL
sFID_
IST
Proc.T
Rec.
Fin]
.FTD]
1ST
Prec.
Ro:
FID]
.FIDL
151
Pres11
Rec:
Da PE*
162.50
14.91
0.15
0.27
188.42
191.58
0.06
0.11
316.22
158.
0.10
0.04
DIT VisionNTK*
45.81
50.42
99.92
0.48
0.42
124.83
113.88
37.79
0.22
0.39
174.68
139.23
16.28
0.10
0.30
23.45
53.25
158.46
0.63
0.35
64.91
88. 59
70:04
0.36
0.34
109.00
109.00
38.38
0.21
0.30
RIFLE**
64.36
0.66
92:25
0.42
0.35
107.32
0.17
0.30
LEDIT
15.08
10.94
138.25
0.75
0.31
64.84
48.06
103:05
0.51
0.25
56,02
65.99
63.26
0:43
0.21
LEDTT
12.07
30.47
188,15
6.80
0.31
20.91
48.37
152.57
0.69
0.25
34.29
64.10
110.04
0.56
0.22
DIT Sin/Cas
87.03
116.67
44.93
0.31
0.31
168.23
145.45
15.25
0.12
0.23
213.77
168.51
7.98
0.06
0.13
DIT VisioNNTX
71.25
30,59
67.42
0.35
0.51
184.29
122.99
16.94
0.30
0,41
206.56
144,99
8.82
0.04
0.17
Visioon
15.35
0.71
50.81
0.56
0.35
64.63
DIT RIPLEX
57.88
77.27
75.88
0.37
0.55
186.76
129.17
17.13
0.09
0.41
251.92
163.77
10.39
0.04
0.12
TVT22 VisionNTK
38.43
47.09
107.88
0.45
0.54
179.01
115,12
18.20
0.08
0.42
257.63
171.10
6.72
0.01
0.21
VisioaYRN
35.13
157.93
0.55
0.48
71.94
64:72
64.49
0.29
0.51
80
118.21
20.76
0.11
0.27
LEDIT
9.34
25.02
281.09
0.78
0.39
17.62
39.43
214.99
0.66
0.34
33.25
54.36
138.01
0.32
0.31
and fine-tuning for 100K iterations. we set CFG=1.5, generate 10K images at 512x512 resolution, and report FID-IOK and IS-10K.
Components ablations. Table la shows the influence of each component of LEDiT. Removing PE (NoPE) degrades DiT severely. Both causal attention and convolution can significantly enhance extrapolation performance. Combining these two components decreases the FID from 378.95 to 35,86 and increases the IS from 3.79 to 139.91, yielding the optimal performance. Figure illustrates the impact of causal attention and convolution. DiT with NoPE generates noise-like images, indicating that without PE, DiT cannot capture token positional information. Incorporating causal attention yields structurally coherent objects but insufficient high- freuufrey details, since causal attention provides global ordering but struggles with local distinctions (see Section (33) Conversely, introducing convolution provides adequate high-frequency details, but leads to duplicated objects
due to the lack of a global receptive field. When both are combined, the generated images exhibit realistic object structures well as fine-grained details. Therefore, we use both causal attention and convolution as the default setting.
Causal scan variants. Table Lb presents a quantitative comparison of different scan variants. Except for variant (b), 2D scan variants (c) and (d) outperform the ID variant (a) in both FID and IS. Figure & shows that variants (c) and (d) produce more coherent object structures and finer details than variant (a). Although variant (b) is also a 2D scan, leads to blurred images, resulting in lower FID and sFID scores. We adopt variant (d) as our default.
Multi-dilation strategy. As shown in Figure T8c when adapting multiple receptive fields, the multi-dilation strategy enhances LEDiT's performance and significantly mitigates object duplication. Table Tc shows that LEDIT with the multi-dilation strategy achieves better FID and IS scores. We adopt the multi-dilation strategy as the default setting.
Block design. Table Id compares different orders of causal and self-attention blocks. The first two rows use sequential blocks, whereas the last two employ an alternating arrangement. Both orders achieve strong performance, but sequential order exhibits higher variance while alternating orders more stable. We thus use the alternating order with self-attention preceding causal attention as our default.
Multi-dilation probability. Table Te shows the result of different multi-dilation probabilities p. where y = O disables the strategy. As p increases, FID initially decreases and then rises. At 0.5. the receptive field alternates equally between 3x3 and 5x5. The experimental results show that frequent conv parameter changes during training slow convergence and introduce instability, leading to worse performance, while smaller p mitigates object duplication and ensures stable training. So we adopt p= 0.1 as the default setting.
Dilation rate. We also evaluate different dilation rates r to accommodate multiple receptive fields (Table If. For instance,
(2, 3) means there is a p/2 probability of choosing r = 2 or r = 3. It can be seen that both r = 2 and r = (2.3) can improve performance, but r = (2,3) is less effective than = 2, likely due to the increased complexity of handling multiple dilation values. Nevertheless, successfully adapting convolution to multiple receptive fields may further benefit extrapolation. In this paper, retain 1 by default.
8

---

Cin PE
DIT DiE-VisinNIK
DTF VasieYaRN
DIT RIFLEX
FIL22 VisionNTK
FTTv2 VaionNNiinNNRN
LEDIT
1026x 512x512
Class Labe
Cluss Label 38
Figure 9: Qualitative comparison with other methods. The resolution and class label are located the left of the image. We use the model trained on 256x256 ImageNet to generate images at 512x512 resolution, and the model trained on 512x512 ImageNet generate images at x024 1024 resolution. We set CFG=4.0. Best viewed when zoomed in. See Figure 19 and Figure 20 for more comparison.
4.3 Main Results
256 x 256 ImageNet. In Figure 9 we present qualitative comparison between LEDiT and other methods. Vanilla DiT (DiT-Sin/Cos PE) suffers from severe image quality degradation. When combined with VisionNTK, VisionYaRN or RIFLEx, DiT generates images with detailed textures but introduces unrealistic object structures. FiT produces images with severely degraded quality. In contrast, LEDiT produces images with realistic object structures and rich details. The quantitative comparison is reported in Table LEDiT substantially outperforms previous extrapolation methods. At resolution of 384x 384, LEDIT reduces the previous best FID-50K of 13.51 (achieved by DiT- Vit-ViiionYaR1 to 9.34. As the resolution increases, LEDiT further widens the performance gap, lowering the best previous score from 49.86 33.25 512x 512. When trained from scratch, LEDiT also achieves significantly better FID and sFID scores compared to its counterparts, demonstrating the effectiveness of LEDiT in both fine-tuning and waining-from-ssreee minor architectural refinement during training from scratch can significantly improve extrapolation performance (see Section F). We present the comparison with LookHere Section We report the result of 512 x 512 ImageNet in Section H
scenarios. We further find that
Arbitrary Aspect Ratio Extension.
Table 3: Comparison of state-of-the-art extrapolation meth- ods and our LEDiT trained on 256x256 ImageNet at arbi- trary aspect ratios. We set CFG=1.5.
Beyond generating square images, we evaluate the generalization abilities of LEDiT across different aspect ra- tios. Unlike FiT (28 43), do not apply multiple aspect ratio train- ing techniques. Instead, we directly use the LEDiT-XL/2 model trained on the center-cropped 256x256 Im- ageNet dataset. This highlights the model's inherent generalization abili- ties. The quantitative results, reported in Table 3 demonstrate LEDiT's su- periority. It achieves the best FID scores, with 20.29 and 18.82 at res- olutions of 512x384 and 384x512, respectively, notably outperforming VisionNTK, VisionYaRN, and RIFLEx. These results confirm that LEDiT can generate high- quality images across diverse aspect ratios even without various aspect ratio training techniques. See Figure 20 for the qualitative comparison.
Model
Resolution
FID.
<FID]
IS1
Prec:1
Rec: 1
DiT-Sin/Cos
153.74
144.66
16.52
0.13
0.27
DIT VisionNTK
179.71
117.81
15.88
0.09
0.36
YRR
512x384
25.69
46,22
176.19
0.58
0.36
DIT RIFLEX
163.38
117.22
20.36
0.11
0.44
0.08
YaR
56.04
51.05
81.96
0.35
0.47
LEDIT
20.29
38.52
191.69
0.63
0.35
DiT-Sin/Cos
DIT VisioNTTK
150.70
110.67
25.88
0.14
0.41
DET-Vision DIT-RIFLEX
22.02
48.72
202.03
0.61
0.35
384x512
143.84
116.93
27.42
0.14
0.45
FTT22 VisioNNTK
177.44
114.56
17.14
0.08
0.40
YARN
49.67
$7.07
99.29
0.39
0.41
LEDIT
18.82
42.64
205.38
0.64
0.36
Text-to-image generation. We further evaluate the performance of our method on the text-to-immn to-image generation task. Table 4 shows that LEMMDiT perform favorably compared to -tate-oflie ffte--of-theaite the-art extrapolation methods. Specifically, LEMMDiT achieves an FID of 29.89 and an sFID of 39.98, representing substantial reduction compared to Vanilla MMDiT and RoPE-based variants. The CLIP Score 34 shows that LEDiT consistently outperforms other methods in terms of semantic we present qualitative comparison between LEMMDiT and other methods.
coherence. In Figure 21

---

Vanilla MMDiT produces images with severe quality degradation. VisionNTK generates images with fine details but suffers from object duplication. Vision YaRN and RIFLEx yield more plausible object structures but lose fine-grained details. In contrast, LEDiT generates images with reasonable structures and rich details.
5 Conclusion
In this paper, we introduce a novel Diffusion Transformer, named B Diffusion Transformer (LEDiT). LEDiT does not require explicit positional encod- ings such RoPE. By combining causal attention and a locality enhancement module, LEDiT can im- plicitly encode positional information, which facilitates length extrapolation. Conditional and text-to extt-to-mmage gen- eration shows that LEDiT supports up to 4x inference resolution scaling. Compared to previous extrapolation methods, we can generate images with more coherent object structures and richer details. We hope that LEDiT's principled departure from explicit positional encoding paradigms will not only advance the frontier of length extrapolation, but also inspire new perspectives on the foundational design space of transformer architecture.
Table 4: Texl-lo-image comparison of state-of-the-art extrap- olation methods and our LEMMDIT trained 256 COCO. The inference resolution is 512x512. CFG=2.0.
We set
Model
FID1
sFID]
ISt
Proc.1
Roc:1
CLIP
MMDIT
160.72
156.38
9.09
0.07
0.09
22.94
MMDiT VIYARN
163.88
93.39
10.35
0.11
0.30
23.37
AMDIT-RIFLES
34.58
52.58
21.75
0.40
0.36
27.14
LEMMDIT
29.89
39.98
24.11
0.44
0.31
27.82
References
[1] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth 2023.
words: A vit backbone for diffusion models. In CVPR, pages
[2] bloc97. Ntk-aware scaled rope allows llama models have extended (8k+) con- text size without any fine-tuning and minimal perplexity degradation. https //www.reddit rwddit.ddet...
allows /tt/t/////////oaaaa///////o/ http:///mmeee///a/////ttaaaaaaaa sccaaaaaarreeaaaeeeepreereeeeeee
hacpe_oddel_ss_eeeeeeee taatellsttt_heeee/eee 2023.
[3] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to-strong training of diffusion transformer for text-to-image generation. In ECCV, pages 74-91. Springer, 2024.
[4] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR, 2024.
[5] Ta-Chung Chi, Ting- Han Fan, Li-We Chen, Alexander Rudnicky, and Peter J Ramadge. Latent positional information the self-attention variance transformer language models without positional embeddings. In ACL, 2023.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009.
[7] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In NeurlPS, pages 8780-8794, 2021.
[8] Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond million tokens. In ICML, 2024.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
10

---

[10] Ruoyi Du, Dongliang Chang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. Demofusion: Democratising high-resolution image generation with no $$S. In CVPR. pages 6159-6168, 2024.
[11] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In ICML, 2024.
[12] Anthony Fuller, Daniel Kyrollos, Yousef Yassin, and James Green. Lookhere: Vision transform- ers with directed attention generalize and extrapolate. In NewrPPS, 2024.
[13] Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. Contextual position encoding:
Learning to count what's important. arXiv preprint arXiv:2405. 2024.
[14] Langing Guo, Yingqing He, Haoxin Chen, Menghan Xia, Xiaodong Cun, Yufei Wang, Siyu Huang, Yong Zhang, Xintao Wang. Qifeng Chen, et al. Make cheap scaling: self-cascade diffusion model for higher-resolution adaptation. In ECCV, 2024.
[15] Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. In EMNLP, 2022.
[16] Yingqing He, Shaoshu Yang, Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang, Ran He, Qifeng Chen, and Ying Shan, Scalecrafter: Yuning-free higher-resolution visual generation with diffusion models. In ICLR, 2024.
[17] Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In ECCV, pages 289-305. Springer, 2024.
[18] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NeurIPS, 2017.
[19] Marius Hobbhahn, Agustinus Kristiadi, and Philipp Hennig. Fast predictive uncertainty for classification with bayesian deep networks. UAI, 2022.
[20] Linjiang Huang, Rongyao Fang, Aiping Zhang, Guanglu Song, Si Liu, Yu Liu, and Hongsheng Li. Fouriscale: A frequency perspective on training-free high-resolution image synthesis. In ECCV, 2024.
[21] Md Amirul Islam, Sen Jia, and Neil DB Bruce. How much position information do convolutional neural networks encode? In ICLR, 2020.
[22] Zhiyu Jin, Xuli Shen, Bin Li, and Xiangyang Xue. Training-free diffusion model adaptation for variable-sized text-to-image synthesis. In NeurIPS, 2023.
[23] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. In NeurlPS, 2023.
[24] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In NeurIPS, 2019.
[25] Black Forest Labs. Flux: Efficient latent space modeling for image generation. 2024.
https
//github.
[26] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755, 2014.
[27] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
[28) Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. Fit: Flexible vision transformer for diffusion model. In ICML, 2024.
11

---

[29] Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng Chen, Base of rope bounds context length. arXiv preprint arXiv:2405.44591, 2024.
[30] Charlie Nash, Jacob Menick, Sander Dieleman, and Peter Battaglia. Generating images with sparse representations. In ICML, pages 7958-7966 1968.. PMLR, 2021.
[31] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pages 4195 4205, 2023.
[32] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. In ICLR, 2024.
[33] Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables imput length extrapolation. ICLR, 2022.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[36] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. MICCAI, pages 234-241. Springer, 2015.
[37] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NewrlPS, 2016.
[38] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. In NeurlPS, 2022.
[39] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputingg 568: 127063, 2024.
[40] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. In NeurIPS, 2016.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[42] Shuai Wang, Zexian Li, Tianhui Song, Xubin Li, Tiezheng Ge, Bo Zheng, and Limin Wang. Exploring den-like architecture for fast image generation with arbitrary resolution. In NeurIPS, 2024.
[43] ZiDong Wang, Zeyu Lu, Di Huang, Cai Zhou, Wanli Ouyang, et al. Fitv2: Scalable and improved flexible vision transformer for diffusion model. arXiv preprint arX2v:2410.13925. 2024.
[44] Ge Wu, Shen Zhang, Ruijing Shi, Shanghua Gao, Zhenyuan Chen, Lei Wang, Zhaowei Chen, Hongeheng Gao, Yao Tang, Jian Yang. et al. Representation entanglement for generation: Training diffusion transformers is much easier than you think. arXiv preprint arXiv:2507.01467, 2025.
[45] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In ICCV, pages 22-31, 2021.
[46] Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. In ICLR, 2025.
[47] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, pages 12077-12090, 2021.
12

---

[48] Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. In ICI.R, 2025.
[49] Shen Zhang, Zhaowei Chen, Zhenyu Zhao, Yuhao Chen, Yao Tang, and Jiajun Liang. Hidiffu- sion: Unlocking higher-resolution creativity and efficiency in pretrained diffusion models. In ECCV, pages 145-161. Springer, 2024.
[50] Min Zhao, Guande He, Yixiao Chen, Hongzhou Zhu, Chongxuan Li, and Jun Zhu. Riflex: A free lunch for length extrapolation in video diffusion transformers. In /CML, 2025.
[51] Chuanyang Zheng, Yihang Gao, Han Shi, Minbin Huang, Jingyao Li, Jing Xiong, Xiaozhe Ren, Michael Ng, Xin Jiang, Zhenguo Li, et al. Dape: Data-adaptive positional encoding for length extrapolation. In NeurlPS, 2024.
13

---

A Proof of Theorem 3.1
Assumptions. In Theorem 3.1 demonstrate that Causal Attention introduces position wosition-dependent variance in attention outputs, allowing the Transformer to encode positional information implicitly.
To facilitate the subsequent derivations, we introduce the following assumptions:
Stochastic Initialization Assumption: We assume that the attention scores S OK
vd, are inde- pendently and identically distributed (i.i..). Analogously, the value V is assumed to be i.idd with E[V] My and Var(V) ov-
Mutual Independence: We assume that the attention scores {Sig} and the value V are mutually independent.
Proof. Consider a sequenee of length n. For S ii S n, the causal attention matrix Asy is defined by
exp( (ij))
i2j.
Aij
=1= exp (Si
(13)
<.j.
Zij
Let Zi exp(Sij) and Wi
Given the assumption on S, the elements {Sig} are
Zj-m1 Ziz'
assumed be ii.d., where
Sy
Qim jm-
(14)
v dd
Then the attention output at position i in dimension is Ys Ej Wiy Vgr. The variance (Y₁))
Var(Yu)= = Var
WoyV = Σ Var(W.,Kss),
(15)
j=1
since Wiy and Vj are independent for each i, Furthermore, Var(W,, Vga) = E[W3] u Var(Wi)). Hence,
Var(Ya) Σ (ETW3) pe Var(W:;)).
(16)
Because Zij are i.i.d. and positive, have Ej Wij Σj= Zi
1, hence ElW 1/i.
EE= Ziz
can be approximated by Dirichlet(1,... .., 1) distribution. This approximation is conceptually aligned with the analytic framework proposed by Hobbhahn [19], which establishes mapping from a distribution over logits to Dirichlet distribution the corresponding softmax outputs. Reasonably, whenever the exponentials (exp(S;))) do not differ too sharply and remain roughly exchangeable, this leads to the uniform-symi symmetric Dirichlet scenario. In practice, it provides convenient closed-fom E[W3]]
Besides, the normalized vector(Wn1, Wa. ..... Ws))
Z,1 25 Z,y' -y 2,y'
Zu
2
Then Var(W E[W33 (E(W,))2
2
$. Substituting into the sum, one obtains
Var(Yn) Σ
2
of + #p
[
(1(i+1) 22)
2
-
i 1+1 + (i+1)) p².
(17)
i(i +1
As increases, we can approximate
((+1) + leading to the reasonable approximation
Var(Yn)
(18)
i+1 1
14

---

where the constant C' - 20 + u.y
B Justification of Mutual Independence Assumption
We observe that the correlation between the attention matrix and the value vectors is low during the early stages of denoising. This empirical observation supports the validity of independence assumption. As discussed in the main paper, our analysis primarily focuses on the early stage, where we show that causal attention mainly encodes positional information.
The input sequence
[z1..... can be approximated as i.dd Gaussian noise in the early denoising stage. Each 2; is an independent Gaussian vector. The queries and keys are computed as: q: =,,W, kg x, Wg Since the 2; are i..d., the sets qi are also i.id. after linear transformation, the same ky. The attention score is: Siy 9f k,. Because the K3 are ii.d., the set Sty for 1,... are identically distributed random variables. After applying softmax A = Softmax (S), the attention matrix Ay approaches uniform distribution, and E[A,,] = 1/n. From this perspective, the attention matrix and the value have low correlation: changes in the value vectors do not significantly affect the attention matrix, resulting in low correlation. We conduct toy experiment with a sequence length of 256 and a head hidden dimension of 72, consistent with the DiT-XL/2 configuration. We randomly initialized W,, WE. W... generated random Gaussian noise œ; and computed the correlation coefficient between the attention scores S and the values V over 1000 trials. The average correlation coefficient was 0.03, indicating very low correlation. This provides theoretical support for our assumption.
In practice, we also observe several layers show low correlation between the attention matrix and the value vectors during the early denoising stages of trained diffusion transformer. As shown in Figure 10 we report the correlation coefficients between the attention matrix and the value vectors across different timesteps and layers. The correlation remains low in the early stages of denoising. While the correlation gradually increases later stages stages-whet where the independence assumption no longer holds-we have demonstrated in the main paper that causal attention primarily encodes positional information during the early denoising steps. Therefore, this does not affect the validity of justification.
0.35
Laver 0
Laver
0.30
Laver
Corriation
0.25
0.20
0.35
0.30
0.85
0.60
999
Timestup
Figure 10: Correlation across timestep.
C Variance Distribution Across Timesteps
As shown in Figure II we present the variance distribution of causal attention outputs across different timesteps and layers. During the early stages of denoising, the variance distribution mainly follows our proposed theorem. In the late denoising stage, especially T 100, the variance distribution deviates from the theorem and becomes irregular. We attribute this to the higher independence among values V in the early stages, which aligns with the theorem's assumptions. In the later stages, the increasing correlation between tokens (position and semantic relationships) violates the assumptions. This raises a question: What is the potential impact of variance deviation in later denoising steps on image quality? To further investigate this, we conduct additional experiments. Specifically, we introduce switching threshold 7' in the denoising process. Given that the denoising timestep T decreases from 1000 to O, design the attention mechanism follows: when T > I", we use
15

---

causal attention; when T < T", we switch to self-attention. In this setup, a smaller T" corresponds to switching later the denoising process.
We choose different values T to train an LEDiT-XL/2 on 256x 256 ImageNet and use it to generate 512x512 images. As shown in Figure [2] we find clear trend: the later the switch, namely the smaller the 7", the better the generated image quality. Notably, when T" = 100, the generated images are comparable to those of LEDiT with full causal attention (T" 0). This suggests that the positional information is primarily acquired in the early phase, and the causal attention with variance deviation in the later steps (T 100) has minor effects on the resolution extrapolation ability.
T-991,1-2 1--27
T-955, L-12
T-871, L-13
T-835, L-7
1200
1000
100
120
100
600
80
400
25
69
200
20
40-
20
20
50
100
150
200
250
50
100
150
200
250
100
200
150
250
Position
Position
Position
Position
T-742, I-11
T-646, L--4
T-477, L-15
T-417, L-6
1.2
100
Var
69
20
69
-9
10
20
20
50
100
150
200
250
50
100
150
200
259
100 150
200
250
150
200
250
Position
Position
Position
Position
T-353, 1.-9
T-309, 1-11
T-88, L-13
T-0, 1-20
140
70
225
100
69
60
200
175
40
30
50
100
150
200
250
50
100
150
200
250
0
50
100
150
200
250
100
150
200
250
Position
Position
Position
Position
Figure 11: Variance distribution across different timesteps and layers.
7' 400
7' 200
T' - 100
T 0
Figure 12: Switching threshold ablations. The resolution is 512x512. We set CFG=4.0. Variance deviation in the later steps (T' 100) has minor effects on the resolution extrapolation ability.
D Variance Distribution of Variant
The variance distribution under the "Mask Lower-right Corner" order is consistent with our main findings. As this is 2D scan variant, the variance is expected to decrease progressively along the
16

---

Table 5: Positional index regression.
Position Regression
2D Position Regression
Method
Training Loss
Test Error
Training Loss
Test Error
DIT-NoPE
5091.24
5265.85
12.57
13.30
DiT-Cau. Atten.
97.42
112.08
1.20
1.36
demonstrates that the 2D causal scan variants still exhibit the existence of causal attention in DiT that satisfies the conditions outlined our theorem.
height or width axis. Figure 13
(a) Timestep: 943, In-
(b) Timestep: 750,
Layer: 20.
Layer: 15. Out-of-
distribution (256x256)
distribution (512x512)
settings.
settings.
Figure 13: Variance distribution of variant
E
Positional Index Regression
We further conduct positional index regression experiment verify that causal attention encode positional information. Specifically, we trained an MLP to predict the position index of each token using the outputs of causal attention from well-trained DiT input. We conducted experiments both ID and 2D position regression tasks on ImageNet- 256 to validate the effectiveness.
For ID causal attention, the MLP predicts the ID position index of each token (e.g.. 1, 2, 256). For 2D causal attention, the MLP predicts the 2D position index (e.g., 1,1, 1,2...., 16,16) for each token. The MLP is trained using L2 loss.
We performed these experiments using DiT with ID causal attention (variant (a) in the main paper) and DiT with 2D causal attention (variant (d)), and compared the results with DiT-NoPE, which cannot encode positional information. If the causal attention variants outperform DiT-NoPE, demonstrates that the outputs of causal attention contain implicit positional information.
During inference, we generated images using both DiT with causal attention and DiT-NoPE. At each of the 250 denoising steps, the MLP predicts the positional index from the features output by causal attention. We generated 100 images, resulting in 2,500 tests in total. We report the L2 loss between the predicted and ground truth position indices. Since the MLP is trained to predict positional indices, cannot generalize to unseen positional indices when extrapolating to higher resolutions. Therefore, we perform inference with the MLP at the training resolution. Nevertheless, the significant performance gap compared NoPE provides strong evidence that causal attention can implicitly encode positional information. As shown in the table below, causal attention demonstrates significant advantage over NoPE in both training loss and test error. This indicates that the position regressor can effectively learn positional information from the outputs of causal attention, providing further evidence that causal attention can implicitly encode positional information.
Architecture Refinement
observe that minor architectural modifications to LEDiT during training from scratch can signifi- cantly improve extrapolation performance, as detailed in Table 6 (i) Adding layer normalization
after convolution stabilizes training and enhances extrapolation. (ii) Using only single causal layer
achieves strong extrapolation, slightly outperforming the 14-layer setting. (ii) The multi-dilatio
17

---

strategy reduces sFID but leads to slight increase in FID. This finding contrasts with the fine-tuning scenario Table Ic We hypothesize that, during training from scratch, the dilation perturbations may hinder convergence, whereas in the fine-tuning setting, where the model is already well-trained, dilation has less impact on convergence. promising future direction is to adopt progressive multi-dilation strategy, which we leave for future work.
Table 6: Ablation study architecture refinement. The inference resoluton is 512x512. The models are trained on x 256 ImageNet, and we report results with 10K samples.
Causal Layers
Conv Post Norm
Multi-dilation
FID
sFID
ISt
Prec.t
Rec.1
59.32
78.35
62.13
0.42
0.36
14
40.45
72.98
104.24
0.56
0.39
39.84
80.41
104.26
0.55
0.38
36.79
76.98
112.82
0.56
0.39
G In-distribution Comparison
In the main paper, we compared the results of various methods at resolutions higher than the training resolution. this section, we compare the performance LEDiT-XL/2 and LEMMDIT at the training resolution. Tables and 8 shows the result of LEDiT-XL/2
ImageNet and LEMMDiT COCO. The FID of LEDiT T and LEMMDiT increases slightly at 256 x 256. Although both LEDiT (with causal attention) and DiT (with standard self-attention) have approximately the same number of parameters, their computational complexities differ substantially. For an input sequence of length T. and hidden dimension d, the self-attention mechanism computes attention scores for all possible pairs, resulting in per-layer computational complexity of (I.2d). In contrast, causal attention restricts each position attend only previous positions (including itself), leading reduced number of attention computations. Specifically, the total number of attention weights is reduced from L to L(L 1)/2, and the corresponding computational complexity becomes O(L2d/2). This O(L?d) halves the theoretical compute cost compared to self-attention i.e., O(L*d/2) = 2 While this reduction improves efficiency theoretically, it may also limit the model's ability capture
dependencies, which can explain the slight performance gap between LEDiT and DĪT. Nevertheless, illustrated in Figure 14 LEDiT still produces high-fidelity samples, demonstrating that causal attention still achieves competitive generative quality despite its lower computational complexity.
Table 7: Comparison of performance on 256 x56 resolution. The models are trained on 256 x266 ImageNet. We set CFG= We report results with 50K samples.
Model
Resolution
FID
sFID
IST
Prec.
Rec.
DiT-Sin/Cos PE DiT-RoPE
2.27
4.60
278.24
0.83
0.57
2.33
4.58
272.02
0.83
0.58
LirrLeeaeable PE DiT-LH-180
256x256
2.38
4.69
275.05
0.82
0.58
2.54
4.94
248.47
0.82
0.57
LEDiT
2.38
4.58
268.66
0.83
0.58
18

---

Figure 14:
256x 256 samples generated from our LEDiT-XL/2 trained on ImageNet 256x256 resolution with CFG 4.0.
Table Comparison of performance on 256x256 resolution. The models trained on 256
COCO. We set CFG=2. We report results with 40,504 samples.
Model
Resolution
FID
sFID
1St
Prec.
Rec.1
MMDiT
6.32
11.77
30.01
0.65
0.49
MMDiT-ROPE
256x256
5.39
11.68
32.32
0.67
0.50
LEMMDiT
6.35
11.61
31.54
0.65
0.48
H 512x512 ImageNet
We fine-tune new LEDiT-XL/2 model on 512x512 ImageNet for 100K iterations using the same hyperparameters the 256x model. The qualitative comparison among vanilla DIT, VisionNTK, Vision YaRN, and L.DiiT is shown in Figure As resolution increases from 512x512 512 to 1024 1024, vanilla DiT exhibits further quality degradation, with significant noise artifacts. HTTv2-Visionyy generates images with duplicated objects, while FiTv2-Vision YaRN produces blurry images with severe
detail loss. DiT-VisionNTK, VisionYaRN, RIFLEx generate higher-quality images but exhibit object duplication local structures. In contrast, LEDiT maintains more realistic structures and finer details. The quantitative results are reported in Table Due to the heavy quadratic computational burden, we generate 10K images for evaluation. LEDiT consistently achieves superior metric scores across all resolution settings. For instance, at resolution of 768 x 768, LEDiT improves the previous best FID of 28.94 (achieved by DiT- NiKionNTK) to 21.75.
Table Comparison of of-te-of-hhe-are extrapolation methods and our LEDiT trained on 512x512 ImageNet at various resolutions beyond the training image size. We set CFG=
768 768
$96 899
1024 x 1024
Model
HDD
+F113]
IST
Pice.f
Rec.t
F11.,
+F113]
1S7
Pice.
H1D
[+113]
Pice."
Rec.1
DIT-SHn PE
159152
187.92
7.76
01.12
0.24
220.93
217.70
3:27
1103
0.08
251.57
240.17
2.16
1101
0.03
DVT-ViseoNTK
DIT ViT-VisionaaR
29.46
61.37
181.32
0.56
0.53
65 58
91.48
83.21
0.50
0.51
104.62
118.05
43.04
0.47
DIT RIFLEX
31.30
80.71
154.01
0.56
0.49
64.84
115.61
62.90
0.43
0.47
114.84
152.9
23.87
0.27
0.36
FTV2 VisceNTX
251.73
195.83
3:44
0.12
309.13
230.84
2.54
0.01
0.01
369.76
2.43
0.11
VTTVV-Vision VaRN
215.72
FDNT
21.75
49.51
176.26
0.71
0.52
48.64
73.25
97.54
0.56
0.50
91.11
108.7 70
48.13
0..00
0.44
I Comparison with Learnable Positional Embeddings
I Learnable positional embeddings have been widely adopted in the original VIT 9 and Swin Trans- former (27]. We replace the Sin/Cos PE in DiT with learnable positional embeddings to conduct comparison with our method. For length extrapolation, interpolate the leamable positional
19

---

embeddings to higher resolutions to ensure compatibility, We report the in-distribution performance in Table and the oft-t--distributtinn performance in Table At the training resolution. Learnable PE and LEDiT exhibit nearly comparable performance. When extrapolating to 512x512, we observe a significant drop for Learnable PE. This likely because the interpolated positional embeddings at new spatial locations are not seen during training, leading to degradation.
Table 10: Comparison with learnable positional embeddings on 512x resolution. The models are trained on 256x256 ImageNet, and we report results with 10K samples.
Model
FID
sFID
1St
Prec.
Rec.1
DiT-LLerraabe PE
208.45
139.38
5.23
0.02
0.02
LEDiT
35.86
67.97
139.91
0.52
0.51
J Comparison with LookHere
LookHere-1800
LEDiT
LookHere- 180
LEDIT
Class Label 140
Class Label 217
Figure 15: Comparison with LookHere at 512 resolution. The models are trained on 256x256 ImageNet. We set CFG=4.0. iindicates training from scratch. Best viewed when zoomed in.
Both LEDiT and LookHere 12 explore causal attention to enhance length extrapolation. LookHere carefully designs various combinations of causal masks to provide directional inductive biases. intro- duces AliBi (33 to penalize attention scores and demonstrates extrapolation improvement. LookHere conduct extensive experiments to demonstrate the extrapolation capabilities in classification.
However, we clarify that there are key differences. We provide a rigorous theoretical framework that explains why causal attention is capable of encoding positional information and enabling length extrapolation. This not only facilitates effective image generation with robust extrapolation capabilities, but also offers valuable insights into the underlying mechanisms of length extrapolation. In contrast, LookHere does not investigate the reasons behind causal attention's 's ability to encode positional information; merely states that attention with 2D masks can "limit the distribution shift that attention heads face when extrapolating".
We directly adapt LookHere to image generation tasks and find does not yield effective length extrapolation. We select LookHere- -itterre--0--th 180-the best-performing variant-as the representative method and compare its performance with LEDiT. We (i) train the randomly initialized LookHere/LEDiT for 400K steps or (ii) fine-tune LookHere/LEDD EDDTT for 100K steps. Quantitative results are presented in Table [T] In the fine-tuning scenario. LEDiT outperforms LookHere. In the training-from from-scratch scenario, LookHere achieves a lower sFID, while LEDiT achieves lower FID. Specifically, sFID leverages intermediate spatial features from the Inception network, capturing fine-grained image details, whereas FID is computed using the spatially-pooler poolled layer, reflecting more global structures. We interpret that the lower sFID LookHere indicates fine-grained image details, while the lower FID of LEDiT suggests more coherent object structures. As shown in Figure [5 when extrapolated
20

---

to 512x512 resolution, samples generated by LookHere preserve fine image details but suffer from severe object duplication, whereas those generated by LEDiT exhibit more coherent structures, indicating that LEDiT shows better extrapolation performance. We believe a promising direction to integrate Here and LEDiT, aiming to generate images with both high-quality details and coherent object structures.
Table 11: Comparison with LookHere on 512x512 resolution. The models are trained on 256 ImageNet. We set CFG=1.5. We report results with 10K samples. indicates training from scratch.
Model
FID
sFID
1St
Prec.
Rec.
DiT-LH-180
66.93
83.09
82.39
0.39
0.36
LEDiT
35.86
67.97
139.91
0.52
0.51
DiT-LH-180
41.97
71.42
111.44
0.56
0.33
LEDiT
36.79
76.98
112.82
0.56
0.39
K 16x Length Extrapolation
We evaluate LEDiT trained on ImageNet-256x22 256 and extrapolated to 1024 1024 resolution (a 16x length extrapolation). As shown in Table 12 LEDiT outperforms other methods. However, the high FID suggests that aggressive resolution extrapolation remains challenging and warrants further exploration.
Table 12: Comparison with other method on 1024x 1024 resolution. The models are trained on 256 x56x256 ImageNet, and we report results with 10K samples.
Model
FID
sFID4
1St
Prec.t
Rec.1
DiT-Sin/Cos PE
281.57
240.17
2.16
0.01
0.02
DiT-Learnable PE
284.07
230.15
2.41
0.08
0.01
DiT-VisionNTK
333.01
244.23
1.96
0.22
0,00
DiT-vn YaRN
228.41
199.62
7.60
0.03
0.09
DiT-RIFLEx
335.30
214.46
4.86
0.01
0.12
F1 VisionNTK
342.54
260.28
2.75
0.01
0.00
VTv2-VisionYaRN
338.12
241.12
2.93
0.01
0.00
LEDiT
212.97
169.88
10.14
0.05
0.14
L FID over Fine-tuning Steps
We plot FID over fine-tuning steps from 25K to 200K at both training resolution (256x256) and beyond (512x512), shown in Figure T6 At 256x256 resolution, FID generally decreases with more fine-tuning steps. A1512x512. performance plateaus around 50K steps, then gradually increases with further fine-tuning. We chose 100K steps to balance performance in-distribution and out-of- distribution.
M
Ablation Study on Attention Scaling
Following the widely adopted practice length extrapolation, we also apply attention scal- ing (22] Figure 17 shows the ablation of attention scaling. Without attention scaling, LEDIT can still generate reasonable images. Attention scaling primarily improves image quality and mitigates local structural issues (c.g., the dog's mouth).
21

---

8.5
8.0
50
2 7.0
45
6.5
40
5.5
35
25K
50K
75K
100K
150K
200K
25K
50K
75K
100K
150K
200K
Taining Steps
Training Steps
Training Resolution (256x256)
Beyend Training Resolution (512x512)
Figure 16: FID-10K over fine-tuning steps of LEDiT at the training resolution (256x256) and beyond the training resolution (512x512).
w/o attention scale
w/ attention scale
Figure 17: Attention scaling ablation. The resolution is 512x512 generated by LEDiT-XL/2 trained on 2566 We set CFG=4.0.
N Ablation Study on CFG
Following prior work, FiT, we set CFG=1.5, Additionally, we test model performance across various CFGs, where CFG=1 indicates no classifier-free guidance. As shown in Table 13 LEDiT outperforms VisionNTK, Vision YaRN and RIFLEx across different CFGs, demonstrating LEDiT's robust extrapolation capability.
Table 13: Ablation study on CFGs. The models are trained on 256x256 ImageNet. The inference resolution is 512x512 and report results with 10K samples.
Model
FTD1
sFIDL
1S1
Prec.
Rec.7
FTD]
s[ID]
IS1
Prec.1
Rec."
RID
.FID]
IS"
Prec.
Rec.1
VisionNTK
125.73
125.19
32.78
0.19
0.48
53.75
76.62
107.75
0.41
0.53
27.12
$4.04
211.85
0.61
0.46
D/T RIFLEx
302.35
197.81
256.1
172.54
10.58
0.06
214.30
0.05
0.21
FTTV2- Viv--ViiiNNTK
214.17
265.40
6.58
205.64
150.89
9:22
0.03
0.3
FTTV--Vison YaRN
246.06
180.58
8.43
0.04
0.29
163.27
139.45
20.42
0.12
0.35
97.49
95.19
46.50
0.22
0.52
More Training Steps
In this section, we use LEDiT without architecture refinement. We further extended the training from scratch to 600K and 1000K steps, as presented Table [4] LEDiT consistently outperforms VisionNTK, Vision YaRN and RIFLEx, and we do not observe FID saturation.
22

---

Table 14: Comparison of state-of- tfae-of-the--a ahe extrapolation methods when training from scratch on 256 256 ImageNet. We extend the training steps to 600K and 1000K. The inference resolution is 512x We report results with 10K samples.
Model
Training Steps
FID4
sFIDL
ISt
Prec.f
Rec.
DiT-Sin/Cos
600
244,22
193.22
2.74
0.19
0.04
DrT-VisioNTTK
600
183.16
146.21
16.18
0.09
0.33
DIT: VisionYaRN
600
118.44
123.36
36.68
0.20
0.43
DIT-RIFLEx
132.27
129.26
25.94
0.15
0.40
LEDIT
56.34
72.14
74.10
0.44
0.38
VisionNTK
1000
190.65
151.52
14.19
0.09
DIT- DrT-VisionYan YaRN
1000
131.03
127.05
32.13
0.17
0.44
DIT-RIFLEX
1000
160.42
117.45
21.37
0.11
LEDIT
1000
54.29
74.53
79.35
0.45
0.42
P Performance on Smaller Models
We conduct experiments on smaller models, namely DīT-B and DīT-S, under the same training settings LEDiT-XL/2. As shown Table 15 LEDiT consistently outperforms state-of-the-ar extrapolation methods across all evaluation metries in DiT-B, and shows substantial improvements in key metrics in DiT-S, with comparable FID to DiT-RoPE-NTK. It delivers stable performance gains across DiT-S, DřT-B, and DiT-XL, demonstrating robustness and scalability. These results highlight LEDiT's strong length extrapolation capabilities and generalizability to different model scales.
Table 15: Comparison of performance on 512x512 resolution using DiT-B and DiT-S. The models trained 256 256 ImageNet for 400K steps, and we report results with 10K samples.
Model
FIDJ
sFID
Prec.
Rec:f
DiT-S-Sin/Cos
253.20
186.79
2.36
0.02
0.01
DĪT-S- VisToNiii
121.42
197.59
11.61
0.12
0.23
DIT---VisionYaRR
161.41
129.67
13.40
0.11
0.20
DIT-S-RIFLEX
313.10
192.36
6.34
0.02
0.20
LEDIT-S
124.71
97.76
18.57
0.16
0.22
DiT-B-Sin/Cos PE
214.05
188.24
3,27
0.02
0.07
164.51
193.89
6.98
0.09
0.24
DiT-B-Visions YaRN
126.52
132.84
21.37
0.18
0.31
DiT-B-RIFLEx
433.49
225.40
4.00
0.01
0.09
LEDIT-B
86.63
84.93
35.21
0.29
0.30
More Ablation Visualization
We present all visualizations of the ablation study in Figure 18
R More Qualitative Visualization
We present more comparison results with DiT-Sin/Cos PE, DiT-VisionNTKK DiT-VisionYaRN, DiT- REFLEx, @iTv2-VisionNY and FiTv2- Visii-isionnon YaRi demonstrate the effectiveness of LEDiT, as shown in Figure 19 Figure 20 and Figure 21 LEDiT outperforms other methods in both fidelity and
local details.
Additional Samples
We present more samples generated by LEDiT-XL/2 in Figure 22 and Figure 23
23

---

T Limitations and Future Work
Constrained by limited resources, train LEDiT only on the ImageNet and COCO datasets. We did not test how LEDiT will perform on a larger-scale dataset such as LAION-5B 38 The generative capabilities of LEDiT when training with higher resolutions have not been explored. Subsequent research can focus on how to integrate LEDíT into modern powerful diffusion models or LLMs to achieve more amazing outcomes. Moreover, other learnable modules capable of capturing positional information, similar to causal attention, should be explored to further enhance the performance of length extrapolation.
U Broader Impacts
Our diffusion-basee approach can advance generative modeling, enabling applications in image synthesis, data augmentation, and scientific discovery, which may benefit research and industry. At the same time, method could be misused for generating misleading harmful content, such as deepfakes or synthetic data for malicious purposes. We discuss these risks and suggest possible mitigation strategies.
NcPE
Causal
Coay
Causal Conv
(4)
(h)
(4)
(a) Component ablations.
(b) Casual scan variants.
Multi-Dita
CAXL/2-SAXL/2
SAxLLE-CAAUU
(CA, SAy-12
ISA, CAMA22
(c) Multi-dilation strategy.
(d) Block design.
Dilation 12,
Multi-dilation probability.
Dilation rate.
Figure 18: Ablation visualization of Table 1
We set CFG=4.0, Best viewed when zoomed in.
24

---

Cos
DIT-VisloNNK DrE-VisOOYERR
RITVZ-ViSRENTK FIT2--HHOn'aRN
Chess Lakel 2786
Class Labe) 42
448 X06-48
Class Label 209
512x35
Class Label 130
760x7765
Class S9
89659996
Class 200
102 1024
Figure 19: More qualitative comparison with other methods. The resolution and class label located to the left of the image. We use the model trained on 256x256 ImageNet to generate images with resolutions less than equal to 512x512, and the model trained on 512x512 ImageNet to generate images with resolutions greater than 512x512 and less than or equal to 1024x 1024. We set CFG==.0. Best viewed when zoomed in.
DissSin/Cos PE
DiiiVisioNNTK
DiE-VisinnaRNN
DIT RIFLLS
Filv2 VisioNNIK
FID2 Visioniii
LEDIT
Closs Label 344
512x0164
Class Label Label133 335
384x513
Figure 20: More qualitative comparison with other methods on generating non-square images. The resolution and class label located the left of the image. We use the model trained on 256x256 ImageNet to generate images at 512x384 and 384 x 512 resolutions. We set CFG=4.0. Best viewed when zoomed in.
25

---

MMDIT
VisionNTK
VisionYaRN
RIFLEX
LEMMDIT
A pizza covered in veggies on white plate sittin.
A pupple and orange bus is traveling down the road.
There is wood bench at the edge forest path.
A track station.
Figure 21: Qualitative comparison with other methods on text- text-to-image task. The prompts are located the left the image. We the model trained on x56 COCO to generate images 512x512. We set CFG=6.0. Best viewed when zoomed in.
Figure 22: More arbitrary-reoolution
resolution samples (512², 512x384, 384x512, 512x256, 256x 384
2562, x28x.56). Generated from our LEDiT-XL/2 trained on ImageNet 256x256 resolution. We set CFG = 4.0.
26

---

Figure 23; More arbitrayy-reoolution samples (10243, 1024x 768 1024, 1024x512, 512x 1024,
7682, 5122, 256x512). Generated from our LEDiT-XL/2 trained on ImageNet 512x512 resolution. We set CFG = 4.0.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contributions are accurately reflected in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
28

---

Justification:
See Section A for the proof of Theorem 3.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The experiment setup information is introduced in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
Justification:
We release the source code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
htapii//guiides//eeSubbmm wtw.eeeuuuusssiioooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctpppp.ccppulli//uuiiie wwwiiccgguie///eeeeeee CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experiment setup information is introduced in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
30

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The hardware resource is introduced in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics haaps:://euripss/ httpp//ccccppp...tttttccccceeeee
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
31

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.ree.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
32

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
33

---

Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htteas/s//eerrippoccc///o/rreerp for what should or should not be described.
34


Reviews:
[{"review_id": "NVcIz17ima", "reviewer": "Reviewer_xXsT", "summary": "LEDiT introduces a novel No Positional Embedding (NoPE) Diffusion Transformer (DiT) for image generation, enabling models trained on low-resolution data to generate high-resolution images during inference. It overcomes the limitations of explicit positional encodings (PEs) by:\n\n1. Implicit Positional Encoding: Leveraging an observed per-token variance property, LEDiT replaces self-attention with causal attention during specific timesteps. This implicitly encodes positional information, removing the need for explicit PEs.\n\n2. Enhanced Locality: A single dilated convolution layer is added after patchification to capture fine-grained local features, ensuring a balance of global and localized information.\n\nThis combination allows LEDiT to achieve impressive length extrapolation and arbitrary resolution generation, a critical advancement for high-resolution image synthesis.  The authors show significant improvements over NOPE in baselines when applying a combination of causal attention + self-attention with a single dilated convolutional layer. \n\t\nBy combining these strategies, LEDiT enables DiTs trained on low-resolution data to generate high-resolution images during inference without loss of quality, making them more adaptable to diverse output requirements.", "questions": "1. The authors have also shown results on text to image tasks, however, it would be good to see how does the T2I alignment metric perform with the proposed architecture. It would be good to understand if the architecture would compromise semantic coherence for dense prompts ? \n\n2. Could you elaborate on potential trade-offs of replacing self attention layers with causal attention's unidirectional flow.\n. Is there any insight on why the recall drops ?", "limitations": "The paper is well motivated and well written. While the combination of causal attention with convolution has been applied in litterature, the strength of the paper lies in demonstrating that these building blocks could also be applied in context of length extrapolation without positional embeddings. The analytical insights into per-token variance are particularly helpful in strengthening the claim of novelty. However, I have a few concerns I have raised for which I give a borderline accept.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\n1. The paper offers valuable analytical and empirical insights into per-token variance, highlighting the temporal impact of positional information during diffusion. The causal attention's significant computational savings over full attention is a notable practical advantage.\n\n2.  The combination of causal attention and a convolutional layer (\"Enhanced Locality\") demonstrably boosts performance over baseline DiTs without explicit PEs, effectively tackling length extrapolation.\n\n3. LEDiT shows compelling improvements in image fidelity (FID, IS) when generating images at significantly higher resolutions than trained, confirming its primary claim of length extrapolability and arbitrary aspect ratio support.\n\nWeakness: \n\n\n1. LEDiT is a continuous diffusion model that uses a specific type of masked attention (causal attention) to achieve length extrapolation by implicitly encoding positional information. Most recent multi-modal generation models follow bi-directional attention increases contextual representation.\n\n2. Training on smaller datasets (ImageNet, FFHQ, LSUN) rather than large-scale, diverse sets like LAION-5B raises concerns about LEDiT's generalizability and real-world applicability at scale.  \n\n3. The unidirectional nature of causal attention, compared to bidirectional attention in leading multimodal models, might limit the model's ability to capture full contextual representations, potentially reducing generative diversity (recall). This trade-off needs further analysis and discussion.", "was_revised": true, "final_justification": "As mentioned in my review, the method is simple and elegant approach to context extrapolation without Positional Embedding. \nThe authors in the rebuttal have responded to two of my concerns\n1.  Impact of causal attention on T2I alignment: Table on clip scores shows consistant improvement against existing PE baselines. The authors have also compared against baselines with learnable positional embeddings. \n2. Increasing in the number of causal attention layers indicate improvements in the quality on default configuration\n\nDue to extensive empirical investigation I have raised my score.", "rebuttal": "We thank the reviewer for their positive feedback and are pleased that they found our paper to be well-motivated and well-written. We appreciate the recognition of our novel implicit positional encoding and the analytical insights provided, which enable robust length extrapolation and demonstrate substantial improvements over existing methods.\n\n>[W1&W3&Q2]: The trade-offs of replacing self-attention layers with causal attention's unidirectional flow needs further analysis and discussion. Whether causal attention might potentially reducing generative diversity (recall); The insight on why the recall drops.\n\nThank you for your insightful comments. **We have found that, by appropriately balancing causal attention and self-attention, our method achieves in-distribution performance comparable to RoPE, while significantly outperforming existing methods on extrapolation tasks.**\nTo provide a more comprehensive analysis, we conducted controlled ablation studies by varying the number of causal attention layers in LEDiT. Our results indicate that **increasing the number of causal layers enhances length generalization, with only a minor reduction in in-distribution performance. Compared to previous methods, our approach achieves a more favorable trade-off between these aspects.**\nTable 1 compares different LEDiT variants and other positional encoding methods under both in-distribution and out-of-distribution settings. Notably, with just a single causal layer, LEDiT significantly outperforms all previous methods at higher resolutions. At the training resolution, LEDiT matches the widely adopted RoPE in performance, **demonstrating that LEDiT preserves strong in-distribution results while providing superior extrapolation capability**. Increasing the number of causal layers further enhances extrapolation, with only a slight reduction in in-distribution performance. While the optimal trade-off may vary by application, our results indicate that the default configuration described in our paper (i.e., 14 causal layers) achieves an effective balance between in-distribution and out-of-distribution performance.\n\n| Method         | 512x512 FID↓   |  sFID↓   |  IS↑     |  Precision↑ |  Recall↑ | 256x256 FID↓ | sFID↓ | IS↑ | Precision↑ | Recall↑ |\n|------------------------|--------|---------|------------|---------|----------|---------|----------------|-------------|-------------|-------------|\n| Learnable PE       |   208.45    |   139.38     |    5.23      |   0.02     |   0.02  | 2.38 |     4.69    |  275.05      | 0.82|0.58|\n| Sin/Cos PE         |   215.69    |  180.75      |    8.09       |    0.05    |   0.15  | 2.27 |       4.60    |    278.24    |0.83|0.57\n| VisionNTK (RoPE-based)          |   251.85   |   153.40   |    8.73       |   0.04    | 0.26    | 2.33 |    4.58       |  272.02      |0.83|0.58\n| VisionYaRN (RoPE-based)          |   53.75   |   76.62    |    107.75      |   0.41    |   0.53  |  -  |    -        |       -  |- |- \n| RIFLEx (RoPE-based)             |   256.1    |   172.54    |    10.54      |   0.04    |  0.18   |  -  |    -        |       -  |- |-\n| **LEDiT (1 Causal layer)**   |   **37.70**    |   **65.20**    |    **137.57**       |   **0.52**    |  **0.52**   | **2.33** |   **4.53**        |  **273.84**      |**0.83**|**0.57**|\n| LEDiT (2 Causal layers)      |  38.60    |    68.92    |   134.46        |     0.50   |   0.51  | 2.35 |     4.61      |  268.69      | 0.83 | 0.57 |\n| LEDiT (4 Causal layers)      |   41.91   |   69.58     |    127.40       |  0.49      | 0.53    | 2.39 |   4.55        |   270.02     |0.83 | 0.57|\n| LEDiT (8 Causal layers)      |  39.87    |    69.18    |    133.82       |   0.51     |  0.51   | 2.34 |   4.57        |   267.69     | 0.83| 0.58|\n| **LEDiT (14 Causal layers)**     |  **35.86**    |   **67.97**     |    **139.91**       |  **0.52**      | **0.51**    | **2.38** |   **4.58**        |   **268.66**     | **0.83** | **0.58**|\n| **LEDiT (28 Causal layers)**       |  **32.61**    |   **61.06**     |    **162.58**       |   **0.57**     |  **0.51**   | **2.40** |   **4.65**        |   **264.81**     | **0.83**| **0.57**|\n\nTable 1: Out-of-Distribution (512x512) and In-distribution (256x256) comparison between LEDiT variants and other methods. we report FID-10K for 512x512 and report FID-50K for 256x256\n\n> Whether causal attention might potentially reducing generative diversity (recall); The insight on why the recall drops.\n\n**Causal attention is not the cause of recall reduction.** In fact, we emphasize that LEDiT even achieves slightly better recall than DiT in the in-distribution setting (see Table 1), indicating that causal attention does not compromise sample diversity.\n\nIn the length extrapolation scenario, both Table 2 of the FiT paper [1] and Table 1 of our work consistently reveal two distinct method categories:\n\n- **Category I:** Methods with lower FID (e.g., LEDiT, DiT-VisionYaRN) tend to exhibit higher precision but lower recall.\n- **Category II:** Methods with higher FID (e.g., DiT-NTK) tend to exhibit lower precision but higher recall.\n\nFor clarity, we reports the performance of various PE and LEDiT when extrapolated to 384×384 resolution, all trained on ImageNet-256×256.\n\n| 384x384  | FID↓ | Precision↑ | Recall↑ |\n|------------------------|--------|---------|------------|\n| VisionNTK (RoPE-based)          |  71.23   |   0.33  |    0.51    |  \n| VisionYaRN (RoPE-based)          |  13.51  |    0.71 |     0.39     |   \n| RIFLEx (RoPE-based)             |    57.88  |   0.37   |   0.55   | \n| LEDiT  |    9.34   |    0.78  |      0.39    |\n\nCategory I methods generate high-quality samples with limited diversity, whereas Category II methods frequently produce a substantial proportion of broken images, as shown in the visualizations in our appendix. **We argue that Category I methods are preferable, consistent with the statement from the precision/recall paper[2] (page 5): *\"It is unclear which application might favor setup D (Category II), where practically all images are broken, over setup B (Category I), that produces high-quality samples at a lower variation.\"***\n\n**The reduction in recall can be attributed to out-of-distribution (OOD) generalization challenges**. When models are trained on low-resolution data, the learned data distribution  $P_l$ does not fully capture the high-resolution manifold $P_h$. Consequently, when extrapolating to high-resolution, models tend to generate samples concentrated in the intersection region $P_e = P_l \\cap P_h$. Samples from $P_e$ simultaneously satisfy both the higher-resolution requirements and the learned data distribution. However, since $P_e$ is a subset of $P_l$, this restriction reduces diversity, leading to lower recall.\n\n[1] FiT: Flexible Vision Transformer for Diffusion Model, ICML 2024.\n\n[2] Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019.\n\n>[W2]: Training LEDiT on large-scale, diverse sets like LAION-5B to test generalizability.\n\nDue to time constraints and limited computational resources, \nwe currently face challenges in applying LEDiT to large-scale sets like LAION-5B.  Following previous works DiT, SiT, REPA, we conduct extensive experiments on class-conditional and text-to-image generation tasks on ImageNet and COCO dataets. We also believe that LEDiT can improve training on large-scale datasets, which contain images with diverse resolutions and aspect ratios. By removing the constraints of explicit positional encoding, LEDiT can better accommodate images of varying resolutions, which may lead to improved training outcomes.\nIn future work, we plan to progressively extend LEDiT to larer-scale data to evaluate its generalization and scalability in more complex generative settings.\n\n>[Q1]: How does the T2I alignment metric perform with the proposed architecture. It would be good to understand if the architecture would compromise semantic coherence for dense prompts\n\n**LEDiT maintains semantic coherence for dense prompts.** We use 40,504 captions from the COCO dataset to generate images and compute the CLIP score as the text-to-image alignment metric. The comparison is presented in the following table. LEDiT consistently outperforms other methods in terms of semantic coherence.\n\n| 512x512  | CLIP Score↑ | \n|------------------------|--------|\n| MMMDiT        |  22.94  | \n| MMDiT-ViNTK         |  24.91  |  \n| MMDiT-ViYaRN          |    23.37  |  \n| MMDiT-RIFLEx          |    27.14  | \n| **LEMMDiT**  |    **27.82**   |"}, {"review_id": "G2HpFikujR", "reviewer": "Reviewer_JebD", "summary": "The paper demonstrates that causal attention with no positional encoding can efficiently encode global position information and proposes LEDiT architecture to efficiently extrapolate to resolutions higher than training resolution in diffusion models.", "questions": "The authors should include a baseline comparison with learnable position embeddings and analyze the trend in Figure 4 and Appendix B for the default “Mask Lower-right Corner” raster scan. Further, the authors should clarify if NoPE can effectively encode position information at higher resolutions. Finally, justification for the claim that attention scores are independent of inputs in (Lines 453–454) should be provided. I am willing to increase my score if authors can clarify these concerns.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper demonstrates theoretically and experimentally that causal attention with no position encoding can encode global information by utilizing variance in outputs at different positions in the sequence.\n2. The proposed approach leads to notable improvements over baselines. Further, authors provide a comprehensive ablation study of various design choices. \n3. The paper is overall well-written and easy to follow.\n\n**Weaknesses**\n\n1. Missing baseline: The authors should also compare the proposed approach with learnable position embeddings. \n2. Potential issue in scaling: The variance appears to plateau after approximately 150 steps in Figure 4 (left). This raises concerns that at higher resolutions, for example 1024×1024 with 4096 token generation, most tokens may exhibit insufficient variance to encode positional information effectively.\n3. While the assumption that inputs to causal self-attention are Gaussian is reasonable in Theorem 3.1, the assumption that attention scores are mutually independent of the inputs (Lines 453–454) is unclear and lacks justification.\n4. Missing experiment analyzing trend in Fig 4 and Appendix B for “Mask Lower-right Corner” raster scan which is the default order in the paper.", "was_revised": true, "final_justification": "All of my concerns including additional learnable positing embedding baseline and assumption in proofs have been resolved. Hence, I have raised my score.", "rebuttal": "We thank the reviewer for their positive feedback and are pleased that they found our paper well-written and easy to follow. We appreciate the recognition of theoretical and experimental demonstration. We are also encouraged that the reviewer acknowledges the notable improvements.\n\n>[W1]: Missing baseline: learnable position embeddings.\n\nSimilar to ViT and Swin Transformer, we replace the positional encoding (PE) in DiT with learnable positional embeddings. For length extrapolation, we interpolate the learnable positional embeddings to higher resolutions to ensure compatibility. As shown in the table below, at the training resolution, Learnable PE and LEDiT exhibit nearly comparable performance. When extrapolating to 512×512, **we observe a significant drop for Learnable PE**. This is likely because the interpolated positional embeddings at new spatial locations are not seen during training, leading to degradation.\n\n| Method         | 512x512 FID↓   |  sFID↓   |  IS↑     |  Precision↑ |  Recall↑ | 256x256 FID↓ | sFID↓ | IS↑ | Precision↑ | Recall↑ |\n|------------------------|--------|---------|------------|---------|----------|---------|----------------|-------------|-------------|-------------|\n| Learnable PE       |   208.45    |   139.38     |    5.23      |   0.02     |   0.02  | 2.38 |     4.69    |  **275.05**      | 0.82|0.58|\n| **LEDiT**         |  **35.86**    |   **67.97**     |    **139.91**       |  **0.52**      | **0.51**    | 2.38 |   **4.58**        |   268.66     | **0.83** | 0.58|\n\nTable 1: We report FID-10K for 512x512 and FID-50K for 256x256\n\n>[W2]: Scaling to higher resolutions such as 1024x1024.\n\nWe respectfully clarify that a comparison at 1024×1024 resolution is presented in Table 7 of the appendix, where our approach demonstrates a clear performance advantage. Please refer to the appendix for detailed results.\n\nAs discussed in Section 3.3 of the main paper, we have observed that the variance between local tokens becomes indistinguishable at higher resolutions. To address this, we introduce Locality Enhancement, which improves LEDiT's ability to capture local positional information.\n\nFurthermore, we evaluate LEDiT trained on ImageNet-256×256 and extrapolated to 1024×1024 resolution (a 16× length extrapolation). As shown in the table below, LEDiT outperforms other methods. However, the high FID suggests that aggressive resolution extrapolation remains challenging and warrants further exploration.\n\n| 1024x1024  | FID↓ | sFID↓ | IS↑ | Precision↑ | Recall↑ |\n|------------------------|--------|---------|------------|---------|------------|\n| DiT-LearnablePE          |   284.07   | 230.15       |  2.41       |  0.08   | 0.01   | \n| DiT-Sin/Cos PE          |  281.57    |   240.17     |     2.16    |  0.01   |   0.02 |   \n| DiT-VisionNTK          |  333.01    |  244.23      |     1.96    |   0.22  |  0.00  |  \n| DiT-VisionYaRN          |   228.41   |   199.62     |   7.60      |  0.03   |  0.09  |  \n| DiT-RIFLEx          |  335.30    |    214.46    |    4.86     |    0.01 |  0.12  |  \n| FiTv2-VisionNTK          |   342.54   |     260.28   |     2.75    |    0.01 | 0.00   |  \n| FiTv2-VisionYaRN          |  338.12    |    241.12    |     2.93    |   0.01  |   0.00 |  \n| **LEDiT**         | **212.97**   |  **169.88**    |     **10.14**    |   **0.05**    |    **0.14**    |   \n\n\n>[W3]: Justification of the assumption that attention scores and v are mutually independent.\n\nThank you for your insightful comments. **We observe that the correlation between the attention matrix and the value vectors is low during the early stages of denoising**. This empirical observation supports the validity of independence assumption. As discussed in the main paper (lines 128–143), our analysis primarily focuses on the early stage, where we show that causal attention mainly encodes positional information.\n\n**Theoretical Justification:**  The input sequence $\\mathbf{x} = [x_1,...,x_n]$ can be approximated as i.i.d. Gaussian noise in the early denoising stage, as acknowledged by the reviewer. Each {$x_i$} is an independent Gaussian vector. The queries and keys are computed as:\n$q_i = x_i W_q$, $k_j = x_j W_k$\nSince the {$x_i$} are i.i.d., the sets {$q_i$} are also i.i.d. after linear transformation, the same as {$k_j$}. The attention score is: $S_{ij} = q_i^T k_j$.\nBecause the {$k_j$} are i.i.d., the set {$S_{ij}$} for $j=1,...n$ are identically distributed random variables. After applying softmax $A=Softmax(S)$, the attention matrix $A_{ij}$  approaches a uniform distribution, and $\\mathbb{E}[A_{ij}] = 1/n$. From this perspective, the attention matrix and the value have low correlation: changes in the value vectors do not significantly affect the attention matrix, resulting in low correlation.\nWe conduct a toy experiment with a sequence length of 256 and a head hidden dimension of 72, consistent with the DiT-XL/2 configuration. We randomly initialized $W_q$, $W_k$, $W_v$, generated random Gaussian noise $x$, and computed the correlation coefficient between the attention scores $S$ and the values $V$ over 1000 trials. The average correlation coefficient was 0.03, indicating very low correlation. This provides theoretical support for our assumption.\n\n**Empirical Justification:** In practice, we also observe several layers show low correlation between the attention matrix and the value vectors during the early denoising stages of a trained diffusion transformer. Specifically, we report the correlation coefficients between the attention matrix and the value vectors across different timesteps and layers (see Table below). As shown, the correlation remains low in the early stages of denoising. While the correlation gradually increases in later stages—where the independence assumption no longer holds—we have demonstrated in the main paper that causal attention primarily encodes positional information during the early denoising steps. Therefore, this does not affect the validity of our justification.\n\n| Timestep\\Layer | 0 | 14 | 22 |\n| --- | --- | --- | --- |\n| 999 | 0.010 | 0.065 | 0.066 |\n| 887 | 0.036 | 0.081 | 0.012 |\n| 774 | 0.033 | 0.048 | 0.116 |\n| 710 | 0.077 | 0.077 | 0.129 |\n| 642 | 0.064 | 0.120 | 0.121 |\n| 582 | 0.101 | 0.143 | 0.223 |\n| 526 | 0.119 | 0.176 | 0.207 |\n| 449 | 0.156 | 0.199 | 0.192 |\n| 401 | 0.141 | 0.169 | 0.145 |\n| 337 | 0.133 | 0.137 | 0.163 |\n| 265 | 0.160 | 0.084 | 0.170 |\n| 249 | 0.174 | 0.122 | 0.161 |\n| 189 | 0.213 | 0.191 | 0.199 |\n| 124 | 0.293 | 0.273 | 0.234 |\n| 0 | 0.371 | 0.336 | 0.275 |\n\n>[W3]: Missing experiment analyzing trend in Fig 4 and Appendix B for “Mask Lower-right Corner” variant.\n\nThank you for your valuable suggestions. **The variance distribution under the \"Mask Lower-right Corner\" order is consistent with our main findings.** As this is a 2D scan variant, **the variance is expected to decrease progressively along the height or width axis**. Due to rebuttal space constraints, we provide a single table each for in-distribution and out-of-distribution results. Additional visualizations will be included in the revised version. These results demonstrate that the 2D causal scan variants still exhibit the existence of causal attention in DiT that satisfies the conditions outlined in our theorem. \n\nAdditionally, **we conducted a position regression experiment to further demonstrate that causal attention can implicitly encode positional information**. For more details, please refer to our response to [W3] of Reviewer x8vf if interested.\n\n\nHeight/Width | 0 | 2 | 4 | 6 | 8 | 10 | 12 | 14 | 15 | Mean Variance (Height)\n|---|---|---|---|---|---|---|---|---|---|---|\n0 | 62.568 | 61.406 | 61.808 | 61.052 | 61.107 | 58.746 | 55.701 | 52.502 | 51.281 | **59.128** |\n2 | 62.025 | 61.291 | 61.751 | 61.715 | 60.529 | 58.479 | 55.815 | 52.701 | 50.657 | **58.878** |\n4 | 62.141 | 61.784 | 60.561 | 59.886 | 59.104 | 57.109 | 54.493 | 52.129 | 50.380 | **58.047** |\n6 | 62.046 | 61.590 | 60.715 | 58.857 | 58.439 | 56.513 | 54.888 | 51.386 | 50.176 | **57.554** |\n8 | 60.435 | 59.905 | 59.168 | 58.561 | 57.429 | 55.103 | 53.934 | 52.292 | 50.549 | **56.853** |\n10 | 59.075 | 57.629 | 57.258 | 56.201 | 55.485 | 54.238 | 52.989 | 51.017 | 50.692 | **55.301** |\n12 | 55.547 | 55.541 | 54.981 | 54.173 | 54.771 | 53.725 | 51.427 | 51.155 | 50.773 | **53.746** |\n14 | 53.556 | 52.497 | 52.061 | 51.996 | 53.142 | 51.323 | 51.104 | 51.472 | 50.741 | **52.047** |\n15 | 50.885 | 50.879 | 50.614 | 50.155 | 50.687 | 50.706 | 50.610 | 51.098 | 50.415 | **50.715** |\nMean Variance (Width) | **59.215** | **58.649** | **58.246** | **57.213** | **57.041** | **55.336** | **53.573** | **51.788** | **50.670** | - |\n\nTable 2: Timestep: 943, Layer: 20. In-distribution (256x256) settings.\n\n\nHeight/Width | 0 | 4 | 8 | 12 | 16 | 20 | 24 | 28 | 31 | Mean Variance (Height)\n|---|---|---|---|---|---|---|---|---|---|---|\n0 | 89.495 | 79.405 | 74.726 | 71.715 | 67.887 | 62.126 | 55.560 | 45.446 | 28.546 | **65.118** |\n4 | 82.514 | 78.698 | 74.238 | 69.688 | 62.903 | 55.681 | 48.119 | 40.209 | 27.356 | **61.450** |\n8 | 78.711 | 74.260 | 68.082 | 63.037 | 57.270 | 50.776 | 43.420 | 35.626 | 28.701 | **56.517** |\n12 | 74.934 | 69.286 | 63.372 | 57.743 | 51.852 | 45.344 | 38.704 | 31.527 | 26.989 | **51.836** |\n16 | 72.041 | 64.826 | 57.983 | 52.671 | 46.417 | 41.822 | 35.249 | 29.439 | 26.042 | **47.867** |\n20 | 66.171 | 58.735 | 52.689 | 46.901 | 40.970 | 36.559 | 31.926 | 27.396 | 25.330 | **43.194** |\n24 | 57.264 | 52.083 | 45.432 | 39.545 | 35.789 | 31.558 | 28.112 | 26.279 | 27.037 | **37.930** |\n28 | 47.008 | 40.636 | 35.092 | 31.552 | 28.650 | 26.924 | 26.219 | 25.084 | 26.212 | **31.585** |\n31 | 27.496 | 26.517 | 26.821 | 26.373 | 25.777 | 26.950 | 25.257 | 25.255 | 24.567 | **26.298** |\nMean Variance (Width) | **67.905** | **62.102** | **56.820** | **51.894** | **46.846** | **41.982** | **37.188** | **31.573** | **26.563** | - |\n\nTable 3: Timestep: 750, Layer: 15. Out-of-distribution (512x512) settings."}, {"review_id": "eGEYPv26wE", "reviewer": "Reviewer_x8vf", "summary": "This paper introduces LEDiT (Length-Extrapolatable Diffusion Transformer), a novel architecture that overcomes the resolution extrapolation limitations of traditional Diffusion Transformers (DiTs). Standard DiTs rely on explicit positional encodings (PEs) like RoPE, which degrade performance when generating images beyond training resolutions due to positional index distribution shifts. LEDiT replaces explicit PEs with causal attention, which implicitly encodes global positional information by leveraging variance differences in attention outputs. The theorem proves that causal attention’s output variance is inversely proportional to position, enabling implicit ordering. A locality enhancement module (multi-dilation convolution) is added to capture fine-grained details, complementing causal attention’s coarse-grained global information.\nExperiments on ImageNet and COCO show LEDiT supports up to 4× resolution scaling (e.g., 256×256 to 512×512) with superior image quality. It outperforms state-of-the-art methods in FID, IS, and other metrics, even for arbitrary aspect ratios (e.g., 512×384) without multi-aspect-ratio training. Fine-tuning from pretrained DiTs for 100K steps suffices for strong extrapolation, highlighting efficiency.", "questions": "Please see weakness.\n\nThis work is very interesting as it can support length-extrapolation without additional training on these higher-resolution image data. However, LEDiT has not achieved SOT on either 256 or 512 resolutions. The practical significance of this work is not clear. I am glad to raise my score if the authors satisfactorily solve my concerns.", "limitations": "The practical significance of this work is not clear.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 1, "strengths_and_weaknesses": "# Strengths\n1. Stronger Extrapolation Performance: Achieves 4× resolution scaling with better structural fidelity and details than RoPE-based methods (e.g., FiT, RIFLEx). Generates arbitrary aspect ratios without specialized training, demonstrating generalizability.\n2. Theoretical Rigor: Causal attention implicitly encodes positional info, validated by theoretical proof of variance-position relationship. Theorem 3.1 mathematically links causal attention output variance to positional ordering, providing a foundation for implicit PE.\n\n# Weakness\n1. The contribution and significance of this paper are rather vague, because LEDiT has not achieved SOT on both 256 and 512 resolutions, especially at 512, where there are already many powerful models. The practical significance of this work is not clear.\n2. Why Conv can provide positional information? Does kernel size and layer number affect the positional information?\n3. To prove that causal attention can implicitly encode positional information, the author should provide the positional index regression experiments following LLM. The variance distribution in Figure 4 is not enough. \n4. I observe slightly inferior performance on ImageNet-256x256. Causal attention’s reduced computational complexity may slightly degrade in-distribution performance (e.g., 256×256 FID slightly higher than DiT with RoPE). Does causal attention affect the semantic information captured in LEDiT? \n5. Multi-dilation has been widely used in previous vision transformer literature[1,2], limiting the novelty of this work.\n\n[1] Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet.\n[2] Vision Transformer with Progressive Sampling.", "was_revised": true, "final_justification": "This paper introduces an interesting approach to improve the extrapolation. It achieves SOTA extrapolation performance compared with baselines. \n\nDuring rebuttal, the authors provided more experiments to reveal why this method can work. I think the ablations on convolution kernel size and positional index regression could further improve the quality of this work. \nHowever, the model does not achieve SOTA performance on either ImageNet-256 or ImageNet-512 benchmark, limiting the significance and practical applications of this work. Therefore, I will raise my score from 2 to 3 only. \nI suggest the authors incorporate these experiments into the revised version.", "rebuttal": "We thank the reviewer for the thoughtful feedback and are pleased that the reviewer appreciates the theoretical rigor in implicit positional encoding, and superior performance over state-of-the-art methods.\n\nHowever, we respectfully clarify that the concern raised is based on a misunderstanding of our paper. We have addressed and clarified all relevant points in the following response.\n\n>[W1]: The contribution and significance of this paper are rather vague. LEDiT has not achieved SOTA performance.\n\nLet us first clarify the significance of our work: **Our main goal is to improve the generative performance of DiT at resolutions higher than their training resolutions**, addressing a challenging and impactful out-of-distribution scenario relevant to many real-world applications, such as high-resolution medical imaging and high-definition film production. Current powerful DiT models, such as SD3, FLUX still struggle to generate images beyond the training resolution.\n\nTo address this limitation. We introduce **a novel Diffusion Transformer without positional encoding (LEDiT)** for higher-resolution image generation, departing from the commonly used RoPE-based methods. **We provide both theoretical and empirical evidence that causal attention—the core component of LEDiT—implicitly encodes positional information, an that facilitates length extrapolation**.\n\nWe conduct extensive experiments on both class-conditional and text-to-image generation tasks, demonstrating that LEDiT achieves superior extrapolation performance and outperforms state-of-the-art methods. For example, when extrapolating to 512×512 resolution, LEDiT reduces the previous best FID from 49.86 to 33.25. At 1024×1024 resolution, LEDiT surpasses VisionYaRN (the previous best method) by 17 FID points. **We believe our method achieves state-of-the-art extrapolation performance**.\n\nIt is worth mentioning that the significance of our findings has also been acknowledged by other reviewers, who recognized that our work provide insightful theoritial analysis (Reviewer xXsT, JebD\n) and shows compelling improvements in image fidelity (Reviewer xXsT\n, JebD, f3XA). It was also highlighted by reviewer 3ixD, who considered it quite important to the research community.\n\n\n>[W2]: Why Conv can provide positional information? Does kernel size and layer number affect the positional information?\n\nPrevious works [1][2] have demonstrated that convolution with zero padding can leak local positional information, as cited in our main paper. The kernel size does not affect the leakage of positional information. As shown in the table below, the extrapolation performance to 512x512 resolution remains nearly identical across different kernel sizes.\n\n| 512x512  | FID↓ | sFID↓ | IS↑ | Precision↑ | Recall↑ |\n|------------------------|--------|---------|------------|---------|------------|\n| Kernel=(3,3)          |   35.86   |    67.97  |  139.91     |   0.52  |    0.51  | \n| Kernel=(5,5)          | 36.46  |     64.56   |     140.70   |    0.52    |    0.51  |   \n\nWhen it comes to layer number, we conducted experiments by incorporating convolutional layers at different depths of LEDiT and found that adding convolutions to the early layers yields better performance than adding them to the deeper layers. We hypothesize that leveraging local positional information extracted by convolutions at earlier stages is more beneficial for diffusion transformer to leakage positional information, thereby leading to improved overall performance.\n\n| 512x512  | FID↓ | sFID↓ | IS↑ | Precision↑ | Recall↑ |\n|------------------------|--------|---------|------------|---------|------------|\n| Layer=1         |   35.86   |    67.97  |  139.91     |   0.52  |    0.51  | \n| Layer=10          | 49.27  |    79.52   |    93.47   |    0.43   |    0.46  |   \n\n[1] How much position information do convolutional neural networks encode? ICLR 2020.\n\n[2] Segformer: Simple and efficient design for semantic segmentation with transformers. NeurIPS 2021.\n\n\n>[W3]: Provide the positional index regression experiments to prove that causal attention can implicitly encode positional information\n\nThank you for your valuable comments. **The positional index regression experiment further verifies that causal attention can encode positional information**. Specifically, we trained an MLP to predict the position index of each token using the outputs of causal attention from a well-trained DiT as input. We conducted experiments on both 1D and 2D position regression tasks on ImageNet-256x256 to validate the effectiveness.\n\nFor 1D causal attention, the MLP predicts the 1D position index of each token (e.g., {1}, {2}, ... {256}). For 2D causal attention, the MLP predicts the 2D position index (e.g., {1,1}, {1,2},...,{16,16}) for each token. The MLP is trained using L2 loss.\n\nWe performed these experiments using DiT with 1D causal attention (variant (a) in the main paper) and DiT with 2D causal attention (variant (d)), and compared the results with DiT-NoPE, which cannot encode positional information. If the causal attention variants outperform DiT-NoPE, it demonstrates that the outputs of causal attention contain implicit positional information.\n\nDuring inference, we generated images using both DiT with causal attention and DiT-NoPE. At each of the 250 denoising steps, the MLP predicts the positional index from the features output by causal attention. We generated 100 images, resulting in 2,500 tests in total. We report the L2 loss between the predicted and ground truth position indices. Since the MLP is trained to predict positional indices, it cannot generalize to unseen positional indices when extrapolating to higher resolutions. Therefore, we perform inference with the MLP at the training resolution. Nevertheless, the significant performance gap compared to NoPE provides strong evidence that causal attention can implicitly encode positional information. As shown in the table below, **causal attention demonstrates a significant advantage over NoPE in both training loss and test error. This indicates that the position regressor can effectively learn positional information from the outputs of causal attention, providing further evidence that causal attention can implicitly encode positional information**.    \n\n|    | Training Loss | Test Error | \n|------------------------|--------|---------|\n| 1D position regression          |     |      |  \n| DiT-NoPE         | 5091.24  |     5265.85   |\n| **DiT with causal attention**         |  **97.42** |  **112.08**    |\n| 2D position regression         |     |      |  \n| DiT-NoPE         | 12.57    |   13.30   |\n| **DiT with causal attention**        |  **1.20** |   **1.36**   |\n\n\n>[W4]: Multi-dilation has been widely used in previous vision transformer literature[1,2], limiting the novelty of this work.\n\n**We believe T2T-ViT [1] and PS-ViT [2] do not mention or utilize dilation-related concepts or techniques**. The multi-dilation training strategy is only one of the contributions presented in our paper. Our main contribution is improving the generative performance of DiT at resolutions higher than those seen during training. We introduce a novel Diffusion Transformer without positional encoding (LEDiT) and provide rigorous theoretical analysis as well as extensive experimental results to demonstrate the extrapolation capability of LEDiT. We believe our contributions are significant in the domain of length extrapolation."}, {"review_id": "0w5A4G246T", "reviewer": "Reviewer_f3XA", "summary": "This paper introduces LEDiT, which improves the ability of diffusion transformers (DiTs) to generate larger images at test time. The method modifies DiT in two ways:\n\n1. A causal/directional mask in self-attention, which prevents attention to below _and_ to the right of the query token. This mask encodes position information since every token has a different set of visible tokens for attention, thus distinguishing the 2D positions of tokens along the flattened sequence dimension. \n2. A (second) convolution after the initial tokenization/convolution operation, while _not_ adding position embeddings to tokens. \n\nThe paper demonstrates that LEDiT generates higher quality images when extrapolating to larger images than baseline position encoding methods invented for both ViTs and DiTs.", "questions": "None", "limitations": "Some limitations are briefly discussed.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- (I believe) this paper benchmarks many position encoding methods for image size extrapolation for DiTs. The failure cases, e.g. sin-cos embeddings, provide useful information. I don't believe previous DiT papers extrapolate from 224 to 1024 px.\n- The empirical results are strong. LEDiT beats VisionYaRN (the 2nd best method) by 17 FID at 1024 px.\n- The ablations are nice. They show that NoPE alone fails, causal self-attention _or_ the extra convolution performs poorly, but causal self-attention _with_ the extra convolution extrapolates well.\n- The exact causal 2D attention mask is novel (see the discussion on this novelty below)\n- The multi-dilation strategy during training is novel to ViTs/DiTs and seems to help extrapolation a bit\n\nMinor weaknesses:\n- LEDiT is slightly worse than prior methods in-distribution (i.e., when training image size equals testing image size) (Table 5 in the appendix)\n\nMajor weaknesses:\n- LookHere (NeurIPS 2024: https://openreview.net/forum?id=o7DOGbZeyP) is not cited nor benchmarked. I will compare LookHere to LEDiT so that the authors can respond and we can discuss. Both papers introduce position encoding methods for vision transformers with the primary aim of improving image-size extrapolation. Both papers leverage 2D causal attention masks to limit the receptive field of each token, which is found to facilitate image-size extrapolation. LookHere introduced directional attention masks for ViTs, which _are_ 2D causal attention masks. The LookHere paper presents many mask variants, such as \"only attend above\", which LEDiT visualizes in Fig 6A. LEDiT proposes mask variants that did not appear in LookHere, for example LEDiT's _chosen_ variant called \"mask lower right\" (Fig 6D) is the complement of LookHere's \"attend lower right\" mask. The LEDiT submission presents 2D causal attention masks as the key contribution, thus the omission of LookHere in baselines and discussions is critical.\n\nKey differences: LookHere combines 2D attention masks with ALiBi-style distance encoding. LEDiT combines 2D attention masks with a convolutional stem. In both papers, these additional components are critical — which is interesting! LEDiT uses the same attention mask for all heads, LookHere masks heads differently. LEDiT introduces a multi-dilation strategy, which seems to help. LookHere was demonstrated on image recognition tasks — and LookHere is directly useable in DiTs (just like the other _included_ baselines, RoPE, sin-cos, Yarn, etc.). LEDiT is demonstrated on image generation tasks.", "was_revised": true, "final_justification": "This paper introduces a novel position encoding method that improves image-size extrapolation for DiTs. It outperforms SOTA position encoding methods (most of which were invented for ViTs).\n\nDuring the rebuttal period, the authors demonstrated that LEDiT outperforms LookHere (NeurIPS 2024) on image generation fine-tuning. In light of LookHere, the LEDiT method is not as novel as the paper claims as LEDiT is _not_ the first to introduce causal attention masks for ViTs for extrapolation. That being said, the method as a whole is novel and the empirical results are important, thus I recommend the paper is accepted.", "rebuttal": "We thank the reviewer for their positive feedback and are pleased that they appreciate our comprehensive comparison of positional encoding methods for extrapolation in DiTs, the strong empirical results, and the thorough ablation studies, the design of LEDiT framework.\n\n>[W1]: The omission of LookHere in baselines and discussions is critical.\n\nThank you for your valuable comments and constructive feedback. We agree there are some conceptual similarities between LEDiT and LookHere.  Both LEDiT and LookHere explore causal attention to enhance length extrapolation. LookHere carefully designs various combinations of causal masks to provide directional inductive biases. It introduces AliBi to penalize attention scores and demonstrates extrapolation improvement. LookHere conduct extensive experiments to demonstrate the extrapolation capabilities in classification tasks. \n\nHowever, **we clarify that there are key differences. Notably, directly adapting LookHere to image generation tasks does not yield effective length extrapolation**. We have provided a comprehensive comparison between LEDiT and LookHere in both methodology and performance.\n\n**The key differences between LEDiT and LookHere are summarized in the following table**:\n| Method      | Theoretical Contribution | Complementary Strategy to Causal Attention                     | Receptive Field   | Is Suitable for Extrapolation in Image Generation    |\n|-----------|----------|-------------------------------|------------|-------------|\n| LookHere     | No       | ALiBi | Restricted local  receptive field  | No    |\n| LEDiT  | Yes    | Multi-dilation Convolution                  | Global  receptive field      | Yes       |\n\n- **Theoretical Contribution:** We provide a rigorous theoretical framework that explains why causal attention is capable of encoding positional information and enabling length extrapolation. This not only facilitates effective image generation with robust extrapolation capabilities, but also offers valuable insights into the underlying mechanisms of length extrapolation. In contrast, LookHere does not investigate the reasons behind causal attention's ability to encode positional information; it merely states that attention with 2D masks can \"limit the distribution shift that attention heads face when extrapolating\" (Page 2).\n\n- **Complementary Strategy to Causal Attention & Receptive Field**: LEDiT not modify the causal attention mechanism itself. Instead, we introduce multi-dilation convolution as a complementary module to causal attention, **ensuring that the global receptive field of causal attention remains unaffected**. LookHere incorporates AliBi into causal attention. AliBi can be interpreted as a form of soft window attention, where an attention score penalty increases with distance, inherently restricts the receptive field of causal attention—even when LookHere explores different combinations of causal masks to cover the entire image.\n\n- **Is Suitable for Extrapolation in Image Generation**: \nwe conduct detailed experiments and observe that LookHere fails to achieve effective length extrapolation in image generation tasks— whereas LEDiT demonstrates robust and reliable extrapolation capabilities.\n\n\n**Detailed Experimental Comparison**:\n\nWe evaluate all LookHere variants (LH-180, LH-90, LH-45) and compare their performance with LEDiT. We fine-tune all models for 100K steps on ImageNet-256x256 by loading the DiT-XL/2 pretrained weights. Since AliBi is a key component of LookHere, we also report results for AliBi to facilitate a comprehensive discussion. As shown in the table below, **LEDiT outperforms both LookHere and AliBi at the training resolution as well as at higher resolutions**. We observe that, when extrapolated to higher resolutions, samples generated by LookHere preserve fine image details but suffer from severe object duplication. Since we are unable to provide visualization of generated images, we focus on quantitative comparisons using FID and precision metrics, which primarily reflect the structural accuracy and quality of the generated images. LookHere achieves lower FID and precision scores compared to LEDiT.\n\n| Method         | 512x512 FID↓   |  sFID↓   |  IS↑     |  Precision↑ |  Recall↑ | 256x256 FID↓ | sFID↓ | IS↑ | Precision↑ | Recall↑ |\n|------------------------|--------|---------|------------|---------|----------|---------|----------------|-------------|-------------|-------------|\n| AliBi            |   241.38  | 123.21     |  6.26   |   0.05  | 0.16   |  109.70  |  35.14          |  14.28      | 0.21 | 0.68|\n| LH-45            |  75.66   |   81.28   |   70.13  |  0.33   |  0.37  | 2.77  |   5.08         |   236.65     | 0.81|0.57\n| LH-90            |  80.84   |   84.13   |  67.74  | 0.34   | 0.37   |  2.72 |    4.93        |   242.92     | 0.82| 0.56\n| LH-180            |   66.93  |   83.09   |   82.39  |   0.39  | 0.36   | 2.54  |   4.94         |  248.47      |0.82 | 0.57\n| **LEDiT**     |  **35.86**    |   **67.97**     |    **139.91**       |  **0.52**      | **0.51**    | **2.38** |   **4.58**        |   **268.66**     | **0.83** | **0.58**|\n\nTable 1: Out-of-Distribution (512x512) and In-distribution (256x256) comparison between LEDiT and other methods. we report FID-10K for 512x512 and report FID-50K for 256x256\n\nThis raises an interesting question: **why LookHere demonstrated on image recognition tasks but fails on image generation tasks?**  We present some insights:\n\n- **Consistent Higher Training Loss for LookHere:**  We report the mean loss over the last 1,000 training steps, as shown in the table below. On the training set, LookHere shows a higher loss compared to LEDiT and Sin/Cos PE, indicating that LookHere is relatively underfitting. Increasing the training steps to 150K does not further reduce the training loss, suggesting that the model has already converged. **The persistently high training loss suggests that the LookHere architecture may not be directly applicable to the image generation task**.\n\n| Method         | Training Loss   |\n|------------------------|--------|\n| **Sin/Cos PE (100K)**            |   **0.14005**  |\n| AliBi (100K)            | 0.14557    |\n| LH-45  (100K)          |  0.14301   |\n| LH-45  (150K)          |  0.14296   |\n| LH-90  (100K)         |   0.14283  |\n| LH-90 (150K)           | 0.14275    |\n| LH-180 (100K)           | 0.14134    |\n| LH-180 (150K)           | 0.14150    |\n| **LEDiT (100K)**           |   **0.14018**  |\n\n- **Image generation requires more precise positional information than image classification.** In image classification, the input is a clear image, and the model can rely on strong image content priors to make predictions. In contrast, image generation starts from pure noise, providing no prior information. Therefore, stronger positional encoding is essential for generation tasks. \n\n- **LookHere uses causal attention and AliBi, but these mechanisms are unable to encode precise positional information.** As discussed in our main paper, causal attention cannot accurately encode positional information. We also observed that AliBi performs poorly on in-distribution settings, indicating that it does not provide accurate positional information. We believe that while the combination of causal attention and AliBi works for image classification, their ability to capture positional information is limited, which makes LookHere unsuitable for image generation.\n\n\n- **The limited local receptive field of LookHere restricts its extrapolation capability.** A global receptive field is crucial for enabling the network to capture global context, which is essential for robust length extrapolation. The causal attention in LEDiT provides access to the entire global context, resulting in strong extrapolation ability. Although LookHere uses directional attention to cover the whole image, it penalizes long-range attention scores, which impairs its ability to model long-distance dependencies. Consequently, LookHere tends to generate repeated objects and exhibits inferior performance during extrapolation.\n\nIn contrast, LEDiT combines simple causal attention with multi-dilation convolutions, achieving favorable performance in training resolution settings and demonstrating superior extrapolation ability. **Compared to LookHere, LEDiT exhibits better positional encoding capabilities in image generation tasks**. Thanks for your insightful comment. We will add the discussion and performance comparison with LookHere in the revised version.\n\n\n>[W2]: Slightly worse than prior methods in-distribution.\n\nThank you for your insightful comment. **We have found that, by appropriately balancing causal attention and self-attention, our method achieves comparable in-distribution performance to RoPE while significantly outperforming existing extrapolation methods**. We conducted controlled ablation studies by varying the number of causal attention layers in LEDiT. With only one causal layer, LEDiT significantly outperforms all previous methods at higher resolutions. At the training resolution, LEDiT achieves performance comparable to the widely used RoPE, **demonstrating strong in-distribution performance while exhibiting superior extrapolation capabilities**. Due to space limitations, we respectfully refer you to our response to Reviewer xXsT's first question for further details."}]

Author rebuttal:

Meta-review: The paper introduces LEDiT, a diffusion transformer architecture that aims to enable diffusion models to generalize to lengths beyond their training regime. LEDiT enables resolution extrapolation up to 4 times with improved image quality over state-of-the-art methods, including support for arbitrary aspect ratios without multi-aspect-ratio training. The reviewers originally raised various concerns, including the relationship with prior work that was not cited. I believe the authors have now become aware of this work and they will include it in the camera-ready version. Another concern was the scale of the experiments as raised by multiple reviewers. After the rebuttal the reviewers have been satisfied, while I do believe ImageNet is a reasonable dataset for the experiments. Therefore, this paper should be revised for the camera-ready but it presents an elegant solution.
Decision: Accept (poster)