UltraViCo: Breaking Extrapolation Limits in Video Diffusion Transformers
Authors: Min Zhao
Hongzhou Zhu
Yingze Wang
Bokai Yan
Jintao Zhang
Guande He
Ling Yang
Chongxuan Li
Jun Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: video diffusion model
video generation
diffusion transformer
video length extrapoaltion
Primary area: generative models
TL;DR: We present a training-free, plug-and-play method that pushes the practical extrapolation limit from 2× to 4×.

Abstract: Despite advances, video diffusion transformers still struggle to generalize beyond their training length, a challenge we term video length extrapolation. We identify two failure modes: model-specific periodic content repetition and a universal quality degradation.
Prior works attempt to solve repetition via positional encodings, overlooking quality degradation and achieving only limited extrapolation. In this paper, we revisit this challenge from a more fundamental view—attention maps, which directly govern how context influences outputs. We identify that both failure modes arise from a unified cause: attention dispersion, where tokens beyond the training window dilute learned attention patterns. This leads to quality degradation and repetition emerges as a special case when this dispersion becomes structured into periodic attention patterns, induced by harmonic properties of positional encodings. Building on this insight, we propose UltraViCo, a training-free, plug-and-play method that suppresses attention for tokens beyond the training window via a constant decay factor. By jointly addressing both failure modes, we outperform a broad set of baselines largely across models and extrapolation ratios, pushing the extrapolation limit from $~2\times$ to $4\times$. Remarkably, it improves Dynamic Degree and Imaging Quality by 233\% and 40.5\% over the previous best method at $4\times$ extrapolation. Furthermore, our method generalizes seamlessly to downstream tasks such as controllable video synthesis and editing.

Published as a conference paper at ICLR 2026
ULTRAVICO: BREAKING EXTRAPOLATION LIMITS IN VIDEO DIFFUSION TRANSFORMERS
Min Zhao H.ngzhou Zhu
1:2- 1.2 Yingze Wang's ' Bokai Jintao Zhang! Guande He*. Ling Yang". Chongxuan Li Jun Zhu
Dept. of Comp. Sci. & Tech., BNRist Center, THU-Bosch ML Center, Tsinghua University. ShunggShu. "Goliin School of Artificial Intelligence, Renmin University of China.
"The University of Texas at Austin. 5 Princeton University.
grace zhaol 9970 0 gmmal. com, zhuhz22@ma 1s t t tsinghua, edu. on
ABSTRACT
Despite advances, video diffusion transformers still struggle to generalize beyond their training length, challenge we term video length extrapolation. We iden- tify two failure modes: -pdel-s periodic content repetition and universal quality degradation. Prior works attempt to solve repetition via positional encod- ings, overlooking quality degradation and achieving only limited extrapolation. In this paper, we revisit this challenge from a more fundamental view attention maps, which directly govern how context influences outputs. We identify that both failure modes arise from a unified cause: attention dispersion, where to- kens beyond the training window dilute learned attention patterns. This leads quality degradation and repetition emerges as a special case when this dispersion becomes structured into periodic attention patterns, induced by harmonic prop- erties of positional encodings. Building on this insight, we propose Ultra ViCo, training-free plug-and-play method that suppresses attention for tokens beyond the training window via a constant decay factor. By jointly addressing both failure baselines largely across models and extrap-
modes, we outperform a broad olation ratios, pushing the extrapolation limit from 2x to 4x. Remarkably, it improves Dynamic Degree and Imaging Quality by 233% and 40.5% over the previous best method at 4x extrapolation. Furthermore, our method generalizes seamlessly to downstream tasks such as controllable video synthesis and editing. Project page is available at
INTRODUCTION
Building upon the expressive power of diffusion transformers (DiTs) [aoo et al. 2023 Peebles & Xie 2023). recent advances text-to-video (T2V) generation Bao et al. (2024); Zheng et al. (2024b) Brooks et al. 2024);; Wan et al. (2025): Kong et al. (222)) Hong et al. (2022) have en- abled models to synthesize high-fidelity videos. However, these models are typically trained on
fixed maximum sequence length (e.g., seconds Wan et al. (2025), Kong et al. (2024); Hong et al. (2022) and struggle to generate videos beyond their training length, task we term video length extrapolation, which is critical for practical applications.
To investigate the core challenges of this task, we conduct experiments on a range of models and identify two failure modes: (i) model-specific periodic content repetition, where short clips loop indefinitely in certain models; and (ii) a universal quality degradation, manifested as blurred spatial details and frozen temporal dynamics across all models. Both failures become increasingly severe as the extrapolation length grows. Prior work, such as RIFLEx (Zhao et al. (0025), tackles repetition from the perspective of positional encodings, while overlooking quality degradation and therefore achieving limited extrapolation. We contend, however, that positional encodings play only an indi- rect role by perturbing queries and keys to influence attention. In contrast, attention itself-directly aggregating contextual information to generate outputs- -fffesss a more fundamental view.
"Equal contribution.

---

Published as a conference paper at ICLR 2026
3 x extrapolation: Ours (top) vs. RIFLEx (bottom)
4X extrapolation: Ours (top) RIFLEx (bottom)
(a) Extending T2V models to 4x, where existing method yields nearly static, low-quality videos. Controllable video generation
Masked video-to to-o-oo-video editing
(b) Generalization to downstream tasks 3x.. See more tasks Appendix C.44
Visual results. UltraViCo achieves significant extrapolation improvement on (a) T2V models and (b) downstream tasks. See prompts and videos in supplementary materials.
Figure 1:
Therefore, we revisit extrapolation failures through the lens of attention maps. Our systematic anal- ysis of attention maps shows that both failure modes arise from unified mechanism: attention dispersion. This occurs when new tokens beyond the training length dilute the learned attention patterns. This leads to quality degradation and repetition arises special case when dispersion becomes organized into periodic attention patterns. Specifically, this happens when positional en- coding frequencies form harmonics, enabling the largest-amplitude frequency and its harmonies to accumulate amplitude and contribute substantially to the overall amplitude.
Building on this unified view, we propose Ultra-extrapollteed Video via Attention Concentration (UI- tra ViCo), plug-and-play method that suppresses attention for tokens beyond the training window with a constant decay factor. This adjustment reallocates attention reliable in-window context while naturally breaking periodic patterns, thus simultaneously addressing both failure modes. No- tably, standard attention implementations encounter oft-of-memory errors when modifying logits for long video sequences. We therefore develop memory-efficient CUDA kernel that enables scal- able applications on large video models.
To validate our approach, we conduct comprehensive evaluations various T2V models (Kong (et al. 2024 Yang et al. 2024; Wan et al. 2025) and extrapolation ratios, against large family of baselines (Chen et al. 2023b bloc97 2023 Zhuo et al. 2024 Peng al. 2023 Zhao et al. 2025). Experiments demonstrate that our method consistently surpasses all baselines in all settings by simultancously addressing both failure modes. Notably, while prior methods collapse beyond 3 extrapolation and yield static videos, ours maintains fluid motion, effectively extending the practical limit from 2x 4x. Remarkably, it improves Dynamic Degree and Imaging Quality by 233% and
2

---

Published as a conference paper at ICLR 2026
40.5% over the previous best method at 4x extrapolation. Beyond this, method also generalizes seamlessly to downstream tasks such as various controllable video synthesis and editing.
Cannyuaniides
Wan
Normal length
Video of frames
Video frames
3
extra.
(a) Periodic content repetition quality degradation.
(b) Quality degradation.
Imaging Quality
80
Hunyuan Video
Dynamic Degroe f
80
Hunyuan Video
Repetition Count
Hunyuan Video
70
Wan
60
Wan
60
Variable extra.
50
40
1x
2x
3x
4x
5x
1x
2x
3x
4x
5x
1x
2x
3x
4x
5x
(c) Both quality repetition worsen as the extrapolation grows from 1x 5x.
Figure 2: Failure modes video length extrapolation. Some models exhibit periodic content repetition, while quality degradation occurs universally. Both failure modes intensify with longer extrapolations. "extra." denotes extrapolation. See Appendix C.I for additional models.
PRELIMINARY
Attention mechanism with rotary position embedding. Modern video diffusion models largely built on DiTs whose core the attention mechanism (Vaswani et al.] 2017] [i al. (025a). The input video is patched into L. tokens, each projected into queries, keys, and values. To encode the position information, DiT's mainly adopt Rotary Position Embedding (RoPE) (Su et al.) 2024), which injects position into queries and keys through complex rotations. Concretely, for each query or key vector x E pD at position t, RoPE maps it to _ as
PRoPF (): (((
2'2
(2(()
cos( oxt) - sin( (o.l)) cos(
1))
(1)
r2i+1
Here, each frequency depends exponentially on and used encode the (2i, 2i++) components of æ. After RoPE, the queries and keys form matrices Q E RR4DD and K E RLLD Their interaction yields the attention logits E R which are normalized by the softmax function obtain the
attention scores LLL These scores are then applied the value matrix V E R to
produce output O E BLXD'
S VD"
S QKT P softmax(
P
(2)
For videos with temporal and spatial axes, Multimodal RoPE (M-RoPE) Wang et al. 2024a) parti- tions the dimension D = dr + du + dw and encodes each subspace separately. Since we focus on temporal extrapolation, we consider only the temporal axis and denote dy d for simplicity (see details in Appendix B.2)..
video length extrapolation. Despite advances, DiT-based video generation mod- els struggle to produce videos longer than their training duration. This task, known as video length extrapolation (Zhao et al. 2025). aims to adapt a pre-trained model to generate high-quality videos of sequence length L' that exceeds its training length L, with the extrapolation ratio defined as s = L'/L > 1. Notably, video length extrapolation targets the model's intrinsic ability to gen- erate longer sequences in single forward generation, which is orthogonal to prior methods (Qiul ct al. 2023 Wang et al. 2023 Kim et al. 2024 Wang et al., [024c] Lu et al. 2024) that rely on inference-time modifications. See Appendix A for more related work.
Problem setting:

---

Published as a conference paper at ICLR 2026
3 METHOD
3.1 FAILURE MODES OF VIDEO LENGTHI EXTRAPOLATION
In this section, we investigate the core challenges of video length extrapolation on range of SOTA video diffusion transformers. including Wan (Wan et al. 2025), Hunyuan Video (Kong et al. 2024). and VigeeX (Yang et al. (024).
Qualitative results in Fig 2 and 2 reveal two distinct failure modes. The first is a periodic content repetition, which occurs in certain models such as ElunyuanVider and CogVideoX. The second a universal quality degradation, characterized by compromised spatial fidelity and tem- poral dynamics across all models. To further investigate their trends across extrapolation lengths, we perform a quantitative analysis on 10 prompts using metrics including Imaging Quality (Huang al. 2024), Dynamic Degree (Huang et al. 2024), and Repetition Count. Fig. 2 confirms that both failures become more severe as the extrapolation factor increases.
These findings raise three critical questions: First, why does periodic content repetition only manifest in specific models? Second, what is the underlying cause of the universal quality degradation? Most importantly, is there a unified cause behind these two seemingly independent failure modes?
Existing work such as RIFLEx addresses only content repetition, neglecting quality degradation, which limits both model generalization and extrapolation capacity. While RIFLEx attributes repeti- tion to positional encoding periodicity, we argue that positional encodings play only an indirect role by modulating queries and keys. Instead, as Eq. ( shows, the attention map itself fundamental, since it directly determines how context is aggregated. This motivates us to revisit extrapolation failures through attention analysis.
3.2 ATTENTION ANALYSIS OF THE CAUSE
In this section, we first focus on the specific issue of periodic content repetition (Sec. 3.2. Through an in-depth attention analysis of its underlying mechanism, we find, surprisingly, that the solution designed to resolve repetition also improves video quality. This key finding then allows us to un- derstand the cause of the more universal problem of quality degradation (Sec. (.2.2). and ultimately reveals the intrinsic connection between the two failure modes.
3.2.1 THE CAUSE OF CONTENT REPETITION; PERIODIC ATTENTION PATTERNS
Periodic attention induces output repetition. We analyze the cause of content repetition by in- specting the attention map P L''L' during 4x extrapolation, where L' is the extrapolated sequence length (i.e., video features flattened into 1D sequence). The entry at row i, column j of P. denoted P.j. is the attention score from query i to key j. As shown in Fig. 33, the attention map of Hunyuan Video reveals two properties that jointly induce periodic outputs.
First, the map exhibits distinct row-wise periodicity. Specifically, for any query at position its at- tention scores to key positions j and j T are nearly identical: P., Rs P.j+T, where T corresponds to the observed repetition period in Sec. 3.1 As indicated in Fig. 31, the blue and purple circles high- light nearly equal scores. Second, the map shows relative positional invariance: query-key pairs with the same relative displacement p yield approximately equal scores, P.j P.tp.it- This RoPE-induced property appears as uniform values along diagonals and subdiagonals; for example, when p = T,, the scores marked by the blue and green circles are nearly identical.
Combining these properties, we can derive that entire query rows also repeat periodically: PAtTI PV P.j. as shown by the green and purple circles. Thus, rows and T retrieve nearly the same weighted information from the value V, leading to periodic outputs (see Appendix B.T for details):
4'1 O Σ PHT.VVS Σ P.vV= J=0
F-1
(3)
This periodicity is directly reflected in repeated content in pixel space. Larger extrapolation ratios traverse more periods, thus increasing repetition counts, which is consistent with our observations in
4

---

Published as a conference paper at ICLR 2026
Model
Attention maps
Statistical row-wise attention analysis
Individual Frequencies
Final composite attention
Pai
PIao
5.0
0,cost A,))
S(An= sid
0.0
Hun.
25
75
100
125
25
100
125
Temporal distance
Temporal distance dr
(a) Periodic atteation:
(b) Harmonic RoPE frequencies (d:/óx- 1 E
amplify the lagest-ampliuude
Pi, s PLş+T
frequency its harmonics (dushed line), inducing periodic composite attention.
Individual frequencies
Final composite attention
Pij
Pig-t
a,000t0 (eoest,,00 Ar+
Skon mi
MMM
Wan
20
Temporal distance
Temporal distance
(c) Non periodic attention:
(d) Inharmonic RoPE frequencies (db// N disperse spectrum (dashed
P. & Pas+r
line), yielding non-periodicity in the final composite attention.
Figure 3: Periodic attention patterns as cause of content repetition. Left: unlike Wan, Hun- yuan Video exhibits row-wise periodic attention during 4x extrapolation, causing repeated outputs. Right: statistical row-wise attention can be expressed as linear combination of trigonometric func- tions of RoPE frequencies, whose properties govern this periodicity. Hun. denotes Hunyuan Video.
Sec. 3.1 By contrast, the attention map of Wan (Fig. 3b) does not display such row-wise periodicity, and accordingly its outputs remain free of repetition.
Origin of periodic attention patterns. Next, show that such model-specific row-wise period- icity originates from the RoPE frequencies. To reveal the core row-wise attention structure from noise, we construct statistical row attention pattern 5(A)), which captures the relation between a query and keys at the same spatial location but At latent frames apart. This is achieved by tak- ing the expectation of the pre-softmax attention logits across all layers, heads, and query positions. As derived in Appendix B.3 (based on Eq. ) this quantity admits the following trigonometric decomposition:
4/2-1
5(At) Σ
a, cos cos( (t b.) +C,
where [0:) d//2-1 are the RoPE frequencies defined in Sec. and {ar) 1d/2-1 (b,11//211 d/2-1 C are con- stants determined by the statistics of queries and keys from models, with h; typically close to zero. Visualizations of these frequency components for Hunyuan Video and Wan highlight crucial dif- ference (Fig. 3p.d, left). The periodicity such a superposition decided by the frequency rela- tionships, as formalized in Proposition
Proposition (Period and Amplitude of Harmonics). For a function f(AI) = N as = N+
where a; db > 0 and min; Φi ON-1, i, if and only if Vi, 0z/ØN a set of harmonics), f f(At) is periodic with period Tx-1 0N-1
(i.e., they form 2n In this case, max f(At) =
EN-1 04, wwwnever At = mTTN-1. m E Z. (i.e., whenever At is at harmonic alignment positions)
We find that Hunyuan Video's 's frequencies satisfy this harmonic condition in Proposition 1 allowing amplitude accumulation of the aagest-amppitude frequency da and its harmonics (i < 3) at harmonic alignment positions mf (dashed line in Fig. 3D), where E Z. This yields dominant component that contributes 79.6% of the total amplitude, producing strongly periodic composite attention pat- tern (Fig. Bp. right). A similar harmonic alignment is also observed in CogVideox (Appendix (B6).. In contrast, Wan't 's frequencies are not harmonically aligned, resulting in a dispersed spectrum where frequency dominates (largest 31.6 31.6%), and thus clear periodicity emerges (Fig. 3d) Notably,
5

---

Published as a conference paper at ICLR 2026
while the strict periodicity of Hunyuan Video is determined by the lowest frequency, its small am- plitude and long period make negligible; the observed periodicity is effectively governed by the dominant frequency (sce Appendix [B6).
In summary, our analysis establishes the causal chain: RoPE-induced frequency harmonics lead to periodic aftention patterns, which in turn produce periodic output features and ultimately mani- fest as content repetition. To validate this, we mask tokens at harmonic alignment positions mT. Breaking these constructive interference points disrupts periodic attention and, as shown in Fig. 4a. effectively mitigates repetition.
Model
Generated videos: baseline intervention
Attention maps: baseline intervention
Hun.
(a) Non-repetition and improved video quality after intervention
(b) Attention focused centrally after intervention
Wan
Improved video quality after intervention
(d) Attention focused centrally after intervention
Figure 4: Fixing repetition reveals attention dispersion the fundamental cause. Left: our intervention, initially targeting repetition, surprisingly enhances video quality in both models. Right: the shared mechanism revealed, where the intervention refocuses diffuse baseline attention toward the central training window. This suggests attention dispersion as the unified cause.
3.2.2 THE CAUSE OF QUALITY DEGR ADATION: ATTENTION DISPERSION
Surprisingly, we find the above both models (Fig. 4h, c). This finding suggests more profound hypothesis: content repetition and quality degradation may arise from shared, fundamental underlying mechanism.
intervention also improves video quality across
comparison of attention maps shows our intervention consistently concentrates the initially dif- fuse attention (Fig. Fb. d). This occurs because masking the harmonic peaks forces softmax re- normalization, which sharpens the attention distribution by proportionally increasing the remaining scores. To further identify where this sharpened focus is most beneficial, systematically masked different attention regions and found that concentrating attention within the original central training window yielded the strongest improvements (see details in Appendix (B.)). This leads us hypoth- esize that attention dispersion is the underlying issue. New tokens during extrapolation dilute the learned attention patterns within the original training window. This dispersion has detrimental effects. Spatially, the model needs to consider far-away extrapolated frames, which makes it diffi- cult focus on fine details and results visual blurriness. Temporally, taking these distant frames into account mixes local motion with unrelated movements, causing the video to appear static and unnatural. These effects are consistent with the quality degradation observed in Sec. 3.1
To validate this hypothesis, we conduct a controlled experiment where we progressively mask at- tention scores for tokens outside the training window, thereby foreing the attention concentrate centrally. The results, presented in Fig. 5] demonstrate clear positive correlation: more concen- trated attention (i.e., by increasing the proportion of masked out-of-window scores) consistently improves both the visual quality and motion dynamics of the generated video. This provides strong evidence that attention dispersion is the cause of quality degradation. Consequently, as the extrap- olation ratio increases, attention becomes more dispersed, leading to worse quality, consistent with the observations Sec. 3.I
periodic attention analysis, we can unify both failure modes under single perspective: attention dispersion is the fundamental cause of extrapolation failure, with periodic attention patterns representing special
A unified view:
of attention dispersion. Building upon the above

---

Published as a conference paper at ICLR 2026
Imaging Quality
70
63
60
55
highly focused (100% masking ratio)
50
Focus degree (masking ratio)
Dynamic Dogreef
60
moderately focused (60% masking ratio)
40
20
0
20%
60%
100%
Focus degree (masking ratio)
weakly focused (20% masking ratio)
(a) Quantitative results.
(b) Qualitative results.
Figure 5: Validation of attention dispersion as the cause quality degradation. Both (a) quan- titative and (b) qualitative results show that video quality improves monotonically as the degree of attention central focusing (i.e., the masking ratio of out-of-window scores) increases.
case. Specifically, when RoPE frequency contributes substantially the overall amplitude (e.g., due to harmonic alignment), it induces a strongly periodic attention pattern; otherwise, the model exhibits generic, non non-periode periodiiood dispersion.
3.3 ULTRAVICO
Building the above unified view, propose Ultra-extrapolated Video via Attention Concentration (UlraaiCa a simple yet effective method that suppresses attention for tokens be- yond the training window via decay factor, thereby restoring the model's 's focusing ability. To achieve this, we introduce a position-dependere decay factor Xij applied to the original attention logits Sij. yielding the corrected attention Siy:
[1, S or Sig < 0, a, otherwise,
S = Aey Sig, where Xij
(5)
where a < 1 is a constant decay hyperparameter and L is the training length. Here, Aij is set to be for all pairs within the training window, preserving the model's core learned dynamics. For out-of- window tokens, only positive logits (Siy 2 0) are down-scaled because multiplying negative logits Siy by
can undesirably increase its value, while multiplying a 1 or 1 for negative logits has negligible effect. We also experimented with various decay strategies, such as linear decay, but found the constant form sufficient, indicating that the key distinguishing in-window from out-of-winde tokens rather than the decay shape itself (see Sec. 4.2 for details).
However, in models showing periodic repetition (Sec. 3.2.I harmonic alignment positions m'T attract disproportionately high attention. Applying uniform small decay would overly suppress all oft-owwwindw context, harming temporal consistency. To address this, we apply stronger decay specifically to these risky positions mI,, while keeping for other out-of wiindow tokens:
1, if jl < L/2 or Sij 0, 8, else if (i.j) Paak, a otherwise,
Xij
where Prisk = (i.j)) mf 7 < i-j mf +7, m m Z, N+ denotes the set of positions within 7 frames around the harmonic alignment positions mT and 3 adjustment reallocates attention to reliable in-window context while eliminating spurious periodic patterns, allowing UltraViCo to mitigate both failure modes simultaneously.
1. This targeted
Efficient CUDA implementation. UltraViCo requires modifying attention logits, but standard Py- Torch attention is infeasible for long sequences. At 3x extrapolation (~~200 tokens for Hunyuan-

---

Published as a conference paper at ICLR 2026
Table I: Quantitative illustrative results on VBench for Hunyuan Video and Wan. For Wan, which does not exhibit content repetition, we omit the NoRepeat Score. Additional results for more extrapolation ratios and models are provided Appendix [C3] Consist., Dyn., Qual., Over. and NoRe. denote Consistency, Dynamics, Quality, Overall and NoRepeat Score respectively. Normal. indicates the training length for reference.
Wan2.1- 1-1.3B
Hunyuan Video
Method
Consist.
Dyn.
Qual.
Over
User
Consist.
NoRe.
Dyn."
Qual.
Over."
User
Normal.
0.9554
51
70.34
24.25
0.9786
71
69.31
26.81
3x extrapolation
PE
0.9419
56.28
18.53
3.82
0.9795
53.17
51.85
21.62
3.96
PI
0,9667
52.16
17.48
4.69
0,9787
90.23
46.30
21.29
4.91
NTK
0.9437
57.73
18.50
4.40
0,9802
84.80
53.11
22.14
3.74
YaRN
0,9676
53.46
17.53
4.71
0.9790
88.74
47.05
21.42
5.05
TASR
0.9434
6
57.41
18.48
4.47
0.9807
80.74
22
51.95
22.02
4,65
RIFLEx
0.9431
5
53.79
17.54
4.90
0,9823
73.97
17
50.57
21.22
4.67
Ours
0.944
62.43
23.21
1.01
0,9465
100.0
65.00
26.45
1.02
4x extrapolation
PE
0.9415
11
55.25
16.65
3.75
0.9891
31.41
14
47.12
17.61
3.70
PI
0.9711
12
50.44
16.34
4.87
0.9885
70.93
42.19
17.83
4.82
NTK
0.9477
55.37
16.09
4.24
0.9915
72.39
50.01
18.92
4.23
YaRN
0.9729
51.16
16.69
4.57
0.9877
62.87
41.37
18.53
5.03
TASR
0,9495
55.18
16.16
4.72
0.9911
$1,28
46.81
18.47
4.51
RIFLEx
0.9453
10
51.05
15.83
4.84
0.9906
52.84
41.02
16.47
4.69
Ours
0.9484
47
59.36
21.61
1.01
0.9468
99.87
42
66.54
24.52
1.02
Video), for instance, materializing a 200K x 200K attention mask consumes over 80GB of mem- error. To address this, we integrate Ultra ViCo into Triton-based FlashAttention (Dao et al.) 2022) and SageAttention (Zhang et al.) 2024b), where the online-softmax formulation avoids explicit mask construction. This yields scalable, memory- efficient computation, enabling UltraViCo on large video models.
ory in bf 6, causing an immediate out-of-
EXPERIMENTS
4.1 SETUP
Evaluation. We evaluate methods on three video diffusion models, including Hunyuan Video, Wan2.1-1.3B and CogVideoX-5B. Following RIFLEx, we use 100 prompts sampled from VBench (Huang et al. 2024). For quantitative evaluation, following RIFLEx, we adopt Imaging Quality (Quality). Dynamic Degree (Dynamics), and Overall Consistency (Overall) from VBench, along with the NoRepeat Score for models prone to content repetition. Notably, our NoRepeat Score variant of that in RIFLEx, tailored for multiple-repetition (see Appendix C.2 for details). Fi- nally, we conduct a user study with 10 participants on 10 prompts, where users rank (User) the overall quality of videos across all methods. More details are provided in Appendix
C.2
Implementation Details. The decay factor a is set to 0.9 for Wan and Hunyuan Video at 3x and 4x extrapolation. For Hunyuan Video. set 4 for all ratios, and 8 0.6 at 3x and 0.8 4x. Our baseline configurations follow RIFLEx. Further details are provided in Appendix
4.2 RESULTS
Performance comparison. We compare wide range of length extrapolation baselines on three SOTA models (Kong et al. 2204 Yang et al. 2024; Wan et al. 2025 across various extrapolation ratios, including PE (Zhao et al. 2025), PI (Chen et al. 2023b). NTK (bloc97 2023), TASR (Zhuo) et al. 2024)) YaRN (Peng al. 2023). and RIFLEx. Tab. reports 3x and 4x results on Hunyuan- Video and Wan, while Fig. shows qualitative samples on Hunyuan Video. Results for additional ratios and models are provided in the Appendix C33
As shown in Tab. I our method consistently outperforms all baselines across models and extrap- olation ratios, simultaneously improving video quality and eliminating content repetition. Specifi-
8

---

Published as a conference paper at ICLR 2026
cally, PE suffers from repetition, reflected in low NoRepeat Scores. In contrast, method achieves substantially higher scores, effectively removing repetition. Beyond repetition, unlike RI- FLEx which targets only this issue, method delivers broader gains in both visual quality and motion quality. For instance, it improves Dynamic Degree and Imaging Quality on Hunyuan Video by 233% and 40.5% over the previous best method 4x extrapolation, respectively. Notably, Wan beyond 3x extrapolation, while prior methods collapse and yield static videos (Dynamic De- gree < 12), our method restores fluid motion. By addressing both core failure modes, our method extends the extrapolation limit from 2x to 4x. These improvements are further corroborated by user rankings (Tab. [ and qualitative visualizations (Fig. 6 which consistently confirm the supe- rior quality of our generated videos over baselines.
PE
PI
NTK
YaRN
TASR
RIFLEx
Ours
(a) 3X extrapolation
(b) 4x extrapolation
Figure 6: Qualitative results on Hunyuan Video. The baselines produce nearly static videos with poor visual quality, whereas our method achieves significantly better quality by addressing extrapo- lation failure modes. Additional qualitative results for other models are in Appendix C.4
parabolic
linear
constant
0,8
0.9
0.95
Figure 7: Ablation studies. Top row: different decay strategies have minor impact, suggesting simple constant decay suffices. Bottom row: small a harms consistency while large a offers limited gains. An intermediate value (a = 0.9) enhances quality while preserving consistency.
Ablation studies. We ablate the decay strategy and the decay factor a on Wan at 3x extrapolation. As shown Fig. 7 (top), different decay strategies yield minor differences, indicating that simple constant decay suffices. As shown in Fig. (bottom), strong decay harms consistency (i.e., the spare tire of the car disappears) while weak decay offers limited gains. An intermediate value (a = 0.9) enhances quality while preserving consistency. Further details are provided in Appendix C.2 A sensitivity analysis for and 3 (Fig. 8) shows a stable trend: a 2 0.9 and 2 0.6 improve visual quality and motion dynamics while keeping temporal consistency near baseline. We adopt
= 0.9 and 0.6 as robust defaults, with small adjustments possible (e.g., 3 = 0.8 for stronger consistency, 0.85 for better quality). Although larger and / may introduce mild reduction

---

Published as a conference paper at ICLR 2026
in consistency, values above 0.94 remain visually stable, aligning with common long-video settings (e.g.. Wan's training-horiza consistency = 0.95). See more metrics of a, B in Tab. Fig. 17
and
Connection with other long-video generation methods. UltraViCa aims to extend the effective training window of video diffusion transformers and is therefore orthogonal to existing long-video generation techniques such as FreeNoise (Qiu et al. 2023). FIFO-Diffusion (Kim et al. 2024). and
sliding-window. As demonstrated in Table 2 enlarging the context window via Ultra ViCo consis- tently improves the long-term temporal consistency of these methods, without negatively affecting other performance. Table 2 all methods follow the same evaluation setup (6x extrapolation for 30-second videos on Wan), where UltraViCe extends the base model's training window by 3x.
Generalization to downstream tasks. Our method enhances the model's inherent ability handle longer sequences, making it naturally applicable to downstream tasks. As shown in Fig. based on VACE (Jiang et al. 2025b), UltraViCo enables extrapolation controllable generation and video editing. See Appendix C.4 for additional results.
Consistency
Comsistency x
Dynamies
Quality
Dynamicss
Quality
(a) Illustration of the a sensitivity curve.
(b) Illustration of the sensitivity curve.
Figure 8: Illustration the hyperparameter sensitivity curve. (a) When a 2 0.9, motion dynam- ics improve while consistency stays stable; below 0.9, consistency drops sharply. (b) When 3 > 0.6, dynamics remain high with comparable consistency; below 0.6, consistency degrades significantly.
Table 2: Application of UltraViCa on existing long-video methods.
Method
Consisteney!
Dynamicst
Quality
Overall
Sliding Window
0,8478
56
62.94
23.57
+ ViCo
0.9183
54
62.85
23.95
FreeNoise
0.9243
38
63.09
23.75
+ ViCo
0.9431
41
62.12
23.92
MiFO-Diffusion
0.9131
53
61.31
23.81
+ ViCo
0.9319
51
63.09
24.24
CONCLUSION
In this paper, we identify attention dispersion as the unified cause behind video length extrapolation failures. Based on this insight, we propose training-free method that suppresses attention scores for tokens beyond training length. Experiments show that it significantly improves video quality, extending the practical extrapolation limit from 2x 4x.
ETHICS STATEMENT
This paper advances the field of video generation, while emphasizing the importance of responsi- ble use to avoid potential negative societal impacts, such as the creation of misleading or harmful content.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by Fundamental and Interdisciplinary Disciplines Breakthrough Plan the Ministry of Education of China (No. JYB2025XDXMI01) National Natural Science Foundation of China (Nos, 62522609, 92470118) the Beijing Natural Science Foundation (No. L247030); and the fund for building world-class universities (disciplines) of Renmin University of China.
REFERENCES
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 22669-22679) 2023.
Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to- video generator with diffusion models. arXiv preprint (iX.::2405.04233: 2024.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rom- bach. Stable video diffusion: Scaling latent video diffusion models to large datasets. NONE, 2023.
bloc97. NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) con- text size without any fine-tuning and minimal perplexity degradation. 2023. URL https: //ww:: reddit.com/ Local LLaMA/ http:///oooooooaaaaaa////////a//
scaled_rege wwwpe_elleee____a______ http://ws__aaaamddde___________. sasaaa__mdels_____aeeeee have/
Tim Brooks, Bill Peebles. Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024.
Minghong Cai, Xiaodong Cun, Xiaoyu Li, Wenze Liu, Zhaoyang Zhang, Yong Zhang, Ying Shan, and Xiangyu Yue. Ditctrl: Exploring attention control in multi-modal diffusion transformer for tuning-free multi-prompt longer video generation. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7763-7772. 2025.
Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitz- mann. Diffusson forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 77722081-224125, 2024a.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan. Videocrafterl: Open diffusion models for high-quality video generation, 2023a.
Haoxin Chen, Yong Zhang. Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024b.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint MwXiiv22306155000 2023b.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in neural information processing systems, 15:16344-1639, 2022.
Jianxiong Gao, Zhaoxi Chen, Xian Liu, Jianfeng Feng, Chenyang Si, Yanwei Fu, Yu Qiao, and Ziwei Liu. Longvie: caltimoodal-guidee controllable ultra-long video generation. arXiv preprint arXiv:2508.00 033404 2025.
Yuchao Gu, Weijia Mao, and Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503. /9325, 2025.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. 2022.

---

Published as a conference paper at ICLR 2026
Roberto Henschel, Levon Khachatryan, Hayk Poghosyan, Daniil Hayrapetyan, Vahram Tadevosyan, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 2568-2577, 2025.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang. Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv: 2210.02303, 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. arXiv preprint arXiv:2205.15866 2022.
Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-fest gap in autoregressive video diffusion. arXiv preprint arXiv:2506.08000 2025.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818 2024.
Jiaxiu Jiang, Wenbo Li, Jingjing Ren, Yuping Qiu, Yong Guo, Xiaogang Xu, Han Wu, and Wang- meng Zuo. Lovic: Efficient long video generation with context compression. arXiv preprint arXiv:2507.12952 2025a.
Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing, arXiv preprint arXiv:2503.0759%. 2025b.
Jihwan Kim, Junoh Kang, Jinyoung Choi, and Bohyung Han. Fifo-diffusion: Generating infinite videos from text without training. Advances in Neural Information Processing Systems, 37: 89834-89868, 2024.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming- Chiu, et al. Videopoet: large language model for zero-shot video generation. arXiv preprint urXiv:2312.14122 2023.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03600 2024.
Xingyang Li, Muyang Li, Tianle Cai, Haocheng Xi, Shuo Yang. Yujun Lin, Lvmin Zhang, Songlin Yang. Jinbo Hu, Kelly Peng, et al. Radial attention: O(nlog n) sparse attention with energy decay for long video generation. arXiv preprint ar0iv:2506. 19852, 2025a.
Zhuoling Li, Hossein Rahmani, Qiuhong Ke, and Jun Liu. Longdiff: Training-free long video generation in one go. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 17789-17798, 2025b.
Yu Lu, Yuanzhi Liang, Linchao Zhu, and Yi Yang. Freelong: Training-free long video generation with spectralblend temporal attention. Advances in Neural Information Processing Systems, 37: 131434-131455 2024.
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: -osied text-to-video generation using pose-free videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4117-4125, 2024a.
Yue Ma, Hongyu Liu, Hongfa Wang. Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung- Yeung Shum, Wei Liu, et al. youroji cine-controllable and expressive freestyle portrait animation. In SIGGRAPH Asia 2024 Conference Papers, pp. 1-12, 2024b.
12

---

Published as a conference paper at ICLR 2026
Yue Ma, Kunyu Feng, Zhongyuan Hu, Xinyu Wang. Yucheng Wang, Mingzhe Zheng, Xuanhua He, Chenyang Zhu, Hongyu Liu, Yingqing He, et al. Controllable video generation: A survey, arXiv preprint arXiv:2507.16869, 2025.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on compuler vision, pp. 4195-4205, 2023.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. International Conference on Learning Representations, 2023.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoory Vyas, Bowen Shi, Chih-Yar Yao Ma. Ching-Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Ja- gadeesh, Kumpeng Li, Luxin Zhang, Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang. Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sum- baly, Sai Saketh Rambhatla, Sam Tsai, Samaneh Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy, Shelly Sheynin, Siddharth Bhattacharya, Simran Motwani, Tao Xu, Tianhe Li, Tingbo Hou, Wei-Ning Hsu, Xi Yin, Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo, Yen- Cheng Liu, Yi-Chiao Wu, Yue Zhao, Yuval Kirstain, Zecheng He, Zijian He, Albert Pumarola, Ali Thabet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo, Boris Araya, Breena KerT, Car- leigh Wood, Ce Liu, Cen Peng, Dimitry Vengertsev, Edgar Schonfeld, Elliot Blanchard, Felix Juefei-Xu, Fraylie Nord, Jeff Liang, John Hoffman, Jonas Kohler, Kaolin Fire, Karthik Sivaku- mar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du. Movie gen: cast of media foundation models. arXiv preprint arXiv: 2410.13720, 2024.
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. / /2409, 2021.
Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu. Freenoise: Tuning-free longer video diffusion via noise rescheduling. arXiv preprint arXiv:231015119, 2023.
Yang Song, Jascha
Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1. 2020.
Jianlin Su. Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 11398-14409 2024.
Jiangtong Tan, Hu Yu. Jie Huang, Jie Xiao, and Feng Zhao Freepea: Integrating consistency infor- mation across long-short frames in training-fres long video generation via principal component analysis. In Proceedings of the Computer Vision and Pattrrn Recognition Conference, pp. 27979- 27988, 2025.
Zhenxiong Tan, Xingyi Yang, Songhua Liu, and Xinchao Wang. Video-infinity: Distributed long video generation. arXiv preprint arx6v:24406.16660, 2024.
Genmo Team. Mochi 1. httpp://ggithhbb.commggemg// /mmp:// 2024a.
The FastVideo Team. Fastvideo: A frameook
video generation, April 2024b.
URL
Hansi Teng, Hongyu Jia, Lei Sun, Lingzhi Li, Maolin Li, Minggiu Tang, Shuai Han, Tianning Zhang, WQ Zhang, Weifeng Luo, et al. Magi-1: Autoregressive video generation at scale. arXiv preprint ar arXiv:2505.13211 2025.
13

---

Published as a conference paper at ICLR 2026
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Team Wan, Ang Wang. Baole Ai, Bin Wen, Chaojie Mao, Chen-Wee Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.203 14, 2025.
Fu- Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-I-video: Multi-text to long video generation via temporal co-denoising. arXiv preprint arXiv:2305. 2023.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- model's perception of the world at any resolution. arXiv preprint arXiv:2409. /2/91, 2024a.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need.
arXiv preprint a4Xi::2499. /8869, 2024b.
Yuqing Wang, Tianwei Xiong, Daquan Zhou, Zhijic Lin, Yang Zhao, Bingyi Kang, Jiashi Feng, and Xihui Liu. Loong: Generating minute-level long videos with autoregressive language models. arXiv preprint 21X0022410.02757, 2024c.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104. 14806, 2021.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In European conference computer vision, pp. 720-736. Springer, 2022.
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint ar arXiv:2409 24090.4449, 2024.
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. 2023.
Feihong Yan, Peiru Wang, Yao Zhu, Kaiyu Pang. Qingyan Wei, Huiqi Li, and Linfeng Zhang. Gen- eration then reconstruction: Accelerating masked autoregressive models via two-stage sampling. arXiv preprint arXiv:2510. 2025a.
Feihong Yan, Qingyan Wei, Jiayi Tang, Jiajun Li, Yulin Wang, Xuming Hu, Huiqi Li, and Linfeng Zhang. Lazymar: Accelerating masked autoregressive models via feature caching. In Proceed- ings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 15552-15561. October 2025b.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104. 10157,200 2021.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint arXiv:2408.0600 2024.
Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. In Proceed- ings of the Computer Vision and Pattern Recognition Conference, pp. 22963-22974, 2025.
Jintao Zhang, Haofeng Huang, Pengle Zhang, Jia Wei, Jun Zhu, and Jianfei Chen. Sageatten- tion2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. arXiv preprint arXiv:2411. 100958, 2024a.
14

---

Published as a conference paper at ICLR 2026
Jintao Zhang, Jia Wei, Haofeng Huang, Pengle Zhang, Jun Zhu, and Jianfei Chen. Sageat- tention: Accurate 8-bit attention for blug-and-play inference acceleration. arXiv preprint arXiv:2410.02367, 2024b.
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-imad tramslation via energy-guided stochastic differential equations. Advances in Neural Information Processing Systems, 35:3609-3623. 2022.
Min Zhao, Rongzhen Wang, Fan Bao, Chongxuan Li, and Jun Zhu. Controlvideo: Adding condi- tional control for one shot text-to-video editing, arXiv preprint arXiv:2305.17095 2(3), 2023.
Min Zhao, Hongzhou Zhu, Chendong Xiang, Kaiwen Zheng, Chongxuan Li, and Jun Zhu. Iden- tifying and solving conditional image leakage in image-to-video diffusion model. Advances in Neural Information Processing Systems, 37:30300-30326, 2024.
Min Zhao, Guande He, Yixiao Chen, Hongzhou Zhu, Chongxuan Li, and Jun Zhu. Riflex: A free lunch for length extrapolation in video diffusion transformers. arXiv preprint arXiv:2502. 15894,
2025.
Zangwei Zheng, Xiangyu Peng, Tiajji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.20400 2024a.
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv: 2412.20404, 2024b.
Yuan Zhou, Qiuyue Wang, Yuxuan Cai, and Huan Yang. Allegro: Open the black box of commercial-level video generation model. arXiv preprint arXiv: 2410.15458, 2024.
Le Zhuo, Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongjie Huang, Wenze Liu, Lirui Zhao, Fu-Yun Wang, Zhanyu Ma, et al. Lumina-nexi: Making lumina-12x stronger and faster with next-dit, Advances in Neural Information Processing Systems., 2024.
USE OF LARGE LANGUAGE MODELS
We used a large language model solely to assist in polishing English writing and improving clarity. All research ideas, experiments, results, and interpretations are entirely our own.
A RELATED WORK
Text-to-video Diffusion Transformers. Beyond autoregressive approaches Yan et al. (2025a 5a b).
the recent advances text-to-video generation have been primarily driven by diffusion models (Ho
et al.
2020
Song et al.
20200 Ho et al. 2022, He et al.
2022; Zhao et al. 2022: 2023 Blattmann
al.
2023 Xing et al.) 2023 Chen
[023a Zhao al. [024]| et al. 2024 Zhou al.]
2024
Team 20244 Chen et al. 2024b; Ma et al. 2024a b 2025). With the development of diffu-
sion transformers (DiTs) (Bao et al. 2023 Peebles & Xie 2023), DiT-based text-to-video diffusion models have achieved remarkable performance, such Sora (Brooks al. 2024). Vidu (Bao al.) 2024). VogVideoX (Yang et al., 2024) and Open-Sora (Zheng et al. 2024a).. Although achieving high quality, leading models trained only fixed maximum sequence length, limiting long- term capacity. During video length extrapolation, they suffer from repetition or quality degradation, underscoring the need for length extrapolation.
Length Extrapolation in Transformers. The goal of length extrapolation is to enable trans- formers to generate sequences longer than those seen during training in a single forward (Press) et al. 2021). This is typically achieved by modifying positional encodings. For example, position interpolation (PI) (Chen et al.] (2236) improves performance by interpolating the frequencies in RoPE that they remain within the training range even under extrapolation. NTK (bloc97 2023
15

---

Published as a conference paper at ICLR 2026
YaRN (Peng et al. 2023), and Tame- Scaled RoPE (TASR) (Zhuo al. 2024 combine inter- polation with direct extrapolation, incorporating adjustments along the token dimension, denoising timesteps, and other factors to achieve better results. However, these methods perform poorly image and video DiTs, often leading to content collapse or repetition. RIFLEx (Zhao et al. 2025) mitigates repetition by identifying and attenuating the intrinsic RoPE frequency, yet it still suffers from degraded visual quality. In contrast, our method effectively addresses both content repetition and quality degradation.
Long Video Generation. There also exist many approaches to long video generation (Qiu et al.
2023 Wang et al., 2023 Henschel et al. 2025 Kim et al.) 2024 Tan et al. 2024: Yin et al. 2025
2024c Cai et al. [225|] Li al. 2025b] Lu et al., 2024 Tan et al. 2025 Jiang et al. 2025a Gao et al. 2025 Gu et al. 2025), ) most of which intervene in the diffusion inference pro-
Wang et al.
For instance, Free (Qu al. 2023) enhances temporal consistency via noise initial- ization, FIFO-Diffusion (Kim et al. 2024 feeds frames sequentially into a denoising window of training length, and Video-Infinity TTan et al. 2024) exploits distributed computation scale up video length. While effective for generating long videos, these methods are orthogonal to our length extrapolation strategy, which extends the intrinsic capacity of DiTs longer sequences and can be readily integrated with them.
In addition diffusion- based approaches to long video generation, alternative modeling paradigms
such as autoregressive methods (Wu et al., 2 Yan et al.) 2021 Hong et al. 2022 Wu et al.)
2022 Kondratyuk et al. 2023 Wu al. 2024 Sun et al. 2024 Wang al. 2024b) and diffusion
forcing (Chen et al., 2024a Huang et al. [025 Teng et al. (025) are also capable of generating long videos. Although our method designed for diffusion models, it may also offer insights into length extrapolation for these alternative paradigms.
B MORE DETAILS OF OUR METHOD
B. DERIVATION OF THE PERIODIC OUTPUTS
In this section, we present a formal derivation of Eq. Specifically, the attention score matrix
P E p/'x'' satisfies the following properties up to negligible error:
Prop.1 (Row-wise periodicity): P.j P.j+T, Vi {0................ {O...., L'
11.j {O...., L' L' -T T 1}, where T E N+ corresponds to the observed repetition period in Sec. 3.1
Prop.2 (Relative positional invariance): Pt, Pi+pitp: Vi E (O..... L'-p- 1}.j (0...., L.....,L'' L' p-1}, where E N+ is the relative displacement. In the ffollowing derivation we instantiate p = T.
On basis of the above properties, we derive the periodicity of the attention scores and outputs as follows. Vi E (0. (O...., T
O = Σ 1=0
1'
P
V/TT- PATSVV + Σ j= P4IIV
(8)
1=0
Prop.1
W-T-1 PiTTIHTVV+ + Dj PHTSV
1'-1
(9)
1=0 4-T-1 V, + Σ Ljsue P.j-TV,
Prop.2
(10)
1=0
Prop.1
W-
P. V, + Σ 1-11
-=-/-T P.VV,
(11)
Σ o=
P.
(12)
O,
(13)
16

---

Published as a conference paper at ICLR 2026
B.2 DETAILS THE MULTIMODAL ROTARY POSITION EMBEDDING
In this section, we provide the details of the Multimodal RoPE (M-RoPE) (Wang et al. 2024a) introduced in Sec. 2 Specifically, for token position (1, hi w), the input vector RD is divided into three subspaces of dimensions dr, dx, dw. respectively assigned to tem- poral, height, and width encodings. Each subspace is modulated by its own frequency series Jir {o? Ji [ (d tdx 10: D-1 i=
+d=d Concretely, we define
(Cos(ofPu) Po) - of (sin Pa)
fRaPE (x, 1 w) = R. (pa)
221 R (pa)
(14)
#2i+1]
cos P
where (T.H. W) indexes the temporal, height, and width dimensions with corresponding positions Pa E {t, h, w and frequency components { The index ranges are
E /T....,,T/22-11
a = T, a = H.
dT dx / - 1}. (2 + d2/2. D/2 - 1}, a W.
(15)
After M-RoPE encoding, the queries and keys form Q RR''D and K E RL'xD, As in Eq. 2 they produce the attention logits matrix S E R xI', where the attention logit between the query at (t, h, w), denoted (tt,,,)) and the key at (t (++ At, h + Ah, + Aw), denoted At expands explicitly as:
dr 42/2-1
S (cth
(2:2:+1)T (1.h.w))
R (A (2626+1)
20
dt 224d2/122-
Σ
(2:2:+1)
R (Ah)),
(2:2++11)
(1thh,w)
(t+A,, Ah, +
i 1/2
D/2-1 Σ
(26:2+1)T
(A (1+ (16)
((t
/
d 12/2-1
Σ
A1
cos( A1) + X sin( At) +
/2+dy/2-1
Σ
X cos( ( sin( Ah) sin(o?AAh)]
imdy/2
D/2-1
Σ
(200) cos(o) Aw) + 2 sin( Aw)
(17)
i= /2
where
X (26) (22))
lit.h,w) (E+ AA
((t.h.ar) (e+ h Aww+)
(2i+1) (2i+1)
(18)
(2) (2i+1) (21)
(2)) (2i+1)
((thh, (++ Ar,
(t+ At, AR
(19)
B.3 DERIVATION OF THE STATISTICAL ATTENTION PATTERN S(A!)
In this section, we present the derivation of Eq. in Sec. 3.2.1
We investigate the row-wise pattern of attention logits by examining the expectation of the attention logits between queries and keys at relative temporal distance At (i.e., E (t At.4Att.... (t
This expectation is taken across attention layers, heads, and query positions. In Appendix B.4 we further show that when the true variance is taken into account, the actual attention logits still follow the same patterns as indicated by this expectation.
Strictly speaking, the analysis should target (4..) for all Ah, Aur but as the phc- nomena similar across Ah, Aw, focus S (1.,,,)))))4,..... for simplicity.
17

---

Published as a conference paper at ICLR 2026
Specifically, basis of the formula of M-RoPE (i.e., Eq. (6)). the target expectation is given by
dr/2-1
Br,
Σ
(t.h.,))
1262++1T
(2:224 R. (At)
(t-Ath
d /2t2d//22222 /2-1
D/2-1 Σ
Σ
(2:2141)T
(0)A
(t)
(1
(tt
(2::(+1)T
R (0)A (0))((4-2tt)
(1+Ath),))))
(20)
(2 12
b=yy /2+du/2
dr 12-1
D/2-1
Σ
D1 cos (o. + E sin (o) [t)
+
Σ E
(21)
i=d7/2
where
E = Et
(t.h. (1+ + (2i-1) k (2i+1)
(2i) (2))
((,h,,w) ((+At,h,
(22)
(())) Bt ((thh,w) (f+At,h,w) (ithh, (t+)t,
(26+1) (2))
(26) (2i+1)
(23)
In practice though the integrands these expectations are actually functions At, the empirical statistics in Fig. (col. 1) indicate that their variances with respect At are negligible. Hence,
approximate E and Fi(e) as constants up to negligible error, which is defined by
E ELA,,,
(tth.w) K
(28)
(26) (1+A6h
((t,h,,w) (t+)t,h,))) =: E(()
(26+1) (25+1)
(24)
E E E.
(t.h.ww) K (1+Arh,,) lir ((+At,
12++1)
(26)
(2i)
k (2++1)
E
(25)
By substituting these two expressions into Eq. (22 and Eq. 23 the expected attention logits can be well approximated as S(D(), where
dr/2-1
D/2-1
S(At) =
Σ 1=0
E cos (t. + E sin (TAtt) +
Σ E
(26)
i=d7/2
To simplify the expression, employ the auxiliary angle formula rewrite the two trigonometric functions as one, i.c.,
dr dr/2-1
S(At) = Σ
as cos bi)] + C.
(27)
where a;
+
(1)
bb atan2(-Er) ( E(1)). Interestingly, as shown in Fig.
(col. 2), E remains consistently close to zero, which in turn makes b; nearly vanish (for example,
bo is 0.039 for Hunyuan This observation allows us apply Proposition [ in Sec. 3.2. up
to an error of negligible magnitude. Detailed statistical data for E()) E(") a, by are shown in Fig.
(col. 2, 3, 4).
B.4 CONSISTENCY OF ACTUAL ATTENTION PATTERN WITH S(A()
In this section, we investigate the actual attention scores under the true variance, demonstrating that they preserve the same characteristics as the averaged values described in Sec.[3.2.) As shown in Fig. I0 when the standard deviation over attention layers, heads, and query positions is incorporated into the mean, the attention logits of Hunyuan Video still exhibit clear periodicity at their peaks, whereas those of 1 1 remain non-periodie. Therefore, the conclusions drawn in Sec. 3.2.1 from the mean-based analysis hold with strong generality in practice.
For brevity, omit layer and head indices in the expectation notation.
18

---

Published as a conference paper at ICLR 2026
Variance E.t ". with respect dr
Value of E... F
Amplitudes RoPE froquencies
Bias angles
Vartance respest
Variane of 6 cg
ropect
Frequency index
Frequency index
freqeency index
Frequency index
(a) Statistics of Hunyuan Video.
Variance Elf 6.eh with respoct
Value of E
Amplitudes RoPE froquencies
Bias angles
Variines reqpect
of ropectto
Varimnce of topurtio de
15
Frequency
Frequency index
Frequency
Prequency index
(b) Statistics of Wan.
Figure 9: Statistics of attention logits in Hunyuan Video and Wan. The variances of E E with respect to A/ (col. are negligible compared their expectations (col. 2), making the approx- imation in Eq. (24). Eq. (25) accurate. The bias angles by (col. 4) are close to zero, except for by and 61s Wan whose impact is negligible since the corresponding ag, 415 are near zero (col. 3).
HunyyannVide
Wan
(((r)) (mean std)
S(A/) (mean std)
20
-20
25
50
100
125
20
60
50
Temporal distance A/
Temporal distance Ar
Figure 10: Attention logits under actual variance. Even with standard deviation across layers, heads, and query positions, Hunyuan Video retains clear periodic peaks while Wan 2.1 remains non- periodic, confirming the general validity of the mean-based analysis in Sec. 3.2.1
B.5 PROOF OF PROPOSITION 1
Proposition I is well-known in harmonic analysis and signal processing, and we provide the proof here only for completeness.
Proof. Sufficiency, If di/áN-1 E N+ for all i, write di KKŌN-I with Kr E N+. Let TN-1 2n 2 Then for each i,
cos (0i(A/ + T)) = cos At kT) = cos():A1), VAIER,
(28)
so = f(At), VAt E IR. Hence - is period of f.
Necessity, Suppose Tv-1 2T/ JoN is period of f. Then for all At.
N-1
0= f(At + Tw-1) - = Σ ail cos((o,Atttooooo xTTN-1) - cos().At)).
(29)
1=0
Using cos(x + v) - -cos x = (coss y - 1) 000
sin sin
0 = ag (cos(0,Ty=2) 1) cossoo, (t) -i sin(())))) VALER.
(30)
19

---

Published as a conference paper at ICLR 2026
Model
Attention maps
Statistical row attention analysis
Individual frequencies
Final composite attention
P.s P.
10
S(A0
Hun.
Pam
10
20
30
40
50
10
20
30
40
50
Temporal distance
Temporal distance
(a) Periodic attention:
(b) Approximately harmonic RoPE frequencics (do/d1 du ss N amplify the largest
P.j Ra P.5-T
amplitude da (dashed line), inducing approximately periodic composite attention.
Figure 11: Periodic attention patterns of CogVideoX. The RoPE frequencies of Cog VideoX ap- proximately satisfy the harmonic condition, which amplifies the largest-amplitude component and thereby induces periodic attention patterns.
The family {cos(dr.)) (n((o))); with distinct positive de is linearly independent over RR (e.g., via independence of etodd), Hence for each
cos cos( =1 0,
sin( =
(31)
so 6.TN-1 E 2TZ. Substituting TN-1 = 2T/ŌN-1 yields
Φi e
(32)
ΦN-1
as 0; >
B.6 REMARKS ON PROPOSITION
Relaxed conditions under which the proposition holds approximately. Although the strict dition for forming harmonics in Proposition [ is 0:/áN-1 E N+, in this section we highlight ap- proximate conditions that likewise induce dominant frequency leading content repetition in videos. Specifically, if <i/ (xx 1 is sufficiently close to an integer, constructive amplification can still occur for small |t| (e.g., [t] S. -1) For example, for Cog VideoX, the ratio of the first two frequencies is / Φ1 3.16, which is close to the integer 3, thereby producing a dominant compo- nent that accounts for 50.80% of the total amplitude. This gives rise to an approximately periodic composite attention pattern (Fig. which in turn leads to content repetition (Fig. 13 right).
Remarks on the strict period of HunyuanVidev We herein examine the strict periodicity of Hunyuan Video. Strietly speaking, its fundamental frequency is d7, with ratios o 27-;, ;i (0....,7) 7)- According to Proposition [ the theoretical period of ( is T7 = d However, as shown in Fig. 19 (col. 3), the amplification contributed by dn is very small, accounting for only 6.677%, which makes its impact negligible. Moreover, its period of 804 is far larger than the extrapo- lation length (c.g., 132 at 4x extrapolation). rendering the variation of the corresponding component almost imperceptible within this range. The same reasoning applies o, for E {4. 5, 6). Conse- quently, our analysis focuses Φ with i E (0, 1, 2.2.3) 3), whose ingle-frequeny contributions are both large enough in amplitude and sufficiently oscillatory shape S(At).
B.7 NECESSITY OF CONCENTRATING ON THE TRAINING WINDOW
In this section, provide detailed experimental evidence supporting the discussion in Sec. 3.2.2 on where sharpened attention focus is most beneficial. Specifically, on Wan with extrapolation ratio = 3. test four strategies for sharpening attention: concentrating the leading of each row, the trailing 1 . the training window, and the top-! tokens according the original attention scores. As shown in Fig. 12 concentrating on the leading or trailing 1 of each row causes the & video to collapse, while top yields poor visual quality with little dynamics. contrast, restricting attention to the training window leads to the most significant improvement in video quality.
20

---

Published as a conference paper at ICLR 2026
concentrating on the leading _
segment
concentrating on the trailing segment
concentrating on top-
tokens
concentrating on the training window
Comparison of attention concentration strategies on Wan at = 3. Concentrating on the leading or trailing 1 of each row collapses the video, and top- yiclds poor quality with little dynamics. Restricting attention to the training window proves most effective.
Figure 12:
C MORE DETAILS OF EXPERIMENTS
C.I FAILURE MODES OF COGVIDEOX
In this section, we present the manifestation of the failure modes of video length extrapolation as discussed in 3...3.1oon on an additional model, CogVideoX. As shown Fig. [3] when extrapolated to three times the normal training length, the generated videos exhibit a sharp decline in both dynamic degree and visual quality, along with noticeable content repetition.
normal length
3X extrapolation
Figure 13: Failure modes of Cog VideoX under 3x extrapolation. The generated videos show degraded visual quality, reduced dynamics, and clear content repetition, consistent with the failure modes discussed in Sec. 3.1
C.2 MORF IMPLEMENTATION DETAILS
In this section, we provide further details of Sec.
The implementation of NoRepeat Score. The NoRepeat Score implemented in RIFLEx ZZaao et al. 2025) is only applicable when the content repeats once, which makes it unsuitable for longer extrapolation tasks. We therefore modify it accordingly. Specifically, the computation of the NoRe- peat Score consists of two steps: static-video filtering and repeated- frame ratio calculation. In the first step, we uniformly sample frames across the video; the mean pairwise L2 distance among them falls below threshold, the video is considered static and discarded. This prevents completely static videos from interfering with subsequent repetition detection. In the second step, we measure the ratio of repeated frames to the total frame count, which defines the NoRepeat Score. Following RIFLEx, we first search around the dominant-frequency period for the frame with the minimal L distance to the first frame. This frame then taken the start of candidate repeated sequence. We then compare each frame in this candidate sequence with the corresponding frame at the beginning of the video; frames whose Laz distance below the threshold are counted as repetitions. Empir- ically, threshold of 55 was found to align better with human perception and was consequently applied to both steps. Finally, we report the mean NoRepeat Score across vidcos as the final result. The detailed implementation code is included in the supplementary material.
The implementation of RIFLEx and ViCo on Wan. Since Wan does not exhibit content repetition, it is not applicable determine the dominant frequency from the repetition period per-
21

---

Published as a conference paper at ICLR 2026
formed in Zhao al. (2025), Instead, following Sec. Sec.3.2.1 we take the aappett-a frequency @ as the dominant frequency.
For UltraViCo the first frame's decay factor is set negative to fix its blurring. We hypothesize that this is caused by the causal design of the video VAE, where the first frame is encoded indepen- dently and without temporal compression. As result, exhibits different statistical properties from subsequent frames and becomes more sensitive to perturbations.
Details of the ablation study. Herein, we detail the setup the ablation study in Sec. 4.2 Specif- ically, as shown in Fig. 7 (top), we compare three decay strategies parabolic, linear, and constant. The parabolic strategy takes the following form:
if - 1./2 Sij <0, X = jal(i=///L')-- a2 - - jil-))))))3), otherwise,
(1,
Sij Suj. where
(33)
whereas the linear strategy takes the following form:
8/g = >ug Sig, where Xu -
if - j| v L/ Sij < 0, aaili (aili-jl/Li + a2 - i j-ji/LL), otherwise,
1,
(34)
and the constant strategy is
1, if i - jl - L/2 or Stg < 0,
Si = >ij Sij, where Xij =
(35)
ax otherwise.
We set 0.9 for the constant strategy, and a1 0.85, 02 0.95 for the parabolic and the linear strategies. As shown Fig. (top), parabolic, linear, and constant decay yield only minor differences, indicating that the key is distinguishing in-window from out-of-window tokens rather than the decay shape.
C.3 ADDITIONAL EXPERIMENTS OF DIFFERENT EXTRAPOLATION RATIOS AND MODELS
to 5 and models based on 25 prompts from VBench (Huang et al.) 2024). To evaluate the generality of UltraViCo, we test 2x extrapolation on Hunyuan Video, Wan, and VodVid, as well as 3x and 4x extrapolation on Cog Video. In addition, we assess 5x extrapolation
Settings. In this section, we provide some additional extrapolation ratios from
Hunyuan Video. For Wan, we set & = 0.9. For Hunyuan Video, we use & = across all ratios, with a = 0.95, 8 = 0.6 at 2 and 0.9, B 0.8 at 5x. For Cog CogVideoX,
y and / = 0.6 for all ratios, with = 0.9 at 2 and 3x, and a = 0.85 at 4 x. The configurations of other baselines follow Sec. 4.1
Results. We compare ViCo with the baselines in Sec. 4.2 As shown in Tab. 3 UltraViCo achieves the best performance across all models and extrapolation ratios, not only avoiding content repetition but also substantially improving video quality. For example, Cog VideoX exhibits nearly static videos at 4x extrapolation (Dynamic Degree 16) with poor visual quality (Imaging Quality < 56), whereas our method significantly enhances both temporal dynamics and visual quality, with Dynamic Degree and Imaging Quality improving by 200% and 13.48%. respectively. Furthermore, at 5x extrapolation, Ultra ViCo also demonstrates strong performance, surpassing the best bascline scores by 350% in Dynamic Degree and 47.59% in Imaging Quality, indicating the potential of our method to extend to larger extrapolation ratios.
C.4 MORE QUALITATIVE RESULTS OF OUR METHOD
As shown in Fig. and Fig. 15 whether under 3x or 4x extrapolation ratios, and across Wan and Cog VideoX, our method consistently achieves substantially superior visual quality and temporal dynamics com- pared to the baselines. For example, shown in Fig. [4] the videos generated by various baselines for 3x and 4x extrapolation on Wan are nearly completely static, whereas our method produces highly fluid and natural large-scale motion. Similarly, as shown in Fig. [5 the videos from the base- lines are very blurry with dull colors, while our method generates realistic, natural results with rich details.
In this section, we provide additional qualitive results for the experiments in Sec. 4.2
22

---

Published as a conference paper at ICLR 2026
Table Quantitative results on VBench for more models and extrapolation. Note that NeRepeat Score is essentially binary indicator: red entries indicate visually obvious repetitions, while others show no noticeable repetition.
Wan with 2x extrapolation
CogVideoX with 3x extrapolation
Method
NoRepeat
Dynamict
Quality?
Overall
NoRepeat
Dynamic?
Qualityf
Overall
PE
N/A
32
58.13
23.22
82.52
16
57.91
19.59
PI
N/A
32
54.23
21.52
99.07
54.27
18.17
NTK
N/A
44
59.59
23.52
86.07
55.24
19.33
YaRN
N/A
24
55.14
21.57
97.47
53.96
18.05
TASR
N/A
36
59.97
23.70
97.93
55.75
19.24
RIFLEx
N/A
16
48.15
20.34
97.86
55.31
19.03
Ours
N/A
68
66.88
25.28
99.38
32
60.09
24.77
Hunyuan Video with 2x extrapolation
Cog VideoX with 4x extrapolation
Method
NoRepeatt
Dynamict
Quality
Overallf
NoRepeat
Dynamic?
Qualityt
Overall
PE
80.43
40
62.67
24.36
76.57
16
55.25
17.27
PI
98.87
4
52.35
23.55
88.53
46.82
16.63
NTK
94.97
32
65.47
24.62
78.89
52.74
18.14
YaRN
97.99
4
52.87
23.26
94.75
47.36
16.90
TASR
94.85
36
64.55
24.59
99.13
46.75
17.28
RIFLEx
97.27
36
65.19
24.52
97.00
50.59
16.66
Ours
97.53
44
66.50
24.82
96.79
48
62.70
25.39
CogVideoX with 2x extrapolation
Hunyuan Video with 5x extrapolation
Method
NoRepeat
Dynamict
Quality?
Overall
NoRepeat"
Dynamic*
Quality
Overall
92.31
28
64.28
22.83
30.78
39.04
15.64
PI
98.85
8
57.11
21.88
81.58
36.63
16.76
NTK
94.66
16
63.04
23.55
71.54
43.43
17.78
YaRN
98.81
8
58.83
77.70
37.88
17.85
TASR
95.91
16
62.17
23.44
35.31
42.88
17.88
RIFLEx
99.42
16
60.30
23.28
53.65
40.55
15.71
Ours
98.92
32
64.39
25.36
99.44
24.16
Moreover, present another downstream task in Fig. 16 where generation is performed based on given pose. Our method achieves high quality and dynamic results while closely following the given conditions.
C.5 ACCELERATION OF ULTRAVICO VIA SPARSE ATTENTION AND DISTILLATION
Building upon recent advances in ppare-attenntion based video acceleration and distillation Team 2024b), Ultra ViCo achieves about 16x speed-up without compromising performance (see Table
C.6 RUNTIME AND MEMORY COST
As shown in Table g built on top of FlashAttention (Dao et al. 2022) and SageAttention (Zhang
al. 2024b a). Ultra ViCo incurs almost no additional overhead in either latency or memory usage.
D FURTHER DETAILS OF ULTRAVICO
D.I ULTRAVICO WITH EFFIEIENT ONLINE ATTENTION
Uitr does not require materializing the full attention matrix and can be seamlessly integrated into efficient online attention kernels. Herein, we present its implementation based on FlashAtten- tion, illustrated by Algorithm
23

---

Published as a conference paper at ICLR 2026
PI
NTK
YaRN
TASR
RIFLEx
(a) 3X extrapolation
(b) 4X extrapolation
Figure 14: Qualitative results on Wan. The baselines produce nearly static videos with poor visual quality, whereas our method achieves significantly better quality and much more motion.
PE
PL
NTK
YaRN
TASR
RIFLEx
Ours
(a) 3X extrapolation
(b) 4X extrapolation
Figure 15: Qualitative results on Cog The baselines produce nearly static videos with poor visual quality, whereas our method generates realistic results with rich details and fluid motion.
24

---

Published as a conference paper at ICLR 2026
Our method for pose-guided video generation. Our method closely aligns with the given pose conditions, while ensuring high dynamic range and excellent visual quality.
Figure 16:
Algorithm 1 UltraViCo FlashAttention Kernel
Require: Matrices Q, K, Nxx block bg, ble.
1: Divide Q into Tm = N/bb blocks (Qm), h. and divide K, V into Tn Nibko blocks {Ka} and (Va):
2: for m in [1, Tm do
3: for in Tx. do
4: 7= by rangeeO, bq); = bkv range((,, be.)), Z R R xxx R Rxxbr
5:
Initialize X Xb to O;
6: A Eq (.j)
7:
ST. AQ
8:
Ph maxxpm n-1
.roomax 5m ))
9:
Pm exp(S". pmn)
10:
m ei -hh rowsum (!)) Im
11:
t diag(e?" 10= Pm Vn
end for
13:
Om diag(1 ) -T
14:
end for
15:
return O }:m};
D.2 ABLATION ON HYPERPARAMETERS
In this section, present detailed illustrative ablation results for the hyperparameters and 3. The detailed sensitivity curve is shown in Fig, 17 while the illustrative ablations on the independent effects of and the main experiments are reported Tab. 6
Consistency
Dynamies
Quality
8
Over
epeat
(a) Schematic diagram the a sensitivity curve.
Consistency
Dynamics
Quality
Overall
8
(b) Schematic diagram of the sensitivity curve.
Figure 17: Illustration of the hyperparameter sensitivity curve.
25

---

Published as a conference paper at ICLR 2026
Table 4: Illustrative sensitivity analysis of a on Hunyuan at extrapolation. We set 3 equal to a, Le., a single decay factor is shared globally.
Consistencyt
Dynamicst
Qualityt
Overall
NoRepeat
1.0
0.9795
16
51.85
21.62
53.17
0.95
0,9663
25
54.92
24.07
100
0.9
0.9647
32
57.53
26.25
93,34
0.85
0.9298
68
69.93
26.89
99.53
0.8
0.9231
70.35
26.96
100
Table 5: Illustrative sensitivity analysis of 3 on Hunyuan at 3x extrapolation. We set = 0.9 across all settings.
8
Consistencyt
Dynamicst
Quality
Overallt
NoRepeat"
1.0
0.9716
28
55.23
24.52
57.42
0.9
0.9647
32
57.53
26.25
93.34
0.8
0.9510
45
59.35
26.42
97.25
0.75
0.9496
51
62.11
26.98
95.77
0.6
0.9465
62
65.00
26.45
100
0.45
0.9349
65
68.34
26.99
100
0.3
0.9318
66
70.45
26.98
100
Table 6: Illustrative ablation experiments that independently examine the individual effects of
a and B.
Method
Consistencyt
Dynamicst
Qualityt
Overall
NoRepeatt
Hunyuan Video with 3x extrapolation
a 1.9 1
0.9795
16
51.85
21.62
53.17
a 0.9, 0 =1
0.9716
28
55.23
24.52
57.42
1,3 0.6
0.9784
25
55.13
23.13
93.52
a 0.9, 3 0.6
0.9465
62
65.00
26.45
100
W-n..1-1.3B with 3x extrapolation
a = 1
0.9419
6
56.28
18.53
a 0.9
0.9444
46
62.43
23.21
Table 7: Illustrative performance when combined with recent video-acceleration methods on
Hunyuan Video.
Setting
Time Cost.
Consisteneyt
Dynamicst
Quality
Overall
NoRepeatt
3x
GPU-hours
0.9465
62
65.00
26.45
100
3x with FastVideo
0.3 GPU-hours
0.9432
64
63.89
25.98
100
4x
GPU-hours
0.9491
42
66.54
24.52
99.87
4x with FastVideo
0.5 GPU-hours
0.9399
40
62.24
24.83
96.32
26

---

Published as a conference paper at ICLR 2026
Table 8: Illustrative runtime and memory comparison. Note that SageAttention is optimized for 4090-like architectures; A800, its runtime is comparable FlashAttention.
Model / Method
Time (s /iter)
Memory (per GPU)
HlnyyanVidee (3x extrapolation)
SageAttention
341.2
73188M
SageAttention + Ours
349.6
72346M
FlashAttention FlashAttention Ours
349.3
76030M
355.3
75932M
Wan (3x extrapolation)
SageAttention
32,13
24342M
SageAttention + Ours
34.12
24342M
FlashAttention
32.64
24349M
FlttsAttention Ours
33.74
24346M
(a) Performance of the vidco-continuation haseline alone.
(b) Illustration of combining Ultra ViCr with the video-continuation method.
Figure 18: Application of UltraViCa to segment-wise long-video generation. (a) Wan2.2-T12V
uses only a few ending frames, causing identity drift; (b) UltraViCo alleviates this issue.
27


Reviews:
[{"review_id": "ihoEQabxQM", "reviewer": "Reviewer_ju43", "summary": "The paper tackles video length extrapolation in diffusion transformer–based text‑to‑video models, which suffer from repetition and quality loss beyond training length. It identifies attention dispersion as the root cause and introduces UltraViCo, a simple training‑free decay on out‑of‑window attention logits. This method extends extrapolation from 2x to 4x and significantly improves motion and visual quality across multiple models.", "questions": "1) How can alpha, beta, gamma be selected without manual tuning? Could you propose a lightweight heuristic or adaptive scheme that works across unseen models?\n2) Why stop at 4x only? How does model perform beyond this threshold? If not, why do you think the method is not suitable for extreme long video generation task?\n3) What do you think about other works/directions that are trying to tackle long video generation (e.g., Diffusion Forcing, Frame Pack)? Do you think augmenting/optimizing attention maps (the approach taken in this paper) is a more promising direction as opposed to others?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper is very well written; the figures are very helpful to understand the motivation behind the proposed method.\n2) This work provides a compelling analysis that connects two failure modes (repetition and quality degradation) under the concept of attention dispersion, offering a deeper understanding of underpinnings of video diffusion transformer models.\n3) The proposed solution UltraViCo is training-free, simple to implement, and integrates seamlessly with existing popular SOTA video DiTs.\n4) Very good empirical results and comprehensive evaluation of the method and baselines.\n5) The approach demonstrates applicability to downstream tasks like controllable video generation and video editing without retraining.", "weaknesses": "1) UltraViCo relies on manually chosen decay factors (alpha, beta) and harmonic band width (gamma), which differ across models. The paper does not provide a fully automated procedure for selecting these values, raising concerns about generalization to unseen architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OGNZc3S5jz", "reviewer": "Reviewer_apZp", "summary": "This paper proposes a length-extension method for video generation models, aiming to generate longer videos without additional training. The authors achieve this by analyzing and modifying the attention mechanism. In the experiments, they compare their method with other training-free video extension approaches and demonstrate its effectiveness.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This study follows a clear methodology: observation, analysis, improvement, and validation, with rigorous logic and well-structured presentation.\n* Experiments demonstrate the effectiveness of the proposed method on Wan and HunyuanVideo, outperforming baseline approaches.", "weaknesses": "* The authors’ modifications to the computation process may introduce potential performance issues. Their improvements focus on the attention map, but in efficient attention implementations (e.g., FlashAttention-3), this matrix is not fully materialized. Consequently, the proposed changes could negatively impact computational efficiency.\n* Generating long videos is no longer a major challenge, as several publicly available models already support segment-wise long-video generation—such as Wan2.2-S2V, Wan-Animate, and LongCat-Video. The authors should consider including these models in their baseline comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oY1ZS1YA4I", "reviewer": "Reviewer_Tf16", "summary": "This paper tackles video length extrapolation for generating videos longer than the training sequence length in diffusion-based video transformers. State-of-the-art text-to-video diffusion models are trained on fixed-duration clips (e.g. 5 seconds) and struggle beyond that length. The paper highlights the two failure modes that emerge as sequence length grows: (i) periodic content repetition, and (ii) quality degradation affecting all models. These issues worsen at longer extrapolations, limiting practical use. Prior work such as RIFLEx addressed repetition via positional encoding tweaks but did not solve the widespread quality drop. The author(s) of this paper identify a unified cause behind both failures, i.e, attention dispersion. When generating beyond the trained window, new frames dilute the model's learned attention focus, causing it to spread attention too broadly. In extreme cases, Rotary Positional Embedding (RoPE) harmonics can organize this dispersion into a periodic pattern, explaining the looping behavior. More generally, dispersion leads to the model losing focus on fine details and mixing unrelated temporal contexts. Building on this insight, the paper introduces UltraViCo. It is a training-free, plug-and-play method to restore the model’s focus during long video generation. The author(s) claim impressive gains in video quality and temporal dynamics on multiple text-to-video diffusion models (HunyuanVideo 1.3B, CogVideoX 5B, and Wan 2.1) and various extrapolation lengths.", "questions": "1. Does suppressing distant-frame attention risk any long-term coherence issues? In particular, how does UltraViCo ensure that important elements present in the first few frames like background setting, main character identity, etc. are still represented in later frames, given those early frames’ influence is decayed?\n\n2. The paper positions as complementary to other long-video generation techniques. Have the author(s) tested or considered combining UltraViCo with such methods? For instance, could one apply UltraViCo in a sliding-window generation to further improve consistency at the window boundaries, or use it alongside a method like FreeNoise to additionally boost temporal stability?\n\n3. The implementation mentions setting the first frame decay factor to a negative value to counteract blurring. Could the authors clarify why the first frame needed special treatment?\n\n4. How should one choose the decay parameters $(\\alpha, \\beta, \\gamma)$ for a new model or a different extrapolation ratio? The results show different optimal values for different scenarios (for e.g.  stronger decay for 5x on HunyuanVideo, $\\alpha = 0.85$ for CogVideoX at 4x).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a unification of two major failure modes under the single concept of attention dispersion. To me, this is a good contribution contending the earlier positional encoding-centric explanations.\n\n2. The approach presented in the paper requires no additional training or model modifications, making it a plug-and-play inference-time fix. Despite its simplicity, it tackles both identified issues simultaneously.\n\n3. The paper gives importance to practical implementation. Integrating the attention decay into a custom FlashAttention kernel shows strong engineering effort so that the method works on large-scale models without running out of memory.", "weaknesses": "1. UltraViCo down-weights the influence of very distant frames. A possible side effect is that the model might lose some long-term context or consistency. The paper asserts that important content is preserved while removing irrelevant far-context influence but it does not rigorously quantify scene consistency over extremely long videos. In one ablation, using too small an $\\alpha$ caused a car tire to disappear in later frames, indicating that overly concentrating attention can indeed harm persistent content.\n\n2.  The approach requires a massive computational cost for generating long videos. The authors report needing ~8 A800 GPU-hours per video for 4x extrapolation on the largest model, which forced them to use a reduced number of samples for evaluation.\n\n3. The paper focuses comparisons on inference-time extrapolation techniques (RoPE/PE variants and RIFLEx), which is appropriate, yet omits direct comparison to some alternative approaches for long video generation. For example, sequential generation methods like FIFO-Diffusion or diffusion re-initialization schemes (NUWA-XL, Flexible Diffusion Modeling for long videos Neurips 2022, interactive video generation via domain adaptation, and others) could also produce long videos. The authors cite these works and justifiably note they are different.\n\n4. The theoretical analysis stops short of fully justifying the chosen solution. The idea of suppressing out-of-window attention is intuitively motivated and empirically validated, but there isn’t a formal framework proving it as an optimal or necessary fix. For example, the harmonic analysis (Proposition 1) explains when periodic repetition arises, but there’s no analogous theory for why a constant decay factor is the best way to concentrate attention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H5gUFNZ3HB", "reviewer": "Reviewer_3Ey3", "summary": "Video Length Extrapolation adapts pretrained video diffusion transformers to generate sequences longer than training in a single pass, but models typically loop content and lose quality beyond trained length 𝐿. The paper attributes both failures to attention dispersion. UltraViCo, a training-free method that decays attention to tokens beyond the training window, restores focus, extending extrapolation from 2× to 4×. This outperforms baselines across models and extrapolation ratios.", "questions": "1. Given existing approaches for long video generation that rely on inference-time modifications (e.g., FIFO), what is the specific motivation and unique significance of studying video length extrapolation as a separate paradigm? \n\n2. In Figure 3, when visualizing Eq. (4), how many video samples were used? Was the visualization averaged over multiple videos, or based on a single example? Because Eq. (4) involves both queries and keys, it seems necessary to report statistics over multiple videos to confirm consistency of the observations.\n\n3. For Eq. (6), is there an analysis of how varying 𝛼 and 𝛽 affects the results (e.g., sensitivity curves or robustness across ranges)?\n\n4. Regarding the decay factors introduced in Eq. (6), could you include an ablation with quantitative metrics in Table 1 to show their individual contributions?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Training-free, plug-and-play: Requires no additional data or fine-tuning.\n\n- Repetition mitigation via RoPE insight: Identifies the correlation between attention periodicity observed across models and RoPE-induced harmonics, effectively reducing repetition during video length extrapolation.\n\n- Quality preservation through attention concentration: Alleviates attention dispersion beyond the training window, mitigating quality degradation in long-horizon generation across diverse video generation models.", "weaknesses": "1. Non-uniform evaluation prompts: Prompts are not standardized across models (e.g., UltraViCo uses 100 prompts, HunYuan uses 25), complicating fairness and direct comparability.\n\n2. Limited analysis of Eq. (6): The paper provides insufficient sensitivity/ablation on how changes in 𝛼 and 𝛽 affect outcomes, leaving robustness and trade-offs unclear.\n\n3. Hyperparameter dependence: 𝛼 and 𝛽 must be tuned per model and per extrapolation ratio, reducing generality and increasing deployment friction—hindering active, broad use despite the training-free claim.\n\n4. Minor suggestions: The notation in Eq. (6) makes it unclear whether the constant 1 or 𝛽 has precedence (i.e., which term is intended to dominate or be applied first). It would be clearer to revise the expression to make the precedence explicit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)