Lumos-1: On Autoregressive Video Generation with Discrete Diffusion from a Unified Model Perspective
Authors: Hangjie Yuan
Weihua Chen
Jun CEN
Hu Yu
Jingyun Liang
Shuning Chang
Zhihui Lin
Tao Feng
Pengwei Liu
Jiazheng Xing
Hao Luo
Jiasheng Tang
Fan Wang
Yi Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Video generation
autoregressive models
unified models
Primary area: generative models

Abstract: Autoregressive large language models (LLMs) have unified a vast range of language tasks, inspiring preliminary efforts in autoregressive (AR) video generation. Existing AR video generators either diverge from standard LLM architectures, depend on bulky external text encoders, or incur prohibitive latency due to next-token decoding. In this paper, we introduce Lumos-1, an LLM-based unified model for AR video generation with efficient discrete diffusion. Firstly, to fit videos with LLMs, we identify that 1D RoPE is ill-suited for visual spatiotemporal correlation modeling, and while demonstrated to be useful, naive 3D RoPE exhibits imbalanced frequency spectra. Therefore, we propose MM‑RoPE, which preserves the original textual RoPE while seamlessly accommodating video data with comprehensive frequency spectra and scaled 3D positions. Secondly, to fit the video data's nature and overcome the inefficiency of next-token decoding, we adopt a parallel and mask-based discrete diffusion with the intra-frame bidirectional and inter-frame causal attention masks. Based on this attention mask, we uncover the frame‑wise loss imbalance issue caused by spatial information redundancy and propose Autoregressive Discrete Diffusion Forcing, which introduces temporal tube masking during training with a compatible inference‑time masking policy to avoid quality degradation. Despite using only 48 GPUs for pre-training, limited data and a discrete tokenizer, Lumos-1 achieves results surpassing those of Show-o2 on GenEval, COSMOS-Video2World on VBench-I2V, and OpenSoraPlan on VBench-T2V.

Published as a conference paper at ICLR 2026
LUMOS-1: ON AUTOREGRESSIVE VIDEO GENERATION WITH DISCRETE DIFFUSION FROM A UNIFIED MODEL PERSPECTIVE
Hangjie 1.233 Weihua Chen Jun Cen Hu Yu', Jingyun Liang!.2 Shuning Chang1.2.3 Zhihui Lin Tao Feng'. Pengwei Liu Jiazheng Xing Hao Luo!.? Jiasheng Tang 1.2 Fan Wang', Yi Yang"
DAMO Academy, Alibaba Group. HHupan Lab, *Zhejiang University. "Tsinghua University
t Corresponding author. pj.yuanEzju.ecm .d
Code and models: https:/ /tas:://gitae //ggtttthbb....// com/ a Liiiaaa--aae danaa--da--aaae darabbb--ddd-aaaadyy/ynmmmm
ABSTRACT
Autoregressive large language models (LLMs) have unified vast range of lan- guage tasks, inspiring preliminary efforts in autoregressive (AR) video generation. Existing AR video generators either diverge from standard LLM architectures, depend on bulky external text encoders, or incur prohibitive latency due to next- token decoding. In this paper, introduce Lumos-1, LLM-based unified model for AR video generation with efficient discrete diffusion. Firstly, to fit videos with LLMs, we identify that ID RoPE is ill-suited for visual spatiotemporal correlation modeling, and while demonstrated to be useful, naive 3D RoPE exhibits imbalanced frequency spectra. Therefore, we propose MM-RoPE, which preserves the original textual RoPE while seamlessly accommodating video data with com- prehensive frequency spectra and scaled 3D positions. Secondly, to the video data's 's nature and overcome the inefficiency of next-token decoding, we adopt a parallel and mask-based discrete diffusion with the intra-frame bidirectional and fnter-frame causal attention masks. Based on this attention mask, we uncover the trime-wise loss imbalance issue caused by spatial information redundancy and propose Autoregressive Discrete Diffusion Forcing, which introduces temporal tube masking during training with a compatible inference-time masking policy avoid quality degradation. Despite using only 48 GPUs for pre-training and fine-tuning, limited data and discrete tokenizer, Lumos-1 achieves results sur- passing those of Show-o2 on GenEval, COSMOS-Video2Wee on VBench-I2V, and OpenSoraPlan on VBench-T2V.
INTRODUCTION
Autoregressive (AR) models have demonstrated significant advancement in the field of language
processing OpenAl (2023); Touvron et al. (2023a Ouyang et al. (2022); Bai al. (2023) ; Liu et al.
(2024a) by unifying diverse language tasks into one single framework and expanding the scale of
large language models to unprecedented sizes. Following this trail of research, significant efforts have emerged in autoregressive visual generation Li al. (024a):(Chang et al. (2023) 2022 Luo
al. (2024). By utilizing similar architectural and generation designs those used LLMs there substantial potential for advancing LLMs towards a unified model capable of both visual generation
and understanding Wang al. (2024c); Xie al. (2024): Zhou al. (2024): Tong et al. (2024).
Pioneering research efforts have been dedicated to instantiating AR visual generation, with focus on key aspects, spanning improving autoregressive paradigm Pang et al. (2024); Tian et al. (2024); Chang et al. (2022): Luo et al.) (2024), improving tokenizer format Li et al. (2024a) (e.g.. from discrete to continuous), and enhancing tokenizer capabilities Tang et al. (2024); Wang et al. (2024a); Shi et al. (2024). However, an AR video generation paradigm that is fully compatible with LLM- based unitied models (e.g., Chameleon Team 2024) and EMU3 Wang et al. (2024c) remains underexplored. Preliminary attempts either exhibit architectural distinctions with LLM architectures (e.g., NOVA Deng et al. (2024) and Phenaki Villegas et al. (2022)) ), rely on external text encoders (e.g., Illmm (un al. (2024) and Fluid Fan et al. (2024) or trail in generation efficiency due

---

Published as a conference paper at ICLR 2026
to next-token prediction (e.g., Loong Wang et al. (202dd). Therefore. the introduction of LLMs to visual generation motivates careful designs for modeling the spatiotemporal visual space.
Adapting LLMs for video generation presents two fundamental challenges. First, the standard positional encoding in LLMs, such as ID Rotary Position Embeddings (RoPE) Suu et al. (2024), is inherently designed for sequential text and is ill-suited for modeling the complex 3D spatiotemporal correlations of video. Second, the standard autoregressive paradigm of next-token prediction, while effective for text, is notoriously inefficient for visual data and fails to properly model videos' unique properties: spatial bidirectionality within a frame and strong temporal causality across frames. Addressing these two challenges- RRoP representation and prediction paradigm- is critical for building an effective and efficient model for video generation.
To account for the spatiotemporal na- ture of videos using the RoPE tech- nique, we start with systematic exper- iments and analysis (in Sec. 3.1 of
Table 1: Design comparison with other types of RoPE
RoPE
Computiable with Text RoPE
Comprehensive
Strategic Scaling
Type
Structure
Frequency Allocation
extending 1D RoPE to a vanilla 3D RoPE Hong et al. 2022); Kong et al. (2024). revealing that ID RoPE is far from optimal, and the incorporation 3D RoPE facilitates autoregressive generative learning. However, existing 3DD RoPE still suffers from imbalanced frequency spectrum ranges for temporal and spatial modeling, as shown in Sec. [.] Building on this insight, we pro- pose MM-RoPE, new family of RoPE that better accommodates the structure of video data. As summarized in Tab. [ the core enhancement lies in more distributed frequency allocations for more comprehensive context modeling. substantially improving the visual generation capability. More- over, we design a principled scaling to the 3D positions, which improves vision-languaee modality balancing. Though MM-RoPE modifies RoPE for visual tokens, the RoPE for text tokens remains consistent with native L.Mss, therefore preserving the language learning capabilities.
M-RoPE
x
X
U-RoPE
x
x
IL-RoPE
X
X
VideoRoPE
HaPE
X
MM-RoPE
To account for the nature of videos (i.e., spatial bidirectionality and temporal causality), we build our model upon the intra-frame bidirectional and inter-frame causal token dependency strategy, so that we can introduce the use of the efficient and parallel discrete diffusion Chang et al. (2022). Discrete diffusion requires training with masks Li et al. (2024a). However, the naive global random mask prediction leads to the loss imbalance issue, which means that masked visual tokens in the later frames tend have much lower loss, impeding effective video modeling. We attribute this videos' spatial information redundancy, as spatial information leakage allows the model to easily predict the masked token by attending unmasked ones in previous frames. To address this, we propose Autoregressive Discrete Diffusion Forcing (AR-DF). The core of AR-DF training involves temporal tube masking Tong et al. 2022), which repeats frame-level random mask pattern across the temporal axis. Such a design prevents masked tokens in later frames from learning ones. The core of AR-DF inference is the strategic inference-ting time masks that align with partial observation of history during training, enabling inference without degraded frame quality and motion.
mehortcut-coyyinnn previous
Collectively, we propose Lumos-1, 11, model building on Ilama Touvron et al. (2023b) and discrete diffusion to achieve AR video generation. Through a stage-wise training with GPU frieelyy techniques, we pre-train Lumos-1 from scratch on 60 million images and 10 million videos using only 48 GPUs, achieving performance comparable to EMU3 on GenEval, COSMOS- Video2World on VBench-I2V, and OpenSoraPlan on VBench-T2V.
Our contributions can be summarized as: 1) We propose Lumos-1, a pure LLM-based unified architecture with discrete diffusion for AR video generation. 2) We identify that although being useful. naive 3D RoPE exhibits imbalanced frequency spectra. Therefore, we propose EM-RoPE with more distributed frequency allocations and principled scaling, to achieve better dependency modeling and modality balancing. 3) We identify the loss imbalance issue during mask prediction training, and propose AR-DF to enforce training with temporal modeling and introduce inference-time masks avoid quality degradation. 4) We validate the efficacy of the above designs through extensive experiments. Lumos-1 achieves competitive results on T2I, 12V and T2V tasks, while using limited training computing, data and a discrete tokenizer.
2

---

Published as a conference paper at ICLR 2026
Index: 5,
M-RoPE
Channel
Token sequence
Text token
Zero position
Text tokens Visual tokens
Visual token
Scheme
Channel
MM-RoPE
Channel
Global position
Text token
Text token
Visual token
Visual token
position
Scheme
Channel
Meta MM-RoPE
H position
Text token
Arrangement
Visual token
of channels
W
W position
(a)
(b)
Figure I: (a) Initial exploration of 3D RoPE in autoregressive video generation. The sequence is comprised of text tokens and visual tokens (default arrangement in this paper), with global indices starting from 0. Two schemes are exemplified using one text token and one visual token. In Scheme and temporal, height and width positions start from O. (b) Details of MM-RoPE compared to M-RoPE. The figure illustrates the distributed channel allocation of MM-RoPE, Temporal, height and width positions are indexed after text tokens.
RELATED WORK
Autoregressive video generation. Autoregressive video synthesis follows a trajectory similar to images (from a dedicated architecture unified one like LLMs), but the substantial computational overhead encouraged researchers to explore different granularities of autoregression. At the macro level, methods such as CausVid Yin et al. (2024). Pyramidal Flow Jin al. (2024), and MAGI- 1 [Sand-Al (2025) generate video clips by recursively invoking diffusion models. A second line of work adopts a hybrid AR-diffusion strategy that couples pre-trained diffusion backbone with an external autoregressive planner, as exemplified by ARI.N [.i et al. (2024b) and Mardini Liu et al. (2024c). Our focus, however, is on micro AR approaches Gu et al. (2025); Zhou et al. (025);; Yu et al. (2023a b): Villegas et al. )022);; Kondratyuk et al. (2023); Wang et al. (202cc); Deng et al. (2024); Wang et al. (2024d), which treat the entire spatiotemporal token sequence as a single context and generate videos end-to-end with one autoregressive transformer (e.g., Phenaki Villegas al. (2022). VideoPoet Kondratyuk et al. (2023), Loong Wang et al. (2024d), EMU3 Wang et al. (2024c). NOVA Deng al. (2024)). Building on this paradigm, systematically study the of RoPE, and further introduce AR-DE, which ensures efficiency and is compatible with intra-frame bidirectionality and inter-frame temporal causality, enabling effective training and inference with LLM architectures.
RoPE for vision-language data. RoPE was first proven effective for LLMs Su et al. 2024))
Touvron al. (2023)); Huang et al. (2025a) and later adopted in DiTs Peebles & Xie (2023) to
inject spatial priors. However, its potential for AR video generation is heavily underexplored. Related works, including M-RoPE Wang et al. (2024b). U-RoPE Tang et al., RoPE-2D Agrawal et al..l.22000 (2024), VoPE Chen et al. (2024b). TAD-RoPE Gao et al. (2024). Io-RoPE et al. (2025). VideoRoPE Wei et al. (2025) and HoPE Li al. (2025a), incorporated structure priors into RoPE (i.e., spatiotemporal information or its subsets) or improve the frequency allocation of RoPE for better languaee understanding. However, they bear drawbacks: 1) some of them are not compatible with the original text RoPE; 2) their frequency allocations remain suboptimal for local- global dependency modeling: 3) their scaling strategy remains suboptimal for spatiotemporal data. We propose MM-RoPE, a distributed and scaled RoPE technique that improves AR video synthesis while remaining plug-and-play for unified models.
3 LUMOS-1
3.1 SPATIOTEMPORAL CORRELATION INJECTION VIA MM-RoPE
Preliminaries of 3D ROPE. RoPE Su al. (2024) aims to encode the absolute position with rotation matrix while incorporating the explicit relative position dependency in the attention mechanism. If we denote (zm, m) and fu (2 n) as query and key features encoding positions m and n, the attention calculation in RoPE can be rewritten as:
fg (Dm, m) fa n) = x W R W.T.,
m
(1)
where W is the projection matrix; R is the rotary matrix with pre-defined parameters (0, 20- / 112,...dd22 (d is the feature dimension and @ is the base frequency). We
3

---

Published as a conference paper at ICLR 2026
Intial Expreration of 30 ROPE Sania
30 HOPE Frequsncy Aloccaioon
Enboldinn Retorion or Unit Circle (High vs. Mis
fotetio Drsciest
10
1:10 -0000
1055
120
20
50002244 0.00002
2003
0060
4030 8030 13033
-2.50
-0.23
0.50 1.50
Charnel Indoc
(a)
(b)
(c)
Figure 2: (a) Validation loss curve of toy experiment exploring the necessity of 3D RoPE using the
0.5B model: (b) Frequency allocation in the vanilla 3D RoPE: (c) Comparison of rotary speed of the first dimension for temporal and height dimensions (high and middle frequencies) in vanilla 3D RoPE.
formulate R using base rotary matrix R with I as the frequency and T as the relative position:
Ro
0
Rege
R
0
R
R
cos 70 sin T8
sin r cos 7
Reaya
However, applying the original RoPE to modeling visual data remains suboptimal considering the spatiotemporal nature of visual tokens. Diffusion models Ho et al. (2022) improved upon this by proposing 3D RoPE that jointly injects spatiotemporal latent coordinates during attention calculation Hong et al. (2022); Kong al. (0024). If we slightly abuse the annotation by denoting x W and Wn as X,, m and Xn, we can write the attention calculation based on 3D RoPE as:
Re
X
mitale
Xnan X muhich
Xadishh + X
X
Ro..
Row.
(3)
where [t.,t,) = {0, Pe d) fhr hr] = (e fs and [w,, { Jdd denote the start and end dimension index for encoding temporal, height and width relative position; XT,,,:4, denotes the submatrix extracted from X using row indices [2t,, 2tz); other matrices are similarly defined.
Initial exploration of incorporating 3D ROPE. Our preliminary exploration starts with the intro- duction of 3D RoPE to AR video generation. We follow previous works Fan et al.] (2024); Li et al. (2024a) to utilize the validation loss to observe the effectiveness due to its strong correlation with evaluation metrics. By default, we use the cross-entropy loss (C-Loss), following standard LLM training. We compare the vanilla LLM RoPE with three schemes, as shown in Fig. [ 1) Scheme 1, which allocates the first 1/2 channels to encode the global position (i.e., the index in the global sequence) and the second 1/2 channels to encode the temporal, height and width positions with a ratio of 2 : 3 3. For text tokens, we only use the first half channels encode the global positions to ensure the language modeling capability, while for visual tokens, we only use the second half to encode 3D positions; 2) Scheme 2, which extends upon scheme 1 by leveraging the first half channels of the visual tokens to encode the global positions. 3) M-RoPE Wang et al.) (2024b), which uses all channels of the visual tokens to encode the 3D positions. From Fig a(a), we can observe that: 1) The incorporation of the spatiotemporal correlation in RoPE can significantly improve the performance of the model fitting data by comparing vanilla RoPE and Scheme 1. 2) The injection of the raster-scan order position information to visual tokens (i.e., the global position in Scheme 2 of Fig, D [)) can degrade the performance. 3) A comprehensive channel utilization (M-RoPE) is better than partial channel utilization (Scheme 1). Therefore, it is promising to inject such priors in Lumos-1.
Peering into 3D RoPE and identifying its limitations. Although 3D ROPE proves effective in practice, its design remains suboptimal. In Fig. 2[bb, we visualize how the frequencies are allocated to model temporal, height, and width dimensions. We observe that the temporal channels dominate large frequency spectrum, whereas the height and width channels are relegated to near-zero frequencies. For the sine function, the relative positions 7 (when T >= 0) should not exceed one period to avoid ambiguity since radians beyond 2TT start repeating patterns in the function. Beyond this range, the model cannot distinguish fine- -nn-grdined positional differences since rotary embeddings of this channel lose uniqueness. For the low-indexed channels, the embeddings rotate significantly faster than
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 AR-DF Training Procedure
Algorithm 2 AR-DF Inference Procedure
Require: viden dataset D. training mask ratio Pirus generative model Gg., number latent frames T
Require: text prompt X,. trained model Gg. inference mask ratio Piv. mumber of latent frames number generation steps Nytepa. number of tokens lotent frame y. KV cacbe C = R. generated latent list V = Ø
1: for each video X E D do
2: X, X. tokenize text. frames
1: Initialize text cache:
3: Sumple mask pattern for all frames:
2: Generate text causal mask Attr for Xp
4: P ~ Uniform( (eftormm[peeea [Pera: l)):
3: C C (X, Attn Mask
p Store cache for the prompt
5:
Bernoulli(1
p]
4: Sample cache mask:
6:
for t- = 1 -1--7 T do
> Apply M to all frames
6: for Min/ ~ Bernoulli(1 Pin.))
7:
x = M Φ x
+ (1
M) O
[MASK]
7: Initialize all tokens in x as [MASK]
8:
end for
8: Generate temporal causal mask Attn Mask for (Xp. x(())
9:
Form X pXp. X
9:
Generate 1-th frame x! and append it to
10:
Generate attention mask Mask for X
11:
X +- G,(X, Attr Mask)
10:
Cache partial observation the generated frame:
11:
x Manf x + Min5) @ [MASK]
12:
Compute loss L(X, M)
13:
Backprop & update
12: C Gx Attn C) Store for t-th frame
end For
13:
end for
Figure 3: Left: Training algorithm with AR-DF, For simplicity, this only includes training on vidcos. Right: Inference algorithm with AR-DE. A more detailed version for inference is included in the Appendix.
high-indexed channels (as shown in Fig. [c). leading to accelerated aliasing and loss of embedding uniqueness. For the high-indexed channels, the embeddings rotate so slowly that they lack sufficient resolution to model subtle local changes. Finally, while height and width are symmetrically important, they occupy disproportionately small and different segments of the overall frequency spectrum, reducing their capacity to capture spatial details effectively.
MM-RoPE: a distributed and scaled 3D RoPE mechanism. To elegantly solve the aforementioned limitation, we propose MM-RoPE, distributed 3D RoPE mechanism. Compared with M-RoPE Wang et al. (2024b) that is widely adopted in
models, one core idea of MM-RoPE is to encode relative positions across comprehensive frequency spectra for all 3D information. As illustrated in Fig. [b). the RoPE for text tokens in MM-RoPE follows the standard design of an LLM,, whereas the RoPE for visual tokens comprised of multiple meta MM-RoPE components. Within each meta MM-RoPE, we keep the ratio of 3D information identical to 3D RoPE (i.e., 2: : 3), while minimizing the overall dimensionality to maintain a more distributed design. Concretely, we first allocate channels for temporal modeling, then symmetrically interleave height and width channels to model spatial information. We can formulate the attention calculation of the first meta AM-RoPE as:
X
m,
Ro O
Roz.rs
XT Xn.022+
m,,24
Rog.rn 0
Ros.T
X
Rog.on
0
ReTor O
0
x m
X + M,,8:8
Xn,0.8
Reg.Tw
Røg.rw
where each meta MMPRoPE component comprises 16 channels; additional components are defined analogously and collectively form the RoPE strategy for visual tokens.
Moreover, for model that jointly processes text and visual tokens, the interplay between two modalities is significant to ensure the vision- language alignment. However, the range of posi- tions for representing texts visual data tends to differ. Contemporary visual generation systems are typically trained with extremely long and descriptive captions Betker et al. (2023): Team (2025), despite the low latent resolution of visual data (e.g., video of resolution 448 x 256 x 25
Scaled 3D Position
Atfention Mark
Text
how hone
de past jonn
[b)
Figure 4: (a) Details scaled 3D positions MM- RoPE. (b) Temporal causal mask used in 1. becomes 56 x 32 x 7 after a 8 x 4 compression). To balance the two modalities, we propose scaling the 3D positions to ensure a balanced learning. Specifically, we empirically scale the latent 3D positions to the RGB space by multiplying the compression ratio, as shown in Fig. 4 a). This simple scaling operation, from another perspective, improves the RoPE resolution for visual tokens by slightly accelerating the rotation speed. Experiments in the experiment section demonstrate its efficacy, thereby showing the importance of balancing the two modalities from the RoPE perspective.
5

---

Published as a conference paper at ICLR 2026
We acknowledge, however, that given the autoregressive generation nature of videos, this scaling may not be the optimal solution. More advanced and sophisticated solutions are left for future work.
3.2 AUTOREGRESSIVE DISCRETE DIFFUSION FORCING
The most naive generation paradigm (i.e next-token prediction) suffers from incompatibility with the nature of videos and low generation efficiency, rendering it impractical for autoregressive visual generation. In this work, we resort to discrete diffusion Xie et al. (2024); [Swerdlow et al. (2025) to generate visual content, together with spatially bidirectional and temporally causal token dependency, However, due to the autoregressive nature of Lumos-l. naive random masking (i.e., a globally random mask) or temporally independent masking (i.e., diffusion forcing Chen et al. (2024a); [Song et al. (2025) both lead to significant loss imbalance, ie., visual tokens in the later frames tend to have much lower loss. Since the task difficulty of predieting frames with ample history frame context considerably easier than predicting the first image given text prompt, the model would lean towards optimizing simpler tasks, leading to a degradation of temporal learning.
Training scheme. To resolve this issue, we build upon the basic nature of videos-spatial information redundancy. The core issue of imbalanced loss during training stems from spatial information leakage. It is worth noting that the original video diffusion transformer [Song al. (2025) that adopts diffusion forcing does not encounter this issue due to its usage of bidirectional dependency. Facing this challenge, we introduce Autoregressive Diserete Diffusion Forcing (AR-DF), which adopts temporal tube masking during the training of AR video generation. Concretely, for every video, we randomly generate mask pattern for the first frame and then apply this mask pattern repeatedly later frames in this video. If we denote the multi-modal token sequence X composed of text tokens X, and visual tokens X, y and sample a mask ratio p. the training masking strategy in AR-DF can be formulated as:
M, ~ Bernoulli( - p) fori= 1o.i=,....,.. N₁
(5)
X
M X + (1-M) @ [MAK], t= L.....,,
(6)
X x X.O, _
X!
(7)
where Ny and T denote the number of tokens in a latent frame and the number of latent frames in X,; x denotes the visual tokens of the 1-th frame: X denotes the masked multi-modal token sequence prepared for training: @ indicates Hadamard multiplication; M denotes the mask pattern; [MASK] denotes the mask token. After the preparation of the token sequence, it is fed into the model for processing. To ensure consistency with native LLMs and temporal causality in videos, we adopt temporal causal mask AttnM ask for attention processing, as shown in Fig. 4b) To train the model, we use cross-entropy loss and compute the loss only on masked tokens, denoted as L(X, X, MM) and X is the token sequence after model processing. The algorithm is formalized in Algorithm |
Inference scheme. After training with AR-DF, the most naive inference scheme (i.e., autoregressively generating video frames) would result in significant frame quality and motion degradation. We observe that this is caused by inconsistent inference with training. During training, later frames consistently have partial observation of history frames, while the inference stage does not align with this observation pattern. Therefore, given a caption, we first generate the first frame by running multiple steps and then randomly replace a pre-defined ratio pin/ of tokens with the [MASK] token for the generated image. We infer the model with this partially observed image and cache the Keys and Values of this image for swift inference. This process is repeated until the entire video is generated. The algorithm is formalized in Algorithm 2
3.3 IMPLEMENTATION
Architecture. Lumos-11 1's architecture follows Llama Touvron et al. (202b)); Grattafiori et al. (2024). To stabilize training, we integrate QK-Norm following Chameleon Team (2024). We have models of three scales (O.5B, IB and 3B), whose architectural details are placed in the Appendix.
Tokenizers. We adopt the discrete Cosmos Tokenizer Agarwal et al. (2025) that achieves spatiotempo- ral compression rates of 8 8 4. For text tokens, retain Chameleon text tokenizer. Lumos-1's total codebook size is 129,536, partitioned into 65,536 text tokens and 64,000 visual tokens.
Stage-wise training. Due the autoregressive nature of Lumos-1, the training of video generation can be categorized into training two capabilities: 1) text-to-image and 2) image/images to video. Although the incorporation of AR-DF training substantially ameliorates the imbalanced learning

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison on GenEval. The "#Params" column shows the parameter counts of the generative model and the language encoder for a fair comparison to unified models following Zhou et al. (2024). Gen Rep denotes the generation representation type: continuous or discrete. denotes results without inference-tine scaling. denotes supervised fine-tuning small-scale data.
Model
#Params
Vimages
Gen Rep
Overall!
Single Obj.
Two Obj.
Counting
Colors
Pusition
Attr. Bind
Difftsion models
SD
v1.5 Roohbshh ccal,
2002
0.8 a 0.18
2B
Costatuoss
0.43
1.97
0.38
0.35
11.76
11.44
01:06
SD
XL Peast 2009
2.08 + 0.88
Consatoss
0.55
0.98
0.74
0.39
0.25
1.335
0.23
SD
Fier 2004
8.28 2.8R
Contimos
0.68
0.98
0.54
0.66
0.74
0.40
0.43
DALL-Z
5023
4.28 + 1.08
650M
Contitous
0.52
0.94
0.56
0.49
0.77
0.30
0.19
SANA
168 2.60
SOM
Coniinooss
0.66
R 7029
125+
Continooss
0.67
0.99
0.85
0.75
0.77
0.20
0.42
18 +0443
210M
Discrete
0.54
0.99
0.42
0.30
0.22
Muddi 2XA5
10 +040
50M
Discrete
0.61
0.58
0.54
0.32
0.19
0.41
UniDit
1.48
0.42
0.92
0.19
Discrate
0.05
0197
0.50
MMaaba 2015
0.61
Autoregmesive models
SEED-X0ruaal
170
0.69
Continuoss
0.63
Fa 2354
10.58 4.7B
Costiuous
0.69
0.96
0.51
Caniiuous
0.76
0.58
0.52
0.62
U Mann 2024
0.88
60M
Discrete
0.32
0.1
0.21
0.58
0.07
0.04
Show-d X2
RU733
1,36
1B
Discrete
0.68
0.98
0.80
0.66
0.84
0.31
0.50
Chamoban Tien M74
34R
Discrete
0.39
Luman mGr 1 2e
2024h
70
1.4B
Discrete
0.56
0.77
0.27
0.32
Lamiso-nfiP11 217 Am
10
0255
78
Disercte
0.73
1.00
0.87
0.49
0.85
11.52
0.62
EMU 2MU al
2029
$B
Discrete
0.66
0.99
0.81
0.42
0.50
0.49
0:45
Lumos-: (352 x 3521
1.50
80M
Discrete
0.601
0.559
0,733
0.375
0.774
0,365
0.400
Lumos-1 (352 x 3521
3.66
80M
Discrete
0.664
0.955
0.806
0.463
0.506
0.483
0.475
Tumos-" (512 x 512V
1.58
50M
Discrete
0.725
0.984
0.869
0.519
0.862
0.558
0.558
Lumos-1 (512 512y'
3.6
Discrete
0.791
0.991
0.856
0.678
0.840
0.683
0.688
Table 3: Performance comparison
VBench-12V benchmark. We list partial metrics duc to space limits.
Midel
#Params
Viideox|
| Gen Rep
Total:
12V Score Quality Score [2VV Sob. E2V Back. Soh. Cons. Back. Coms.
Img. Quality
Dillusion models
VideoCrafier P2 Ctal
2023a
2,68
20M
Comiinuus
$2.57
85.31
78.84
91.17
91.31
97.85 95.27
98.79 58.28
71.64
Consigl Kes 2m A00ele
1.3 0.30
10M
Comtiuras
$4.07
91.91
76.22
95.81
65.05
56.92
SEINE
0.96 0.11
10M
Comiinuas
64.88
90.39
72:37
95.57
96.80
94.2
97.26
70.97
DIVGeFFXT Zhmn
14B 1.08
35M
Comiiuuas
85-28
90:11
78.44
95.48
96.81
18
97.00
89: 14
CagVidosX Yang
SAR
4.88
Continnns
96.70
94.79
78.61
97.19
96.74
94.34
96.42
70.01
Autoregresire models
COSMOS st 2024
11B
Comiinous
VideoMARITn 200g
Comiinoos
97.85
97:20
62.34
(672
1.58
Discrete
$4.16
96.06
96.25
67.06
3,68
64.72
97.42
96.91
69.23
issue, still observe that the later task is relatively easier. Therefore, a stage-wise training scheme is mandatory to ensure successful video generation training. Our training consists of three pre- training stages and supervised fine-tuning (SFT) stage. The pre-training stages focus on learning the generation capabilities from a vast amount of data: 1) text-to-image training at 256p. 2) joint image-vider training 256p, and 3) joint fine-tuning at 384p. The final SFT stage focuses on small-scale fine-tuning on data of high quality. We use around 100k images and 20k videos. This process teaches the model to more precisely follow instructions and adhere to specific aesthetic styles. We find that both models trained on 256p or 384p data can be boosted by simple SFT.
Details on sequence formatting and GPU memory friendly implementation are placed in the Appendix.
4 EXPERIMENTS
4.1 EXPERIMENTAL DETAILS
-1aooss-1100 Datasets. To train Lumos 1, we curate a image dataset containing 60 million images and a video dataset containing 10 million videos. We preserve their original aspect ratios, with videos being clipped to 25 frames for training. To ensure fine-grained vision-language alignment Ramesh et al. (2022): Team 2025), the visual data is re-captioned using langulel models Lin et al. (2024b) to obtain long and descriptive captions.
Training, inference and model evaluation. We train the model from scratch and the basic training hyper-parameters are placed in the Appendix. During AR-DF training, the training mask ratio Ptre is set to 0.7 following Li et al. 2024a)). During AR-DF inference, classier- guidance (CFG) is utilized by default to enhance the generation quality. The inference mask ratio fin is set to 0.7 by default. The number of steps to generate one latent frame is set to Nstepe = 50 by default. When evaluating on GenEval et al. (2023) and VBench Huang et al. (2024), the guidance scale is set to 16 if not otherwise specified. Since we train our model only on detailed and descriptive captions, it is mandatory for us rewrite captions using Qwen 32B Yang et al. (002a)) when evaluating on
7

---

Published as a conference paper at ICLR 2026
Table 4: Performance comparison on VBench-T2V benchmark. We list partial metrics due to space limits. f denotes supervised fine-tuning on small-scale data.
Model
/Viduu Rep |Tatal' Quality Semantic Cure. Cons. Img. Quality Obj-
Diffasion models
ModetScope (27 Wonn atal 20050
LaB +.33
10M
Continacos
15.75
78.05
06.54
89:57
05.24
50.57
52.24
51.72
25.67
memrgnVstte YOSuELEIR 2022
1:0 0..000..00 0..01
10M
Continuuu
76,61
11.50
15.81
45. 15.300
16.97
00:01
73.26
77:14
19.91
LaVis Ware drala 40MA
258 0.58
25M
Castinasu
77.08
78.78
70.31
91:41
97.47
61.90
91.82
56.39
26.41
Open
2.7 15B
NM
Cintinens
05.63
97.19
07:36
56.31
Hot Mi
70
01:30
50.38
10
CogVideoN Ting 270100
Continoos
$1.91
77.33
90.45
03.35
$3.03
27.05
Autorsgresive models
CogVida (Bor al 2022
9B
5.4M
Crntitnaou
67.01
72.06
46.83
92.19
96,20
41,03
73.40
79.57
7,70
NOVA KUAL3
0.68 2.58
20M
Castinaous
80.12
$0.39
79.05
92.00
EMU Wing F1 10050)
Discrete
80.56
$4.09
08:43
45.32
V7.60
56. 17
Lunos-1 1672 264
1.38
10M
Discrete
70.34
78.27
08.00
96.04
96.24
26.10
90.43
77.89
25,07
Lunos-1
1672
25)
3,68
TOM
Disorne
78.32
79.53
75.51
95.51
96.50
50.06
50005
K2000
25.29
Tunos-1
1672
1.58
10M
Disoris
78.17
79
73.45
95. 16
96.0K
50.21
77.1%
Tonos-1
(672
3.68
10M
Divertts
78:50
79.81
75.21
96.78
96.50
51.92
35.57
GenEval Ghosh et al. (2023). When evaluating on VBench Huang et al. (2024), we use its official
long captions by default.
4.2 COMPARISON WITH OTHER METHODS ON VISUAL GENERATION
Text-to-image generation. We compare Lumos-1 with competitive image generation methods in Tab. [ Compared with diffusion models, we observe that Lumos-1 outperforms competitive models (e.g. SD-XL Podell et al. (2024) and FLUX Labs (024)) using continuous or discrete tokenizers by a clear margin, even if Lumos 1 only uses a discrete tokenizer. Compared with autoregressive models, we observe that Lumos 11 outperforms EMU3 Wang et al. (2024c), Fluid Fan et al. 2024) and Lumina-mGPT series Xin et al. (2025); Li et al. 2024b), while using substantially fewer training images and retaining small model. Lumos-1 achieves superior results in terms of position and attribute binding, demonstrating excellent language understanding and vision-language alignment even without textual pre-training. We also observe that with small-scale supervised fine-tuning using high-quality data, the performance can be largely boosted on GenEval, especially on challenging metries like position and attribute binding.
Image-to- video generation. Thanks to the autoregressive nature of Lumos 1, we can perform amage-to-video generation by specifying the first frame, although we did not specifically train on this task. Results are listed in Tab. 3] Lumos outperforms the popular VideoCrafter-I2V model and is on par with the leading COSMOS-VVdeo2Wott model, which uses substantially more data (100M> 10M) and training resources (10000 H100s>48 H20s), demonstrating the promising performance of Lumos-1.
Text-to-video generation. We list the comparison in Tab. 4 Although Lumos utilizes discrete tokenizers and does not rely on a heavy pre- trained text encoder, it can still be on par with diffusion models like Open- SoraPlan. Due the autoregressive nature, the video quality can be en- sured by first frame quality, enabling Lumos-3
Table 5: Comparison with different training methods for AR video generation. Results are measured on GenEval, Vhench- Overall Consistency (OC), and Vbench- Imaging Quality (IQ).
Training Methods
GenEval Vbench-OC
Vbench-IQ
Random masks
0.593
0.232
0.424
Linear decay loss Step decay loss Diffusion forcing masks
0.588
0.228
0,410
0.584
0.234
0.464
0.590
0.241
0.540
AR-DF
0.591
0.249
0.559
-1 1 to excel in object-centric metries (object class and color).
4.3 ANALYSIS AND ABLATION STUDIES
Qualitative visual comparison. We compare Lumos-1 with popular video generation methods in Fig. 5 For T2V, the visual quality of our 384p videos does not trail 512p videos from LTX- Video. In the provided case, Lumos-1 generates better natural motion (water waves) and aligns with prompts better (skier in red and waves). For 12V, Lumos-1 handles multiple-object (multiple floating hot air balloons in example 1) and fine-grained motion (subtle ripples around the shoreline in example 3) substantially better than Stable Video Diffusion Blattmann al. (023) (SVD). which generates global camera movement only. In example 2, SVD produces significant blurs, whereas Lumos-1 animates subjects smoothly. More visualizations are placed in the Appendix.
Effectiveness of temporal-tube masks during AR-DF training. In Fig. 6(a). we compare the frame-wist validation loss (frame 0, 3, 6) when using global random masks and temporal tube masks.
8

---

Published as a conference paper at ICLR 2026
Text-o-Vidao
Lumos-1 (672-384-25]
LTX-Video (760-512-25
skilled skier red
glides down sunli
snowy slope amid
alpine peoks,
building
nest from hwigs and
leaves.
beautiful coostol
spring,
lopping on sand, black
and white
Image
Lumos (672-384-25)
Stable Video Diffusion (1024-576-25)
to
-Video
Figure 5: Visual comparison of Lumos-1 with other methods on text/image-tor to- tasks.
Effectivaness of Temporal Tube Maska AR-OF Training
Effectiveness of MIM-RPPE
Impact of CPG Scale
Dertfral Scor
1100
Naliccs 13860
Score
Opfinol
CFC Scale
[a]
(b)
[c]
Figure 6: Effectiveness temporal-tube masks during AR-DF training (on videos) using the 0.5B model; (b) Effectiveness MoPR,PE; (c) Sensitivity analysis of CFG scale on GenEval using model.
VBench Metrics with Varylng
Frame
Frame 17
Frame
25
Frame
Frame 17
Frame 25
Syea Sogar
w/ AR-DF Infer eeeeeence Mask
w/o
uPt) frome quslq
Videes
Figure 7: (a) Selected VBench metrics with varying inference-time mask ratio Pinfi (b) Text-to-video visualization comparing the effect of inference-time masks. (wlo: Prnf = 0.0, w/: Pinf 0.7)
For random masks, the loss frame decreases immediately and becomes lower than that of earlier frames. This rapid fall indicates pronounced information leakage: the model can reconstruct masked token by attending to unmasked tokens in neighbouring frames instead of modelling genuine temporal dynamics, rendering the task overly easy. For temporal tube masks, frame is the hardest because has the longest context to learn from, and all pixels in the same spatial locations are masked across the temporal axis, eliminating the learning shortcut. As iterations proceed, the gap between frames narrows and eventually levels out, demonstrating that the model is learning to propagate information through time rather than copying it.
Table 6: Inference time analysis with different RoPFs and decoding strategies. Results reported on H20 with batch size to 1, Natege 50 CFG used by default.
Effect of AR-DF inference masks and sensitivity to Pin.f. AR-DF re- quires the same partial-context mask- ing at inference as during training; omitting these masks severely harms quality. Fig. ()), we observe that the "w/o inference mask" setting pro-
(1B%
RePP (18
MM-RoPB:
Tmags (648 256) Video (G48 x 236
7.kc/ 16
7.7c/ 16
7.5/1800 180s
75.1s/ 1a11//1730 173.6s
77.66 178.55
77.86 / 178.55
Telen
KY Cochte
900.0s

---

Published as a conference paper at ICLR 2026
duces visible artifacts and flickering, whereas using masks preserves coherence. In Fig. [a), select two VBench metrics (imaging quality and dynamic degree) to assess the impact of pinf. y. A broad plateau between 0.3 and 0.7 yields smooth, visually pleasing videos. When Pinf < 0.3, insufficient context causes motion and per-frame quality degradation, thus inflating dynamic degree values. When Pinf 0.7, aggressive masking disrupts temporal continuity, driving up dynamic degree values. We set Pinf to 0.7 to ensure obvious motion.
Training strategy comparison. Temporal tube masking is key in AR-DF training. In Tab. 5 we compare it with other methods, including global random masks, loss reweighting (step decay loss Wang et al. (2024d) and its modified version, linear decay loss), and diffusion forcing masks, by performing joint training on Lumos IB using 8 H20 for 10k steps. We observe that loss reweighting cannot solve the information leakage issue. Diffusion forcing can ameliorate this, but temporal tube masking in AR-DF is particularly suited for videos, thus surpassing all other methods.
Efficacy of MM-RoPE. Fig. b(b) plots the val- idation loss of the 0.5B model under four RoPE settings. Note that M-RoPE means that both designs are removed. We can observe that MM-RoPE consistently converges faster and set- tles at the lowest loss, confirming the benefit of modelling fne-grained spatiotemporal informa- tion. Although ablating either component raises
Table 7: Ablation study on MM-RoPE.
Distributed Scaled
GenEval
Vbench-OC
Vbench-1Q
0.310
0.122
0.350
0.414
0.211
0.451
0.566
0.245
0.535
0.591
0.249
0.559
Scaling
GenEval
Vbench-OC
Vbench-IQ
(1. 1.
0.566
0.245
0.535
the loss, dropping the distributed design hurts more than dropping the design of scaled posi-
(2, 4, 4,
0.583
0.248
0.545
(4, 8)
0,591
0.249
0.559
(8, 16, 16)
0.593
0,250
0.553
tion, indicating that comprehensive frequency
allocation the dominant factor. Removing both enhancements gives the slowest convergence and the highest plateau, showing that the two mechanisms are complementary for effective video generation. We quantitatively ablate on the MM-RoPE. design in Tab. 7 by performing joint training on Lumos- 1B using 8 H20 for 10k steps. We can find that the distributed design contributes significantly to the boost and speeds up the model's convergence. By varying the scaling factors, we confirm that (4,8,8) is suitable and well-justified choice for visual generation.
Inference time analysis. Tab. 6 we compare the inference speed using the vanilla ID ROPE, M-RoPE and MM- LMMRoPEE. We observe that: 1) Compared with ID RoPE, the incorporation of 3D priors introduces only 3.5%-4. 1% inference latency. 2) Compared to M- M-RoPE, MM-RoPE introduces additional latency. We also compare the speed using different decoding strategies. We observe clear efficiency boost compared with vanilla next-token prediction by using discrete diffusion (mask prediction) and KV cache.
Comparison with different RoPE designs. To validate the efficacy of MM-RoPE, we benchmark it against M-RoPE Wang et al. (2024b), Vide- oRoPE Wei al. (2025), U-RoPE Tang al. IL- RoPE Liao et al. (2025) and HoPE Li et al. (2025a). As shown Tab. 8 MM-RoPF consistently outper- forms them. This superiority stems from its unique design, which holistically incorporates a comprehen- sive frequency allocation strategy and strategic scal- ing for modality alignment and resolution enhance- ment. A detailed breakdown of each method and full analysis are provided in Appendix E.2
Table 8: Performance comparison with other types of RoPE on GenEval and VBench.
RoPE Type
GenEval
Vbench-OC
Vbench-IQ
M-RoPE U-RoPE
0.310
0.122
0.350
0.402
0.165
0.423
IL-RoPE
0.541
0.225
0.513
VideoRoPE
0.569
0.243
0.540
HoPE
0.570
0.246
0.545
MM-RoPE
0.591
0.249
0.559
Sensitivity analysis of CFG scale. We study the impact of the guidance scale on GenEval using IB
model in Fig. ((.. We find that scale values from 13 to 16 (default value) lead decent results.
CONCLUSION
In this paper, we introduce Lumos-1, which utilizes the LLM architecture for AR video generation. We propose MM-RoPE for better spatiotemporal dynamics modeling and propose AR-DF for effective training and inference considering intra-frame bidirectionality and inter-frame temporal causality. We anticipate that Lumos-t - represents significant step toward building foundational unified model.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our work introduces Lumos-1, autoregressive model for text-to-video generation. Similar to other generative models, this technology carries societal risks if misused. The primary concerns include the potential for creating deceptive misleading content for misinformation campaigns, the generation of harmful or disturbing visuals, and the potential amplification of societal biases present in the training data. To address these, we have taken several steps and advocate for further safeguards, as outlined Appendix F.3 We firmly state that Lumos -1 is currently eeeearch-oriente project. We recommend that any future public release of this technology must be preceded by rigorous safety measures, also detailed in Appendix F33
REPRODUCIBILITY STATEMENT
We committed to ensuring the reproducibility of our work. All essential details for reproducing our results are provided within the paper and the appendix. The details of architectures and their training details are presented in and Sec. 3.3 Sec. 4.1aand and Appendix Our technical contribution, MM-RoPE and AR-DF, are described with comprehensive details Sec. The evaluation protocols detailed in Sec. 4.1 We will ensure the release of our source code, pre-trained model weights, and evaluation scripts upon publication of this work. We can also provide any source code if any reviewer asks for a detailed implementation.
REFERENCES
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chat- topadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.0357. 2025.
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint 1rX0v:2410.07073, 2024.
Jinbin Bai, Tian Ye, Wei Chow, Enxin Song, Qing-Guo Chen, Xiangtai Li, Zhen Dong, Lei Zhu, and Shuicheng Yan. Meissonic: Revitalizing masked generative transformers for efficient high- resolution text-to-image synthesis. In The Thirteenth International Conference Learning Representations.
Jinze Bai, Shuai Bai, Yunfci Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Gc, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309, 16609, 2023.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science, 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv preprint arXiv:2311.15127 2023.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11315-118 1325, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William Freeman, Michael Rubinstein, et Muse: Text-to-image generation via masked generative transformers. arXiv preprint 2301:0300000704. 2023.
Boyuan Chen. Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitz- mann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. arXiv preprint arX1v:2407.01392, 2024a.
Dengsheng Chen, Jie Hu, Tiezhu Yue, and Xiaoming Wei. High-resolution image synthesis via next-token prediction. arXiv preprint 1rXiv:24/1. 14808, 2024h.
11

---

Published as a conference paper at ICLR 2026
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang. Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXhv:2310.19512. 2023a.
Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang. Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset. arXiv preprint arXiv:2505. 09568, 2025.
Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In The Twelfth International Conference on Learning Representations, 2023b.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2337.0869/) 2023.
Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. Autoregressive video generation without vector quantization. arXiv preprint arXiv:24/2.14100 2024.
Patrick Esser, Sumith Kulal, Andreas Blatmannn Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. 2024.
Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410. 13863, 2024.
Mingze Gao, Jingyu Liu, Mingda Li, Jiangtao Xie, Qingbin Liu, Bo Zhao, Xi Chen, and Hui Xiong. Te-llava: Rethinking the transfer from image to video understanding with temporal considerations. arXiv preprint arXiv:2409.0322 03206, 2024.
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song. Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprini arXiv:2404. 02244044.4396. 14396, 2024.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36: 52132 52152, 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Yuchao Gu, Weijia Mao, and Mike Zheng Shou. Long-contex autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503. 0tt.:25533.192.,,, 2025.
Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Ltx-video: Realtime video latent diffusion. arXiv preprint S001::25010000000 2024.
Yefei He, Yuanyu He, Shaoxuan He, Feng Chen, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Neighboring autoregressive modeling for efficient visual generation. arXiv preprint arXiv:2503. 10696, 2025.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint 2.X0::2204.03458, 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205. 15868, 2022.
12

---

Published as a conference paper at ICLR 2026
Hangjie Yuan DanDan Zheng Weilong Chai Jingdong Chen Kecheng Zheng Feng Zhao Hu Yu, Biao Gong. Videomar: Autoregressive video generatio with continuous tokens. Advances in neural information processing systems, 2025.
Jie Huang, Xuejing Liu, Sibo Song, Ruibing Hou, Hong Chang. Junyang Lin, and Shuai Bai. Revisit- ing multimodal positional encoding in vision-language models. arXiv preprint arXiv:2510.23095, 2025a.
Zhihao Huang, Xi Qiu, Yukuo Ma. Yifu Zhou, Chi Zhang, and Xuelong Li. Nfg: Autoregressive image generation with next-frequency prediction. arXiv preprint (rXiv:2503.07072 2025b.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-211 21818, 2024.
Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song. Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint 140.059544 2024.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: large language model for zero-shot video generation. arXiv preprint arXiv:2312. 14125, 2023.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXfv:2412.03603, 2024.
Black Forest Labs. Flux. httpss//gitthbb..om/llaaat
2024.
Haoran Li, Yingjie Qin, Baoyuan Ou, Lai Xu, and Ruiwen Xu. Hope: Hybrid of position embedding for length generalization in vision-language models. arXiv preprint arXiv:2505. 2025a.
Tianhong Li. Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406. 1838, 2024a.
Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang. Dual diffusion for unified image generation and understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 2779-2777777 2790, 2025b.
Zongyi Li, Shujie Hu, Shujie Liu, Long Zhou, Jeongsoo Choi, Lingwei Meng, Xun Guo, Jinyu Li, Hefei Ling, and Furu Wei. Arlon: Boosting diffusion transformers with autoregressive models for long video generation. arXiv preprint arXiv:2410.20502, 2024b.
Chao Liao. Liyang Liu, Xun Wang. Zhengxiong Luo, Xinyu Zhang, Wenliang Zhao. Jie Wu, Liang Li. Zhi Tian, and Weilin Huang. Mogao: An omni foundation model for interleaved multi-modal generation. arXiv preprint arXiv:25055 05475, 2025.
Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang. Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:24/2.0013// 2024a.
Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 26689-26699, 2024b.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina- mgpt: Illuminate flexible photorealistic -o-itee-mmae generation with multimodal generative pretraining, arXiv preprint wrXiv:240..06657 2024b.
13

---

Published as a conference paper at ICLR 2026
Haozhe Liu, Shikun Liu, Zijian Zhou, Mengmeng Xu, Yanping Xie, Xiao Han, Juan C Pérez, Ding Liu, Kumara Kahatapitiya, Menglin Jia, et al. Mardini: Masked autoregressive diffusion for video generation at scale. arXiv preprint arXiv:2410.20280, 2024c.
yyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018
Zhuoyan Luo, Fengyuan Shi, Yixiao Ge. Yujuu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing auto-regressive visual generation. arXiv preprint arXiv:2409.0444 04440004440.... 2024.
Xu Ma, Peize Sun, Haoyu Ma, Hao Tang, Chih-Yar Ma, Jialiang Wang, Kunpeng Li, Xiaoliang Dai, Yujun Shi, Xuan Ju, et al. Token-shuffte Towards high-resolution image generation with autoregressive models. arXiv preprint 1rXiv:250417788. 2025.
OpenAl. GPT-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27740-27744, 2022.
Yatian Pang, Peng Jin, Shuo Yang, Bin Lin, Bin Zhu, Zhenyu Tang, Liuhan Chen, Francis EH Tay, Ser-Nam Lim, Harry Yang, et al. Next patch prediction for autoregressive visual generation. arXiv preprint arXiv:2412.1533
2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 0205,-4255400 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching- Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arX1v:2410.13720, 2024.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference machine learning, pp. 8821-8831. Pmir, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125, 2022.
Sucheng Ren, Yaodong Yu, Nataniel Ruiz, Feng Wang, Alan Yuille, and Cihang Xie. M-var: Decoupled scsee-wise autoregressive modeling for high-quality image generation. arXiv preprint arXiv:2411. 10433, 2024a.
Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang-Chich Chen. Beyond next- token: Next-x prediction for autoregressive visual generation. arXiv preprint arXiv:2502.20388. 2025.
Weiming Ren, Huan Yang, Ge Zhang, Cong Wei, Xinrun Du Wenhao Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-video generation. arXiv preprint arXiv:2402.04324, 2024b.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Sand-AL. Magi- Autoregressive video generation at scale, 2025. URL htthttss///staic..m world/ http:///sstaaitiileeeseeese 1. p.fdf
Fengyuan Shi, Zhuoyan Luo, Yixiao Ge, Yujiu Yang, Ying Shan, and Limin Wang. Taming scalable visual tokenizer for autoregressive image generation. arXiv preprint arXiv:2412.02692. 2024.
14

---

Published as a conference paper at ICLR 2026
Qingyu Shi, Jinbin Bai, Zhuoran Zhao, Wenhao Chai, Kaidong Yu, Jianzong Wu, Shuangyong Song, Yunhai Tong, Xiangtai Li, Xuelong Li, et al. Muddit: Liberating generation beyond exx-to--maae with unified discrete diffusion model. arXiv preprint arXiv:2505.23666, 2025.
Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion. arXiv preprini arXi::2502.06760 2025.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 568: 127063, 2024.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06552, 2024.
Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, and Katerina Fragki- adaki. Unified multimodal discrete diffusion. arXiv preprint 25l.:25500.083333 2025. doi:
10,445550arxxi,25532255355
Anni Tang, Tianyu He, Junliang Guo, Xinle Cheng, Li Song. and Jiang Bian. Vidtok: A versatile and open-source video tokenizer. arXiv preprint arXiv:2412.13061, 2024.
Feilong Tang, Xiang An, Haolin Yang, Yin Xie, Kaicheng Yang. Ming Hu, Zheng Cheng, Xingyu Zhou, Zimin Ran, Imran Razzak, et al. Univit: Unifying image and video understanding in one vision encoder. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
Chameleon Team. Chameleon: Mixed-modee early-fusion foundation models. arXiv preprint
08XY::2005.09818, 2024.
Wan Team. Wan: Open and advanced large-scale video generative models. 2025.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404. 02900,, 2024.
Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412. 14164, 2024.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders data- efficient learners for sulf-super video pre-training. Advances in neural information processing systems, 35:10078-1009; 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 207.0023,, 2023b.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2022.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope ext-to-video technical report. arXiv preprint arXiv:2308. 06571, 2023a.
Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image-video tokenizer for visual generation. arXiv preprint arXiv:2406.09933 2024a.
Junke Wang. Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu-Gang Jiang. Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. arXiv preprint 2504 1455, 2025.
15

---

Published as a conference paper at ICLR 2026
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- language model's ' perception of the world at any resolution. arXiv preprint arX1v:2409.12111, 2024b.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang. Zhen Li, Qiying Yu, al. Emu3: Next-token prediction is all you need. arXiv preprint 24X9v:2409. / 18869, 2024c.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu. Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309. 15/03, 2023b.
Yuqing Wang, Tianwei Xiong, Daquan Zhou, Zhijie Lin, Yang Zhao, Bingyi Kang, Jiashi Feng, and Xihui Liu. Loong: Generating minute-level long videos with autoregressive language models. arXiv preprint arXiv:2410.02757, 2024d.
Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, et al. Videorope: What makes for good video rotary position embedding? arXiv preprint ar Xiv::500200577.. 2025.
Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng YU, Ligeng Zhu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, Han Cai, al. Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer. In Forty-second International Conference on Machine Learning.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
Jinheng Xie, Zhenheng Yang. and Mike Zheng Shou. Show-o2: Improved native unified multimodal models. arXiv preprint arXiv:2506. /5564, 2025.
Yi Xin, Juncheng Yan, Qi Qin, Zhen Li, Dongyang Liu, Shicheng Li, Victor Shea-Jay Huang, Yupeng Zhou, Renrui Zhang, Le Zhuo, et al. Lumina-mgpt 2.0: Stand-alone autoregressive image modeling. arXiv preprint ar 2xx7::2507.17800
2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 15X1v:2412.15115. 2024a.
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505. 15809, 2025.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang. Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint 2rXiv:2408.06072 2024b.
Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. arXiv preprint arXiv:2412.07772. 2, 2024.
Hu Yu, Hao Luo, Hangjie Yuan, Yu Rong, and Feng Zhao. Frequency autoregressive image generation with continuous tokens. arXiv preprint arXiv:2503.05305, 2025.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zinu Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich sxxttto-image generation. arXiv preprint ar arXiv:2206. 10789, 2(3):5, 2022.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang. Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14499-10469 2023a.
16

---

Published as a conference paper at ICLR 2026
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXtv:2310.057377 2023b.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chich Chen. Randomized autoregressive visual generation. arXiv preprini urXiv:2411.00770 2024.
Hangjie Yuan, Shiwei Zhang, Xiang Wang, Yujie Wei, Tao Feng, Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, and Dong Ni. Instructvideo: Instructing video diffusion models with human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6463-6474, 2024.
Lymin Zhang and Maneesh Agrawala. Packing input frame context next-frame prediction models for video generation. arXiv preprint arXiv:2504. /2626, 2025.
Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. arXiv preprint arXiv:2206.055: 05564, 2022.
Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang. Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.041 2023.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprini arXiv:2408./ 2024.
Deyu Zhou, Quan Sun, Yuang Peng, Kun Yan, Runpei Dong, Duomin Wang, Zheng Ge, Nan Duan, Xiangyu Zhang, Lionel M Ni, et al. Taming teacher forcing for masked autoregressive video generation. arXiv preprint arXiv:2501. 12389, 2025.
Xianwei Zhuang, Yuxin Xie, Yufan Deng, Liming Liang, Jinghan Ru, Yuguo Yin, and Yuexian Zou. Vargpt: Unified understanding and generation in visual autoregressive multimodal large language model, arXiv preprint arXiv:2501.12322 2025.
17

---

Published as a conference paper at ICLR 2026
APPENDIX
In this Appendix, we provide additional content organized as follows:
Appendix presents more related work and discussions:
More related work on autoregressive image generation.
The reasons for choosing and retaining the LLM architecture.
Discussion on other LLM-based visual generation architectures.
Discussion on AR-DF's relatedness to Diffusson Forcing and FramePack.
Appendix presents detailed version introducing vanilla 3D) RoPE.
Appendix presents detailed version of the AR-DF inference algorithm.
Appendix D
presents architectural and implementation details.
Model architectural details and more training details.
Detailed introduction of the token sequence formatting.
GPU memory friendly implementation.
Appendix E
presents more analysis and ablation studies.
More ablation studies, including
* The effect of the number of meta MM-RoPEs in MM-RoPE.
The effect of the scaling factors in MM-RoPE.
More analysis, including
Lumos-1's 's robustness aspect ratios.
Lumos - 1's comparison with other RoPE designs.
Lumos-1 1 training resource comparison.
More visualizations, including
Qualitative visual comparison text-to-image generation.
* More text-to-image visualizations.
* More image-to-video and text-to-video visualizations.
Appendix presents more discussions.
A road map unified models for understanding and generation.
Limitations and future work.
Potential societal impact and safeguards.
Appendix G presents disclosure of LLM usage in this paper.
A MORE RELATED WORK AND DISCUSSIONS
As a supplement to the main paper, we provide 1) more related work on autoregressive image generation, 2) the reasons for choosing LLMss 3) the comparison with L.LM-based visual generation models and 4) AR-DF's relatedness to Diffusion Forcing and FramePack.
Autoregressive image generation. Spurred by the progress of LLMs, visual generation research is rapidly shifting from diffusion paradigms Rombach et al.) (2022); Ho et al. (2020); Zhang et al.
(2022 Esser al. (2024) an autoregressive one Sun et (2024); Tian et (2024); Yu al.
(2025 Huang et al. (025b);; Ren et al. 2025): Zhou et al. (2025): Yu et al. (2024); He et al. (002)) ;
Ma al. (2025): Wang al. (2025); R Ren et al. (2024a). due to potential being integrated into unified models Team (2024); Zhuang et al. (2025) with minimal modification. Preliminary research, such as Parti Yu et al. (2022). DALL-E Ramesh et al. (202T), MaskGiT [Cang al. (2022) and Ramesh LLamaGen Sun et al. (2024), have demonstrated the efficacy of autoregressive generation using discrete tokenizers. MAR Li et al. (2024a) incorporates a diffusion-style objective into autoregressive training, thereby accommodating continuous tokens. VAR Tian et al. (2024) and FAR Yu al. (0255) reformulate the learning target as next-scale and next-frequency prediction, respectively, improving generation fidelity. Unlike them, dive into the design of the RoPE technique in visual generation.
The reasons for choosing and retaining the LLM architecture. One significant group of unified models, like Chameleon Team (0024) and EMU3 Wang al. (2024c), adopts the original L.MM
18

---

Published as a conference paper at ICLR 2026
architecture to unify understanding and generation tasks. Some hybrid architectures combining an autoregressive transformer and a diffusion head, like HalP3o-Nexx(Cherr et al. (2025). also rely on the autoregressive visual generation capabilities. Although the autoregressive LLM transformers are very capable language models since they are designed for language tasks, the visual generation results are not satisfactory due to the lack of research exploration. We align our model architecture with the LLM so that the insights obtained and techniques designed (e.g., MM-RoPE and AR-DF) in this paper could transfer smoothly to the unified models to come (but we also want mention that techniques like MM-RoPE can be transferred to other types of visual generation models). This architectural alignment also means that Lumos-1 is not just specialized generator; the addition of understanding -oriineed data to its pre-training can unlock both generation and comprehension capabilities, which a promising future work.
Discussion on other LLM-bbsed unified visual generation architectures. We briefly compare our model with representative L.LM-based unified generative models, including DALL-E] Ramesh et al.) (202I). LlamaGen Sun et al. (2024), Loong Wang et al. (2024d) and Lumina-mGPT Liu et al. (2024b) in Tab. AT to clarity the key distinctions. This comparison highlights three key advantages of Lumos- I's design: 1) Truly unified architecture: Unlike hybrid models like LlamaGen which stitch together a T5 encoder and an LLM (for visual generation), Lumos-1 uses a single, end-to-end LLM backbone. This is simpler and more elegant design. 2) Advanced positional encoding (MM-RoPE): Lumos-1 is the only model that incorporates a native 31D-aware RoPE (MM-RoPE) that is also compatible with standard text RoPE used in LLMs. This allows for principled spatiotemporal modcling for visual data, a capability absent in models using ID, 2D. or non-RoPE embeddings. 3) Efficient prediction paradigm: Crucially, Lumos-1 is the only model in this comparison leverage fast, parallclizable mase- prediction. Other models are constrained by the slow, sequential nature of next-token prediction, which becomes prohibitively inefficient for videos. These targeted architectural choices allow Lumos-1 to function as fast and versatile image-video generator while being consistent with the core LLM structure.
Table Al: Comparison of different LLM-based visual generation models. The table highlights their architecture. position embedding techniques, prediction paradigms, generated media types, and performance on the GenEval benchmark.
Model
Architecture
Pasition Embedding
Prediction Parudigm
Generated Media
GenEval
DALLE
oos PE:
Nest-token prodiction (Slow)
Inage
LamaGeen
L.M (2.98 0.8R)
Naive 2D RoPF
Next-token prodiction (Slow)
Image
Loong
LLM
ID RoPE
Next-token prediction (Slon)
Image Video
LGPP
LLM (7B)
ID RoPE
Next-token prediction (Slow)
Image
0.56
Lunes-1
(1.5))
MM RoME (distributed and scaled RoPE)
Misk prediction (Fust)
Image Video
0.725
Discussion on AR-DF's relatedness to Diffusion Foreing and FramePack. For Diffusion Forc- ing Chen et al. (2024a) and AR-DE, they are related since both of them are methods to overcome the fundamental drifting problem in video generation: A small error made early on gets passed down and magnified over time, causing the final video to "drift" far away from the intended or realistic result. The temporal dependency of the video data, to some extent, leads to the drifting problem since later frames' predictions rely previous frames. We identify that this is exacerbated by frame-wise loss imbalance since the model tends to optimize those easier tasks, encouraging later frames to rely on "copying" previous frames. The temporal tube masks in AR-DF or the per-frame independent noise in Diffusion Forcing Transformer (Song et al. (2025) both aim to ameliorate this issue by reducing the reliance of later frames on previous frames. The inference-time masks in Lumos-1 also mitigate error accumulation by dropping partially generated context. Our temporal tube masking is particularly suited for video data, as it directly targets the spatial information redundancy between frames. This can be revealed by the experiments in Tab. [5] where we compare our model with variant trained by following Diffusion Forcing Transformer. We observe that they have similar performance on image generation, while Lumos-3 performs better on video generation. For FramePack Zhang & Agrawala (2025), it compresses the history in a context with fixed length. Due to its strategic method for history compression, it focuses more overcoming the forgetting problem (i.e., later frames forget history context), which is not the focus of Lumos -1. It also proposes anti-driftin sampling to get rid of drifting. However, it relies on access future frames.
19

---

Published as a conference paper at ICLR 2026
B DETAILED PRELIMINARY OF 3D ROPE
One de facto design of contemporary LLM RoPH et (2024), whose overall aim is to encode the absolute position with rotation matrix while incorporating the explicit relative position dependency in the attention mechanism. This can be formulated as:
(fg m). fz n)) g( nn n.)
(8)
where [g m)) encodes the position m for the embedding 2'm to obtain the query feature (Ss n) is analogously defined); g En, n) is a function that defines the inner product between the query and key vectors that explicitly encodes the relative position m n. The resultant form of function f(a can be formulated as:
f(a.A) m) Ri... Wg.k g.
where W is the projection matrix; Ri the rotary matrix with pre-defined parameters {0, - i= [1, 2, .... d/2]} with d acting as the dimension of the embeddings and / acting as the base frequency. In such formulation, the attention calculation can be rewritten as:
(a m fi (Ir,, r2) = ,., W,J R WKTn.
7
m
(10)
where the detailed formulation R can be formulated using a base rotary matrix Ra with 0 as the frequency and 7 as the relative position:
R 0
RO
O
Jcos 70 sin 70
sin 70 cos T
R
RØ
(1I)
O
Reara.T
However, the application of the original RoPE to modeling visual data remains suboptimal considering the spatiotemporal correlation of visual tokens. One popular type of generative models, diffusion modelsHHo et al. 20222) 2020), improved upon this technique by proposing 3D RoPE that jointly injects spatiotemporal latent coordinates during attention calculation and demonstrates its effectiveness Hong et al. (2022) Kong et al. b0244; Team 2025). If we slightly abuse the annotation by denoting x;,, W and Wi W. as X and X,, we can write attention calculation Eq. based on RoPE as:
Ra
Ro
xm.a.
N. + Xmmsoon
X
Ro.
Ro
Re
(12) where (la, = (0, Pe d). (hs, he) (Z 16 d) and (wa, w.] d {d) denote the start and end dimension index for encoding temporal, height and width relative position; XT Mmtait. denotes the submatrix extracted from XT m using row indices [2t,, 2t.);; other matrices are similarly defined.
C DETAILED AR-DF INFERENCE ALGORITHM
In this subsection, we present the detailed version of the AR-DF inference algorithm. After the text prompt X, is encoded once and its KV pairs cached (Lines 1-3), iterate over the T latent frames (Line 6). Each frame is generated by running Nateps iterations (Lines 9-22). During the frame generation process, the model predicts a token distribution, samples tokens only positions that are still masked, gathers their confidences, and sets the scores of already-reveale tokens to +00. After adding Gumbel noise, re-mask the lowest-confidence tokens. As the generation proceeds, fewer tokens are re-masked as the running steps grow and the frame becomes sharper and clearer. The partially revealed frame is then added to the cache before the next frame is generated. Once all frames are finished, they are decoded to RGB space to form the final video.
D MORE ARCHITECTURAL AND IMPLEMENTATION DETAILS
Model architectural details and more training details. For all of our models, we adopt the Adam Loshchilov & Hutter (2088) optimizer to optimize the model, the weight decay to 0.1 and
20

---

Published as a conference paper at ICLR 2026
Algorithm 3 AR-DF Inference Procedure
Require: text prompt x. trained model Gg. inference mask ratio pinf, number of latent frames T. number of generation steps Natepe, number of tokens in latent frame Nç, v KV cache C Ø, generated latent list V 2
1: Initialize text cache:
2: Generate text causal mask AttnMask(() for X,
3: C+ (x(Xp, Altt.p..tMMassst AltnMask!!))
D. Store cache for the prompt
4: Sample cache mask:
/ Bernoulli(1 (1-pm)))
6: for to T do
> Reused for all frames
7: Initialize all tokens in X as [MASK]
8: Generate temporal causal mask Attn.ask") for (Xp, x!")
9: for n1 - Nateps do
10:
P Go (X. AttnmMss
Predicted token distribution for all tokens
11:
sampled ids < multinomial((? (P,
12:
sampled_ie ids +- where x [MASK], sampled ids, x!
13:
Q < Gather (P, sampled ids)
Select confidence score of every chosen token
14:
Q - -ettuoowlnffnity Q, x ( + [MASK]) Set +oo where the token was not masked
Mask the lowest confident (Q) tokens sampled ids with randomness:
16:
Perturb Q with gumbel noise
17:
a - cos cos 2 Natena
D Cosine schedule factor in [0, b Number of tokens to mask in this iteration
18:
U + floor(a a N2) low low_conf_pon sampled_ids siodddddsllloww ltwpiissllo_ooo_pooo x! sampled_ids
19:
SeelecLowConfideneeee U)
20:
[MASK]
21:
22:
end for
23:
Append x V
24:
Cache partial observation of the generated frame:
25:
X = Minf c x(" + (1 Man)) & [MASK]
26:
C G. Attm.Mask!),
Store cache for the 1-th frame
27:
end for
28: Decnde T latents RGB space
29: return the generated video frames
Figure C1: Detailed inference algorithm with AR-DF.
set 31 and B2 to 0.9 and 0.95. Tab. D1 presents the architectural details of three versions of Lumos- and their corresponding training details, including their training batch sizes and their learning rates. For 256p joint training, we use all 60M images and 10M videos. For 384p joint training, curate higher-quality subset from 256p training data, consisting of 8M images and 0.6M videos. Following the alternating strategy of MovieGen Polyak et al. (2024), interleave image and video batches during joint training-
Detailed token sequence formatting. The visual tokens and text tokens interleaved within sequence Liu et al. (2024b), with the text tokens specifying metadata, including the text prompt, video resolution, video fps and the number of frames in this video. With this design, train the model us- ing images and videos of varying aspect ratios without resizing them. The text tokens are formatted as "Generate video with a resolution of siddeoresolution >, consisting of < video frames > frames at < undco_fps > frames per second, according to the following prompt:'n < text pect_pprppttteeee >" The visual tokens are extracted and organized according to the following struc- tured format: "< starrslsarrr_eenn >, video catpouddrrrroooooe
Liu
>, < video__pss fps__kken >, < Trame_loken ?..... wideo video_end__ie end_loken >" < frraee_ooee tokens > contains tokens for each latent frame, which are formatted as: "< image_start, token >, h h_ token >, w __ggri__token >, < manage_contens content_tokens >, < image_end_ token >". where tokens
raster-scan visual tokens with new line token inserted after each row of visual tokens. To harmonize text and visual tokens, we utilize MM-RoPE, which encodes spatiotemporal coordinates for visual tokens and global positions for text.
<
GPU memory friendly implementation. By default, we leverage flash attention Dao 2023 to accelerate attention calculation and reduce memory overhead during training and inference of
21

---

Published as a conference paper at ICLR 2026
Table DI: Model architectural and training details. Batch sizes (*/*) are images and videos during joint training. Learning rates (*/*) are for 256p and 384p. The 0.5B version is only for fäst ablation studies.
Model
#Params
#Layers
Hidden size
#Heads
Head dim
Batch (256p)
Batch (384p)
Learning rate
Lumos-1 0.5B
0.5B
16
1024
16
64
896 128
192 02/332
5e-5 / 2.5e-5
Lumos-1 IB
1.5B
16
2048
32
64
640 / 96
192 /2
5e-5 / 2.5e-5
Lumos 1 3B
3.6B
3072
128
768
288 48
1c-4/ 5.0e-5
Effect Number MM-RoPEs
Effect Scaling Factors MM-RoPE
11.25
HMARPPE wi distibuted design
11.25
MH--RUE
MM6-t7E (emeta MaN-RPPt 1)
MH--OPE scale (2,4Al
HW-ROFF (éreea 1MV RoPE
NM ROPE scale (4.8.)
22.00
HMA-DDE (emeta NIN-RUPE
11:03
HM--uPE scale (8 (8.16.16)
KIN RoPE scate (16.32.32)
10.75
Losss Value
C-LoGS Valuee
1025
30.00
10.23
6.79
9.53
2002
2000
4230
0000
Valid Iterations
Iterationn
(a)
(b)
Figure E1: (a) Effect of the number of meta MM-RoPEs in MM-RoPE: (b) Effect of the scaling factors in MM-RoPE. Both curves are plotted using the 0.5B model trained on videos.
Lumos-1. Moreover, we observe significant amount of GPU memory consumption during the training of a model with large codebook size. To address it, eliminate the use of language-related loss (i.e., next-token prediction on texts), reducing the final logit matrix size to match only visual tokens. While text token embeddings. which map the text token index token embeddings, remain trainable, this focuses the model on video generation. This loss can be added if we target a unified model that capable of learning within the language modality. Finally, we observe substantially high GPU memory consumption during loss computation over 129 token types, which easily leads to the out-of- -f-mooyy issue. To solve this, we utilize chunked cross-entropy loss, which maintains full softmax accuracy by upcasting and calculating softmax logits one chunk of token sequences at a time. This approach significantly reduces peak memory usage. By default, we set the chunk size to 2,000.
MORE ANALYSIS AND ABLATION STUDIES E
E. MORE ABLATION STUDIES
Effect of the number of meta MM-RoPEs in MM-RoPE, MM-RoPE slices the embedding channels into several meta groups. More groups mean that one kind information (temporal, height or width) receives wider and more comprehensive frequency spectrum for modeling. Fig. ET () plots the validation loss for a 0.5B model under four settings: 1) w/o distributed design: This setting uses the previous design, which allocates the first channels for temporal modeling and channels for height and width modeling, respectively. 2) #meta MM-RoPE- /:: This setting equips a meta MM-RoPE of 64 channels, while maintaining the ratios (2 3) for modeling temporal. height and width information. This variant improves upon the previous one by enabling interleaved height and width channels, which increases the frequency spectrum range for the two spatial dimensions. 3) #meta MM-RoPE=2: This setting equips meta MM-RoPE of 32 channels. This variant improves the frequency spectrum range for temporal, height and width information compared with the previous one. 4) #meta MM-RoPE= This setting is the default design that keeps the number of channels for every meta AMM-RPPE minimal (16 channels). This means that the frequency spectrum range is most comprehensive for the temporal, height, or width dimension. These results confirm that widening the frequency spectrum for every dimension through increasing the number of meta MM-RoPES substantially improves spatiotemporal modeling and overall training effectiveness.
22

---

Published as a conference paper at ICLR 2026
Effect of the scaling factors in MM-RoPE. Fig. [E][) presents the validation loss curves of
Table E1: GenEval score across different aspect ra- tios using IB model.
varying the scaling factors for modeling tempo- ral, height and width positional information in
448 x 256 (7 352 x 352 (1 1) 256 448 (4: 7)
GenEval
0.605
0.601
0.569
MM-RoPE. Two clear trends emerge: Moving
from (1, 1, + (2, 4) + (4, 8, 8) steadily lowers the curve, but further enlarging to (8, 16, 16) or (16, 32, 32) yields no additional gain since the three curves almost overlap throughout training. Therefore, a moderate scale of (4, 8, 8) is sufficient to balance vision- ranges and fully realize the benefit of higher- resolutnesolution RoPE. while avoiding unnecessary frequency inflation. We therefore adopt (4, 8,8) 8) as the default scaling for MM-RoPE.
E.2 MORE ANALYSIS
Robustness to aspect ratios. Although the aspect ratio of training data is mostly 7 4, Tab. indicates that Lumos-1 1 IB adapts well to visual generation with different aspect ratios due to the unified codebook design.
Comparison with other RoPE de- signs. To demonstrate the superiority of MM-RoPE, we compare it with five other RoPEs, i.e., M-RoPE Wang al.
Table E2: Design comparison with other types of RoPE.
RoPE Type
Compatiable with
Comprehemsive Frequency Allocation
Strategic
RoPK
Structure
Scaling
M-RaPE
X
(2024b). VideoRoPE Wei et al. (2025). )
U-RoPE
x
U-RoPE Tang et al., RoP Liao et al.
IL-RoPE
VideoRoPE
(2025) and HoPE Li et al. (2025a). We
HoPE
X
show their design comparison in Tab. E2
MM-RoPE
We mainly compare these methods on four aspects: 1) their compatibility with text RoPE, so that models can align with contemporary LLMs; 2) the inclusion of 3D information into RoPE to enable enhanced spatiotemporal modeling: 3) the comprehensive frequency allocation that enables more balanced local-global context modeling; 4) the strategic scaling of spatiotemporal indices (or their subsets) account for different granularities of vision and text information. Based on the above analysis, MM-RoPE holistically incorporates the four properties.
Moreover, we conduct a quantitative comparison Tab. E3 using our 1B model to perform joint train- ing for 10k steps. Note that M-RoPE is our baseline, equal to eliminating the distributed design and the scaling design from MM-RoPE. U-RoPE improves upon M-RoPE by reallocating high-frequenc, com- ponents for modeling local spatial details, which is useful. IL-RoPE allocating partial channels for mod- eling spatial information with interleaved design, while it only allocates partial channels for modeling texts and visual data, hurting the performance. The concurrent work, VodeoRoPE, proposes several tech- niques for understanding tasks: low- freeureeuency temporal allocation, diagonal layout and adjustable temporal spacing. We use VideoRoPE's default hyperparameters. We observe that VideoRoPE slightly trails MM-RoPE in our task. We attribute this to the difference in frequency allocation and adjustable temporal spacing. Low-frequency temporal allocation enables more local modeling for spatial features and global modeling for temporal features. while it is important maintain both local and global modeling for spatiotemporal features, which RoPE performs better at. Moreover, adjustable temporal spacing increases the resolution for better modeling, and we believe that adjusting the hyperparameter in adjustable temporal spacing can increase its performance, which a future work for VideoRoPE. HoPE improves upon VidcoRoPE with a zero-frequenc, strategy for temporal modeling, therefore slightly outperforming VideoRoPE. It trails MM-RoPE because the frequency spectra that HoPE use for modeling spatiotemporal dependency are not as comprehensive as MM-RoPE.
Table E3: Performance comparison with other types of RoPE. Results are measured on GenEval, Vbench-Overall Consistency (OC), and Vbench- Imaging Quality (IQ).
RoPE Type GenEval
Vbench-OC
Vbench-IQ
M-RoPE
0.310
0.122
0.350
U-RoPE
0.402
0.165
0.423
IL-RoPE
0.541
0.225
0.513
VideoRoPE
0.569
0.243
0.540
HoPE
0.570
0.246
0.545
MM-RoPE
0.591
0.249
0.559
Comparison of training resources. We compiled a detailed comparison of training costs against popular text-to-image and text-to-video models. To ensure fair comparison, normalized all
23

---

Published as a conference paper at ICLR 2026
photo bench (ninlight liber through leafy branches, caslngg dloppeed shadows on banch and ground.)
photo green leneis block dog
photo orange a yellow stop sign
photo potted A photo of cheir plant through window, coslng gerls Shodows)
photo dog above
photo person an apple shoulderlength brown
left zebra
hal, weers simple, sorfvlooed drows)
EMU3 [720w720]
SIOP
SD3 Medium
[512-512]
STOP
Lumos-1 [512x512]
STOP
Figure F2: Visual comparison of Lumos-1 with EMU3 and Stabel Diffusion 3 (Medium) on the text-to-image task. We place the simple version of the prompts above the images for reference. Fragments of the detailed prompt are placed in parentheses, with critical attributes highlighted in red. All images are generated using the detailed prompts for all models.
training times 8xA 100 GPU-Days based on their respective FP16 Tensor Core performance. The We can observe from the table that, compared with text-to-image
results are placed in Tab. E4 diffusion models or larger-scale AR models, Lumos-1 is more compute--fficient in terms of training and obtains better performance on GenEval. Compared with Open-Sora Plan v1.2, Lumos-] achieves similar performance, while reducing the training cost significantly. Unlike diffusion models, which rely on powerful, external text encoder, Lumos-1 learns language understanding from scratch. Its high efficiency, even with this added learning burden, highlights the effectiveness of our discrete tokenization and MM-RoPE alignment strategy.
E.3 MORE VISUALIZATIONS
Qualitative visual comparison on text-to- image generation. We compare Lumos-1 with popular text-to-image generation methods in Fig. B2 All models are configured to their de- fault inference settings and use the same version of detailed prompts to ensure quality generation.
Table E4: Comparison of training efficiency for text- to-image and text-to-video models.
Time
Method
Params
(8xA100 GPU-Days)
GenEval
SD-15
0.9B 0.1B
781.2
0.43
SD-2.1
0.98
0.38
1041.6
0.50
We can observe that: 1) Although EMU3 is sig- nificantly larger than Lumos - Lumos - gen- erates images that reproduces fine prompt details more faithfully, such as the dappled shadows on the bench (example 1) and the correct "STOP" text and the yellow color on the road sign (ex- ample 3). EMU-3 sometimes drops or warps these details and exhibits occasional anatomical
D
1.08
5208.9
PixAnta
0.68 4.7B
94.1
0.48
LGP
7B
4488.8
0.56
Lumos-1
L.58
25.9
0.725
Time
Method
Params
(8xA100 GPU-Day))
VBench
Open-Sora Plan v1.2
2.7B
138
592
75.98
Lumos-1
1.58
85.4
78.17
distortions, such the distorted zebra (example 5) and malformed fingers on the woman holding an apple (example 7). 2) Stable Diffusion 3 offers slightly crisper textures, which is expected from its continuous tokenizer, but it frequently ignores descriptive constraints, such the dappled shadows (example 1) and gentle shadows (example 4), and the incorrect dress color (example 7). By con- trast, Lumos- - meets these constraints while maintaining a convincing overall composition. These examples illustrate that Lumos-1 delivers strong lasionnlanguage alignment compared with both diffusion and autoregressive baselines, while remaining competitive in visual quality.
More text-to-image visualizations. Fig. E presents more results generated by Lumos-1 using descriptive captions. Due to the unified codebook, can train Lumos-1 using the original
24

---

Published as a conference paper at ICLR 2026
photo
a photo of a bird
photo of train
photo person
photo of bear
a photo of dog a photo dining table
photo of three handbage
photo four vases
photo four dogs
poned plant bockpock
photo of left a
photo truck of refrigerator
of photo zebro lof elephart
photo
two beds
Figure E3: More text-to-image results generated by Lumos-1. All images generated using detailed prompts, but we only place short prompts here space limits. The resolutions for three rows of images are 384 x 672, 672 x 384 and 512 x 512.
resolution of the visual data, enabling generation with different aspect ratios such 7, 7 and 1 1, although we do not have a significant amount of data with aspect ratio 4 : 7. Even though Lumos is trained from scratch, it transforms lengthy descriptions into complex scenes that aligned faithfully with the prompts. This can be exemplified by the second row of images, where these images clearly reflect the prompts:
"a photo of three handbags" contains the description: To the left, classic black leather handbag with a structured design and gold-tone hardware catches the eye. the center, a vibrant red quilted handbag with chain strap adds pop color and touch of luxury. On the right, a casual beige canvas tote bag with simple, clean lines offers a more relaxed aesthetic.
"a photo of four vases" contains the description: The first vase the left is tall, cylindrical glass vase with a slight teal tint, showcasing clear transparency and modern design, Next it is vintage ceramic vase with off-white base adorned with hand-painted blue floral patterns, adding a touch of traditional elegance. The third vase is a short, squat terracotta pot with rustic, earthy texture and visible firing marks, evoking a natural, bohemian feel. Finally, a sleek, metallic silver vase with a geometric shape completes the set, providing contemporary contrast.
"a photo of four dogs" contains the description: To the left is golden retriever, its fur glowing warmly in the sunlight, with a gentle, friendly gaze. Next to it is a sleek black Labrador, its coat shining with health, head tilted slightly as if listening intently. In the center, a small, energetic Jack Russell terrier bounces with excitement, its white and brown fur clearly define. On the right, a large German Shepherd sits calmly, its regal posture and dark, expressive eyes adding dignified presence to the group.
These qualitative results underscore the strong vaniun-e alignment achieved by Lumos-1
More amage-to-vided and text-to-video visualizations. Fig. E4 presents more image-to- video results
and Fig. ES presents more text-to-video results generated by Lumos-1. The prompts span a wide semantic and dynamical range, containing close-up animal/human scenes (example 4, 5 in Fig. ES 2) fast human and vehicle motion (example 1, 4 in Fig. E4 and example 1, 2 in Fig. E5). 3) fludd
dynamics (example 1, 5 in Fig. [4 and example in Fig. E5 and 4) complex multi-object collective
25

---

Published as a conference paper at ICLR 2026
mon flyboards above a lake, propelled by jet ski, against serene backdrop.
blue and white airplane descends under clear sky.
group of fish swimming over coral reef.
red bus driving down snowy street night.
Two people in lake with mountains in the background.
Figure E4: More image-to-video results generated by Lumos-1. images are generated using detailed prompts, but we only place short prompts here due to space limits. The results encompass videos of resolution 672 x 384 and 768 x 320.
A motorcyclist rides through lush countryside with green fields and vibrant red flowers.
A surfer experfly navigates a wave on vibrant beach with builling cocalline,
school silver fish creates mesmerizing patterns deep blue ocean scone.
man enjoys food at a lively market filled with stalls and vibrant signs.
yellow duckling pecks dirt, surrounded by gross, sunlight.
Figure E5: More text-to-video results generated by Lumos-1. All images generated using detailed prompts, but we only place short prompts here due to space limits. The resolution 384.
motion (example in Fig. E4 and example 3 in Fig. ES). The results demonstrate that Lumos-1 11 can 1) handle videos of different aspect ratios and 2) generate single-subject and multi-subject motion that is temporally coherent and physically plausible.
26

---

Published as a conference paper at ICLR 2026
F DISCUSSIONS
F.1 A ROAD MAP TO UNIFIED MODELS FOR UNDERSTANDING AND GENERATION
Building upon our current work, a complete unified model can be achieved through a principled, staged approach:
Stage 1: Foundational Model Selection and Codebook Unification. The starting point is base model with strong language capabilities. There are two primary paths: 1) Path A (Pragmatic Start): Initialize with a pre-trainee VLM that already possesses strong visual and textual understanding. The main task is then integrate our generative capabilities by extending its vocabulary with a discrete visual codebook and fine-tuning with our proposed AR-DF and MM-RoPE on generation tasks. 2) Path B (Purist, End-to-End): Start with a pure LLM. This requires developing -par-pue discrete visual tokenizer-one that produces tokens rich enough for both high-fidelit reconstruction (for generation) and high- level semantics (for understanding). This is challenging but crucial research direction for true end-to-end unification.
Stage 2: Multi-Tass Data Curation. A successful unified model requires a diverse, high- quality dataset covering all target modalities and tasks. This involves curating a mixture of: 1) Language-only Data: To maintain and enhance the model's 's core reasoning and prevent catastrophic forgetting. 2) Image/Video-
Pairs: For training both understanding (e.g., captioning, VQA) and generation (T2I, T2V) tasks. 3) Interleaved Multi-modal Documents: To teach the model to seamlessly process and generate complex sequences of text and visuals.
Stage 3: Joint Multi- and Staged Training. The final step is to train the model to be a versatile multi-task model. This should be done in stages: 1) Understanding Foundation (if using Path B): First, train the LLM on understanding tasks (e.g.. VQA, captioning) to establish a strong vision- laiion-a alignment baseline. 2) Joint Generation and Understanding Training: In the main stage, train the model jointly on mix of understanding and generation objectives. Our proposed AR-DF and MM-RoPE would serve as the core engine for all generative and understanding tasks. By sharing the same transformer backbone, the model is forced to develop a rich, shared representation space where concepts learned from understanding tasks can positively transfer to generation, and vice versa.
F.2 LIMITATIONS AND FUTURE WORK DISCUSSIONS
We recognize that Lumos 1, as an initial endeavor this area, comes with its limitations. Most prominently, its training corpus (60 million images and 10 million videos) is modest compared with datasets used by recent foundation models Team (202)) Labs (2024), which usually contain billions :
of samples. Consequently, Lumos-1 1 human actions or highly intricate scene dynamics. Considering this, our immediate research plan therefore includes scaling along three axes: 1) Data volume and diversity expanding both image and video coverage to narrow the generalisation gap. 2) Model capacity- training larger backbones while retaining the MM-RoPE and AR-DF designs. 3) Multimodal knowledge infusion initializing with strong vasion-language models or co-training with visual understanding tasks so the generator can better ground its outputs in world knowledge.
meder-gene in scenarios that require fine-grained
F.3 POTENTIAL. SOCIETAL IMPACTS AND SAFEGUARDS
As the pioneering efforts autoregressive visual generation, Lumos-1 can be a significant step towards large-scale unified model for general visual understanding and generation. Through substantially large-scale training. the obtained model could stimulate intelligence that exceeds existing models trained on unimodal data. The obtained model could serve a foundation for various downstream applications, thereby reducing the carbon and economic cost that society spends on training various specialized models.
At the same time, unrestricted deployment carries non-trivial risks: the model might be used to fabricate deceptive media, produce disallowed disturbing visuals, amplify harmful biases present
27

---

Published as a conference paper at ICLR 2026
in the training data. To mitigate these concerns, recommend the following safeguards before any real-world release: 1) Aligmment runing apply preference- or instruction-tumie so that outputs respect human aesthetic and moral preferences Yuan et al. (2024). 2) Dual-stage filtering - ii) prompt filters that refuse generation requests involving illicit content, and ii) post-generation classifiers that block watermark unsafe outputs. 3) Red-team evaluation continuous adversarial testing to uncover failure modes in new domains or under distribution shift. Lumos 1 is rescarch-oriented work, and any broader deployment should proceed only with thorough oversight and evaluation to ensure responsible and ethical use.
G DISCLOSURE OF LLM USAGE
We utilized a LLM to assist in the preparation of this manuscript. The primary role of the LLM was to improve the language, clarity, and readability of our writing. All intellectual contributions, including the research ideation, experimental design, data analysis, and the core arguments presented in this paper, are entirely our own. The L.M was used solely as writing-enhancemenn tool and did not contribute to the scientific aspects of the work.
28


Reviews:
[{"review_id": "IZyHJXYeFF", "reviewer": "Reviewer_gCo6", "summary": "The paper presents Lumos-1, a unified autoregressive model for video generation using a discrete diffusion process. It combines text and video vocabularies and proposes two main techniques: MM-RoPE for better spatiotemporal position encoding, and AR-DF (Autoregressive Discrete Diffusion Forcing) to handle loss imbalance during masked diffusion training. The model is tested on T2I, I2V, and T2V tasks, showing comparable results to larger diffusion-based systems despite using fewer resources.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The study of RoPE frequency allocation is genuinely interesting. The proposed MM-RoPE makes sense — it fixes some imbalance in 3D RoPE and doesn’t really cost extra compute.\n2. The setup is efficient: 60M images and 10M videos, trained on 48 H20 GPUs.\n3. The experiments are complete, covering main benchmarks (T2I, T2V, I2V) and solid ablations.", "weaknesses": "1. The biggest weakness is the performance. Even though the paper claims efficiency, Lumos-1 doesn’t really beat diffusion or existing AR models on any benchmark. The visuals still look blurry and the motion often feels weird or distorted. It’s not convincing that this setup improves things beyond simplicity.\n2. The experimental part only uses validation loss to validate different component like AR-DF and MM-RoPE. Including benchmark metrics would better demonstrate effectiveness since validation loss does not always aligns with down stream performance, even for AR models.\n3. The frequency allocation for MRoPE is actually getting more and more attention nowadays. For example, [1] proposes to use U-RoPE. A concurrent work [2] gives a much more in-depth discussion about this, including HoPE and IL-RoPE. This paper only compares with VideoRoPE, weaken its contribution.\n4. AR-DF looks like a combination of discrete diffusion and diffusion forcing, only with tube masking. But tube masking actually breaks the unidirectional dependency in AR models. This could hurt the scalability of the model.\n\n[1] UniViT: Unifying Image and Video Understanding in One Vision Encoder\n[2] Revisiting Multimodal Positional Encoding in Vision–Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VDe5OTEfh6", "reviewer": "Reviewer_NXEV", "summary": "The paper presents Lumos-1, a unified autoregressive video generation model built upon a LLM architecture, designed to address key limitations of existing autoregressive video generators. To overcome decoding inefficiency, the authors integrate masked diffusion with causal attention. They further introduce MM‑RoPE that preserves standard 1D RoPE for text while effectively capturing spatiotemporal correlations in video. Additionally, the model mitigates error accumulation through a specialized form of diffusion forcing that shares both timestep and mask perturbation across frames. Lumos-1 achieves performance comparable to models such as Emu3, Cosmos1, and OpenSoraPlan v1.3.", "questions": "- See weaknesses.\n- In short, the work presents Lumos-1 model for autoregressive video generation,  the contributions of this are work are not sufficiently substantiated by the experimental results: \n1. A unified architecture for visual generation with strong performance.\n\n2. Asynchronous AR-DF for mitigating the training-inference inconsistency.\n\n3. The difference between IL-RoPE and MM-RoPE on interleaved frequency allocation.\n\nIf the authors provide stronger model performance or a more thorough analysis of the weaknesses, I would be happy to raise my score.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Lumos-1 demonstrates that a pure LLM architecture can be employed for autoregressive video generation.\n- Lumos-1 addresses the frequency limitations of the original M-RoPE.\n- Lumos-1 proposes AR-DF scheme to mitigate the training-inference inconsistency.", "weaknesses": "- **Reconsidering the Masking Strategy of AR-DF.** Diffusion forcing[1] models the conditional probabilities between frames by adding independent noise to each frame. However, due to the issue of information leakage in mask-based approaches, AR-DF can only apply the same noise to all frames, which deviates from the core idea of diffusion forcing. In my view, AR-DF is more like a form of data augmentation or an exploration of better masking strategies for video generation. I believe the authors need to reconsider and clarify their masking strategy.\n- **Regarding the novelty of MM-RoPE.** I consider that interleaved frequency allocation (IL-RoPE) was first introduced in Mogao[2], the authors should cite this work and provide a comparative analysis between MM-RoPE and IL-RoPE.\n- **Suboptimal performance of unified architechture.** Although Lumos-1 employs architecture without a text encoder, its text-to-image and text-to-video performance remains suboptimal.  The authors could substantially improve performance to better align with the promise of its unified architecture. Moreover, some of the reported baselines appear outdated or unfair, for instance, Show-o[3]’s GenEval score is 0.68 without rewritting, and NOVA[4] achieves a VBench score of 80.12 using long prompts.\n\n- [1] Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. arXiv preprint arXiv:2407.01392, 2024a.\n- [2] Chao Liao, Liyang Liu, Xun Wang, Zhengxiong Luo, Xinyu Zhang, Wenliang Zhao, Jie Wu, Liang Li, Zhi Tian, and Weilin Huang. Mogao: An omni foundation model for interleaved multi-modal generation. arXiv preprint    arXiv:2505.05472, 2025. \n- [3] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. In ICLR, 2025.\n- [4] Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. Autoregressive video generation without vector quantization. In ICLR, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ARGDoKvqU7", "reviewer": "Reviewer_uiBr", "summary": "This paper introduces Lumos-1, an autoregressive video generator based on an LLM architecture with minimal modifications, which aims to create a unified multimodal model. A key achievement is the demonstration of video generation using LLM architecture, which paves the way for a truly unified foundation model and eliminates the need for external text encoders. \n\nAdditionally, Lumos-1 incorporates two innovations:\n1.  It proposes MM-ROPE to address imbalanced frequency spectrums in 3D RoPE, enhancing spatiotemporal correlation modeling via distributed channel allocation and scaled 3D positions.\n2. Lumos-1 employs autoregressive discrete diffusion forcing (AR-DF) to mitigate frame-wise loss imbalance and spatial information redundancy in autoregressive video generation training.\n\nThe paper provides detailed descriptions of the methods employed, thorough experimental evaluations against robust benchmarks, and insightful ablation studies that effectively demonstrate the merits of the innovations. However, Lumos-1 is trained from scratch despite a high structural similarity with Llama, so the model needs to learn both language and vision simultaneously, which can introduce instability and inefficiency during training. \n\nIn summary, this is an innovative work with significant community impact, and I appreciate its architectural exploration; however, I remain concerned about its training efficiency and stability, and I am curious about its performance with LLM initialization.", "questions": "1. Since the model structure is highly similar to LLM, why not initialize the parameters with pretrained Llama?\n2. Based on the experimental results, the unified architecture improves semantic consistency significantly, yet the visual quality still lags behind some baseline methods. Have you attempted to explain this trade-off or conducted a deeper analysis of its causes?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This work proposes Lumios, which demonstrates the effectiveness of video generation using LLM architecture, paving the way for a truly unified foundation model and eliminating the need for external text encoders. \n2. It proposes MM-ROPE to address imbalanced frequency spectrums in 3D RoPE, enhancing spatiotemporal correlation modeling via distributed channel allocation and scaled 3D positions.\n3. Lumos-1 employs autoregressive discrete diffusion forcing (AR-DF) to mitigate frame-wise loss imbalance and spatial information redundancy in autoregressive video generation training.\n4. Comprehensive experiments and complete ablation studies validate the advantage of MM-ROPE and AR-DF, and the effectiveness of LLM-based video generation.", "weaknesses": "1. Although structurally similar to Llama, Lumos-1 is trained from scratch, requiring simultaneous learning of language and vision, which may lead to training instability and inefficiency. The validation curve in Figure 7(a) supports concerns about the instability. Furthermore, as a foundation model, the full cost of the training, particularly in terms of time, is not disclosed, which is a cause for concern in terms of the inefficiency of the training.\n2. The paper claims minimal structural modification from LLM for a unified text-visual model, but it lacks a comprehensive structural comparison with other established unified generative models or LLM-based autoregressive models, making the claimed advantage less substantiated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6pc6uW6QCB", "reviewer": "Reviewer_J6ru", "summary": "The paper introduces Lumos-1, a unified, LLM-based autoregressive video generator that addresses the central challenge of spatiotemporal positional encoding. To align with the dual nature of video (i.e., spatial bidirectionality and temporal causality), the authors propose two key contributions: 1) MM-RoPE: a distributed, scaled 3D Rotary Position Embedding that preserves standard textual RoPE while interleaving temporal, height, and width channels in repeated \"meta\" groups to broaden the frequency spectrum. It further scales latent 3D positions by the tokenizer's compression ratio to harmonize visual–text rotary rates, correcting the frequency imbalance inherent in naïve 3D RoPE. AND 2) AR-DF: a mask-based discrete diffusion scheme that replaces next-token decoding under an intra-frame bidirectional and inter-frame causal attention mask. During training, temporal tube masking mitigates late-frame loss collapse caused by spatial redundancy. At inference, consistent partial-context masking combined with KV caching stabilizes quality and motion, outperforming random masking, loss reweighting, and diffusion-forcing baselines.\n\nLumos-1 is trained on 60M images and 10M videos using 48 GPUs, which achieves competitive text-to-image performance on GenEval compared to diffusion and AR models, and produces image-to-video and text-to-video results on VBench comparable to COSMOS-Video2World and OpenSoraPlan/LTX-Video, despite substantially lower compute and data budgets. The paper also presents extensive, thorough ablation studies.", "questions": "1. Does the design in AR-DF affect long-form video generation, especially complex scenarios like shot transitions, and metrics related to dynamics?\n\n2. I recommend providing playable example videos on the project page to enable more effective qualitative visual comparisons.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of MM-RoPE is both clear and rigorous. It lays out a coherent, step-by-step narrative: from RoPE fundamentals to naive 3D extensions, diagnosing frequency-spectrum imbalance, and culminating in MM-RoPE, so readers see exactly why each design choice is made. The claims are substantiated by targeted experiments and diagnostics, including validation-loss comparisons across vanilla RoPE, Scheme 1, Scheme 2, and M-RoPE, as well as visual analyses of frequency allocation and rotary speeds that reveal the imbalance.\n\n2. Extensive and convincing ablations: 1) Efficacy of MM-RoPE (showing distribution is primary and scaling complementary); 2) Effectiveness of temporal-tube masks during AR-DF training; 3) Effect of AR-DF inference masks. The \"Inference time analysis\" suggests substantial speedups for mask-based discrete diffusion with KV caching over next-token decoding.\n\n3. By retaining standard text-side RoPE and an autoregressive transformer, and augmenting them with plug-and-play MM-RoPE for visual tokens and AR-DF for training–inference alignment, the system integrates seamlessly with pretrained LLMs. This composability makes it straightforward to scale to larger multimodal LLMs, leveraging broader pretraining and alignment to improve both understanding and generation.", "weaknesses": "1. Despite the title emphasizing \"from a unified model perspective\", the paper offers relatively little analysis or evidence on the understanding side of a unified system (e.g., text capability, multimodal understanding). While the architecture remains LLM-compatible and preserves text-side RoPE, the work does not demonstrate positive transfer to or from understanding tasks. Given page limits, a full \"generation + understanding\" suite isn’t necessary, but minimal support would help: Expanding this section with small-scale evaluations or a concrete discussion/roadmap would better align the title with the evidence.\n\n2. Both the fourth and fifth paragraphs of Introduction begin with \"To account for the nature of videos,\" which disrupts the logical flow. I recommend inserting a paragraph before them that explicitly outlines the key considerations for modeling the spatiotemporal visual space. Then, follow with separate paragraphs that elaborate on each aspect, resulting in a clearer and more coherent structure.\n\n3. The current comparisons do not include 2025 state-of-the-art models, which weakens the empirical case. Please add or reference 2025 baselines (both AR and diffusion/hybrid) with matched settings where possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)