OmniGen-AR: AutoRegressive Any-to-Image Generation
Authors: Junke Wang
Xun Wang
Qiushan Guo
Peize Sun
Weilin Huang
Zuxuan Wu
Yu-Gang Jiang
Venue: NeurIPS 2025 poster ()
Keywords: autoregressive visual generation
Primary area: deep_learning

Abstract: Autoregressive (AR) models have demonstrated strong potential in visual generation, offering competitive performance with simple architectures and optimization objectives. However, existing methods are typically limited to single-modality conditions, \eg, text or category labels, restricting their applicability in real-world scenarios that demand image synthesis from diverse forms of controls. In this work, we present \system, the first unified autoregressive framework for Any-to-Image generation. By discretizing various visual conditions through a shared visual tokenizer and text prompts with a text tokenizer, \system supports a broad spectrum of conditional inputs within a single model, including text (text-to-image generation), spatial signals (segmentation-to-image and depth-to-image), and visual context (image editing, frame prediction, and text-to-video generation). To mitigate the risk of information leakage from condition tokens to content tokens, we introduce Disentangled Causal Attention (DCA), which separates the full-sequence causal mask into condition causal attention and content causal attention. It serves as a training-time regularizer without affecting the standard next-token prediction during inference. With this design, \system achieves new state-of-the-art results across a range of benchmark, \eg, 0.63 on GenEval and 80.02 on VBench, demonstrating its effectiveness in flexible and high-fidelity visual generation.

OmniGen-AR: AugoRegees Any-to-Image Generation
Junke Wang! Xun WangS, Qiushan Guo?, Peize Sun',
Weilin Huang", Zuxuan Wu Yu-Gang Jiang
Institute of Trustworthy Embodied AI, Fudan University
Shanghai Collaborative Innovation Center Intelligent Visual Computing Bytedance Seed, "The University of Hong Kong
Abstract
Autoregressive (AR) models have demonstrated strong potential in visual gener- ation, offering superior performance with simple architectures and optimization objectives. However, existing methods typically limited to single-modality ditions, e.g., text, restricting their applicability in real-world scenarios that demand image synthesis from diverse controls. In this work, present OmniGen-AR, a unified autoregressive framework for Any-to-Image generation. By discretizing various visual conditions through shared visual tokenizer and text prompts with a text tokenizer, OmniGen-AR supports broad spectrum of conditional inputs within a single model, including text (text-to-image generation), spatial signals gegmentation-to-maag and (epth-to-image) and visual context (image editing, frame prediction, and text-to-video generation). To mitigate the risk of information leakage from condition tokens to content tokens, we introduce Disentangled Causal Attention (DCA), which separates the full-sequence causal mask into condition causal attention and content causal attention. It serves as training-time regular- izer without affecting the standard next-token prediction during inference. With this design, OmniGen-AR achieves new state-of-the-art or at least competitive results across a range of benchmark, e.g., 0.63 on GenEval and 80.02 on VBench, demonstrating its effectiveness in flexible and high high-fidelity visual generation.
Introduction
In recent years, deep generative models [32, 21, 100, 61, 52] have experienced rapid development and revolutionized the way create visual contents. Among them, autoregressive models (AR) [15, 100, 66, 82] have demonstrated the capability for high-quality image synthesis through sequential token prediction. The superior performance, flexbility, and compatibility with multimodal inputs, position them as competitive alternatives to diffusion models [25, 64, 13, 51, 70].
Despite their potential, existing autoregressive (AR) visual generation methods primarily focus on single-modality conditioning, such as category labels [15, 18, 101, 81, 79] or text prompts [60, 100, 66, 82]. While these models achieve strong performance within the respective domains, they fall short of the versatility required in real-world applications, where visual generation often respond to diverse set of conditional inputs [106, 107, 89, 94], such semantic masks, reference images, or history frames. In other words, building a unified AR generative model that accommodates various inputs remains under-explored.
To this gap, this work presents OmniGen-AR, an autoregressive framework for Any-to-Image generation. In addition to text, OmniGen-AR also supports a wide range of visual conditions
t: corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table 1: A system-level comparison between OmniGen-AR and other methods. Compared to OmniGen [94]. OmniGen-AR additionally supports video generation.
Text-to-lmage generation
Condition
Method
Type
Text
Ref
Spatial
GLIGEN [40]
Diff
(2) Text-to-video generation
Change to watercolor pointing
ControlNet [106]
Diff
X
video prediction
(3) Image editing
X
Uni-ComtrolNet [107]
Diff
OmniGen [94]
Diff
AA
LLamaGen [66]
AR
SimpleAR [82]
AR
X
(4) Depth-to-image generation
(5) Seg-to-image generation
ControlAR [41]
AR
Figure 1: OmniGen-AR could handle types of gener- ation tasks within single model.
Ours
AR
including segmentation masks, depth maps, and reference images, by discretizing them with shared visual tokenizer. An overview of our method provided in Table and Figure I. While this approach allows model preserve the simplicity of autoregressive modeling, the serial nature of prediction introduces a potential risk of information leakage from condition tokens to content tokens. This becomes particularly problematic in tasks like image editing and frame prediction, where much of the output remains unchanged relative to the input. Consequently, the model may converge to suboptimal solutions that exploit shorteut patterns between conditioning and prediction signals, instead of producing meaningful and cnterructoo-foolowiiiii results.
To address this, introduce Disentangled Causal Attention (DCA), which separates the causal attention over the entire sequence into condition causal attention and content causal attention. This disentanglement prevents the information flow from content tokens to condition tokens while still allowing the latter to retain awareness of their relative positions. During training, we randomly replace the vanilla causal attention with DCA, which regularizes the model by discouraging over-reliance conditional context and promoting more instruction compliant predictions. The inference process of our model still follows the standard next-token prediction.
We validate the effectiveness of OmniGen-AR on six representative visual generation tasks, including text-to-image generation, text-to-video generation, frame prediction, image editing, depthhconditioned image generation, and potneeetationccoddiioon
image generation. The results demonstrate that achieves state-of-the- thetthe-art or at least competitive results the prevalent benchmarks, e.g., 0.63 on GenEval [20] and 80.02 on VBench [29]. OmniGen-AR not only maintains the inherent ficxibility and scalability of autoregressive models but also enables the seamless integration of various control signals, providing a unified and effective solution for universal visual generation.
2 Related Work
2.1 Autoregressive Visual Generation
Autoregressive (AR) models have become a popular paradigm in generative modeling for both language [56, 57, 72, 3] and vision [100, 66, 82], owing to their strong capability modeling complex distributions. Early efforts in AR-based visual generation model images sequences of pixels [74, 10], which achieves satisfactory results but suffer from inefficiency and limited scalability. Subsequent approaches such VQ-VAE [75] introduce discrete visual tokenization to autoregressive visual generation, enabling the use transformer- based language models for image synthesis. These bosen-based methods significantly improve the generation quality and training stability, attracting series of work that leverage learned codebooks for autoregressive image generation [60, 15, 100, 66].
Recent works have explored autoregressive generation with continuous representations [39, 108), scale-wise autoregressive modeling [69, 23), and menfoorcementllerrnnng for improved generation quality [82]. Despite these advances, existing AR models mainly focus single-mon modellty conditions

---

(e.g., text or class labels), restricting their applicability in real-world scenarios requiring multi modal controls. The most relevant literature is EditAR [49], which also employs autoregressive transformers and support multiple conditional image generation tasks. Ours work differ from them in two aspects: 1) EditAR is specifically designed for image editing and low-level control tasks (e.g., depth-to-image, edge-to-imagee segmentation-to -to-image)), while our model is unified Any-to-Image framework that handles a broader range of input modalities. 2) EditAR aims to improve text-image alignment by introducing distillation loss, while OmniGen-AR hopes prevent information leakage through DCA, a novel training-time attention mechanism that disentangles condition and content attention paths.
2.2 Diffusion Models for Any-to-Image Generation
Recent advances in diffusion models [52, 14] have significantly improved the quality and control- lability of image synthesis from diverse conditioning signals [106, 40, 85, 104]. ControlNet [106] firstly introduces framework that injects spatial control (e.g., edge maps, segmentation masks) into a pretrained diffusion model without compromising generation quality. demonstrates strong performance in aligning generated content with spatial priors but still requires separate adapters for each conditional modality. To address this, Uni-ControlNet [107] proposes a unified architecture that supports multiple spatial controls within a single framework by learning modality agnostic representation space. It improves the generality and ficxibility across tasks, but still requires the separate training procedures for different types of condition. More recently, OmniGen [94] pushes toward general-purpose image generation by unifying diverse tasks, e.g., eext-to-image synthesis, image editing, and subject-driven generation, within diffusion framework that eliminates external modules. Inspired by this, we explore the autoregressive framework for any-to-image generation. We adopt autoregressive modeling as it provides more natural fit for handling sequential inputs and enabling interleaving generation.
modasity-agnostic
2.3 Unified Models for Multimodal Understanding and Generation
The belief in scaling data and model size 31, 102] has driven the community towards building unified and even general multimodal models [2, 68, 1]. CLIP [55] first demonstrates that large-scale contrastive pretraining on image-text pairs could yield powerful vision-language representations. Subsequent works [38, 80, 88, 12, 4] extend this paradigm to support a broader range of vision- language tasks such as captioning and VQA, across both image and video domains.
LLaVA [44, 43] opens up another chapter, iee. visual instruction tuning, by aligning pretrained vision encoders [57, 103] with large language models [72] to enabling open-ended multimodal understanding. Recently, Chameleon [67] steps beyond the scope of understanding tasks to unified multimodal understanding and generation, seamlessly integrating both modalities in a token- based framework. Following work improve the design of unified multimodal language models through better multimodal fusion [111, 95, 71] and visual encoding [92, 93]. These advancements showcase the growing potential of general multimodal artificial intelligence, pushing the boundaries of both understanding and generation tasks across multiple domains.
3 Method
Our goal is to unify conditional image generation (i.e., text, spatial, image) within a single autoregres- sive framework. To this end, we propose OmniGen-AR, which consists of a text and visual tokenizer to discrete various inputs. With this, we model the dependency between multimodal tokens using an autoregressive transformer. The architecture of OmniGen-AR is illustrated Figure 2.
3.1 Visual and Textual Tokenization
To enable the unified processing and generation of diverse modalities, the first question is how to represent them in a compatible format. Unlike previous work [106, 41] that rely on separate encoders to encode visual condition V E IR H x x3 (segmentation masks, depth maps, image to be edited) and the image to be generated X E & x3 we adopt the same visual tokenizer [16] to convert them into discrete visual tokens: E R and E N2, where N1 = N2 denote the sequence length. Here we omit the loop-up and flatten operations for simplicity. While for the textual inputs, tokenize them with a language model tokenizer [98] to obtain the text tokens & M.
3

---

11
12
15
16
18
19
21
AutoRegressive Transformer (Causal & Disentangled Causal Attention)
10
14
15
18
21
Text Tokenizer
Visual Tokenizer
Text Tokenizer
Visual Tokenizer
A woman wearing a yellow lop and white headscarf is pouring soup into a bowl.
Turn the photo into film style.
Figure OmniGen-AR consists of a text tokenizer, visual tokenizer, an autoregressive transformer.
3.2
Autoregressive Transformer for Multimodal Generation
We adopt decoder-only transformer model for multimodal generation, which consists of stacked attention blocks [76]:
qk" /dz
Attention(q.ke v) softmax
+ m
(I)
where g, k E Nxdx represent the query, key, and value embeddings, and RNN is the attention mask. In modern language models [57, 72] and AR-based visual generation models [100, 66, 82], is usually implemented lower triangular matrix to mask the future positions
0, -0, otherwise
i
mi.j =
This mechanism ensures that each token attends only to itself and preceding tokens the sequence, preserving the left-to-right generation order. However, in the context of conditional image generation, the plain causal attention can lead to unintended information leakage: given access to previous condition tokens, the model may learn exploit trivial correlations between them and the content tokens to be predicted, instead of generating meaningful tokens that follow instructions faithfully [17, 96, 37, 84].
To alleviate this problem, we modify the attention mask to prevent information flow from condition tokens to content tokens, while still preserving autoregressive modeling within each. Taking the image editing task as an example, we concatenate text, condition, and content tokens along the sequence dimension the token sequence: It, x] RA, where L = M N1 N2, and design the attention mask (M+ M N2) Na) in the following manner.
0, if < and (i.j) AUBUC miij DC, if C, jEB
(3)
DC, otherwise
where A MO, B
C C= N1, N,, N₁ N₂)
Such masking scheme permits content tokens to attend to preceding text tokens while blocking access other condition tokens, thereby reducing the risk shorteut learning. During training, we randomly apply DCA in place of vanilla causal attention as regularization. Please see Figure for a better illustration. Notably, the proposed DCA differs from classifier-free guidance [26] in its treatment of condition tokens. First of all, content tokens remain aware of positional information, as the condition tokens are not dropped entirely. addition, DCA applied only during training and has no impact the inference process.

---

plain causal attention mask
(b) disentungled causal attention mask
(c) classifier-free guidunce
Figure 3: Comparison between plain causal attention, the proposed disentangled causal attention, and classfier-free guidance. T. 11, la represent text, spatial (image) condition and content tokens.
3.3 Training and Inference
For different generation tasks, construct the token sequence z by interleaving text, condition, and content tokens according to task task-specific formats. Specifically, spatial- and mnage-conditioned tasks follow the manner: 2 it, v, z². while text-conditioned tasks use 2 [t. z]. OmniGen-AR is trained to autoregressively predict the next token over these sequences using language modeling loss [57, 100].
During inference, the tokens are sequentially sampled based on the learned conditional probability: argmax pe (2) After that, we feed them the decoder of visual tokenizer to generate images. Classfier-Fres Guidance (CFG) [26] is adopted to improve the generation quality, following previous work [66, 82].
4 Experiments
4.1 Experimental Setup
Training data. The training of OmniGen-AR includes three stages: 1) single image stage (SI), where we pretrain our model on large-scale image datasets, involving CC3M [62], CC12M [7]. [46]. We also incorporate video datasets, i..,, a 9M subset of Panda70M [11] and HD-VILA-IOOM [97], and randomly sample frame for each video. 2) image-video joint stage (IV), where we maintain the datasets used in the first stage but sampled 9 frames from the videos. 3) multi-task stage (MT), where train our model on wide- range of high-quality datasets, including text-to-image datasets (JourneyDB [65]. Synthetic-dataset IM [53], and 10M internal data), image editing datasets (MagicBrush [105], Instruct-Pix SEED-Edit [19]), depth-to-image datasets (MultiGen-Deee Depth [54]), segmentation- -o-ma-att- datasets MllltiGenAADE00 [54] and MultiGen-COCOOttuf [54]), and text-to-video datasets (OpenSora- pexels-45k [28], Open Vid- 1M [50], and 0.5M high-quality internal data). We recaption all the images and videos using QWen2-VL [83].
Openlmages [36], SAMIB [33], and Negalith-hugginge
[5],
Implementation details. We adopt Qwen2.5 [98] as the text tokenier and transformer model. While for visual tokenizer, we use image-video joint tokenizer, i.e., Cosmos DV8 x 16x 16 [16], which allows us to tokenize different controls, images, and videos with the same codebook. During the SI and IV stages, we train our model on 512 resolution, and the learning rate is set to le-4. While for the MT stage, we increase the resolution to 1024 and decrease the learning rate to 2e-5. We train model on 64 A100 GPUs, the global batch size 256 for all stages, warm up learning rate decay are used. AdamW [45] is employed for optimization. During the IV and MT stages, we replace the standard causal attention mask with disentangled causal attention mask with probability of 10%, and similarly drop the text conditions for classifier-free guidance with the same probability. We set CFG scale 6.0 during inference.
4.2 Comparison with State-of-the-arts
Text-to-image generation. Table 2, we compare OmniGen-AR with existing image generation models GenEval [20], challenging and popular text-to-image (T2I) benchmark. The results
5

---

Table 2: Text-to-image generation GenEval, Results markdd with result are using prompt rewriting.
Table 3: Video generation on VBench.
Method
Par.
Qua.
Sem.
Total
Method
Par.
Two.
Pos.
Color.
Overall
CogVideo [27]
9B
72.06
46.83 67.01
SDv1.5 [61]
0.9B
0.38
0.04
0.06
0.43
LaVie [87]
3B
78.78
70.31 77.08
PixArt-alpha [9]
0.6B
0.50
0.08
0.07
0.48
OpSoraP V1.3 [42]
2.7B
80.14 65.62 77.23
SDv2.1 [61]
0.9B
0.51
0.07
0.17
0.50
CogVidcok [99]
5B
83.05
77.33 81.91
LeamaGen [66]
0.8B
0.34
0.07
0.04
0.32
Hunyuan [35]
13B
85.09
75.82 83.24
SimAR-SFT [82]
0.5B
0.75
0.20
0.24
0.53
Ours
0.5B
0.74
0.20
0.29
0.55
Mira [30]
1.1B
78.78
44.21 71.87
LDM [61]
1.4B
0,29
0.02
0,05
0.37
TF-12V [8]
1.8B
80.05
56.69 75.38
DALL-E 2 [59]
6.5B
0.66
0.10
0.19
0.52
OpSora V1.2 [109]
1.1B
81.35
73.39 79.76
Show-o [95]
1.3B
0.80
0.31
0.50
0.68
AniDiff V2 [22]
0.9B
82.90
69.75 80.27
Infinity [23]
2B
0.85
0.491
0,57
0.73
VidCrafter-2.0 [8]
1.4B
82.20
73.42 80.44
Janus [92]
1.5B
0.68
0.46
0.42
0.61
CogVidcoX [99]
2B
82.18
75.83 80.91
Emu3 [86]
8.5B
0.81
0.491
0.45t
0.66
Wan2.1 [78]
1.3B
85.23
75.65 83.31
SimAR-SPP [82]
1.5B
0.87
0.27
0.33
Ours
0.5B
76.60
67.20 74.72
Ours
1.5B
0.94
0.30
0.40
0.63
Ours
1.5B
81.51 78.08 80.02
Table 4: Frame prediction on Kinetics-600 (left. denotes zero-shot evaluation), image editing on Emu-Edit set (middle), and spatial-conditioned generation (right). CT: CLIP text similarity between edited image and edited prompt, CI: CLIP image similarity between edited image and condition image.
Method
FVD
Method
CT
CI
Method
Mask mloU (+)
Depth RMSE (L)
LVT [58]
225
I-Pix2Pix [5]
0.22 0.83
Uni Uni-ControlNet [107]
19.39
40.65
ViTrans [91]
170
MagBrush [105]
0.22 0.84
GLIGEN [40]
23.78
38.83
CogVideo [27]
109
PnP [73]
0.09 0.52
EditAR [49]
22.62
34,93
ViVQVAE [77]
64
Null-Text [47]
0.24 0.76
ControlNet [106]
32.55
35,90
OmniTok [81]
33
Emu-Edit [63]
0.23 0.86
ControlAR [41]
39.95
29.01
VideoPoet-8B" [34]
687
OmniGen [94]
0.23 0.83
OmniGen [94]
40.06
31.71
O.rs-1.5B*
429
Ours-1.5B
0.23 0.84
Ours-1.5B
35.28
37.42
show that OmniGen-AR significantly outperforms all other models with fewer than 1B parameters, including both diffusion models (e.g..SDDv2 [61]) and autoregressive models (e.g., LLamaGen [66]). When scaled to 1.5B size, the overall performance of OmniGen-Ai ARR is improved from 0.57 to 0.63, highlighting its promising scalability when more training computes are available.
Text-to-video generation. We also evaluate OmniGen-AR on VBench [29] for text-to-vided gen- eration, and the comparison with existing video generation models is shown in Table 3. With only 0.5B parameters, our model achieves 74.72 7. total score on VBench, surpassing the previous SOTA AR-based models, Le., Cog Video [27], by 11 % while using much fewer parameters (0.5B v.s. 9B). Similar to what we have seen on T2I, the results of T2V could be signigicantly improved to 80.02 using 1.5B parameters, even beating diffusion models like OpenSora V1.2 [109]. It is also worth noting that it is the first time that a vanilla autoregressive model using discrete tokens could achieve 80+ score on VBench.
Frame prediction and image editing. To evaluate the image generation capability given visual context (image condition). choose two types of tasks: frame prediction on Kinetics-600 [6] and image editing on Emu-Edit test set [63]. The results in Table show that OmniGen-AR achieves much lower Fréchet Video Distance (FVD) than VideoPoet [34] for frame prediction. While for the more challenging image editing task, OmniGen-AR also achieves competitive results, i.e., 0.23 CLIP text similarity [55].
Segmentation and depth-to-imaae generation. We follow previous work [41, 94] to report the segmentation-to-ineet and depth-to-image generation performance on ADE20K [1]0] and MultiGen-

---

Depth-Eval [54] in Table 4 (right). Compared to text or image condition. spatial conditions provide more structured and fine-grained instructions, thus posing a greater challenge the model to geometrically accurate and contextually coherent content. We can see that OmniGen-AR achieves competitive results on both tasks, outperforming diffusion counterparts [48, 40).
4.3 Ablation Studies
Effects of disentangled causal attention. To better illustrate the potential information leakage in image-conditioned generation tasks, compute the token match ratio (TMR) of MagicBrush [105], popular image editing dataset, and visualize in Figure 4. TMR defined as the fraction identical tokens in the same position between condition and content images: TMR N₂ Σ; [e; cal. where 2 and c; denote the i-th token from the condition and content images respectively, and N1 is the total number tokens. The x-axis of Figure 4 denotes binned TMR ranges (e.g., . 0.80-0.85), and the y-axis shows the proportion of samples falling into each bin. As can be seen, significant portion of samples exhibit high TMR values, suggesting substantial overlap between condition and content images, which may imply unintended information leakage.
16
Averaged 0.37%
Table 5:
OmniGen-AR w/ and w/o DCA on different generation tasks.
14
Percentage
10-
Method
VBen
Emu-CT
Mask
0%
70.33
0.15
24.76
5%
74.55
0.17
25.78
0000.0
0.10-.15
0.002225
0.30-335
4.4.4.55
0.50-0.55
0.760.75
a500
0.90.0.95
10%
74.72
0,20
25.33
20%
73.28
0,21
25.16
Token Match Ratio
30%
71.69
0.19
21.49
Figure 4: Token similarity on MagicBrush [105].
As mentioned in 1ec.4.1, 1, we randomly replace standard causal attention with the proposed Dis- entangled Causal Attention (DCA) during training. We also conduct experiments with varying replacement probabilities using the 0.5B model across different tasks. shown in Table5, adopting DCA with 10% probability improves the CLIP text similarity on Emu from 0.15 to 0.20, indicat- ing the encouragement robust conditioning without significantly limiting the access to informative context. Interestingly, DCA also yields slight gains in segmentation-to-ing hypothesize results from its ability to reduce over-reliance on exact segmentation inputs and thus improving the robustness of model. Unless otherwise specified, we adopt default replacement probability of 10% in all experiments.
generation, which
Table Joint or separate training.
0.58
photo
The Shanghai
Method
GEval
VBen Emu Mask
green
Goght
Joint
0.55
74.72
0.20 25.33
Text
0.58
1.58
T21
0.57
T2V
77.18
Edit
0.18
Depth
0.58
1.58
Segmentation
0.58
1.59
Seg
22.59
Figure 5: Effects of model scaling.
Synergy between different tasks. We study the synergy between different generation tasks by comparing joint training and separate training, both initialized with the 2nd stage checkpoint. As shown in Table 6, joint training leads to degraded performance on text-to-image and text-to-video benchmarks, possibly due to the lower visual quality of editing and spatial-conditioned datasets. In contrast, it improves the results editing and segmentation-to-imag generation tasks, suggesting that strengthening the foundation capability, i.e., text-conditioned generation, can facilitate better generalization to broader range of downstream tasks.

---

high resolution portrait. Her hair is styled in full platinum blonde waves that frame her intensely expressive, pale-skinned visage with high detail.
Girl with Pearl Earring, adorned with a pearl drop earring. The soft texture of her pale skin with dark, liquid like background.
captivating portrait showcasing a young girl with ethereal beauty, gracefully suspended amidst the soft, billowy clouds.
painting dominated by rich, deep colors featuring gothic clwn girl as central figure.
An intricately detailed ail painting that captures whimsical essence feline super math wizard.
Two golden-brown spring rolls with perfectly crispy texture sit invitingly bamboo mat.
curious vessel, with architecture reminiscent glant green broccoli, basks in the bright sunlight.
A bright yellow tennis ball lies in stark contrast on vibrant green mowed grass court.
The sun radiates down a sandy beach where a chocolate ice cream beginning to melt down.
vibrant green plant with broad leaves and sturdy stem, situated at the bottom of clear, gently flowing stream.
beautiful coastal beach in spring, waves lapping on sand, tilt down
cute happy Corgi playing in park, sunset, watercolor painting
Figure Visualization of text-to-image and text-to-video results generated by OmniGen-AR.
Model scaling. Figure 5, we qualitatively compare the models with 0.5B and 1.5B parameters. can be that scaling the model size could effectively improve the generation results various tasks, leading to improved instruction-follovinn capability and more aesthetically pleasing images.
4.4 Qualitative Results
We display some visualization results in Figure 6 and 7. OmniGen-AR could synthesize high-quality images based on various types of conditions, showcasing both versatility in handling diverse inputs and the ability to maintain semantic coherence with contexts. Several failure cases are also shown in Figure 8, which can be broadly categorized into two types: 1) Instruction-fol fordioonf capability. For instance, the first row of Figure 8, the instruction is "Remove the bag on the bench next to the person sitting at the bus stop". but the model removes the person instead. This indicates failure grounding fine-grained spatial and referential cues from language into visual modifications, 2) Low-quality generations under sparse control signals. Examples in the second row (depth-to-image and sigmentation-to-iaant show blurry or structurally inconsistent results, which likely stem from noisy supervision and sparse training coverage for these conditions. These failure modes suggest two

---

Delete the apples.
Make into abstract style.
Change green.
depth
image
fused
depth
image
fused
segmentation
image
fused
segmentation
image
fused
Figure 7: Image editing. depth-to topph- and segmentation- to-mam generation results.
Rernove the bag on the bench next to person sitting stop.
Add the word 'room' to the back wall
Turn the round donuts into tennis balls.
Figure Failure cases generated by OmniGen-AR.
potential directions for future work: 1) Scaling up the model and training data to build stronger base model with improved generalization and instruction-followinn ability across diverse visual tasks. 2) Leveraging chain-of-thought (CoT) [90] to improve the reasoning ability on complex prompts.
5 Conclusion and Broader Impacts
This paper presented OmniGen-AR, a unified autoregressive framework any-to-image generation. OmniGen-AR represents wide spectrum of conditional inputs, i.e., text prompts, spatial controls, and visual contexts, as discrete tokens, and trains a unified autoregressive transformer to model the dependencies between these conditions and the target image tokens. To mitigate the potential information leakage in conditional generation tasks, we proposed Disentangled Causal Attention (DCA), which separates the attention pathways between condition and content tokens to facilitate the learning of Instructioonfoolooiing generation. Comprehensive experiments demonstrate that OmniGen-AR achieves state-of-the the-art at least competitive performance across wide range of visual generation tasks.
Acknowledgement This project was supported by NSFC under Grant No. 62032006 and No. 624B2043.

---

References
[1] M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, J. Hewett, M. Javaheripi, P. Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:441..08005 2024.
[2] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint ar arXiv:2303.08774 2023.
[3] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[4] D. Bolya, P.-Y. Huang, P. Sun, J. H. Cho, A. Madotto, C. Wei, T. Ma, J. Zhi, Rajasegaran, Rasheed, et al. Perception encoder: The best visual embeddings are not the output the network. arXiv preprint 2rXiv:2504.13188 2025.
[5] T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning follow image editing instructions. In CVPR, 2023.
[6] J. Carreira, E. Noland, A. Banki-Horvath, C. Hillier, and A. Zisserman. A short note about kinetics-600. arXiv preprint urXiv: 100..0088.0000 31040, 2018
[7] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
[8] H. Chen, Y. Zhang, X. Cun, M. Xia, X. Wang, C. Weng, and Y. Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusson models. In CVPR, 2024.
[9] J. Chen, J. Yu, Ge, Yao, Xie, Y. Wu, Z. Wang, J. Kwok, P. Luo, H. Lu, et al. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.0042
2023.
[10] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In ICML, 2020.
[11] T.-S. Chen, A. Siarohin, W. Menapace, E. Deyneka, H.-w. Chao, B. E. Jeon, Y. Fang, H.-Y. Lee, J. Ren, M.-H. Yang, et al. Panda-70m: Captioning 70m vidcos with multiple cross-modast cross-modality teachers. In CVPR, 2024.
[12] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, Xing, M. Zhong, Q Zhang, X. Zhu, L. Lu, et al. Internvl: Sealing up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, 2024.
[13] P. Dhariwal and A. Nichol. Diffusion models beat gans image synthesis. In NewrIPS, 2021.
[14] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICLR, 2024.
[15] P. Esser, Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[16] N. et. al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575 2025.
[17] Z. Fu, W. Lam, A. M.-C. So, and B. Shi. A theoretical analysis of the repetition problem in text generation. AAAI, 2021.
[18] S. Ge, T. Hayes, H. Yang. X. Yin, G. Pang, D. Jacobs, J.-B. Huang, and D. Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. In ECCV, 2022.
[19] Y. Ge, S. Zhao, C. Li, Ge, and Y. Shan. Seed-data-edit technical report: A hybrid dataset for instructional image editing. arXiv preprint PrXiiv:240504000 2024.
10

---

[20] D. Ghosh, H. Hajishirzi, and L. Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. In NeurIPS, 2024.
[21] J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NeurlPS, 2014.
[22] Y. Guo, C. Yang, A. Rao, Z. Liang, Y. Wang. Y. Qiao, M. Agrawala, D. Lin, and B. Dai. Animatediff: Animate your personalized text-to-image to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.0472. 04725, 2023.
[23] J. Han, J. Liu, Y. Jiang, B. Yan, Y. Zhang, Z. Yuan, B. Peng, and X. Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. In CVPR, 2025.
[24] T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse. J. Jackson, H. Jun, T. B. Brown, P. Dhariwal, S. Gray, et al. Scaling laws for autoregressive generative modeling, arXiv preprint arXiv:2010. 14701, 2020.
[25] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
2020.
[26] J. Ho and T. Salimans. Classifier-fre free diffusion guidance. arXiv preprint 2022.
[27] W. Hong, M. Ding, W. Zheng, X. Liu, and Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205. 2022.
[28] hpcai tech. open-sora-pexxll-45k. open open-sora- peoor-peeees--oe 45k, 2025.
https://muggingface http:/accc..cctttttttts////aapae
[29] Z. Huang, Y. He, J. Yu, F. Zhang, C. Si, Y. Jiang, Y. Zhang, T. Wu, Q. Jin, N. Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In CVPR, 2024.
[30] X. Ju, Y. Gao, Z. Zhang, Z. Yuan, X. Wang, A. Zeng, Y. Xiong, Q. Xu, and Y. Shan. Miradata: A large-scale video dataset with long durations and structured captions. In NeurIPS, 2024.
[31] J. Kaplan, McCandlish, T. Henighan, T. Brown, B. Chess, Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.083611 2020.
[32] D. P. Kingma, M. Welling, et al. Auto-encoding variational bayes, 2013.
[33] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, C. Berg, W.-.. Lo, et al. Segment anything. ICCV, 2023.
[34] D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, G. Schindler, R. Hornung, V. Birodkar,
J. Yan, M.-C. Chiu, et al. Videopoet: A large language model for zero-shot video generation. In ICML, 2024.
[35] W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. Hunyuanvideo: systematic framework for large video generative models. arXiv preprint arXiv:2412.03600 2024.
[36] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, A. Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
[37] Li, T. Lan, Z. Fu, D. Cai, L. Liu, N. Collier, Watanabe, and Y. Su. Repetition in repetition out: Towards understanding neural text degeneration from the data perspective. In NewrIPS, 2023.
[38] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-imas -maga pre-training for unified lision-language understanding and generation. In ICML, 2022.
[39] T. Li, Y. Tian, H. Li, M. Deng, and K. He. Autoregressive image generation without vector quantization. In NeurIPS, 2024.

---

[40] Y. Li, H. Liu, Q. Wu, F. Mu, J. Yang. J. Gao, C. Li, and Y. J. Lee. Gligen: Open-set grounded text-to-image generation. In CVPR, 2023.
[41] Z. Li, T. Cheng, S. Chen, P. Sun, H. Shen, L. Ran, X. Chen, W. Liu, and X. Wang. Controlar: Controllable image generation with autoregressive models. In ICLR, 2025.
[42] B. Lin, Y. Ge, X. Cheng, Z. Li, B. Zhu, S. Wang, X. He, Y. Ye, S. Yuan, L. Chen, et al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.00131, 2024.
[43] H. Liu, C. Li, Li, and Y. Lee. Improved baselines with visual instruction tuning. In CVPR, 2024.
[44] H. Liu, Ci, Q. Wu, and Y. Lee. Visual instruction tuning. In NeuPPS, 2023.
[45] I. Loshchilov and F, Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.051011 2017.
[46] madebyollin.
https://muggingface http:/aaac..oooaaaaaaaaee catastet
beaebyoliinmmggaliie 10m, 2024.
[47] Mokady, A. Hertz, Aberman, Y. Pritch, and D. Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In CVPR, 2023.
[48] C. Mou, Wang, L. Xic, Y. Wu, Zhang, Z. Qi, and Y. Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In AAAI, 2024.
[49] J. Mu, N. Vasconcelos, and X. Wang. Editar: Unified conditional generation with autoregres- sive models. CVPR, 2025.
[50] K. Nan, R. Xie, P. Zhou, T. Fan, Z. Yang, Z. Chen, X. J. Yang, and Y. Tai. Openvid- Im: A large-scale high-quality dataset for text-to-vided generation. arXiv preprint 3rXiv:2407.023711 2024.
[51] W. Peebles and S. Xie. Scalable diffusion models with transformers. In CVPR, 2023.
[52] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rom- bach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint ar arXiv:2307.019522 2023.
[53] ProGamerGov.
captions.
https://huggingfare co/ da//ttsstss//////ssssoass.aassst hagaaatt----llllee--hh--------a
htt//aatassss//oooaaarroooooo PPto/oarrrreeo///// 2022.
[54] C. Qin, S. Zhang, N. Yu, Y. Feng, X. Yang, Y. Zhou, H. Wang, J. C. Niebles, C. Xiong, S. Savarese, al. Unicontrol: unified diffusion model for controllable visual generation the wild. arXiv preprint arXiv:2305. 147, 2023.
[55] A. Radford, W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In ICML, 2021.
[56] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. OpenAl Blog, 2018.
[57] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
[58] R. Rakhimov, D. Volkhonskiy, A. Artemov, D. Zorin, and E. Burnaev, Latent video transformer. arXiv preprint arXiv:2006. 10704, 2020.
[59] Ramesh, Dhariwal, A. Nichol, C. Chu, and Chen. Hierarchical text-conditional image generation with clip latents, arXiv preprint 27X1v:2204.06115, 2022.
12

---

[60] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
[61] R. Rombach, A. Blattmann, D. Lorenz, Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[62] P. Sharma, N. Ding, Goodman, and R. Soricut. Conceptual captions: cleaned, hypernymed. image alt-text dataset for automatic image captioning. In ACL, 2018.
[63] S. Sheynin, A. Polyak, U. Singer, Y. Kirstain, A. Zohar, O. Ashual, D. Parikh, and Y. Taigman. Emu edit: Precise image editing via recognition and generation tasks. In CVPR, 2024.
[64] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[65] K. Sun, J. Pan, Y. Ge, H. Li, H. Duan, X. Wu, R. Zhang, A. Zhou, Z. Qin, Y. Wang, al. Journeydb: A benchmark for generative image understanding. In NeurIPS, 2023.
[66] P. Sun, Y. Jiang, S. Chen, S. Zhang, B. Peng, P. Luo, and Z. Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06652. 2024.
[67] C. Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint
arXiv:2405.094 00080, 2024.
[68] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
[69] K. Tian, Y. Jiang, Z. Yuan, B. Peng, and L. Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:244.02205,, 2024.
[70] R. Tian, Q. Dai, J. Bao, K. Qiu, Y. Yang, C. Luo, Z. Wu, and Y.-G. Jiang. Reducio! generating 1k video within 16 seconds using extremely compressed motion latents. In ICCV, 2025.
[71] Tian, M. Gao, M. Xu, J. Hu, Lu, Z. Wu, Y. Yang, and A. Dehghan. Unigen: Enhanced training & test-time strategies for unified multimodal understanding and generation. In NeurIPS, 2025.
[72] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, Hambro, F. Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1381
2023.
[73] N. Tumanyan, M. Geyer, S. Bagon, and T. Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In CVPR, 2023.
[74] A. van den Oord and N. Kalehbrenner. Pixel mn. In ICML, 2016.
[75] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. In NeurlPS, 2017.
[76] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention is all you need. In NeurlPS, 2017.
[77] J. Walker, A. Razavi, and A. d. Oord. Predicting video with vqvae. arXiv preprint arXiv:2103.01950, 2021.
[78] A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen. F. Yu, H. Zhao, J. Yang, J. Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint 2xXiv:2503.203/44 2025.
[79] H. Wang, S. Suri, Y. Ren, H. Chen, and A. Shrivastava, Larp: Tokenizing videos with learned autoregressive generative prior. In ICIRR 2025.
[80] J. Wang, D. Chen, Z. Wu, C. Luo, L. Zhou, Y. Zhao, Y. Xie, C. Liu, Y.G.. Jiang, and L. Yuan. Omnivl: One foundation model for langaaaa and video-language tasks. NeurIPS, 2022.
13

---

[81] J. Wang, Y. Jiang, Z. Yuan, B. Peng, Z. Wu, and Y.GG. Jiang. Omnitokenizer: A joint image-vides tokenizer for visual generation. In NeurIPS, 2024.
[82] J. Wang, Z. Tian, X. Wang, X. Zhang, W. Huang, Z. Wu, and Y.-G. Jiang. Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. arXiv preprint arXiv:2504.114553 2025.
[83] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint 1rXiv:2409.12191 2024.
[84] W. Wang, Z. Li, D. Lian, C. Ma. L. Song, and Y. Wei. Mitigating the language mismatch and repetition issues in Ilm-based machine translation via model editing. arXiv preprint arXiv:2410.07054 2024.
[85] X. Wang, T. Darrell, S. S. Rambhatla, R. Girdhar, and I. Misra. Instancediffusion: Instance- level control for image generation. In CVPR, 2024.
[86] X. Wang, X. Zhang, 7. Luo, Q Sun, Y. Cui, J. Wang, F. Zhang, Y. Wang. 7. Li, Q. Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409. 18869, 2024.
[87] Y. Wang. X. Chen, X. Ma, S. Zhou, Z. Huang, Y. Wang, C. Yang. Y. He, J. Yu, P. Yang, et al. Lavie: High-quulity video generation with cascaded latent diffusion models. IJCV, 2025.
[88] Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wang, al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03111, 2022.
[89] Y. Wang, H. Xu, X. Zhang, Z. Chen, Z. Sha, Z. Wang, and Z. Tu, Onnicontrolnet: Dual-stage integration for conditional image generation. In CVPR, 2024.
[90] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
[91] D. Weissenborn, O. Täckström, and J. Uszkoreit. Scaling autoregressive video models. arXiv preprint arXiv: /9tp:/990..0000 02634, 2019.
[92] C. Wu, X. Chen, Z. Wu, Y. Ma, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, C. Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. In CVPR, 2024.
[93] Y. Wu, Z. Zhang, J. Chen, H. Tang, D. Li, Y. Fang, L. Zhu, E. Xie, H. Yin, L. Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. In ICLR, 2025.
[94] S. Xiao, Y. Wang, J. Zhou, H. Yuan, X. Xing, R. Yan, C. Li, S. Wang, T. Huang, and Z. Liu. Omnigen: Unified image generation. arXiv preprint arXiv:2409. 11340 2024.
[95] J. Xie, W. Mao, Z. Bai, D. Zhang, W. Wang, K. Q. Lin, Y. Gu, Z. Chen, Z. Yang. and M. Z. Shou. Show-o: One single transformer to unify multimodal understanding and generation. In ICLR, 2025.
[96] J. Xu, X. Liu, J. Yan, D. Cai, H. Li, and J. Li. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. In NeurIPS, 2022.
[97] H. Xue, T. Hang. Y. Zeng, Y. Sun, B. Liu, H. Yang, J. Fu, and B. Guo. Advancing high- resolution video-language representation with large-scale video transcriptions. In CVPR, 2022.
[98] A. Yang, B. Yang, B. Zhang. Hui, B. Zheng. B. Yu, Ci D. Liu, Huang, H. Wei, et al. Qwen2. 5 technical report. arXiv preprint 24Xi:2412..11115, 2024.
[99] Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong, X. Zhang, G. Feng, et al. Cogvideox: Text-to- vidvo diffusion models with an expert transformer. arXiv preprint (rXi::2488.06072, 2024.
14

---

[100] J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid. Z. Wang, V. Vasudevan, A. Ku, Y. Yang. B. K. Ayan, al. Scaling autoregressive models for content-rich text-to-image generation. In ICLR. 2024.
[101] Q. Yu, M. Weber, X. Deng, X. Shen, D. Cremers, and L.-C. Chen. An image is worth 32 tokens for reconstruction and generation. In NeurIPS, 2024.
[102] X. Zhai, A. Kolesnikov, N. Houlsby, and L.. Beyer. Scaling vision transformers. CVPR, 2022.
[103] Zhai, Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre- training. In ICCV, 2023.
[104] H. Zhang, D. Hong. Y. Wang, J. Shao, X. Wu, Z. Wu, and Y.G. Jiang. Creatilayout: Siamese multimodal diffusion transformer for creative layout-to-image generation. ICCV, 2025.
[105] Zhang, L. Mo, W. Chen, H. Sun, and Y. Su. Magicbrush: manually annotated dataset for insrucction-uiided image editing. In NeurlPS, 2023.
[106] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
[107] S. Zhao, D. Chen, Y.C.. Chen, J. Bao, S. Hao, L. Yuan, and K.-Y. K. Wong, Uni-controlnet: All-in-one control text-to-image diffusion models. In NeurIPS, 2023.
[108] P. Zheng, J. Wang. Y. Chang, Y. Yu, Ma, and Wu. Rethinking discrete tokens: Treating them conditions for continuous autoregressive image synthesis. In ICCV, 2025.
[109] Z. Zheng, X. Peng. T. Yang, C. Shen, S. Li, H. Liu, Y. Zhou, T. Li, and Y. You. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.20404 2024.
[110] B. Zhou, H. Zhao, X. Puig. S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
[111] C. Zhou, L. Yu, Babu, K. Tirumala, M. Yasunaga, L. Shamis, J. Kahn, X. Ma. L. Zettlemoyer, and O. Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. In ICLR, 2025.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly and accurately state the contribution and scope in the abstract and introduction (Sec. 1).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations by showing failure cases in the experimental section (See. 4.4) and conclusion section (Sec. 5).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We specify the model configurations, hyperparameters, and data in Sec. 4, supporting the reproduction of main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
17

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
will release the full codebase upon publication, including implementation details and training scripts necessary to reproduce the main experimental results. The training data includes combination of public datasets and internal data that cannot be released due to licensing constraints. We will provide clear documentation specifying how the datasets were used and how public data can be substituted to approximate our results.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
public/ mublic/guides /tt/dde/Coeeeuuniiieeeeeee/o/eee (Cooeeubiissiiiee heabbmmsssonoooooo.
ceiioonooiiiii Polilllyyy)). for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: Coo///ooddeeseseeeeeeeeeee / /nipss. httpcc/pppllccccc / wwwisssssnnpoooooo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification:
We specify the model configurations, hyperparameters, and data in Sec. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars would be too computationally expensive to report. We claim that gains in our experimental results are consistent and significant.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper specifies details about computer resources in Sec. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htappp::////uuuieeeee /ww.rrnur..../// cc ccc/publc//BBe Coubblc/Ethiccee http:/htthsiiiiiieeeeeeeeeeeeeae csCuidelinest
Answer: [Yes]
Justification: Ethics.
We make sure to conduct the research strictly following the NeurIPS Code of
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in Sec. 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all hirrd-prtty assets (e.g., code, datasets, pretrained models) used in the paper. While license terms explicitly listed in the current draft, will ensure that all licenses and terms of use are properly documented and respected in the final version upon publication.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: / //neruurpp.... ccc/onnferencee/222//11111 for what should or should not be described.
22


Reviews:
[{"review_id": "QnXeVdQuJX", "reviewer": "Reviewer_EJMS", "summary": "The paper introduce a unified autoregressive model for any-to-image generation. The input for the model is a discrete sequence of text and condition image (depth/semantic map), then, generate a sequence of discrete visual tokens. In addition, the model is capable of generating videos from text. The author also propose disentangle causal attention to mitigate information leakage.", "questions": "- The paper mentioned \"adopt Qwen2.5 as the text tokenier and transformer mode\", does that mean the author finetune a pretrained Qwen2.5 for image generation or initialized a model with similar architecture as Qwen2.5 and trained it from scratch? If the author finetuned  a pretrained model, how does it perform on natural language understanding tasks after \"fine-tuning\"? If the model is trained from scratch, what is the reasoning of not using pre-trained weights? \n- What is the inference speed of the introduced model? how is it compared to AR and Diffusion/Flow matching model of same scale?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n- The paper is well written and easy to follow.\n- The experiment and evaluation is fairly extensive.\n\n## Weaknesses:\n- The author claim the introduced model is the first AR for Any-to-Image generation, which is incorrect. There are several attempts that train an autoregressive image generative model for both text and images as input [1,2,3].\n- The author claim to achieve state-of-the-art on benchmarks, which doesn't seem to be true. For instance, their model gets (substantial) lower performance compared to several models of the same size both AR [4], Diffusion [5], and hybrid model [6].\n- The novelty of the paper is limited. The author simply extend the conventional next-token prediction on latent from VQ-VAE. Thus, I don't see how the model can tackle the inherited problem of previous AR/next-token image generation models e.g., slow inference speed and unidirectional generation.  The novelty is from the DCA, which is borderline in my point of view.\n- For me, the main purpose of using AR image generation over diffusion or VAR is the simplicity when implementing join training with language modeling task. This may enable several useful properties such as in-context learning and complex language instruction. The paper doesn't demonstrate any convincing reasons for using AR. In fact, the performance and speed seems to be worse than diffusion/flow matching.\n\n\n[1] Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks\n\n[2] Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action\n\n[3] Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining\n\n[4] Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling\n\n[5] SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers\n\n[6] Show-o: One Single Transformer to Unify Multimodal Understanding and Generation", "was_revised": true, "final_justification": "The author addressed most of my concerns in the rebuttal so I slightly increase my score", "rebuttal": "**Q1: The claim of \"the first AR for Any-to-Image generation\"**  \nA1: Thank you for the valuable comment. We agree that there have been prior autoregressive models that support multiple input modalities, including text and image. However, our goal is to develop a unified autoregressive visual generation model that explicitly accommodates three types of conditions, text, spatial, and visual contex, in a single framework. While models like Unified-IO and Unified-IO 2 indeed support text and spatial inputs, they do not explicitly unify visual context as a generative condition. Lumina-mGPT is trained with diverse inputs, but to the best of our knowledge, it only reports performance on text-to-image generation, without demonstrating generalization to the broader range of input conditions explored in our work. We will revise our claim in the paper to better reflect this nuance.   \n\n**Q2: The claim of \"achieving state-of-the-art on benchmarks\"**  \nA2: Thank you for your comment. We will revise the paper to more accurately characterize our results in comparison to prior work, below we would like to provide some important context regarding the comparisons mentioned: 1) diffusion models like [5] typically rely on additional text encoders that are typically not counted in the model size. For example, [5] uses a strong 2B-parameter text encoder (Gemma-2B), which is non-negligible compared to the diffusion models themselves. In contrast, our model uses a single causal transformer without external text encoders, keeping the architecture clean and parameter-efficient. 2) AR model [4] and hybrid model [6] are trained on much more data than our model. Specifically, our model is trained on approximately 56M text-to-image pairs and 13M examples for image editing and control generation. In contrast, [4] reports using around 140M samples, while [6] uses roughly 2B examples. Despite this significant parameter or data gap, our model achieves competitive performance across a range of benchmarks, especially given our unified support for diverse conditional generation tasks. \n\n**Q3: Novelty of this work**  \nThank you for the thoughtful comments. We would like to clarify the novelty and motivation of our work: we do not seek to alter the autoregressive modeling manner, but to build a unified AR framework that accommodates diverse conditional inputs. To alleviate the information leakage issue from condition tokens to content tokens in causal modeling, we introduce Disentangled Causal Attention (DCA), a training-time regularization scheme that carefully preserves the causal nature of AR generation while enabling the model to learn condition-aware generation without overfitting or shortcutting. Though lightweight by design, DCA is proved to be effective in improving the instruction following capability of our model, as evidenced by our ablation studies.  \n\n**Q4: The advantage of using AR for image generation**  \nThanks for pointing this out. Our choice of an autoregressive (AR) model is motivated by its ability to flexibly accommodate diverse input conditions within a single causal transformer, in addition to the advantage you noted. There is currently no clear consensus in the community regarding which family of generative models, AR, diffusion, or VAR, is fundamentally superior to visual generation. Each has its own trade-offs in terms of quality, extensibility, and inference speed. We believe AR models remain an important and under-explored direction, especially in the context of multi-conditional visual generation, and our work aims to push this frontier forward.  \n\n**Q5: Qwen initialization**  \nThank you for the question. We fine-tune a pretrained Qwen2.5 model for image generation tasks. Specifically, we adopt Qwen2.5 both as the text tokenizer and as the initialization for the transformer decoder. During training, we compute the autoregressive loss on the entire sequence, including both the text tokens (input prompts) and the visual tokens (generated image latents). \n\n|               | GenEval | MMMU |\n|---------------|---------|------|\n| Qwen2.5-0.5B  | - | 47.5 |\n| Pretrain w/o init | 0.29    | -    |\n| Pretrain w/ init  | 0.31    | 12.7 |\n\n- Performance with and without Qwen initialization: we conducted experiments using both Qwen-initialized and randomly initialized models and found that both setups can achieve similar performance on text-to-image (T2I) benchmarks after convergence (0.29 vs 0.31 on GenEval after 512 resolution pretraining). However, Qwen initialization leads to more stable training in the early stages, with lower loss.\n- Natural language understanding performance after T2I fine-tuning: after fine-tuning Qwen2.5 on T2I data, we observe a significant drop in language understanding performance, with MMLU accuracy decreasing from 47.5 to 12.7. This degradation is expected as we do not include any text-only data during fine-tuning. Although we compute loss on text tokens, the learning signal is relatively simple and repetitive, and lacks the diversity and complexity of language modeling tasks. We believe that the text understanding capability can be better preserved by mixing in text-only data for joint training, and we leave a systematic study of its impact on both language and T2I performance for future work.  \n\n**Q6: Inference speed**  \nA6: Thank you for the question. We evaluate the inference speed of our model alongside other models of comparable scale. Since Janus only supports 384×384 resolution, we only report its speed based on 24×24 tokens. As shown below, benefiting from the usage of KV cache, the inference speed of our model clearly outperforms Show-o. While our generation speed is still slower than SANA, deploying with vLLM could significantly narrow the gap.\n\n| Method              | # of Tokens | Speed (Sec/Image) |\n|---------------------|-------------|--------------------|\n| Janus (1.5B)        | 24×24       | 11.92              |\n| Show-o (1.3B)       | 32×32       | 269.10             |\n| SANA (1.6B)         | 32×32       | 1.77               |\n| Ours (1.5B)         | 32×32       | 43.55              |\n| Ours + vLLM (1.5B)  | 32×32       | 5.40               |"}, {"review_id": "ooSUm3WF2r", "reviewer": "Reviewer_SePx", "summary": "This paper introduces OmniGen-AR, the first unified autoregressive (AR) model that can generate images from a wide variety of inputs like text, segmentation maps, or video frames, all within a single framework. Its success comes from two key ideas: a shared tokenizer that turns all visual inputs into a common language, and a special training technique called Disentangled Causal Attention (DCA) that stops the model from simply copying the input. The model achieves state-of-the-art results on numerous tasks, proving that AR models are a powerful alternative to diffusion models for universal image generation.", "questions": "1. What is the total amount of training data used by the model? A brief comparison with other methods is needed.\n2. Is it too straightforward to mask out the condition? Is it possible to use loss constraints in a way that prevents information leakage?\n3. Are there more quantitative experimental comparisons between the 1.5B model and the 0.5B model?", "limitations": "See Questions", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Streangths:\n1. It is the first work to systematically develop and validate a unified autoregressive framework for such a broad range of conditional image generation and understanding tasks.\n2. The authors astutely identify \"information leakage\" as a novel and crucial challenge in a unified AR setting. The proposed Disentangled Causal Attention (DCA) is an elegant and effective solution.\nWeaknesses:\n1. The ablation study (Table 6) commendably reports that multi-task training, while beneficial for editing and spatial control, slightly degrades performance on the core text-to-image task. This indicates the presence of negative transfer or task interference, an important issue that is not fully resolved. It suggests that simply mixing all tasks together may not be the optimal strategy and that more advanced multi-task learning techniques might be necessary.\n2. There are still some meaningful ablation experiments that can be conducted.", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Total amount of training data and a brief comparison with other methods**  \nA1: Thanks for your suggestion, below we compare the training data with several baselines:\n\n| Method         | Unified | T2I  | T2V  | Editing                                                                                     | ControlGen                                     |\n|----------------|---------|------|------|---------------------------------------------------------------------------------------------|------------------------------------------------|\n| **LLamaGen** | ❎      | Laion (50M) + Internal (10M) | -    | -                                                                                           | -                                              |\n| **OmniGen**        | ✅      | DataComp(56M) + SA(11M) + Laion(4M) + ShareGPT4v(1.26M) + ALLaVA-4V(1M) + DOCCI(15) + DenseFusion(1M) + JourneyDB(4M) + Internal(16M) | - | MagicBrush + Instruct-Pix2Pix + SEED-Edit + SomethingSomething + HR-VITON + FashionTryon + StyleBooth | MultiGen |\n| **Unified-IO2**       | ✅      |  LAION-400M + CC3M + CC12M + RedCaps + OBELICS | YT-Temporal-1B + ACAV100M + AudioSet + WebVid-10M + HDVILA-10M + Ego4D | -                                              |\n| **EditAR**         | ✅      | initialized from LLamaGen  | -    | SEEDData-Edit-Unsplash(1.5M) + PIPE (1.8M), COCOStuff + MultiGen                         | -                                              |\n| **ControlAR**      | ✅      | initialized from LLamaGen | -    | -                                                                                           | LAION-Aesthetics, ImageNet + ADE20K + COCOStuff + MultiGen |\n| **Janus-Pro (1B)** | ❎      | Laion(12M) + ImageNet(1M) + SA(11M) + OpenImages(8M) + Megalith(8M) + YFCC(15M) + JourneyDB(4M) +  Dalle3-high-quality-captions(1M) + PixelProse(16M) + Internal(72M) | -    | -                                                                                           | -                                              |\n| **Show-O (1.3B)**  | ❎      | ImageNet(1.28M)+ CC12M + SA(11M) + Laion(35M) + DataComp + COYO(2B) + JourneyDB(4M) | -    | -                                                                                           | -                                              |\n| **Ours**           | ✅      | CC3M + CC12M + OpenImages(8M) + SA(11M) + Megalith(8M) + JourneyDB(4M) +  Dalle3-high-quality-captions(1M) + Internal(10M) | Panda + HD-VILA(9M) + OpenSora pexels(45k) + OpenVid-1M + Internal(0.5M)                      | MagicBrush + Instruct-Pix2Pix + SEED-Edit  | MultiGen |\n\n**Q2: Ablation studies on using loss constraints to prevent information leakage**  \nA2: Great suggestion! We conducted experiments to apply loss reweighting between condition and content tokens—specifically, assigning a lower weight (0.1) to condition tokens and a standard weight (1.0) to content tokens during training. The results below show that the proposed DCA outperforms loss reweighting on all benchmarks by clear margins.\n\n| Method   | VBench | Emu-CT | Mask  |\n|----------|--------|--------|-------|\n| baseline | 70.33  | 0.15   | 24.76 |\n| DCA      | 74.72  | 0.20   | 25.33 |\n| Loss     | 72.54  | 0.14   | 24.94 |\n\n**Q3: Quantitative experimental comparisons between the 1.5B model and the 0.5B model**  \nA3: Below we report more quantitative experimental comparisons between 0.5B and 1.5B models, 1.5B model consistently achieve better results than 0.5B model.\n\n| Model Size | FramePred | Emu-CT | Mask |\n|------------|--------------------|------------|----------|\n| 0.5B       | 613                | 0.20       | 25.33    |\n| 1.5B       | 429                | 0.23       | 35.28    |"}, {"review_id": "6R65kdy79z", "reviewer": "Reviewer_eCVT", "summary": "This paper presents OmniGen-AR, a unified autoregressive framework for any-to-image generation that can handle diverse conditional inputs including text prompts, segmentation masks, depth maps, and reference images within a single model. The key technical innovation is the use of a shared visual tokenizer for different visual conditions, and a Disentangled Causal Attention (DCA) mechanism that aims to prevent information leakage between condition tokens and content tokens during training, improving instruction-following behavior. OmniGen-AR is evaluated on a wide variety of tasks (text-to-image, text-to-video, image editing, frame prediction, depth-to-image, segmentation-to-image) and demonstrates strong performance across established benchmarks, even surpassing diffusion-based competitors with comparable or fewer parameters.", "questions": "1. Can you provide a more formal or theoretical justification for DCA beyond the intuition of “blocking leakage”? For example, how does it relate to causal language modeling guarantees or to other attention regularizers?\n2. In Section 4.4, you mention failures in removing objects or correctly following edits. Could you analyze why these errors happen — is it data sparsity, model capacity, or the autoregressive formulation? This would help readers understand the model’s true limitations.\n3. AR models are generally slower than diffusion models for image synthesis due to token-by-token decoding. Could you quantify the latency or token generation speed in practice, compared to diffusion models? This would be important for practitioners considering adoption.\n4. The results on segmentation-conditioned tasks are slightly worse than ControlAR or OmniGen. Do you have insights on how your approach might be adapted to improve geometric consistency in such spatially precise tasks?\n5. The paper mentions failure cases, but are there potential misuse or bias issues with a general-purpose any-to-image generator? Please consider adding more discussion on this front.", "limitations": "Yes, it's discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper shows a solid empirical evaluation over many tasks, with quantitative comparisons to both AR and diffusion baselines, as well as reasonable ablation studies.\n2. The writing is clear, with well-motivated design choices (e.g., why DCA is needed), and reasonable diagrams.\n3. The work demonstrates that autoregressive models can be scaled to unified, multi-conditional image and video generation, which is significant given most recent focus on diffusion models.\n4. While unifying multiple conditions is not a new direction, applying a shared visual tokenizer with disentangled causal attention within an AR framework is novel and technically interesting.\n\nWeaknesses:\n1. The empirical evaluation, although extensive, still shows weaker results on some metrics (e.g., segmentation-to-image mIoU is worse than ControlAR/OmniGen) and does not fully analyze why.\n2. The treatment of failure cases is rather brief; the analysis in Section 4.4 is minimal, leaving unclear how to address the observed editing failures.\n3. The DCA mechanism is an elegant solution, but the rationale and its theoretical justification are a bit ad hoc, relying on intuition rather than rigorous justification.\n4. The paper lacks deeper discussion on training cost, sampling latency, or practical deployment of a large AR model for real-time tasks, which would help judge its feasibility in practice.", "was_revised": true, "final_justification": "I thank the authors for the reply, and part of my concerns are resolved, hence I raise my rating accordingly. However, I agree with other reviewers' comments that the quality and novelty of the paper is yet to be improved.", "rebuttal": "**Q1: Weaker results on some metrics and further analysis**  \nA1: Thanks for pointing this out! We acknowledge that our model shows weaker performance on segmentation-to-image in terms of mIoU compared to ControlAR and OmniGen, and will add more analysis in the revised paper.\n- For ControlAR, it leverages a dedicated control encoder initialized from DINOv2-S, which provides strong representations tailored for visual understanding. In contrast, our model uses a unified causal transformer without condition-specific encoders, aiming for architectural simplicity and generality across a wide range of tasks—not only spatial control but also text, video, and editing-based generation. Moreover, ControlAR is limited to controllable generation tasks, while our framework supports a broader set of inputs, including text-to-image, image editing, depth, segmentation, and even text-to-video.\n- For OmniGen, the performance advantage stems in large part from the scale and diversity of its training data. OmniGen is trained on a massive self-constructed dataset, covering a wide spectrum of tasks including text-to-image (~80M), image editing, human motion, virtual try-on, style transfer, subject-driven generation (e.g., GRIT-Entity, Web Images), and structured control generation (e.g., LAION with depth, segmentation, pose, and canny annotations, RefCOCO, ADE20k, and ReasonSeg). In comparison, our model is trained on a more modest dataset: ~56M T2I samples and ~13M for editing and control generation, using only publicly available sources.\nDespite these differences, our model still achieves strong and competitive results across most benchmarks, while maintaining a clean, unified autoregressive architecture that handles diverse input modalities without task-specific heads or encoders. We believe this highlights the scalability and generality of our approach.  \n\n**Q2: The analysis in Section 4.4 is minimal, leaving unclear how to address observed editing failures**  \nA2: Thank you for pointing this out. We will expand Section 4.4 in the revised version to include a more detailed analysis of failure cases: As shown in Figure 8, failure cases can be broadly categorized into two types: 1) Instruction-following capability. For instance, in the first row of Figure 8, the instruction is “Remove the bag on the bench next to the person sitting at the bus stop”, but the model removes the person instead. This indicates a failure in grounding fine-grained spatial and referential cues from language into visual modifications. 2) Low-quality generations under sparse control signals. Examples in the second row (depth-to-image and segmentation-to-image) show blurry or structurally inconsistent results, which likely stem from noisy supervision and sparse training coverage for these conditions.\nThese failure modes suggest two potential directions for future work: 1) Scaling up the model and training data to build a stronger base model with improved generalization and instruction-following ability across diverse visual tasks. 2) Leveraging chain-of-thought (CoT) to improve the model’s reasoning ability on complex prompts.\n\n**Q3: The rationale of DCA mechanism**  \nA3: Thank you for your comment. Disentangled Causal Attention (DCA) is designed to address a concrete and critical issue in conditional autoregressive generation: the risk of information leakage from condition tokens to content tokens during training. In standard causal attention, the model may inadvertently learn to exploit trivial correlations or positional cues between condition and target tokens, rather than learning meaningful conditional generation.\n\nDCA introduces a principled modification to the attention mask, splitting it into condition and content causal regions. This separation enforces a clearer dependency structure, encouraging the model to learn robust alignment between inputs and outputs without relying on token overlap or order-based shortcuts. While DCA is simple in form, it is grounded in a clear rationale: to preserve the autoregressive training signal while regularizing the attention pathway in a way that reflects the asymmetric role of condition versus content tokens. Its effectiveness is consistently validated across multiple conditional tasks, as shown in our ablation results, and it plays a key role in enabling our model to generalize across diverse inputs.\n\n**Q4: Discussion on training cost, sampling latency, or practical deployment of a large AR model**  \nA4: Thanks for your valuable suggestions! We will include the discussion on training cost, sampling latency, and practical deployment in our revised paper:\n- Training cost: We train our model on 64 A100 GPUs. The 512 pretraining of 0.5B/1.5B models took 6/10 days separately, while the 1024 sft stage took 4/6 days.\n- Sampling latency and practical deployment: below we show that the AR visual generative model is compatible with the vLLM framework and deploying with it could significantly improve the inference speed.\n\n| Method           | # of Tokens | Speed (Sec/Image) |\n|------------------|-------------|-------------------|\n| SANA (1.6B)      | 32×32       | 1.77              |\n| Ours (1.5B)      | 32×32       | 43.55             |\n| Ours+vLLM (1.5B) | 32×32       | 5.40              |"}, {"review_id": "4pKcebUSyV", "reviewer": "Reviewer_xn72", "summary": "The authors propose OmniGen-AR, a unified autoregressive framework capable of generating images from various input modalities, including text prompts, spatial conditions (e.g., segmentation masks, depth maps), and visual contexts (e.g., image editing, frame prediction, video generation). OmniGen-AR utilizes a shared visual tokenizer for discretizing various visual inputs and incorporates a separate text tokenizer for text prompts. A key challenge addressed is the tendency of autoregressive models to overfit to input images in editing tasks, leading to poor instruction adherence. To address this issue, the authors propose Disentangled Causal Attention (DCA), an attention mechanism that probabilistically masks the input image during the training phase. This prevents the model from excessively relying on the input image, thereby enhancing its ability to follow instructions accurately.", "questions": "Please refer to W1, W2, and W3.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nS1. The authors tackle an important problem in image generation by proposing a unified framework that flexibly accommodates diverse input modalities.\n\nS2. The authors provide comprehensive experiments revealing that autoregressive image generation models tend to overfit to input images, leading to poor instruction adherence.\n\nWeakness\n\nW1. The paper lacks comparison with existing models.\nThe paper lacks a thorough review and comparison with existing autoregressive image generation models. In particular, EditAR [1] also encodes multiple visual modalities using a unified visual tokenizer and employs an autoregressive transformer for conditional generation tasks, including diverse image-editing operations. The authors should cite this work and clarify the differences between OmniGen-AR and EditAR. \n\n\n[1] Mu, Jiteng, Nuno Vasconcelos, and Xiaolong Wang. \"EditAR: Unified Conditional Generation with Autoregressive Models.\" arXiv preprint arXiv:2501.04699 (2025).\n\nW2. Experiments need to be improved.\nW2.1. The authors should include EditAR[1] as a direct competitor, as mentioned in W1, to support their claims on performance and generality.\nW2.2. Table 5 reports results only on text-to-video, image-editing, and mask-to-image tasks, omitting key tasks such as text-to-image, frame prediction, and depth-to-image. These omissions weaken the claim that DCA improves robustness across diverse generation tasks by reducing over-reliance on segmentation inputs.\nW2.3. Although the authors describe conceptual differences between DCA and classifier-free guidance [2], they do not provide any empirical comparison, making it difficult to assess the effectiveness of the proposed technique.\n\n\n[2] Ho, Jonathan, and Tim Salimans. \"Classifier-Free Diffusion Guidance.\" NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications.\n\nW3. The experimental analysis seems incomprehensive.\nW3.1. In Table 4 (right), ControlAR [3], which is cited as the most relevant baseline, outperforms the proposed OmniGen-AR on both mask-to-image and depth-to-image tasks. However, the authors do not address or explain this discrepancy, making it unclear what advantages OmniGen-AR offers.\nW3.2. In Table 6, the authors attribute performance degradation in T2I and T2V tasks under joint training to the “lower visual quality of editing and spatial-conditioned datasets,” but providing actual sample outputs would help validate this claim.\nW3.3. In Table 5, the effect of varying DCA probabilities is reported, but the underlying trade-offs (e.g., why performance drops after 10%) are not analyzed.\n\n[3] Li, Zongming, et al. \"Controlar: Controllable image generation with autoregressive models.\" arXiv preprint arXiv:2410.02705 (2024).", "was_revised": true, "final_justification": "I raised my rating from 3 to 4, as the rebuttal successfully addressed key concerns.\n\n**Resolved Issues:** \n- Provided comparisons with existing models\n- Additional results on text-to-image, frame prediction, and depth-to-image\n- Empirical comparison between DCA and classifier-free guidance\n- Analysis of the discrepancy with ControlAR\n- Analysis of DCA probability effects.\n\n**Points that would further strengthen the paper:** \n- Add direct experiments with EditAR beyond seg-to-img and depth-to-img, such as image editing and edge-to-image.\n- Provide a qualitative comparison among OmniGen-AR, EditAR, and ControlAR.", "rebuttal": "**Q1: Comparison with existing models**  \nA1: Thank you for pointing this out. We appreciate the reviewer bringing EditAR to our attention and agree that it is relevant work that should be cited and discussed. We will include a citation and clarify the distinctions in the revised version of the paper:\n\nWhile EditAR and our method both employ autoregressive transformers and support multiple conditional image generation tasks, they are fundamentally different in two aspects: 1) EditAR is specifically designed for image editing and low-level control tasks (e.g., depth-to-image, edge-to-image, segmentation-to-image), while OmniGen-AR is a unified Any-to-Image framework that handles a broader range of input modalities. 2) EditAR aims to improve text-image alignment by introducing distillation loss, while our model hopes to prevent information leakage through DCA, a novel training-time attention mechanism that disentangles condition and content attention paths.\n\n**Q2: More results on text-to-image, frame prediction, and depth-to-image**  \nA2: Thanks! We will add more quantitative results in the revised paper:\n\n| Method | T2I | VBench | FP  | Emu-CT | Mask | Depth |\n|--------|-----|--------|-----|--------|------|--------|\n| 0%     | 0.53 | 70.33 | 779 | 0.15   | 24.76 | 30.35 |\n| 5%     | 0.54 | 74.55 | 682 | 0.17   | 25.78 | 31.57 |\n| 10%    | 0.55 | 74.72 | 613 | 0.20   | 25.33 | 32.94 |\n| 20%    | 0.52 | 73.28 | 610 | 0.21   | 25.16 | 31.64 |\n| 30%    | 0.54 | 71.69 | 628 | 0.19   | 21.49 | 29.31 |\n\n**Q3: Empirical comparison between  DCA and classifier-free guidance**  \nA3: Thanks for your great suggestion! The proposed DCA is fundamentally different from CFG, as it does not directly drop the conditions. Instead, it calibrates the attention between condition and content tokens to allow the model better disentangle and leverage conditional information. Below we show empirical comparison between them. As can be seen, compared to CFG, DCA leads to significantly better results on tasks like frame prediction, editing, and controllable generation.\n\n| Method | T2I  | VBench | FP  | Emu-CT | Mask | Depth |\n|--------|------|--------|-----|--------|------|--------|\n| DCA    | 0.55 | 74.72  | 613 | 0.20   | 25.33| 32.94  |\n| CFG    | 0.56 | 72.63  | 854 | 0.17   | 23.43| 27.98  |\n\n**Q4: Analysis of the discrepancy with ControlAR**  \nA4: We acknowledge that ControlAR achieves higher mIoU on both mask-to-image and depth-to-image tasks. However, it’s important to highlight that ControlAR is specifically designed for controllable generation and relies on an additional control encoder (initialized from DINOv2-S), which is pretrained for dense visual understanding. This encoder introduces strong spatial priors that benefit structured control tasks but are task-specific and less generalizable to other modalities.\n\nIn contrast, OmniGen-AR maintains a unified architecture without any task-specific heads or encoders, supporting a wider variety of inputs, including text, segmentation, depth, canny edges, image editing, and video frame prediction, all using a single causal transformer. While we may not outperform ControlAR on a few specific spatial tasks, we offer greater flexibility, simplicity, and generality, which are essential for Any-to-Image generation.\n\n**Q5: Visualization of editing and spatial-conditioned datasets**  \nA5: Thanks! We will add this to our revised paper.\n\n**Q6: Analysis of the effects of DCA probabilities**  \nA6: We thank the reviewer for raising this point. While DCA helps prevent condition-content leakage during training, applying it too frequently disrupts the model’s ability to fully leverage causal dependencies, especially when condition tokens carry rich information (e.g., dense segmentation or image context). In our experiments, 10% DCA strikes a balance: it encourages robust conditioning without significantly limiting the model’s access to informative context. We will update the paper to include more detailed analysis."}]

Author rebuttal:

Meta-review: This paper proposes OmniGen-AR, a unified AR framework for any-to-image generation that can take as input diverse conditions including text prompts, segmentation masks, depth maps, and reference images within a single model. The key technical innovation is the use of a shared visual tokenizer for different visual conditions, and a Disentangled Causal Attention (DCA) mechanism to prevent information leakage between condition tokens and content tokens during training, improving instruction-following behavior.

The strengths of the work include clear motivation (unified AR model with any input), moderate novelty (shared visual tokenizer with DCA), elegant solution (DCA), solid empirical evaluations, and clear writing.
Some negative aspects include some weaker results on some metrics, some unaddressed failure cases, lack of theoretical justification, lack of computational cost analysis, etc., most of which are solved by the author rebuttal. Therefore, all the reviewers are finally satisfied with the work and give Accept (x1) and Borderline Accept (x3). Based on the above analysis, the AC think it deserves an Accept.
Decision: Accept (poster)