ORION: Decoupling and Alignment for Unified Autoregressive Understanding and Generation
Authors: Taihang Hu
Mengting Chen
Jinsong Lan
Xiaoyong Zhu
Kaifu Zhang
Ming-Ming Cheng
Bo Zheng
Yaxing Wang
Venue: ICLR 2026 Poster (2026)
Keywords: unified generation and understanding
Primary area: foundation or frontier models, including LLMs

Abstract: Unified multimodal Large Language Models (MLLMs) hold great promise for seamlessly integrating understanding and generation. However, monolithic autoregressive architectures, despite their elegance and conversational fluency, suffer from a fundamental semantic–structural conflict: optimizing for low-level reconstructability in generation leads to catastrophic forgetting of high-level semantic understanding. We present ORION, a unified framework that resolves this conflict through Decoupling and Alignment. A non-linear vision head decouples structural pressures from shared representations, while a novel Representation Consistency Loss explicitly aligns semantics during generation. Together with a curated progressive training recipe and high-quality multimodal data, our method enables balanced optimization of both capabilities. Built purely on a monolithic autoregressive backbone without task-specific separate parameters, ORION achieves performance on par with or exceeding recent state-of-the-art unified models that rely on more complex designs. These results validate monolithic autoregression as a simple, effective, and competitive path toward truly integrated multimodal intelligence.

Published as a conference paper at ICLR 2026
ORION: DECOUPLING AND ALIGNMENT FOR UNIFIED AUTOREGRESSIVE UNDERSTANDING AND GENERATION
Taihang Hu'; Mengting Chen 2* Jinsong Lan', Xiaoyong Zhu?, Kaifu Zhang?
Ming-Ming Cheng", Bo Zheng"i, Yaxing Wang
VCIP, College of Computer Science, Nankai University Alibaba Group
{ (hutaihang00) @ @ggaall.com {cmm, yaxing) @ mankai.edu.cn {cm(27 1286, jinsonglan.ljs @aaibaba-inc.com
ABSTRACT
Unified multimodal Large Language Models (MLLMs) hold great promise for seamlessly integrating understanding and generation. However, monolithic autore- gressive architectures, despite their elegance and conversational fluency, suffer from a fundamental semanticstructtuaal conflict: optimizing for low-level recon- structability in generation leads to catastrophic forgetting of high-level semantic understanding. We present ORION, a unified framework that resolves this conflict through Decoupling and Alignment. A non-linear vision head decouples structural pressures from shared representations, while a novel Representation Consistency Loss explicitly aligns semanties during generation. Together with curated progres- sive training recipe and high-quality multimodal data, our method enables balanced optimization of both capabilities. Built purely on a monolithic autoregressive backbone without task-specific separate parameters, ORION achieves performance on par with or exceeding recent state-of-the-an unified models that rely on more complex designs. These results validate monolithic autoregression as a simple, effective, and competitive path toward truly integrated multimodal intelligence.
INTRODUCTION
In recent years, multimodal artificial intelligence has advanced significantly along two parallel tracks. On one hand, Multimodal Large Language Models (MLLMs) have demonstrated powerful capabilities in complex cross-modal understanding tasks(Hurst et al., 2024; Liu et al., 2023; Bai et al., 2025; Chen et al., 2024b). On the other hand, diffusion models have achieved unprecedented realism and controllability in image generation(Ho al., 2020; Rombach al., 2022; Podell et al., 2023; Esser et al., 2024; Wu et al., 2025c; Zhang et al., 2023; Huang et al., 2024; Wu et al., 2025a). However, this functional separation of understanding and generation capabilities represents a critical bottleneck on the path toward general-purpose multimodal intelligence.
Fusing these two capabilities within a unified framework is the next frontier in the field. Recent explorations have primarily revolved around three architectural choices, as shown in Fig. 1. Cascaded architectures( Wu et al., 2025a; Lin et al., 2025a; Pan et al., 2025: Liu al., 2025) serially connect understanding and generation models, essentially using the MLLM as a multimodal text encoder for the diffusion model, which makes difficult to support native multi-round dialogue with interwoven text and images. Parallel pechtectures(Deng et al., 2025; Shi et al., 2024; Mo et al., 2025) decouple the parameters required for generation and understanding, but face challenges of high training costs and scalability. In contrast, a monolithic autoregressive architecture(Sun et al., 2023; Tong et al., 2024; Team, 2024), capable of processing and generating arbitrarily interleaved image-text sequences in end-to-end manner, may be the most promising path forward. This approach treats images as another "language", predicting them through a unified autoregressive process, which provides the natural framework for fine-grained, controllable, and conversational content creation. Although prior works like Emu(Sun et al., 2023) and Chameleon(Teaall (Team, 2024) have explored this direction, their performance has not yet reached satisfactory level, suggesting that we still need
PPojecc leader. Co-corresooding authors.

---

Published as a conference paper at ICLR 2026
to explore more effective training strategies to unlock the full potential of monolithic architectures. Among the challenges, one of the most acute and prevalent is that when a powerful, pre-trained MLLM is generatively fine-tuned by adding regression loss for visual tokens, the model's original understanding capabilities suffer from catastrophic forgetting.
We formalize this problem representational conflict between Semantic Fidelity and Structural Reconstructability within the model. Understanding tasks, driven by a cross-entropy loss, pursue high-level semantic fidelity, which requires the model's hidden representations to be highly separable in semantic space to facilitate classification. In contrast, generation tasks, driven by Mean Squared Error (MSE) loss, pursue structural reconstructabillity, which demands that hidden representations contain sufficient low-level informationcan to precisely reconstruct their coordinates in a continuous embedding space. These two conflicing optimization objectives create a "tug-of-war" within the shared representation space. Our experiments provide clear evidence for this conflict. As shown in Fig. 3, naive generative fine-tuning causes severe semantic drift, with categorical predictions for visual tokens deviating from the plausible text semantics of the base model. Likewise, the ablation study (Tab, 4, first row) shows that simple full-parameter fine-tuning yields poor understanding and generation performance, underscoring the impact of the semanticstructural conftict.
To address this, we propose a systematic framework designed to reconcile this conflict. Methodologi- cally, the core of this framework lies in decoupling and alignment. First, we decouple the direct pressure on the shared representations from these two tasks using high-capacity non-linear vision head. This vision head is dedicated translating the high-level semantic information from the LLM backbone into precise, low-level structural vectors. Second, we introduce a representation consistency supervision mechanism that explicitly aligns the confficting objectives during optimization, ensur- ing the model does not lose its semantic understanding of visual content while learning structural reconstruction. To effectively implement this framework, we further construct a complete and effi- cient training regime. This includes: a curated three-stage progressive training strategy designed to introduce the generation task to preserve pre-trained semantic knowledge smoothly; a comprehensive collection and cleaning of high-quality, open-source datasets for both image understanding and generation, providing a solid foundation for learning complex tross-modal representations; and a series of engineering optimizations to ensure the efficient training of this complex system.
Evaluations on widely-used multimodal understanding and generation benchmarks Yue et al., 2024; Liu et al., 2024c; Ghosh et al., 2023) show that model, with native autoregressive monolithic architecture and no separated parameters, achieves performance in both image understanding and gen- eration that is comparable to recent unified models and surpasses previous monolithic architectures.
The main contributions of this paper are as follows:
Conceptually, we are the first to identify and define the "semantic-strune sttuutura representation conflict" in unified autoregressive MILMS and argue that the monolithic autoregressive architecture is key path toward supporting native multi-round image-text dialogue.
Technically, we propose and validate a methodology of "decoupling and alignment" complemented by a curated, complete progressive training strategy and data support, forming systematic solution, and will open-source all training code, data, and models to give back to the community
Performatively, under a monolithic autoregressive architecture without separable parameters model achieves combined understanding and generation performance comparable to that recent works using other architectures, strongly validating the significant potential of this technical route.
RELATED WORK
2.1 UNIFYING GENERATION AND UNDERSTANDING
Recent efforts to unify visual generation and understanding have pursued three primary architectural paradigms (Fig. I): CaCaeaddd Architectures. This approach (Wu et al., 2025a; Lin et al., 2025a) cascades Multimodal Large Language Model (MLLM) with separate diffusion model, where the MLLM acts as a text encoder to guide image synthesis. While leveraging specialized models, this design cannot comprehend its visual outputs. precluding multi-tur interaction. 2) Parallel Architectures. These models (Deng et al., 2025; Shi et al., 2024; Mo et al., 2025; Lin et al., 2025b) use partially independent modules for different tasks within unified framework. For instance, some
2

---

Published as a conference paper at ICLR 2026
Image
Image
Image
Diffusion
Diffusion
Text
VAE
Joint Attention
MILMM
MILM
Und. Branch
Gen. Branch
Text
Image
Text
Image
e.g., QwenImage Edit, UniWorld,
e.g., BAGEL, LmFusion, X-fusion
e.g,, Emu, Chameleon, MctaMorphy, X- X-Omni
OmniGen2, Stepi-x
Input Text Tokens
Input Image Tokens
Predicted Image Tokens
Figure 1: Three main architectural for recent unified models. (Left) Cascaded Architectures (Middle) Parallel Architectures. (Right)Monolithic Autoregressive Models
only share attention mechanisms between vision generation and understanding (Deng et al., 2025; Shi et al., 2024). This strategy typically results in large parameter counts and complex training pipelines. 3) Monolithic Autoregressive Architectures. This paradigm (Sun al., 2023; Team, 2024; Kou et al., 2024; Geng et al., 2025) processes interleaved text and image sequences with a single set of parameters under a unified autoregressive objective. Images are handled either by predicting continuous embeddings for separate decoder (Sun et al., 2023) or by using discrete visual tokens for end-to-end training (Team, 2024). This unified design is elegant and naturally supports seamless multi-turn dialogue.
2.2 CONTINUOUS TOKENS VERSUS DISCRETE TOKENS
Early autoregressive models like LLamaGen(Sun al., 2024) and Lumina-mGPT(Liu et al., 2024b) used a VQ-Van(Van Den Oord et al., 2017) to discretize images, framing generation as a text- like prediction task with unified cross-entropy loss. However, these VQ-VAE tokens prioritize low-level textures over high-level semantics, hindering the image understanding of models like Chameleon (Team, 2024). More recently, X- X-Omni(Geng et al.. 2025) attempted to mitigate this by discretizing the outputs of a semantically-riche encoder (c.g., SigLIP(Zhai et al., 2023)) and using diffusion decoder for image synthesis. While this improved multimodal comprehension, it still lagged behind methods using continuous semantic tokens. To maximally preserve multimodal understanding, our work leverages continuous image tokens.
2.3 PREDICTING IMAGE TOKENS AUTOREGRESSIVELY OR IN PARALLEL
MetaQuery(Pan et al., 2025) and Seed-X(Ge et al., 2024) simplify training by using a fixed set of query tokens to parallel regress image tokens. However, this creates a critical representational mismatch: the model generates from proxy tokens but must understand real image tokens. This inconsistency prevents the model from comprehending its own output, thus failing in coherent, multi-turn dialogues. In contrast, the purely autoregressive approach of serially predicting each visual token, while more challenging to train, ensures representational consistency, which crucial for enabling seamless multimodal interaction.
METHODOLOGG
3.1 OVERVIEW OF MODEL ARCHITECTURE
The overall model architecture is depicted in Fig. 2, which is built upon pretrained autoregressive MILMM. This model is capable of processing arbitrarily interleaved sequences text and images. At the input stage, text is tokenized into sequence of text tokens T = (t1, 12. fn). while images are first encoded by a visual encoder into a series of continuous visual embedding. which are then mapped into sequence of visual tokens V = {V1, U2.... Um) decoder that autoregressively predicts the next token based on all preceding tokens, whether visual or textual. We denote the output from the final hidden layer of the Transformer at step t he, Within our unified framework, the model must handle two fundamental prediction tasks, supervised by the following two loss functions:
The core of the model is Transformer
3

---

Published as a conference paper at ICLR 2026
That Claodiccainn Head
Viearn Regrnaiio Hoal
tokens
Prsteine XLL.X
Last hiblee
Text Read
COOD
Autoregressive Und. and Gen. Model
diffusion decoder
(riPan))
Laci laddonn
Test Heal
Vision Encoder
Modil
Utiffed
the uninal caterqillar metamurplooss
Figure 2: Method Overview. (Left) Our unified autoregressive model processes mixed and visual tokens using two heads; a Text Classification Head for understanding and a Vision Regression Head to predict visual tokens for image synthesis. (Right) The Representation Consistency Loss prevents semantic drift by aligning our model's text output with frozen pretrained model at visual token positions, ensuring semantic integrity during generation.
Text Understanding Loss (CCr): When the next token to be predicted is text, the model computes probability distribution over the vocabulary via a classification head (i.e., the language model's s output embedding matrix). This process is supervised by the standard cross-entropy loss, which aims to maximize the log-likelihood the correct text token. This constitutes the model's semantic learning objective.
CCE
Nsent Σ
1
(1)
Image Generation Loss (LMSE): To endow the model with generative capabilities, when the next token be predicted is visual token, we require the model directly regress its embedding vector. This process is supervised by the mean squared error loss, which minimizes the Euclidean distance between the predicted vector 0j+1 and the ground-truth vector 2/+1. This forms the model's '' structural learning objective.
LMSE
1
Σl1((h,) 1y+1l12
Nvisson
Here, f(+) represents the vision head that maps the hidden representation hy to the predicted visual token.
Additionally, we pretrain a separate diffusion model to act as an image decoder. This decoder takes the autoregressively generated sequence of visual tokens (01,.... 0m conditional embedding render the final pixel-level image during inference.
3.2 REPRESENTATION ALIGNMENT
Representation Conflict. When training the model by naively combining Lee and LMSE, we observe a severe degradation in its image under- standing capabilities post-training. This degra- dation is also reflected a significant shift in its internal visual representations. As shown in Fig. 3, the original base model's text classification head can map visual tokens reasonable textual semantics. However, after naive generative fine- tuning, the model's 's classification predictions for the same visual tokens devolve into nonsensical outputs, indicating a profound semantic shift. To address this, we propose two key improvements, intervening at the decoder architecture and loss function levels.
naive fine-tunine
Pretrained MLLM
last hillen
Text Head
6ir
deg
Figure 3: Semantic Drift. The rich semantic predic-
tion for visual token from pretrained MLLM col- lapses into a meaningless distribution after naive generative fine-tuning
Vision head. Prior works such as Emu (Sun al., 2023) and Nexus-Gen (Zhang et al., 2025) use linear vision head, creating a representational bottleneck. This directly propagates "seman- tically blind" pressure from the MSE gradient to the shared L.M representation h4. We draw
4

---

Published as a conference paper at ICLR 2026
inspiration from the text token prediction process, where the path from the hidden state to the final token (hidden + logits (up-projection) + sof tmax (activation) + embedding (cokkkupdooow-pooeeeeeee
can be viewed implicit MLP regression. We therefore replace the this linear layer with a single-hidden-layer MI.P head. This MLP serves as a nonlinear structural decoder, acting key-value memory(Geva al., 2020) map the high-level semantic h, to low-level visual token vectors. This design relaxes constraints on hy; the MLLM no longer needs linearly decodable semantic space, but only needs output sufficient information generate the next visual token. We find well-pretrained MLP regression head significantly mitigates the decline image understanding performance.
Representational Consistency Loss (CKL). At positions where visual tokens predicted, the model lacks explicit semantic supervision from the cross-entropy loss. We therefore introduce a representational consistency loss. The core idea is that when the model regresses visual token, must also simultaneously comprehend the textual semantics corresponding to that visual token. We achieve this through knowledge distillation, using frozen, original base model as a teacher. During training, at each visual token position j, we enforce that the probability distribution Pstudent (w/ (w)))) produced by our student model's text classification head should remain consistent with the distribution Pheacher ( (u(hh) produced by the teacher model. This consistency is measured by minimizing the KI.- divergence between the two distributions.
LKL
Σ seniisioniindices
(Peeacee (w]hy || Pstudent (1))
Nvision
This loss term acts as a powerful semantic anchor, ensuring that as the model pursues structural reconstructabilityy its representations do not deviate from meaningful semantic trajectory. In doing so, it aligns the two conflicting optimization objectives.
Overall Training Objective.
The final loss function is a weighted sum three loss terms:
Goaal
LCB + AMSE + XKL
where AMSE and λKI are hyperparameters that balance the contributions of the different tasks.
3.3
MODEL TRAINING
3.3.1 DATA CURATION.
Our training corpus encompasses a diverse range of data types to ensure balanced development across all model capabilities. We carefully organize the public dataset to adapt to multi-task, which includes:
a) Understanding Data: To maintain and enhance comprehension abilities, we utilize the publicly available LLVVA-OV dataset(Li et al., 2024), which comprises 3.5M samples.
b) Text-to-lmage Data: For training text-to-image generation, we filter the pre-training and fine- tuning datasets from BLIP-3o(Chen et al., 2025a), yielding 20M image-text pairs. Additionally, to ensure prompt diversity and high image quality, we leverage FLUX. 1-devLLabs, 2024) regenerate 5M images from the LAION-COCO aesthetic dataset(Guangyil 2022), then recaption by Qwen2.5
VL.Bai et al., 2025).
Image Editing Data: To cover various image editing tasks, we incorporate 1.2M samples from the UniWorld(Lin et al., 2025a).
3.3.2 PROGRESSIVE TRAINING RECIPE.
We repurpose the diffusion decoder from Nexus-Gen(Zhin et al., 2025), which is tasked with image reconstruction conditioned by visual token sequences encoded by the VIT. To smoothly inject generative capabilities into the pretrained Qwen2.5- its original understanding abilities, we adopt a curated multi-stage training recipe using a dynamic mixture of the curated data described above. Tab. shows the configuration of different training stages in detail. We carefully adjusted the data and hyperparameters of each stage to achieve better performance.
et al., 2025) while maximally preserving
5

---

Published as a conference paper at ICLR 2026
Table I: Detailed hyperparameters for different Training recipe
Hyperparameters
Warm-up
Pretrain
SFT
Max learning rate
1 x 10 3
1.0 x 10
1.0 x 10
Min learning rate
1 1 10-3
1.0 10-5
LR scheduler
Constant 0.1
Cosine decay
Cosine decay
Weight decay
0.1
0.1
Gradient norm clip
1.0
1.0
1.0
Optimizer
A (3 = 0.9, 82 = 0.95, c <= 1.0 x 10 10-15)
Loss weight (CE: MSE KL)
0:1:0
1:1:0.01
1:1:0.01
Warm-up ration
5%
5%
5%
Training epoch
2
2
Batch size(all devices)
2048
4096
4096
Gen resolution (min and max long side)
(252,252)
(252, 252)
(252, 252)
Und resolution (min and max long side)
(56, 1024)
(56, 1024)
Stage I: Vision Head Warm-up. We freeze the full MLLM backbone and exclusively train the newly added MLP vision head. The training objective in this stage is solely the MSE loss (CMSE). This step is designed to sufficiently warm up the MLP, equipping it with a preliminary capability for structural prediction and thereby laying solid foundation for subsequent joint training. This stage used 20M images with lower quality but diverse text-to-image data to sufficiently warm up the vision head, which we found to be crucial for subsequent performance.
Stage 2: Full-Parameter Pretrain. In this phase, we unfreeze all model parameters and conduct fine-tuning using data mixture heavily weighted towards understanding tasks. The primary objective to adapt the entire model the new architectural components and tasks while strongly anchoring its original semantic understanding capabilities. We use high-quality 3.5M image understanding and 5M text-to-inn to-------imme-eee data for this phase.
Stage 3:
Multi-task Supervised fine-tuning. This is the final stage. We employ a comprehensive mixed dataset incorporating all data types (understanding, generation, and editing). The goal is to balance and jointly enhance all the model's diverse task capabilities in unified manner. This stage introduces 1.2M image editing data based on the Stage 2.
4
EXPERIMENTS
4.1 EXPERIMENT SETUP
Implementation details. Our model is built upon the 7B Qwen2.5-VI. (Bai et al., 2025) backbone and the Nexus-Gen (Zhang et al., 2025) diffusion decoder. For generation, we represent images as 81 visual tokens, following Zhang et al. (2025), while preserving native multi-resolution inputs for understanding tasks. We train the model using the DeepSpeed ZeRO-3 framework (Aminabadi et al., skip training steps with anomalous gradients (Liu et al., 2024a)
2022). For stability and efficiency, and employ sequence packing (Krell et al., 2021) to maximize GPU throughput.
Baselines. We compare our model against three categories of state-of-the-art (SOTA) baselines: MILMSs like Qwen2.5-VL. (Bai et al., 2025) and VLLVVAOV (Li al., 2024), to assess the preservation of understanding abilities; 2) Generation-Only Models such as SDXL (Podell et al., 2023) and Flux. dev (Labs, 2024), to benchmark image synthesis quality; and 3) Unified Models spanning the cascaded, parallel, and monolithic paradigms (Sec. 2.1) for direct, holistic comparison.
1) Understanding-
4.2 RESULTS
Text-to-Image Generation We conduct a comprehensive quantitative evaluation of our model's text-to-image (T2I) generation performance on the established GenEval benchmark(Ghosh et al., 2023). As presented in Tab. 2, the quantitative results indicate that our model achieves highly com- petitive performance on this benchmark. Firstly, surpasses previous monolithic unified models and demonstrates competitive performance against the latest counterparts, establishing new performance benchmark for this technical approach. Furthermore, our model exhibits capabilities comparable to recent unified models that employ more complex architectures and is superior to several strong

---

Published as a conference paper at ICLR 2026
Text to image
Foods that squirrela love Clothes suitable for going winter
The animal that caterpillar transforms into
Classic Thunksgiving delicacies
The national flag of the country where Yellowstone National Park located
Image Editing
Change
stip
Change Ere style
Figure 4: Overview of image generation capabilities. Our model can not only generate images from prompts (first two rows), but also generate images based world knowledge reasoning (third row). addition, it has certain local editing and global editing capabilities (bottom row).
generation-only baselines. This result provides strong evidence that the monolithic architecture, when optimized via our proposed framework, is fully capable of achieving top-tier generative quality without compromising its architectural simplicity and conversational fluidity.
Beyond the quantitative metrics, Fig. provides qualitative showcase of our model's synthesis capabilities. The examples in the first two rows demonstrate high prompt fidelity, with the model accurately rendering specified object compositions and complex scene layouts. Moreover, the model showcases impressive generalization when processing abstract prompts that require real-worke knowledge, testifying to the effective synergy between the MLLM's reasoning core and the visual generation module(third row). More detailed results can be found in Appendix Fig. 6.
Image Understanding We evaluated our model's multimodal understanding performance across five comprehensive benchmarks: MMBench(Liu et al., 2024c), MMStar(Chen et al., 2024a), MMVet(Yu Yu al., 2023), SEED-Bench(Li et al., 2023), and RealWorldQA((XAA, 2024). As presented in Tab. 3, the results show that our model not only largely preserves the strong understanding capabil- ities of base model, Qwen2.5-VL, but also significantly outperforms all prior monolithic unified models. This key result validates that our proposed framework successfully enables image generation while effectively mitigating catastrophic forgetting, thereby overcoming the critical performance trade-offs that have previously hindered the development of monolithic approaches.
Image Editing Following multi-task SFT, our model also demonstrates certain image editing capabilities (Fig. 4, bottom row), performing both local(object add) and global(color change/style transfer) instruction instruction-guided guided edits while preserving high fidelity in non-edited regions.
7

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation of text-to-image generation ability on GenEval benchmark, following (Deg et al., 2025) we use LLM rewriter.
Method
Single
Two
Counting
Colors
Position
Color Attr.
Overall
Gen. Only Models
(PXlePo al., 2023)
0.98
0.74
0.39
0.85
0.15
0.23
0.55
DALLe3(Betker 2023)
0.87
0.47
0.83
0.43
0.45
0.67
DDS-medium(Esser al., 2024)
0,94
0.72
0.89
0.33
0.60
0,74
PLUX.I-dev(LAn)) 2024)
0.98
0.75
0.93
0.68
0.65
0.82
Unified Models
Cascaded Architectures
OmniGen2 Wu al., 2025h)
0.99
0.96
0.74
0.98
0.72
0.75
0.86
MetaQuery(Pan et 2025)
0.80
BLIP3-o( et al., 2025a)
0.84
UniWorld-Ve (Ln al., 2025a)
0.99
0.93
0.81
0.89
0.74
0.71
Parallel Architectures.
Mogao(Liao et al., 2025)
1.00
0.97
0.83
0.93
0.84
0.80
0.89
BAGEL(Deng al., 2025)
0.98
0.95
0.84
0.95
0.78
0.77
0.88
Monolithic Architectures
Janus-Pro(Chen et al., 2025b)
0.89
0.59
0,90
0.79
0.66
0.80
Eman(Wang al., 2024)
0.99
0.42
0.80
0,49
0.45
0,66
Show-o2(Xie al., 2025)
0.87
0.58
0.92
0.52
0.62
Chameleon(Team. 2024)
0.39
Orthus( Kou al., 2024)
Ours
1.00
0.86
0.66
0.93
0.69
0.77
0.82
Table 3: Comparison with state-of- the-an methods multimodal understanding benchmarks.
Method
Base MILMM
MMB
MMStarf
MMVet?
SEED?
RWQAT
Und. Only Models
Qwen2.5 (L(Bai et al., 2025)
79.1
63.9
67.1
79.5
68.5
LLaVA-OV(Li al. 2024)
80.8
61.7
57.5
75.4
66.3
Unified Models
Cascaded Architectures
OmniGen Wu et al., 2025h)
Qwen2.5-V1. 7B
79.1
63.9
67.1
79.5
68.5
MetuQuery(Pan et al., 2025)
Qwen2.5-VL. 7B
79.1
63.9
67.1
79.5
68.5
ELIP3-o(Chen et al., 2025a)
Qwen2.5-VL. 7B
79.1
63.9
67.1
79.5
68.5
UniWorld- VI et al., 2025a)
Qwen2.5-VL VL 7B
79.1
63.9
67.1
79.5
68.5
Parallel Architectures.
Mogao(Liao et al., 2025)
75.0
74.6
BAGEL(Deng et al., 2025)
Qwen2.5-VL 7B
85.0
67.2
Monolithic Architectures
Banus-Pro(Cheee (Chen al., 2025b)
DeepSeck-LLMM 7B
79.2
50.0
72,1
Emu3(Wane et al., 2024)
EmuChat 7B
58.5
37.2
68.2
57.4
Xh(xe al., 2025)
Qwen2.5-7B
79.3
56.5
69.8
Chameleon Team, 2024)
From scratch 7B
35.7
8.3
Orthus Kou et al., 2024)
Chameleon-7B
Ours
Qwen2.5-VL. 7B
83.7
64.2
64.5
78.1
67.4
4.3 EMERGENT CAPABILITIES OF THE UNIFIED REPRESENTATION
Beyond quantitative benchmarks, our unified training framework gives rise several zero-shot emergent capabilities. These abilities, not explicitly present the training data, serve as evidence for the generalization power of the learned unified representation.
Zero-shot Interleaved Generation and Understanding, The model can engage multi-round dialogue that seamlessly interleaves generation and understanding (Fig. 5, first row). It can generate an image and immediately answer questions about its own creation, despite being trained only on decoupled single-tun annle--tun generation and multi-tum understanding data. We attribute this zero-shot capability to the alignment of text and image representations; the model treats its own generated visual tokens understandable context for the next round.
8

---

Published as a conference paper at ICLR 2026
understanding generation
User: What the most
Generute smulpture
User: Generate imagee of Fiffel Tower the evening
you deseribe
firwwrrk night sky the original imagee. Output:
famous aculyture in Boudl??
image?
Output:
Output:
Output:
Output:
One Brazil's most famous seulptures the statue of Cluist the Redeemer near Coparabona Beach in Rio de lanctro, officially named "Chvrist the Redlecmer of Janeiro," 'Standing. meters tall
This picture show the beautiful secnery of the Eiffel Tower dusk. The Eiffel Tower, illuminoted by lights, appears particularly magniffomnt The displays gradient light blue and pink, offering
tranquility and the foreground,, lights, adding depth soeme.
largest
landmark
round
Third
Multilingual image generation
In context multi image edit
WORLD
REL Bi0 0 C4
RB 18.21922 2: WESOOLE12R1
A cov bokii Twilie eign, and sige
Transforms Imaget into the sbye of Imageez.
knd 25 & No 2
Figure 5: The zero-shot emergent capabilities.
Cross-lingual Generalization. Trained exclusively on English text-to-image data, the model exhibits zero-shot cross-lingual generation from prompts in other languages like Chinese and Japanese (Fig.5, second row, left). This capability is inherited from the powerful multilingual alignment already present in the Qwen2.5-VL backbone and successfully extended the generative modality by our framework, showcasing robust generalization.
Contextual Multi-image Editing. Similarly, the model generalizes from its training on single-image editing data to performing contextual edits across multiple images in single turn (Fig.5, second row, right). This underscores the strong compositional properties endowed by the unified representation space, allowing learned skills to be applied to more complex, novel scenarios.
4.4 ABLATION STUDIES
We systematically investigate the impact of two core components: the vision head architecture and our proposed representation consistent loss LKL- The results are presented in Tab.4.
Table 4: Ablation Study: Impact of Vision Head and Training Strategies.
Config Vison head
Lxz
Stage Data
Understand"; MMB
Gen.
Impact of Vision Head Architecture and Warm-up. The vision head is responsible for decoding the MLLM's hidden states into vi- sual tokens. We compare proposed MLP head against two alternatives: a simple Linear layer and
MMS
GenEval
Baselines and Altematives
A
Linear
5M
71.6
54.3
0,62
B
Lineur
x
20M
71.5
56.8
0.65
Q Former
X
5M
77.3
60.7
0.75
complex (-rammerrprrammttrr- matched with MLP). When warm-up vision head with relatively small dataset of 5M sam- ples, the Q-Former (Config C: 77.3 MMB) shows a slight advantage understanding met- rics over the MLP head (Config E: 76.4 MMB). However, as we scale the head Stage-1 data to 20M samples, the MIP head's performance (Config F) not only closes the gap but clearly surpasses that of the Q-Former (Config D) across all metrics. This suggests that while Q-Former may more safple-efficient initially, the MLP exhibits superior scaling properties and generalization potential. It underscores our conclusion that sufficiently warm-up vision head is critical to unlocking the full capabilities of the monolithic framework.
Former
X
20M
78.6
59.6
0.77
Our Proposed Approach
E
MLP
X
None
74.1
57.5
0.73
E
MLP
X
SM
76.4
59.3
0.76
MLP
X
20M
79.8
61.0
0.79
G
MLP
20M
83.7
63.2
0.82
Efficacy of LKL- The second key component of our framework is the LKL, designed to mitigate the cemantic-structure conflict. By comparing final model (Config G) with its direct counterpart trained without this loss (Config F). the inclusion L yields substantial and consistent gains across the board: remarkable +3.9 point increase MMBench and +2.2 2 MMStar for understanding.

---

Published as a conference paper at ICLR 2026
coupled with solid +3.0 improvement in the GenEval score for generation. This concurrent enhancement of both understand and generative tasks provides strong empirical evidence for our central hypothesis: the LK1. 1. effectively aligns the representations required for both tasks, ensuring that the model maintains semantic fidelity even as it learns to perform structural reconstruction.
5 CONCLUSION
We tackled the core challenge of unifying multimodal understanding and generation in monolithic autoregressive models, where a semanticstructural conflict often leads to catastrophic forgetting. To address this, introduced framework of Decoupling and Alignment, combining non-linear vision head, a representation consistency loss, and a progressive training recipe. Our approach enables purely monolithic model to preserve strong understanding while achieving competitive generation and editing performance, surpassing prior monolithic designs and rivaling more complex unified models. These results demonstrate that monolithic autoregression, when carefully aligned, is simple yet effective path toward integrated multimodal intelligence.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed -inerennee enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking. Storage and Analysis, pp. 1-15. IEEE, 2022.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint Stt::2502.1392... 2025.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science. htlps://cdn. openai. com/papershuulle-3. pdf. 2(3):8, 2023.
Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: family of fully open unified multimodal models-architecture, training and dataset. arXiv preprint arXiv:2505.0955 2025a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056-2706 2024a.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling, arXiv preprint arXiv:2501. 17811, 2025b.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024b.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. 14683, 2025.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari. Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404. 24X::22004.4336. 2024.
Zigang Geng, Yibing Wang, Yeyao Ma, Chen Li, Yongming Rao, Shuyang Gu, Zhao Zhong, Qinglin Lu, Han Hu, Xiaosong Zhang, et al. X-omni: Reinforcement learning makes discrete autoregressive image generative models great again. arXiv preprint arXiv:2507.22055: 2025.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint 20Xv:20012.14913. 2020.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36: 52132-52152, 2023.
Guangyil. laion-coco-aesthetic. 2022. URL http::://uggiinffe...o//ttttstss
laion-coca coco- aecco-aesthetc.. Hugging Face Datasets.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jie Hu, Shen, and Gang Sun. gqueeze-and-excitee networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
11

---

Published as a conference paper at ICLR 2026
Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Huanzhang Dou, Chen Liang, Yutong Feng. Yu Liu, and Jingren Zhou. In-context lora for diffusion transformers. arXiv preprint arXiv:2410.23775 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Siqi Kou, Jiachun Jin, Zhihong Liu, Chang Liu, Ye Ma, Jian Jia, Quan Chen, Peng Jiang, and Zhijie Deng. Orthus: Autoregressive interleaved image-text generation with modality-specific heads. arXiv preprint arXiv:2412.0012 2024.
Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. Efficient sequence packing without cross- cross-contamination: Accelerating large language models without impacting performance. arXiv preprint arXiv:2107.02027, 2021.
Black Forest Labs. Flux. https://gitie www.:ttt........aaaaaaeeeea www.://aaaaooorettaaaaa..s forest-labs/flus 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang. Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.04 03332. 2024.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench. Bench- marking multimodal Ilms with generative comprehension. arXiv preprint 2rX1v:2307.16125, 2023.
Chao Liao, Liyang Liu, Xun Wang. Zhengxiong Luo, Xinyu Zhang, Wenliang Zhao, Jie Wu, Liang Li, Zhi Tian, and Weilin Huang. Mogao: An omni foundation model for interleaved multi-modal generation. arXiv preprint 2XXv:2505.05472 2025.
Bin Lin, Zongjaan Li, Xinhua Cheng, Yuwei Niu, Yang Ye. Xianyi He, Shenghai Yuan, Wangbo Yu, Shaodong Wang, Yunyang Ge, al. Uniworld: High-resolution semantic encoders for unified visual understanding and generation. arXiv preprint arXiv:2506.03 2025a.
Han Lin, Jaemin Cho, Amir Zadeh, Chuan Li, and Mohit Bansal. Bifrost-1: 1: Bridging multimodal lims and diffusion models with patch-level clip latents. arXiv preprint arXiv:2508. 059554 2025b.
Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Chase Lambert, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving text- to-image alignment with deep-fusion large language models. arXiv preprint arXiv:2409.10695, 2024a.
Dongyang Liu, Shitian Zhao, Le Zhuo. Weifeng Lin, Yi Xin, Xinyue Li, Qi Qin, Yu Qiao. Hongsheng Li, and Peng Gao. Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining. arXiv preprint arXiv:2408. 02657, 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916 2023.
Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, al. Steplx-edit: A practical framework for general image editing. arXiv preprint a50iv:2504.1 17761, 2025.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et Mmbench: your multi-modal model an all-around player? In European conference computer vision, pp. 216-233, Springer, 2024c.
Sicheng Mo, Thao Nguyen, Xun Huang, Siddharth Srinivasan Iyer, Yijun Li, Yuchen Liu, Abhishek Tandon, Eli Shechtman, Krishna Kumar Singh, Yong Jac Lee, et al. X-fusion: Introducing new modality to frozen large language models. arXiv preprint vrXiv:2504.2099) 2025.
Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, et al. Transfer between modalities with metaqueries. arXiv preprint arXiv:2504.062. 062566, 2025.
12

---

Published as a conference paper at ICLR 2026
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and puttern recognition, pp. 10684-10695, 2022.
Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu. Lmfusion: Adapting pretrained language models for multimodal generation. arXiv preprint arXiv:2412.15188, 2024.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 04t1::2006.062255, 2024.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang, Emu: Generative pretraining in multimodality. arXiv preprint urXiv:2307.05222, 2023.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09815 09818, 2024.
Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint 11X::22412.14144 2024.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, al. Emu3: Next-token prediction is all you need. arXiv preprint 2409 18869, 2024.
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report. arXiv preprint arXiv:2508.02.22 2025a.
Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang, Yexin Liu, Junjie Zhou, et al. Omnigen2: Exploration to advanced multimodal generation. arXiv preprint arXiv:2506. 1555066.88777 18871, 2025b.
Ge Wu, Shen Zhang, Ruijing Shi, Shanghua Gao, Zhenyuan Chen, Lei Wang. Zhaowei Chen, Hongcheng Gao, Yao Tang, Jian Yang, al. Representation entanglement for generation: Training diffusion transformers is much easier than you think. arXiv preprint arXiv:2507.01444 066,77 2025c.
X.AI. G15 vision preview, 2024. URL httph: ai/blag/bbbgggeee watpg//rooo11.510
Jinheng Xie, Zhenheng Yang. and Mike Zheng Shou. Show-o2: Improved native unified multimodal models. arXiv preprint arXiv:2506. 15564, 2025.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.022 02490, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training In Proceedings of the IEEE/CVE international conference on computer vision, pp. 11975-11986 2023.
13

---

Published as a conference paper at ICLR 2026
Hong Zhang, Zhongjie Duan, Xingjun Wang, Yuze Zhao, Weiyi Lu, Zhipeng Di, Yixuan Xu, Yingda Chen, and Yu Zhang. Nexus-gen: A unified model for image understanding, generation, and editing. arXiv preprint arXiv:2504.21356, 2025.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference computer vision, pp. 3836-38444 2023.
A APPENDIX
A.1 LIMITATIONS AND FUTURE WORK
Limitations While our model achieves competitive performance in both understanding and gen- eration within a purely monolithic and autoregressive framework without separate image encoders or task-specific parameters, this architecture exhibits notable limitations. The primary challenge lies in image editing tasks that require high fidelity the source image. The visual representations fed to the diffusion decoder are derived from a semantic-rich vision encoder. Consequently, these representations, while excellent for generation, lack the fine-grained, low-level structural details necessary for precise, consistency-preserye engineering solutions, such as concatenating low-level VAE tokens from the source image into the diffusion decoder, this approach introduces considerable complexity. For instance, in multi-turn dialogue, determining which image a user intends to edit becomes a non-trivial challenge. The root of this issue lies in the absence of image tokenizer that excels both semantic understanding and pixel reconstruction. As the research community progresses towards this goal, our proposed framework is designed seamlessly integrate such future advancements.
edits. Although this can be partially mitigated through
A second limitation pertains to generative diversity. Since visual tokens are predicted via regression, the generation process is deterministic and lacks the inherent stochasticity of sampling-base based methods. We contend that this is not major drawback. A potential solution is to structure the training data such that the model learns to take a concise user prompt and generate more descriptive, elaborated prompt, which then conditions the image output. Diversity can then be achieved by sampling from the distribution of these elaborated prompts, effectively leveraging the language model's generative variety.
Future Work The primary advantage of the monolithic autoregressive architecture its natural ability to support multi-turn, interleaved understanding and generation. Encouragingly, our model demonstrates this capability as an emergent property, even without being trained on relevant data. However, this conversational proficiency is still nascent and requires further development. Our immediate future work will focus on constructing a dedicated dataset for interleaved multimodal dialogue and subsequently fine-tuning the model to explicitly enhance these interactive capabilities.
A.2 DATASET CONSTRUCTION DETAILS
In addition to leveraging publicly available datasets, we constructed a high-quality synthetic dataset of million eext-to-mmaee pairs to enhance our training corpus. This dataset was generated through a carefully designed three-stage pipeline to ensure both conceptual diversity and high image-text alignment.
Initial Prompt Generation: create diverse collection of prompts covering a wide range of visual concepts, first employed the Qwen2.5-VL 7B model(Bai et al., 2025) to recaption images from the LAION-COCO aesthetic dataset(Guannge 2022).
2. Image Synthesis: Next, these generated captions were used prompts for the FLUX.1- dev(Labs, 2024) model to synthesize a new, high-quality set of images.
3. Final Recaptioning: Finally, to ensure strong text-image consistency, performed second recaptioning step. We used Qwen2.5-VL 7B once more to generate clean and accurate descriptions for the newly synthesized images.
14

---

Published as a conference paper at ICLR 2026
This pipeline ensures that our final dataset possesses both rich visual diversity and robust image-text aligmment. The training data for generative tasks follows a conversational format, similar to that used for understanding tasks. An example is provided below.
Data Format Example
"id": 0,
"i "image": "./000. jpg", " "conversations"
"from": "human", "value": "Can you make an image of ancient library hidden in a deep forest, where bookshelves reach the canopy.
},
{
"from": "gpt", "value": "<image>"
}
},
To enhance the model's robustness natural language, the human-provided prompts in the "conversations" array were constructed from a diverse set of templates. These templates cover range of interaction styles, including polite requests (e.g., "I'd d like direct commands (e.g., "Generate an image of ......),. "), and questions (e.g., "Can you produce picture of of..."). This variety ensures the model is not overfitted a single command structure.
a drawing of... "),
A.3 EXPLORATION OF AN ALTERNATIVE APPROACH
In alternative approach, explored leveraging features from multiple layers of the MILM backbone, motivated by the hypothesis that different layers capture visual-semantic information at varying levels of granularity. We implemented a mechanism compute a learnable, weighted average of the hidden states from all layers, like SENet(Hu et al., 2018). This aggregated feature vector was then passed the vision regression head.
Interestingly. this method achieved lower final convergence loss during training compared to our baseline approach, which uses only the final layer's representation. Despite this promising training dynamic, the resulting model exhibited complete failure generalize during inference, rendering it incapable of performing ext-to-image synthesis. We hypothesize that this phenomenon attributable to the compounding of prediction errors during autoregressive inferencea discrepancy between training on ground-truth embeddings and inferencing on self-generated ones. plausible that the minor errors in each predicted token are propagated and amplified through the network's 'ss successive layers. Our weighted- aaeereiiiveriiigggg mechanism may then inadvertently aggregate and exacerbate these emors from across the network's depth, leading to catastrophic breakdown of the generative process.
A.4 VISION HEAD ARCHITECTURE
The vision head is implemented
Lingle-hidden-layer Multilayer Perceptron (MLP). We utilize the GELU activation function and found empirically that setting the intermediate layer's dimension eight times that of the input dimension yielded strong balance of performance and efficiency. Further expansion of this intermediate dimension offered only marginal performance gains.
In our ablation studies (see Section 4.4), this MLP architecture was benchmarked against Q-Former of a comparable parameter count. The Q-Forerr which consisted of stacked self-attention layers, was configured to use single query token to autoregressively predict the subsequent visual token.
15

---

Published as a conference paper at ICLR 2026
A.5 MORE QUALITATIVE RESULTS
Fig.6 presents diverse array of images generated by our model. These results highlight the model's capability to produce high-fidelity images across a range resolutions, all closely adhering to their corresponding textual descriptions.
Figure Visualization results of Ours model at various resolutions
A.6 STATEMENTS
Broader Impacts While our work can enhance creative tools and human coman-c interaction, acknowledge the significant societal risks. The technology could be exploited to generate
16

---

Published as a conference paper at ICLR 2026
misinformation, and the model may perpetuate societal biases learned from web-scale training data. We encourage the community to focus on developing robust safeguards and mitigation strategies for these potential harms.
We acknowledge the ethical risks of misuse and bias inherent in this work. To promote transparency and enable further research into model safety and fairness, we commit the full open-soureing of our code, models, and newly curated data. We advocate for the responsible development and deployment of this technology within the research community.
Ethical Statement
Reproducibility Statement. To ensure full reproducibility, research pipeline publicly available. We will release all training code, curated dataset files, and final model checkpoints upon publication. Our model is built upon the publicly accessible Qwen2.5-VE backbone and the diffusion decoder from Nexus-Gen. All experiments were conducted using publicly available datasets, including LLaVA-OV, BLIP-30, LAION-COCO, and UniWorld, detailed in the methodology section.
commit making entire
LLM usage statement large language model was utilized solely as writing assistant help polish the manuscript and improve the clarity and fluency of the English prose. The LLM was not involved in any core aspects of the scientific work, including the formulation of the initial hypothesis, experimental design, data analysis, or the derivation of our findings and conclusions. All intellectual contributions were made exclusively by the authors.
17


Reviews:
[{"review_id": "n9SH2SycAm", "reviewer": "Reviewer_FMab", "summary": "This paper proposes ORION, a unified autoregressive model that aims to reconcile the inherent conflict between semantic understanding and structural reconstruction in multimodal large language models (MLLMs). The core claim is that a \"semantic-structural representation conflict\" hinders monolithic architectures, which the authors address via \"decoupling and alignment.\" Specifically, they introduce a non-linear MLP vision head to decouple structural prediction and a Representation Consistency Loss to align semantics during generation. The model, built on Qwen2.5-VL and a Nexus-Gen decoder, is evaluated on standard benchmarks, showing competitive results in both understanding and generation without task-specific parameters.", "questions": "- Can you provide quantitative results on a standard image editing benchmark to support the claim \"certain image editing capabilities.\"  \n- Can you design an ablation that separates  \"decoupling\" and \"alignment?\" For instance, applying the KL loss to a model with a linear head to see if \"alignment\" alone can compensate for poor \"decoupling.\"\n- Can you provide a discussion of the total training cost compared to simpler fine-tuning or other unified models (e.g., parallel architectures).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clearly formalizing the semantic-structural conflict in unified autoregressive models.\n- The model is evaluated on a wide range of standard benchmarks for both understanding and generation.\n- The paper is well-written and easy to follow.", "weaknesses": "- The paper fails to discuss and contrast its approach with other recent works that pursue a similar monolithic, autoregressive vision (such as GLaMM or Emu3).\n- Presenting image editing as a result without quantitative evaluation.\n- The ablation study does not full isolate the mechanisms of \"decoupling\" and \"alignment.\"\n- The paper lacks of discussion on training cost. The progressive three-stage training, while effective, appears computationally intensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d3oWfapzM6", "reviewer": "Reviewer_YV3K", "summary": "The paper proposes ORION, a unified multimodal framework that keeps a monolithic autoregressive backbone while attempting to reconcile a central tension between semantic fidelity for understanding and structural reconstructability for generation. The method introduces two key pieces. First, a non-linear MLP vision head decouples low-level regression pressure from the shared backbone. Second, a representation consistency loss uses a frozen teacher to align the student’s text-prediction distribution at visual token positions, aiming to prevent semantic drift during generative training. The approach is trained with a progressive three-stage recipe that warms up the vision head, then performs full-parameter pretraining with an understanding-heavy mix, and finally multi-task supervised fine-tuning that includes editing. Experiments report competitive text-to-image scores on GenEval and strong preservation of understanding across MMBench, MMStar, MMVet, SEED-Bench, and RealWorldQA, alongside ablations that attribute gains to the MLP head and the consistency loss.", "questions": "1. Teacher details for the consistency loss. What teacher is used exactly, and does it share weights with the initialization of the student backbone. Do you match logits with a temperature or probabilities directly. How sensitive are results to the loss weight λKL and to masking strategies at visual token positions.\n2. Data governance. Please specify data sources, licenses, and any filtering for copyrighted or sensitive content. Clarify whether FLUX regenerated images and Qwen recaptions appear in the test distributions and how you prevented leakage into GenEval or understanding benchmarks. \n3. Robustness and safety. Did you assess failure modes like hallucinated text in images, biased generations, or unsafe outputs during editing. Any guardrails applied during training or inference would be useful to report.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper isolates a concrete failure mode in unified autoregressive models, namely semantic drift during generative fine-tuning, and frames it as a representation conflict. The proposed remedy is conceptually simple yet targeted. The MLP head reduces direct gradient pressure on shared representations, and the representation consistency loss anchors semantics where cross-entropy supervision is absent. The training curriculum is pragmatic and clearly described, including data composition and hyperparameters, which improves reproducibility. Reported results indicate that ORION narrows the gap between monolithic models and more complex unified systems, while maintaining multi-turn interleaving abilities. The ablations are helpful and show that both the warm-up and the consistency loss matter, with measurable gains on understanding and generation metrics.", "weaknesses": "First, the representation consistency loss depends on a frozen teacher distribution at visual token positions. The paper should clarify whether the teacher sees exactly the same tokenization and masking, and whether temperature scaling or label smoothing is used. Without that detail, it is hard to judge stability and potential confirmation bias, since the student is nudged toward the teacher’s pretraining priors.\nSecond, the training recipe relies on large data mixtures that include regenerated images and recaptions. The paper should detail data licensing, deduplication against evaluation sets, and the exact share of synthetic captions. This is important for fairness and for assessing potential overfitting to benchmarks. \nThird, although the ablation table is useful, it collapses several factors at once. For example, Q-Former vs MLP comparisons change the warm-up corpus size, and the effect of the loss weights is not isolated. A factorial ablation on λ values and warm-up sizes would make the causal story more convincing. \nFinally, some claims about emergent multi-turn and cross-lingual behavior are illustrated qualitatively. These are compelling, but a small, controlled quantitative probe would help, for instance by measuring understanding accuracy on the model’s own generations and by using non-English GenEval-style subsets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MAl8JX878d", "reviewer": "Reviewer_QFw6", "summary": "This paper introduces ORION, a unified framework that resolves the conflicts between understanding and generation through two decoupled vision heads for semantical and generative objectives separately.\nThe core contribution is:\n1. The authors explicitly define the \"semantic-structural representation conflict\" in unified autoregressive MLLMs.\n2. To resolve the conflict, the authors decouple semantic-preserving and low-level generation objectives by two vision heads:\n+ The semantic-preserving is realized by aligning the semantic prediction distribution using the raw language heads with a frozen MLLM teacher of the same initial weights, with a proposed Representational Consistency Loss, i.e., $\\mathcal{L}_{KL}$.\n+ The low-level generation objective is realized by aligning the outputs of an additional vision head with the input embeddings of the MLLM, i.e., $\\mathcal{L}_{MSE}$.\n3. The authors dedicatedly curated a progressive training pipeline to warm up the vision heads and conduct full-parameter training.\n\nThe authors conduct experiments on several benchmarks and show that ORION can achieve SOTA generation ability while preserving its base MLLM's understanding ability.", "questions": "1. Can you add more discussion about the difference between your work and SEED-X?\n2. Can you show results on:\n+ Semantics-oriented benchmarks other than GenEval, for instance, WISE and DPG-Bench.\n+ Non-semantics-oriented benchmarks, such as MJHQ-30K and MS-COCO, these benchmarks should reveal if ORION learns aesthetic-aligned representations from a low-level view.\n\n**I am willing to raise my score based on other reviewers' comments as well as the rebuttal quality.**", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper writing is very clear and easy to follow.\n2. The figures are easy to read.\n3. All designs, i.e., the vision regression head and the $\\mathcal{L}_{KL}$, are rationalized well.\n4. Promising results on both understanding and generation benchmarks.", "weaknesses": "1. **Incremental to SEED-X without discussion**. \nThis work adopts nearly the same architecture as SEED-X[1] without discussing the differences. Please correct me if I misunderstand something:\n+ Both SEED-X and ORION receive native tokenized texts and ViT embeddings as the inputs.\n+ For the text side, they are exactly the same, i.e., supervised by the Next-Token-Prediction objective.\n+ For the vision side\n    + SEED-X regresses the $N$ learnable queries with $N$ ViT embeddings using MSE to foster generation ability. They ignore the raw $N$ corresponding vision tokens because at that time, MLLMs are not well-developed, and the backbone is purely text-based, i.e., Llama-2. Therefore, the vision tokens do not have a decodable semantic space. \n    + Since this work adopts a pre-trained MLLM as the backbone, the $N$ vision tokens become meaningful (decodable), and the learnable queries are no longer needed. Therefore, ORION switches to directly regress the $N$ vision tokens using the same MSE loss. \n+ Both SEED-X and ORION fed vision outputs to a diffusion decoder to produce pixel output.\n\nI recognize the contribution of the naturally extended $\\mathcal{L}_{KL}$ objective because the vision tokens are decodable under the MLLM-as-backbone scenario. But for other parts, it looks like simply switching the regression target from $N$ queries (SEED-X) to $N$ raw vision outputs (ORION), which seems a bit incremental to SEED-X.\n\n[1] SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation\n\n2. **Overclaimed contribution.**\nI do not agree with the authors' claim:\n\n*\"Conceptually, we are the first to identify and define the “semantic-structural representation conflict” in unified autoregressive MLLMs\".*\n\nThere is a semantic-structural representation conflict between understanding and generation has become a consensus in the community. Several well-known works have already raised such an issue. For example, in Janus[1] (which is not cited by this paper):\n\n```\nThe output of understanding task not only involves extracting information from images but also involves complex semantic reasoning. Therefore, the granularity of the vision encoder’s representation tends to mainly focus on high-dimensional semantic representation. By contrast, in visual generation tasks, the main focus is on generating local details and maintaining global consistency in the image. The representation in this context necessitates a low-dimensional encoding that is capable of fine-grained spatial structure and textural detail expression. Unifying the representations of these two tasks within the same space will lead to conflicts and trade-offs.\n```\n\nMany other papers regarding unified models have similar statements, so the authors should not claim they are **the first one** to discover it. \n\n[1] Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation\n\n3. **Inaccurate description of Cascaded Architectures.**\n\nIn L115-L116, the authors comment:\n\n*The MLLM acts as a text encoder to guide image synthesis. While leveraging specialized models, this design cannot comprehend its own visual outputs, precluding multi-turn interaction.*\n\nBut many MLLM-based unified models receive multimodal input instead of pure text. Though some query-based works, e.g., MetaQuery, cannot directly understand the query, they indeed can understand the output images. By combining the output image with previous conversations and new user instructions as a multimodal input, they can also realize multi-turn interaction.\n\n4. **Misclaimed architecture type.**\n\nThe authors claim ORION is a monolithic autoregressive model, but the generation requires a separately trained diffusion model, i.e., Nexus-Gen. From Figure 1, there is no difference between the Cascaded Architectures (Left) and the Monolithic Autoregressive Models (Right). From my perspective, true Monolithic Autoregressive Models should not rely on external diffusion models, e.g., Transfusion[1] and Show-o2[2]. ORION should be categorized as a Cascaded Architecture. \n\n[1] Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model\n\n[2] Show-o2: Improved Native Unified Multimodal Models\n\n5. **Does not truly resolve the conflicts.**\n\n+ From Table 3, we can observe that compared to Qwen2.5-VL-7B, the performance on most benchmarks, i.e., MMVet, SEED, and RWQA, shows a clear degradation, indicating that the semantic-structural representation conflict is not truly resolved. \n+ From Figure 4 Image Editing part, the results are quite similar to those of MetaQuery, where the objects and backgrounds are significantly altered. For instance, the background of the robot becomes much brighter, and the robot's shape is changed. For the second image, the buildings on the left are totally different after editing. This evidence suggests that the learned representations do not truly preserve low-level details; instead, they still focus on semantic accuracy, and the model is doing regeneration instead of editing.\n\n6. **Potential benchmark hacking because of the usage of BLIP-3o training data**.\n\nBLIP-3o training data is a highly controversial one in the community, because it directly distills the GenEval benchmark. Training on such a corpus and evaluating on GenEval is unfair for other baselines, e.g., Janus-Pro and Show-o2. Also, as a unified model, evaluating only on GenEval is not sufficient. Please consider evaluating on other benchmarks, including those non-semantic-oriented ones, to show the true generation quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)