Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Heads
Authors: Siqi Kou
Jiachun Jin
Zhihong Liu
Chang Liu
Ye Ma
Jian Jia
Quan Chen
Peng Jiang
Zhijie Deng
Venue: ICML 2025 poster (2025)
Keywords: multimodal foundation model
interleaved image-text generation
diffusion
Primary area: deep_learning->foundation_models
TL;DR: Orthus is a unified multimodal model for interleaved image-text understanding and generation. It generates content across modalities by routing the outputs from its shared transformer backbone to modality-specific heads.

Abstract: We introduce Orthus, a unified multimodal model that excels in generating interleaved images and text from mixed-modality inputs by simultaneously handling discrete text tokens and continuous image features under the \textbf{AR} modeling principle. The continuous treatment of visual signals minimizes the information loss while the fully AR formulation renders the characterization of the correlation between modalities straightforward. Orthus leverages these advantages through its modality-specific heads---one regular language modeling (LM) head predicts discrete text tokens and one diffusion head generates continuous image features. We devise an efficient strategy for building Orthus---by substituting the Vector Quantization (VQ) operation in the existing unified AR model with a soft alternative, introducing a diffusion head, and tuning the added modules to reconstruct images, we can create an Orthus-base model effortlessly (e.g., within 72 A100 GPU hours). Orthus-base can further embrace post-training to craft lengthy interleaved image-text, reflecting the potential for handling intricate real-world tasks. For visual understanding and generation, Orthus achieves a GenEval score of 0.58 and an MME-P score of 1265.8 using 7B parameters, outperforming competing baselines including Show-o and Chameleon.

Orthus:
Autoregressive Interleaved Image-Text Generation with Heads
Siqi Kou" Jiachun Jin'
Zhihong Liu Chang Liu
Ye Ma Jian Jia
Quan Chen
Peng Jiang?
Zhijie Deng'
Abstract
1. Introduction
Multimodal models have shown promise in image-to-text and/or text-to-image generation, with LLaVA (Liu al., 2024d;c), Emu2 (Sun et al., 2024c), and NExT-GPT (Wu et al., 2013) as popular examples. These abilities are essen- tial for handling complex real-world understanding and gen- eration problems. Yet, existing approaches can suffer from significant modeling redundancy due to the trivial combina- tion of specialized large models (e.g.. CLIP-ViT (Radford et al., 2021), Stable Diffusion (Rombach et al., 2022), and LlaMa (Touvron al., 2023a;b)). Doing so also undermines the benefits brought by cross-modal learning and introduces considerable inefficiency for both training and inference.
We introduce Orthus, a unified multimodal model that excels in generating interleaved images and text from mixed-modality inputs by simultane- ously handling discrete text tokens and continu- ous image features under the AR modeling prin- ciple. The continuous treatment of visual signals minimizes the information loss while the fully AR formulation renders the characterization of the correlation between modalities straightforward. Orthus leverages these advantages through its modality-specifft heads one regular language modeling (LM) head predicts discrete text to- kens and one diffusion head generates continu- ous image features. We devise an efficient strat- egy for building Orthus-by substituting the Vec- tor Quantization (VQ) operation in the existing unified AR model with soft alternative. intro- ducing diffusion head, and tuning the added modules to reconstruct images, we can create an Orthus-bass model effortlessly (e.g., within 72 A100 GPU hours). Orthus-base can further em- brace post-training to craft lengthy interleaved image-text, reflecting the potential for handling intricate real-world tasks. For visual understand- ing and generation, Orthus achieves GenEval score of 0.58 and an MME-P score of 1265.8 using 7B parameters, outperforming competing baselines including Show-o and Chameleon. Our code is available at https:/ http:://github. htggttgttb.cooo com/ zhi taiijjie-rrou group//rthus.
There is ongoing interest in jointly modeling visual under- standing and generation with unified, compact model. One strategy is map both images and texts to discrete tokens for simple autoregressive (AR) modeling (Liu al., 2024f; Team, 2024; Wang et al., 2024) (left of Figure 1). However, the image tokenizer, often equipped with a vector quanti- zation (VQ) bottleneck, can cause inevitable information loss and easily lead to suboptimal performance on vision tasks concerning high-frequency details (e.g., OCR and hu- man face generation). Alternatively, recent works, including Transfusion (Zhou et al., 2024) and Monoformer (Zhao et al., 2024) (middle of Figure 1), propose to integrate AR modeling on discrete text tokens and diffusion modeling continuous image features within single transformer. Nonetheless, the nature diffusion modeling to process noisy images (Ho al., 2020) makes the joint modeling of image-to- exxt-to-image, and more complicated inter- leaved image-text challenging.
This paper proposes Orthus' to bridge the gap. Orthus con- joins lossless continuous image features and the unified, cross-modal AR modeling by decoupling diffusion from the transformer backbone. This circumvents the noise dis- turbance and renders the characterization of the correlation between modalities straightforward, making it more suitable for interleaved image-text modeling. Specifically, Orthus
"Work done during an internship at Kuaishou Technology. 'Qing Yuan Research Institute, Shanghai Jiao Tong University 2 "Kuaishou Technology. Correspondence to: Zhijie Deng <zhi- jied @ood@jjtuu sjtu.edu.cn>.
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Orthus a loyal two-headed guard dog Greek mythology.

---

Orthus: Autoregressive Interleaved Image- Teage-Text Generation with Modality-Speciffe Heads
LM head
Dif fusion head
LM head
Diffusion (full atm)
Autoregressive (cousal attn)
Autoregressive (causall attn)
Autoregnessive (cousal att))
PEOO
Figure/ 1. Comparison of existing unified multimodal models with Orthus. Left: Fully AR models (Liu et al., 2024f; Team. 2024; Wang et al., 2024) convert visual signals to discrete image tokens via vector quantization for joint modeling with text tokens, but this causes information loss. Middle: AR-diffusion mixed models (Zhou et al., 2024; Xie et al., 2024) perform next-token prediction for text generation and image patch denoising for image generation, but the involved noise disturbance on images makes the concurrent image-to-tens text and text-to-image generation challenging. Right: Orthus operates in a fully AR manner while circumventing vector quantization and noise disturbance to preserve input information and modeling flexibility.
embeds both discrete text tokens (from an off off-the-shelf to- kenizer) and continuous patch-wise image features (from a pre-trained variational autoencoder (Kingma & Welling, 2013)) into the same representation space, where an AR transformer is then invoked to model the inter- and intra- modality interdependence. On top of the backbone, Orthus defines two medality-specifit heads, with one the regular language modeling (LM) head to predict discrete text tokens and the other as novel diffusion head to craft continuous image features. During inference, Orthus autoregressively predicts the next text token or image patch according to the indication of special transition tokens.
ing capabilities for unseen tasks. Furthermore, Orthus demonstrates a strong ability to generate logically coherent interleaved image-text content with high relevance. For vi- sual understanding and generation, Orthus is substantially superior to Chameleon and Show-o (Xie et al., 2024) across multimodal understanding and generation tasks. Notably, Orthus achieves GenEval (Ghosh et al., 2024) accuracy of 0.58 and POPE score of 79.6, even surpassing specialized text-to- tomge models SDXL (Podell et al., 2023) and the performant MnstructBIIIP-133 (Dai et al., 2023).
To summarize, our contributions are as follows:
Notably, the investigation into diffusion head for superior image generation draws striking analogy to the recent masked AR (MAR) approach (Li et al., 2024), yet with focus shift from image-only generation to mixed mixed-modality one. On the other hand, our Orthus differentiates from MAR and its variant (Yang et al., 2024a) in that it characterizes the correlation with fully AR formulation instead of mask- based modeling, which avoids expensive hyperparameter specification and eases the modeling of interleaved data.
We introduce Orthus for interleaved image-text genera- tion. Orthus models the correlation between modalities through AR principle and generates discrete text tokens and continuous image features with dedicated heads.
We propose an efficient strategy to build Orthus by ex- ploiting its connection with existing unified AR models, which reduces the cost to merely 72 A A100 GPU hours.
Compared to related works such as Chameleon (Team, 2024) and Show-o (Xie et al., 2024), Orthus outper- forms them across various visual understanding and generation benchmarks, while also demonstrating ex- tra capabilities in mixed-modality understanding and generation, positioning it as promising approach for unified multimodal modeling.
The other important contribution of this work is super- efficient strategy to build Orthus. Inspired by that Orthus differentiates from the representative token-based AR model Chameleon (Team, 2024) only in the input embedding mod- ules and output heads, we propose to substitute the VQ operation with soft alternative and augment the model with extra diffusion head to instantiate Orthus. We tune only the embedding modules and diffusion head (with 0.3B parameters in total) to reconstruct images on a 10k dataset to effortlessly obtain an Orthus base model. Orthus-base can further adopt post-training to bolster its ability to model interleaved images and text.
2. Related Work
Visual understanding. To enable multimodal large lan- guage models (MLLMs) to comprehend modalities beyond text, prior work has introduced methods that leverage pre- trained, modality-specinf encoders (Radford et al., 2021; Li et al., 2022; Yu et al., 2022; Chen et al., 2024) to generate latent representations for each modality. These representa- tions are then projected into a pre pre-trained LLM's 's input space through trained adapters, allowing for multimodal informa- tion alignment within the language model, understanding
We have performed extensive studies evaluate Or- thus. For mixed-modality understanding and generation, Orthus outperforms the editing-specific model Instruct- pix2pix (Brooks et al., 2023) and exhibits in-context learn-
2

---

Orthus: Autoregressive Interleaved Image-Tex Generation with Spdelity-Speciffe Heads
are handled within the transformer backbone (Liu et al., 2024d; Zhu et al., 2023; Dai et al., 2023; Driess al., 2023; Chen et al., 2023b; Liu et al., 2024b; Lin et al., 2024). This framework allows LLMs to perform complex multimodal tasks while maintaining the language-based reasoning capa- bilities inherent to their architecture.
3. Preliminary
Unified multimodal modeling aims to cope with a blend of images and texts with single compact model (Liu et al., 2024f; Team, 2024; Wang et al., 2024; Zhou et al., 2024; Xie et al., 2024). The model usually includes vision au- toencoder, specified with an encoder E and a decoder D, a text tokenizer, and a transformer network (Vaswani, 2017). The encoder E' is used to map the input image to a se- quence of patch-wise features V := [vi...., Unl. Vi E Rd for effective information compression, where d, is the fea- ture dimension and n is the number of patches. The text tokenizer maps the input text into sequence of text to- kens U := [u,,..., Umm] with m as the sequence length. The transformer is then asked to process U and V simul- taneously to yield meaningful outputs, which can be then detokenized as texts or decoded by D to produce images. There are primarily two strategies for the learning of the transformer, detailed as follows.
Visual generation. The generation of visual content has long been a central focus within the deep learning research community (Kingma, 2013; Goodfellow et al., 2014; Karras et al., 2019; Vahdat & Kautz, 2020). Over the past few years, research in visual generation has focused on decomposing visual signals in a more sophisticated manner and generat- ing them iteratively. Diffusion models
et al., 2015; Ho et al., 2020; Song et al., 2020b; Dhariwal& & Nichol, 2021; Rombach et al., 2022; Peebles & Xie, 2023; Esser et al., 2024) transform generation into reverse diffusion process from noise to data, gradually refining an initial noise input through a series of denoising steps. While another line of work aims to emulate the success of AR modeling from language modeling within the visual domain (Parmar et al., 2018; Razavi al., 2019; Ramesh et al., 2021; Yu et al., 2023; Sun et al., 2024b). Specifically, images are first transformed into sequence of vector-quantized tokens (Van Den Oord et al.. 2017; Esser et al., 2021: Tian et al., 2024; Yu et al., 2024), after which AR modeling is then performed on the discrete-valuee token space (Touvron et al., 2023a). To mitigate generation quality degradation caused by in- formation loss during the VQ process, MAR replaces the per-token categorical distribution modeling with a diffusion procedure (Li et al., 2024; Fan et al., 2024). Our proposed method generalizes MAR to mross-md generation.
Observing that the AR principle excels in the generative modeling of discrete content, seminal works, including LWM (Liu et al., 2024e) and Chameleon (Team, 2024), propose to leverage the Vector Quantization (VQ) (Van Den Oord et al., 2017) technique to transform the continuous image features V as discrete tokens to enable a fully AR modeling of the mixture of images and texts. Specifically, VQ introduces set of K codes {cj E Rd 1 and solves the following problem for continuous -isscrrete transformation:
Fully AR models.
0 = arg min d(ve, Cÿ) for i = I,...,7. n, jE(L.....K
(1)
Unified visual understanding and generation. To en- able a model to possess both understanding and generation capabilities, one kind of approach aims to connect L.MMs with multimodal adapters and diffusion decoders (Sun et al., 2023; Ge et al., 2024; Ye et al., 2024). However, using mul- tiple distinct components can lead to modeling redundancy. Consequently, recent studies seek to leverage single trans- former for unified understanding and generation. A straight- forward approach is to apply vector quantization to con- tinuous visual signals to enable visual tokens, like discrete text tokens, to be trained within a unified token space us- ing eross-entropy loss. Representative works are LWM (Liu et al., 2024e), Chameleon (Team, 2024), Anole (Chern et al., 2024), and VILA-U (Wu et al., 2024b). Alternatively, some works have explored combining AR with diffusion mod- eling. Show-o (Xie et al., 2024) unifies AR and discrete diffusion modeling for multimodal understanding and gen- cration within one single transformer. Transfusion (Zhou et al., 2024) trains one shared transformer for both discrete text autoregression and continuous image diffusion. Our proposed method circumvents the potential information loss caused by quantization and noise disturbance.
where .(.,,) is distance metric.
Let V = ( Un denote the discrete image tokens. The fully AR model embeds both V and U as de- features. Specifically, the embedding corresponding to o; is
h: = Σ wy ور
(2)
where (w, E TR: K.K refer to the embedding weights. The embeddings for text tokens can be similarly gained, yet with another set of embedding weights. The transformer then processes these embeddings with causal attention, where the output head naturally yields the prediction of the next token. For training, the objective is simply the AR loss.
Despite being simple, the fully AR models can suffer from information loss (Liu et al., 2024a; Team, 2024), because VQ makes the transformer unable to directly look at the image features d,.
A AR-diffusion mixed models. Another line of unified multi- modal models is AR-diffusion mixed models (Zhou et al., 2024; Xie et al., 2024; Zhao et al., 2024), which integrates
3

---

Orthus: Autoregressive Interleaved Image-Text Generation with Sodality-Speciffe Heads
Text
Image
[BOI]
Text Tokenizer
vision encoder
[EOi] token
discrete text token
leornable codebook
continueus image patch
text embedding module
vision embedding module
image embedding layer
condition ft V
Transformer
edd hoise
M2-1
MLP
LM head
disfis head
noise prediction
diffusion modeling for ((v)al)))
Yl
Figure 2. Architecture of Orthus. Orthus is composed a text tokenizer, a vision autoencoder, two modality specific embedding modules, transformer backbone, and two modality-specifff heads. Orthus tokenizes texts into discrete text tokens and encodes images into continuous patch-wise features. They are then embedded as a sequence of vectors and processed by the transformer backbone with causal attention, generating sequence of output vectors. The vectors routed modality-specifie heads, with the LM head to predict the next text token categorically and the diffusion head to predict the next image patch feature through conditional diffusion modeling.
diffusion modeling on images (Ho et al., 2020; Peebles & Xie, 2023) and AR modeling on text within shared transformer. Take Transfusion (Zhou et al., 2024) for ex- ample, its inputs are noisy version of the image features V. denoted V, and the text tokens U. To facilitate the simultaneous processing of V and U. the attention mask of the transformer adopts a unique configuration- with full- attention structure among V and causal structure among U. Then, the outputs from V are directed to an output projector to predict the noise on V, whereas the outcomes linked to U are channeled to an LM head for next-token prediction. The training objective the combination of AR loss and denoising loss with a balancing factor. During inference, the model operates as a diffusion model to craft images, with special tokens indicating mode switching.
avoids the pathologies caused by the quantized image fea- tures V or noisy image features V. U and V are embedded into the de-dim representation space with a differentiable vision embedding module (detailed in the next subsection) and the aforementioned discrete embedding module respec- tively. Subsequently, the embeddings are fed into the trans- former backbone with causal attention for the modeling of both inter- and intra-modality interdependence. Given the output states of such backbone contain enough informa- tion about the multimodal context, Orthus sends them to two modality-speciff heads a diffusion head and an LM head-to to predict the next image patch or the next token.
Specifically, let fx denote the output state corresponding to the input image feature v, and cø denote the diffusion head employed by Orthus with parameter 0. The goal of en is to predict for the next patch feature l conditioning on f.. According to common practice (Ho et al., 2020; Dhariwal Nichol, 2021), the learning objective for the diffusion head can be formalized as:
AR model to generate texts and
However, diffusion modeling inherently requires feeding noisy inputs to the model, hindering joint modeling of vi- sual understanding (requiring clean images) and generation (requiring noisy ones). For example, Transfusion identifies nearly 15% performance drop in image captioning when full-range noise is introduced during training.
Cui Ee,elllc co(Voru44 VI- f. (3)
where E ~ (N I) a Gaussian noise and is a randomly sampled timestep. ar follows pre-defined noise sched- ule (Ho et al., 2020). In practice, eg be shallow mul- tilayer perception (MIP) with three inputs (the condition ft, the scalar timestep f, and the noisy state). On the other hand, the LM head remains the compact linear projection followed by a softmax transformation to yield the predictive probability of the next token over the entire vocabulary.
4. Method
We introduce Orthus to address the issues of existing works. This section begins with an overview of Orthus and then elaborates efficient training recipe for Orthus. We will also illustrate post-training pipeline of Orthus.
4.1. Overview of Orthus
4.2. An Efficient Strategy for Constructing Orthus-base
As shown in Figure Orthus directly takes the continuous image features V and discrete text tokens U as input, which
The differences between Orthus and fully AR models exist in the vision embedding module and the output head. Given
4

---

Orthus: Autoregressive Interleaved Image-Text Generation with Spdality-Seecifte Heads
that pre-training multimodal model from scratch can be frustratingly costly but the fully AR models like LWM (Liu et al., 2024e) and Chameleon (Team, 2024) are readily ac- cessible from the open-source community, we are naturally interested in deriving Orthus based on them at a minimal ex- pense. This section elaborates on a hard-to-soft adaptation trick and an efficient training strategy to enable this.
question answering), images (e.g., image editing), or even both images and text (e-g., storybook generation) from mixed-modality inputs. For example, given [V, U] as user imput and [V, U, V, UJ as model output, we surround image features V with the embeddings of special tokens [BOT] and [ЕOI] before the concatenation with U. A [SEP] token is used to separate user input and model output in each conversation.
Differentiable vision embedding module. It is easy to note that the embedding yielded by Equations 1 and 2 can be equivalently obtained via softmax-based transformation
Let Lur denote the AR loss on the text tokens. The entire training objective of Orthus is then Lorthus = Lar + XLaiff where À is a balancing coefficient. All parameters except for those of the vision autoencoder are tuned. Hereinafter, we will denote the model trained following this objective as Orthus, distinguishing it from Orthus-base.
Σ w d(v.o///
h =
(4)
with 5 - O. Increasing T gradually from 0 then naturally lifts the information bottleneck from the image features 1% to the model outputs fi, while rendering the reuse of the pre-trained weights and codes of fully AR models possible. This way, the codes {- also become part of the input module, so we can leverage gradients to directly push them to adapt to the multimodal learning tasks. This contradicts fully AR models which froze the codes during training.
During inference, Orthus alternates between next-token pred- ition and next next-patch prediction to seamlessly generate inter- leaved texts and images. When [BOT] is sampled during the next-token prediction process, the algorithm moves to next-patch prediction. Once a fixed number of n image patches are generated, [EOI] is appended and the algo- rithm switches back to next-token prediction.
K
Training strategy. With the above trick, we start with a pre- trained fully AR model, transform its input module into a differentiable one, and introduce an output diffusion head to initialize Orthus. These modifications primarily focus on the visual part, thus we recommend fine-tuning the initialized model on a collection of images. In particular, we imput only the image into Orthus acquire the hidden states f; and utilize the diffusion loss in Equation 3 to recover the next patch to train the vision embedding module and diffusion head. The temperature T is set to 1 during training.
5. Experiments
In this section, we evaluate Orthus's performance in inter- leaved image-text modeling as well as visual understanding and generation. Both quantitative and qualitative results demonstrate the effectiveness of Orthus.
5.1. Implementation Details
We implement the diffusion head as an MLP consisting of residual blocks, each sequentially applying AdaLN (Peebles & Xie, 2023), a linear layer (width of 1536 channels), SiLU activation, and another linear layer. The condition vector f. is added to the diffusion time embedding, which is then incorporated through AdaLN. The diffusion noise schedule is linear following (Rombach et al., 2022), with 1000 steps at training time. X is set to 100 to balance the order of magnitude between Laif and Car during post-training. During inference, we use greedy decoding to generate text. For image generation, we adopt the DDIM (Song al., 2020a) sampler with 100 steps. We employ classifier-free guidance (CFG) (Ho & Salimans, 2022) with the scale set to 5 during sampling. All images are generated at a resolution of 512x512. More training and evaluation details are provided in Appendix B.
Initialized with the typical Chameleon-7BB (Team, 2024), Orthus can acquire image processing capabilities while pre- serving the text generation capacity after 9-hour training on 10k high-quality images (laion-coco aesthetic) using 8 A 110 GPUs We designate this model as Orthus-base, pre-trained model capable of generating continuous image features and discrete text tokens.
Although the decoder in the VQ-VAE (Van Den Oord et al., 2017) used by Chameleon can reconstruct the raw image pixels given the patch-wise features V to some extent, it can be suboptimal due to the quantization-awwe aware training. To address this, we advocate further tuning its decoder to reconstruct high-quality images directly based on V. The comparison between the capacity of the original VQ-VAE and ours is exhibited in Appendix A.
4.3. Multimodal Post-training
5.2. Interleaved Image-Terr Text Generation
Orthus-base can be further post-trained to unlock its poten- tial for interleaved image-text modeling in complex down- stream tasks. These include generating text (e.g., visual
Compared to existing unified models, such as Janus- series (Wu al., 2024a; Ma et al., 2024), that focus ex- clusively on visual understanding and generation, we inves-
5

---

Orthus: Autoregressive Interleaved Image- Text Generation with Speelity-Speciffe Heads
In dn-domain editing
In conteetxt editing
story:
Please continue this story:
sureyy momning,
the ond hix friend deciped busti ing ciay park. ofverturee by eept oriing colorful playgroand.
leorge stood dork forest; torge, cest on the crees, on the greund. projecting tie Fri et houette
According to the Fullowt vollowing sumplas::
loy
ratrboo
MAk the suit red.
while. George, decidde tase hreck and gross, enjoytng the posceful troes and buildingg.
be scanal oasily.
steed inerde Huell cose, ready
Ploase continue
on a ancay day, Flgand nol winter help his friled, the and nod scurf. They wore cataide their"
The bn of tight cast seft glon, sparkle
his syea curtastity.
Out-of-domann editing
Generate For inputi
In the shadeey Geonge spected bell. ta quictty gresbed
Ris now Find. Trees [ E'Tuiisstto Light sounco, surnuncedd him.
the in the orange his stood cor bortollly landucape, his sette bright agetieet
Remove the cot.
foce, George stood in the cave outstreicche: He engoying odvontune by
also out In the snowy Lendscope, wore and gloves, vordering if eore way.
stalactitee glowingg light.
Figure 3. Qualitative results on mixed image-text understanding and generation of Orthus, Left: Image editing results after fine-tuned on Instruct-Pix2Pix (Brooks al., 2023). Notably, Orthus exhibits in-context learning capacity by performing image editing successfully when provided with examples rather than explicit instructions, which is not included in the training dataset. Right: Interleaved story book creation results after finetuned on the StoryStream (Yang et al., 2024b) dataset. Results show that Orthus excels in generating logically coherent interleaved image-text with high relevance.
Table 1. Comparisons of CLIP similarities (Ruiz et al., 2023; Gal al., 2022) between fätting--peciffi diffusion models Orthus on the test dataset of Instruct-Pix2PPx.
when provided with examples instead of explicit instruc- tions that do not match the formats seen during training, successfully completes the task. This highlights Orthus's strong capability for interleaved data modeling and its great potential foundation multimodal model. More editing examples and comparisons provided in Appendix F.
Model
-Tt
-If
-Df
PnP (Tumanyan et al., 2023)
0.156
0.76
0.023
SDEdit (Meng et al.)
0,229
0.84
0.047
Image-Text + Image- Tewe---mage-eeeeee Text. To further vali- date Orthus's ' superiority in modeling interleaved data, we fine-tune Orthus-base with the unified learning objective on the StoryStream (Yang et al., 2024b) dataset, which in- cludes a collection of images and corresponding narratives from cartoon series. As shown in Figure 3, after training, Orthus can generate contextually consistent scenes paired with narrative text given an initial image-text pair and the instruction "Please continue this story." Notably, there is a strong aligmment between images and text (e.g.. the smile on the monkey's 's face) as well as consistent detail across images (e.g., the boy's orange hat and red scarf). These re- sults highlight Orthus's ability to generate long sequences of contextually relevant images and text. This capability opens up potential applications such as report generation, educa- tional content creation, and other tasks requiring seamless mixed-modality content generation.
I-Pix2Pix (Brooks et al., 2023)
0.233
0.88
0.045
Orthus (Ours)
0.238
0.87
0.049
tigate Orthus's flexibility and extensibility model inter- leaved images and text on two representative downstream tasks: image editing and storybook generation.
Image-Text > Image We compare the performance of Orthus with an editing-specific diffusion model after train- ing Orthus-base on the 400k Instruct--ix2pix (Brooks al., 2023) training dataset. Table shows that the images edited by Orthus align well with both the given instruction and the input image, performing comparably to even surpassing the -diting-specific diffusion model. Moreover, Figure 3 il- lustrates Orthus's strong generalization ability to edit images in zero-shot real- -mage domains. Notably, Orthus exhibits in-context learning capacity as a unified multimodal model:
6

---

Orthus: Autoregressive Interleaved Image- Generation with Modality-Seeciffe Heads
Table Evaluation on visual understanding benchmarks. Und. and Gen. denote "understanding" and "generation", respectively. Models using external wre-trained diffusion models are marked with and Chameleon' is post-trained with the same dataset as Orthus. The results in bold and underline are the best second best results, respectively. The results correspond the exact match accuracy.
Type
Model
Params
POPE
MME-Pt
VQAv21
GQA
MMMU1
LlaVa (Liu et al., 2024d)
7B
76.3
809.6
LlaVA-v 1.5 (Liiu al., 2024b)
7B
85.9
1510.7
78.5
62.0
35.4
InstructBLIP (Dai al., 2023)
7B
49.2
Und. Only
Qwen-VL-Chat (Bai al., 2023)
7B
1487.5
78.2
57.5
Emu3-Chat (Wang et al., 2024)
8B
85.2
1243.8
75.1
60.3
31.6
InstructBIIIP (Dai et al., 2023)
13B
78.9
1212.8
49.5
Emu" (Sun al., 2023)
13B
52.0
NEXT-GPT" (Wu al., 2013)
13B
66.7
Gemini-Nano-1 (Team et al., 2023)
1.8B
62.7
26.3
Show-o (Xie et al., 2024)
1.3B
73.8
948.4
59.3
48.7
25.1
Und. and Gen.
LWM (Liu et al., 2024e)
7B
75.2
55.8
44.8
Chameleon
7B
77.8
1056.9
57.8
49.6
26.7
Orthus (Ours)
7B
79.6
1265.8
63.2
52.8
28.2
Table 3. Comparison with state-of-the-arts on visual generation benchmarks. Model using external pre-trained diffusion model is marked with and Chameleon is post-trained with the same dataset as Orthus. The results in bold and underline are the best and second- best results, respectively.
5.3. Visual Understanding and Generation
In this section, we validate the effectiveness of Orthus on visual understanding and generation by post-training Orthus- base with a mixture of AlaVA-v1.5-663 (Liu et al., 2024d) and high-quality text-to-image data (JourneyDB (Sun et al., (laion-coco aesthetic) recaptioned from ShareGPT-4v (Chen et al., 2023a)). We also tne-tune pre-traine trained Chameleon (Chern et al., 2024) with the same mixed dataset as Orthus to provide an apple- to-apple baseline.
Type Model
Res. GenEval HPS
2024a) and AATONCCOCO-assti
SDv1.5 1.5 (Rombach et al., 2022)
512
0.43
27.0
SDv2.1 (Rombach et al., 2022)
512
0.50
27.2
Gen. Only
DALL-E (Ramesh et al., 2022)
512
0.52
26.9
Emu3-Gen (Wang et al., 2024)
512
0.54
Image + Text. Table shows that: (i) Compared to Chameleon post-trained with the same dataset, Orthus consistently demonstrates superior performance across all benchmarks. Besides, inspecting OCR-related tasks in MME-P, we witness significant superiority of Orthus over Chameleon (with scores of 70 vs. 45). These results validate the superiority of Orthus's modeling by adopting lossless representations for images. (ii) Orthus outperforms other unified models using single transformer like LWM and Show-o across all benchmarks, highlighting its efficacy for unified modeling. (iii) Compared to larger unified models using an external diffusion model, such as NEXT-GPT-13B, Orthus achieves decent results on the VQAv2 benchmark. reasonable to speculate that Orthus's potential for mul- timodal understanding problems can be further unleashed by scaling up training compute and data.
SDXL (Podell et al., 2023)
512
0.55
30.9
SD3(d=30) (Esser et al., 2024)
512
0.64
SEED-X* (Ge et al., 2024)
448
0.49
LWM (Liu et al., 2024c)
256
0.47
26.1
Und.
Show-o (Xie et al., 2024)
256
0.53
27.3
&
Gen.
Transfusion (Zhou al., 2024)
256
0.63
Chameleon
512
0.43
26.9
Orthus (Ours)
512
0.58
28.2
HPSv2. This advantage can be attributed to the utilization of continuous image representations and diffusion based continuous modeling, which facilitates the generation of high-quality images with richer detail and stronger align- ment with human preferences. (jii) Compared with other unified models such as SEED-X. LWM. and Show-o, Orthus obtains significantly better performance, highlighting the ad- vantages of its modeling strategy. (iv) Qualitative results in Figure 4 showcases images generated by Orthus alongside results from other unified models, including Chamelcon and Show-o. Results show that Orthus is capable of generat- ing diverse, engaging, and realistic visual imagery at the resolution of 512x512.
Text - image. Table 3 shows that: (i) When compared with strong competitors specialized for text-to-image gen- erations such as DALL-E and SDXL, Orthus achieves an improvement of 0.06 and 0.03 on GenEval, respectively. (ii) Compared Chameleon and its post-trained version, Orthus demonstrates significant superiority on both GenEval and

---

Orthus: Autoregressive Interleaved Image-Tea Text Generation with Sppecit Heads
Show-o
Chameleon
Orthus
detailed ink illustration of hedgehog.
A hyinc fursone sits in savernahh sunset anidst the grass.
pointing portrois young sunset mountains background.
Figure 4. Left: Comparison between images generated by Show-o, Chameleon, and Orthus based on the same prompts. Samples produced by Orthus contain more details. Right: Text-to-image gallery Orthus.
Table Comparisons of the performance of Orthus via separate trining and unified training across multimodal benchmarks.
Table S. Ablation study on the choice vision embedding modules visual understanding tasks.
Type
Lail
fa
POPEЧ
MME-P GQAT GenEval
Type
POPE
MME-P1
VQAv21
GQAt
MMMU1
Und. only
X
/
78.7
1244.2
51.9
softmax
78.7
1244.2
60.8
51.9
28.0
Gen. only /
x
0.56
argmin
77.6
1064.8
57.9
50.1
26.7
Und. & Gen.
79.6
1265.8
52.8
0.58
linear
70.4
800.7
50.3
44.5
22.3
Loss design. To test the necessity of diffusion modeling for the image features, we train the MLP head with straightfor- ward Mean Squared Error (MSE) loss between predictions and target features. As shown in Appendix C, the model trained with MSE loss generates degraded samples that lack details and exhibit limited color diversity. The reason is that the deterministic nature of MSE loss leads to mode collapse.
5.4. Ablation Studies
Separate training vs. unified training. To validate the effi- cacy of Orthus for unified multimodal modeling, we com- pare baselines using identical training data but with different learning objectives: (i) generation-only baseline focused solely on text-to-image generation; (ii) an understanding- only baseline dedicated to visual understanding tasks; and (iii) unified training objective, the default setting in Orthus. Table shows that unified training outperforms -asc-spcc training both visual understanding and generation, high- lighting the superiority of Orthus which facilitates informa- tion gains from bidirectional cross-modal learning.
6. Conclusion
In this paper, we propose Orthus, a unified multimodal model for interleaved image-text understanding and genera- tion. Orthus generates content across modalities by routing outputs from shared transformer backbone to modality- specific heads. Its continuous treatment of visual signals pre- serves input integrity and its unified AR modeling approach for both discrete text tokens and continuous image features enables superior performance across various multimodal understanding and generation benchmarks. For future work. plan scale Orthus by increasing its parameters and leveraging larger, interleaved datasets to maximize its poten- tial. We also aim to broaden its capabilities by incorporating more modalities, including video and audio.
Impact of vision embedding modules on visual under- standing tasks. In this section, we ablate the impact of different choices of vision embedding modules to build Or- thus from fully AR models on visual understanding. When we retain the original embedding module fully AR models ("argmin" in Table 5), performance drop is observed due to the information loss. Moreover, replacing the embedding module with randomly initialized linear layer also leads to suboptimal performance due to the significant distribution shift between the embedded space and the transformer's 's input space. This misalignment may necessitate training with more image-text pairs to mitigate.
Limitations. The main limitations of this work are as fol- lows. Orthus exhibits relatively high inference latency intro-
8

---

Orthus: Autoregressive Interleaved Image-Texx Generation with Sodality-Speciffe Heads
duced by the need for multiple forward passes through the diffusion head. Besides, Orthus is limited to 7B parameters due to constrained computational resources.
Dai, w., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general-purpose vision-language models with instruc- tion tuning, 2023. URL bettps://axxiv.orrt /bs/ 2.305 2.305.0550..
Impact Statement
This work presents a challenge in machine learning and proposes a solution, the potential negative consequences apparent. While it is theoretically possible for any technique to be misused, the likelihood of such misuse occurring at the current stage is low.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances neural information processing systems, 34:8780-8794, 2021.
Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q. Yu, T., et al. Palm-e: An embodied multimodal language model.
Acknowledgements
arXiv preprint arXiv:2303.03375 2023.
This work was supported by NSF of China (Nos. 92470118, 18, 623061700 16)), Natural Science Foundation of Shanghai (No. 23ZR1428700 4288700), CCF-Zhipu Large Model Innovation Fund (No. CCF-Zhipu202412), and CCF-ALIMAMA TECH Kangaroo Fund (NO. CCF-ALIMAMA OF 2025010).
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
References
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versa- tile langualanguage model for understanding, localiza- tion, text reading, and beyond, 2023. URL https: // //arxiv.. org/ abs 2208 12966.
Fan, L., Li, T., Qin, S., Li, Y., Sun, C., Rubinstein, M.. Sun, D., He, K., and Tian, Y. Fluid: Scaling autoregressive text-to text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410. 13863, 2024.
Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18400 18402 2023.
Gal, R., Patashník, O., Maron, H., Bermano, A. H., Chechik, G., and Cohen-Or, D. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG), 41(4):1-13, 2022.
Chen, L., Li, J., Dong, X., Zhang. P., He, C., Wang. J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311. /t..::2311.12993.. 2023a.
Ge, Y., Zhao, S., Zhu, J., Ge, Y.. Yi, K., Song, L., Li, C., Ding, X., and Shan, Y. Seed-x: Multimodal models with unified multi granularity comprehension and generation. arXiv preprint arXiv:2404. 14396, 2024.
Chen, X., Djolonga, J., Padlewski, P., Mustafa, B., Chang- pinyo, S., Wu, J., Ruiz, C. R., Goodman. S., Wang, X., Tay, Y., et al. Pali-x: On scaling up multilingual vision and language model. arXiv preprint arXiv:2305.18556. 2023b.
Ghosh, D., Hajishirzi, H., and Schmidt, L. Geneval: An focuet-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024.
Chen, Z., WU, J., Wang, W. Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q. Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12.2207..229900 12598, 2022.
Chern, E., Su, J., Ma, Y., and Liu, P. Anole: An open, autore- gressive, native large multimodal models for interleaved image-text generation, arXiv preprint 2XXi::2477006133 2024.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6683 6851, 2020.

---

Orthus: Autoregressive Interleaved Image- mmat Generation with Modality-Seeciffe Heads
Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In Pro- ceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4401-4410 2019.
Ma, Y., Liu, X., Chen, X., Liu, W., Wu, C., Wu, Z., Pan, Z.,, Xie, Z.,, Zhang, H., Zhao, I.., et al. Janusflow: Har- monizing autoregression and rectified flow for unified multimodal understanding and generation. arXiv preprint arXiv:241 1.00975, 2024.
Kingma, D. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6111 2013.
Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, 2013.
Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A., and Tran, D. Image transformer. In Interna- tional conference machine learning, pp. 4055 6065-006. PMLR, 2018.
laion-coco aesthetic.
URL https://
huggingface co/datasets /ggurgggi/////g///// laion- calloncoco--ao aeoobetts
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International confer- ence machine learning, pp. (8888-12900) PMLR, 2022.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195 4205, 2023.
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregres- sive image generation without vector quantization. arXiv preprint ar arXiv:2406.1 11388, 2024.
Podell, D., English, Z., Lacey. K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for regolutsotttion image synthesis. arXiv preprint urXiv:2307.01952 2023.
Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., and Han, S. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 26689- 26699, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Liu, D., Zhao, S., Zhuo, L., Lin, W., Qiao, Y., Li, H., and Gao, P. Lumina- Lumina-mgpt: Illuminate flexible photorcalis- tic text-to-image generation with multimodal generative pretraining. arXiv preprint arXiv:2408.0266 2024a.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In International conference on ma- chine learning, pp. 8821-883 Pmlr, 2021.
Liu, H., Li, C., Li. Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20296-26306, 2024b.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical coxticonditional image generation with clip latents. arXiv preprint arXiv:2204.06612 206.2200..,,, 1(2):3 2022.
Liu, H., Li, C., Li, Y. Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Leava-next: Improved reasoning, ocr, and world knowledge, 2024c.
Razavi, A., Van den Oord, A., and Vinyals, O. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Liu, Li, C., Wu. Q., and Lee, J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024d.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEFF/CV con- ference on computer vision and pattern recognition, pp. 2022.
Liu, H., Yan, W., Zaharia, M., and Abbeel, P. World model on million-length video and language with ringattention. arXiv preprint 2rXiv:2402.08266 2024c.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500-22510 2023.
Liu, H., Yan, W., Zaharia, M., and Abbeel, P. World model on million-length video and language with blockwise ringattention, 2024f. URL https: http///xxivvoo/// org/ abs/2402. 08268.
10

---

Orthus: Autoregressive Interleaved Image-Tex Generation with Sodality-Speciffe Heads
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference machine learning, pp. 2256-2265 PMLR, 2015.
Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T. Plug- and-play diffusion features for text-driven image-to- image translation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, PP. 1921-1930, 2023.
Song. J., Meng, C., and Ermon, S. Denoising diffusion im- plicit models. arXiv preprint arXiv:2010.02502, 2020a.
Vahdat, A. and Kautz, J. Nvae: A deep hierarchical vari- ational autoencoder. Advances in neural information processing systems, 33:19667-11679, 2020.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score- generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13456, 2020b.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep-
resentation learning. Advances in neural information processing systems, 30, 2017.
Sun, K., Pan, J., Ge, Y., Li, H., Duan, H., Wu, X., Zhang, R., Zhou, A., Qin, Z., Wang, Y., et al. Journeydb: A benchmark for generative image understanding. Advances in Neural Information Processing Systems, 36, 2024a.
Vaswani, A. Attention is all you need, Advances in Neural Information Processing Systems, 2017.
Wang, X., Zhang, X., Luo, Z., Sun, Q. Cui, Y., Wang, J., Zhang, F., Wang, Y., Li, Z., Yu, Q., et al. Emu3: Next-token prediction all you need. arXiv preprint arXiv:2409, 18869, 2024.
Sun, P., Jiang, Y., Chen, S., Zhang, S., Peng, B., Luo, P., and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06555, 2024b.
Wang, Z., Bovik, A. C.,, Sheikh, H. R., and Simoncelli, E. P. Image quality assessment: from error visibility to struc- tural similarity. IEEE transactions on image processing, 13(4):600-62, 2004.
Sun, Q. Yu, Q. Cui, Y., Zhang, F., Zhang, X., Wang, Y., Gao, H., Liu, J., Huang. T., and Wang, X. Emu: Genera- tive pretraining in multimodality. In The Twelfth Interna- tional Conference on Learning Representations, 2023.
Wu, C., Chen, X., Wu, Z., Ma, Y., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C., et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410. 13848, 2024a.
Sun, Q. Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J., Huang, T., and Wang, X. Generative mul- timodal models are in-context learners. In Proceedings of the IEFE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14398-14409, 2024c.
Wu, S., Fei, H., Qu, I., W., and Chua, T.-S. Next-gpt: Any-to-any multimodal 11m. In Forty-first International Conference on Machine Learning, 2013.
Team, C. Chameleon: Mixed-modal early- farly-fusion foundation models. arXiv preprint arXiv:2405.098/1 2024.
Wu, Y., Zhang, Z., Chen, J., Tang, H., Li, D., Fang, Y.,
Zhu, L., Xie, E., Yin, H., Yi, L., et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.0444
Team, G., Amil, R., Borgeaud. S., Alayrac, J.-B., Yu, Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11800 2023.
2024b.
Xie, J., Mao, W., Bai, Z., Zhang, D. J., Wang. W., Lin, K. Q., Gu, Y., Chen, Z.,, Yang, Z., and Shou, M. Z. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 240.:::408./22288 /2528, 2024.
Tian, K. Jiang, Y., Yuan, Z., Peng, B., and Wang, L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404 240000, 2024.
Yang, J., Yin, D., Zhou, Y., Rao, F., Zhai W., Cao, Y., and Zha, Z.-J. Mmar: Towards lossless multi-modal auto-regressive prababilistic modeling. arXiv preprint urXiv:2410.10790 2024a.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint 2rXhv:2302.13971 2023a.
Yang, S., Ge, Y. Li, Y., Chen, Y., Ge, Y. Shan, Y. and Chen, Y. Seed-story: Multimodal long story generation with large language model. arXiv preprint arXiv:2407.08683, 2024b.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint XxXiv22377009288, 2023b.
Ye, H., Huang, D.-.., Lu, Y., Yu, Z., Ping, W., Tao, A., Kautz, J., Han, S., Xu, D., Molchanov, P., et al. X- vila: Cross modsalty alignment for large language model. arXiv preprint arXiv:2405. 19335, 2024.

---

Orthus: Autoregressive Interleaved Image-Tex Generation with Sodelity-Secciffe Heads
Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y. Coca: Contrastive captioners are image- text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Yu, L., Lezama, Gundavarapu, N. B., Versari, L., Sohn, K. Minnen, D., Cheng, Y., Birodkar, V., Gupta, A., Gu, X., et al. Language model beats diffussion-ookeiizriis.. key to visual generation. arXiv preprint arXiv:2310.05737, 2023.
Yu, Q.. Weber, M., Deng, X., Shen, X., Cremers, D., and Chen, L.-C. An image is worth 32 tokens for reconstruc- tion and generation. arXiv preprint 40X.v.2406.07550, 2024.
Zhao, C., Song, Y., Wang, W., Feng, H., Ding, E., Sun, Y., Xiao, X., and Wang, J. Monoformer: One transformer for both diffusion and autoregression. arXiv preprint Stti::2409.128860 2024.
Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and dif- fuse images with one multi-modal model. arXiv preprint arXiv:2408. 1109, 2024.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
12

---

Orthus: Autoregressive Interleaved Image-Text Generation with Sodality-Seeciffe Heads
A. Comparison of Vision Autoencoder
To construct vision autoencoder capable decoding high-quality images based on continuous image features V, we freeze the encoder of Chameleon's VQ-VAE, drop the quantization step, and finetune the decoder only reconstruct images, transforming it into a conventional continuous autoencoder effectively. The decoder is trained on -eshN-Aesthetic dataset (laion-coco aestheticJusing learning rate of 1e-5, batch size of 256, and total of 15,000 training steps. Table 6 shows that our vision autoencoder achieves better reconstruction quality compared to the original VQ-VAE. The evaluation is conducted on a subset of the LAION-Aesthetie consisting of 10,000 images that are excluded from the training dataset.
Table Comparison of reconstruction quality for vision autoencoders: the discrete is worse than the continuous variant.
Model
PSNR
SSIM (Wang et al., 2004)f
VQ-VAE (Team, 2024)
23.7
0.80
Ours
26.1
0.84
B. Training details
The images for training Orthus-bass the first 10k from laion-coco aesthetic. Both training and evaluation are carried on servers equipped with 8 NVIDIA A100 80GB GPUs.
Table Training details for constructing Orthus-base and the instruction tuned one visual understanding generation
Model
Orthus-base
Instruct-tuning
Optimizer
AdamW (6, 0.9, 8/2 = 0.99)
Learning Rate
lc-4
le-5
Batch Size
32
16
Training Steps
15,000
35,000
C. Diffusion Loss v.s. MSE Loss
Figure 5. Text-to-image results from models trained with MSE loss. The text prompt is "Generate image of snowman."
D. Examples on Visual Generation
Figure shows examples of images generated from Orthus post-trained Section 5.3.
E. Examples on Visual Understanding
In addition to quantitatively evaluating Orthus in Section 5.3 on domain-specifee tasks, we also assess its performance in general chat scenarios in Figure 7.
F. Examples on Image Editing
Figure shows random examples of image editing by Orthus-bass post-trained Instruct-Pix2Pix (Brooks et al., 2023).
13

---

Orthus: Autoregressive Interleaved Image-Text Generation with Sodelity-Seeciffe Heads
A pho torealiii image of giant floating glass sphere in rocky londscape surrounded by gentle mist.
Animals fashioned from gems,colarfu colorful and shapely, depicted natural lighting, with slight effervescence.
A portrait painting of South Indian woman wearing sari with intricate details and eerie sense of horror, created in ultra-realistic style by artgerm.
A serene meadow with tree, river, bridge, and mountains the background under slightly overcast sunrise sky.
gems,
A painting of Persian cat dressed as Renaissance king, standing around skyscraper overlooking city.
stylized building
watercolor gouache, featuring interesting shopes and forms, located in desolate landscape with food stall in an Asion-style alleyway.
A cobblestone street with tree over the sea sunset, illuminated by sun rays, in a colorful illustration by Peter Chan Artstation.
Portrait of Archduke Franz Ferdinand by Charlotte Grimm, depicting his detailed face.
glowing dry tree stands alone under starry sky in a detailed fantasy artwork
A girl looks out from the edge of mountain onto a large city at night.
A gangster squirrel is counting his money in angle film still.
A neon-colored frog cyberpunk setting.
Digital painting furry deer character FurAffinity.
An anthropomorphic frog wizard wearing a cape and holding wand.
Portrait of monkey wearing an astronaut helmet.
close-up ail painting of littlest pet shop fuzzy skunk in field.
Figure 6. Generated 512 512 images from Orthus. Results demonstrate its ability to generate diverse, engaging, and realistic images.
14

---

Orthus: Autoregressive Interleaved Image- Text Generation with Sodality-Seeciffe Heads
Can tell what kindd muild mountiol
Fruidd help the ireee iacge? Alioo,o think woald sulesblle chnt os pee?
for thi aind Form?
earipment this, footaring and harrou- truiler transport the tervested shat. oeher cruciel pleces of equipment sight include tractors with specializes cant qurations for wheas celsivation, tractors contuning hervessed prain crops
grain
Besed or the tasge descriptton, the d2g 15 snaggy, brow,, and ton puppy, It cosld shoggy idencl
flaitty sheggy puppy's aspcorance the treed soar conds ard derecsoristics suct as size, nose shope, eye shape, sor shope, and other physsoal queistses.
commenty would
cour cross, combire orills to evenly diatr focilltane
honey cquianere, and safityy por the operonors end workers the rarm In the saage, norrow is aiso vestale, deccratrating the presence of this cructal piece wheeled Form equipaent.
bockpocis
Intelligence, gose thol 1ib6l1ity by.. and gertle, Friendly denecnor, waking thee popsilrr croice pet. If this iedeed sngggy del Retrlever, excel in homes
tralits. For romplle, coldendieddees
for their offectionnte
Figure 7. Examples of Orthus visual understanding. Results demonstrate that Orthus exhibits strong instruction-followiii capabilities robust generalization abilitics.
Have animals made chocolate
Add lightning storm
Chonge the house to castle
Make more opocalyptic
Origin
Orthus
I-pix2pix
Figure 8. Examples of Orthus image editing. Compared to editing-specific diffusion models (Brooks et al., 2023), Orthus demonstrates better fidelity to the original image in regions where no editing is required.
15


Reviews:
[{"review_id": "c7b960f1u2", "reviewer": "Reviewer_eJsk", "summary": "This paper proposes Orthus, an interleaved image-text generation model with modality-specific heads. Orthus shows that language model heads for discrete tokens and diffusion heads for continuous image generation can work together. By fine-tuning from Chameleon, Orthus obtain good performance on both image understanding and generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback and useful suggestions! We address your concerns point by point below.\n\n**Weakness:** The idea is relatively not novel enough, just combining diffusion head and lm head, which shows limited insight into unified image understanding and generation. \n\n- We would like to emphasize that **our combination of diffusion head and lm head leads to a unified model for flexible interleaved image-text modeling, mitigating the issues of existing fully AR and AR-diffusion mixed models.** Such abilities are important for downstream tasks like in-context editing and storybook generation, as proven by the recent GPT-4o, but have been inadequately explored in research community.\n- The incorporation of diffusion head enables the processing of continuous visual signals. And, experiment results in Table 2&3 show that lossless continuous visual signal benefits both image understanding and generation. Namely, we yield the insight that the diffusion head can be essential when we need a unified multimodal generation model. \n\n**Missing baselines:** Missing VILA-U and Janus as baselines.\n\nSorry for the missing comparisons and we include them below:\n- We first clarify that VILA-U and Janus focus solely on image understanding and generation, lacking support for interleaved image-text generation, where Orthus demonstrates strong capabilities. Additionally, Janus decouples visual understanding and generation by using separate encoders, limiting its flexibility in handling interleaved data—where images must be generated and understood simultaneously.\n- We add a comparison with VILA-U and Janus on visual generation and understanding in table below. As shown, Orthus generates images with higher human preference scores at a resolution of 512. For visual understanding, while Orthus still lags, this may be attributed to training solely on the LlaVa-v1.5-mix-665k dataset, whereas the baselines leverage much more high-quality instruction-tuning data [1,2,3]. We plan to incorporate these datasets to further enhance performance.\n\n| **Model**        | **Res.**       | **GenEval $\\uparrow$**                | **HPSv2$\\uparrow$**           | **POPE$\\uparrow$**                   | **MME$\\uparrow$**  | **GQA$\\uparrow$**  |\n|-------------- |---------------------------|:-------------------------:|:-------------------------:|:--------------------------:|:-------:|:-------:|\n| Orthus   | 512     | 0.58                     | 28.2                     | 79.6                     | 1265.8 | 52.8  |\n| VILA-U | 256    | 0.40                     | 25.3                     | 83.9                     | 1336.2 | 58.3  |\n| Janus | 384                         | 0.61                     | 27.8                     | 87.0                     | 1338.0 | 59.1  |\n\nWe will include these comparisons in the revised version. Thank you for your valuable suggestions.\n\n[1] Kvqa: Knowledge-aware visual question answering. AAAI 2019.\n\n[2] Llava-onevision: Easy visual task transfer. TMLR 2025.\n\n[3] Screenqa: Large-scale question-answer pairs over mobile app screenshots. arXiv:2209.08199.\n\n**Question:** In the ablation study, are the data all the same for understanding-only and generation-only tasks compared with unified training?\n\nYes, for understanding-only and generation-only tasks, we use the same understanding and generation data as in the unified training setting.\n\nGiven these reclarifications and improvements in response to your initial concerns, we hope you would like to raise your score.", "claims_and_evidence": "N/A", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "Show the effectiveness of combining the diffusion head proposed in MAR and token prediction.", "essential_references_not_discussed": "Missing some important baseline, including VILA-U[1], Janus[2].\n\n\n[1] VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation.\n[2] Janus: Decoupling visual encoding for unified multimodal understanding and generation.", "other_strengths_and_weaknesses": "Strength: \n1. The idea is easy to follow and understand.\n2. The model shows good performance in image editing, generation, and understanding.\n\nWeakness:\n1. The idea is relatively not novel enough, just combining diffusion head and lm head, which shows limited insight into unified image understanding and generation.", "other_comments_or_suggestions": "1. In the ablation study, are the data all the same for understanding only and generation only tasks compared with unified training?", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ReOaXYuAXU", "reviewer": "Reviewer_iJo1", "summary": "This paper introduces an architecture to conduct unified multimodal understanding and generation. Specifically, it introduces a dedicated diffusion head to generate continuous visual token during image generation. By doing so, the proposed approach can enable native image generation with LLM at a low training cost.\n\n## update after rebuttal\nGiven the clarification and experimental results provided by the authors, which largely resolved my concerns, I would like to keep my score as weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback and useful suggestions! We are glad you thought our proposed design achieved impressive results. We address your concerns point by point below.\n\n**W1&Q1:** Numerical comparisons on the inference latencies of the proposed approach and other baseline methods.\n\nThanks for your suggestion. We estimate the practical inference time (sec/image) for generating images at the resolution of 256 with Orthus-7B and Chameleon-7B under the hugging face inference framework. Transfusion-7B employs bidirectional attention and functions as a DiT for image generation. This design prevents the use of kv-cache optimization, resulting in high latency at each diffusion sampling step. However, when the number of sampling steps is significantly lower than the number of generated image tokens, acceleration over AR frameworks could be possible. We will test the exact inference time of Transfusion-7B and provide a detailed comparison once it is open-sourced.\n\n| **Sampler**       |  **Orthus** | **Chameleon** |\n| -------| :---------:| :---------:|\n| DDIM 100 steps | 38.6 | 14.2 |\n| DDIM 50 steps | 26.2 | 14.2 |\n| DPM-Solver++ 10 steps | 16.8 | 14.2|\n\nAs shown in the table above, Orthus exhibits slightly higher latency than Chameleon, due to the diffusion head (41M parameters) requiring ~0.001s per forward pass. **However, this latency can be significantly reduced by decreasing the number of forward passes with advanced diffusion samplers or consistency distillation (requiring only 1–4 steps for sampling [1]) and by reducing the per-step computation cost with a more lightweight MLP.** It is also worth noting that the transformer backbone of Orthus and Chameleon shares the same architecture as LLaMA, which is already supported by the vLLM serving framework (achieving up to 1k tokens/s, enabling image generation in less than 1s). This allows for further acceleration of inference speed. We will include these analyses in the revised version and will actively implement these improvements in the future.\n\n[1]Song, Yang, et al. Consistency Models. ICML 2023.\n\n**W2&Q2:** The ability to scale to higher resolution.\n\nThanks for pointing this out. Currently, Orthus supports image generation up to a resolution of 512, which is already advanced among unified multimodal models (e.g., 256 for Transfusion, 256 and 512 for Show-o, and 384 for Janus). \n\nFor even higher resolutions, such as 1024×1024, we have checked that our used VAE is capable of encoding and decoding them. Thus, the main challenge lies in the increased sequence length required for higher-resolution images. To address this, we can interpolate the RoPE positional embeddings of our transformer, following [1], and fine-tune the model to extend its capability for processing and generating longer sequences. We can compress the sequence length in attention computation with KV token compression. Alternatively, we can use another VAE with a higher spatial compression rate, such as a partitioned VAE[2], to reduce sequence length. We will explore this in future work.\n\n[1] Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis. CVPR 2025.\n\n[2] Diffusion-4K: Ultra-High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2503.18352.\n\n**Q3:** Following fine-tuning for interleaved text-image generation tasks (e.g., image editing, storytelling), is it possible for the model’s visual understanding and text-to-image generation performance to remain at the same level as before fine-tuning?\n\nIt is challenging to maintain the same level of performance after downstream SFT due to the common issue of catastrophic forgetting[1,2]. We conduct a toy experiment to mitigate this by incorporating the original instruction-tuning data during storytelling finetuning, yet still observe a performance drop of approximately 10% on the POPE benchmark. This can possibly be alleviated by better data mixture and regularization strategies.\n\n[1] An empirical investigation of catastrophic forgetting in gradient-based neural networks. ICLR 2014.\n\n[2] Overcoming catastrophic forgetting in neural networks. PNAS 2017.\n\nGiven these clarifications and improvements to your initial concerns, we hope for your reconsideration in raising your score.", "claims_and_evidence": "Yes, the claims are supported by proper evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense for the problem.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "Yes, I have checked soundness of the experimental designs.", "relation_to_broader_scientific_literature": "The proposed architecture could inspire future works on architecture design of unified visual understanding and generation model. Although previous works such as Transfusion achieves relatively good performance, the inference cost is too high. With the design of the dedicated diffusion head, the generation efficiency could be better.", "essential_references_not_discussed": "All related references are properly discussed.", "other_strengths_and_weaknesses": "Strength:\n1) The proposed design achieves impressive results on simultaneous visual understanding and generation.\n2) The proposed architecture has the potential to achieve good visual quality in a relatively efficient manner.\n3) The proposed approach generalises well to interleaved text-image generation.\n\nWeakness:\n1) A numerical comparisons on the inference latencies of the proposed approach and other baseline methods should be provided.\n2) With the current architecture with VAE, it might be still challenging to scale to higher resolution for generated images.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. See weakness (1), (2). My concern is still mainly focusing on the efficiency of the proposed approach and the ability to scale to higher resolution.\n\n2. Following fine-tuning for interleaved text-image generation tasks (e.g., image editing, storytelling), is it possible for the model’s visual understanding and text-to-image generation performance to remain at the same level as before fine-tuning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lt5MRD4Tec", "reviewer": "Reviewer_ofPN", "summary": "The paper introduces Orthus, a unified multimodal model that autoregressively generates interleaved images and text. The key idea is to handle discrete text tokens and continuous image features within a single transformer framework by employing modality-specific heads. One head is dedicated to language modeling for predicting text tokens, while the other is a novel diffusion head designed to generate continuous image patches. This design avoids the information loss typically associated with vector quantization and mitigates the noise issues inherent in diffusion models when applied jointly with text. An efficient training strategy is proposed where a pre-trained autoregressive model is adapted by replacing hard vector quantization with a soft, differentiable alternative and adding a diffusion head and fine-tuning it on a modest image dataset. Experimental results demonstrate that Orthus outperforms existing unified models on benchmarks for both visual understanding and generation, including tasks like image editing and storybook creation, showing its potential for coherent interleaved image-text generation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the time to read our paper. We address your feedback point by point below.\n\n**W1:** Overclaim of lossless.\n\nBy \"lossless\", we primarily emphasize the continuous treatment of visual signals compared to discrete tokens. We will clarify this in the revised version of the paper.\n\n**W2&Q1:** Training efficiency.\n\nPretraining a unified multimodal model from scratch is highly computationally intensive. For example, Show-o-1.3B-base uses 35M image-text pairs and requires training on 48 A100 GPUs for 500k steps, consuming thousands of GPU hours. In contrast, through adaptation of the pre-trained Chameleon, Orthus-7B-base only requires 72 GPU hours, showcasing its super efficiency.\n\n**W3&Q3:** Quantitative evaluations for interleaved image-text generation.\nThank you for your suggestion. For quantitative analysis, we use GPT-4V to compare the storybook generation quality of Orthus and MM-Interleaved[1] and report the win rate of Orthus in the table below. Results show that Orthus excels in generating logically coherent interleaved image-text with high relevance and we will add this table in the revised version.\n\n[1] MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer\n\n|   | Image quality & consistency | Text quality & continuity | Text-Image alignment |\n|---- | :-----:| :-----:| :-----:|\n|Win rate | 0.72| 0.51 |0.66|\n\nHere's the prompt we use for evaluation:\n```\n\"Please act as an impartial judge and evaluate the quality of the generation story contents provided by two AI assistants. Your job is to evaluate which assistant's generation is better. Your evaluation should consider image quality and consistency. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. After providing your explanation, output your final verdict by strictly following this format: \\\"[[A]]\\\" if assistant A is better, \\\"[[B]]\\\" if assistant B is better.\"\n```\n\n**W4&Q2:** Temperature sensitivity and noise management in the diffusion head.\n\nTo analyze the sensitivity of temperature, we ablate different choices and compare their performance in the table below. Results indicate that the model performance is robust to variations in this hyperparameter. We adopt the typical DDPM training SNR schedule[1] in the diffusion head without any tuning. More studies on these will be added to the revision.\n|  | T=0.1 | T=1 | T=10 |\n|---- | :-----:| :-----:| :-----:|\n|POPE | 78.2 |78.7  |78.0|\n\n[1] Denoising Diffusion Probabilistic Models, NIPS 2020.\n\n**S3:** Detailed information on the dataset selection.\n\nThe details of training data can be found in line322, line355-358, and line622. \n\n**S4:** Explain for d=30.\n\nThis is the model depth of the baseline method SD3.\n\n**Q4:** Robustness in imbalanced or partially missing interleaved inputs. \n\nOrthus demonstrates generalization ability for unseen interleaved inputs. As shown in Figure 3, it successfully completes the task even when provided with unseen interleaved formats during training.\n\n**Q5:** Potential limitations.\n\nOrthus exhibits slightly lower image generation efficiency than Chameleon due to the introduction of the diffusion head. Potential solutions include adopting a faster diffusion sampler or consistency distillation.\n\n**Essential References Not Discussed:** Latest benchmarks including MMIE and OpenING.\n\nThanks for your suggestion. We primarily focus on fine-tuning Orthus on downstream domain-specific tasks such as image editing and storybook generation to demonstrate its effectiveness in modeling interleaved image-text. We also finetune Orthus on the open-source [WebSight](https://huggingface.co/datasets/HuggingFaceM4/WebSight) dataset to generate webpages (HTML code equipped with images) based on text prompts. Here is an [anonymous link](https://0x0.st/828x.html) showcasing results on this downstream application for your reference. Regarding benchmarks such as MMIE and OpenING, they primarily evaluate interleaved image-text understanding and generation capabilities in more general domains, which are not our main focus. Nevertheless, we evaluate Orthus on MMIE and present the results in the table below.\n\n|  | Orthus | MiniGPT-5 | GILL |\n|---- | :---:| :---:| :---:|\n|MMIE-PBL | 0.568 | 0.551  | 0.576|\n\nDespite demonstrating reasonable performance, Orthus sometimes generates text-only responses or replies with \"Sorry, I cannot assist with that.\" due to the lack of instruction tuning or downstream SFT. We plan to finetune Orthus with more diverse and complex interleaved datasets  (e.g., MMC4) for further evaluation afterward. Moreover, OpenING evaluates interleaved generation methods through pair-wise comparisons. Since other models' results are not yet open-source, evaluation is infeasible now. We will conduct tests once they are released.\n\nGiven these improvements and additional experiments to your initial concerns, we hope you would like to raise your score.", "claims_and_evidence": "Overall, many claims are supported by extensive experiments and ablation studies. However, a few claims are still with some issues:\n\n1. The paper states that using continuous image features avoids “information loss” and is entirely lossless. While benchmark improvements (e.g., higher MME-P and OCR scores) suggest benefits, the claim is potentially exaggerated since the diffusion process itself might introduce trade-offs that aren’t fully analyzed.\n\n2. The claim that the Orthus-base model can be built “effortlessly” in only 72 A100 GPU hours is not contextualized with comparisons to similar models, making it hard to assess the real efficiency gain.\n\n3. The demonstration of robust in-context learning for tasks such as image editing and storybook generation is mainly qualitative. More rigorous quantitative evaluation or statistical analysis would strengthen the evidence for these capabilities.\n\nThese points, if addressed, would make the paper’s claims even more convincing.", "methods_and_evaluation_criteria": "The proposed methods are well-suited for the interleaved image-text generation problem. The use of modality-specific heads, which is a language modeling head for text tokens and a diffusion head for continuous image features, provides a clear and innovative way to address the shortcomings of vector quantization and noise issues in existing models. The evaluation criteria, including metrics like GenEval, MME-P, CLIP similarity scores, and OCR performance, are appropriate to assess both visual understanding and generation. However, additional quantitative analysis, especially regarding in-context learning, would further strengthen the evidence supporting these methods.", "theoretical_claims": "The paper does not offer new, formal theoretical proofs but builds on established formulations. For example, the derivation of the softmax-based alternative to vision embedding (Equation 4) is consistent with the standard property that a softmax with a temperature approaching zero approximates an argmax operation, as used in Equation 1. Similarly, the diffusion loss (Equation 3) follows common practice in diffusion models. No rigorous proofs regarding convergence or guarantees of “losslessness” are provided; instead, these claims are supported by empirical evidence. Overall, the standard derivations appear correct, though the paper could benefit from a more formal discussion of any theoretical guarantees.", "experimental_designs_or_analyses": "The experimental design appears generally sound. The paper evaluates Orthus across multiple tasks, such as interleaved image-text generation, visual understanding (using benchmarks like VQAv2, GQA, and OCR-related metrics), and text-to-image generation, with comparisons against established baselines. Ablation studies, like those on the choice of vision embedding modules and the impact of unified versus separate training, also add depth to the analysis. However, there are a few points to note:\n\n1. The evaluation of in-context learning and qualitative tasks (e.g., storybook generation) relies largely on qualitative observations with limited quantitative backing.\n\n2. There is limited discussion on hyperparameter sensitivity and statistical significance of the reported improvements.", "relation_to_broader_scientific_literature": "The paper’s contributions build directly on a range of well-established ideas in multimodal learning. First, it extends the vector quantization‐based autoregressive approaches (e.g., VQ-VAE methods as in Van den Oord et al., 2017) by replacing the hard quantization step with a soft, differentiable alternative. This adjustment seeks to mitigate the known information loss issues of converting continuous image features to discrete tokens. Second, by incorporating a dedicated diffusion head for generating continuous image patches, the paper leverages advancements in diffusion models (as seen in Ho et al., 2020 and Dhariwal & Nichol, 2021), which have proven effective in high-fidelity image synthesis.\n\nMoreover, the work positions itself against recent unified AR-diffusion models (like Transfusion and Monoformer) by decoupling the noise-inducing aspects of diffusion from the core transformer backbone. This contrasts with approaches that mix noisy image inputs with text tokens, which can hurt performance in tasks such as image captioning. The paper also parallels the masked autoregressive (MAR) models but argues that its fully autoregressive formulation better captures interleaved image-text dependencies without complex hyperparameter tuning.\n\nFinally, the efficient adaptation of pre-trained AR models—requiring relatively low training compute—aligns with current trends in making large-scale multimodal models more accessible and versatile. Overall, the paper synthesizes and extends prior findings in both language modeling and image generation, integrating them into a unified framework for interleaved image-text processing.", "essential_references_not_discussed": "Please consider discussing (or even supplying experiments) the latest benchmarks, including MMIE and OpenING.\n\nXia, Peng, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang, Wenhao Zheng, Zhaorun Chen et al. \"Mmie: Massive multimodal interleaved comprehension benchmark for large vision-language models.\" arXiv preprint arXiv:2410.10139 (2024).\nZhou, Pengfei, Xiaopeng Peng, Jiajun Song, Chuanhao Li, Zhaopan Xu, Yue Yang, Ziyao Guo et al. \"GATE OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation.\" arXiv preprint arXiv:2411.18499 (2024).", "other_strengths_and_weaknesses": "Strengths: \n1. The paper presents an innovative integration of autoregressive modeling with modality-specific heads, effectively combining continuous image representation and discrete text generation.\n2. It offers a practical and efficient training strategy by adapting pre-trained AR models, which could lower the barrier for building unified multimodal models. \n3. Extensive experiments, including ablation studies and diverse benchmarks, provide solid empirical support for the approach. \n4. The method clearly advances interleaved image-text generation, a challenging yet important task in multimodal learning.\n\nWeaknesses: \n1. Some claims, such as achieving “lossless” image representation, may be overstated given that the diffusion process could introduce its own trade-offs. \n2. The efficiency claim (e.g., 72 A100 GPU hours) lacks sufficient comparative context with similar models, making it hard to gauge its significance. \n3. Evaluation of in-context learning and qualitative tasks relies heavily on visual and anecdotal evidence, with limited rigorous quantitative analysis. \n4. The description of certain architectural details and training trade-offs could be more explicit, particularly regarding hyperparameter sensitivity and noise management in the diffusion head.", "other_comments_or_suggestions": "1. Some minor typographical and formatting issues could be addressed (e.g., consistent capitalization in headings and figure references).\n2. The discussion on temperature scheduling for the softmax-based vision embedding could be expanded for clarity. \n3. More detailed information on the dataset selection and preprocessing would improve reproducibility.\n4. Some key parts are not explained. E.g., d=30.", "questions_for_authors": "1. Could you provide more context or comparative baselines regarding the claim of 72 A100 GPU hours? How does this compare to similar models trained from scratch?\n2. What is the rationale behind the specific temperature schedule used in the softmax-based vision embedding module, and how sensitive is the model performance to variations in this hyperparameter?\n3. Can you offer additional quantitative evaluations to support the claims of in-context learning, particularly for tasks like storybook generation?\n4. How does the model handle imbalanced or partially missing interleaved inputs, and have you evaluated its robustness in such scenarios?\n5. Are there any failure cases or limitations observed during experiments that were not discussed in the paper? How might these impact the model's application in real-world tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FKo5mWPSZ3", "reviewer": "Reviewer_GAez", "summary": "This paper introduce Orthus, which is  a unified multimodal LLM for generating interleaved images and\ntext from mixed-modality inputs by simultaneously handling discrete text tokens and continuous image features under the AR modeling\nprinciple.\n\n## update after rebuttal\nAfter considering the authors' response, I am inclined to maintain my original rating of 'Weak Accept.' This decision is based on the model's capability for mixed image-text understanding and generation, but relatively weak vision embedding module.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your attentive comments! We are glad you thought our paper was well-written and easy to follow. We address your  concerns point by point below.\n\n**W1:** Orthus does not perform exceptionally well in multimodal comprehension, which may be limited by its relatively weak vision embedding module.\n\nYes. The features encoded by VAE are suited for visual generation, but they can be suboptimal for multimodal comprehension compared to CLIP-ViT due to the lack of semantic information. However, in this paper, we focus on building a unified model capable of interleaved image-text understanding and generation, where images need to be understood and generated simultaneously. Therefore, we embed images using VAE instead of CLIP-ViT. **A possible improvement is to incorporate semantic loss during the training of the VAE.** Additionally, we train solely on the LlaVa-v1.5-mix-665k dataset for multimodal comprehension currently, demonstrating the advantages of the improved soft vision embedding module and the continuous treatment of visual signals. In the future, we plan to incorporate more high-quality visual understanding data for further improvement.\n\n**W2:** The authors build Orthus based on Chameleon and substitute the VQ operation with a soft alternative and only tune the parameters of vision embedding module and diffusion head. Have the author tried using a pre-trained multimodal LLM based on a continuous ViT representation, with an added diffusion head to achieve unified multimodal understanding and generation?\n\nThank you for this interesting suggestion. It is indeed possible to train an additional diffusion head to predict representations encoded by CLIP-ViT autoregressively. However, **decoding features encoded by CLIP-ViT back into the original image is challenging.** Previous work like VQKD[1] has attempted this but the reconstructed images suffered from significant blurring and an evident loss of high-frequency details, which is attributed to the emphasis on encoding semantic information at the expense of fine-grained details. \n\nAlternatively, prior studies [2] have successfully reconstructed images using a CLIP encoder and a diffusion-based DiT decoder. However, **this design introduces redundancy by modeling the correlation between image patches both in the transformer backbone and diffusion decoder**, and the inclusion of two separate diffusion processes significantly slows down image generation.\n\n[1] Beit v2: Masked image modeling with vector-quantized visual tokenizers, arXiv:2208.06366.\n\n[2] 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities, NIPS 2024\n\n**Question:** Why does unified training perform better than understanding training only on the multimodal comprehension benchmark as listed in Tab 4? This seems rarely seen in unified multimodal understanding and generation work. For example, the Emu series conducts separate instruction tuning for understanding and generation.\n\nThe synergy may stem from unified cross-modal learning, which renders the characterization of the correlation between modalities. Moreover, both generation and understanding tasks contribute to the alignment of text and image in a shared representation space. On the other hand, previous work DreamLLM [1] also observes a synergy between generation and understanding. It suggests that training LLMs with stronger visual comprehension capabilities leads to more fine-grained encoding of text, which in turn improves text-to-image tasks.\n\n[1] DreamLLM: Synergistic Multimodal Comprehension and Creation, ICLR 2024.\n\nWe hope these reclarifications and explanations of our method in response to your initial concerns can convince you to increase your score.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The methods and evaluation criteria make sense for the unified multimodal LLM .", "theoretical_claims": "No proofs for theoretical claims in this paper.", "experimental_designs_or_analyses": "The experimental designs and analyses are reasonable to me.", "relation_to_broader_scientific_literature": "LLM-based unified multimodal comprehension and generation is a very important research direction, especially for interleaved image-text generation given mixed-modality input. This approach is quite inspiring as it overcomes the limitations of diffusion modeling by equipping a multimodal LLM with an additional diffusion head.", "essential_references_not_discussed": "The authors discussed sufficient related works.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is generally well-written and easy to follow, with clearly illustrated figures.\n2. Orthus is capable of  mixed image-text understanding and generation such as in-context editing and interleaved storybook\ncreation.\n\nWeaknesses:\n1. Orthus does not perform exceptionally well in multimodal comprehension, which may be limited by its relatively weak vision embedding module.\n2. The authors build Orthus based on Chameleon and substitute the VQ operation with a soft alternative and only tune the parameters of vision embedding module and diffusion head. Have the author tried using a pre-trained multimodal LLM based on a continuous ViT representation, with an added diffusion head to achieve unified multimodal understanding and generation?", "other_comments_or_suggestions": "No.", "questions_for_authors": "Why does unified training perform better than understanding training only on the multimodal comprehension benchmark as listed in Tab 4? This seems rarely seen in unified multimodal understanding and generation work. For example, the Emu series conducts separate instruction tuning for understanding and generation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents Orthus, an interleaved image-text generation model with modality-specific heads. After rebuttal, it received scores of 3334. All the reviewers are generally happy about the paper, commenting that (1) the idea is easy to follow and understand, (2) the model shows good performance in image editing, generation, and understanding, it also generalizes well to interleaved text-image generation. The newly reported experimental results during rebuttal is also helpful to further support the paper’s contributions. 

On the other hand, I also agree that Orthus does not perform that well in multimodal understanding, which may be limited by its relatively weak vision embedding module. Also, the benchmarks reported for multimodal understanding is a bit limited. For example, most of the popular benchmarks for testing text-rich image understanding are not reported, possibly due to performance is not strong on them. 

Overall, given the positive feedback from the reviewers and the importance of the topic, the AC agrees that the paper is in a good shape, and would like to recommend acceptance of the paper. The authors are highly encouraged to discuss the limitation of the methods in the revision as well.
Decision: Accept (poster)