Show-o: One Single Transformer to Unify Multimodal Understanding and Generation
Authors: Jinheng Xie
Weijia Mao
Zechen Bai
David Junhao Zhang
Weihao Wang
Kevin Qinghong Lin
Yuchao Gu
Zhijie Chen
Zhenheng Yang
Mike Zheng Shou
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal understanding and generation
large language model
Primary area: foundation or frontier models, including LLMs
TL;DR: One single transformer to unify multimodal understanding and generation

Abstract: We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model.

Published as a conference paper at ICLR 2025
SHOW-O: ONE SINGLE TRANSFORMER TO UNIFY MULTIMODAL UNDERSTANDING AND GENERATION
Jinheng Xie't Weijia Maolt
Zechen Bai David Junhao Zhang'1 Weihao Wang²
Kevin Qinghong Lin' Yuchao Gu' Zhijie Chen' Zhenheng Yang² Mike Zheng Shou
1 Show Lab, National University of Singapore 2 ByteDance
ABSTRACT
We present a unified transformer, i.e. Show-o, that unifies multimodal under- standing and generation. Unlike fully autoregressive models, Show-o unifies au- toregressive and (discrete) diffusion modeling to adaptively handle inputs and out- puts of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual guestion-answering text-to-image generation, text-guided appainiiggexttapolattion and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding generation. This significantly highlights its poten- tial as a next-generation foundation model. Code and models are released at http:///ghhub..oo///oo/wbbb/////
INTRODUCTION
"Alone we can do so little; together we can do so much." " Helen Keller
Over the past few years, significant advancements have blossomed in the two key pillars of multi- modal intelligence: understanding and generation (Fig. (a) and (b)). For multimodal understanding, Multimodal Large Language Models (MLLMs) like LLaVA (Liu et al., 2024c) have demonstrated exceptional capabilities in vision-language tasks such as visual guestion-answering (VQA). For the other pillar of visual generation, denoising diffusion probabilistic models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020b) have revolutionized the traditional generative paradigms (Kingma & Welling, 2013; Goodfellow et al., 2014), achieving unprecedented performance in text-to- image/video generation (Podell et al., 2023; Esser et al., 2024; Ho et al., 2022; Wu et al., 2023a).
Given these achievements in individual fields, it is natural to explore the potential of connecting them. Recent works (Wu et al., 2023b; Gt al., 2024; Ye et al., 2024a; Dong et al., 2024) have tried assemble expert models from different domains to form unified system that can handle both multimodal understanding and generation. However, existing attempts mainly treat each domain independently and often involve individual models responsible for understanding and generation separately (as shown on the left of Fig. I(c)). For instance, NEXT-GPT (Wu et al., 2023b) employs base language model for multimodal understanding but requires an additional pre-trained diffu- sion model for image generation. Nonetheless, the mainstream understanding models like LL are of transformer architecture (Vaswani al., 2017b) while each leading generation models like Stable Diffusion 3 (SD3) (Esser et al., 2024) are just another transformer. This motivates research question: can one single transformer handle both multimodal understanding and generation?
Very recently, Chameleon (Team, 2024) has demonstrated this is possible. Specifically, Chameleon enables an early fusion of different modalities to generate both text and image tokens through the same manner of autoregressive modeling. While it is reasonable to model text tokens autoregres- sively (Touvron et al., 2023; Liu et al., 2024c), it is less clear whether it is better to model im- age/video patches (or pixels) autoregressively
well. An apparent and significant bottleneck of autoregressively predicting an image is the large number of sampling steps required due to its causal attention, particularly when dealing with images/videos in higher resolution. Further, (continuous)
Equal Contribution
Corresponding Author

---

Published as a conference paper at ICLR 2025
[+) Understanding Only
(b) Generation Only
Veicn
Langunge
LLM [AR]
Language
Vsiin
Visin
E.g. LLaVA
e.g. Stable Diffusion 3
e.p., UamaCen
[c] Uniffed Model [Understanding & Generation)
Language
Vison
Langinge
Vislon
Langlage
LLM
Diffusion
LLM (AR)
LLM [AR Diffusion)
Larigsinge
Lungsinge
Langsigge
e.g. NEXT-GPT, SERT-GGT,SFFDD. SEED-X
egg. IWM, Chameleon
Show-o (Ours)
Figure 1: Characteristics comparison among understanding only, generation only, and unified (un- derstanding & generation) models. "Vision" and "Language" indicate the representations from spe- cific input modalities. In this context, "Diffusion" represents both continuous and discrete diffusion.
diffusion models (Podell et al., 2023; Esser et al., 2024) have exhibited superior capabilities in visual generation than autoregressive ones and are full attention.
This motivates us to ponder: can such one single transformer involve both autoregressive and diffusion modeling? Here we envision a new paradigm that text is represented as discrete tokens and modeled autoregressively, same with large language models (LLMs), and continuous image pixels are modeled using denoising diffusion. However, it is non-trivial to integrate these two distinct techniques into single network due the significant differences between discrete text tokens and continuous image/video representations. Another challenge lies the fact that existing state- of-the-art diffusion models typically rely on two distinct models, i.e., text encoder encode text conditional information and a denoising network to predict noise.
To this end, present a novel unified model, i.e., Show-o, capable of addressing both multimodal understanding and generation tasks simultancously with mixed autoregressive and diffusion mod- cling (as shown in Fig. 2). Specificallyy, Show-o is built upon a pre-trained LLM and inherits the autoregressive modeling capability for text-based reasoning. Inspired by Gu et al. (2022); Chang al. (2022), we employ a simplified discrete denoising diffusion, similar to MaskGIT (Chang al., 2022), to model discrete image tokens instead of continuous diffusion used in existing works (Ge et al., 2024; Dong et al., 2024). Besides, Show-o inherently encodes text conditional information, eliminating additional text encoders. To accommodate diverse input data and variations of tasks, text tokenizer and image tokenizer are employed to encode them into discrete tokens, and a unified prompting strategy is proposed further to process these tokens into structure sequences input. Consequently, given an image accompanying questions, Show-o gives the answers autoregressively. When provided only text tokens, Show-o generates images style of discrete denoising diffusion.
Quantitatively, Show-o demonstrates comparable even better performance to individual models with an equivalent or larger number of parameters across benchmarks. In contrast autoregressively gen- crating an image, Show-o requires approximately 20 times fewer sampling steps, exhibiting inherent potential in acceleration. Besides, shown in Fig. Show-o naturally supports various downstream applications like text- extt-guided inpainting and extrapolation, without any fine-tuning. Moreover, we have demonstrated that Show-o has the potential for video keyframe generation with text descriptions, video understanding. and video generation. This demonstrates the potential of the unified model as feasible paradigm for long-form video under- standing and generation. Beyond, we investigate the impact of dataset scale, image resolution, and different types of image representations (discrete or continuous) on the multimodal understanding performance, presenting systematic insights for the design of unified model in the future.
generation like interleaved
In Fig. 1, we present a comparison of model characteristics between Show-o and existing represen- tative methods across various domains. One observe that Show-o is unified model that flexibly involves existing advanced techniques to comprehensively address multimodal understanding and generation. Collectively, the main contributions of this paper can be summarized as:
2

---

Published as a conference paper at ICLR 2025
We present unified model, i.e. Show-o, which unifies multimodal understanding and generation using one single transformer.
Show-o innovatively unifies autoregressive and (discrete) diffusion modeling within one single transformer, demonstrating versatility in handling both text and images distinctly.
As unified model, Show-o demonstrates comparable even better performance to indi- vidual baseline models with an equivalent larger number of parameters in multimodal understanding and generation benchmarks.
Show-o inherently supports various downstream applications like text-based inpainting and extrapolation, without necessitating any fine-tuning. Besides, also demonstrates the po- tential for mixed-modalin generation, video understanding, and video generation.
We explore the impact of dataset scale, image resolution, and different types of represen- tations (discrete or continuous) on multimodal understanding, providing valuable insights for improving multimodal understanding capabilities of a unified model.
2 RELATED WORK
2.1 MULTIMODAL UNDERSTANDING
Significant advancements in large language models (LLMS) (Touvron al., 2023; Brown al., 2020; Chowdhery et al., 2023) have inspired the development of multimodal large language mod- els (MLLMs) (Li al., 2024; Yin al., 2023; Bai al., 2024). Early MLLM efforts, such as LLaVA (Liu et al., 2024c), MiniGPT-4 (Zhu et al., 2023a), and InstructBLIP (Dai et al., 2023), demonstrate notable multimodal understanding capabilities. To integrate LLMs into multimodal do- mains, these studies explored projecting features from pre-trained modal-specific encoder, such as CLIP (Radford et al., 2021), into the input space of LLMs, enabling multimodal understanding and reasoning within the transformer backbone. There are various design choices of MLLM (McKinzie et al., 2024; Tong et al., 2024) in vision encoders, feature alignment adapters, and datasets.
2.2 VISUAL. GENERATION
Autoregressive models. Transformer models (Vaswani et al., 2017a; Raffel et al., 2020; Brown et al., 2020; Touvron et al., 2023) have demonstrated great success of autoregressive modeling in natural language processing. Inspired by such progress, previous studies (Parmar al., 2018; Esser et al., 1021; Ravuri Vinyals, 2019; Chen et al., 2020; Kondratyuk et al., 2023) directly apply the same autoregressive modeling learn the dependency of image pixels for image/video generation. For instance, VideoPoet (Kondratyuk et al., 2023) also employs the decoder-only transformer archi- tecture for synthesizing high-quality videos from multimodal inputs. More recently, LlamaGen (Sun et al., 2024) has demonstrated ALM-architectue based image token autoregression.
Diffusion models. In recent years, diffusion-baar based methods (Rombach et al., 2022; Ramesh al., 2022b;a; Peebles & Xie, 2023; Bao et al., 2023; Podell et al., 2023; Chen et al., 2024; Nichol et al., 2021: Xue et al., 2024; Xie et al., 2023; Wu al., 2023a) have demonstrated exceptional capabili- ties in text-to-imagee iaa-immaavvveoo generation. Typically, the denoising diffusion process is operated on the continuous latent space, which the model is tasked with predicting the added Gaussian noise. In contrast, D3PM (Austin et al., 2021), Mask-predict (Gl (Ghazvininejad et al., 2019), ARDM (Hooge- boom et al., 2022), MaskGIT (Chang al., 2022), UniD3 (Hu et al., 2023), and Copilot4D (Zhang et al., 2024) formulate discrete corruption process as alternative Gaussian diffusion.
2.3 UNIFIED VISION-LANGUAGE FOUNDATION MODEL.
In recent years, an increasing number of studies (Wu et al., 2023b; Tang et al., 2024; Ye et al., 2024a; Aiello al., 2024; Lu et al., 2024) have focused on unified multimodal language models capable of both comprehension and generation. Some efforts (Zhu et al., 2023b; Sun et al., 2023b;a) use con- tinuous representations interleaved with text tokens for autoregressive modeling to generate images. SEED-X (Ge et al., 2024) proposes a unified and versatile foundation system capable of handling both multimodal understanding and generation tasks. DreamL_.N. (Dong al., 2024) also explores
3

---

Published as a conference paper at ICLR 2025
Multimodal Understanding (Caploning. VQA ...)
Visual Generation
Mixad-modality genoration
(Text-lo (Tes--iolmmge Generation Text-guided Inpainking and Extragodetion)
(Video keyframe generation with lest description)
Q1: Meae dboortee the
frog ahouting into
Hagg ixflerea roe image??
Stongg
nicrsphene while sta boukder.
foat
clerr
19ka
Text Tokenizer & Image Tokenizer
Show-o - (Causal & Full Attention)
LULH
Text De- Tokenizer Image De- Tokenizzz
Al: The mag fustoros scang prlusting hweichiingg....
outling arocado with
eoceadess entis pllle.
AZ: theme ritioos inags,
Special
dislingusshnng
Image
Figure 2:
An overview of Show-o, The input data, regardless of its modalities, is tokenized and then prompted into formatted input sequence. Show-o processes text tokens autoregressively with causal attention and image tokens (discrete) diffusion modeling via full attention, and then gener- ates the desired output. Specifically, Show-o can handle image captioning, visual question answer- ing, text-to-image generation, text-guided inpainting/extrapolatio, and mixed modality generation.
the potential of enabling multimodal comprehension and creation. Chameleon (Team, 2024) intro- duces bakee-based mixed-modal models capable of comprehending and generating images.
3 METHODOLOGY
Preliminaries. Instead of continuous diffusion, this work employs mask token prediction used in MaskGIT as simplified discrete diffusion modeling enable a more unified learning objective, i.e, predicting discrete tokens within one single transformer. We draw the connection between mask token prediction used in this work and discrete diffusion modeling Appendix
3.1 TOKENIZATION
Show-o is built upon pre-trained LLMs (Li al., 2023), it is natural to perform the unified learning on the discrete space. We maintain a unified vocabulary to include discrete text and image tokens.
Text Tokenization. Show-o is based pre-trained LLM such that utilize the same tokenizer for text data tokenization without any modifications.
Image Tokenization. Following MAGVIT-v2 (Yu al., 2023), train lookup free quantizer using a large-scale image data. The quantizer maintains a codebook of size K = 8, 192 and encodes images of 256 256 resolution into 16x 16 discrete tokens (option (a) in Fig. 3).
An alternative approach is to use different tokenizers for understanding and generation, respectively, Inspired by existing studies (Liu al., 2024c:b), also extract the con- tinuous image representations from the pre- trained MAGVIT-v2 and CLIP-ViT (Radford et al., 2021) encoder as input for exploring the improvement of multimodal understand- ing capabilities (options (b) and (c) in Fig. 3). We will present more details and discuss this exploration in Section 4.6. In the following sections, the default Show-o employs discrete image tokens input for both multimodal understanding and generation (option (a) Fig. 3). For simplicity, only elaborate the default Show-o in the methodology sections.
Option
Option
Image
Image
Image
Conwiet
Corwiet
Quantitzzion
Quardeation
Embedding (indes embedd
Projection
Projection
Continuous
Continouss
Continous
Figure 3: Optional inputs for multimodal understanding.
4

---

Published as a conference paper at ICLR 2025
Multi-madal Understanding
Visual Generation
501
Special task tokenss
image
tokens
tokene
Image tokens
Start & end toxt tokenss
Mixed-Modality Generation
SC1
E07
901
ECk
Start & end image tokens
Image
Image
Figure 4: Illustration the proposed unified prompting format.
3.2 ARCHITECTURE
Show-o inherits the architecture of existing LLM (Li et al., 2023) without any architecture modifi- cations except for prepending a QK-Norm operation (Dehghani al., 2023; Wortsman al., 2023; Team, 2024) each attention layer. We initialize Show-o with the weights of a pre-trained LLM and expand the size of the embedding layer by incorporating 8,192 new learnable embeddings for dis- crete image tokens. Unlike saae-of-the-asd diffusion models that require additional text encoder, Show-o inherently encodes text conditional information by itself for rext-to-image generation.
Unified Prompting. To perform unified learning multimodal understanding and generation, design unified prompting strategy to format various kinds of input data. Given an image-text pair y), is first tokenized into M image tokens the image and text tokenizer, respectively. We form them into an input sequence according to the type of task in the format illustrated in Fig. 4. Specifically, [MMU] and [I2I] are pre-defined task tokens that indicate the learning task for the input sequence. [SOT] and [EOT] serve as special tokens denoting the start and end of tokens, respectively. Similarly, [SOI] and [EOI] are pre-defined special tokens marking the start and end of image tokens.
{u, } and N text tokens v {u; J by
By employing this prompt design, we can effectively encode various input data for multi-modal understanding, text-to-image generation, and mixed-modalis generation as sequential data. This setup enables unified learning to operate seamlessly within sequences across these various tasks. Once trained, can accordingly prompt Show-o to handle various vision-language tasks including visual question answering and text-lo-image generation (as shown Fig. 2).
1ooae wo dbmut
Image
Image
Image
Image
Multimodal Lindesstaddin
Text-to-Image Generation
Language Modeling
Maxad-Madaliiyy Geseration
Figure 5: Omni-Attention Mechanism (The dark squares represent 'allow to attend', while white squares indicate 'prevent from attending'). It is versatile attention mechanism with causal and full attention that adaptively mixes and changes according to the format the input sequence.
Omni-Attentin Mechanism. Different from existing works (Touvron et al., 2023; Team, 2024) that model sequence auto-regressively only, we propose an omni-attention mechanism to enable Show-o model various types of signals in distinct ways. It is a comprehensive attention mech- anism with causal and full attention that adaptively mixes and changes according to the format of the input sequence. We illustrate omni-attention examples for different input sequences Fig. 5. Specifically, Show-o model text tokens within the sequence via causal attention. For image to- kens u, Show-o processes them via full attention, allowing each token to comprehensively interact with all others. Given formatted input sequence, it is apparent that in multimodal understanding (Fig. 5(a),, text tokens in a sequence can attend to previous image tokens, and in text-to-image generation (Fig. 5(b)), image tokens able to interact with all preceding text tokens. When given only text tokens, degrades to causal attention (Fig. 5(c)).
Training Objectives. To perform both auto-regressive and (discrete) diffusion modeling, employ two learning objectives: i) Next Token Prediction (NTP) and ii) Mask Token Predic- tion (MTP). Given sequence with M image tokens (u,, "2, , uus)
and N text tokens
5

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation on multimodal understanding benchmarks. Show-o currently built upon Phi-1.5 and thus we implement LLaVA-v1.5-Phii-.. as our apple-to-apple baseline. Und. and Gen. denote "understanding" and "generation", respectively. denotes the improved Show-o that employs CLIP-ViT continuous representations, We highlight the model size of Show-o and LLaVA baseline in green, and we use blue to highlight the larger model size than ours.
Type
Model
# Params
POPET
MME
Flickr30kt
VQAV2aau
GQAT
MMMUT
LLaVA.vI v1.5
2024h)
1510.7
62.0
35.4
InstructBL.I TP (Dai et ol., 2023)
13B
78.9
1212.8
49,5
Und. Only
Qwen VI. Bai
(2023)
1487.5
57.5
mPLUG-Owl2 (Yc et al., 2024b)
7B
85.8
1450,2
79.4
56.
LI Phi-1..
1128.0
56.5
30.7
Gomini-Nano-1 (Anil al., 2023)
18B
62.7
26.3
CoDI (Tang et al., 2024)
2024)
12.8
Emu (Sun ct al., 2023c)
138
77.4
57.2
NEXT-GPT (Wu et al., 2023b)
138
84.5
66.7
SEED-X (Ge ct 2024)
178
84.2
1435,7
52.3
47,9
35,6
Und. and Gen.
DreamLLMM (Dong at. 2024)
7B
72.9
VILA-U (Wu ct al., 2024)
2024)
7B
85,8
14018
79.4
60.8
31,6
Emu3 (Wang et 2024)
75.
60.3
Chamelcon (Team, 2024)
34B
74.7
66.0
Show-o (Ours)
1097.2
62.5
58.0
26.7
Show-o (Ours)
1.3B
84.5
1232.9
67.6
74.7
61.0
27.4
uN J for multimodal understanding. we maximize the likelihood of text tokens by employing the standard language modeling objective:
== fui, 02
LNTP
Σ
logpo 11 uM),
(1)
where p(-l-) indicates the conditional probability which is modeled by the weights 0 of Show-o and stochastic gradient descent is used to train the model. Note that, the input sequence involves only text tokens, there
conditional terms image tokens u {uz, ug. 1......,,00.... ,us))
With the proof in Appendix A, we seamlessly integrate the simplified discrete diffusion modeling within Show-o by employing the mask token prediction a learning objective. Hence, for modeling us } within the input sequence, we first randomly replace the image tokens with the [MASK] token, notated as u,, at random ratio (controlling by time step) create masked sequence u, {u., U2, U. uM) }. An illustration can be found in Fig. 9. Next, we aim reconstruct the original image token from the masked tokens conditioning on unmasked regions and preceding text tokens by maximizing the following likelihood:
image tokens u = {ug, "2 ...
LMTP
logpo (u, [u., U2 U. UM U1, ".. UN).
Note that the loss is only applied to the masked tokens. Specifically, we follow the sampling strategy used by MaskGIT Chang et al. (2022; 2023) to mask image tokens and reconstruct them via the information from all text and unmasked image tokens within the input sequence. Following the classifier-free guidance introduced by Ho & Salimans (2022), we randomly replace the conditioned text tokens using a null text u" with some probability.
Given batch size of input sequences, the overall training loss is the combination of CMIP and LNTP:
L LMTP + aLNTP,
where a is the hyper- paremmerreeter weighting the loss term LNTP. The training schedule mainly involves three stages, and we provide more details in Appendix C.
Inference Stage. In multimodal understanding, given an image accompanying visual questions, Show-o autoregressively predicts textual answers. In visual generation, we use all [MASK] tokens as initial input for Show-o, in which [MASK] tokens will be iteratively replaced by the predicted image tokens within T steps. More inference details are provided in Appendix D.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. We assemble two scales of datasets, i.e., around 35M and 2.0B image-text pairs, and col- lect around 2M high-quality data for multimodal understanding and generation fine-tuning. Besides,

---

Published as a conference paper at ICLR 2025
User:
Do you think the image is unueual or not?
Show-o
Chameleon:
The image is unusual because It depicts living room with cauch and tible in the miodle of the cosan. sumouunded by large body water. This for living indoora designed for milccalion antertinnent. presence
The image unuuuul becsuse of the transpanent and colorful fish coral the The
typical
while clouds. water. unusual objects the Image fish and coral
because of clear objects
living
usually
image
souch and lable unique and unoxpacted clement to soane, making stand out from typical living room selupe.
ocean adds
water. unusual objects the image are the fish and voral the water. The unusual objects in the image are the fiss and coral in the water.
Figure 6: Comparisons of VQA capabilities between Chamelcon and Show-o
RefinedWeb (Penedo et al., 2023) is adopted as text corpora to maintain the language modeling pability. Appendix E provides more details about these datasets.
Evaluation Details. Following LLaVA (Liu al., 2024b), we evaluate the multimodal understand- ing capabilities of Show-o on POPE, MME, Flickr30k, VQAv2. GQA, and MMMU benchmarks. Besides, we adopt Fréchet Inception Distance (FID) on MSCOCO dataset to evaluate the genera- tion fidelity of Show-o. Further, we follow SD3 (Esser et al., 2024) to evaluate the text-to-image generation capabilities of Show-o on the GenEval (Ghosh et al., 2023) benchmark.
Implementation details. Current version of Show-o is based on Phi-1.5 (1.3B) (Li et al., 2023). In the following, the default Show-o employs discrete image tokens as input for both multimodal understanding and generation. Show-o! and Show-o+ indicate the use of continuous image repre- sentations from the pre-trained MAGVIT-v2 and CLIP-ViT (corresponding to options (b) and (c) in Fig. 3), respectively, for multimodal understanding. Training details can be found in Appendix F.
4.2 MULTIMODAL UNDERSTANDING
Quantitative Evaluation. Table presents the multimodal understanding capability of Show-o on public benchmarks, such as image captioning and visual question-answerre tasks. i) Thhe current version of Show-o is built upon Phi-1.5 and thus we follow LLaVA to train Show-o's o's understanding only counterpart as our direct baseline, namely LLaVA-v1.5-Phi-L.5 The proposed Show-o exhibits comparable performance in all evaluation metrics to the baseline LLLVA-v ---i-111--P--1.... which is ded- icated and optimized to only multimodal understanding. This demonstrates the great potential of our framework to unify multimodal understanding and generation in one single transformer. ii) When comparing with understanding only models including InstructBLIP, Qwen- VL-Chat, and mPLUG- Owl2 on multimodal understanding, our model with much smaller model size also achieves com- petitive performance on POPE, MME, Flickr30k and VQAv2 benchmarks and performs better on GQA benchmark. iii) Compared with unified models with much larger number of parameters, such as NEXT-GPT-13B and Chameleon-34BB our model also achieves decent performance on Flickr30k benchmark and performs much better on VQAv2 benchmark.
Qualitative Results. We present Show-o's visual question- answering capability and make compar- isons with Chameleon in Fig. 6. It is evident that when presented with query image, Show-o can respond to commonly asked questions, even addressing the unusual aspects within the image. In the example of Fig. 6, when asked, "Do you think the image unusual or not", Chameleon fails to correctly identify the unusual aspect. In contrast, Show-o's 's response, "as living rooms are usually indoors and designed for relaxation and entertainment" is more accurate.
4.3 VISUAL GENERATION
Table 2: MSCOCO zero-shot FID. Und. and Gen. denote "understanding" and "generation", respectively.
Results MSCOCO 30K. We present zero-shot FID of Show-o on MSCOCO 30K in Table 2. It can be observed that, compared to genera- tion models trained with larger num- bers of parameters and training im- ages such as GLIDE and DALL-E 2,
Type
Method
A Params
# Imnges
FID-3IK]
DALL-E (Ramesh al., 2021)
250M
27.50
GLIDE (Nichol et al., 2021)
5B
250M
12.24
LDM (Rombach et al., 2022)
1,4B
400M
12.64
DALL 2 (Ramesh et 2022a)
6.50
650M
10.39
Only
(Romhach et 2022)
0.98
2000
9.62
GigaGAN (Kang et., 2012)
0.90
2700M
9.09
Pix (Chen 2114)
0.8
25M
7.32
Show-o achieves a better FID, i.e.
Imagen (Saharia
RAPHAEL (Xue 2024)
2022)
3B
38
5000M+
860M
7.27
6.61
9.24, with only 1.3B parameters and 35M training data. Though Giga-
Colli (Tang et al., 2024)
400M
22.26
LWM (Liu al., 202
7B
12.68
Und.
SEED-X
202()
GAN, Imagen, and RAPHAEL ob- tain relatively better performance
DreamL.LM (Dong et al., 2024)
7B
8.76
Show-o (Ours)
1.3B
35M
9.24
7

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation on the GenEval (Ghosh al., 2023) benchmark. Und. and Gen. denote "understanding" and "generation", respectively. We highlight the model size of Show-o in green, and we use blue to highlight the larger model size than ours. Obj.: Object. Attri.: Attribute.
Type
Method
# Params
Single Oit.
Two Obj.
Counting
Colors
Position
Color Aftri
Overallt
LlamaGen (Sun et 2024)
D.5B
0.71
0.34
0.21
0.58
0.07
0.04
0.32
LDM (Roenbach et 2022)
L:13
0.92
0.29
0.23
0.70
0.02
0.05
0.37
(Rombach
2022)
0.9B
0.97
0.38
D:04
0.06
Gen. Ouly
SDv2.1 1 (Romhach et 2122)
PixArt-alhha (Chen et 2024)
0.98
0.6B
0.98
0.98
0.50
0.51
0.44
0.44
0.85
0.80
D.07
0.08
0.17
0.07
0.50
0.48
DALL-E (Ramesh
2022a)
6.5B
0.94
0.66
0.49
0.77
0.10
0.19
0.52
SDXI (Podell et al., 2023)
2,68
0.98
0.74
0.85
0.15
0.23
0.5
SD3 (d-24) (Esser ot al,, 2024)
2B
0:98
0.74
0.63
0.67
0.34
0.36
0.62
CoDI (Tang et al., 202()
0.89
0.16
0.16
0.65
0.02
0.01
0.31
LWM (Lis
2024s)
7B
0,41
SEED-X (Go et al. 2024)
17B
0.97
0.58
0.26
0.80
0.19
0.14
0.49
Und. and Gen.
Emu3 Wang et al. (2024)
8B
0.66
Trunsfusion Zhou al. (2024)
7.3B
0.63
Chameleon (Team, 2024)
7B
Show-o (Ours)
1.3B
0.98
0.80
0.66
0.84
0.31
0.50
0.68
Show-ol (Ours)
1.38
0.98
0.85
0.67
0.81
0.28
0.55
0.69
than Show-o, they are much larger in model size (3B v.s. 1.3B) and trained with much more data. In comparison to unified models, Show-o also exhibits improvement. However, it is worth noting that FID on MSCOCO 30K may not be a comprehensively accurate assessment of generation fi- delity. The reason lies in the fact that existing generation models are commonly fine-tuned with high-quality and aesthetic images that do not align with the distribution of the MSCOCO dataset.
Results on GenEval. One can observe in Table that when comparing to the model in a similar size such as LDM (I.4B). Show-o obtains significantly better performance in all six metrics, with an improvement of around 0.24 overall. Besides, Show-o achieves a better performance than DALL-E 2, which is 5 times larger in model size, and SDXL. Further, Show-o, with only 1.3B parameters, achieves comparable performance to models with around two times larger number of parameters such SD3 (2B). It indicates that our unified model's generative capabilities are comparable or even surpass those of specialized generation models. In comparison to unified models such as CoDI, SEED-X, and Chameleon, Show-o also demonstrates significant improvements.
Qualitative Results. We show image samples generated by Show-o in Fig. 7. One can observe that Show-o capable of generating diverse, interesting, and realistic visual content in a resolution of 512x512. For example, Show-o can generate a futuristic style of car, highly detailed face, cute objects, and vivid scenery with vibrant contrast.
Text-guided Inpainting and Extrapolation. As mentioned, Show-o naturally supports text-based inpainting and extrapolation without requiring any fine-tuning. We illustrate examples Fig. (a). As shown on the top of the figure, given an input image and inpainting mask, Show-o can inpaint the original red trolley car to a blue sports car with sleek curves and tinted windows based on the user-provided text prompt. Specifically, we first tokenize the original image, mask those tokens to be inpainted, and then Show-o will gradually replace the masked tokens with predicted image tokens. Besides, Show-o is capable of extrapolating the original image horizontally/vertically based on the given text prompt. These cases significantly demonstrate the inherent advantages of Show-o over those autoregressive models for downstream applications.
4.4 MIXED-MODALLIY GENERATION OF VIDEO KEYFRAMES AND CAPTIONS
Here, we explore the mixed-modality generation ability of Show-o based on the text descriptions and video keyframes in the GenHowTo To dataset. Given a sequence of interleaved text descriptions and video keyframes (as shown at the bottom of Fig. 4), Show-o is trained to predict the next text tokens or keyframe tokens conditioning on all preceding tokens. Thus, Show-o can generate mixed- modality of text descriptions and video keyframes. Examining single frame, these tokens are generated in a diffusion manner. When considering the modeling of long sequences, as subsequent keyframes are produced based on all preceding text and image information, this can also be viewed as form of temporal auto-regressive modeling.
We have tried to train Show-o using instructional examples and present qualitative examples in Fig. 8 (b). For example, given question "Can you guide me through making Avocado and Apple Juice", Show-o exhibits the capability to generate video keyframes with text descriptions related to the question. It is apparent that the generated keyframes are temporally consistent.
8

---

Published as a conference paper at ICLR 2025
Figure Images generated by Show-o. Text prompts are provided in Appendix G.
4.5 VIDEO UNDERSTANDING AND GENERATION
Beyond image understanding and generation, have explored Show-o support video understand- ing and generation. We inflate the existing pre-trained MAGVIT-v2 to support encoding videos into discrete tokens, compressing an 8 FPS video tensor of 17 256 256 into 5 16 16. this way, following the sequence format and omni-attention of multimodal understanding and generation introduced in Section 3.2, it is convenient to fine-tune the existing Show-o involve video under- standing and generation. One can observe visual examples in Figs. 8 (c) and (d) that Show-o can accurately comprehend and describe the variations in the video and generate consistent video frames with "a jeep car is approaching from a distance". More examples can be found in Appendix H.
4.6 ABLATION STUDIES
Impact of dataset scale and image resolution on multimodal understand- ing. As only discrete image tokens extracted from the vision tokenizer, it is re- quired learn image token embeddings in Show-o from scratch. Unlike aligned image representations from the CLIP well- trained on a large-scale image-text dataset, Show-o necessitates the multimodal alignment between image and text embeddings during the pre-training stages. Here, we study the impact of the dataset scale and image resolution on the learning discrete image token embeddings for multimodal un- derstanding in Table 4. One can observe that the multimodal understanding capabilities of Show-o are consistently improved when increasing the data scale and image resolution. This reveals that it is required to involve more image-text pairs for multimodal alignment and more image tokens to represent an image for better comprchending the image information.
Table 4: Impact of dataset scale and image resolution on the learning of discrete image token embeddings for multimodal understanding.
Image-text
Resolation
POPE
MME
Flickr30k
VQAv2
GQA
MMMU
256
948.4
25.1
2.0B 2.0B
256
76.2
1014.9
48.9
64.7
54.2
25.0
5124
80.0
1097.2
62.5
69.4
58.0
26.7
As illustrated in Fig. 3(a), the default Show-o adopts the pre-trained MAGVIT-v2 to tokenize input image to discrete tokens, which are then passed to the embedding layer to obtain embeddings as input provide systematic exploration of different design
for multimodal understanding. Beyond, multimodal understanding. Specifically, as shown in choices for the input of Show-o to enhance Fig. 3(b) and (c), instead of discrete image tokens, extract the continuous image representations

---

Published as a conference paper at ICLR 2025
Original Image
"A "Ablue sports with sieck surves and ahited windows." "A oemanns natural lanowecape festuring clear Mue lake surrouundee sked by lunt Irwat."
Extrapoiaton
(a) Text-guided Inpainting and Extrapolation
User: Can you guide through making Avocado and Apple Juice?
Input image
Apple
(2) person cutting apple.
a person slicing apple plate.
(4) Avocado Peeling/Slicing.
a woman cutting avocado plate.
(6) food processor with avocado in
Peeling/Cutting.
Generated Text Descriptions with Corresponding Video Keyframes (b) Mixed- Generation
User: Describe the video content.
sitting seat. holding an closed, suggesting she is eteer aslecp or simply taking a moment to relax. The car seat she is sitting on black, and the background the video blurred, indicaing that the focus is on the girl and her ice cream, The lighting the car bright, and the car seet posilioned in front of window, providing view the oulside envirommntt.
Show-o:
appears anjoying treat, eyes
(c) Video Understanding
"Ajeep car spproaching from distance"
(d) Vidco Generation
Figure 8: Examples of text-guided oppainiingextrapolaation mixed-modality generation, video un-
derstanding and generation. 6 frames are sampled from the (generated) video for illustration.
from the pre-trained MAGVIT-v2 and CLIP-VIT VIT, respectively, as input for Show-o when dealing with multimodal understanding. We provide experimental results and insights in Appendix I.
Additionally, we present qualitative examples to illustrate the impact of sampling steps and classifier- free guidance for text-to-imas image generation in Appendix I. In observation, increasing the sampling steps can allow the synthesis of an image that closely adheres to the prompt and improve fidelity. Besides, the classifier-free guidance can significantly make the colors and contents more diverse and consistent with the given text prompt.
We also discuss the failure modes of Show-o in Appendix K.
5 CONCLUSION
This paper proposed unified transformer, i.e., Show-o, to unify multimodal understanding and generation. Show-o for the first time unified autoregressive and (discrete) diffusion modeling that handle different modalities in distinct ways. Extensive experimental results demonstrated that Show-o comparable to even better than individual expert models across a wide range of vision- language tasks. This highlighted its potential next-generation foundation model.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENTS
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No.AIIGG3-RP-202-0000)
We would like to express our sincere gratitude to Henry Hengyuan Zhao for his valuable discussions and insightful feedback on multimodal understanding, Mingrui Wang for his patient assistance in helping us set up the development environment, and Han Yao for providing preprocessed video datasets.
REFERENCES
Emanuele Aiello, LILI YU, Yixin Nic, Armen Aghajanyan, and Barlas Oguz. Jointly training large autoregressive multimodal models. In ICLR, 2024.
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Saan-Baptsste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 1, 2023.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. NeurlPS, pp. 17981-17993, 2021.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. vl: A frontier large vision-language model with versatile abilities. CoRR, abs/2308. 1tt.22088.1966,,, 2023.
Zechen Bai. Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey, arXiv preprint
arXiv:2404. 18930, 2024.
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu, All are worth words: vit backbone for diffusion models. In CVPR, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. pttpp:://iithuie comtub..com//i kaao///kkabbain////
1oyo-dataset.222 2022.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. NewrIPS, pp. 28266-28279, 2022.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, pp. 11315-11322 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- eration via masked generative transformers. arXiv preprint arXiv:2301.0m 007004, 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web- scale image-text pre-training recognize long-tail visual concepts. CVPR, pp. 3558-3568,
2021.
Junsong Chen. Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie. Zhongdao Wang. James T. Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR. OpenReview.net, 2024.
11

---

Published as a conference paper at ICLR 2025
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-moda models with better captions. arXiv preprint arXiv:2311.12793, 2023.
Mark Chen, Alec Radford. Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, pp. 1691-1703, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer. Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard. Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 14:240:1-2444 11, 2023.
Douglas
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpont vision-language models with instruction tuning, 2023.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer. Michael Tschannen, Anurag Arnab, Xiao Wang. Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver, Utku Evei, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko, Vighnesh Birodkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetic, Dustin Tran, Thomas Kipf. Mario Lucic, Xiaohua Zhai, Daniel Keysers, Jeremiah J. Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters. In JCML, pp. 7480-751 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. DreamLLM: Synergistic multimodal comprehension and creation. In ICLR, 2024.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pp. 12873-12883, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. NeurIPS, 2024.
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularig comprehension and generation. arXiv preprint arXiv:2404.1 14396, 2024.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In EMNLP, pp. 6111-6120, 2019.
Dhruba Ghosh, Hanmanch Hajishirzi, and Ludwig Schmidt. Geneval: An object- framework for evaluating text-to-image alignment. In NewrlPS, 2023.
12

---

Published as a conference paper at ICLR 2025
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR, pp. 10696- 10706, 2022.
Jonathan Ho and Tim Salimans. 1XXV:22077.15995 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, pp. 6840-6851, 2020a.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, pp. 1840-6851, 2020b.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. NeurlPS, 2022.
Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In ICLR. OpenReview.net, 2022.
Minghui Hu, Chuanxia Zheng, Zuopeng Yang, Tat-Jen Cham, Heliang Zheng, Chaoyue Wang, Dacheng Tao, and Ponnuthurai N. Suganthan. Unified discrete diffusion for simultaneous vision- language generation. In ICLR, 2023.
Minguk Kang, Jun-Jay Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In CVPR, pp. 10124-10134, IEEE, 2023.
Diederik Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational
bayes.
arXiv preprint
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et Segment anything. In ICCV, pp. 4015-4026, 2023.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: large language model for zero-shot video generation. arXiv preprint arXiv:2312. 14/25, 2023.
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, al. Multimodal foundation models: From specialists to general-purper -urppss assistants. Foundations and Trends@ in Computer Gruphics and Vision, 16(1-2): 1-214, 2024.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint 2xXi:22399004633 2023.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, pp. 26296-26306, 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2024c.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEF/CV Conference on Computer 2024.
Vision and Pattern Recognition, pp.
Brandon McKinzie, Zhe Gan, Jean- Phinlphi Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal Ilm pre-training. arXiv preprint 2rXiv:2403.06//1 2024.
13

---

Published as a conference paper at ICLR 2025
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.107441 2021.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML. pp. 4055-4064, 2018.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pp. 4195- 4205, 2023.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refined- web dataset for falcon LLM: outperforming curated corpora with web data only. In NeurlPS, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pp. 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1 67, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ICML, pp. 8821-8831 Pmlr, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. CoRR, abs/2204.06125. 2022a.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2xXi:2204.006155, 1(2):3, 2022b.
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. NeurlPS, 32, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 35:36479-36494, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In /CML, pp. 2256-2265, 2015.
Tomáš Souček, Dima Damen, Michael Wray, Ivan Laptev, and Josef Sivic. Genhowto: Learning to generate actions and state transformations from instructional videos. In CVPR, pp. 6561-6571, 2024.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06555, 065.00 2024.
Quan Sun, Yufeng Cui. Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models in-context learners. CoRR, abs/2312, 13286, 2023a.
14

---

Published as a conference paper at ICLR 2025
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. CoRR, abs/2307.05222, 2023b.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang. Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In ICLR, 2023c.
Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. NeurlPS, 36, 2024.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.
arXiv preprint
arXiv:2405.098/1 2024.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal lims. arXiv preprint arXiv:2406. 16860, 2024.
Hugo Touvron, Thibaut I Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, /tbs/2302.1397/.. 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 30, 2017a.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017b.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang. Yueze Wang. Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409, 18869, 2024.
Mitchell Wortsman, Peter Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John Co- Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint 2rXiv:2309.14332 14322, 2023.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In ICCV, 2023a.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multi- modal Ilm. arXiv preprint XXXx:22009005199, 2023b.
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409. 240.2490,,,422, 2024.
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffu- sion. In ICCV, pp. 1452-74611 2023.
Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. NeurlPS, 36, 2024.
Hanrong Ye, De-An Huang. Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, et al. X-vila: Cross-modality alignment for large language model. arXiv preprint arXiv:2405.19335, 2024a.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mpluu-ow12. Revolutionizing multi-modal large language model with modality collabo- ration. In CVPR, pp. 13040-13051, 2024b.
15

---

Published as a conference paper at ICLR 2025
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey multimodal large language models. arXiv preprint WrXiv:2306.13544 2023.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Language model beats diffusion-
Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, tokenizer is key to visual generation. arXiv preprini arXiv:23 2023.
Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun. Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion. In ICLR, 2024.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. 2024. URL https://api.
semanticscholer wrgg/orppsII: 2095555
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. CoRR, abs/2304.10592, 2023a.
Jinguo Zhu, Xiaohan Ding, Yixiao Ge, Yuying Ge. Sijie Zhao, Hengshuang Zhao, Xiaohua Wang, and Ying Shan. VL-GPT: A generative pre-trained transformer for vision and language under- standing and generation. CoRR, abs/2312.002557 2023b.
16

---

Published as a conference paper at ICLR 2025
Image corruption by adding different level mask (noise) tokens
Image generation by iteratively removing mask (noise) tokens
Figure 9: Illustration of image corruption by adding different level mask (noise) tokens and image generation by iteratively removing mask (noise) tokens the absorbing discrete diffusion paradigm.
APPENDIX
A PRELIMINARIES
In recent years, denoising diffusion prohabilistic models (DDPMs) (Ho et al., 2020a) have demon- strated unprecedented performance in text-to-image/video generation in continuous state spaces, particularly exemplified by the popular Stable Diffusion series (Podell et al., 2023; Esser et al., 2024). Concurrently, discrete denoising diffusion probabilistic models (D3PMs) (Austin al., 2021) have also shown impressive capabilities in modeling data in discrete form, featuring models like VQ- Diffusion (Gu et al., 2022) and Copilot4D (Zhang al., 2024). Further, MaskGIT (Chang et al., 2022) and Muse (Chang et al., 2023) have demonstrated a simplified discrete diffusion that can ef- fectively model discrete image tokens. Our Show-o model built upon MaskGIT so that both such discrete visual and textual tokens can share unified learning objective format. In the following, provide preliminaries for diffusion models and draw the connection between discrete diffusion and mask token prediction employed in MaskGIT.
In diffusion models, the forward process q(x1:T) xn) = II Xo (xx)) into latent variables X1. XT in different noise level. The reverse Markov pro- cess is learned iteratively remove the noises added the latent variables towards the real image distribution (Xo). In the continuous scenario, the transition distribution (xx:/xx-1) is commonly characterized by a Gaussian distribution:
r=1 g(x,|xx->) x+-1) corrupts the image data
(4) where the mean is vT- -- and the variance is Br. For images tokenized into K (i.e. the codebook size) categorical random variables x,, x,-1 E (1..... K} and given [MASK] state, the transition distribution is instead formulated by stochastic transition matrix Q (K(K (K++))
g(x, xxx]xx,1)) Nxxx - B,I),
(5) where [Qly q(x: j|x-11 1). x; indicates the row vector-matrix product, and
q(x, x,,)) Cat(x//x,,
(x Q.) is categorical distribution over the one-hot row vector x, given by 1 When the transition matrix Q applied to each image token sequence, the marginal and posterior at time step and & respectively, are formulated as:
q(x; (xo) = Cat (xx|xo Qx)
where Q = = .......... .Q...
q(x--1 |x,, xo) = g(x, X, Xx)) Xx)
Xi Xn xnQ(-11
Cat X(-1|
(6)
(xx (x))
Xy Q,
where 9(x,-1 X,, xo) 9(x_11 xx) because of the Markov property.
the following, we introduce the Absorbing-Unitne Uniform Discrete Diffusion by defining the stochas- tic transition matrix Q as follows:
Q Q Q
where em is one-hot vector with value of at the index of [MASK] token, Q (1- ar + agleT and Q I Bi(I TT)
8 (1 em)(1 em) Here, a, and B, represent the ) (K+1)
17

---

Published as a conference paper at ICLR 2025
probabilities of image token transforming into the [MASK] token and non-[MASK] token time step t, respectively, Specifically, the matrix form of Qu can be written as:
We + 1/6
1t
Vt
1t
01
Wi
1/
01
WE + 1/1
12
Q2
(8)
1/
We
1/
01
(K+1)) Intuitively, during the corruption process, each image token in the sequence has a probability of a to be replaced by the [MASK] token, a chance of yt to be uniformly diffused, and probability of we + 1 remain unchanged. Besides, if token turns into a [MASK] token, it will stay in the same [MASK] state during the following corruption process. Likewise, Q₂ Q Q₂ can be accordingly derived. An illustration of the image corruption process using [MASK] token provided in Fig. 9.
where wi (1 04 Bi) and v4
8
The evidence-lower bound (ELBO) for the variational diffusion models is:
- LELLOO (Xo, = ( [x0]
(Xo,
DKL DKI(1(xx|xD)) |l p(xT)]
log pe xx/xx))
2x
La
T
Σ 1x xo) ll po (x
(9)
2m
Considering the proposition in (Camphell et al., 2022) and the following parameterization of the reverse process:
px Σ 2(xx-11xx xo)Po (Xo xx);
(10)
the variational lower bound can be further expressed under the image distribution (xx) (referring to the proof provided by Zhang et al. (2024) detailed in the Appendix B);
E (log 2 CCELL))(CC 0)] 2 Σ E Pop C. (11)
When deriving this lower bound, the discrete diffusion paradigm can further simplified by restrict- ing each image token to be either unchanged or replaced with the [MASK] token, with possibility of becoming other categorical variables. The resulting lower bound is effectively the Cross-Entropp loss used in MaskGIT (Chang et al., 2022), which is the mask token prediction to learn a neural net- work pe to reconstruct masked regions x₂ from the noised x, In this work, we follow MaskGIT to integrate this simplified discrete diffusion paradigm into Show-o because of its simplicity. Fur- ther, Muse (Chang et al., 2023) has successfully scaled up such a paradigm for text-to-iine t--to-image models of 3B parameters using 460M image-text pairs.
18

---

Published as a conference paper at ICLR 2025
B ALTERNATIVE LOWER BOUND FOR THE VARIATIONAL DIFFUSION
Bxx()) log po (xo)]
(xx(o)) Bog / po(xo: X1 xx )dx xT)
-Bo(xo) { log (xcurtox))
po p p(xTT)
((x1:T/ xx)
log po 0x (
2 (x(xx))(xxx))
log ((XT)
((x1.T Xo)
log po (x) log log p(xT)
(xoo)))
(1Xx:(x,-))
T
(Xoor))
log Po (x (x, xx) + logp(xT)
log gl (xx xx- 1)
121
T
T
(g(())))
log
xl 80) 20),0
+ (x(xerr)
logp(xT)
log x(xx|xx-1)
121
&
121
Ci
g( ( 1, xu) x11
(B((xor)
log
Do (801xx) Oi
{(xo|xx
121
110 11/0((0)))
T
q (xx1 x2-1)
=Eg(xoo:)
log
((x1-11x1)
Po
+C1
xxx
12
%
Bx(xxor))
Σ
(xEn xx- x1-1) log
((xx111xxx)
po (((80/x1)) +Cr
+
xc)
121
&
T
g(xx-1|xx)
Bx(xo:T)
Σ
Σ
(xxم 10g8
C1 Bx(xoor)
Σ
Σ q(30 (x(-)) log
g xx)
121
X0
12
C2
T
By(xox 1xx1)
Σ 1o
+ +C1 +2
121
X0
T
Ey(xo.xi 180)0(800x pllogpo(zo/x))) +C C2
Σ By(xo 1) 1)t( ()( 1) 1og (x01x)) C2
T
Σ
B |x dat 1800 log (x) + +
121
T'
Σ
(log
(oo + + + C2
12
19

---

Published as a conference paper at ICLR 2025
The constants C1 and C2 are:
C1 = (x((o
log q (x+ x1-1) +
log p(xx)
t=1
Note pxxx)=((XT))
log (x,,((())))) X x1-1) Σ
Σ log (xxx)
C2 (xx:))
log g( x (xx_
Σ 1=1 X0
(xx0 xx-1) log g ((
(xx:))
Σ log (x1; x(-1) Σ log g(xx))
C + C2 = Eg(xoe)) [log (xx)) - Et= log ((x|/x)))
The alternative lower bound can be derived as:
E gxx)) [log Po (xo)]
xx) llog + (x(xorr) [log ((x))
Σ
log (xxo [x)]
1=1
E bea C.
This proof is provided by Zhang et al. (2024).
C TRAINING PIPELINE
Given that the embedding of image tokens is newly initialized, it necessitates large-scale pre-training to align for multimodal understanding and generation. Besides, Show-o eliminates the text encoder to extract text embeddings for text-to-image generation, which poses significant challenge for achieving effective alignment between text and image content within one single transformer. To this end, we employ three-stage approach to progressively and effectively train Show-o:
Image Token Embedding and Pixel Dependency Learning: We employ RefinedWeb (Penedo et al., 2023) dataset to train Show-o to maintain the language modeling ability. Meanwhile, ImageNet-1K 1K dataset (Deng et al., 2009) and large-scale image-text pairs are adopted to train Show- for class-conditionss image generation and image captioning, respectively. Here, we directly lever- age the class names from ImageNet-1K K as textual inputs for learning class-conditional image gen- eration. This stage primarily involves the learning of new learnable embeddings for discrete image tokens, pixel dependency for image generation, and alignment between image and text for image captioning.
ii) Image-Text Alignment for Multimodal Understanding and Generation: Building upon the pre-trained weights, we proceed to involve training of text-to-image generation on the image-text data instead of the ImageNet- I. This stage mainly focuses on image and text alignment for both image captioning and text-to-image generation.
iii) High-Quality Data Fine-tuning: Lastly, we further refine the pre-trained Show-o model by incorporating filtered high-quality image-text pairs for text-to-image generation and instructional data for multimodal understanding and mixed-modally generation.
D INFERENCE DETAILS
In inference, two types of predictions, i.e., text and image tokens, are involved Show-o. In mul- timodal understanding, given the conditional image and questions, text tokens auto-regressively sampled from the predicted tokens with higher confidence. visual generation, given N text to- kens and M [MASK] tokens initial input, Show-o predict M logits &t [M M. in parallel, where
20

---

Published as a conference paper at ICLR 2025
E E RI and t is the time step. Following the work (Chang et al., 2023), compute both the conditional logit Ex and unconditional logit é, for masked tokens. The final logit of each [MASK] token is obtained by the following equation with guidance scale
= (1 + w) wlf
(12)
For each [MASK] token at location i, we sample an image token u from the codebook based on the predicted probability p = softmax(45) and indicate its predicted score 8, E R as the confidence of this sampled token. The confidence of the unmasked token is set as 1.0. Next, we compute the number of image tokens m that should be re-masked based on the mask scheduling function 7 where = [n( MT More details of mask scheduling functions can be found in MaskGIT (Chang et al., 2022). Subsequently, we replace the predicted image tokens with [MASK] token u, based on the following metrie:
(++1) = Ur: if 9; (s;t 0 = -u4f uuf otherwise.
(13)
The resulting sequence, consisting of the remaining image tokens and [MASK] tokens, will be fed back to Show-o for the subsequent round of prediction until reaching the final time step T. The finalized image tokens are decoded by the image tokenizer into an image.
E DATASET DETAILS
Three types of data are adopted for training Show-o: i) Text-only Data: We employ the publicly available Wef dataset (Penedo et al., 2023) to preserve the text reasoning capabilities of the pre-trained LLM. This dataset comprises approximately billion instances (equivalent to 968 million individual web pages) and totals 2.8 terabytes of curated text data. ii) Image Data with Class Names: Show-o learns pixel dependencies using 1.28M images sourced from the ImageNet- IK (Deng et al., 2009) dataset. iii) Image-Text Data: For pre-training tasks correspond to multi- modal understanding and generation, we assemble roughly 35M image-text pairs from the publicly available datasets including CC1 2M (Changpinyo et al., 2021), SA1B (Kirillov et al., 2023), and LeION-aesthetics- 12M Additionally, we further increase the data scale to around 2.0B by incor- porating DataComp (Gadre et al., 2024) and COY0700M (Byeon et al., 2022) with some filtering strategies. Note that, we employ ShareGPT4V (Chen et al., 2023) to re-caption these datasets. Addi- tionally, around 1M internal image-ter texttetext pairs serve high-quality text-to-image generation datasets for the final fine-tumine tine-tunng. Following LLa 1.5 (Liu et al., 2024b), we incorporate LLa VA-PPretrann 558K and V.L.VA-vl..5mm 1A-v1l5--mix-665k for instruction tuning. Moreover, the GenHow'T To dataset (Souček et al., 2024) is utilized for mixed-modality generation.
F IMPLEMENTATION DETAILS
We initially conduct joint training of Show-o using the RefinedWeb, a collection of image-text pairs, and the ImageNet-Ie 1K for language modeling, image captioning. and class condiisoni image gener- ation, respectively, over 500K steps. Subsequently, we replace the class-conditional generation with the training for text-to-image generation using the around 35M image-text pairs for an additional 1,000K steps. The base model is trained on 48 A100 (80GB) GPUs with total batch size of 1,152. We employ the AdamW optimizer with weight decay of 0.01, 5,000 steps of warm-up, and an initial learning rate of le-4 with cosine scheduling. Finally, we fine-tune Show-o with around IM internal high-quality image-text pairs and adhere to the configuration of LLaVA-v1.5 for instruction data tuning. Note that, the current version of Show-o is based Phi-1.5 (Li et al., 2023). In the following experiment sections, the default Show-o employs discrete image tokens as input for both multimodal understanding and generation. Show-o and Show-of indicate the use of continuous image representations from the pre-trained MAGVIT-v2 and CLIP-ViT (corresponding to options (b) and (c) in Fig. 3), respectively, for multimodal understanding and we discuss this exploration in Section 4.6.
Based on the pre-trained Show-o, we continue train it on the 2.0B image-text pairs for 500K steps and then we increase the image resolution to 512 x 512 and train Show-o for an additional
http::/mieeiiiiteeeeeaaaaaeeeeer 12m-umpp
21

---

Published as a conference paper at ICLR 2025
500K steps. Finally, fine-tune Show-o with around 1M internal high-quality image-text pairs and adhere to the configuration of L for instruction data tuning.
G TEXT PROMPTS
"A 3D render of futuristic made of glass, driving through city of mirrors."
"A colorful cartoon of a tiger camouflaged in an abstract art painting, its stripes merging with the wild brushstrokes."
"The image features a stylized stained glass illustration of a hummingbird with vibrant colors, set against backdrop of swirling patterns and large sun. The composition includes floral elements and intricate details, creating a vivid and dynamic scene that emphasizes the beauty of the bird. The colors range from greens reds, enhancing the lively and artistic aesthetic of the piece."
"A 3D render of a surreal explosion scene on the shore a beautiful white sand beach with crystal clear water. The explosion has spatter of oil paint with pastel colors and thick consistency. The explosion is in quiet and serene environment. A beautiful Japanese woman with a dress compacted the sea is seen. There are butterfly petals and flowers with an ethereal glow and bioluminescence. There are pink and blue roses, and the overall image has a surreal and dreamlike quality. "
"A hyper-realistic close-up photograph of woman's face, focusing on the left side. The image is highly detailed and realistic, showing voluminous glossy lips slightly parted, a well-defined nose, and open eyes with long eyelashes that cast shadows on the skin. The eye color is crystal clear almond green. The skin texture is crisp, with incredible detail of natural, lush skin and pores and freckles, with subtle highlights and shadows that give a realistic, close-up appearance.
"A 3D render of cute, round rice ball character named Mochi, with big, sparkling eyes that convey curiosity and joy. Its body is soft, fluffy white with slight sheen, resembling freshly cooked rice. Mochi has small, rosy cheeks that give it warm, friendly expression. A tiny smile brightens its "waist," adding playful touch. Mochi's
face, and it often sports a colorful ribbon tied around arms and feet are cartoonishly short, allowing it to bounce adorably around its surroundings. This time, Mochi is placed against a background that is a vibrant explosion of colors, with bright hues of fuchsia, turquoise, lemon yellow, and emerald green creating a canvas of vibrant contrasts and playful energy. The clashing colors make Mochi's soft white body and rosy cheeks stand out even more, inviting viewers into world of cheerful exuberance and visual delight.""
H MORE EXAMPLES OF VIDEO
We provide more examples of video understanding and generation in Fig. 10.
I ABLATION STUDIES
Impact of Vision Encoder for Multimodal Understanding. The default Show-o employs MAGVIT-v2 to encode images into discrete tokens for both multimodal understanding and gen- eration. Inspired by the literature (Liu et al., 2024b), we investigate the impact of the most popular design choice of vision encoder, i.e., the pre-trained CLIP ViT (Radford et al., 2021), for mul- timodal understanding. We first compare the two settings using our Show-o model. In Table 5, the comparison between Exp 2 and Exp 4. Exp 3 and Exp 5 clearly demonstrates that continuous representations from CLIP-Vit ViT have significantly better performance on multimodal understanding than that of MAGVIT-v2. This mainly attributes to: i) The CLIP-ViT is pre-trained on much larger dataset (400M) than that of our pre-trained MAGVIT-v2 (35M); ii) In contrast to image re- construction learning objective in MAGVIT-v2, the discriminative loss, i.e., image-text matching, in CLIP-ViT makes the extracted representations easier to be adapted for multimodal understanding.
Impact of Various Representations for Multimodal Understanding, In typical multimodal un- derstanding models like LLaVA, the image representation extraction and cross-modal alignment usually happen in the continuous space. However, image tokenizers such as MAGVIT-v2 naturally yield discrete image tokens. As shown in Table 5, we compare the two types of input, i.e., con- tinuous representations and discrete tokens, in the multimodal understanding scenario. In Exp 6
22

---

Published as a conference paper at ICLR 2025
Describe the video content.
The video features man in a red shirt who is making drink by adding various ingredients to glass. He pours the mixture into strainer and a lemon slice the glass. The man then drinks drink from the glass.
Describe the video content.
The video features woman who is sitting in front of a computer and talking to the camera. She is wearing a blue shirt and has her The background
Throughout the video, video is filled with various ltems, including clock, a mirror, and shelf with clothes. The woman's expressions and gestures vary throughout
making gestures speaking
they focused
Describe the video content.
The video features woman in kitchen who cooking omelette. She is shown stirring the bowl, adding butter and grated cheese, and flipping the omelette. The video also cartoon character who is shown cooking an omelette.
blue car drives past white pickel fence on a sunny day"
'There are two dogs, one brown and one black, playing together in fenced area."
Figure 10: More examples video understanding and generation. Only some key frames sampled for illustration.
Table 5: Ablation studies of various vision encoders and kinds input representations for multimodal understanding. Note that this experiment is based on the Show-o pre-trained on 35M image-text data in resolution of 256 x 256.
# Exp
Method
Vision Encoder Unified Pretrain
Feature type
POPE
MME
Flickr30k
VQAv2,4)
GQA
MMMU
LLaVA
CLIP-VIT
X
Continuous
84.1
1128.0
69.6
73.0
56.5
30.67
2
Show
CLIP VIT CLIP-VIT
/
Continaous
84.5
1182.7
64.3
71.9
57.5
27.4
Show-ot
Continuous
84.5
1161.6
68.5
4
Show of
MAGVIT-22
/
Continuous
74.3
947.8
33.9
59.4
51.0
26.7
5
Show-o!
MAGVII-2
X
Contimuous
63.1
800.0
12.3
50.8
43.9
24.6
Show-o
MAGVII-2
Discrete
73.8
948.4
36.2
57.8
48.7
25.1
7
Show-o
MAGVIT-2
X
Discrete
63.8
689.
4.5
46.1
40.5
28.1
and 7, we use the pre-trained MAGVIT-v2 to extract discrete tokens and train an embedding layer to embed the tokens into the continuous embedding space of the LLM. In Exp 4 and 5, modify MAGVIT-v2 to output continuous representations without quantization. The cross-moue cross-modal projection layer follows the setting of LLaVA. The comparison between Exp 5 and Exp 7 reveals that discrete
23

---

Published as a conference paper at ICLR 2025
"A colorful cartoon of tiger camoufaaged in abstract art painting, its stripes merging with the wild brushstrokes."
Step
Step 25
Step 50
W 1.0
W 3.0
W ==
Illustration of generated samples using different sampling steps and classifier-free guid- ance scale w. This experiment is based on the Show-o in a resolution of 512 512.
Figure 11:
Failure Cases Multimodal Understanding
Failure Cases Visual Generation
ROADSTE
CLOSING DOWN
ALL STOCK MUST GD
MREW
What are the words this image? How many people in this picture?
'The word 'mardefly on coffee mug"
"A group of seven people slope, wearing skis posing for picture."
Show-o:
The words image
There people
"olosing stock must down." picture.
Figure 12:
Illustration of failure cases Show-o multimodal understanding and generation.
tokens show much worse performance on most benchmarks. We attribute the performance gap to that popular multimodal understanding datasets, e.g. LLaVV--rettaain5588K are not sufficient to align discrete image tokens into the language space, leading to an unsatisfactory cross-modal under- standing. In contrast, continuous representations, already lying in well-shaped embedding space, are much casier to align.
Impact of Unified Pre-training for Multimodal Understanding. Our training pipeline involves two-stage unified pre-training to learn image token embedding and image-text alignment for multi- modal understanding and generation (as described in Section C). Here we elaborate the impact of the unified per- per-training with different vision encoders and types of representations:
CLIP-ViT with Continuous Representations. The comparison between Exp 2 and Exp 3 shows that the unified pre-training has small negative effect on the CLIP ViT-based understanding, as the performance on most benchmarks has marginal degradations. We hypothesize that the MAGVIT-v2 token-based pre- pre- training and the CLIP ViT-based tuning happen in nearly orthogonal dimensions, and the capability of the backbone has been spared to maintain the compatibility the two tasks.
MAGVIT-v2 with Continuous Representations, In the comparison between Exp 4 and Exp 5, we also notice a performance improvement brought by the unified pre-training, even though the pre- pre-training uses discrete tokens while the experiments here continuous fea- tures. This comparison further validates the hypothesis that unified pre-training enhances the multimodal understanding and reasoning capabilities of the backbone by diverse mul- timodal interactions during pre-training.
MAGVIT-v2 with Discrete Tokens. The comparison between Exp 6 and Exp shows that the unified pre-training has significantly boosted the multimodal understanding per- formance. This is intuitive since the pre-training also adopts MAGVIT-v2 discrete tokens as image representation. Specifically, we attribute the performance gain to that unified pre-training learns better cross-modal alignment with large-scale data and enhances the multimodal understanding capabilities of the backbone.
Impact of Sampling Steps. We present generated results at 512 512 resolution with varying sampling steps on the left of Fig. 11. With just five steps, Show-o can produce an image that is roughly related to the given prompt. Increasing the sampling steps 25 allows the synthesis of an image that closely adheres to the prompt. When the sampling step is set as 50, the generated image becomes more detailed and realistic. In contrast, auto-regressive models Team (2024); Sun
24

---

Published as a conference paper at ICLR 2025
Original image
green apple
Asmal
Original Image
cartoonable
Figure 13: Mask free image editing.
al. (2024) require 1024 sampling steps generate an image of the same resolution when the downsampling rate is 16, which is around 20 times more steps than our approach.
Impact of Classifier-free Guidance. The visual variations of generated images with different classifier-free free guidance scales w are illustrated on the right of Fig. 11. It can be observed that the object in the generated images lacks detail without classifier-free guidance. As the classifier-free guidance scale w is gradually increased to 3 and 5, the colors and contents become more diverse and consistent with the given text prompt.
J MASK-FREE IMAGE EDITING
Given an image, it can be converted into discrete image tokens, which can then be subject to iterative random masking for sampling purposes. This process allows for image editing without the need for predefined masks. For instance, as illustrated Fig. 13, Show-o enables local-region modifications such as "changing the red apple to green" "replacing the apple with mouse." Furthermore, Show-o facilitates global style adjustments like "transforming the original image into a cartoon or oil painting style."
K FAILURE CASES
We provide failure cases of Show-o in multimodal understanding and generation in Fig. 12. The current version of Show-o sometimes cannot accurately recognize the text and count the object instances and exhibits challenges in generating correct belongings such skis for each instance. One can observe that Show-o fails to identify the phrase "closing down" in the left of Fig. 12(a) and unable generate the term "mardefly" (as shown left of Fig. 12(b)). This limitation is mainly attributed to the insufficiency of specific data tailored to these scenarios, as our model relies on limited of image-text pairs sourced from publicly available datasets and utilizes automatically generated captions. Enriching such kind of data holds promise for addressing these failure modes in Show-o, an aspect that will explored in the future.
25


Reviews:
[{"review_id": "JqciuUYiGF", "reviewer": "Reviewer_4YnC", "summary": "The work introduces Show-o, a unified transformer model that can do multimodal understanding and generation in one shared transformer model. Show-o combines autoregressive and diffusion modeling to handle various input and output modalities effectively. It supports diverse vision-language tasks such as visual question-answering, text-to-image generation, and mixed-modality tasks, showcasing competitive performance across benchmarks. The paper highlights Show-o's potential as a foundational model for the next generation and mentions forthcoming release of associated code and models.", "questions": "Please see the weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is good and clear.\n\n2. The paper introduces a compelling study on Show-o, a unified MLLM, which compares with various MLLM models, showcasing its promise in both multimodal understanding and generation tasks.\n\n3. A wide array of experiments is conducted to showcase the effectiveness of Show-o.", "weaknesses": "1. This work lacks comparisons with other unified models like Vila-u, Transfusion, and Emu3...\n\n2. Show-o appears to utilize special tokens like [mmu] for multimodal understanding and [T2I] for visual generation to handle different tasks separately. The design raises questions about its necessity, as it explicitly reveals task information. Should a new special token be introduced for each new task?\n\n3. I'm curious about Show-o's ability to manage interleaved input and output.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VB7qQjApsC", "reviewer": "Reviewer_vpwM", "summary": "This paper introduces Show-o, a unified autoregressive model for multimodal understanding and generation. Show-o integrates autoregressive and discrete diffusion modeling with an omni-attention mechanism, allowing it to utilize and output multimodal content. This design enables Show-o to handle a wide range of vision-language tasks like visual question answering, text-to-image generation, inpainting, extrapolation, and video generation with better benchmark results.", "questions": "- Are there any synergies or conflicts that arise from unifying understanding and generation, e.g., does this approach improve instruction-following capacity for generation, or reasoning capacity for text-only and multimodal inputs? Also, is there any conflict between the objectives of generation and understanding?\n- For the LLaVA-v1.5-Phi-1.5 baseline, please clarify if the same pretraining data were used for Show-o? (I noticed that the same SFT data appears to be used)\n- Please elaborate on the motivation and potential applications for unifying these two modalities, given that the current performance on both tasks still lacks usability (compared to VLM and diffusion models).\n- Additional benchmark results would be helpful. For example, FID on MS-COCO for image generation; MMBench, SEED for visual understanding (reported VQA benchmarks have strong variance or bias); GSM8K, MMLU, HumanEval for text-only tasks.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Solid experiments, comprehensive comparison with existing works.\n- Investigate a new paradigm of combining auto-regressive and discrete diffusion modeling for unifying understanding and generation.\n- Extensive ablations for scaling up data, resolution, and visual representation (discrete/continuous and different pertaining).\n- The paper is well-written and easy to follow.", "weaknesses": "- The model incurs significant training costs, making it challenging to scale, with potentially slower inference speeds.\n- The benefits of unifying understanding and generation in a single model are unclear, especially given its lower performance in both areas compared to specialized unimodal models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UAILw8b7zW", "reviewer": "Reviewer_3aSb", "summary": "The paper introduces a model called Show-o, designed to unify multimodal understanding and generation tasks within one transformer. Unlike traditional models that focus on either autoregressive or diffusion-based generation, Show-o combines both approaches to handle a wide range of inputs across tasks, including visual question answering, text-to-image generation, and text-guided inpainting.\n\nShow-o can process performs competitively on standard benchmarks, often matching or exceeding models that are specialized for either understanding or generation. Additionally, Show-o supports mixed-modality tasks, like creating video frames with text descriptions. Overall the model offers a unified approach to multimodal transformers, aiming to simplify handling different types of media in a single framework.", "questions": "- The paper discusses the use of continuous image tokenizer in a version of their model, however it's unclear to me as to how is the continuous tokenization used for image generation as Show-O does discrete diffusion? If it's only used for understanding tasks, is this model trained on less data?\n\n- Can the authors perform human evals for understanding and generation tasks?\n\n- For understanding and generation evals in Table 1 and 2, is classifier free guidance being used for baselines and Show-O. Further how does varying the guidance score affect the results.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper compares against a lot of existing baselines on a lot of benchmarks thus indicating dense evaluation.\n- The results are strong and competitive on image-text understanding and generation tasks across strong baselines.\n- Good presentation and writing, a lot of well-made figures makes the understanding of the paper easy.\n- Extension to videos and mixed modality makes the paper more applicable", "weaknesses": "- The paper doesn't do a good job at convincing why should one use discrete diffusion on Images instead of AR like Chameleon. While the introduction motivates the use of diffusion in images by discussing the number of forward evals, there are no emperical experiments showcasing this benefit. \n\n- The comparisions with baselines is difficult to asses due to the use of different training/evaluation sets for each of the baselines. A human eval, with human deciding the starting prompt might make the comparisions a lot more fair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eduiCWwfNv", "reviewer": "Reviewer_8ezu", "summary": "The paper introduces Show-o, a unified transformer model designed for both multimodal understanding and generation. By integrating autoregressive and discrete diffusion modeling, Show-o handles diverse inputs and outputs across various modalities, enabling tasks such as visual question answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Show-o achieves comparable or superior performance to specialized models in benchmarks and supports various applications.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation is clear, the proposed method is both sensible and innovative, and the experimental results are extensive and promising.\n\n- The writing is good and easy to follow.", "weaknesses": "1. The discrete diffusion modeling proposed in the paper is essentially similar to MaskGIT’s masked autoregression, rather than a new concept. It’s recommended to rephrase the description or clarify this concept at the beginning of the paper, as it could otherwise increase the readers’ learning burden.\n\n2. Besides inpainting and extrapolation, have the authors tried using [Text tokens, Image tokens, Image tokens] to perform image-to-image tasks like image editing? Can the Show-o framework support such tasks?\n\n3. When using continuous representations, Show-o may not be able to handle generation tasks. However, based on several multimodal understanding benchmarks, continuous representations seem to deliver better performance. From the perspective of unified performance metrics, Show-o currently appears less satisfactory in understanding tasks.\n\n4. Could you provide some experimental analysis of Show-o's inference speed in generation tasks?\n\n5. Does the current design of Show-o support multi-turn interaction? Can it continue text-based question answering and image generation in ongoing dialogues?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)