Beyond Text-to-Image: Liberating Generation with a Unified Discrete Diffusion Model
Authors: Qingyu Shi
Jinbin Bai
Zhuoran Zhao
Wenhao Chai
Kaidong Yu
Jianzong Wu
Shuangyong Song
Yunhai Tong
Xiangtai Li
Xuelong Li
Shuicheng YAN
Venue: ICLR 2026 Poster (2026)
Keywords: discrete diffusion
unified model
Primary area: generative models

Abstract: Unified generation models aim to handle diverse tasks across modalities—such as text-to-image generation and image-to-text generation—within a single architecture and decoding paradigm. Autoregressive unified models suffer from slow inference due to sequential decoding, and non-autoregressive unified models suffer from weak generalization due to limited pretrained backbones. We introduce Muddit, a unified discrete diffusion transformer that enables fast and parallel generation across both text and image modalities. Unlike prior unified diffusion models trained from scratch, Muddit integrates strong visual priors from a pretrained text-to-image backbone with a lightweight text decoder, enabling flexible and high-quality multimodal generation under a unified architecture. Empirical results show that Muddit achieves competitive or superior performance compared to models in both quality and efficiency. This work also highlights the potential of purely discrete diffusion, when equipped with strong visual priors, as a scalable and effective backbone for unified generation.

Published as a conference paper at ICLR 2026
BEYOND TEXT-TO-IMAGE: LIBERATING GENERATION WITH A UNIFIED DISCRETE DIFFUSION MODEL
Qingyu Jinbin Bai Zhuoran Zhao? Wenhao Chai Kaidong Yu², Jianzong Wu', Shuangyong Song", Yumhai Tong', Xiangtai Li Xuelong Li Shuicheng Yan
"Equal Contribution, 'Project Lead, *Corresponding Authors
Peking University, Nattional University of Singapore, 3 Princeton University, 'Collov Labs, TeleAI
8: jinbin. S.nbin.bai@u... nus. bus....s.euuuuuuuu
Model: https://huggingfacc..e http://aa.......eesssssso//o////
Code:
https://github. com/N /Mm AGI Lab/Muddit
ABSTRACT
Unified generation models aim handle diverse tasks across modalities-such as text-to-image generation and image-to-text generation- within a single archi- tecture and decoding paradigm. Autoregressive unified models suffer from slow inference due to sequential decoding. and ron-autoregressive unified models suf- fer from weak generalization due limited pretrained backbones. We introduce Muddit, unified discrete diffusion transformer that enables fast and parallel generation across both text and image modalities. Unlike prior unified diffusion models trained from scratch, Muddit integrates strong visual priors from a pre- trained text-to text-to-image backbone with a lightweight text decoder, enabling flexible and high-quality multimodal generation under unified architecture. Empirical results show that Muddit achieves competitive or superior performance compared models in both quality and efficiency. This work also highlights the potential of purely discrete diffusion, when equipped with strong visual priors, as a scalable and effective backbone for unified generation.
INTRODUCTION
Unified generative models have recently emerged as a promising paradigm for multimodal data, en- compassing both text and images. Most existing approaches adopt the autoregressive (AR) frame- work (Touvron et al., 2023; Zhou et al., 2024), where modalities are represented as discrete token sequences and generated sequentially in raster order. While this paradigm is well-suited for lan- guage, it introduces severe inefficiencies in image generation: producing an image requires step- by-step prediction of thousands of tokens, leading substantial computational cost. Moreover, the imposed rasterized order is poorly aligned with the inherently spatial structure of images. These limitations hinder speed/quality trade-offs and restrict flexible conditional generation, such as painting, thereby constraining the practical applicability of unified models in interactive or real-time scenarios.
To mitigate these issues, recent works (Chen et al., 2025a; Pan et al., 2025; Chen et al., 2025b) have proposed hybrid approaches that couple AR with diffusion-based image modeling methods (Ho et al., 2020; Fsser et al., 2024), shown in Fig. 1 (a). However, such "glue" architectures fall short of true unification, as they introduce additional complexity (e.g., special tokens in the template) into the training and inference pipeline. The gap between text and image modeling methods further increases the complexity, leaving current unified models without a principled multimodal generative paradigm.
As shown in Fig. (b), recent work like Dual-Diffusion (Li et al., 2024c) explores unifying multi- modal generation under the diffusion model, but ultimately relies continuous diffusion for im- age (Esser et al., 2024) and discrete diffusion for text (Swerdlow et al., 2025b; Shi et al., 2024). This fundamental mismatch in generative principles undermines its claim of true unification paradigm.

---

Published as a conference paper at ICLR 2026
Langusgge
Muddit The first Viuuti Uniteed Dacrets Diftaiion Model
Howeo
Discrete Image Medendion Gan Tulk!
neet
@ feentouingge Goveoszze
R Image Castioing
Large
Conaration
Modul
Showes, Aanuk
Hode
D Ctretiaee Disceete
Bl Vausl Qu
We propose Muddit, the first unified discrete diffusion model with a visual prior. Com- pared to language prior models like Show-o (Xie et al., 2024), Muddit demonstrates strong perfor- mance at image captioning and visual question answering. It also delivers clearer gains (7.0 vs 3.0) in image generation over the visual prior model D-DiT (Li et al., 2024c).
Figure I:
UniDisc (Swerdlow et al., 2025a) takes more promising step by applying discrete diffusion over multimodal token spaces'. This allows parallel refinement of text and image tokens, improving infer- ence efficiency and enabling more flexible conditioning. However, the overall quality of UniDisc's generation remains far from satisfactory. For example, it fails to match the fidelity of early diffusion models such Stable Diffusion v1.5 (Rombach et al., 2022), and lacks support for vision- Aision-lngguage reasoning tasks such as visual question answering (VQA). We attribute these shortcomings to the pretrained model's lack of prior knowledge. Without modular components carrying rich priors, these models face generalization and scalability bottlenecks.
Taken together, the two dark clouds: ineffective unified paradigm and the lack of strong prior knowl- edge, highlight the need for a new generation of unified models. In this work, we present Muddit, MaskGIT-style unified discrete diffusion transformer equipped with lightweight text decoder. By combining the strengths of parallel discrete diffusion and semantically rich visual priors from a pretrained Mcissonic text-to-image backbone (Bai et al., 2025b), Muddit enables scalable, efficient, and flexible sampling while significantly improving alignment and quality across modalities and var- ious tasks such as high- resolutoon text-to-image generation, image captioning, and visual question answering, as shown in Fig. I (c).
We systematically detail the training objective of unified discrete diffusion models, the mask- ing strategy, and the shared inference sampling strategy across three tasks. Finally, we conduct comprehensive evaluations with current popular unified models on several benchmarks, including GenEval (Ghosh et al., 2024), CIDEr (Lin et al., 2014), VQAv2 (Antol et al., 2015), GQA (Hudson & Manning, 2019), MME (Fu et al., 2023), and MMMU (Yue et al., 2024), demonstrating Muddit's superior performance and efficiency, validating that the unexplored purely discrete diffusion ap- proach can rival much larger autoregressive based unified models. While concurrent unified gener- ation models (Yang al., 2025) often build upon a language modeling leverginng pretrained dLLMs as the backbone-wy we instead take a visual-first approach. Muddit is built upon an image generation prior, offering new path toward unifying vision and language tasks within discrete diffusion framework. We hope that this work inspires a new trend for unified generative modeling, grounded in discrete diffusion, beyond the boundaries of traditional text-to-image generation (Bai et al., 2025b).
METHOD
2.1 DISCRETE DIFFUSION WITH UNIFIED IMAGE AND TEXT PERSPECTIVE
In discrete diffusion, sample E X' is treated as a one-hot vector x, where X = (1, (1...., N). For language models, N equals the vocabulary size. While for image models, N is the number of discrete image token IDs obtained from a tokenizer or VQ codebook. At each diffusion step, we stochastically corrupt the tokens, gradually transforming the data distribution into a maximally entropic categorical prior; the generative model then learns to invert this corruption. Following
GIT, MaskAR, RandomAR, and Discrete Diffusion share significant conceptual and practical over- laps, often differing only in decoding order or architectural nuances. We elaborate on their connections in the next section. While Meissonic (Bai et al., 2025b) follows the naming convention of MaskGIT (Chang et al., 2022), we standardize terminology in this paper by referring to all such models under the umbrella of Discrete Diffusion.
2

---

Published as a conference paper at ICLR 2026
recent works (Lou et al., 2023; Bai et al., 2025b) that cast token corruption as continuous time Markov chain (CTMC) over the finite alphabet X, we let
d Pt Qe Pt. dt
(1)
where Pt RN+1 is the distribution of 22, and the time-dependent matrix Qi transports the data distribution Po R Pdata to the maximally entropic "noise" distribution p1 = Pstationary- We adopt the absorbing-state (masked) diffusion variant that has proved particularly effective in text modelling: every symbol can jump to a dedicated mask token m (O...., O, 1) but never leaves it, i.c. is an
absorbing class.
Forward posterior. Marginalizing gives
g(z, x) Cat(ar 04 + mm))))
Cal(.) denotes categorical distribution; returns one-hot token sampled from the probability vector inside the parentheses. ag [0, 1] is the survival probability, i.e. the probability that an individual token has not yet been masked by time Thus 2, equals the original clean token with probability ar and equals the mask token m with probability 1
Reverse process. For any 8 <t < 1, the CTMC induces an analytic posterior
Cat(x, xt).
It # m,
q(z, x1,x)
Catl x. (1-a,)m (a, - )xx)x 1 1-
(3)
Te = m,
Ty and Va are the corrupted tokens at times and (s t). If v, is already a real vocabulary token (I, o m) it stays unchanged going backwards; otherwise, when Ie m, the distribution over convex combination of the mask and the clean token x, weighted by their respective survival probabilities a, and Ciz.
Training Objective.
We employ a masked-token predictor I0( at) z x, which leads to the continuous-time negative ELBO (Ou et al., 2024; Sahoo et al., 2024; Shi et al., 2024)
.NLLBO = E xx) [f/ of a- at
log(xo(22, at) dt
where a; da, and x is the one-hot vector of ground truth. s0 an) E RR is the model's predicted categorical probability vector for the clean token given the corrupted input (I,, ar).
dt
During generation, we start from all-mask sequence (t 1) and integrate the reverse CTMC towards / = 0, repeatedly replacing every masked position with the model's categorical prediction. Because the corruption schedule and objective are identical for any discrete alphabet X, the same diffusion backbone unifies text and image generation. In the following section, we present Muddit, unified framework that leverages discrete diffusion to model the generation tasks for both text and image jointly.
2.2 MUDDIT
2.2.1 UNIFIED ARCHITECTURE
As shown in Fig. 2, our architecture comprises text encoder Exxe, image encoder Eing. transformer generator G, sampler S, text decoder Dext, and image decoder Dang. The generator is a single MM-DiT model, following the dual-/single-stream design of FLUX (Labs, 2024). Importantly, the generator is initialized from Meissonic (Bai et al., 2025b), which has been extensively trained for high-resolution text-to-image generation. This initialization brings in strong pretrained im- age prior, capturing rich spatial structures and semantic correlations across image and text tokens, which significantly enhances sample quality and accelerates convergence in the multimodal setting. Consequently, the same MM-DiT predicts the masked tokens for both modalities, which produces a shared generator for text and image synthesis.
3

---

Published as a conference paper at ICLR 2026
(a) Training
(b) T21 Inference
(c) I2T Inference
Detokenizer
The Canosian
MgDes
odtronour lands
Dec
Text Deo
MM-DIT
Crose- Peree-Entrrp
MM-DIT
MM-DDIT
Me-weihheed Cross Entropy
x staps
X steps
Eno
Mask
ostrona londs
The
MH.DIT
VQ Enc
Text
Tokenizer lands
astronnuu lands the motn
The moon.
Mask tokenn
token
Text token
Trainable
Frozen
Figure 2: The training and inference architecture of Muddit. (a) During training, we randomly mask tokens from one of the two modalities. MM-DiT is trained to predict the masked tokens using a re-weighted cross-entropy loss, which jointly optimizes both the MM-DiT backbone and lightweight text decoder. (b) In text-to-image inference,
initialize the image latent features using all-masked tokens and iteratively predict each latent token via MM-DiT. (c) In image-to-text inference, similarly initialize all text tokens masked and generate the text through the same iterative decoding process. Specifically for VQA tasks, we append mask token IDs to the end of the question and predict all masked token IDs the final answer.
To reduce the computational cost of high-resolution imagery and lengthy captions, quantize both modalities into compact discrete space. A pre-trained VQ-VAE acts as the image encoder Eing. mapping pixels to codebook indices, while the CLIP text model, as Ers, provides the text token embeddings. The MM-DiT predicts clean tokens this shared space, which lightweight linear head Duxt converts back to text tokens.
2.2.2 UNIFIED TRAINING
Masking strategy. We model the forward posterior in Eq. 2 both modalities using time-dependent hyperparameters ag, with the mask ratio defined 7t employs a fixed mask ratio of 15%, this setting is suitable for token completion but insufficient for generation. support generative tasks, the design of Y must satisfy the following criteria:
a₁, While BERT (Devlin, 2018)
1. 7t must be a continuous function, bounded between O and 1, for t E [0, 1].
2. ?t should monotonically decrease with respect t, with boundary conditions % (initially clean data) and 71 - 1 (masking all tokens).
Several strategies for masking and sampling have been proposed to meet these criteria (Chang al., 2022). We adopt cosine scheduling strategy. During training, timestep / E [0, 1] is sampled from truncated arccos distribution, with the density function:
Tt (1-(111)))
(5)
During training, a mask ratio ?t E [0, 1) is randomly sampled for each modality (either image or text tokens), and the forward process (Eq. 2) applied by randomly replacing clean tokens with mask tokens to obtain Zt--
Unified training objective. Let e denote the conditioning: the text embedding when synthesizing an image, or the image embedding when generating caption. We randomly sample mask ratio by Eq. 5. Then we corrupt the target sequence Xo (image or text tokens) with the CTMC described in Eq. and train single masked-token predictor G(I,, Ot2, c) to reconstruct xp. Both directions- text + image and image- + text share the identical continuous-tinn time negative ELBO
Lumified = (x((ss/))) [f/"
of log( On, c) c).x) dt
(6)
1-
4

---

Published as a conference paper at ICLR 2026
where all symbols input. Key point: switching from text-> image to image + text merely changes the conditioning signal e; the loss Eq. 6 itself is unchanged. This symmetry keeps optimization identical across tasks and allows us to train a single parameter set jointly for both generation directions. During inference again start from all-mask sequence (!=1) and integrate the reverse CTMC towards t=0, feeding in the desired condition to obtain either an image or sentence from the same diffusion backbone.
in Eq. but the G now receives the cross-modal condition additional
2.2.3 UNIFIED INFERENCE
Sampling strategy. During inference, we apply the time-reversed posterior as defined in Eq. 3.
Cat(rr xz)) (1 - (ss)m + (at, - -4 -a) Cat Is
It o
S(G, Fe. t) = po (2's (x))
Ft = m,
1
where A denotes the parameters of G. is the multimodal condition, and 0e in Eq. is applied sequentially with taking values 1, _,,- ..... 7. where T is the total number of reverse steps. At each timestep t, Muddit predicts fraction 7 - % of the masked tokens by G and updates the masked tokens x, by S continuing iteratively until all masked tokens are recovered. This dynamic approach offers several advantages over autoregressive methods, which require the model to learn conditional probabilities P(x; x<c)) based on a fixed token ordering. In contrast, random masking with variable ratio enables the model to learn P(æ; IA), where A denotes arbitrary subset of observed tokens. This ficxibility is essential for parallel sampling, allowing multiple tokens to be predicted simultaneously rather than sequentially.
Our Muddit supports three tasks with a single generator and sampler S: text-> image, (ii) image text (captioning), and (iii) visual questioon answering (VQA). The only change across tasks is the conditioning source provided to G; the diffusion process and guidance logic are shared.
(i) Text image. Given text prompt tp T. the text encoder Exxt produces text token embed- ding Ctxt Exxt (tp). Starting from fully masked sequence T1, the generator produces logits
4 = G( aiz Cext),
-t- SS(,, 24, t),
(8)
for k = 1, T-1
T. ..... T. After steps we obtain visual tokens xo. which the image decoder Ding converts to a pixel-space image Dag (zo)).
(ii) Image-1 - text. For captioning, an input image I E I is tokenized by the image encoder Eing* Cing Eing(1) (1). The generator now conditions the visual tokens while progressively decoding text:
4 = G( atz Cing),
- S(4, 2z, t),
yielding text token sequence 2g which Dxx: maps to caption Detokenize (za))-
(iii) Image question - answer (VQA). For visual-question answering we supply both an image and question: Cing Eing (?) and Cexe = Eex: (q). They are concatenated and fed to the generator, which outputs logits over answer tokens Zş:
4 G( Cr, [Cing: Gext]),
xz = S(11, (),
(10)
until the full answer a is produced and decoded by a = Dettokenizz((Eee(r))))
Classifier-free guidance. At each decoding step, we apply the same guidance rule, independent of modality:
4 G(Ek, aik c) + G(G(z,, an c) G(Za, Ock, Cneg))
(11)
where 2 (image or text tokens) is the partial target sequence, e is the positive condition (prompt, image, or image +question), Cuog is the corresponding negative condition, and X is the guidance scale. Because the loss, decoding schedule, and guidance operator are identical all three scenarios-on -only the conditioning signal changes- our framework realises genuinely unified multimodal generator.
5

---

Published as a conference paper at ICLR 2026
3 EXPERIMENT
3.1 EXPERIMENTAL. SETUP
Implementation details. We build Muddit top of the open-source Meissonic models (Bai al., 2025b). The MM-DiT backbone is initialized with pretrained weights, and a lightweight linear head is added text decoder. Following Meissonic, we adopt the CLIP (Radford al., 2021) text encoder and VQ-VAE as image encoder and decoder, keeping them entirely frozen throughout all ex- periments. To support discrete denoising, we append special <mask> token CLIP's vocabulary for text masking, while the image mask token is inherited directly from Meissonic's 's initialization. We observe that, even without training, the <mask> embedding can already be predicted into a coherent sentence during training. Therefore, for simplicity, we freeze the <mask> embedding. During training, we use a constant learning rate of 1 10 and a weight decay of x 10-2, Gradi- ent accumulation is applied in both pretraining and supervised fine-tuning, resulting in an effective batch size of 1024. We train on 16 H100 GPUs for days. During inference, we adopt the default Meissonic configuration, using cosine masking scheduling, 64 sampling steps, and classifier-free guidance (CFG) scale of 9.0 and 1.5 for text-to-image and image-to-text generation, respectively.
Training data. We train Muddit in two stages using combination of publicly available and in- ternal datasets, including JourneyDB (Pan et al., 2023), LAION-Art (Schuhmann et al., 2022), CC12M (Changpinyo al., 2021), and others. The final dataset is filtered based aesthetic score, resolution, and aspect ratio, resulting in approximately 10 million image-text pairs. Both stages are optimized with the unified training objective defined Eq. 6. Below, we describe the datasets and settings for each stage in detail.
Pretraining. We pretrain Muddit for 100K steps with batch size of 1024, using the unified objective across both modalities. Text inputs are truncated to a maximum of 77 tokens, and images are resized to 312x512. The pretraining corpus consists of 8 million image text pairs, re- captioned using Qwen2.5-VL-33 for improved consistency. Each batch is evenly split between text-to-image and image-to-text samples to enable joint training in both directions.
2. Instruction tuning. After pretraining, we fine-tune the model on combination of 1 mil- lion followwi samples, including LLaVA- IAAA-Insttut-15000 150K, ALLaVA, SA-1B, and the VQAv2 training set. During this stage, only the answer portion of each prompt is masked. Ad- ditionally, we construct a curated dataset of 1 million high-quality image-text pairs to support multi-task training VQA and image generation. Following the task instructions embedded in each sample, Muddit learns to produce long-form answers, concise replies, and image captions via task-specific prompting.
We present both quantitative and qualitative results for the T2I and 12T tasks in the following sec- tions. Additional experiments and ablation studies are provided in the Appendix.
3.2 TEXT-TO-IMAGE GENERATION
Quantitative results. Following prior work, we evaluate our 512 x 512 model on GenEval (Ghosh al., 2024) after supervised fine-tuning in Tab. 1. Muddit attains an overall accuracy of 0.61, surpassing prior discrete diffusion models such as Monetico (0.44) and Meissonic (0.54), and closely matching Stable Diffusion 3 (0.62) with only 1B parameters. It further shows strong compositional reasoning (0.72 "Two Objects", 0.54 on "Counting"), and benefits from joint multimodal training. which enhances T2I performance. These results demonstrate the effectiveness of Muddit as the first unified discrete diffusion model for both text and image modalities.
Qualitative results. We present diverse generations from our model conditioned on rich textual prompts in Fig. 3. The outputs exhibit strong text-image alignment, capturing fine details in both realistic and imaginative scenes. Our model effectively renders complex structures, lighting, and textures across various domains.
3.3 IMAGE-TO-TEXE GENERATION
We present a comprehensive comparison of our model Muddit against other multimodal models across four benchmarks: MS-COCO (image captioning) (Lin al., 2014), VQAv2 (Antol et al.,

---

Published as a conference paper at ICLR 2026
Human: medioval town nestied on an eepansive bey, closked in blanket of snow and icc. ley winds whip through cobolastane streats
stone walls intricate carvings. Steam rises from cozy hearths as villagers hudcle around fires, their laces illuminated by lamterns, The sky is deep ind flecked with atars full moon casts etherest
bulidings.
the scane.
Human: towering, robotic whole sculated with gears and coge stands amidst bustling futuristic cityscage bathed in neon light. Its colossal form detalled riople as it glidea across shimmaring Its sye stalks glow array of glowing LEDs.
gleaming chrome
Human: An organic, bioluminescant paviion, crafted from gleaming chroma and pai paiiahed metal, risea from the tranquil depits of misty Sunbeams plarce swirting Illuminating shimmering structures in int intricate curves forms, ethereal sheen of ambient light, preating a hazy, captivating scene.
below
Human: quality photograph capturing malestic Plum Architesture
city antroopomerphic mammals. architecture is intricate and detailed, with towering structures made of natural meterials like stone wood, against nackdrop of lush the
Human: slice of creamy cheesesake, adorned with a vibrant yellow layer of lemon and flecks of zesty lemon zest. The surface is smooth and inviting, eaptured realistic detail. lighting soft, sinamatic, highllghting the intricate toxture and high definition sharpness.
photography ultra-
Human: A 3D digitally rendered portrait in the style of "glass fragments" art. The face depicted with fractured, cracked surface reminiscent of shattered glass, using muned
aplit-tonad adds depth and di ame, highlighting the emotional intonsity of the subject's expression. The image altra-high cefinitionn (4K] and rendered Photoshop.
Figure 3: Samples of Text-to-Image Generation by Muddit.
Human: How many people are there in this image?
Human: What is the color of this mug?
Assistant: The color of the cup white with blue picture on it.
Assistant: There are two people this image.
Human: What this statue made of?
Human: What on the table?
Assistant:
There are cup, saucer, and a vase flowers on the white
Assistant: statue made marble.
Figure Samples of Visual Question Answering by Muddit.
2015), MME (Fu al., 2023). GQA (Hudson & Manning, 2019), and MMMU (Yue et al.. 2024) in Tab. 2. Notably, Muddit is the first unified model to employ discrete diffusion for both text-to-image and image-to-text generation, demonstrating that this approach is highly competitive.
Quantitative comparison. Muddit demonstrates strong performance across benchmarks despite its compact size (IB parameters). achieves a CIDEr score 59.9 on MS-COCO, surpassing diffusion-based baselines such as D-DiT (56.2). On VQAv2, it attains 68.2% accuracy, outperform- ing Show-O and D-DiT. Moreover, it reaches 1107.4 on MME and 57.5 on GQA, underscoring its competitiveness across multimodal reasoning tasks. These results highlight the effectiveness of Muddit as a unified diffusion- based model that balances efficiency with high-quality task perfor- mance.

---

Published as a conference paper at ICLR 2026
Table Evaluation of text-to-image generation performance on the GenEval (Ghosh al., 2024).
Model
Text Gen
Image Gen
Parans
Overall
Objects
Counting Colors Position
Color t Attribution
Single
PixArt-o (Chen ct al.,, 2024b)
DifTusion
0.6
0.48
0.98 0.50
0.44
0.80
0.08
0.07
SD 2.1 (Rombach
2022)
Diffusion
0.9
0.50
0.98 0.51
0.44
0.85
0.07
0.17
DALLEB (Ramesh
2022)
Diffussion
SDXL (Hodell ot al., 2025)
Diffusion
2.6
0.55
0.98 0.74
0.39
0.85
0.15
0.23
DALL-B3 (Betker
2025)
DifTusion
0.67
0.96 0.87
0.47
0.83
0.43
0:45
(lisser al., 2024)
Diffission
LWM (Lii ot al., 2024c)
AR
AR
0.47
0.93 0.41
0.46
0.79
0.09
0.15
SEED-X (Ge
2024)
AR
AR
17
0.49
0.97
0.58
0.26
0.80
0.19
0.14
Chameleon (Team, 2024)
Show-O (Ki at al., 2024)
AR
Discrete Diff.
1.3
0.68
0.98 0.80
0.66
0.84
0,31
0.50
Transfusion Zhou 202()
AR
Diffusion
0.67
2024b)
Diffusion
Monctico (512 512) (Bai ot al., 20250)
Diserete Dift.
0.44
0.92 0:48
0.26
0.78
0.06
0.13
Meissoii (1024 1024)
20256)
Diserete Diff.
0.54
0.66
0.22
UniDise (512 512) (Swerdlow et al., 2025b) Discrcte Diff. Diserete Diff.
0.42
0.92
0.47
0.15
0.67
0.13
0.19
Muddit (512 512)
Discrete Diff. Discrete Diff.
0.61
0.98
0.72
0.54
0.82
0.19
0.41
Assistant:
The
holding dos and smiling, is wearing yellow and wet jacket, and he has happy expression. The background consiets of calm body water with possibly
Assistent: image depicts two
standing
lush, forasted aroa. the person on the left is older and
wearing light blue shirt and dark while the
right dressad
both
appear engaged discussion diacussing something on the elccmmnnss, with the person on the left hoiding tablet.
The image depicts a group of sheep waiking along road near a body of water, possibby lake body water watter, possinll sea or sea. the road lined with small rocks scattered vegetation, a are rolling hills calm water with fence.
Assistant:
background,
Assistant: image depists
motorcycles parked parking lot. matorcycle a white model with black and white solor scheme, identifiable by distinctive design and modern model model, evidenized by the branding on the side, the motoreycle background includes building, branding branding
Assistant:
The image depicts a brown horee standing emidst lush, green forest, the thoroughbred
characterized by dense features visible, is equipped with a bridle and a decorativ tag on its forchead. the background is rilled with dense tellage and small plants, suggesting natural, foreforested environment. the overall
apoears
Figure 5: Samples of Image-to- mnxxe-to-Text Generation by Muddit.
Qualitative results. We present example captions generated by our model across diverse scenarios in Fig. 5, including humans, animals, vehicles, and natural landscapes. The model demonstrates strong visual grounding and fine- graingr descriptive ability, accurately capturing attributes such as clothing, expressions, background context, and object relationships. Fig. 4 illustrates our model's ability to accurately answer visual questions across various domains, including object counting, color recognition, material identification, and compositional reasoning.
3.4 ABLATION STUDY AND ANALYSIS
Analysis of the inference timesteps. As shown in Tab. 5, performance generally improves with more diffusion steps, plateauing around T 32. GenEval and CIDEr see large gains from T T 32, with diminishing returns afterward. VQAv2 remains stable across timesteps, indicating
8

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation of image captioning, visual question answering on multimodal benchmarks.
Params
Text Gen
Image Gen
MS-COCO CIDEr
VQAv2
MME
GQA
MMMU
Model
(B)
Arch
Arch
Acc.
Acc.
Acc. T
Acc. 1
IntemV1.2200 al. 2025e)
LLaVA-Next (Liu et al., 2024d)
13
AR
82.8
1575.0
65.4
36.2
BLIP-2 (Li 2023)
QWEN-VL (Bai et al,, 2023)
AR
78.2
1487.5
57.5
35.9
OpenFlamingo (Awadalla al., 2023)
AR
65.5
43.5
28.7
Flamiggo (Alayae al., 2022)
AR
79.4
51.8
Chameleon (Team, 2024)
AR
AR
18.0
LWM (Liu al., 2024c)
AR
AR
55.8
Show-O (256x250) (Xie et al., 2024)
1.3
AR
Discrete Diff.
64.7
1014.9
54.2
Show-O (512x512)
al., 2024)
Transfusion (Zhou et al., 2024)
AR
Diffusion
29.0
2024b)
Diffusion
D-DIT (512x512) (Li ct al. 2024b)
Discrete Diff.
Diffusion
56.2
60.1
1124.7
39.2
UniDiie (Swerdlow al., 2025b)
Discrete Diff
Discrete Diff.
46.8
Muddit (512x512)
Discrete Ditl.
Discrete Dit..
59.9
68.2
1107.4
57.5
27.6
Muddit (1244)1024)
Discrete Diff
Discrete Diff
60.1
70.2
1139.2
57.8
28.7
Table 3: Impact of text loss weight. We apply the same loss weight during both pretrain- ing and instruction tuning.
Table 4: Effect of joint training. We denote -maaeeo-mmae as T2I and image-to-text as 12T, respectively.
Benchmark
0,2
0.4
0.6
Benchmark
T21 only
12T only
Joint training
GenEiaal
60.1
60.5
61.6
60.8
58.3
GenEival MS-COCO
99.3
28.3
61.6
MS-COCO
51.4
52.1
59.9
59.4
60.1
59.9
VQAr2
62.7
66.2
68.2
68.4
69.2
VQAv2
69.1
68.2
that fewer steps suffice for discriminative tasks. good balance between accuracy and efficiency.
Overall, moderate number of steps provides
Analysis of the text loss weight. As shown in Tab. 3, moderate text loss weights (around 0.6) yield the best overall performance. CIDEr and GenEval peak near this value, suggesting that both insufficient and excessive weighting harm generation quality. VQAv2 continues to im- prove with stronger text supervision but begins plateau beyond 0.6. Overall, while discrimina- tive tasks benefit from heavier textual guidance, generative tasks require balanced mix of visual and textual hignals-highlighting the importance of grounding language in multimodal learning.
Table 5: Performance across different diffusion timesteps.
Sample steps
GenEval
CIDEr
VQAv2
T=8
51.6
53.9
T=16
58.5
59.3
57.4
T=24
59.3
59.4
62.3
T=32
61.9
59.7
65.4
T=40
61.7
60.1
66.8
T=64
61.1
59.9
68.2
Analysis of joint training. Joint optimization over both text-to-image (T2I) and image-to-text (I2T) objectives is essential. As shown in Tab. 4, joint training yields the highest GenEval score, outperforming both T21-only and 12T-only vari- ants. Notably, I2T-only causes GenEval to drop sharply from 61.6 28.3 18.3-more more than twofold decrease- while MS-COCO CIDEr remains nearly unchanged and VQAv2 declines only slightly. These results show that separating the objectives severely weakens moos- integration, under- scoring the need for unified optimization to maintain strong multimodal coherence.
3.5 THE SCALABILITY OF MUDDIT
To demonstrate the scalability of our approach, we curate roughly 10 million image text pairs from LAION-ART (Schuhmann et al., 2022), JourneyDB (Pan et al., 2023), CC12M (Changpinyo et al., 2021) and internal datasets. We filter out samples with aesthetic score below 7, height or width under 512 pixels, or an aspect ratio above 2. All images are re-captioned using Qwen2.5-VL. 7B (Bai et al., 2023). We pretrain Muddit on this dataset with batch size of 512 and resolution of 1024, applying random masking to both image and text modalities. The image and text loss weights are set to 1.0 and 0.3, respectively. Training for 100K steps.

---

Published as a conference paper at ICLR 2026
Table 6: Training hyperparameters across different training stages.
Hyperparameters
Stage-I (Pre training)
Stage-Il (Instruction tuning)
Learning Rate
1.0 10-1
1.0 x 10-1
Scheduler
Constant
Constant
Weight Decay
0.01
0.01
Max Gradient Norm
10.0
10.0
Optimizer
AdamW (9) = 0.9, 0.999)
Batch Size
$12
512
Training Steps
100K
15K
Training GPUs
16xH100
16xH100
Gen. Resolution
1024
1024
Under. Resolution
1024
1024
Table 7: Quantitative comparison with other unified models.
Model
Params
Base model
Architecture
Data scale
Geneval w/ TTS
VQAV2
MMR
MMMU
Lamin-DaMM0 (Xin es al., 2025b)
SB
LLabM aDA
Discrete Dtff.
80M
0.92
1534.2
53.6
MMaDA (512 512) (Yang al., 2025)
8B
Diserete Tofly
Unknown
0.66
76.7
1410.7
30.2
Show O (512=512)) (as et al, 2024)
1,38
Pai 1.5
AR Discreta Didf
35M
69.4
1097.2
27.4
D-DiT (512x512) (Li 2024b)
2B
SD3 mediumm
Discrate Diff. Diff.
40M
1124.7
Madiit (512x5122
Meissonic
Dift.
0.64
1107.4
27.6
Muddit (1224xx002)
Meissonic
Discrete Ditt
70.2
1139.2
28.7
For instruction tuning, we collect about 6M samples from
(Liu et al., 2024e),
ALLaVA LAION (Chen et al., 2024a), SA-1B (Kirillov et al., 2023). ART500K (Mao al., 2017), ScienceQA (Lu et al., 2022), Chart2Text (Kantharaj et al., 2022), and VQAv2 (Antol et al., 2015). Muddit is then trained with a batch size of 512 at resolution of 1024, with masking applied only to the answer text. We also add 2M high-quality image dataset for high-quality fine-tuning. Further training configurations are provided in Tab. 6. All experiments are conducted on 16 H100 GPUs.
We evaluate the scaled Muddit model against other comparably sized unified models and state-of- the-art unified discrete diffusion models (Xin et al., 2025b; Yang et al., 2025), as shown in Tab, 7. Across established benchmarks, Muddit exhibits consistent improvements in both image generation and image understanding, empirically validating the scalability of our model. Furthermore, we com- pare Muddit with unified models of similar parameter sizes, all of which rely on hybrid architectures. Despite being trained on substantially less data, Muddit achieves superior performance.
We attribute this data efficiency to two key factors. First, our visual prior naturally maintains strong text-following capability for text-to-image generation, enabling robust alignment between image and text modalities. From the perspective of unified modeling, we prioritize cross-modal alignment over isolated single-modality ability, which allows Muddit to reach higher performance with less training data. Second, Muddit adopts fully unified modeling paradigm: the model learns by pre- dicting mask tokens based on context across all tasks (text-to-image -o-ae and image-to-text), In contrast, hybrid architectures must simultaneously handle next-token prediction alongside velocity or mask prediction, and often introduce additional special tokens (e.g., < soi >, < coi >), which increases architectural complexity and hinders optimization.
4 CONCLUSION
In this work, we present Muddit, a unified generative framework that employs discrete diffusion to bridge text and image modalities. By unifying image and text generation within a single model, Muddit demonstrates strong performance across text-to-image, image-to-text, and VQA tasks. No- tably, it matches outperforms the capabilities of significantly larger autoregressive models, while enabling fast, parallel inference. Our results validate the effectiveness of discrete diffusion as a general-purpose modeling strategy and highlight its potential to serve a scalable backbone for future multimodal systems that are equipped with strong visual priors.
Acknowledgement, This work is supported by the National Key Research and Development Pro- gram of China (No. 2023YFC3807600) and the National Natural Science Foundation of China under Grant No. 62320106007.. In addition, this work is supported in part by NUS Start-up Grant A-0010106-00-00.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Lean-Baptistt Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Ruther- ford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Ja- cob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning- In Adv. Neural Inform. Process. Syst., 2022. URL http:: nips cc wow.ettttoooeeeeeeeeeem html.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp.
2015.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open- source framework for training large autoregressive vision-language models, 2023. URL https: //arxiv. org/ abs/ 013900
Jinbin Bai, Yu Lei, Hecong Wu, Yuchen Zhu, Shufan Li, Yi Xin, Xiangtai Li, Molei Tao, Aditya Grover, and Ming- Hsuan Yang. From masks to worlds: A hitchhiker's 's guide to world models. 2025a.
arXiv preprint Ming-Hsuan
Jinbin Bai, Tian Ye, Wei Chow, Enxin Song, Qing-Guo Chen, Xiangtai Li, Zhen Dong, Lei Zhu, and Shuicheng Yan. Meissonic: Revitalizing masked generative transformers for efficient high- resolution text-to-image synthesis. ICLR, 2025b.
Jinbin Bai, Yixuan Li, Yuchen Zhu, Yi Xin, Qingyu Shi, Aosong Feng, Xiaohong Liu, Molei Tao, Jiannu Xue, Xiangtai Li, et al. Prism: Efficient test-time scaling via hierarchical search and self- verification for discrete diffusion language models. arXiv preprint arXiv:2602.01882,, 2026.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond, 2023. URL https: //tp://aaiv..a//ii//aaaeeeeeaaaa org/ abs /208 12966.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.0822 2021.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang. Juntang Zhuang, Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science., 2:3, 2023.
Huiwen Chang. Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11315-11325, 2022.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhi- hong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing sptdv-synthesized data for a lite vision-language model, 2024a.
Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. Blip3-o: A family of fully open unified multimodal architterc training and dataset, 2025a. URL https: / //ara //arxiv.eo org/ abs/ 22505.0.5... 095688.
Junsong Chen, Jincheng Yu, Chongjian Ge. Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo. Huchuan Lu, et al. Pixart-a: Fast training of diffusion transformer for photore- alistic text-to-image synthesis. In ICLR, 2024b.

---

Published as a conference paper at ICLR 2026
Liang Chen, Shuai Bai, Wenhao Chai, Weichu Xie, Haozhe Zhao, Leon Vinci, Junyang Lin, and Baobao Chang. Multimodal representation alignment for image generation: Text-image inter- leaved control is easier than you think. arXiv preprint arXiv:2502.20112, 2025b.
Sixiang Chen, Jinbin Bai, Zhuoran Zhao, Tian Ye, Qingyu Shi, Donghao Zhou, Wenhao Chai, Xin Lin, Jianzong Wu, Chao Tang, et al. An empirical study of gpt-4o image generation capabilities. arXiv preprint 25X7522504.0597, 2025c.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 2025d.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang. Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance bound- aries of open-source multimodal models with model, data, and test-time scaling, 2025e. URI. https: //tx/i/axxvvvv www.svv...../ooooooo.ooo abs/ / 2212.42122 2.0271..
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. 2018.
arXiv preprint
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi. Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309. 1499, 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprini arXiv:2410. 20t.:2400...3.3.. 13863, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song. Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi granuliranularityy comprehension and generation. arXiv preprint arXiv:2404.1 10004.14390 14396, 2024.
Dhruba Ghosh, Hannanch Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused frame- work for evaluating text-to-image alignment. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
Jianyuan Guo, Hanting Chen, Chengcheng Wang. Kai Han, Chang Xu, and Yunhe Wang. Vi- sion superalignment: Weak-to-strong generalization for vision foundation models. arXiv preprint arXiv:2402.037499 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 16000-16009, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
12

---

Published as a conference paper at ICLR 2026
Shankar Kantharaj. Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. Chart-to-text: A large-scale benchmark for chart summarization. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pp. 4005-4023. Association for Computational Linguistics, 2022.
Alexander Kirillov, Bric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In ICCV, 2023.
Black Forest Labs. Announcing black forest labs, 2024. https://blackforestlbbb aii/ anrouncing-blast blw.kllcck----o forest-
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 19730-19742. PMLR, 2023, URL https:
/ //proceedings... mlr. press //22//22////// 2ttp//1i3q.... html.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37: 66424-56445 2024a.
Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang. Dual diffusion for unified image generation and understanding, 2024b. URL htthtttts//aaaa
org/ //a//b//// 2501.2111 00289.
Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang- Dual diffuusion for unified image generation and understanding. arXiv preprint arXiv:2501.00288 2024c.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. ECV, 2014.
Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal gener- ative pretraining, 2024a. URL https: /ttp://aaaavvv/ieaeeaeeeeeeaieva http///aaii..orraa/////aa//////e 02656
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08266 2024b.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint, 2024c
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024d. URL https:// 1 lllaaavl..g thub. oo/ htt/bblgg/222--0111 30- llava- -
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2024e.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310 10. 16834, 2023.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Kai Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, 2022.
Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang- dkv-cache: The cache for diffusion language models. arXiv preprint arXiv:2505. 2025.
Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, al. Janusflow: Harmonizing autoregression and recti- fied flow for unified multimodal understanding and generation. arXiv preprint arX.v:2411.07975, 2024.
13

---

Published as a conference paper at ICLR 2026
Hui Mao, Ming Cheung, and James She. Deepart: Learning joint representations of visual arts. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1183-1191. ACM, 2017.
OpenAl. Addendum to got system card: 4o image generation, 2025. URL com/ com/index/ weiiiii///iie gppp/ppptpppttt 40- wwwiiimaaaeeeeeeeeeee generation- seseeratio--sssee------- card- aaardddddeddum/..... Accessed:
2025-04-02.
Jingyang Ou, Shen Nie. Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint M4Xx:22066033366 2024.
Junting Pan, Keqiang Sun, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou. Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Journeydb:: A benchmark for generative image understanding, 2023.
Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, et al. Transfer between modalities with metaqueries. arXiv preprint arXiv:2504.0622 2025.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhor,, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:1 37:130136 1100361111118800 130184, 2024.
Christoph Schuhmann, Richard Vencu Beaumont, Romain Gordon Vencu, Clayton Coombes, Arun Katta, Robin Kaczmarezyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next-generation image-text models. https: http//aiinnaaaaaaaaa...aaaaaaa ai/blgg laion-5b/, 2022. Ac-
cessed: 2025-09-25.
Jiaxin Shi, Kehang Han, Zhe Wang, Amaud Doucet, and Michalis Titsias. Simplified and general- ized masked diffusion for discrete data. Advances in neural information processing systems, 37: 103131-103167, 2024.
Qingyu Shi, Size Wu, Jinbin Bai, Kaidong Yu, Yujing Wang, Yunhai Tong, Xiangtai Li, and Xuelong Li. Rectok: Reconstruction distillation along rectified flow. arXiv preprint arXiv:2512.1344 2025.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang. Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint wrXiv:2307.05222 2023.
Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, and Katerina Fragki- adaki. Unified multimodal discrete diffusion. arXiv preprint urXiv:2503.2085. 2025a.
Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, and Katerina Fragki- adaki. Unified multimodal discrete diffusion. arXiv preprint arXiv:2503.20853, 2025b. doi:
14

---

Published as a conference paper at ICLR 2026
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818. 2024.
Shengbang Tong.
David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412. 2444:22412.14164, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, and Hang Xu. Illume: Illuminating your Ilms to see, draw, and self-enhance. arXiv preprint 2.Xi::2412.06673, 2024a.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409. 18869, 2024b.
Yecheng Wu, Zhuoyang Zhang. Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409, 240900429, 2024.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
Yi Xin, Siqi Luo, Qi Qin, Haoxing Chen, Kaiwen Zhu, Zhiwei Zhang, Yangfan He, Rongchao Zhang, Jinbin Bai, Shuo Cao, al. dmilm-tts: Self-verified and efficient test-time scaling for diffusion multi-modal large language models. arXiv preprint arXiv:2512.. 19433, 2025a.
Yi Xin, Qi Qin, Siqi Luo, Kaiwen Zhu, Juncheng Yan, Yan Tai, Jiayi Lei. Yuewen Cao, Keqi Wang, Yibin Wang, et al. Lumina-dimoo: An omni diffusion large language model for multi-modal generation and understanding. arXiv preprint arXiv:2510.066. 10.06000., 2025b.
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505. 15809, 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024.
Chuyang Zhao, Yuxing Song, Wenhao Wang, Haocheng Feng, Errui Ding, Yifan Sun, Xinyan Xiao, and Jingdong Wang. Monoformer: One transformer for both diffusion and autoregression. arXiv preprint arXiv:2409.166 2024.
Mengyu Zheng, Yehui Tang, Zhiwei Hao, Kai Han, Yunhe Wang, and Chang Xu. Adapt without for- getting: Distill proximity from dual teachers in vision-language models. In European Conference on Computer Vision, pp. 109-125. Springer, 2024.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 1039, 2024.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
APPENDIX OVERVIEW
This appendix provides additional discussions, results, and analyses to complement the main paper. It is organized as follows:
We review unified multimodal models for understanding and gen- eration, with focus on autoregressive and diffusion-based paradigms, as well as recent advances in masked image modeling.
Related Work (Sec. A):
Additional Qualitative Results (Sec. B): We present extended visualizations for several tasks, including image captioning, text-to-image generation, visual question answering, and image-guided text editing.
Additional Experimental Results (Sec. C): We present more experimental results.
Additional Ablation Studies (Sec. D): We present extended ablation studies.
Inference Time Analysis (Sec. E): We analyze inference efficiency by comparing autore- gressive decoding with discrete diffusion, providing FL.OPs complexity and speed bench-
marks.
Generated Results Step by Step (Sec. F): We illustrate the reverse discrete diffusion pro- cess in detail, showing intermediate decoding steps and examples of progressive generation.
Discussion (Sec. G): We reflect on the limitations of our approach and its broader impacts, including potential applications and risks of misuse.
Use of Large Language Models (Sec. H): We clarify the role of large language models during paper preparation.
A RELATED WORK
A.1 UNIFIED MODELS FOR GENERATION AND UNDERSTANDING
The success of LLMs in language modeling has inspired efforts to extend unified generation to mul- timodal domains. However, the divergence between autoregressive and diffusion-based paradigms presents fundamental architectural trade-offs. Autoregressive models naturally handle language, and several works (Sun et al., 2023; Wang et al., 2024a; Tong et al., 2024: Ge et al., 2024; Dong al., 2023; Chen et al., 2025b) extend this by connecting vision modules to L.LMs via adapters or instruc- tion tuning, with LLMS serving as planning modules that produce intermediate representations for image generation. While effective to some extent, these paradigms often exhibit limited interaction between text and image modalities and struggle with content consistency, particularly in image- synthesis. To address these limitations, recent
to-image generation and complex instruction-bart research explores unified generation models that integrate understanding and generation within single architecture. We categorize these into four major paradigms (see Fig. 6):
Fully Autoregressive: Both text and image are tokenized into discrete sequences and modeled with an AR Transformer (Liu et al., 2024b; Team, 2024; Wu et al., 2024; Wang et al., 2024b; Chen al., 2025d; Liu et al., 2024a; Guo et al., 2024; Zheng al., 2024). These models achieve strong cross-modal generation but suffer from high latency due to sequential decoding.
Text AR, Image Diffusion: LLMs generate text tokens while image synthesis delegated to pre- trained continuous diffusion backbones (Zhou et al., 2024; Zhao et al., 2024; Ma et al., 2024) or discrete diffusion (Xie et al., 2024). Though visually strong, these models are not truly unified, as they rely separate architectures and token spaces.
Image Diffusion, Text Discrete Diffusion:
Emerging models experiment with discrete diffusion for text and images (Li al., 2024c), though many, like Dual-Diffusion (Li et al., 2024c), still use continuous diffusion for image synthesis, failing to realize true modality symmetry.
Fully Discrete Diffusion: Recent work like UniDise (Swerdlow et al., 2025a) pioneers full-token discrete diffusion over shared Transformer backbones. These models support parallel sampling and native integration, but currently lag behind in generation fidelity and scale.
16

---

Published as a conference paper at ICLR 2026
(8)
Text & Image
Text
Image
Image
Text
Text & Image
Continuous/
Continuous
Discrete Diff.
Autoregressive (AR)
AR
Discrete Diff.
Discrete Diff.
Diff.
Text & Image
Text Image
Text & Image
Text & Image
Figure 6: Four types of unified generative models. More details can be found in Sec. A.
Among these, the GPT-4o (OpenAI, 2025) model represents significant advance as a unified mul- timodal generative system. However, its closed-source nature obscures critical architectural and training details, and its success may be largely attributable scale rather than architectural nov- eity (Chen et al., 2025c).
A.2 MASKED IMAGE MODELING
Masked Image Modeling (MIM) has emerged as a powerful self-supervised learning paradigm in computer vision (Bai et al., 2025a), drawing inspiration from the success of Masked Language Mod- eling (MLM) in NLP, notably BERT (Devlin, 2018). The fundamental principle of MIM involves obscuring portions of an image, which could be raw pixels (MAE (He et al., 2022)), latent patches of pixels, or even discrete latent tokens (BEiT (Bao et al., 2021), MaskGIT (Chang et al., 2022)), and training model, typically an autoencoder, to predict or reconstruct this missing information by leveraging the context provided by the visible parts.
MaskGIT (Chang et al., 2022) introduced parallel decoding via iterative token refinement, inspiring discrete diffusion models. Recent work such as RandomAR (Fan et al., 2024) and MAR (Li et al., 2024a) formalize this as random-order or masked autoregressive generation, blending AR and MIM principles. The major conceptual difference between RandomAR/MAR and MaskGIT is in the scanning order inference time.
This class of techniques forms the conceptual foundation discrete diffusion over tokenized spaces (Shi et al., 2025) and plays critical role in modern unified models. We will introduce discrete diffusion in the next section.
A.3 RELATIONSHIP TO CONCURRENT WORK
Our main contribution is to demonstrate that unified, visual-prior fully discrete diffusion model can be both effective and data-efficient for image understanding tasks, rather than just text-to-image generation tasks. Regarding the distinction from concurrent discrete diffusion works (Yang et al., 2025; Bai et al., 2026; Xin et al., 2025b;a), we think that unified models should allow for multiple design choices. Our goal is to demonstrate that a visual-first, fully discrete diffusion backbone can be practical and competitive alternative to the more common "LLM-first" unified paradigm, and we believe this a fundamental design choice.
Specifically, prior unified discrete diffusion models, such UniDisc (Swerdlow et al., 2025b), trained from scratch on multimodal data and therefore lack strong visual priors. As a result, they significantly underperform early diffusion baselines such as Stable Diffusion 1.5 (Rombach et al., 2022) and do not support visual question answering tasks (Antol et al., 2015). In contrast, Muddit the first unified discrete diffusion model built top of pretrained high-resolution text-to-image backbone (Bai et al., 2025b), with a lightweight text decoder on top. This visual prior is not an implementation detail: it improves the scalability and generalization behavior of discrete diffusion through a visual backbone that is well aligned with text.
B ADDITIONAL QUALITATIVE RESULTS
Image-to-text Generation. We present more examples for image-to-text generation in Fig. 7. Text-to-image Generation. We present more examples for text-to-image generation in Fig.
17

---

Published as a conference paper at ICLR 2026
Visual Question Answering. We present examples for visual question answering in Fig. 9. Muddit reliably identifies fine-grained attributes (e.g., "blonde" hair), object categories (e.g.. "bea- gle"), and physical affordances (e.g., answering "No" to crossing at red light). Notably, it also handles commonsense reasoning and spatial localization, such as inferring traffic legality or locat- ing vehicles the street.
Image-guided text editing. Zero-shot text-guided image editing performance is already verified and presented in Meissonic (Bai et al., 2025b). As the successor Meissonic, we present Muddit's performance on the image-guided text editing task, where the model completes a masked sentence based on the input image. shown in Fig. 10, given partially masked caption and an image, Muddit fills in the blanks with semantically and visually grounded phrases.
C ADDITIONAL EXPERIMENTAL RESULTS
We provide a detailed breakdown the MME benchmark results in Tab. 8. Muddit demonstrates strong performance in existence, color, and scene understanding, while also exhibiting solid reason- ing capabilities.
Table 8: Detailed MME results.
Category
Task
Score
Existence
135.00
Count
78.33
Position
53.33
Color
140.00
Posters
62,24
Perception
Celebrity
56.18
Scene
107.25
Landmark
94.50
Artwork
76.00
OCR
52.50
Total
855.34
Commonsense Reasoning
78.57
Numerical Calculation
90.00
Cognition
Text Translation
57.89
Code Reasoning
57.50
Total
283.97
D ADDITIONAL ABLATION STUDIES
D.1 ABLATION STUDY ON THE CFG FOR IMAGE-TO-TEXT GENERATION
As shown in Tab. 9, we report performance on MS-COCO captioning and VQAv2 benchmarks. Moderate CFG values (e.g., 1.5) yield the best results, while higher scales lead to degraded perfor- mance.
E INFERENCE TIME ANALYSIS
As shown Fig. 13, autoregressive multimodal models are inherently limited by token-by-toker decoding, which constrains their inference speed. Muddit overcomes this bottleneck with parallel discrete diffusion decoder, reducing average latency to just 1.49 seconds, achieving a 4x to 11x speedup competitive baselines (4.2x faster than Qwen-2.5-VE 5.6x than Show-o, 8. x than BLIP-2, and 10.9x than LLaV6-).6).
We also present detailed FL.OPs comparison between Autoregressive and Discrete Diffusion.
18

---

Published as a conference paper at ICLR 2026
Table 9: Ablation study on the effect of classifier-free guidance (CFG) scale.
Dataset
CFG
CFG 1.5
CFG=
CFG 2.5
CFG =
MS-COCO
57.2
59.9
58.2
51.3
47.2
VQAv2
65.8
68.2
64.7
55.4
49.2
Table 10: Comparison of model efficiency across different resolutions and steps. We report through- for both text-to-image generation (images per second) and image-to-text tasks (tokens per sec- ond). Muddit achieves the best overall balance, matching the highest text-to-image throughput while significantly outperforming others in image-to-text speed.
Model
Image Res
Steps
Text-to-lmage (img/s)
Image-to- Text (token/s)
Meissonic
1024
32
0,23
UniDisc
512
32
0,89
79.36
Monetico
512
32
1.00
D-DiT
512
28
0.62
26.89
Muddit
512
32
1.00
99.98
Autoregressive (AR) without KV Cache:
At step i, the model attends over previous tokens.
Per-step attention FLOPs: O(t² D).
Total FLOPs:
L =
L D
-0(D-
L(L ((L+1)(22) +1)
O(PD)=
Σ 12
= O(L3D)
6
1=1
Autoregressive (AR) with KV Cache:
step I, Q is computed for token, and attends to K/V keys.
Per-step attention FL.OPs O(I)).
Total FLOPs:
O(LD) = 11 = =0(D-
1
L(L +1++1)
= O(LDD
2
Discrete Diffusion:
Each step updates the full sequence (length L) in parallel.
Per-step attention FLOPs: O(I2D).
Total FLOPs:
T- (LL(()) - O(TLDD) T&L
While discrete diffusion may appear less efficient than autoregressive (AR) models with KV caching terms of theoretical FL.OPs, it offers significant advantage over AR without an L/T speedup by updating the full token sequence in parallel over T iterations. In practice, the higher degree of parallelism leads to competitive, and often faster, inference speed compared to AR models, especially when considering worldw GPU throughput. As KV cache techniques for discrete diffusion are rapidly evolving (Ma et al., 2025), we expect further acceleration in the near future, narrowing the theoretical speed gap with KV-cache AR baselines.
In Tab. 10, we compared Muddit against other non-autoregressive models, running all tests on single A800 80 GB GPU. Muddit demonstrated clear advantage in both image and text generation.
19

---

Published as a conference paper at ICLR 2026
F GENERATED RESULTS STEP BY STEP
Muddit frames text generation as reverse discrete diffusion over a fixed-length sequence of 77 token indices. At inference time, the model performs 16 ''' S 32 denoising steps, starting from max- imally entropic prior where every token is masked. At each step i, parameter-shared transformer predicts a categorical distribution over all positions parallel, and sampler selects the next sequence:
(12) where E V77 is the token sequence at step and denotes conditioning inputs. The logits be tempered or top-k filtered before sampling each token independently. The resulting sequence X,-1 seeds the next step, enabling fast, parallel decoding without autoregressive constraints.
- ((G( C,t), x,, =T
Because all positions are updated in parallel, Muddit preserves global syntactic and semantic struc- ture throughout the reverse diffusion process-unlike left-to-right autoregressive models, which can only condition on past predictions. Empirically. as few as 16 < T < 32 steps are sufficient to approximate the natural language distribution with high fidelity. Thus, Muddit unifies diffusion gen- eration with parallel decoding, effectively overcoming the serial bottleneck that limits conventional autoregressive multimodal models.
We present two examples in Fig. 11 and 12.
G DISCUSSION
G.1
LIMITATIONS
While Muddit advances discrete difffusion for unified multimodal generation, it still has several limi- tations. First, due to its token-level discrete representation, the model may underperform continuous diffusion models in generating photorealistic or high-resolution images. Second, Muddit is initial- ized from pretrained text-to-image foundation model, which offers strong visual priors but limits its capacity for rich text understanding and generation compared to the latest large language models. This makes less suitable for tasks that require long-form understanding and generation deep linguistic reasoning.
G.2 BROADER IMPACTS
Muddit explores a new paradigm in multimodal generation by leveraging a strong visual prior as the backbone, in contrast to the prevailing trend of scaling large language models. This offers complementary path toward efficient, grounded multimodal generation, particularly in vision-centric applications. The model's ability generate aligned visual and textual outputs in fast, parallel manner could benefit downstream tasks, especially in completion-ban based scenarios such as masked captioning, image editing, and code implementation. However, as with all generative models, there remains a risk of misuse in synthetic content creation.
H THE USE OF LARGE LANGUAGE MODELS
During the preparation of this paper, large language models were used only for language polishing and minor editing. All research ideas, methods, and experimental results carried entirely by the human authors.
20

---

Published as a conference paper at ICLR 2026
Assistant:
The image depicts charming street with cobblestone pavement and tail buildings. A white dog is resting. The dog is lying down near the blcycle on the street. To the left of the street, there is an old bicycle parked to the side the street.
The image is black and white photograph featuring three Individuals sitting together on bench. They sppeer to be an urban setting, as indicated by the trees and buildings in the background. The individuals police
Assistant:
with person looking individual from second individual.
first
The image depicts an elderly man standing on a city street, wearing a red jacket and a black cap talking on cell phone to his ear. The background shows crowd people walking around, some whom handbage and others visible in the distance. The crowd appears middle of
Assistant:
The image depicts a close - up of large, round pizza placed on wooden table. The pizza topped with cheese, and there are several pieces background,
Assistant:
people sitting sround.
Assistant: The image depicts tennis match on a blue court. tennis player, wearing shirt and white shorts, mid action and holding tennis racket, possibly preparing for shot. the court has row of seats.
set against backdrop
CHAMPI PI
BOS
Assistant: The image depicts vibrant and colorful arrangement of umbretlas hanging from the nches trees. umbrellas come color and shape, including pink, and yellow,
purple. They erranged circular pattern across the umbrellas, creating visuelly appeating display. The background features a mix of green trees and foliage, suggesting an outdoor.
Assistant: The image depicts bustting urban street scene with several notable foreground, tall green The clock clock face pole, which prominently on indicsting the time. In the background, there are severel, parked cars.
Assistant: The image depicts person surfing surfboard The surfer is in mid-sir, with his arms bent, nevigating the belance of the wave. The bockground shows clear blue sky, adding depth to the overall scene. The surfer seems to be the middle the the wave.
image depicts giraffe standing a grassy outdoor setting. The background features clear blue sky with some scattered clouds, and there are trees the distant distance.
The giraffe appears to be looking towards distant horizon.
The image depicts stop sign in front of grassy field with majestic mountains in the background. The sky clear clouds, end snow-capped mountains riae the background. The scene appears setting.
Assistant:
STOP
clearwith with scattered
Figure 7: Image-to-text generated results.
21

---

Published as a conference paper at ICLR 2026
Human: An oditorial fashion photo portrait of striking avant- avant-garde model with bubble details. Iridescent art and pop surrealism influence the image's ethereal feel. Pestelaesthetic hues of soft pink, tangerine, and green create dreamy backdrop. Seamless pink background, studio lighting emphasizes the model's silhouette against the soft pastel palette.
Human: Dark, heavy rainclouds gather rocky mountain range, obscuring peaks. weathered tant flaps open wind, silhouette against the gray backdrop. pours sheets, blurring the image creating misty effect.
Human: quaint small town nestled amidst snow-capped hills, bathed in the soft, golden hues of dusk. The sky tapestry of twilight blue and orange, casting long shadows across cobblestone streets frosted trees stand bare against the fading light. A sense of quiet peace emanates from the scene.
lake twilight. majestic heron stands the tranquil waters, long legs poised, the last golden rays sunlight paint sky hues blue and purple. The distant mountain range glows warmly with soft, crepuscular light. Sharp focus the heron and water reflection. Wide-angle lens captures panoramic scene.
Human:
An cyanotype photograph capturing
Human: vibrant vibrant Japanese garden, inspired by Van Gogh's switling brushstrokes. Vivid reds, yellows, blues, and greens dominate scene.
Cherry blossoms bloom in full splendor against a backdrop of ancient stone lanterna, rendered with lush foliage and dappled sunlight. The effect heightened by 4K resolution and cinematic depth.
Figure 8: Text-to-image generation results.
22

---

Published as a conference paper at ICLR 2026
Human: Can someone legally walk across the right
Human: What color is kid's hair?
Assistant: color of the kid's hair blonde.
Assistant: No.
What brand of dogs are these?
Human:
What is flying in the picture?
Human:
Assistant:
The brand of dogs are beagle.
Assistant: Kite.
Human:
What eating utensil
Where the cars?
Human:
is pictured?
Assistant: The cars are the street.
Assistant: Fork.
Figure 9: Visual question answering results.
<mask> <mask> <mask> lying on a grassy surface. <mask> <mask> has <mask> fur with darker patches its face and ears. looking directly at <mask> <mask>, the bear's mouth is slightly <mask>, revealing its teeth and tongue. the background shows some green grass.
Muddit
Image-guided text editing A curly bear lying on a grassy surface, the bear has a brown fur with darker patches on its face and ears, looking directly the camera. the bear's mouth is slightly open, revealing its teeth and tongue. the background shows some green grass.
Figure 10: Image-guided text editing results.
23

---

Published as a conference paper at ICLR 2026
User: Please describe this image.
T=1 <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <masko <maske <mask>
<mask>> <mask> <mask> <mask> <maske <mask> <mask> <mask> <mask> <mask> <mask> <mask>
<masko> <mask> <mask> <mask> <mask> <mask> <mask> <masko <mask> <masko <mask> <mask>
<mask> <mask> <mask> <mask> <mask> <mask> <mask> <maske <mask> <maske <mask> <mask>
<mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask? <mask> <mask> <mask> <mask>
<mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask>
<mask> <mask> depicts <mask> <mask> and whimsical <mask> <mask> <maske <mask> <maske
<maske <mask> <mask> <maske, <mask> <mask> <mask> <mask> <maske <mask> <mask>, <maske
<mask> <mask> <mask> <mask> <mask> <mask> <mask> <masko <mask> <masko <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <maske <mask> <maske <maske <mask>
<mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask>
The <mask> depicts <maske and whimsical <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask>, <mask> <mask> <mask> <mask> <mask> <maske <mask>, the <mask> <mask> <mask> <mask> <maske <mask> <maske <mask> <mask> <mask> <maske <maske <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <maske <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <masle <mask>,
<mask> <mask> <mask> <masle <mask> <mask>
The image depicts surreal and whimsical scene in what appears be <mask> <mask?, possibly <mask> <mask> or a dining room. the floor covered with smask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <maske <mask> <mask> <mask> <mask> <mask> <maske <maske <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <maskc> <mask> <mask> <mask>, <mask> <mask> <mask> <mask> <mask
The image depicts surreal and whimsical scene in what appears to be domestic setting, possibly room dining room. the floor is covered with numerous pink flowers, adding touch of <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask?, <mask> to the <mask> <mask> <mask> <mask> <mask> <mask> <mask>
image depicts surreal and whimsical scene appears domestic setting, possibly room or dining room. floor covered with numerous pink flowers, adding touch of <mask>, petals are scattered throughout the room, adding to the dreemlike quality the scene.
The image depicts surreal and whimsical scene in what appears to be domestic setting, possibly room dining room. the floor is covered with numerous pink flowers, adding touch of color. the petals scattered throughout the room, adding the dreamtike quality the
The image depicts surreal and whimsical scene in what appears to be domestic setting, possibly dining room. the flooris with petals are scattered throughout room, adding the dreamtike quality the scene.
pink flowers, adding touch of color. the
T=0
Figure 11: Image-to-text generated results in each step.
24

---

Published as a conference paper at ICLR 2026
User: Please describe this image.
T=1 <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <maske <mask>
<masko <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask>
<masko <mask> <mask> <mask> <mask> <mask> <mask> <mask> <maske <mask> <mask>
<mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask? <mask? <mask>
<mask> <mask> <mask? <mask> <mask> <mask> <mask> <mask> <mask> <mask> <maske <mask>
<mask> <mask> <mask? <mask> <mask? <mask> <mask> <mask> <mask> <mask>
<mask> image
<mask> <mask> <mask> <mask> <mask>
<masko <mask>
<mask> <mask> <mask> <mask> <mask> <mask> <mask> <maske <mask> <masko <mask? <mask>
<mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask? <mask> <mask> <maske <mask>
<mask> <mask> <mask? <mask> <mask> <mask> <mask> <mask> <mask> <maske <maske <mask>
<mask> <mask> <mask? <mask> <mask> <mask> <mask> <mask> <mask> <maske <maske <maske
<mask> <mask> <mask> <mask> <mask> <mask> <mask>
<mask> image depicts <mask> bear lying <mask> <maske <mask> <mask> <mask> <mask>
<mask> <mask> <mask> <mask> <mask> <mask> <mask> <maske <mask> <mask> <mask? <mask>
<mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <maske <mask> <mask> be <mask> <mask> <mask> <mask> <mask> <mask> <mask> <maske <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <masko <mask> <mask> <mask> <mask> <mask> <mask> <mask>.
<mask> image depicts <mask> bear lying <mask> its <mask> <mask> a <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> appears to be resting <mask> <mask> <mask> <mask> <mask> <mask> <mask> <maske <mask> <mask> <maslo <mask> <mask> <mask> <maske <mask> <mask> <mask> <mask> <mask> <maske.
<mask> <mask> <mask> <maske <mask>
The image depicts a bear lying <mask> its back <mask> a <mask> surface <mask> <mask> <maske <mask> <mask> <mask> <mask> <mask> <masko <mask> <mask> <mask> <mask> <mask> <masko <mask> <mask> <mask> <maske, the bear appears to be sleeping smaske with its head <maske on <mask> <mask> surface <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> <mask> or <mask>.
image depicts lying
on <masko surface.
<masko <mask
<mask> <mask> <mask>
<masko <mask> <mask> <mask> <mask> bear appears to be sleeping. the <mask> suggests <mask> the <mask> <maske <mask>, possibly near a <mask> or <mask>.
The image depicts a bear lying on its back on a concrete surface. the bear is lying down, with <mask> head <mask> <mask> its head resting <mask> <mask> <masko, the bear appears to be sleeping. the setting suggests that the scene outdoors, possibly near park <mask>.
The image depicts bear lying on its back appears sleeping. setting suggests that the outdoors,
surface. the bear is lying with its head
resting on the surface. possibly near park or garden.
T=0
Figure 12: Image-to-text generated results in each step.
25

---

Published as a conference paper at ICLR 2026
Inference Speed Comparison (Lower is Better)
Muddit 18
2415
Owen VL. 38
Show-o 1.38
BLIP2-apt-..1
12025
16230
2000
4000
12000
14000
16000
Inference Latency (ms)
Figure 13: Inference speed comparison. We use 32 inference steps for Muddit and fix the sequence length to 77 across all models.
26


Reviews:
[{"review_id": "9QSeMqX4Ic", "reviewer": "Reviewer_9HJx", "summary": "This paper presents Muddit, a unified discrete diffusion model that bridges text and image generation within a single architecture. By leveraging strong visual priors from a pretrained text-to-image backbone (Meissonic) and employing fully discrete diffusion for both modalities, Muddit achieves competitive performance on text-to-image generation, image captioning, and visual question answering tasks. The work demonstrates that discrete diffusion, when equipped with appropriate pretrained knowledge, offers a scalable and efficient alternative to autoregressive approaches for unified multimodal generation.", "questions": "Do you study how the approach scale with model size, training data, and computation resource? I am curious about whether the performance gap can be closed through scaling.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method design is novel. It achieves multimodal unification through fully discrete diffusion for both text and image, unlike hybrid approaches that combine autoregressive and diffusion components with separate architectures.\n\n2. The method enables parallel token generation, achieveing speedup over AR baselines. This is good for evaluation and real-time applications.\n\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The key designs of this work (Initializing from a visual prior rather than LLM, using CLIP as text encoder rather than LLM) fundamentally constrain the model's text comprehension capabilities. This is particularly evident in handling long-form, complex text. These limitations restrict the practical applicability of the method, especially in scenarios requiring sophisticated natural language understanding or generation beyond simple captions and short answers.\n\n2. Text generation is constrained to 77 tokens, limiting applicability for long-form generation tasks.\n\n3. The experiment results are not very strong: there is a gap with AR baselines in several tasks (e.g. VQAv2), and with continuous diffusion models in generation task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tnuhuy3LAh", "reviewer": "Reviewer_TaRs", "summary": "This paper presents Muddit, a unified generative model for text and images that challenges the dominance of autoregressive (AR) frameworks. The authors argue that AR models are slow and that existing unified diffusion models are weak because they are trained from scratch, lacking strong priors. Muddit's core contribution is a \"visual-first\" approach. It uses a single discrete diffusion (MaskGIT-style) transformer backbone that is initialized with the weights of a powerful, pretrained text-to-image model (Meissonic). A new lightweight text decoder is then added, and the entire 1B parameter model is jointly trained (via pretraining and instruction tuning) to handle text-to-image, image captioning, and VQA tasks using a single, parallel decoding process.", "questions": "1. The text decoder is a \"lightweight linear head\" on a visual backbone. Is this the primary bottleneck for the I2T/VQA tasks? Can the authors elaborate on why a more sophisticated text decoder (e.g., a small size Transformer) was not used?\n2. Does the author believe this \"visual-first\" architecture can scale to 7B+ parameters and eventually close the gap on language-heavy tasks, or is it primarily intended for vision-centric tasks where language is secondary?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The \"visual-first\" strategy is a novel and compelling way to build a unified model. It cleverly solves the \"weak prior\" problem of from-scratch diffusion models by inheriting the strong visual capabilities of Meissonic, while using a single, symmetric discrete diffusion objective for all tasks.\n2. The model's primary strength is its efficiency. By replacing sequential AR decoding with parallel discrete diffusion, the paper demonstrates a significant inference speedup over prominent AR models (Fig. 13 and Tab. 7), a practical advantage for real-world applications.\n3. By building on a SOTA T2I model, Muddit's image generation quality is excellent for its size. It achieves a 0.61 on GenEval, making it competitive with much larger models and even SOTA diffusion-only models like SD3 (0.62).", "weaknesses": "1. The model's key strength is also its greatest weakness. By building on a visual backbone, the model's language and reasoning capabilities are fundamentally limited. This is evident in Table 2, where Muddit's VQAv2 (68.2) scores are significantly lower than LLM-based AR models like LLaVA-Next (82.8) or discrete diffusion unified model like Show-o (69.4).\n2. The model is only 1B parameters. The model capacity and visual-first training methodology bring high efficiency but make it lack far behind the leading unified discrete diffusion models like MMaDA (Yang et al., 2025) and Lumina-DiMOO (Xin et al., 2025). As a unified discrete diffusion model, the author should compare to SOTA method like MMaDA in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ihj3TVrvRU", "reviewer": "Reviewer_aAn6", "summary": "This paper presents Muddit, which is a unified discrete diffusion model for text-to-image, image-to-text, and VQA tasks. Achieving competitive performance with existing AR or diffusion or AR/diffusion mixture models, Muddit, with a relatively small model size (1B), demonstrates the potential of a purely discrete-diffusion-based model as a scalable and effective backbone for unified generation.", "questions": "1. I didn’t find explicit evidence in the paper to support the scalability potential of Muddit. Could the authors further elaborate on the meaning of \"scalable\" in the claim: \"highlight Muddit's potential to serve as a scalable backbone for future multimodal systems\" at the end of the conclusion section (and the abstract section)? \n2. Some of the evaluation datasets are split into specific tasks (i.e. MME, which contains multiple categories, such as Existence, Count, Position, Color, OCR, Poster, Celebrity, etc). It would be interesting to also include detailed performance under different task categories.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Muddit is built upon a visual prior rather than a language prior, which offers a new path in bridging vision and language into a unified model.\n- Muddit achieved comparable or superior performance with a relatively small model size.", "weaknesses": "The performance of Muddit is comparable to or slightly better than Show-o (size 1.3B) in both Table 1: Text-to-image generation performance and Table 2: Image captioning and VQA benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7MxBlao5ER", "reviewer": "Reviewer_G1Hq", "summary": "This paper presents Muddit, a unified generative framework that applies discrete diffusion for both text and image modalities. The model leverages a pretrained visual prior to enable fast and parallel generation for variety of tasks. Experimental results show that Muddit achieves competitive or superior performance compared to larger autoregressive models.", "questions": "See the weaknesses two", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and clearly structured, with detailed experiments across multiple multimodal tasks.\n- It demonstrates that a purely discrete diffusion framework can achieve results comparable to autoregressive baselines while offering faster inference.\n- The approach shows practical value by unifying image and text generation within a single framework.", "weaknesses": "Although overall quality of the paper and the proposed methods are good, I have two major concerns.\n- It is not entirely clear why the unified use of discrete diffusion across modalities is fundamentally advantageous. The paper does not convincingly explain what conceptual or empirical benefits this fully discrete (with discrete diffusion) setup provides beyond architectural neatness. For example, from Table 1 and 2, models employing hybrid or alternative architectures perform better in some cases. A more detailed justification or analysis of why full discretization is beneficial would strengthen the contribution.\n- The paper does not sufficiently distinguish its method from prior works that already employ discrete diffusion for both text and image. The differences seem limited to implementation choices and pre-trained priors rather than introducing a fundamentally new formulation. Clearer explanation showing how Muddit’s design overcomes the limitations of previous discrete diffusion models would make the contribution more substantial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)