FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities
Authors: Jin Wang
Yao Lai
Aoxue Li
Shifeng Zhang
Jiacheng Sun
Ning Kang
Chengyue Wu
Zhenguo Li
Ping Luo
Venue: NeurIPS 2025 spotlight ()
Keywords: Multimodal Large Language Models (MLLMs); Discrete Flow Matching
Primary area: deep_learning
TL;DR: a unified multimodal model purely based on discrete flow matching, achieving comparable performance with AR-based MLLMs

Abstract: The rapid progress of large language models (LLMs) has catalyzed the emergence of multimodal large language models (MLLMs) that unify visual understanding and image generation within a single framework. However, most existing MLLMs rely on autoregressive (AR) architectures, which impose inherent limitations on future development, such as the raster-scan order in image generation and restricted reasoning abilities in causal context modeling. In this work, we challenge the dominance of AR-based approaches by introducing FUDOKI, a unified multimodal model purely based on discrete flow matching, as an alternative to conventional AR paradigms. By leveraging metric-induced probability paths with kinetic optimal velocities, our framework goes beyond the previous masking-based corruption process, enabling iterative refinement with self-correction capability and richer bidirectional context integration during generation. To mitigate the high cost of training from scratch, we initialize FUDOKI from pre-trained AR-based MLLMs and adaptively transition to the discrete flow matching paradigm. Experimental results show that FUDOKI achieves performance comparable to state-of-the-art AR-based MLLMs across both visual understanding and image generation tasks, highlighting its potential as a foundation for next-generation unified multimodal models. Furthermore, we show that applying test-time scaling techniques to FUDOKI yields significant performance gains, further underscoring its promise for future enhancement through reinforcement learning.

FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities
Jin Wang"* !
Yao Lai
Aoxue Li
Shifeng Zhang"
Jiacheng Sun?
Ning Kang²
Chengyue Wu
Zhenguo Lit.2
Ping Luo
The University of Hong Kong
2 Huawei Noah's Ark Lab
Abstract
The rapid progress of large language models (LLMs) has catalyzed the emergence of multimodal large language models (MLLMs) that unify visual understanding and image generation within a single framework. However, most existing MLLMs rely autoregressive (AR) architectures, which impose inherent limitations on future development, such as the raster-scan order in image generation and restricted reasoning abilities in causal context modeling. In this work, challenge the dominance of AR-based approaches by introducing FUDOKI, a unified multimodal model purely based on discrete flow matching, paradigms. By leveraging metric-induced probability paths with kinetic optimal velocities, our framework goes beyond the previous masking-based cornuption process, enabling iterative refinement with self-correction capability and richer bidirectional context integration during generation. To mitigate the high cost of training from scratch, we initialize FUDOKI from pre-trained AR-based MLLMs and adaptively transition the discrete flow matching paradigm. Experimental results show that FUDOKI achieves performance comparable to state-of-the art AR- based MILMS across both visual understanding and image generation tasks, highlighting its potential as foundation for next-generation unified multimodal models. Furthermore, we show that applying test-time scaling techniques to FUDOKI yields significant performance gains, further underscoring its promise for future enhancement through reinforcement learning.
alternative conventional AR
Introduction
Driven by the rapid progress of large language models (LLMs) [15] new wave of large-scale multimodal models has emerged, delivering remarkable advances in the two fundamental pillars of artificial general intelligence (AGI): understanding 66-10 and generation [[T-55]. Building on this momentum, growing body of work [16-23] seeks to unify perception and synthesis within single framework, introducing versatile multimodal large language models (MLLMs) that seamlessly integrate visual understanding with image generation.
In prior research, processing multimodal tokens sequentially from left to right for both understanding and generation tasks 26 27
MILMS adopt the autoregressive (AR) architecture of standard LLMss,
While these MILMS deliver strong performance across many multimodal tasks, their
Equal Contribution
Correspondence to: Zhenguo <li.zhenguo@ huawei.com> Ping Luo <pluo@ hkk.. Project page: fudoki-hku.ginn hik.github.io.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Generation
Understanding
Spatial Relation
Question: What the relation between the couch and the coffee table in the
The couch is on top of the coffee table
B. The coffee table in front of the couch
C. The coffee table behind the couch The couch the coffee table
related
Answer: coffee sable front
Prediction
A beautiful modern wooden house, close to the lake, in the mountains at sunrise, anime style
Hypar-realistic photo an abandoned
industrial site during storm
Question: What is the anticipated image?
A. Ho will be released from the police station escape
police station
police
D. be visiting the police station voluntarily
Anower:
The image shows person being escorted by police hands behind their back. This
officers, with the
being to the police station, Therefore, correct answer C, will be arrested and taken to the police station.
person
Scene Understanding
Automobile design drawings, sketch
Portrait of beautiful gif, neon
Question: What the main event that is happening in the image?
A. sward show concert performance graduation ceremony sporting
happening
image
graduation ceremooy. This is evident from the individuals wearing graduation caps and gowns, and the presence of large audience In stadium-like setting, which is typical for graduation performance
Instance Counting
Question: How many people sitting In the background of the image?
None One Three more D. Two
Answer: C. Three more
Physics
Question: What transformation
this image?
A. The water will freeze
B. The water will remain liquid
evaporate
Answer: The Image shows container of water being heated, with blue spheres representing malecules moving from the liquid the
This known boiling Thorefore, the correct answer The water evaporate.
products,
gas
Figure 1: Qualitative Results of Visual Generation and Understanding Capabilities of FUDOKL. FUDOKI designed hased on the framework of discrete flow matching for both visual and textual modalities, capable of performing understanding and generation simultancously under one unified paradigm.
inherent AR design's limitations have become increasingly apparent as shown in recent studies, such as weaker performance in complex reasoning [0], challenges future planning [31), and difficulties with self-correction [32] These shortcomings particularly critical for emerging domains such as embodied AI and autonomous agents, where complex reasoning and deep contextual understanding are essential. This prompts fundamental question for the future of AGI development: what architectural paradigm could define the next generation of MLLMs?
To this end, discrete-spaae space generative flow and diffusion models have gained attention as a promising alternative for generative modeling. These models have seen success the domain text gener- ation 33 38, protein design (3), image synthesis 37 38 and code generation 37 40 Unlike sequential autoregressive models, these models usually begin with a fully corrupted sequence and iteratively denoise the entire sequence in parallel, which allows richer integration of information

---

from both directions to enhance prolonged reasoning. Moreover, these models enable flexible and controllable generation through their inherent iterative refinement process, while offering the potential for accelerated sampling via novel training designs [41443]. Recent studies like L.LDA [44] and
Dream [45] have also scaled discrete diffusion models to 7B parameters, further highlighting their growing potential to overcome the fundamental limitations of autoregressive approaches.
To advance the application of discrete generative flow modeling and challenge the dominance of the AR-based paradigm in MLLMs, we present FUDOKI, a unified multimodal model purely based on discrete flow matching. Different from previous diffusion- -aase unified multimodal models [46 focusing solely on the case of masking as a corruption process, we adopt the novel framework of discrete flow matching [37]38], (38. which substantially expanded the design space of discrete-space generative models by enabling metric- metric-induced probability paths with kinetic optimal velocities. This
design enables better performance than masked construction [8] and allows models to continuously self-correct their responses during the iterative refinement process. Moreover, to mitigate the high training cost of training large discrete flow matching models for multimodal tasks, we leverage the pre-trained AR-based MLLM (20] as the initialization and adaptively transfer it to the discrete flow matching paradigm [49].
The contributions of this paper can be summarized as follows: 1) We introduce FUDOK the first general-purpio unified multimodal model built entirely on discrete flow matching. Unlike traditional approaches that rely on masking-bass based comrpption, FUDOKI leverages metric-induced probability path with kinetically optimal velocities, expanding the design space of discrete multimodal modeling and offering advantages during inference; 2) Through extensive experiments, we show that FUDOKI achieves competitive performance on both visual understanding and text-to-image generation tasks, senggressive-baeeee MLLMs; 3) We apply test-time inference scaling techniques to FUDOKI inspired by [50], which yield substantial improvements across visual generation and understanding benchmarks. This suggests strong potential for future enhancement of FUDOKI via reinforcement learning MG. We believe that FUDOKI provides a compelling foundation for the development of next-generation unified multimodal models.
rivaling
2 Preliminary: Discrete Flow Matching
In this section, we present key concepts and notations in discrete flow matching [3] to facilitate understanding in the following sections. Generally speaking, the objective of discrete flow matching is to approximate the target underlying data distribution g(x) from the source known distribution
p(z), where x = (x x² D) belongs to the discrete space S = TD, where D is the number of discrete variables and Todd=[aee
= [K] = 21. K} represents a finite set of possible discrete values.
Probability Paths. Given a source distribution p(x) and a target distribution q(x) defined over a finite state space S, discrete flow matching defines family of time-indexed probability distributions {p((z))eefo.) to to describe a smooth transformation from p to 9. referred to as probability paths. Each pt (x) is constructed as: pe(a)) = Ex, Es Pt (x In (x(1). where the conditional distribution is factorized across dimensions, namely pe(2r m1) = - Pt (2r" =x)) Here, each pe(x*' x+) defines univariate interpolation between base distribution p(x*) and point mass 6zy (x*), i.e., 5zj (x*) = 1 if x' = o else O. A common design for such interpolations is the mixture path, defined via time-dependent scheduler Ka (x1) E 1]:
Pr | |xf) = (1 - k=(x)))((x') + ke(z))6,; (x'),
where (-)-) and Ni (-) = 1. This class of paths recovers the masked data construction when p(x') = 5m (x*) with m denoting the mask token, which are widely used in previous studies 36].
Probability Velocities. To simulate the generative process that evolves along the prescribed path we consider a continuous-time Markov chain (CTMC) (x, hej.01)) over the discrete space S, such that: 24- ~ pt- Specifically, we describe this CTMC via a probability velocity x.))
tad (FUDOKI) a Japanese term referring to ancient records that comprehensively document and integrate the culture, geography, and traditions of different regions. We name our model FUDOKI to highlight its unified ability to both understand and generate multimodal information, such as interpreting and generating diverse images, mirroring how the original FUDOKI integrates and presents multifaceted knowledge.
3

---

(also known as the rate matrix), describing the rate of probability change of I in its i-th token. Reminiscent of the velocity field in the continuous Flow Matching 42 discrete flow matching features the following definition:
Definition 1. probability velocity ue is said to generate the probability path pe if, for all E [0. 1) and for any sample Tt ~ Pt. the updated sample Fi+h ~ 62((+) + hug( 24) for each coordinate i satisfies the condition that F4th ~ PL+h + o( 1 as h - 0.
Besides, the probability velocity u, should satisty the following rate condition:
Σ [
"a 2) = and (x), 0 vi ( x' #2
such that the updated +h can be sampled from a valid probability distribution. Further, previous studies [37 [3] also demonstrate the Continuity Equation (also known as the Kolmogorov forward
equation) in discrete flow matching, which describes the state probability rate pr(x), x E S by:
p=(x)) + (Peu)) 0.
where div. (pru)) = =EES EP 62)))) [Po(a)u pp)()) =]]. measuring the total out- Szi going flux x + minus the total incoming flux z + for state I E S. Here 82 (x - (2²), which indicates that we only consider æ and when they only differ in the i-th coordinate for calculating the flux (37 34 Intuitively, Eq. expresses that the rate of probability at x is equal to the final remaining probability flux " æ. Previous studies (37] 39] have shown that if the Continuity Equation is satisfied, then Ue is said to generate the probability path pt as in Definition 1.
3 FUDOKI: A Multimodal Model Purely Based on Discrete Flow Matching
This section introduces FUDOKI, new multimodal architecture that unifies vision and language through the novel lens of discrete flow matching. By adopting this framework, FUDOKI enables an integrated approach to both perception and generation across visual and textual modalities.
3.1
Metric-induced Probability Paths with Kinetic Optimal Velocities
Based on the recent theoretical advancement of discrete flow matching 38
adopt a more
general probability path for FUDOKI, instead of the commonly used mask-based mixture paths
33 36
35 46
451. Specifically, we consider the probability paths induced by discrete metrics.
Given a distance function d T T + R20 satisfying d(I', x11) = 0 if and only if x' = z we define a path of conditional distributions via:
(x =1)= softmax(- - 3. 0dddd +1));
(4)
where B, 10 + R20 is monotonic schedule with boundary values 30 = 0, 31 = 00. At 0,
this yields a uniform distribution, and as t + 1, the distribution converges to delta function at 21- Compared to the previous mask-based probability path (i.e., Eq. [T. this metric-induced probability path defines a more semantically meaningful transformation, allowing the probabilities of tokens similar to ri to also increase - 1, when setting d(...) to measure token embedding distances.
After defining the prescribed metric-induced probability path, we then obtain the probability velocities via minimizing the kinetic energy (38. In other words, is expected to minimize the magnitude of flux p,u, for probability velocities to obtain smooth transformation along the probability path. Meanwhile, the obtained velocities should also satisfy several conditions, including the Continuity Equation (i.e., Eq. 3.. the non-negativity of the flux between different states (i.e., Eq. 2 and the boundary conditions for p and q. We leave the detailed mathematical formulations in the appendix. In this way, the kinetic optimal velocity for Eq. Ccan be formulated as follows
38
(10 2 (11) (x x1) 3e [d(2",
(5)
where = = maxf- (- 0) is the ReLU operator and Be is the derivative of Be w.r.t t. Intuitively, for the i-th coordinate 2¹ E T, this velocity ensures that probability mass flows from state 2 to state x' only when x² lies closer to 2 than 2 does, i.e., d(x', z1) < d(2", x1). As a result, the flow monotonically progresses toward zf. After introducing the mathematical foundation of discrete flow matching, now dive into FUDOKI's model structure details.
$ *o(h) refers to function that vanishes faster rate than h as 0, i.e., limb- o(k) h
0.

---

Tegt
Textling
IngiTest
TestBlngg
red panda.
red parda.
Text De-Toknnzzrr
Image
Causal Transformer
Partially Causal Transformer
Dacrada
Inigge Encoder
Text Tokonizer
Text Tokerizer
Image Encoder
Toxt Tokorizer
Text Tokenizzrr
What i?
rad pands.
What is H??
Ared pandl..
(a) Autoregressive (AR): :AR): Janus-Pro, Illume, Illume+, VILA-U (2025); Chameleon, EMUS, LaVIT (2024)
AR (Text) Diffusion (Image) JanusFlow (2025); Transfusion, Show-o (2024)
Ingitient
Ared panda.
Test2ing
Ingelest
And parde.
Tuxt2ingg
Tokanizur
Image Decoder
Image Decoder
Bi-directional Transformer
Bi-directional Transformer
Image Enooder
Text Tokenizor
Text Tokenizer
Toxt Tokerizer
Encodar
Teat Tokerizer
Text Tokenizer
<mask>> <mash>>
What's <masto
Kinaako kmaak>>>
pands.
Caminnauu
cmask>> <mask>
Ared pande.
Daasta
(c) Continuous Discrete Diffusion: D-DIT (2025)
(d) Discrete Diffusion: UniDisc (2024)
Test
Teat2lng
panda.
Decoder
Bi-directional Transformer
Bacoder (Semandic)
Encoder (Pise))
What is It?
prá tion # mont d9 ist
A red pancia.
Dourate
(e) Discrate Flow Matching: FUDOKI
Figure 2: Comparison Model Architectures Unified Multimodal Models. (a) AR-based models [2 26 21 S4 18 5S perform multimodal tasks via sequential token generation under strictly causal context
modeling. (b) Hybrid AR+Diffusion models, such as Transfusion [19] and Show-o (56). integrate AR for
text and diffusion models for images, enabling improved visual generation quality. (c-d) Diffusion-base -assed models: D-DiT [46] applies mask-ban based discrete diffusion
and continuous diffusion to images, while UniDisc (88] employs mask-based discrete diffusson both modalities. (e) FUDOKI adopts unified
discrete flow matching framework for both modalities, leveraging metric-induced probability path to enhance performance in understanding and generation tasks. The inference advantages of FUDOKI over mask-based discrete diffusion modeling used in (c-d) are shown in Fig.
3.2 Architecture Overview
As shown in Fig. Fig. FUDOKI is based on the Janus-1.58 20) architecture, with minor adaptations support unified vision-languag discrete flow modeling. Specifically, facilitate effective learning and accelerate convergence, 1) we adopt a full attention mask instead of the standard causal mask to allow all tokens to attend each other, which helps the model better capture global context; apply a shifting operation 49 to the output logits by one position, so that our model can inherit the next-token prediction capabilities of AR-based MI.LMs much as possible; 3) unlike continuous diffusion models [57 12]. we do not incorporate additional time embedding layers in the model explicitly indicate the noise level in the corrupted input. Following the intuition of mask-based discrete diffusion models [49 (58), we observe that our discrete generative model can also implicitly infer the timesteps from the corrupted input along our defined metric-induced probability path (i.e., 4.. resulting in faster adaptation in experiments. The rest of the architecture remains identical Janus-1.5B. For the text modality, we use the tokenizer with vocabulary size of 102, 400. For images, we decouple the processing paths for understanding and generation. The semantic encoder SigLIP (59) extracts high-dimensional features for image understanding, which are reshaped and mapped into the LLM input space via adaptor. For image generation, we follow LlamaGen (60), employing pixel encoder and decoder to convert images into discrete tokens, with the image token vocabulary size set to 16, 384. Each image token embedding is further transformed into an input
5

---

Now: State Roaal
Thi d
Imasge
centa
Next State Soaal
Trisk
Imagrnal
pendeaal
Lanp Tnion Sulccion
O
Jump Tutan Selcsion
Piedicted Email stane "
This
Image
red
penda
Piedicidd Ernil stane ue "
Thil
imaage
eed
panda
Derkkes Su
Jaep
Jeen
Bi-directional Transformer
Jeegy
Juna
Bi-directional Transformer
No Cansklendd
slways
lineading
Floed after
Image Enserder
Text Beccdar
Image Encoder
Taxtt Encoder
State
That
pansiag
Cumeent Show
(a) Mask-based Discrete Diffusion (MDD): D-DIT, UniDisc
Discrete Flow Matching (DFM): Fudoki
Figure 3: Inference Comparisons between (a) Mask-BBaed Discrete Diffusion Models and (b) Discrete Flow Matching-Basne -aaein FUDOKI. In mask-based discrete diffusion models, once a token is unmasked, typically cannot be modified again, which hinders self-correction. In contrast, our proposed FUDOKI allows its responses to be continuously updated during inference, enabling potential corrections.
feature via a generation adaptor before being fed into the LLM. At the output stage, we use two output heads, text head and an image head, which convert the transformer outputs into discrete categorical distributions. The appropriate head is selected depending on the target modality during inference. Comparisons with previous AR-based and diffusion-based MILMs are shown in Fig.
3.3
Training
We follow the discrete flow matching framework (34 for model training. Our model is initialized from the pretrained weights of Janus-1.5B (20) and further adapted to collected dataset, which contains both text-to-in rext-to-image (generation) and image-to-tex (understanding) data. Specifically, we divide the training of FUDOKI into stages: 1) The main goal of the first stage is quickly relearn the AR-based L.LM such that it can effortlessly support the discrete flow matching paradigm. To this end, we only fine-tune the parameters of the transformer while keeping other parts of the model frozen, including the semantic encoders and embedding adaptors. This can help accelerate convergence and stabilize our training; 2) After the first stage, we further fine-tune the whole model to enhance its overall performance on understanding and generation based on discrete flow matching.
Specifically, in each training stage, the ground-truth target æ1 is drawn from the data distribution q()). where the condition is either a text prompt (for T2I) or an image-questine pair (for 12T). The target x2 is the image token sequence in the T2I setting and the textual token sequence in the 12T setting. At each training step, a time t E [0, 1] is uniformly sampled, and a noised sequence æ, is sampled according the defined probability path pe(- x1) in Eq. 4 We the distance function d(.,)) to measure the 122-ditances between normalized token embeddings, which helps increase the probability of sampling tokens whose embeddings of in the embedding space, thereby making the corruption process more semantically meaningful and facilitating learning. The model then receives I; as input and predicts Iı, outputting per-token logits for each position. The training loss is defined as the expected cross-entropy between the ground-truth sequence x1 and the model's predicted distribution:
close to the corresponding ground-truth token
Σ P (zf mr))
CCE(f) = E,~U10.11, a.
(6)
where p- xx)) denotes the model's predicted categorical distribution for the i-th position, parame- terized by 8, given input 2t.
3.4 Inference
During inference, we apply Euler solver for more robust sampling as suggested in B8]. This solver simulates the continuous -time Markov chain (CTMC) process (Ix (æz)ugrgı- Given that x, ~ Pt. the solver updates the i-th coordinate from time to + h using the following procedure:
Sample æ1 ~ (-(re) from our model;
Compute the total conditional transition rate X Exitet ut agflar)
(see Eq. 5
Draw uniform random variable Z change U0,

---

Sample E as follows: if Zhangs I
1 e
-xx sample 2/4 from "f(aai) x
dag (-));
otherwise set #ith = x.. Here 5zf (-) is a delta function.
We provide a detailed understanding of this inference process as follows. In the second step, X can be interpreted as the intensity with which the probability mass at zi flows to other states x' * xt- The probability that of will change at the current timestep is determined by comparing the threshold 1 1 hx' with uniform random variable Zchange' the larger X is, the more likely a jump will occur. a change happens, d/-n is sampled from all other possible states according to the distribution proportional to w =]]=1), ), as defined in Eq. 5 This means the update tends to move 2444 towards states that are closer to the model's prediction zi. In this way, our sampling process enables the model to: (1) continuously refine its predictions along the probability path, and (2) flexibly adjust tokens towards semantically similar alternatives each timestep. As shown in Fig. [ this is in contrast
to previous mask-based discrete diffusion models [36 35 45), where once a token is unmasked, generally cannot be modified again, even if it contains error.
4 Experiments
4.1 Implementation Details
In both training stages, we use approximately 13M supervised finetuning data to learn our FUDOKI, including 9M in-house generation data for text-t-iimaee generation and 4M public understanding data, which covers various aspects including OCR [6 62]. doc [63], chart [64], screen (65] math [66] 671, language [68], etc. This is less than Chameleon's 's 1.4B data (54) and LWM's 1B data [69]. We leave the detailed dataset collections in the appendix. For text generation, the sequence length for the response is set to 500, while for image generation, it is set 576 match the input size of the image encoder. The text embeddings for calculating the metric distance function d(=..) .) are taken from the original embedding layer of /aous-Pro-7B 26
and the image embeddings are obtained from
with c = 3 and 0.9, suggested in 38 Besides, following previous studies 45 44), for the text modality, we pad each sequence with <eos> (end-of-sequence) and <pad> tokens to the maximum length during training, and compute the loss over model's answer tokens, including these special tokens. After the sampling process, we only keep the model responses ahead of the first <eos> token. The sampling iterations are set as 32 by default, and the resolution of generated images by FUDOKI is 384 x 384. The entire training process spanned approximately 43,000 GPU hours.
the codebook of LlamaGen [60 We set Bg
4.2 Comparison with State-of-the-arts
Visual Generation Performance. We evaluate the generation capabilities of FUDOKI on the widely used GenEval benchmark (75). Table presents the summarized comparisons, where FUDOKI achieved competitive overall performance (0.77), matching the top score of prior models in the category of both the generation-only and the understanding-an and-ogg--n--geannnne and-generation categories. These results underscore our model's advantages in accurate multi-object understanding and attribute binding, making it promising for complex visual generation tasks that go beyond simple object depiction. This be attributed to the discrete flow matching framework of FUDOKI, which allows visual information to integrate in both directions for better layout design of generated images.
Besides, evaluate the visual generation performance of FUDOKI DPG-Bench (76] (Dense Prompt Graph Benchmark), a comprehensive dataset comprising 1,065 lengthy and densely composed prompts specifically designed to assess the fine-grained semantic alignment capabilities of text-to- image models. As shown in Table 2] FUDOKI demonstrates competitive performance compared to both seneration-speccilizee and unified multimodal models. These results highlight FUDOKI's strong ability to handle complex, information-rice prompts, establishing it as a robust and versatile solution for multi-aspect visual generation tasks.
Multimodal Understanding. We evaluate the understanding capabilities of FUDOKI on several benchmarks, including POPE (9]], MME-P (92), SEED 193), MMB [94), GQA 195, MMMU 196),
7

---

Table 1: Visual Generation Performance on the GenEval Benchmark. "Und." and "Gen." denotes "Under- standing" and "Generation". denotes models that integrate external pretrained diffusion model.
Type
Paradigm
Method
Single Obj.
Two Obj.
Counting
Colors
Position
Color Attri
Overall
LlamaCien
60
0.71
0.34
0.21
0.58
0.07
0.04
0.32
Emna3 Eanu3-Gen
0.98
0,71
0.34
0.81
0.17
0.21
0.54
LDD
0.92
0.29
0.23
0.70
0.02
0.05
0.37
Gen. Only
PixArt-a
0.98
0.50
0.44
0.80
0.08
0.48
Diflusion
SDv2.1 12
0.98
0.51
0.44
0.85
0.07
0.17
0.50
DALL-E 70
0.94
0.66
0.49
0.77
0.10
0.19
0.52
SDXL
DALL-E
??
0.96
0.87
0.47
0.83
0.43
0.45
0.67
SD3 Medium
14
0.99
0.94
0.72
0.89
0.33
0.60
0.74
SEED-X X
23
0.97
0.58
0.26
0.80
0.19
0.14
0.49
LWM
0.92
0.79
0.47
ILI.UME
21
0.99
0.86
0.45
0,71
0.39
0.28
0.61
AR
TokenFlow-XI
F4
0.95
0.60
0.41
0.81
0.16
0.24
0.55
Chameleon
Und. and Gen.
Janus
20
0.97
0.68
0.30
0.84
0.46
0.42
0.61
Janus Pro 1B
25
0.98
0.82
0.51
0.89
0.65
0.56
0.73
AR+Diffusion
Show-o
50
0.95
0.52
0.49
0.82
0.11
0.28
0.53
Diffusion
UniDisc
0.92
0.47
0.15
0.67
0.13
0.42
D-DIT H6
0.97
0.80
0.54
0.76
0.32
0.50
0.65
Discrete Flow
FUDOKI (Ours)
0.96
0.85
0,56
0.88
0.68
0.67
0.77
Inferenee Scaling
0.98
0.95
0.73
0.94
0.88
0.78
0.88
Table 2: Visual Generation Performance on DPG-Bench.
Method
Global
Entity
Attribute
Relation
Other
Overall
SDv1.5 P
74.63
74,23
75.39
73.49
67.81
63.18
PixArt-o
82.57
76.96
81.82
74.63
SDXL 71
83.27
82.43
80.91
86.76
80.41
74.65
Playground v2.5 72
83.06
82,59
81.20
84
83.50
75.47
Hunyuan-DiT h0
84.59
80.59
88.01
74,36
86.41
78.87
PixAnt S
30
86.89
82.89
88.94
86.59
87.68
80.54
Emu3 Gen
18
85.21
86.68
86.84
90.22
83.15
80.60
DALL-E3 3
72
90.97
89.61
88.39
90.58
89.83
83,50
SD3 Med-Mmeiumm
[4
87.90
91.01
88.83
80.70
88.68
84,08
Janus
20
82.33
87.38
87.70
85.46
86.41
79.68
88.30
FUDOKI (Ours)
78.00
and MM-Vet 197] Table 3 presents the summarized results 5 Notably, our FUDOKI model (1.5B parameters) achieved highly competitive results, which are on par with or surpass several AR-based MLLMs of similar or even larger scale. This demonstrates that FUDOKI delivered robust multimodal understanding capabilities, which can attributed to the bidirectional reasoning property discrete flow matching. Moreover, we provide generation process comparisons for understanding in Fig. 4 which further highlight the advantages sampling through discrete flow matching for reasoning, e.g., self-correcting the reasoning process for coherency. Our findings highlight the effectiveness and efficiency of FUDOKI, making a strong alternative to the established AR-based MLLMs.
Inference Scaling. We applied test-time inference scaling techniques 550 to FUDOKI, leveraging judge model score multiple candidate outputs and select the highest-scoring responses. The last rows of Table Tand and Table illustrate the impact of inference scaling on visual generation and understanding. For generation, we used the VILA-Judge model [98] to select the top images from 32 candidates per prompt in the GenEval benchmark, resulting in significant performance gains. For understanding, we employed an LLM as the judge choose the best response from 8 candidates in the challenging Ve benchmark, where improvements were observed. These results highlight FUDOKT's potential for further enhancement through reinforcement learning approaches 99
UniDisc 48 not included the table due to their inability conduct visual question answering tasks.

---

Table 3: Multimodal Understanding Performance on Various Benchmarks. "Und," and "Gen." denotes " "Understanding" and "Generation". denotes models that integrate external pretrained diffusion model.
Type
Paradigm
Model
# I.L.M Params POPE:
MME-P1 MMB SEED! GQAT MMMU? MM-Nett
LLaVA-v1.--Phi--
36
1.3B
84.1
1128.0
56.5
30.7
MobileVLM ST
1.4B
84.5
1196.2
53.2
56.1
MobileVLM V2
82
1.4B
84.3
1302.8
57.7
59.3
MobileVLM
81
2.7B
84.9
1288.9 9
39.6
59.0
MobileVLM-VV
2.7B
84.7
1440..5
63.2
61.1
PLa
28,9
Und Only
LLaVA M
76.3
809.6
38.7
33,5
25.5
LL.
78
85.9
1510.7
64.3
58.6
62.0
35.4
31.1
InstructBLIP
7B
36,0
53.4
49.2
26.2
Qwen-VI
851
7B
1487.5
60.6
58.2
$7.5
IDFFICS-9B
X4
48.2
InstructRI IP
13B
78.9
1212.8
49.5
25.6
LaVIT'
37
7B
46.8
MetaMorph
XH
8B
75.2
71.8
Gemini-Nano-
1.8B
26.3
65.1
72.9
TokenFlow-XI.
14
13B
1545.9
68.9
68.7
38.7
40.7
AR
LWM 60
7B
75.2
44.8
9.6
Und. and Gen.
VILAA-U 20
7B
85.8
1401.8
59.0
60.8
33.5
Chameleon 54
7B
22.4
8.3
Janus 20
1.5B
1338.0
69.4
63.7
59.1
30.5
34.3
36.3
AR-Diffusion Show o o-512
Show-o-256
56
1.3B
73.8
948.4
48.7
25.1
56
1.3B
80.0
1097.2
58.0
26.7
Diffusion D-Dit
46
2.0B
84.0
1124.7
59.2
Discrete Flow FUDOKI (Ours)
1.5B
86.1
1485.4
73.9
68.2
57.6
34.3
38.0
+Inference Scaling
(a) Janus (AR) Generation Process
aquation
aquution
toble 3D i ai find the value y. we subelfiute
The last
toble 3D e a. find the value y. we subciifute a equilion
The last
10 d
10 a. 12. find the value
I we sobofilute a 18 into the equalion:
+0
equifion:
a 10
12 12 24
12 S4. the vake of
24 24 22
29
D-DIT (MDD) Generation Process
anatio the the anvolommado nurberr inersalllesr-mm
In the the bestlessoonalo nunbers
In the the budiness to numbers crower, famous-
to-menn t-mmalo--aaao
crawer. wokkemmeenbxmmebbbooo oszaloorssio
10
musto mosoomo lo-ene wow.ttoooooommooooooooooo
probiom
Question: What equatio??
Fudoki (DFM) Generation Process (with "thinking" process)
Ground Truth:
30. Tufled The Vilue ofToN
39 Torind the Value of
10. Tafind the Vakee y. subtracting 24 from Reymand
39 find The wilue
subtracting EventAngg34 from Raymandik (1 essoriontorre
subtroccing from Raymord
subtrast 34 from both aides
equation:
the equatior: y
equatiore y
y Thippher y
dimpiiies
diepiifee
Figure 4: Generation process of different methods. (a) AR-based Janus can only generate tokens sequentially: if an error is made in the initial step, subsequent outputs will consistently propagate this mistake. (b) D-DiT (mask-based discrete diffusion, MDD) cannot revise tokens once unmasked, making errors irreversible and leading to poor generalization. (c) FUDOKI (discrete flow matching, DFM) allows generated tokens be revised subsequent steps, enabling step-by-step reasoning and error correction more accurate answers.
4.3 Ablation Studies
Training Strategies. 1) AR Initialization vs Training from Scratch: As shown in Fig. 5 (left), we compare models initialized with autoregressive (AR) weights 20 against models trained from scratch. The results indicate that AR initialization provided substantial advantage for accelerating model training, leading to consistently lower training loss throughout the optimization process. 2) Effects of Time-embedding Layers: We also evaluate the impact of incorporating time embedding layers into the model architecture. The results Fig. (idddee) show that the model without time embedding layers consistently achieves slightly lower training loss than the version with time embeddings. This

---

AR Initialization vs. Training from Scratch
With vs. Without Time Embadding Layers
Spead-Quality Trade-off
10.
layers
Loss
Gpeect (imghmin) 30
0.7
GanEvvl Score
Way
Spoed tingām nt
Traring
Janus Pro Spood
Janus Pro- Score
Centrall Score
0g
MMMMM
0.4
4000
6000
10
Training Stop
Training Stop
Discrate Flow Matching Timestops
Figure 5: Comparison of training loss and speed-quality trade-off. (Left, Middle) AR initialization and removing time embedding layers both reduce training loss, (Right) With fewer timestops, FUDOKI achieves much higher speed but slightly lower quality than AR; at the optimal timestep, both metrics surpass the AR.
Table Quantitative comparisons between the AR-based models and our proposed FUDOKI in terms of the self-correcting capabilities.
Method
Baseline
+.Jamus-Pree IB to correct
.lanus-Pro--B to correct
FUDOKT to correct
MMVet
37,98
36.33 (-1.65)
38.30 (+0.32)
38.5 (+0.55)
suggests that our discrete generative model can implicitly infer timesteps from corrupted input, and removing time embeddings reduces model complexity.
Quality-Speed Trade-off. Fig. (right) illustrates the trade-off between speed (in images per minute) and quality (GenEval score) in terms of setting different inference timesteps for visual generation. It compares the inference performance of FUDOKI with the autoregressive (AR) baseline, Janus-Pro- 1B (with KV cache enabled). The red solid line (to the left vertical axis) represents the speed of FUDOKI, which decreases as the number of timesteps increases, while the blue solid line (to the right vertical timesteps
axis) represents the generation quality of FUDOKI, which improves and stabilizes increase. We also draw the dashed horizontal lines indicating the baseline values for Janus-Pro-In -B, with the red dashed line for speed and the blue dashed line for quality. Please pay attention to the intersection point of the green arrows. This intersection marks the point where FUDOKI achieves a significant speed advantage over the AR baseline (as the red solid line exceeds the red dashed line) and comparable output quality (where the blue solid line meets the blue dashed line). This can be attributed to FUDOKI's fewer inference steps and richer bidirectional context modeling.
Results on the Self-Correction Capability. We quantitatively evaluated the self-correcting capa- bilities of FUDOKI and performed comparisons with the AR-based models. In experiments, both FUDOKI and AR-based models were tasked with correcting baseline responses where necessary. The baseline responses were obtained from mauus-Pr--IB
the MMVet benchmark, using the Open- Compass VLMEvalKit codebase [00]. To assess their correction abilities: 1) For AR-based models, we appended the following prompt to the original prompt: "Your original response is: <placeholder>, Please correct it if needed. Otherwise, you may keep it the same." The models were then evaluated on their ability to revise or retain the response appropriate; 2) For FUDOKI, responses with the bascline responses (rather than maformly-samplll noise tokens) and performed iterative refinements over 32 steps, as described in the paper. As shown in Table FUDOKI achieved the highest performance improvement, while Janus-Pro- 1B's performance declined and Janus-Pro-7 showed less increase, despite its larger model size than ours. We attribute such results to the increased context length introduced by the baseline responses, which may distract the AR-based model's 's focus. This further highlights the limitations of the AR paradigm for effective self-correction.
initialized the
5 Conclusion
In this work, we introduced FUDOKI, a multimodal model that uses discrete flow matching to unify visual understanding and generation. Unlike conventional autoregressive and masking- based approaches, FUDOKI leverages discrete flow matching for iterative self ceff-correction, bidirectional reasoning, and flexible generation. Experiments show that FUDOKI performs competitively with leading AR-based MILMMs on both visual understanding and text-to-image generation tasks. These results highlight discrete generative flow models exteeliiiee by FUDOKI- as a promising direction for ad vancing multimodal language models and meeting future AGI challenges.
10

---

Acknowledgments
This paper is partially supported by the National Key R&D Program of China No.2022200161000 and the General Research Fund of Hong Kong No. 1200825 and 17209324.
References
[1] DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
[2] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2.5 technical report. arXiv preprint arXiv:2412. 15, 2024.
[3] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, et al. The llama 3 herd of models. ArXiv, abs/2407.21783, 2024.
[4] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, et al. Internim2 technical report, 2024.
[5] OpenAL. Chatgpt. https:://chat.openai.ccom
2023.
[6] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[7] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[8] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
[9] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang. Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint 2rXiv:2403.05522 2024.
[10] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint 2rXiv:2404.1682/ 2024.
[11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-873 8794, 2021.
[12] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/VFF conference computer vision and pattern recognition, pages 10684 10695, 2022.
[13] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic sotttoo-image synthesis. arXiv preprint arXiv:2310 00420, 2023.
[14] Patrick Esser, Sumith Kulal, A. Blattmann, Rahim Entezari, Jonas Muller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. ArXiv, abs/2403.03306, 2024.
[15] Black Forest Labs. Flux. http::////giitubbcoommbbbbee forraalkk--oeee---s/////ll 2024.
[16] Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprint arXiv:2007.08041, 2023.

---

[17] Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang. and Ying Shan. Making Ilama see and draw with seed tokenizer. arXiv preprint arXiv:2310001218, 2023.
[18] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang. Yueze Wang. Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction all you need, 2024.
[19] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 1039, 2024.
[20] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation. ArXiv, abs/2410. 133448, 2024.
[21] Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, and Hang Xu. Illume: Illuminating your ilms see, draw, and self-enhance, 2024.
[22] Rongchang Xie, Chen Du, Ping Song, and Chang Liu. Muse-vl: Modeling unified vim through semantic discrete encoding. arXiv preprint arXiv:2411.1770 2024.
[23] Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu. Lmfusion: Adapting pretrained language models for multimodal generation. arXiv preprint arXiv:24122 18188, 2024.
[24] Jialv Zou, Bencheng Liao, Qian Zhang, Wenyu Liu, and Xinggang Wang. Omnimamba: Efficient and unified multimodal understanding and generation via state space models. arXiv preprint arx::22503 08686, 2025.
[25] Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song. et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. 146833 14683, 2025.
[26] Xiaokang Chen, Zhiyu Wu, Xingehao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. ArXiv, abs/2501. 178111 2025.
[27] Runhui Huang, Chunwei Wang, Junwei Yang, Guansong Lu, Yunlong Yuan, Jianhua Han, Lu Hou, Wei Zhang, Lanqing Hong, Hengshuang Zhao, and Hang Xu. Illume+: Illuminating unified milm with dual visual tokenization and diffusion refinement, 2025.
[28] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
[29] Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality. 2023.
[30] Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction. ArXiv, abs/2403.006633, 2024.
[31] Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning. ArXiv, abs/2410.14157. 2024.
[32] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. ArXiv, abs/2310.01798, 2023.
12

---

[33] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces, Advances neural information processing systems, 34:1 1989-17993, 2021.
[34] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Proceedings of the 41st International Conference on Machine Learning, pages 32819 -2844,, 2024.
[35] Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data, Advances neural information processing systems, 37:103131-103166 2024.
[36] Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 17:130336-10188 2024.
[37] Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. Advances in Neural Information Processing Systems, 37: 37:133345-133385 2024.
[38] Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Karrer, Yaron Lipman, and Ricky T. Q. Chen. Flow matching with general discrete paths: A kinetic-optimal perspective. In The Thirteenth International Conference on Learning Representations, 2025.
[39] Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Genera- tive flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. In International Conference Machine Learning, pages 5453-5512, PMLR, 2024.
[40] Mercury coder, 2025. URL
[41] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. urXiv preprint arXiv:2209.0384 030003 2022.
[42] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, arXiv preprint arXiv:2210.02747, 2022.
[43] Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. 2023.
[44] Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.0990 2025.
[45] Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong, Dream 7b, 2025. httts::////k wtt.:ggittbb..bbb..o.oo wot.:og///oo/oooo 2025/ /dee/
[46] Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang. Dual diffusion for unified image generation and understanding. arXiv preprint arXiv:2501.0023 2024.
[47] Minghui Hu, Chuanxia Zheng, Heliang Zheng, Tat-Jen Cham, Chaoyue Wang, Zuopeng Yang, Dacheng Tao, and Ponnuthurai Suganthan. Unified discrete diffusion for simultancous langunelanguage generation. arXiv preprint wxXiv:2211.14842 2022.
[48] Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, and Katerina Fragkiadaki. Unified multimodal discrete diffusion. arXiv preprint arXiv:2503. 20855,, 2025.
[49] Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusson language models via adaptation from autoregressive models. arXiv preprint arXiv:2410. 17891, 2024.
[50] Enze Xie, Junsong Chen, Yuyang Zhao, Jineheng Yu, Ligeng Zhu, Chengyue Wu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, et al. Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer. arXiv preprint arXiv:2501. 18427, 2025.
13

---

[51] Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, and Ping Luo. Dancegrpo: Unleashing grpo on visual generation, 2025.
[52] Runhui Huang, Chunwei Wang, Junwei Yang. Guansong Lu, Yunlong Yuan, Jianhua Han, Lu Hou, Wei Zhang, Lanqing Hong, Hengshuang Zhao, et al. Illume+: Ulluminating unified milm with dual visual tokenization and diffusion refinement. arXiv preprint arXiv2250.0099: 2025.
[53] Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang. Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, Song Han, and Yao Lu. VILA-u: unified foundation model integrating visual understanding and generation. In The Thirteenth International Conference on Learning Representations, 2025.
[54] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.090 098188 2024.
[55] Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, Bin CHEN, Chengru Song, dai meng, Di ZHANG, Wenwu Ou, Kun Gai, and Yadong MU. Unified language-vision pretraining in LLM with dynamic discrete visual tokenization. The Twelfth International Conference Learning Representations, 2024.
[56] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer unify multimodal understanding and generation. arXiv preprint arXiv:2408.1255 2024.
[57] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-683 6851, 2020.
[58] Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuan-Jing Huang, and Xipeng Qiu. Diffusionbert: Improving generative masked language models with diffusion models. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 4521-45344 2023.
[59] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference computer vision, pages 11975-1998,, 2023.
[60] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06522 2024.
[61] Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint 17X1v:2306.71107, 2023.
[62] Chris Wendler. wendlercheenderetex,, 2023.
[63] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, 2021.
[64] Jason Obeid and Enamul Hoque Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model, 2020. URL /ttp.tttss///arxv.ooog//a/
09142
[65] Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmre: Machine reading comprehen- sion document images. In AAAI, 2021.
[66] Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, and Lingpeng Kong. G-llava: Solving geometric problem with multi-modal large language model, 2023. URIphttps: h //trxiva.vvv///////// www.wvorrgaaao. abs/
2312, 11370
14

---

[67] Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P. Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning, 2022. httpps://aaxii..t.////
abs5 14517
[68] Zhangchen Xu, Fengqing Jiang. Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned Ilms with nothing. ArXiv, abs/2406.08464 2024. URL hwt.h:ttps:/eett /tpp://pp..... http:///..p.ssmaniiiio.mmm
org/ 2
[69] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. The Thirteenth International Conference on Learning Representations, 2025.
[70] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical cext-conditional image generation with clip latents. arXiv preprint 2XXi:22044006155, 1(2):3, 2022.
[71] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint 230.::2307.01952. 2023.
[72] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang. Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://odn. openai. com/papers/dall-e-3 pdf. 2(3):8, 2023.
[73] Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404. 14396, 2024.
[74] Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang. Yi Jiang. Yiming Gao, Hu Ye, Daniel Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03066 2024.
[75] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object- focused framework for evaluating text-to- text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024.
[76] Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with Ilm for enhanced semantic alignment. arXiv preprint arXiv:2403.05133 2024.
[77] Le Zhuo, Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongjie Huang. Wenze Liu, Lirui Zhao, Fu- Yun Wang, Zhamyu Ma, et al. Lumina-Next Making Lumina-T22 stronger and faster with Dext-DiT. arXiv preprint arXiv:2406. 248.1 2024.
[78] Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Play- ground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint 2rXi 2024.
[79] Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-DiT: A powerful multi- resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint arXiv:2405.08748. 2024.
[80] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-a: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision, pages 74-91, Springer, 2024.
[81] Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevim: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv: 16886, 2023.
15

---

[82] Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin. Bo Zhang, al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.0377 2024.
[83] Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Llava-phi: Efficient multi-modal assistant with small language model. arXiv preprint urXiv:2401.023300 2024.
[84] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[85] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv: 12966, 2023.
[86] Hugo Laurençon, Daniel van Strien, Stas Bekman, Leo Tronchon, Lucile Saulnier, Thomas Wang, Siddharth Karamcheti, Amanpreet Singh, Giada Pistilli, Yacine Jernite, and et al. Introducing idefics: An open reproduction of slate-of-the-at visual language model, 2023. URL
coc.:///cccoooo///oo///oooo////a /ddefics
[87] Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, et al. Unified language-viste vision pretraining with dynamic discrete visual tokenization. arXiv preprint arXiv:2309.04666 2023.
[88] Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprini arXiv:2412.1 14164, 2024.
[89] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, dean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
[90] Yecheng Wu, Zhuoyang Zhang. Junyu Chen, Haotian Tang. Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint (eX.:::449904440 2024.
[91] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[92] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint
(Dtv:23006.3399. 2023.
[93] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge. and Ying Shan. Seed- bench: Benchmarking multimodal iims with generative comprehension. arXiv preprint arXiv:2307.16125, 2023.
[94] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modas model an all-around player? arXiv preprint wrXiv2230700628/ 2023.
[95] Drew Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[96] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556 9567, 2024.
[97] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabili- ties. arXiv preprint ar 2308v:2388.00 2308.02490 2023.
16

---

[98] Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang. Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXiv preprint arXiv:2412.04466 2024.
[99] Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv 054000 2025.
preprint
[100] Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang. Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An spencsource toolkit for evaluating large muult--modality models. In Proceedings of the 32nd ACM international conference on multimedia, pages 11 198 1 11201, 2024.
[101] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.0550 0090060..966.... 2022.
[102] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26439-26453 2024.
[103] Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal 11m with discrete sequence modeling, arXiv preprint arXiv:2402.1220
2024.
[104] Kaihang Pan, Wang Lin, Zhongqi Yue, Tenglong Ao, Liyu Jia, Wei Zhao, Juncheng Li, Siliang Tang, and Hanwang Zhang. Generative multimodal pretraining with discrete diffusion timestep tokens. arXiv preprini arXiv:2504. 14666. 2025.
[105] Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jian- jian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. In ICLR, 2024.
[106] Kaizhi Zheng, Xuehai He, and Xin Eric Wang. Minigpt-5: Interleaved rision-and-lane and-language generation via generative vokens. arXiv preprint 23X1v:2310.022.99 2023.
[107] Shenggiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 11m. In Forty-first International Conference on Machine Learming, 2024.
[108) Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. Blip3-o: A family of fully open unified multimodal
training and dataset, 2025.
[109] Yiyang Ma. Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. arXiv preprini arXiv:2411.07977 2024.
[110] Michael Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. ICLR 2023 Conference, 2023.
[111] Aram Davtyan, Sepehr Sameni, and Paolo Favaro. Efficient video prediction via sparsely conditioned flow matching. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 23263-23274, 2023.
[112] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. Jianyuan Zeng, Jiayu Wang. Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang,
17

---

Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models, 2025.
[113] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL lnttps://openai com/ hempresearch////
geder-genertionn models as world s simulators
[114] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Junkun Yuan, Yanxin Long, Aladdin Wang, Andong Wang, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Kai Wang, Mengyang Liu, Pengyu Li, Shuai Li, Weiyan Wang. Wenqing Yu, Xinchi Deng, Yang Li, Yi Chen, Yutao Cui, Yuanbo Peng, Zhentao Yu, Zhiyu He, Zhiyong Xu, Zixiang Zhou, Zunnan Xu, Yangyu Tao, Qinglin Lu, Songtao Liu, Dax Zhou, Hongfa Wang, Yong Yang, Di Wang, Yuhong Liu, Jie Jiang, and Caesar Zhong. Hunyuanvideo: A systematic framework for large video generative models, 2025.
[115] Alexander H Liu, Matt Le, Apoory Vyas, Bowen Shi, Andros Tjandra, and Wei-Ning Hsu. Generative pre- vre-training for speech with flow matching. arXiv preprint 2310:2310.163338 2023.
[116] Matthew Le, Apoory Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 14:110005-4034 2023.
[117] Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang. Xinyue Zhang. Robert Adkins, William Ngan, et Audiobox: Unified audio genera- tion with natural language prompts. urXiv preprint 2.. 2023.
[118] Jason Yim, Andrew Campbell, Andrew YK Foong, Michael Gastegger, José Jiménez- Luna, Sarah Lewis, Victor Garcia Satorras, Bastiaan S Veeling, Regina Barzilay, Tommi Jaakkola, al. Fast protein backbone generation with se (3) flow matching. arXiv preprini arXiv:2310.05297, 2023.
[119] Bowen Jing, Bonnie Berger, and Tommi Jaakkola. Alphafold meets flow matching for generating protein ensembles. In International Conference on Machine Learning, pages 22277-22303. PMLR, 2024.
[120] Avishek Joey Bose, Tara Akhound-Sadegh, Guillaume Huguet, Kilian Fatras, Jarrid Rector- Brooks, Che Liu, Andrei Cristian Nica, Maksym Korablyov, Michael Bronstein, and Alexander Tong. (3)-stochastic flow matching for protein backbone generation. arXiv preprint arXiv:2310.022. 02.09991,10 2023.
[121] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter. Szymon Jakubezak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xi- aoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. #o: vision-language-actoon flow model for general robot control, 2024.
[122] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Internationnl conference on machine learning, pages 8162-8171. PMLR, 2021.
[123] Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[124] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. Advances neural information processing systems, 35:4328-44 4434,4433,400 2022.
18

---

[125] Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. In The Eleventh International Conference on Learning Representations, 2023.
[126] Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. Advances in Neural Information Processing Systems, 36:16693-16715, 2023.
[127] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in neural information processing systems, 34:12454- 12465, 2021.
[128] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1315-11325, 2022.
[129] Tianxiao Shen, Hao Peng, Ruoqi Shen, Yao Fu, Zaid Harchaoui, and Yejin Choi. Film: Fill-in language models for any-order generation. arXiv preprint arXiv:2310.09930, 2023.
[130] Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. In First Conference on Language Modeling, 2024.
[131] Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. score-based continuous- time discrete diffusion models. In The Eleventh International Conference on Learning Repre- sentations, 2023.
[132] Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 0222266-282777 2022.
[133] Shen Nie, Fenggi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.88. 185114 2024.
[134) Jiacheng Ye, Shansan Gong, Liheng Chen, Lin Zheng, Jiahui Gao, Han Shi, Chuan Wu, Xin Jiang, Zhenguo Li, Wei Bi, et al. Diffusion of thought: Chain-of-thought reasoning in diffusion language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[135] Jiacheng Ye, Zhenyu Wu, Jiahui Gao, Zhiyong Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Implicit search via discrete diffusion: A study on chess. In The Thirteenth International Conference on Learning Representations, 2025.
[136] Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning. The Thirleenth International Conference on Learning Representations, 2025.
[137] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi. Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations.
[138] Shanghai AI Laboratory. Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2023.
[139] Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transac- tions of the Association for Computational Linguistics, 2023.
[140] Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for lite lasion- models, 2024.
[141] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In NeurIPS, 2021.
19

---

[142] Junke Wang, Lingchen Meng. Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see to believe: Prompting gpt-4y for better visual instruction tuning. arXiv preprint arXiv:231 2023.
[143] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:231 2023.
[144] Paul Lemerr Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, Jose G Moreno, and Jesús Lovón Melgarejo. ViQuAE, dataset for knowledge- visual question answering about named entities. In Proceedings of The 45th International ACM SIGIR Conference on Research and Development Information Retrieval, SIGIR' 22, New York, NY, USA, 2022. Association for Computing Machinery. doi: 10. 10.1145/34779553331733
[145] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In CVPR, 2019.
[146] Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7W: Grounded Question Answering Images. IEEE Conference on Computer Vision and Puttern Recognition, 2016.
[147] Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V. Jawahar. lcdar2019 competition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE, 2019. doi: 10. 109/icdar:20199 9000444.
[148] Jean-Philippe Thiran Guillaume Jaume, Hazim Kemal Ekenel. Funsd: A dataset for form understanding in noisy scanned documents. In Accepted to ICDAR-OST, 2019.
[149] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty, Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019.
[150] Mlhme-38k, 2025. URL httpp: //ai /tttpp:://ai.00taal. www.cccm..aa.. icdar
[151] A. Mishra, K. Alahari, and C. V. Jawahar. Scene text recognition using higher order language priors. In BMVC, 2012.
[152] Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xiang Bai. Syntax-awar aware network for handwritten mathematical expression recognition. arXiv preprint arXiv:2203.0166 2022.
[153] Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeong Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Conference on Computer Vision (ECCV), 2022.
[154) Jianfeng Kuang, Wei Hua, Dingkang Liang, Mingkun Yang, Deqiang Jiang, Bo Ren, and Xiang Bai. Visual information extraction in the wild: practical dataset and end-to-end solution. In International Conference on Document Analysis and Recognition, pages 36 66-53. Springer, 2023.
[155] U-V Marti and Horst Bunke. The databaaaasse: an english sentence database for offline handwriting recognition. International journal on document analysis and recognition, 5: 39 39-46, 2002.
[156] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension, 2020.
[157] Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.071400 2016.
[158] Markus Diem, Stefan Fiel, Florian Kleber, Robert Sablatnig, Jose M. Saavedra, David Contr- eras, Juan Manuel Barrios, and Luiz S. Oliveira. Proceedings of ieee international conference on frontiers handwriting recognition. In 2014 14th International Conference on Frontiers in Handwriting Recognition, pages 779-784, 2014. doi: 10.1109//FFH..2014....
20

---

[159] Deepform, 2025. URL http:://iaalloos//f/cccco/ooooop
CeepForm-Underttan Struuuuurrtuuuuuddre -ow.aaeooooooooooa----s--aaaaaa
at- Sadoooooeeeeoooooo0000
[160]
Tomasz Stanistawek, Filip Graliński, Anna Wróblewska, Dawid Lipiński, Agnieszka Kaliska, Paulina Rosalska, Bartosz Topolski, and Przemyslaw Biecek. Kleister: key information extrac- tion datasets involving long documents with complex layouts. In International Conference on Document Analysis and Recognition, pages 564-579. Springer, 2021.
[161] Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua. Towards complex document understanding by discrete reasoning. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4857-4866, 2022.
[162] Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508 00305, 2015.
[163] Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structure mathematical reasoning. In International Conference Learning Representations (ICLR), 2023.
[164] Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir Radev. Robut: A systematic study of table qa robustness against human- annotated adversarial perturbations. arXiv preprint arXiv:2306. 14321, 2023.
[165) Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In ACL, 2022.
[166] Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEÉ/CVF Winter Conference on Applications of Computer Vision, pages 1527-1536, 2020.
[167] Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In CVPR, 2018
[168] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawa- har. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697-1706, 2022.
[169] Benny J. Tang, Angie Boggust, and Arvind Satyanarayan. Vistext: A benchmark for semanti- cally rich chart captioning, 2023. URL htt.ttttt////i....oooo argg/bbs/200/ 2ttp//3377000556666
[170] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint 240.:22408..3326, 2024.
[171] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023.
[172] Xingyu Chen, Zihan Zhao, Lu Chen. Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. Websre: a dataset for web-based structural reading comprehension, arXiv preprint arXiv:2101.088 100.:2000..945,, 2021.
[173] Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, Peng Gao, and Hongsheng Li. Mavis: Mathe- matical visual instruction tuning. 2024. URL httph:ttt///eee //ataii///////////aaeaaeaaaaaa www.sii...r./.... aba/aa//000 02407.0800 .87399
[174] Mchran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse; A systematic evaluation of large models for geometric reasoning. arXiv preprint 0rXiv:2312.12224. 2023.
[175] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song- -hhun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning, 2021. URL http//aaxivv..e org abs 2210/ 0211/
21

---

[176] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math- dataset. Advances in Neural Information Processing Systems, 77:95095-951699 2024.
[177] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang. Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.66866 2024.
[178] Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pages 4999-5007, 2017.
[179] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
[180] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram worth a dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 251. Springer, 2016.
[181] Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.0566 2023.
[182] Chandeepa Dissanayake, Lahiru Lowe, Sachith Gunasekara, and Yasiru Ratnayake. Open- bezoar: Small, cost-effective and open models trained on mixes of instruction data, arXiv preprint arXiv:2404.1 12195, 2024.
[183] Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. Advances in Neural Information Processing Systems, 37:90629-90666 2024.
[184] Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng Kuo Yang, et al. Emova: Empowering language models to see, hear and speak with vivid emotions. arXiv preprint arXiv:2409. / 18042, 2024.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see Section and Section 4 for details.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see appendix for details.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
We did not make new theoretical contributions in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please Section and Section 4 for details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer:
[No]
Justification:
We will release the code and model weight when the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.ccppullc//uuuii wwwiiicgguie//eeeeeeeeeic CoodeSubeiiie ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Section for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please see Section 4 for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
25

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please Section for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp//p............tt.cccsueaaa
Answer: [Yes]
Justification: Please the main paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please appendix for details.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
26

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We will put efforts in this when release model in the future.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please see the Reference Section for details
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswipteeeee www.ree.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
27

---

Answer: [NA]
Justification:
We will put efforts in this when we release our model in the future.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
28

---

The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non standad components.
Please refer to our LLM policy hattps://neurips... 200ps//eee//0ooeeeeeee////////// for what should or should not be described.
29

---

A Related Work
A.1 Unified multimodal LLMs
End-to-End and Two-Stage Modeling, Autoregressive (AR) modeling remains core strategy for unified multimodal understanding and generation, but recent advances have led to two distinct AR-based paradigms.
Autoregressive Paradigms:
The first is the end-to-end AR paradigm, in which all modalities images, text, video, and even audio-are tokenized into a unified discrete space and directly modeled within a single AR sequence framework. Representative works such as Unified-IO [OT [02], Chameleon [54]. AnyGPT [103), and Emu3 [18] follow this approach: a transformer autoregressively predicts the next token across modalities, with image tokens directly decoded back to pixels via learned decoders such as VQGAN. DDT-Llama [T04] further improves tokenization by introducing recursive diffusion timestep tokens, enabling better alignment with language modeling and image reconstruction. This approach enables strong performance in both understanding and generation, and supports flexible modality conversion (e.g.. AnyGPT covers speech and music). Building on this foundation, models like Janus [20] and Janus-Pro [266 decouple visual encoding for understanding and generation to address the granularity mismatch, while VILA-U (90), LWM [69], and LaVIT (55) focus on efficient tokenization, unified visual-text alignment, and scaling to long-context and video scenarios.
Illume (21] and Illume+ [52] further enhance data efficiency and token alignment, with Illume+ introducing dual visual tokenization and diffusion-basce decoder for higher-fidelity image synthesis
and editing.
By contrast, the two-stage AR+diffusion paradigm separates sequence modeling and image synthesis: AR models first generate image tokens, which are then used as conditions for downstream diffusion decoders to boost image quality and diversity. Representative works include DreamLLM [105], which enables free-form interleaved multimodal generation; MiniGPT-5 [066], which improves image- text coherence with a two-stage pipeline: NEXT-GPT [107. which supports any-to-any modality conversion by connecting AR sequence modeling with modular diffusion decoders; MetaMorph (88), which efficiently adapts LLMs for unified text and visual token generation; SEED-LLLMMA [17), which aligns image token semantics with text for scalable multimodal autoregression: and SEED-X (73), which further enables arbitrary-size and multi- ceatti-graulaarttty image generation. Recently, BLIP3-o [08 ad advanced this paradigm by generating CLIP-based image features using a diffusson transformer and adopting sequential pretraining to better balance understanding and generation. Collectively, these models demonstrate the flexibility and high image fidelity achievable with the two-stage approach, highlighting distinct trade-off with end-to-end AR models in reasoning and generation quality.
Integrating AR and Diffusion within a Unified Framework. To bridge the gap between the reasoning strengths of AR models and the generative power of diffusion models, hybrid paradigms have emerged that combine both mechanisms in a unified architecture. For example, JanusFlow [109] employs continuous reactified flow for image generation, Show-o [56) adopts discrete MaskGIT-style diffusion, while Transfusion [99 utilizes a continuous U-Net-based DDPM. Despite their differences in diffuusin implementation, these hybrid models all enable more flexible and controllable vision-language generation, further blurring the boundaries between AR and diffusion approaches.
Hybrid Paradigm:
Diffusion Paradigm:
Fully Diffusion- Multimodal Generation. In parallel, fully diffusion- based approaches have also been proposed for unified multimodal modeling. UniDise [88 and D-Dit [46] formulate both text and image generation as a discrete diffusion process, starting from masked sequences and enabling joint inpainting, editing, and controllable multimodal generation. By leveraging the iterative denoising process, diffusion models typically achieve superior generation fidelity and support fine-gri grained,, high-qualit editing. Moreover, unlike autoregressive models that generate tokens sequentially, diffusion- approaches can produce multiple tokens in parallel during inference, improving efficiency and enabling more globally consistent outputs. While these models offer enhanced controllability and flexible inference, they may still face challenges in com- plex instruction following and sequential reasoning. Nevertheless, fully diffusion- paradigms represent promising direction for scenarios requiring fine -rained editing, state-of-the- generation quality, and efficient parallel decoding across modalities.
Comparisons with Bagel [25]. Bagel 25 is a very strong recent advance in unified multimodal understanding and generation. While both FUDOKI and Bagel aim for unified multimodal mod-
30

---

eling, they are based on fundamentally different generative paradigms and architectural choices. Specifically, Bagel employs large
Transformer-Experr (MoT) architecture and follows the autoregressive (AR) modeling paradigm, enabling it to efficiently scale with massive, carefully structured interleaved multimodal data. In contrast, FUDOKI is the first general-purpose unified multimodal model built entirely on discrete flow matching, which allows for bidirectional information integration and iterative self-correction during generation. In terms of empirical performance, Bagel demonstrates strong results on both multimodal generation and understanding, including advanced tasks such as free-form image manipulation. We acknowledge that FUDOKI currently lags behind Bagel, which can be attributed mainly Bagel's Is novel data scaling strategies and substantially larger model size (14B parameters for Bagel vs. 1.5B for FUDOKI). We will explore integrating similar scaling approaches in future work.
A.2 Flow Matching
Flow matching offers fundamentally different approach to generative modeling compared to diffusion models. While diffusion models rely on repeatedly injecting random noise into data and then iteratively denoising it, flow matching instead learns smooth, continuous transformation, formulated through ordinary differential equations (ODEs), that maps simple distribution (such as Gaussian noise) directly to real data. This approach eliminates the need for repeated noise addition and removal.
Pioneering this direction, Lipman et al. 42 introduced Continuous Normalizing Flows (CNFs) and the flow matching framework, which trains neural networks by regressing vector fields along flexible probability paths. This work laid the foundation for subsequent advances in based generative modeling. Building on this, Liu et 41 proposed Rectified Flow, which learns neural ODEs along straight-ling line paths between distributions, enabling more efficient and scalable training for tasks such as image generation and domain adaptation. More recently, Albergo and Vanden-Eijnden 110) presented InterFloow which simplifies training by directly inferring the velocity field from the probability flow of an interpolant density, thus avoiding costly ODE backpropagation and supporting efficient likelihood estimation and high-resolution generation.
key advantage of flow matching its sampling efficiency: by allowing deterministic sampling in just a few ODE steps, it achieves competitive FID scores with orders of magnitude fewer steps compared to diffusion- based samplers. This remarkable efficiency has quickly made flow matching a dominant approach in state-of-the-ait image and video generation models.
Recent studies have also extended flow matching to discrete data domains. Campbell al. [39 introduced Discrete Flow Models (DFMs), which generalize flow matching to discrete spaces using continuous-tine Markov chains, improving multimodal modeling of both continuous and discrete data over discrete diffusion models. Similarly, Gat et al. (37] proposed Discrete Flow Matching, framework that supports general probability paths and scalable con-autoregressvve generation, significantly narrowing the performance gap between discrete flow and autoregressive models on coding benchmarks.
Thanks to these advances, flow matching methods have demonstrated strong performance across wide range of domains, including image synthesis [14 13). video generation [144 speech and audio generation [15H117), protein design [10) and robot control [121]. These successes underscore the broad applicability and effectiveness of flow matching frameworks.
A.3 Discrete Diffusion Models
Diffusion models have achieved remarkable success in continuous domains such as images and audio [57 122 [23]. However, their adaptation to natural language poses unique challenges due to the discrete nature of text. Early attempts to overcome this primarily injected Gaussian noise into token embedding spaces, followed by denoising to reconstruct discrete sequences 124 125 Representative models in this line include Diffusion-LM [124], DiffuSeq [125] and Plaid [26]. While these approaches show promise for controllable generation and to-eeneee----eeeeeee tasks, the need to map between discrete and continuous representations complicates training and inference.
Recent research has shifted to discrete noise-based diffusion models to address these limitations, where noise injection and denoising are directly defined in the symbol space. The most influential
31

---

early works in this direction are Argmax Flows [17]] and D3PM B3]. D3PM, in particular, provides a systematic framework for discrete diffusion, formalizing both absorbing mmask-basede and uni- form (categorical) noise processes for sequence corruption. These foundational studies enable the progressive corruption of discrete sequences through distinct forward processes: in the absorbing (mask-based) process, tokens in the original sequence are gradually replaced with a special absorbing token (e.g.. <MASK>); in the uniform (categorical) process, tokens are progressively replaced with randomly sampled tokens from the vocabulary. The diffusion model is then trained to reverse these processes, denoising the corrupted sequence back to the original data. Building on these foundations, subsequent models such DiffusionBERT [58]. LLaDA 44, and MD4 [35] introduce improve- ments in noise scheduling, scalability, and training objectives. Methods like MaskGIT [128] and FiLM [29]. although originally proposed for vision or general infilling tasks, are methodologically aligned with mask-hased diffusion, employing iterative generation with absorbing masks. These models have achieved performance competitive with, even superior to, autoregressive models in language modeling, infilling, and reasoning tasks.
In addition to mask-based approaches, the uniform (categorical) transition process, also formalized in D3PM, corrupts sequences by progressively replacing tokens in the original data with tokens sampled uniformly from the vocabulary, rather than single mask token. SEDD [34] extends score matching to discrete data via a score entropy loss, achieving state-of-the-ane art results and in some cases surpassing autoregressive baselines. RDM [300] introduces reparameterized sampling framework to improve training and sampling efficiency. Furthermore, recent studies [I3 (
model discrete diffusion continuous- time Markov chain, advancing theoretical understanding and practical efficiency. Most recently, Discrete Flow Matching (DFM) (37 was proposed as a novel discrete flow paradigm for generative modeling of high-dimensional discrete data. Unlike flow matching and paradignn diffusion models designed for continuous domains, DFM introduces a general family of probability paths that interpolate between source and target distributions in discrete space, and provides a unified formula for sampling from these paths using learned posteriors such as probability denoisers and noise predictors. Empirically, DFM demonstrates that adopting uniform (categorical) transition process, rather than an absorbing (mask-basee process, consistently leads to improved generative performance.
Recent scaling studies further demonstrate that, in addition to matching autoregressive models in perplexity and generation quality, discrete diffusion models have achieved strong performance on complex reasoning and planning tasks, underscoring their flexibility and potential as competitive
alternatives for natural language generation and understanding 133 44 35). Recent work [49] explores directly adapting pretrained autoregressive language models into non-autoregressive dif- fusion models via continual finetuning, enabling efficient knowledge transfer between paradigms. Building on this line, Dream 7B [45] further advances diffusion LMS by consistently outperforming previous diffusion models and matching the performance of top autoregressive models of similar size.
B More Comparison with State-of-the-arts
Qualitative Comparisons on Visual Generation. Figure 6 presents qualitative comparisons of visual generation results produced by three models: Janus (20). D-DiT [46), and our method, FUDOKI, across a diverse set of text prompts. Each row corresponds to different prompt, covering scenarios such as animals in unusual environments, cartoon avatars, and objects with specific attributes. As shown in the figure, FUDOKI consistently produced images that more accurately captured the semantics of the prompts, demonstrating superior text-image alignment and higher visual fidelity.
Qualitative Comparisons on Visual Understanding, Figure 7 presents qualitative comparisons of visual understanding capabilities among Janus (AR) [20]. DiD (mask-based discrete diffusion, MDD) 46), and our FUDOKI (discrete flow matching, DFM). The upper section shows selected intermediate outputs from each model's answer generation process, illustrating their reasoning dynamics. The lower section presents additional visual question answering cases, where FUDOKI demonstrates higher reasoning accuracy and better alignment with ground truth answers, highlighting its superior ability to generate reliable and precise responses.
32

---

Janus
D-DIT
FUDOKI
Janus
D-DIT
FUDOKI
Rainbow colored penguin
A blue bird brown bear
An elephant under
shark desert
A fisheye lens view of turtle sitting in a forest
A carrot on the left of broccoli
Girl with long hair, cartoon avatar
A horse running on the beach at sunrise
Teddy bear plush toy with glasses
wears
Figure 6: Qualitative Comparisons Visual Generation. Comparison among Janus 20 D-DIT 46
and FUDOKI various text prompts. The results demonstrate that our method (FUDOKI) achieved superior text-image alignment and aesthetics.
C Further Results
The Denoising Process FUDOKL Fig. Billustrates the iterative refinement process enabled by the discrete flow matching framework in FUDOKI, demonstrating its application to both generation and understanding tasks. The top panel visualizes how images are progressively denoised over iterations, transitioning smoothly from initial noisy prior up to the final high-fidelity image Across diverse generation examples-range ranging from animals to objects-the model incrementally sharpens semantic details and corrects spatial structure at each refinement step. The bottom panel depicts similar iterative refinement for the understanding task, where the model extracts text from an image. Starting from noisy token sequence, irrelevant incorrect tokens are gradually replaced with accurate tokens (c.g., "Sara Lee") as the model converges to the correct answer. The red arrows highlight token-level updates during each step, emphasizing the model's ability to systematically and continuously correct errors and align predictions. This figure showcases how discrete flow matching enables fine-grained control and progressive improvement in both modalities by modeling transitions in discrete space, leading to more accurate and coherent outputs. More cases can found in our project page: fudoki-dffm.ao wtt.dddfw.iihubbboouuookkk
Maze Navigation. this section, we train our proposed FUDOKI model on novel mase navigation-wilt whhch simultancously requires understanding and generation capabilities. To this end,
33

---

Janus (AR) Generation Process
"Ea, surthar surmaarine souna
surmartins
avemariins the
summertise
Thee
clons The
the people wartin fygtt distiing luggett
the people sooring ligit
the people winrring light
clstting suggsd tuat
clutting daggast that
typically
ossociated wilk
(b) D-DIT (MDD) Generation Process
Question: is
emask menskooaakkoon ooce
enssk www.koomonk www.eeeeeeeeeeeerrr........
summertine in the tcans? Antwer it and give the rationole.
ematio mantts
dooc/cc/sccccaaaa crasto enatoo
Ground Truth: No, It looks to be Fall. I think so becouse
(c) FUDOKI (DFM) 6 Generation Process (with "thinking" process)
intlyrentt ad Votice
NOt is nrt prictionnouss
NO, not savmartina the
No. It not suremeetiin the le
there are pumpkins in the background and they are normally associated with fall.
muiir 68 ereetmoet) HiD F
this
Evertheyss
astwvoring that the
le thut Image
Exentorguihng Brees greaeepai
2. overa grove showing lorge
shown lorge Dapotch, whetearre
shows
whech
Dispotth arer (ioered::: 3. dplavecy
clepatci nevecacasore
hypisally karwated hails
are typicaly hurvested the
picturatcue
heatonallt, the surver
coherentere deficentit it Hotton
aiteshowsehsin The Redd
en the image segpests shorter
detting end the redindaats
BE rlbuut canisug? Antion
e wandr coficiect thet:
weather Anton hares istasaatiin
image suppest cooler
bedget
curvise the
t=0
Question: the Image, what does the handshake between the two men symbolize?
Question: Which action is performed
A. The completion of a business deal or on important appointment.
In this Image?
pushing Snowboording
The exchange of personal belongings.
G. biking through snow
C. The start of a friendly conversation. celebration of personal
shoveling
achievement.
Ground Truth:
Ground Truth: A
D-DiT: 

Jenua: 

FUDOKI: A. The completion of business deal important appointment.
FUDOKI: The action performed this image pushing
Question: there yellow brick building image? Please answer yes or Ground Truth: No
ANGIES TP
Question:
ward lago "angle's"?
Please onswer yes or no.
Ground Truth: No
Janus: Yes X
Jonus: Yes x
D-DIT: Yes, there are several yellow brickd buildings positioned next to road in the image.
D-DIT: Yes,
FUDOKI: No.
FUDOKI: No.
Question:
the couch behind
Question: What type of material is the crosswalk the street lamp made of? Ground Trufh: concrete
of?
Ground Truth: dog
Janus: Goats.
Janus: Metal
to street lamp FUDOKI: Concrete
D-DIT: couch front a white dog.
made of materials
FUDOKI: Deg
Figure 7: Qualitative Comparisons on Visual Understanding. The upper part of the figure shows selected intermediate outputs from the answer generation process of different monels-Janus (AR). D-DiT (mask-based discrete diffusion, MDD), and our FUDOKI (discrete flow matching, DFM)-o illustrate their reasoning approaches. Specifically, Janus, the AR- model, is unable to revise its initial incorrect response (i.e., "Yes, it is summertime... ... "), even after generating the correct rationale later (i.e.. "The large pumpkins suggest that it is autumn") "). making its response inconsistent overall. Mcanwhile, D-DiT the mask based diffusion model, fails to handle this reasoning task, often producing empty outputs (i.e., only 
 tokens). In contrast, our discrete flow matching model, FUDOKT, demonstrates coherent and accurate reasoning trajectory, producing consistent and correct answers. The lower part of the figure provides additional qualitative examples on visual question answering tasks. FUDOKI consistently delivers more accurate and well-aligned reasoning with the ground truth.
Fig. 9 presents a series of multimodal decision- maasiion-maiing scenarios where FUDOKI and GPT-40/GPT- Image-1 are evaluated on their ability to reason over spatial layouts and produce both textual and visual outputs. Each case involves frozen lake grid of increasing size (3x3, 4x4, and 5x5), with a defined goal and character's 's current position. The task to select safe move that avoids hazards (dark blue holes) while progressing toward the treasure. We notice that while GPT-4o provided well- reasoned textual explanations that include safety considerations, goal alignment, and environmental awareness, its visual updates lacked consistency with its textual responses, and even altered the maze structure (in the third row of the figure). contrast, FUDOKI consistently predicted plausible directions and generated coherent visual updates aligned with the task constraints, showing basic
34

---

Iterative Refinements for Generation
X1
Iterative Refinements for Understanding
Sample
Answer
X
ATED
wear
Lipschitz
+]
numberOf
weed
1-
Sarafee
Question: Extract
text from the image.
The
text
the
image
weed
Sara
Lee
A. Laura Des
8. Sara Lee
Sweets
D. Mara Treats
X1
text
image
Sara
Lee
Figure 8: Visualization of iterative refinement process enabled by discrete flow matching FUDOKI,
demonstrating denoising process for text-to- -ottt-g--mage generation and visual understanding tasks.
Table 5: Performance Comparisons on the Math Vista Benchmark.
Method
Janus- 1.5B
Janus-Pro-1B
FUDOKI
MathVista
32.4
35.1
38.6
spatial awareness. Furthermore, as shown in Fig. TO FUDOKI is capable of completing the entire maze navigation sequence, moving from the initial position to the treasure step by step.
Results on the Math [177] Benchmark. We also evaluated our proposed FUDOKI on a more challenging mathematical reasoning benchmark, Math Vista (testmini) [177]. As shown in Table 5 we find that FUDOKI achieved the best performance compared to AR-based models at the same scale. We attribute this improvement FUDOKI's discrete flow matching framework, which leverages bidirectional context modeling to facilitate complex reasoning.
D Dataset Collections
Our training set comprises total of 12.62 million samples, divided into two main categories: Generation (8. 78,,, 69%) and Understanding (3.86M, 31%), as shown in Fig. [T The Generation subset, which is entirely composed of in in-house data, is constructed for text-to-image generation tasks. In contrast, the Understanding subset covers diverse set of information extraction and comprehension tasks. This balanced and large-scale collection ensures comprehensive support for both generative and understanding capabilities.
Specifically, the public Understanding of data covers the following aspects:
General (1506.8K, 40.6%): ShareGPT-40 (57.2K) [38], VSR (12.8K) 139 ALLaVA
Instruct (680.4K) [100) IconQA (29.9K) [41,, LVIS-InstruCtAV (10.0K) [42]
ShareGPT4V (613.3K) 113), VIQuAE (18.5K) 144), RAVEN (0.3K) 145), Visual7W
(14.4K) [146), In-house (70.0K)
35

---

Task: safe path through frozen
FUDOKI:
OPT-4o/GPT-Image-1 Based current position (1, 2). the best direction to move is Down to (2, 2) to
reach the goal.
Down (2, 2)
Game Information:
You're 3x3
goal reach grid
safely progress toward goel grid
updated
Avaid Falling into hales (dark blue areas)
after the move:
Current position: (1, 2)
*The character has moved from (1, 2) 2).
Goal position:
you move (Up, Down, Left, Right)??
remains (3, 3).
Also provide the coordinates you'll reach after this move. Then, generate image of the frozen lake ofter making
this
User: Task: Find safe path through the Frozen lake to
FUDOKI: Right (2, 3)
PPTT44/GPPTImmpee Based on the current position (2, 2). the best Right safely progress toward the gool (4, 4) while avoiding the hole at (3, 2). Here the updated frozen lake grid after
reach the
Information:
direction
You're a 4x4 frozen lake grid
Your goal is reath the freesure
Avoid falling into holes (dark blue areas)
Current position: (2, 2)
Gool position: (4, 4)
The character has moved from (2, 2)
What direction should you move (Up, Down, Left, Right??
(2, 3).
Also provide the coordinates you'll reach ofter this move. generate image the frozen after making
"The treasure remains (4,
still present.
this mave.
User: Task: safe path through the frozen lake to
FUDOKI: Left (5. 2)
GPT-40/GPT-Ina IGP--Ioge--:: Based on current position (5, 3), the best
reach the goal.
Game Information:
direction move is Up to (4, 3) safely progress toward goal while avoiding holes.
You're
Frozzen grid
Your goal to reach trecsure
Avoid folling Into holes (dark blue areas)
Here the updated frozen lake grid after the move:
Current poaition: (5. 3)
position:
(5,
What direction should you move (Up, Down, Left, Right)?
(4, 3).
Also provide the coordinates you'll reach after this move.
The treasure remains at (2, 2). The holes remain their respective positions.
Then, generate an image of the frozen lake after making
move.
Figure 9: Comparison of FUDOKI and GPT-40/GPT-Image- frozen lake navigation tasks. GPT- 4o/GPT-Image-1 offered well-reasoned textual outputs with safety and goal awareness generated inconsistent visuals, even altering the maze (e.g. the third row). FUDOKI, by contrast, consistently produced valid directions and coherent visual updates aligned with task constraints. demonstrating stronger spatial consistency.
OCR (428.0K, 11.5%): LLLVAR (59.3K) 61. SROIE (17.1K) 147.
FUNSD (6.8K) 148
OCRVQA (80K) [199]. MLHME-38K (30K) 150,, Rendered Text (10.0K) 662] III5K
36

---

User: Tøsk: Find a safe path through the Frozen lake to reach the goal.
FUDOKI: Right (4,
Information:
You're on a 5x55 frezen lake grid the Aveid falling into holes (dark blue areas)
gool
Current position: (4, 1)
position:
What direction should you (Up, Down, Left, Right)? Also provide coordinates you'll reach after this move.
Then, generate image the frozen after making this
User: Task: Find a safe path through the frozen lake to reach the goal.
FUDOKI: (5, 2)
Information:
You're re on a 5x5 frozen lake grid
goel
the
falling
areas)
Current position: (4, 2)
position:
What direction should
(Up, Down, Left, Right)? Also provide coordinates you'll reach after move.
Then, generate image the frozen lake after making
User: Task: Find safe path through the Frozen lake to reach the goal.
FUDOKI: Right (5, 3)
Information:
You're on 5x5 frozen lake grid
Your reach the treasure
falling holes
areas)
Current position: (5, 2)
Goal position:
What direction should move (Up. Down, Left, Right)? Also provide coordinates you'll reach
move.
Then, generate of the frozen lake after making this
path through
goal.
FUDOKI: Right (5,
Information:
You're on 5x5 frozen lake grid
Your is reach the treasure
folling
(dark areas)
Current position: (5, 3)
position: (5, 4)
What direction should you move (Up, Down, Left, Right)? Also provide coordinates you'll
move.
Then, generate image of the frozen lake after making this move.
Success!
Figure 10: FUDOKI successfully completed the full initial position (4, 1), sequentially selected safe moves Right - Down Right - Right-wi while avoiding holes and progressing toward the treasure at (5, 4). At each step, FUDOKI generated an updated image of the frozen lake, reflecting the character's 's new position and preserving the enviromment' structure, culminating in successful arrival at the goal. Notably, in rows 2 through 4, the input images were taken directly from FUDOKT's previous outputs, demonstrating the model's ability maintain coherent tracking and visual continuity throughout the multistep decision- decision-making process.
navigation task step by step. Starting from the
(6.0K) 151.. HME100K (74.5K) 152., SynthDoG-EN (29.8K) 153 POIE (9.4K) 154
IAM (5.7K) 155]. TextCaps (60.5K) [36], COCO-Text V2.0 (28.
157.
Chrome Writ-
ing (8.8K) 62 ORAND-CAR (2K) 158
Document 8K, 4.2%): DocVQA (122.4K) 63
FUNSD (6.8K)
148 Deepform
(9.2K) [159], Kleister CharityAI (15.2K) [160), TAT-DQA (2.2K) [6]]]
37

---

Training Datasets
Understanding Subsets
B Generation
0 Understanding
General
0.54M
MathiSciencee
Language
3.71M
OCR
0.51M
5.5
Chart
Table
Document
Screen
0.36M
0.18M
6H16H
0.02M
Training Dataset Distribution. The overall training data consists of 8.76M Generation samples (69%) and 3.86M Understanding samples (31%), as shown on the left. The right chart depicts composition of the Understanding subset by category.
Figure 11:
Table (180.2K, 4.9%): TabFact (65.6K)
161
WikiTable (29.5K)
162
TabMWP
(38.4K) [63]. RoBUT WTQ (38.2K) 164 RoBUT SQA (8.5K) 164
Chart (362.6K, 9.8%): ChartQA (62.9K) [65]. Chart2Text (27.0K) 64]. PlotQA
(10K) [1666. DVQA (200K) [67]. Infographic VQA (47.6K) [68], VisText (10.0K) [169]
Diagram Image2Text (0.3K) [70], LRV Chart (1.8K) [7]
Screen (24.6K, 0.7%) WebSRC (5.1K) 1722. VisualMRC (19.5K) 65
Math/Science (544.9K. 14.7%): MAVIS (187.3K) [131. G-LLaVA (162.4K) [66].
GcoQA+ (72.3K) [67]. GeoMVerse (9.3K) [[74), Geometry3KK (3.0K) (175), Math
sion (3.0K) [176), Cambrian Data Engine (50.8K) [177), Textbook QA (21.K) [178).
ScienceQA (19.2K) [179] A12d (18.8K) [18
Language (510.2K, 13.7%): MathInstruct (81.5K) [8I. Evol-Instruct (142.8K)
182
MathPlus (95.2K) [83], Magpie Pro (L3 MT) (50.0K) (68), ShareGPT4 (40.7K)
(84)
Magpie Pro (L3 ST) (50.0K) (68, Magpie Pro (Qwen2 ST) (50.0K) [68
E
Mathematical Formulations of Kinetic Optimal Velocity
To facilitate understanding, state space, with elements x, E T (in the main paper, we have x², z' E T). A probability path is time-varying distribution (1)),, and velocity field ux( z) describes mass transport between states over time. In this way, we have the Continuity Equation as follows.
simplified notation here and let T denote the finite discrete
px(x) + die = 0, VIET
with the discrete divergence given by div (iv) Extrr - - je 2) and (z, 2) the flux, defined by je z) = un (z, z) (z)), which represents the flow of probability mass from a to
(x.=) (((2)
if pf 0 otherwise when x # z and
r. this way, the velocity can be obtained by ux 2)
uz(2,2) - Enzz (z z) to ensure the rate condition in Eq. minimize the kinetic energy during the flow process, namely,
With such notations, we expect to
min / TEula, 2) 2)(2,2)2 2)2 Ps.ft
di Pc(=)
subject to:
Continuity Equation: div (it) (x)
Non-negativity of the flux: je(r, 2) 2 0
Vr #
Boundary conditions: po p, P1
38

---

Here, wx(x, > 0 problem-specife weight controlling the "cost" of mass moving from 2 to x. As evidenced in [38] when p; given and let w, 2) 1/p, (z). the kinetic optimal solution can be obtained via j7 (z, (z) (c(2)po(z))
Væ + 2. In this way, if we apply this kinetic optimal (2, 2) for the probability path in Eq. we can obtain the velocity defined in Eq.
F Limitations and Broader Impacts
Limitations. Despite its promising results, FUDOKI also presents several limitations that warrant further investigation. First, despite the advantages of discrete flow matching-such as being agnostic token order and compatible with bidirectional Transformers- current implementation requires the sequence length to be fixed prior to sampling. This constraint limits flexibility in generation and makes dynamic-length outputs challenging. promising direction for future work to extend the sampling scheme to support variable-length generation, which would broaden the applicability of the model across open-ended tasks and enhance the flexibility on the computational cost during inference. Besides, as shown in Fig. 12 while FUDOKI shows strong performance, it still faces challenges under certain scenarios, such performing text-to-image generation given complex prompts or prompts involving rendering specific texts in images, as well as performing visual understanding tasks that demand expert-level reasoning and domain-specifie knowledge.
Image Understanding
Image Generation
Question: What substance
that is developing on these leaves?
A. Algae
Downy Powdery mildew
Sooty mould
developing let's analyze image. The leoves show yellowish-green patchy appearance, which is characteristic of downy mildew. Downy mildew fungal disease that affects many plants, causing discotoration and fuzzy growth powdery mildew, and
determine
underside
New York Skylina with 'Diffusion' written with sky.
keyboard made of water, the water made light, light
leaves. The other options, such sooty mold, typically this kind of growth. Therefore, the correct answer Downy mildew.
Figure 12: Examples of failed cases on visual understanding and generation. While FUDOKI demonstrated generating images from complex prompts
strong performance, it still struggled with harder involving specific texts, understanding visuals require expert-level knowledge.
Broader Impacts. FUDOKI introduces a novel paradigm for unified multimodal modeling that departs from the long-dominant autoregressive approach, potentially redefining how future multimodal systems are designed. By leveraging discrete flow matching with metric-induced probability paths, FUDOKI enables controllable and interpretable generation processes, which could prove valuable in critical applications such as education, embodied AI, and autonomous driving. Its iterative, self- correcting refinement process aligns well with human reasoning patterns and may support safer, more reliable Al agents in domains requiring high precision, such as medicine and law. Furthermore, FUDOKI's unified architecture for both understanding and generation fosters more integrated, general- purpose agents an important step toward realizing practical artificial general intelligence (AGI). However, as with any generative technology, ethical considerations around bias, misuse, and content safety must be carefully addressed as adoption scales.
39


Reviews:
[{"review_id": "lgVJ269df2", "reviewer": "Reviewer_CdAa", "summary": "This paper introduces a unified model based on discrete flow matching for both generation and understanding tasks by leveraging metric-induced probability paths with kinetic optimal velocities. The model achieves promising results across multiple benchmark datasets.", "questions": "Questions:\n\nAlthough the paper applies flow matching to unified models, it uses the AR-based Janus as the base model for training. Could the authors explain why they did not choose MaskGIT or another diffusion model instead?\n\nBeyond benchmark tests, it would strengthen the paper to include other applications. Could the authors provide results on interleaved image generation or on image editing tasks?", "limitations": "It can be seen in weeknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThis work is the first to use discrete flow matching to train unified models and it achieves promising results on multiple benchmarks.\n\nThe approach leverages metric-induced probability paths with kinetic optimal velocities for self-correction.\n\nWeaknesses:\n\nDiscrete flow matching requires limiting the output length during inference compared with autoregressive methods, which makes it somewhat inconvenient for understanding tasks.", "was_revised": true, "final_justification": "I do not have more concerns. I think this paper propose a effective architecture to unify two tasks using discrete diffusion. However, I think the novelty is a little limited and the motivation using discrete diffusion is a little uncertain. Finally, I stay positive attitude torwards this work and I give boderline accept.", "rebuttal": "Thanks for your great efforts on the review and constructive comments. \nWe will try our best to answer all your questions. \n\n**Please let us know if you are not satisfied with our answers or have further questions, so that we can get back to you as soon as possible.**\n\n$\\textcolor{blue}{\\textsf{Q1: Ask about the fixed output length limitation of the discrete flow matching (DFM) framework.}}$\n\n> \"Discrete flow matching requires limiting the output length during inference compared with autoregressive methods, which makes it somewhat inconvenient for understanding tasks.\"\n\nA1: Thank you for your careful review. We acknowledge that, in its standard form, the DFM framework requires specifying the output length during inference, in contrast to autoregressive models which naturally support variable-length outputs.\nHowever, recent advances—such as EditFlow [cite 1] and DreamOn [cite 2]—have introduced edit operations into the sequence generation process, enabling more flexible and dynamic adjustment of output lengths within non-autoregressive frameworks. \nWe believe incorporating such approaches into FUDOKI is a promising direction for future work, which could further enhance its applicability to a wider range of understanding tasks.\n\n\n[cite 1] Havasi, Marton, et al. Edit Flows: Flow Matching with Edit Operations. Preprint, 2025.\n\n[cite 2] Wu, Zirui, et al. DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas. Blog, 2025.\n\n$\\textcolor{blue}{\\textsf{Q2: Ask about the rationale for choosing Janus as the base model for FUDOKI.}}$\n\n> \"Although the paper applies flow matching to unified models, it uses the AR-based Janus as the base model for training. Could the authors explain why they did not choose MaskGIT or another diffusion model instead?\"\n\nA2: Thank you for your insightful question. \nWe selected Janus as the base model because it demonstrates strong capabilities in both visual understanding and generation. In contrast, adopting MaskGIT or other diffusion-based generation models would require substantial retraining to achieve robust visual understanding, leading to significantly higher training costs. Furthermore, due to our computational constraints—all experiments were conducted on NVIDIA V100 GPUs in the paper—the compact size of Janus (under 2B parameters) makes it a more practical and efficient foundation for our framework.\n\n\n$\\textcolor{blue}{\\textsf{Q3: Ask about other applications of FUDOKI.}}$\n\n> \"Beyond benchmark tests, it would strengthen the paper to include other applications. Could the authors provide results on interleaved image generation or on image editing tasks?\"\n\nA3: Thank you for your question. We appreciate your interest in exploring broader applications of our framework. \nAs an initial step, we conducted preliminary experiments on interleaved image-text generation, i.e., Maze Navigation, which involves one input image for understanding and one output image for generation. The results are presented in Figures 9–10 of the supplementary materials.\nHowever, due to computational constraints with V100 GPUs, we were unable to extend training to more complex multimodal scenarios involving multiple images at a large scale. We recognize the value of demonstrating FUDOKI's versatility and plan to explore advanced tasks such as image editing and large-scale interleaved generation in future work if more computational resources are available."}, {"review_id": "MJ6FdoZcvO", "reviewer": "Reviewer_WC1Q", "summary": "The paper introduce a multimodal model based on kinetic flow matching. The author leverage pretrained AR model and the formulation from [1] to generate both text and image. For efficient finetuning from AR models, they also adopt strategies from [2] i.e., shifting, and discard the time embedding layers. The paper demonstrate the performance of their models on image understanding and generation are competitive with AR-based approaches. \n\n\n\n\n\n[1] Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspective (ICLR 2025)\n\n[2] Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025)", "questions": "- Including a GIF that visualizes the text‐generation process would add valuable insight. From my experience, absorbing‐diffusion models like LLaDA and Dream often behave largely autoregressively, so a clear demonstration of their bidirectional generation capability would be especially informative.\n- It would be helpful if the author add the performance of model with different number of denoising steps. \n- Are experiments run on H100s? Why don't the author fine-tune existing (larger) language diffusion baselines such as LLaDA or Dream (as I believe 43k gpu hours is quite substantial and enough for fine-tuning 7B models)? how straightforward it is to adapt an absorbing diffusion model to the kinetic diffusion?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n- The model performs competitive with AR models of similar scale on various benchmarks.\n- The paper is well-written and the experiment is well-designed\n\n### Weaknesses:\n- The algorithm itself is not novel:  the kinetic flow matching formulation is introduced in [1], the idea of adopting pre-trained AR for discrete diffusion is proposed in [2] (the author use technique from that paper as well), discarding time embedding layers is proposed and investigated in [3] and previous absorbing diffusion. The novelty mostly lie in the fact that the author extend that to visual domain. In my point of view, this makes their contribution borderline.\n- Similar to previous language diffusion models, the proposed approach operates under a fixed compute budget, which is limited for multimodal generation. For instance, a $1024\\times 1024$ pixel image translates to roughly $4096$ tokens, allocate a \"reasonable\" number of generated tokens during inference phase is not straightforward e.g., $4096$ tokens are generally not efficient for language generation task.\n- The model use two type of image encoders (SigLIP and VAE), thus, training pipeline could be rather cumbersome. For instance, multi-turn chat with multiple images, which is not straightforward to separate the understanding and generation task. In addition, it would be difficult to extend image generation tasks beyond text-to-image e.g., (multiple) image to image generation.  \n\n\n[3] Is Noise Conditioning Necessary for Denoising Generative Models? (ICML 2025)", "was_revised": true, "final_justification": "After reading the other review and rebuttal I still believe the novelty is a little limited but the execution is good to me. Therefore, I keep my initial score and lean toward acceptance.", "rebuttal": "Thanks for your great efforts on the review and constructive comments. \nWe will try our best to answer all your questions. \n\n**Please let us know if you are not satisfied with our answers or have further questions, so that we can get back to you as soon as possible.**\n\n$\\textcolor{blue}{\\textsf{Q1: Ask about the contributions of FUDOKI.}}$\n\n> \"The algorithm itself is not novel: the kinetic flow matching formulation is introduced in [1], the idea of adopting pre-trained AR for discrete diffusion is proposed in [2] (the author use technique from that paper as well), discarding time embedding layers is proposed and investigated in [3] and previous absorbing diffusion. The novelty mostly lie in the fact that the author extend that to visual domain. In my point of view, this makes their contribution borderline.\"\n\nA1: Thank you for your comments and for highlighting the relevant prior work.\nWe acknowledge that our approach builds upon several important previous studies. However, we would like to emphasize the key contributions of our work as follows:\n\n- To the best of our knowledge, FUDOKI is **the first unified MLLM built entirely on discrete flow matching for both visual understanding and generation tasks**. This approach brings several unique advantages to multimodal modeling, including **bidirectional information integration**, **parallel inference**, and **iterative self-correction through refinements**. In addition, unlike recent approaches such as MMaDA [cite 1] and LaViDa [cite 2], FUDOKI does not rely on mask tokens. \n- Our experiments demonstrate that **FUDOKI achieves performance on par with autoregressive models at the same scale**. Additionally, we also provide the first empirical validation of inference scaling within the discrete flow matching framework.\n- **We will release both our model and code to benefit and advance the open-source community**, especially considering that the open-source implementations of discrete flow matching are still limited.\n\nWe hope these points clarify the contributions and significance of our work, and we believe that our findings and open-sourced resources can inspire and facilitate further research in this area.\n\n[cite 1] Yang, Ling, et al. Mmada: Multimodal large diffusion language models. Preprint, 2025.\n\n[cite 2] Li, Shufan, et al. Lavida: A large diffusion language model for multimodal understanding. Preprint, 2025.\n\n\n$\\textcolor{blue}{\\textsf{Q2: Ask about the fixed compute budget of the discrete flow matching (DFM) framework.}}$\n\n> \"Similar to previous language diffusion models, the proposed approach operates under a fixed compute budget, which is limited for multimodal generation. For instance, a 1024×1024 pixel image translates to roughly 4096 tokens, allocate a \"reasonable\" number of generated tokens during inference phase is not straightforward e.g., 4096 tokens are generally not efficient for language generation task.\"\n\nA2: Thank you for your careful review. We acknowledge that the standard DFM framework requires specifying a fixed compute budget during inference.\nHowever, recent work—such as EditFlow [cite 1] and DreamOn [cite 2]—has begun to address this issue by introducing edit operations that enable more flexible and dynamic adjustment of output lengths in non-autoregressive frameworks. Additionally, advances in image representation, such as using fewer tokens for images [cite 3–4], offer promising directions for improving generation efficiency.\nWe consider incorporating these techniques into FUDOKI a promising avenue for future research, which could further enhance its flexibility and applicability to diverse multimodal tasks.\n\n[cite 1] Havasi, Marton, et al. Edit Flows: Flow Matching with Edit Operations. Preprint, 2025.\n\n[cite 2] Wu, Zirui, et al. DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas. Blog, 2025.\n\n[cite 3] Yu, Qihang, et al. An image is worth 32 tokens for reconstruction and generation.\" In NIPS 2024.\n\n[cite 4] Bachmann R, et al. FlexTok: Resampling Images into 1D Token Sequences of Flexible Length. In ICML 2025.\n\n\n\n$\\textcolor{blue}{\\textsf{Q3: Ask about the limitations of using separate encoders for images.}}$\n\n> \"The model use two type of image encoders (SigLIP and VAE), thus, training pipeline could be rather cumbersome. For instance, multi-turn chat with multiple images, which is not straightforward to separate the understanding and generation task. In addition, it would be difficult to extend image generation tasks beyond text-to-image e.g., (multiple) image to image generation.\"\n\nA3: Thank you for your comments. We acknowledge that employing two types of image encoders adds complexity to the training pipeline. This design choice was inherited from Janus, aiming to leverage the strengths of each encoder in their respective domains.\nIn contrast, recent studies [cite 1-2] have proposed to develop unified image representations for both understanding and generation, which could simplify the pipeline and enhance the model’s flexibility across various multimodal tasks, such as the mentioned multi-turn chat with multiple images and (multiple) image to image generation.\nWe appreciate the reviewer’s observation and will explore this in future iterations of our system.\n\n[cite 1] Qu L, et al. Tokenflow: Unified image tokenizer for multimodal understanding and generation. In CVPR 2025.\n\n[cite 2] Ma C, et al. Unitok: A unified tokenizer for visual generation and understanding. Preprint, 2025.\n\n\n\n$\\textcolor{blue}{\\textsf{Q4: Ask about the GIF visualizations of the text generation process.}}$\n\n> \"Including a GIF that visualizes the text‐generation process would add valuable insight. From my experience, absorbing‐diffusion models like LLaDA and Dream often behave largely autoregressively, so a clear demonstration of their bidirectional generation capability would be especially informative.\" \n \nA4: Thank you for your comments. \nDuring the submission phase, we included GIF visualizations of the text generation process on our project page, as referenced in the footnote on Page 1 of the paper. \nIn accordance with the NeurIPS 2025 rebuttal policy, the project page has not been updated during the rebuttal phase. \nAdditional examples can also be found in Figure 8 of the supplementary materials. Notably, these visualizations demonstrate that our generation process exhibits less autoregressive behavior, showcasing the model’s bidirectional generation capability.\n\n$\\textcolor{blue}{\\textsf{Q5: Ask about the performance of FUDOKI given different inference steps.}}$\n\n> \"It would be helpful if the author add the performance of model with different number of denoising steps.\"\n\nA5: Thank you for your comments. Please see Figure 5 (right) in the main paper, showcasing the model's performance on the GenEval benchmark with different inference steps (*i.e.*, the blue line).\nResults show that with more timesteps, FUDOKI achieved better generation performance. \n\n$\\textcolor{blue}{\\textsf{Q6:}}$\n\n> \"Are experiments run on H100s? Why don't the author fine-tune existing (larger) language diffusion baselines such as LLaDA or Dream (as I believe 43k gpu hours is quite substantial and enough for fine-tuning 7B models)? how straightforward it is to adapt an absorbing diffusion model to the kinetic diffusion?.\"\n\nA6: Thank you for your comments. Due to our computational\nconstraints, all experiments in the paper were actually conducted on NVIDIA V100 GPUs. As such, fine-tuning larger language diffusion baselines was not feasible within our resources. Regarding model adaptation, in principle, converting an absorbing diffusion model to the kinetic discrete flow matching framework does not require specific architectural modifications. We consider this a promising direction and will explore this in future work if more computational resources are available."}, {"review_id": "Q9H0B7Nro7", "reviewer": "Reviewer_Zj7H", "summary": "The paper introduces FUDOKI, a unified multimodal model for both visual understanding and image generation, built on a discrete flow matching paradigm rather than traditional autoregressive (AR) methods. This shift addresses key AR limitations, such as rigid generation order and weaker reasoning or self-correction.\n\nFUDOKI refines outputs via metric-based probability paths and kinetically optimal velocity fields, allowing for richer bidirectional context and improved generation quality. To ease training, it starts from a pre-trained AR MLLM and transitions gradually to the flow-based framework.\n\nExperiments show that FUDOKI matches or exceeds state-of-the-art AR models on tasks like visual question answering and text-to-image generation. Inference-time strategies (e.g., sampling multiple outputs) further enhance performance, highlighting the model’s flexibility and potential for continued improvement.", "questions": "1.Could the authors clarify the inference speed and step count required by FUDOKI versus an AR model? For instance, how many refinement iterations are typically needed for generation.\n\n2.Do the authors think *any* strong AR language model could be converted to a flow-based model via their adaptation strategy? It would be insightful to discuss if the success depends on specific properties of the chosen base model (Janus-1.5B), or if one could take a larger model or different architecture and apply the same paradigm.\n\n3.The paper introduces multiple novel components. It would strengthen the work to see more ablations.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.It is the first work to build a general-purpose vision–language model entirely on discrete flow matching, which advance beyond prior diffusion models relying on basic mask-based corruption. It addresses key limitations of autoregressive multimodal models.\n\n2.The paper is clearly written, with a well-structured presentation. Section 2 provides a solid overview of discrete flow matching, while Section 3 clearly explains the proposed extensions—metric-induced paths and kinetically optimal velocity fields. The distinctions from prior methods, such as mask-based diffusion versus their continuous refinement approach, are effectively illustrated.\n\n3.In terms of experiments, the paper provides thorough validation. FUDOKI is evaluated on a wide range of well-known tasks and benchmarks. \n\n\n\nWeaknesses:\n\n1.Inference speed remains unclear. While the model benefits from iterative refinement and self-correction and show quality-speed Trade-off, it’s not obvious how this translates to actual inference efficiency. There is no direct comparison with optimized AR models in terms of speed or number of steps required to reach similar output quality.\n\n2.Lack of ablations. While the paper introduces several novel components—such as metric-induced probability paths, kinetically optimal velocity fields, and test-time inference scaling—there is no systematic ablation to isolate their individual impact. For example, it would be informative to see how performance changes if a simpler mask-based path replaces the metric-induced one.\n\n3.Limited Evaluation Scope. While FUDOKI shows promising generalization ability and robustness across several visual generation and understanding tasks, its effectiveness on more challenging reasoning tasks or open-ended question answering scenarios remains underexplored. Further evaluation is needed to confirm whether its strengths extend to these more complex settings.", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns, and I would therefore keep my positive rating.", "rebuttal": "Thank you for your great efforts on the review and constructive comments. \nWe will try our best to answer all your questions.\n\n**Please let us know if you are not satisfied with our answers or have further questions, so that we can get back to you as soon as possible.**\n\n$\\textcolor{blue}{\\textsf{Q1: Ask about the inference speed comparison with the AR model.}}$\n\n> \"Inference speed remains unclear. While the model benefits from iterative refinement and self-correction and show quality-speed Trade-off, it’s not obvious how this translates to actual inference efficiency. There is no direct comparison with optimized AR models in terms of speed or number of steps required to reach similar output quality.\" \"Could the authors clarify the inference speed and step count required by FUDOKI versus an AR model? For instance, how many refinement iterations are typically needed for generation.\"\n\nA1: Thank you for your comments. Please refer to Figure 5 (right) in the main paper, which also compares the inference performance of FUDOKI with the autoregressive (AR) baseline, Janus-Pro-1B (with KV cache enabled). \nAll experiments, including this comparison, were conducted on the NVIDIA V100 GPU in this paper.\nIn the figure, the blue dashed line (to the right vertical axis) indicates the GenEval score of Janus-Pro-1B, while the red dashed line (to the left vertical axis) shows its inference speed, measured in the number of generated images per minute.\nWe kindly direct the reviewer’s attention to the intersection point of the green arrows in Figure 5 (right). This intersection marks the point where FUDOKI achieves a significant speed advantage over the AR baseline (as the red solid line exceeds the red dashed line) and comparable output quality (where the blue solid line meets the blue dashed line).\nMoreover, in practice, we find that FUDOKI typically requires 10 steps to generate plausible images, whereas the AR model, Janus-Pro-1B, requires 576 steps. This demonstrates the efficiency of our approach in terms of both inference speed and step count.\n\n$\\textcolor{blue}{\\textsf{Q2: Ask about more ablation studies of FUDOKI.}}$\n\n> \"Lack of ablations. While the paper introduces several novel components—such as metric-induced probability paths, kinetically optimal velocity fields, and test-time inference scaling—there is no systematic ablation to isolate their individual impact. For example, it would be informative to see how performance changes if a simpler mask-based path replaces the metric-induced one.\" \"The paper introduces multiple novel components. It would strengthen the work to see more ablations.\"\n\nA2: Thank you for your comments. \nDue to time and computational resource constraints, we were unable to complete the training of a mask-based FUDOKI variant during the rebuttal phase. \nNevertheless, the effectiveness of metric-induced probability paths has been systematically evaluated in its original work (*e.g.*, Table 3 in [cite 1]). \nAdditionally, FUDOKI achieves performance comparable to recent mask-based discrete diffusion MLLMs, such as MMaDA [cite 2], even at a smaller scale (8B parameters for MMaDA *vs.* 1.5B for FUDOKI), highlighting the competitiveness of our approach. \nWe appreciate the importance of detailed ablation studies and will include such analyses in future work.\n\n[cite 1] Shaul, Neta, et al. Flow matching with general discrete paths: A kinetic-optimal perspective. In ICLR 2025.\n\n[cite 2] Yang, Ling, et al. Mmada: Multimodal large diffusion language models. Preprint, 2025.\n\n$\\textcolor{blue}{\\textsf{Q3: Ask for evaluations of FUDOKI on more challenging reasoning tasks.}}$\n\n> \"Limited Evaluation Scope. While FUDOKI shows promising generalization ability and robustness across several visual generation and understanding tasks, its effectiveness on more challenging reasoning tasks or open-ended question answering scenarios remains underexplored. Further evaluation is needed to confirm whether its strengths extend to these more complex settings.\"\n\nA3: Thank you for your comments. We have followed your suggestions and conducted new experiments, where we evaluated our proposed FUDOKI on a more challenging mathematical reasoning benchmark, MathVista (testmini) [cite 1].\nAs shown in Table A, we find that FUDOKI achieved the best performance compared to AR-based models at the same scale.\nWe attribute this improvement to FUDOKI's discrete flow matching framework, which leverages bidirectional context modeling to facilitate complex reasoning.\n\n**Table A: Performance comparisons on the MathVista benchmark.**\n\n| Model | Janus-1.5B | Janus-Pro-1B| FUDOKI |\n|:-----|:-------:|:------:|:-----:|\n|  MathVista  |  32.4 | 35.1 | **38.6** |\n\n[cite 1] Lu, Pan, et al. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.\" In ICLR 2024.\n\n$\\textcolor{blue}{\\textsf{Q4: Discussions on the base model of FUDOKI.}}$\n\n> \"Do the authors think any strong AR language model could be converted to a flow-based model via their adaptation strategy? It would be insightful to discuss if the success depends on specific properties of the chosen base model (Janus-1.5B), or if one could take a larger model or different architecture and apply the same paradigm.\"\n\nA4: Thank you for your comments. We have followed your suggestion and conducted new experiments, using another AR model, Qwen2-VL-2B-Instruct, as the base model. \nHowever, we observed that it converged significantly more slowly on the image generation task compared to Janus-1.5B. \nThis observation suggests that the original AR model’s integrated capabilities for both understanding and generation are important for the success of the adaptation strategy."}, {"review_id": "4npReVIOU7", "reviewer": "Reviewer_sZnw", "summary": "This paper presents FUDOKI, the first unified multimodal generative and understanding model based entirely on discrete flow matching (DFM). The model supports both image and text modalities, featuring a mask-free, bidirectional, and dynamically revisable generation process. Experiments demonstrate that FUDOKI matches or surpasses state-of-the-art autoregressive MLLMs in visual understanding and image generation benchmarks.", "questions": "The paper does not compare FUDOKI with recent strong unified models like Bagel [1]; could the authors briefly discuss FUDOKI's relative strengths or weaknesses compared to it?\n[1] Unified Model for Multimodal Understanding and Generation", "limitations": "The paper primarily evaluates standard single-image tasks. Broader validation on diverse, practical, or multi-image scenarios, and deeper discussion of societal impacts, would strengthen the work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n- Technically sound and well-motivated methodology.\n- Unified framework for multimodal generation and understanding.\n- Supports dynamic token revision during generation, improving flexibility.\n- Strong empirical results on standard benchmarks.\n- Novel application of discrete flow matching with kinetic-optimal velocity.\n\n**Weaknesses**\n- The paper does not provide experimental results or analysis for more complex multimodal scenarios, such as multi-image conditioning or compositional tasks.\n- No comparison is made with recent strong unified AR+DiT models like Bagel, leaving it unclear how FUDOKI performs against state-of-the-art baselines on unified generation and understanding.", "was_revised": true, "final_justification": "Authors have resolved most of my concerns. I will remain my rating.", "rebuttal": "Thank you for your great efforts on the review and constructive comments. \nWe will try our best to answer all your questions.\n\n**Please let us know if you are not satisfied with our answers or have further questions, so that we can get back to you as soon as possible.**\n\n$\\textcolor{blue}{\\textsf{Q1: Ask about analysis of FUDOKI on more complex multimodal scenarios and its societal impacts.}}$\n\n\n> \"The paper does not provide experimental results or analysis for more complex multimodal scenarios, such as multi-image conditioning or compositional tasks.\" \"The paper primarily evaluates standard single-image tasks. Broader validation on diverse, practical, or multi-image scenarios, and deeper discussion of societal impacts, would strengthen the work.\"\n\nA1: Thank you for your comments.\nAs an initial step, we conducted preliminary experiments on interleaved image-text generation, *i.e.*, Maze Navigation, which involves one input image for understanding and one output image for generation. The results are presented in Figures 9–10 of the supplementary materials.\nHowever, due to computational constraints—specifically, all experiments in this work were performed on NVIDIA V100 GPUs—we were unable to extend training to more complex multimodal scenarios involving multiple images at a large scale. We recognize the importance of broader validation on such tasks and plan to explore these directions in future work if more computational resources are available.\n\nRegarding societal impacts, we agree that a deeper discussion is important. \nAs multimodal models become more capable, their potential applications—ranging from assistive technology to content creation—grow, but they also bring risks of misuse, bias amplification, and challenges in content authenticity verification. While our work focuses on advancing technical foundations, we recognize the importance of continually assessing these broader impacts and will include such discussions in our revised paper.\n\n\n$\\textcolor{blue}{\\textsf{Q2: Ask for comparisons with recent strong unified models, such as Bagel [cite 1].}}$\n\n\n> \"No comparison is made with recent strong unified AR+DiT models like Bagel, leaving it unclear how FUDOKI performs against state-of-the-art baselines on unified generation and understanding.\" \"The paper does not compare FUDOKI with recent strong unified models like Bagel [1]; could the authors briefly discuss FUDOKI's relative strengths or weaknesses compared to it? [1] Unified Model for Multimodal Understanding and Generation\"\n\nA2: Thank you for your comments.\nBagel [cite 1] is a very strong recent advance in unified multimodal understanding and generation.\nWhile both FUDOKI and Bagel aim for unified multimodal modeling, they are based on fundamentally different generative paradigms and architectural choices.\nSpecifically, Bagel employs a large Mixture-of-Transformer-Experts (MoT) architecture and follows the **autoregressive (AR) modeling** paradigm, enabling it to efficiently scale with massive, carefully structured interleaved multimodal data. \nIn contrast, FUDOKI is the first general-purpose unified multimodal model built entirely on **discrete flow matching**, which allows for bidirectional information integration and iterative self-correction during generation.\nIn terms of empirical performance, Bagel demonstrates strong results on both multimodal generation and understanding, including advanced tasks such as free-form image manipulation. \nWe acknowledge that FUDOKI currently lags behind Bagel, which can be attributed mainly to Bagel’s novel data scaling strategies and substantially larger model size (14B parameters for Bagel *vs.* 1.5B for FUDOKI).\nWe appreciate the reviewer bringing this to our attention and will include a more detailed discussion and citation of Bagel in our revised paper. \nWe will also explore integrating similar scaling approaches in future work if more computational resources are available.\n\n\n[cite 1] Deng C, et al. Emerging properties in unified multimodal pertaining. Preprint, 2025."}, {"review_id": "Et0BsBO6kr", "reviewer": "Reviewer_vmHC", "summary": "The paper introduces FUDOKI, the first unified vision-language model that replaces the standard autoregressive (AR) decoder with Discrete Flow Matching (DFM) decoding. Tokens are noised along a metric-induced probability path in embedding space, and a kinetic-optimal velocity field is learned so the model iteratively transports the corrupted sequence toward the target, allowing bidirectional attention and step-wise self-correction. Trained on Jauns-1B, FUDOKI achieves performance comparable to SOTA AR-based MLLMs across both visual understanding and image generation tasks,", "questions": "The questions are the same as the weaknesses we discussed above. My main concern is whether the motivation for introducing DFM is sufficiently justified. While the self-correction capability is certainly appealing, the paper lacks quantitative experiments to demonstrate the effectiveness of this capability within the unified model. Addressing this issue would  improve my rating of the paper.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. This paper presents the first unified multimodal generation model based on Discrete Flow Matching, introducing DFM techniques into this domain.\n2. The figures in this paper are clear and effectively illustrate the differences between the proposed method and existing approaches.\n\n**Weaknesses**\n1. The novelty of this paper is somewhat limited. The self-correction capability is an inherent property of DFM models, and while the paper applies it to a unified model, it does not demonstrate any unique advantages in this setting. In my view, this reflects a lack of sufficient motivation.\n2. The results presented in this paper are also not particularly strong. Despite initializing from Janus-1B weights, the model undergoes an additional 43,000 GPU hours of training—roughly equivalent to training a 7B autoregressive MLLM from scratch. However, the final performance only matches that of existing AR-based models, and on many benchmarks, the results are not especially competitive.\n3. The main motivation of this paper lies in the self-correction capability of DFM. However, this is only demonstrated through a few qualitative visualizations. While these are intuitive, they lack statistical rigor by academic review standards. The paper does not provide quantitative experiments to convincingly show that self-correction is truly effective and important for MMU tasks.\n4. The Related Work section of this paper lacks citations to some relevant papers, such as Muse-vl, LlamaFusion and OmniMamba. Including references to these papers would help make this section more comprehensive.", "was_revised": true, "final_justification": "The author's new experiment, a quantitative comparison of the self-correction capabilities of AR and FUDOKI, resolved my biggest concern, so I have raised my score for this paper to 4.", "rebuttal": "Thank you for your great efforts on the review and constructive comments. \nWe will try our best to answer all your questions.\n\n**Please let us know if you are not satisfied with our answers or have further questions, so that we can get back to you as soon as possible.**\n\n$\\textcolor{blue}{\\textsf{Q1: About motivations of FUDOKI.}}$\n\n\n> \"The novelty of this paper is somewhat limited. The self-correction capability is an inherent property of DFM models, and while the paper applies it to a unified model, it does not demonstrate any unique advantages in this setting. In my view, this reflects a lack of sufficient motivation.\" \n\nA1: Thank you for your comments. While self-correction is indeed an inherent property of DFM models, we would like to clarify that the motivation behind FUDOKI extends beyond this single aspect. As stated in our introduction, our primary motivation is to **challenge the dominance of the prevailing autoregressive (AR) paradigm** in unified multimodal modeling.\nTo this end, FUDOKI is the first general-purpose unified multimodal model built entirely on **discrete flow matching**, providing several unique advantages over AR approaches that could benefit multimodal tasks, such as **bidirectional information integration** and **parallel, iterative inference**.\nWhile **self-correction** is also one consequence of adopting discrete flow matching, our main motivation is to explore **a new, flexible and generalizable modeling paradigm** for unified multimodal models.\nWe appreciate the reviewer’s suggestion and will further clarify these discussions in the revised manuscript.\n\n\n$\\textcolor{blue}{\\textsf{Q2: Ask for quantitative experiments for evaluating the self-correction capability of FUDOKI.}}$\n\n\n> \"The main motivation of this paper lies in the self-correction capability of DFM. However, this is only demonstrated through a few qualitative visualizations. While these are intuitive, they lack statistical rigor by academic review standards. The paper does not provide quantitative experiments to convincingly show that self-correction is truly effective and important for MMU tasks.\" \"My main concern is whether the motivation for introducing DFM is sufficiently justified. While the self-correction capability is certainly appealing, the paper lacks quantitative experiments to demonstrate the effectiveness of this capability within the unified model.\"\n\nA2: Thank you for your comments. We have followed your suggestions and conducted new experiments, where we quantitatively evaluated the self-correcting capabilities of our proposed FUDOKI and performed comparisons with the AR-based models.\nIn experiments, both FUDOKI and AR-based models were tasked with correcting baseline responses where necessary. The baseline responses were obtained from Janus-Pro-1B on the MMVet benchmark, using the OpenCompass VLMEvalKit codebase [cite 1]. To assess their correction abilities,\n\n- For AR-based models, we appended the following prompt to the original prompt: \"Your original response is: \\<placeholder\\>. Please correct it if needed. Otherwise, you may keep it the same.\" The models were then evaluated on their ability to revise or retain the response as appropriate.\n- For FUDOKI, we initialized the responses with the baseline responses (rather than uniformly-sampled noise tokens) and performed iterative refinements over 32 steps, as described in the paper.\n\nAs shown in Table A, FUDOKI achieved the highest performance improvement, while Janus-Pro-1B’s performance declined and Janus-Pro-7B showed less increase, despite its larger model size than ours.\nWe attribute such results to the increased context length introduced by the baseline responses, which may distract the AR-based model’s focus. \nThis further highlights the limitations of the AR paradigm for effective self-correction.\n\n**Table A: Quantitative comparisons between the AR-based models and our proposed FUDOKI in terms of the self-correcting capabilities.**\n\n| Model | Baseline |+Janus-Pro-1B to correct |+Janus-Pro-7B to correct | +FUDOKI to correct |\n|:-----|:-------:|:------:|:-----:|:-----:|\n|  MMVet  |  37.98 | 36.33 (-1.65) | 38.30 (+0.32)|**38.53 (+0.55)**|\n\n\n[cite 1] Duan, Haodong et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In ACMMM 2024.\n\n\n\n$\\textcolor{blue}{\\textsf{Q3: About the results and training costs.}}$\n\n\n> \"The results presented in this paper are also not particularly strong. Despite initializing from Janus-1B weights, the model undergoes an additional 43,000 GPU hours of training—roughly equivalent to training a 7B autoregressive MLLM from scratch. However, the final performance only matches that of existing AR-based models, and on many benchmarks, the results are not especially competitive.\"\n\nA3: Thank you for your comments. Due to computational constraints, all experiments in our paper were actually conducted on NVIDIA V100 GPUs. \nAs such, training large-scale 7B autoregressive MLLM baselines was not feasible within our resources.\nHowever, we would like to emphasize that our primary goal was to demonstrate the feasibility and potential of discrete flow matching (DFM) in multimodal modeling, rather than to surpass all existing AR-based models. While our current results are on par with AR-based models, we believe this is promising given our smaller model size and limited compute budget.\nWe expect that, with larger models, more extensive training, and improved scaling strategies—such as those used in recent AR-based models like BAGEL [cite 1]—the advantages of DFM-based unified models, including inherent bidirectional reasoning and self-correction, will become more evident.\nWe appreciate the reviewer’s feedback and will clarify these points in the revised manuscript. In future work, we aim to further explore scaling up DFM models if computational resources permit.\n\n[cite 1] Deng C, et al. Emerging properties in unified multimodal pertaining. Preprint, 2025.\n\n\n$\\textcolor{blue}{\\textsf{Q4: Relevant studies are missing.}}$\n\n\n> \"The Related Work section of this paper lacks citations to some relevant papers, such as Muse-vl, LlamaFusion and OmniMamba. Including references to these papers would help make this section more comprehensive.\"\n\nA4: Thank you for your feedback. We will cite and discuss all these works to make the Related Work section more comprehensive in our revised paper."}]

Author rebuttal:

Meta-review: This paper introduces FUDOKI, the first unified multimodal model built entirely on discrete flow matching (DFM). Unlike autoregressive (AR) paradigms, FUDOKI enables bidirectional context modeling and iterative refinement, which is shown to improve reasoning and self-correction capabilities.

All reviewers leaned positively, praising the paper’s clarity, solid technical presentation, and strong empirical results. In particular, the rebuttal strengthened confidence by adding experiments on MathVista and MMVet, which demonstrated improved reasoning ability and self-correction—two aspects highly relevant to the field of multimodal large language models (MLLMs).

At the same time, reviewers noted certain limitations. The novelty is incremental in that FUDOKI extends prior DFM concepts rather than introducing an entirely new paradigm. The computational cost of training is significant, which may limit accessibility and deployment. Moreover, broader ablations and comparisons are missing, for instance against models such as Bagel, which would help position the contribution more clearly.

Overall, I agree with the reviewers that FUDOKI makes an original and timely contribution, showing convincingly that DFM can serve as a viable alternative to AR-based MLLMs. While there are concerns about efficiency, scope of evaluation, and related work coverage, these are issues the authors should address in a revision. On balance, the combination of a novel unified formulation, clear empirical strengths, and demonstrated reasoning improvements merits acceptance as a Spotlight.
Decision: Accept (spotlight)