LaViDa: A Large Diffusion Language Model for Multimodal Understanding
Authors: Shufan Li
Konstantinos Kallidromitis
Hritik Bansal
Akash Gokul
Yusuke Kato
Kazuki Kozuka
Jason Kuen
Zhe Lin
Kai-Wei Chang
Aditya Grover
Venue: NeurIPS 2025 spotlight ()
Keywords: multi-modality
vision-language models
large language models
diffusion language models
Primary area: deep_learning

Abstract: Modern Vision-Language Models (VLMs) can solve a wide range of tasks requiring visual reasoning. In real-world scenarios, desirable properties for VLMs include fast inference and controllable generation (e.g., constraining outputs to adhere to a desired format). However, existing autoregressive (AR) VLMs like LLaVA struggle in these aspects. Discrete diffusion models (DMs) offer a promising alternative, enabling parallel decoding for faster inference and bidirectional context for controllable generation through text-infilling. While effective in language-only settings, DMs' potential for multimodal tasks is underexplored. We introduce LaViDa, a family of VLMs built on DMs. We build LaViDa by equipping DMs with a vision encoder and jointly fine-tune the combined parts for multimodal instruction following.  To address challenges encountered, LaViDa incorporates novel techniques such as complementary masking for effective training, prefix KV cache for efficient inference, and timestep shifting for high-quality sampling. Experiments show that LaViDa  achieves competitive or superior performance to AR VLMs on multi-modal benchmarks such as MMMU, while offering unique advantages of DMs, including flexible speed-quality tradeoff, controllability, and bidirectional reasoning. On COCO captioning, LaViDa surpasses Open-LLaVa-Next-8B by +4.1 CIDEr with 1.92x speedup. On bidirectional tasks, it achieves +59% improvement on Constrained Poem Completion. These results demonstrate LaViDa as a strong alternative to AR VLMs. Code and models is available at https://github.com/jacklishufan/LaViDa

LaViDa: A Large Diffusion Language Model for Multimodal Understanding
Shufan Li Konstantinos Kallidromitis", Hritik Bansal " Akash Gokul Yusuke Kato²
Kazuki Kozuka", Jason Kuen", Zhe Lin", Kai-Wei Chang', Aditya Grover!
UCLA PPanasonic Al Research 3 Adobe Research *Salesforce Research * Equal Contribution
Abstract
Modern Vision-Language Models (VLMs) can solve a wide range of tasks requiring visual reasoning. In real-world scenarios, desirable properties for VLMs include fast inference and controllable generation (e.g., constraining outputs to adhere desired format). However, existing autoregressive (AR) VLMSs like LLLaV struggle in these aspects. Discrete diffusion models (DMs) offer a promising alternative, enabling parallel decoding for faster inference and bidirectional context for controllable generation through text-infilling. While effective in language-only settings, DMs potential for multimodal tasks is underexplored. We introduce LaViDa, a family of VLMs built on DMs. We build LaViDa by equipping DMs with vision encoder and jointly fine-tune the combined parts for multimodal instruction following, To address challenges encountered, VaViDa incorporates novel techniques such as complementary masking for effective training, prefix KV cache for efficient inference, and timestep shifting for high-quality sampling. Experiments show that LaViDa achieves competitive or superior performance AR VLMs multi-modal benchmarks such as MMMU, while offering unique advantages of DMs, including flexible bidirectional reasoning. On COCO captioning, LaViDa surpasses Open-LLaVe Va- Next-Llama3-8B by +4.1 CIDEr with 1.92 speedup. On bidirectional tasks, it achieves +59% improvement on Constrained Poem Completion. These results demonstrate La ViDa as a strong alternative to AR VLMs. Code and models available https////sshbbbooo//ccccsssaa/aa
tradeoff, controllability. and
LaViDa-8B (Ours)
LLaVA-1.6-7 6-78
OpeM--exx-LLMMM3-8
Figure 1: We propose LaViDa, the first family of diffusion- based discrete VLM models. LaViDa models achieve com- petitive performance against AR baselines (LLaVa-1.66 Open- I.aa Va-Nex)) across multiple visual understanding tasks includ- ing MMMU (world knowledge), MathVista (reasoning), ChartQA (OCR), and ScienceQA (science).
80
78.3
73.2
Performance (%))
64.6
60
43.3
44.8
40
37.4
35.1
34.4
20
0
MMMU
MathVista
ChartQA ScienceOA
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Vision-Language Models (VLMs) have shown remarkable utility across diverse domains, from end- user applications like virtual assistants (37. to research tasks such as scientific image captioning and document understanding [46
T6
5
In industrial settings, VLMs support automated
product tagging, content moderation, and quality control in manufacturing [69 2 Their ability to jointly process visual and textual information makes them indispensable for practical applications and sutting-edge research, Currently, nearly all popular VLMs-such as Qwen- VL [S 75 Intern-VL 17 85,, and GPT-4 (58) are built on top of large language models (LLMs) that generate text in an autoregressive (AR) manner, that is, they produce tokens one by one in left-to-right sequence.
While these models have demonstrated strong performance on many tasks, they suffer from several key limitations. First, their sequential generation process is inherently hard to parallelize, resulting in slow inference speed [8] More critically, their left-to-right generation makes difficult handle tasks that benefit from bidirectional context or structural constraints- example, generating poem where each line starts with a specific syllable, or extracting structured information from an image in predefined JSON format, often requires the model to fill in or coordinate content across the sequence. Autoregressive models still struggle to consistently satisfy such constraints, even with carefully crafted prompts and examples.
as text infilling [22]. For
Recently, discrete diffusion models (DMs) have emerged a promising alternative AR LLMs. Most notably, LLaDA (57] and Dream [79] achieved comparable results to AR LLMs across diverse language tasks. Unlike AR LLMs. DMs treat text generation as diffusion process over discrete tokens. A forward process gradually corrupts a sequence of discrete text tokens into a sequence of mask tokens. At inference, we start with a sequence of mask tokens and gradually transform them into a sequence of meaningful text tokens through a learned reverse process.
Compared to AR LLMs, diffusion models offer several theoretical advantages that directly address the limitations of autoregressive generation. While AR LLMs have a fixed throughput- -ggneratngg one token a time-DMs allow flexible control over the speed-quality trade-off by adjusting the number of diffusion steps [79 64 49 57). Moreover, their ability to model bidirectional context makes them well-suited for tasks like text infilling, enabling more effective constrained generation and structured output formatting-capablliic especially valuable in vision-language settings where outputs may need to follow specific schemas.
In this work, we propose LaViDa (Large Vision-Language Diffusion Model with Masking), the first family of VLMs based on diffusion. LaViDa enables pretrained DMs to perceive visual inputs by integrating vision features into the diffusion backbone via a vision encoder aanoddrr-naalouus to how LLaVA 46 44 augments large language models (LLMs) with visual inputs. Specifically, adopt two-stage training pipeline with a diffusion objective: pretraining followed by supervised fine-tuning.
Adapting DMs for vision-language tasks presents several practical challenges. First, standard DM training is data-ineffiiient: the model learns only from the subset of corrupted tokens at each timestep. For example, in corruption process may mask "The [mask] is dog", leaving the key token "dog" unmasked and thus excluded from the loss. This is especially problematic for multimodal tasks, where answer tokens often carry crucial semantic content grounded in the image [61. To address this, we introduce complementary masking scheme that ensures every token in the output sequence contributes to learning, improving data efficiency.
annweerng task where the target answer is 'The answer is dog", the
Second, inference algorithms used by existing DMs are slow in practice due to the lack of KV cache support-an an inherent limitation of bidirectional context modeling [3]. This leads to repeated recomputation over the full prompt every decoding step. While tolerable for short-text settings, it becomes a significant bottleneck in vision-language tasks, where multimodal prompts may include hundreds of visual tokens. To address this, we propose Prefix-DLM decoding, a simple yet effective technique that enables caching of multimodal prompts (i.e., image and question tokens), significantly accelerating inference.
Lastly, DMs offer a unique advantage over autoregressive models: the ability to trade off speed and quality by varying the number of diffusion steps. However, the widely used linear masking schedule-whie unmasks a fixed number of tokens per tep--perfrmss poorly at low step counts. Motivated by text-to-image diffusion models like SD3 [11, adopt timestep shifting strategy that

---

adaptively adjusts how many tokens are unmasked per step. This leads to better sample quality under aggressive step reduction, allowing faster generation without large degradation in quality.
We conducted extensive evaluations of LaViDa across a wide range of vision-language tasks. Results show that LaVil achieves competitive performance on most benchmarks, including MMMU [0]. Math Vista [5 ChartQA [53] and ScienceQA [52), when compared with AR VLMs like LLaVe Va-
1.6-7B
[15], We highlight these results in Figure I On constrained generation tasks, LaViDa greatly outperforms AR baselines (+59% on Poem Completion).
45 43 and Dpee-LLaVa-Neex--Llaee
It also supports flexible speed-qualig, tradeoff, achieving higher quality on COCO image captioning (+ 4.1 CIDEr) and 1.92x speedup (42. In summary, our contributions are:
We introduce La the first family of VLMs based on diffusion models. Our models achieve competitive performance across wide range of tasks compared to AR VLMs, while offering the unique benefits of DMs.
We introduce several novel training and inference techniques for DMs, including comple- mentary masking, Prefix-DLM, and timestep shifting that improve the training efficiency, inference speed, and sample quality of LaViDa.
We conduct a systematic study of various design choices for adapting DMs to vision- language tasks (e.g. image resolution), offering insights for future work in this direction.
2 Background and Related Works
2.1 Vision-Language Models
Vision-Language Models (VLM) extend the capability of Large Language Models to visual under- standing tasks [36] 46
85 58) The common recipe build a VLM is start with a strong large language model and combine it with a vision encoder 46 75 These VLMs typically undergo multiple stages of training, which can be generally categorized into pretraining and finetuning stages. The pretraining data usually consists of text-image pairs for vision-languee alignment, while the finetuning data consists of a wide range of cntrructionfollowiing tasks. There are several dedicated lines of work focusing on different components of this overarching framework, such as the design of To this date, most vision- laiigal models such as LLaVa 46 36 and Qwen-VL. 55 75 series employ an autoregressive training objective.
75
5
vision encoders 266 77
and training techniques 41 72
2.2 Diffusion Language Models
Diffusion models first emerged as a powerful alternative to GANs for generating continuous data such images 62 59 21. Early cxplorations in diffusion language models directly built continuous
diffusion models for latent text embeddings [39 50), with limited success. More recently, discrete diffusion models 4 64 4 have proven be better candidates for language modeling, achieving performance comparable to AR models while offering unique advantages, such as speed-quality tradeoff's and controllability. Most notably, LLaDa-8B and Dream-8B [57] 79 demonstrated that DLMs can achieve competitive performance against AR LLMs at scale.
Formally, given text sequence of L tokens Xo [X] X3... the forward process (X,, gradually converts it to a sequence full of mask tokens "[M]". denoted by X1 = X through the continuous time-interval [o, 1]. with 1 2 t 2 2 O. A neural network pe is used to model the reverse process p(X,, (X.) The diffusion language modeling objective can be defined as:
[ log Po (Xo,
CPLM = -E,,x.,x,
where log (Xo (X,) is assumed to be factorized into I Po Xx) At each training step, we sample uniformly from the interval [o, 1] and sample Xu from some data distribution D. X, is then The loss is only computed over the masked tokens in X,, since pa( (X:)X.) has a closed form representation that does not depend on 0 when X; + [M]. We offer additional background on the details of these formulations in Appendix A.T We also incorporate addition backgrounds on other relevant literature, such as masked generative models 10 II and multimodal diffusion models 688 38), in Appendix C
sampled from the forward process g[xx|
3

---

lovely
grass
[AD]
Prediction
Diffusion Language Model
[Projected Embeddingg]
What is
in
the image
There
is
[M]
dog
on
the
[M]
[M]
[PAD]
[M]
Prompt
Noised Answer X
Vision Encoder
Crop
MLP
Rosize
2
Input Image
Multiple Views
Vision Embeddings
Is Ja....l
Vi.Va...Vs
Figure 2: Overall design of La ViDa. LaViDa's architecture consists of vision encoder, diffusion language model, and an MLP vision projector. The bottom half of the figure illustrates the image encoding process, while the top half depicts the diffusion language modeling process. These two pipelines are described in detail in Sec. 3.1
Method
3.1 Model Architecture
LaViDa's model architecture follows a similar design to common AR VLMs like LLaVa [36]. consists of a vision encoder and diffusion language model. These two parts are connected by MLP projection network. The overall design illustrated Figure
Vision Encoder. Given an input image I and text prompt P. we first resize the image to 7682 and divide it into four non-overlapping views of 3842. denoted 41-1- We also resize the original image These five views are 82 each producing 272 embeddings,
to 384 to obtain fifth view, Is. following the design of prior works 36 45
independently encoded by the vision encoder (SigLIP-400MM denoted Vi.s. In total, this yields 3645 embeddings per image. To reduce sequence length for efficient training, we apply 2 x average pooling on each view, reducing embeddings 142 per view, or 980 total. The embeddings of five views are then flattened and concatenated into a ID sequence before being processed by the projection network to obtain the final visual context of the diffusion language model. This process mirrors the vision encoding process in AR LLMs [26) and is illustrated in the bottom part Figure
Diffusion Language Model. The diffusion language model is a multi- Transformer 7 whose architecture resembles that of LLMs. The only major difference that its attention mask is non- causal, and it uses the diffusion language modeling objective described in Section 2.2 instead of the next-token prediction used in AR models. The input the diffusion language model consists of the projected vision embeddings, the prompt P, and partially masked response X,, The outputs of the last transformer block are passed through final linear layer to obtain token-wise logits po( Po(Xi7, X,) for the unmasked response Xo. In our experiments, we explored LLaDA-8B (default) and Dream-7B as our diffusion language model. This process illustrated the upper half of Figure
3.2 Training Algorithms
Our training objective is based on the diffusion language modeling objective described Section Each training sample consists of image /, text prompt P and clean text answer X, from the training data. For multi-round conversation, we sample one round as the "answer" and treat the history "prompt". We first sample a timestep E [0, 1] and partially masked answer X, using the forward process described in Section 2.2 ViViDa then implements the conditioned reverse process Po Xt). The canonical diffusion lision-language modeling objective formulated as:
2.2
CDVLM = -E,,I.PX...
logpe(Xol44 P.X,)
(2)

---

Img1
DLM
Img2
Pmtl
P1
M
Pmt2
Jkg
Dopy
Ansl
DLM
Ans3
Img1
Img2
Pmt11
Pmt22
Ansl
[M]
Ans3
(a) Complementary Masking
(b) Attention Mask of Prefix-DLM
Figure 3: Technical Details of ViDa. (a) We propose Complementary Masking to ensure loss is calculated over all tokens in the data for training efficiency. (b) We propose Prefix-DLM attention mask that enables KV caching. We visualize the attention mask of image tokens 1-2, prompt tokens (2mt and text and mask tokens in the noise answer X₃ (Ans1, [M], Ans3). Rows represent queries, while columns are keys. Colored squares indicate that queries and keys can interact.
where Po (o(lo P, X,) factorizes into IT Pa P,, following the formulation in Section 2.2 Notably, the loss is only computed over mask tokens where x; [M]. because Po(X) P, X,) dependent on 0 when X * [M].
Complementary Masking. Prior diffusion language models (c.g., LLaDa, Dream) apply stochastic estimator to Equation masking tokens independently across samples in a batch. However, for vision- language tasks, this leads to inefficiencies: (1) only ~50% of tokens contribute to the loss on average, and (2) critical answer lokens-often short and sparse in vision tasks like VQA-ma may not be masked, resulting in misaligned gradients for the vision encoder. For example, "The answer is dog." the key token "dog" might be unmasked in X, and thus ignored in loss computation. To address this, we introduce complementary masking: for each sample, we generate two masked versions X, and X with disjoint corrupted spans (e.g., masks "The [M] [M] dog ". the other [M] answer is [M] [M] ", ensuring all tokens are eventually used in training and improving sample efficiency and gradient flow. When computing the loss over X, and X we copy the encoded vision
embeddings to further boost training efficiency. This process is illustrated in Figure 3a
3.3 Inference Algorithms
At inference time, we first create a sequence of mask tokens as X1. where the response generation length. Then gradually unmask them through K discrete timestamps li-4k, where 4 = and l = 0, until we reach a clean, mask-free sequence Xo. At each timestamp 4, we sample fully unmasked sequence through Po po X.,) and re-mask f++1 tokens to obtain X Both and are hyperparameters for inference. Additionally, we define as "fraction the number of functional evaluations (NFE)" to measure sample efficiency. For example, when NFE 100%, the diffusion model generates one token per forward pass; NFE 50%, generates an average two tokens per forward pass. Overall, the inference process of La is similar to prior DMs such as LLaDa, with two key exceptions:
Frefix-DLM While DLMs theoretically offer superior speed-quality tradeoff's at inference, they are often slower than AR models practice because they cannot leverage KV caching (57). This issue is particularly evident for multimodal prompts containing many visual tokens. To avoid recomputing keys and values for the visual embeddings and text prompts, we propose novel Prefix-DLM scheme inspired by the autoregressive -M. Prefix-Dl DLM adopts a specialized attention mask in which visual and prompt tokens can only attend to other visual and prompt tokens, while answer tokens can attend to all tokens. Figure 3b illustrates this setup. With this design, we can cache the keys and values of the visual and prompt tokens. Empirically, this leads to a speedup of up to 3.9 on COCO captioning tasks. Further details provided in Section 4.5
5

---

Schedule Shift. Diffusion language models (DLMs) allow trading speed for quality via the number of discretization steps K. Prior models like LLaDa and Dream use linear schedule, unmasking & tokens uniformly over E [0, 1]. However, we find this leads to performance degradation at low K
sampling steps. Inspired by SD3 [21] we adopt shifting schedule:
ali
8a(1)))) 1+ (a- 1)t,
Here, (1) is monotonic map with = = 0, lx = 1. When a < 1 (we use a 5) the schedule is convex-leading to more tokens being unmasked earlier. We found that this setup outperforms alternatives. Notably, this conclusion differs from that of continuous diffusion models like SD3 and previous masked diffusion models for image generation [T]] which showed concave schedules > 1) are more preferable. We ensure at least one token is unmasked per step. Further details are provided in Section 4 and Appendix A.2
4 Experiments
4.1 Setup
At a high level, LaViDa employs two-stage training process. In the pretraining phase (stage- 1). only the projector is updated align the visual embeddings with the latent space of the DLM. In the finetuning phase (stage-2), we jointly train all components end-to-end for instruction foolionn-fllowingg. Additionally, we further finetune the stage-2 model for additional steps and obtain two specialized models for reasoning and text-infilling tasks details of these specialized models in Section 4.3 and [4.4] We use 558K image-text pairs as our stage-1 data, and IM visual instruction-followw examples our stage-2 data. Further details on the dataset and training setup are provided in Appendix B
and La LaViDa-FIM), We provide more
We evaluate Va on a wide range of vision-language tasks. Unless otherwise stated, we report re- sults obtained using the stage-2 model with LLaDa-8B as the language backbone. We use 1mms mmms-eve eval package (83) for evaluation and set the sequence length L to be the maximum generation length used for evaluating AR models. We set K L. or NFE-100% by default. Results under differet NFE are explored in Section 4.5 and Appendix
B.3 and B.4
4.2 Main Results
Table reports the results of LaViDa using LLaDA-8B (LaViDa-L) and Dream-7B (LaViDa-D) as the language backbones on vision-une sonoonnnndrtttnningg tasks. We compare with several open-source, open-data models with similar data sizes and parameter counts: LLa [45 43 and Open- LLaVa-Next-Llama>3
[15]. We also include comparisons with frontier open-sourced models of similar size that are trained on larger datasets, namely -naaa--neViiion-7777 [5. and Intern VL-38B 85)
36 Qwen2.5-VL-77
La ViDa demonstrates competitive performance across a wide range of tasks spanning General, Reasoning, OCR, and Science categories. In general vision-language understanding, LL achieves the highest score MMMU 80 (43.3), outperforming all comparable models. VaDia D also ranks second on several benchmarks in this category. For reasoning tasks, both models surpass similarly scaled baselines on math-heavy and spatially grounded benchmarks. In Science, LaViDa achieves the best and second-best scores on ScienceQA (52) (81.4 and 80.2, respectively) while performing on par with Open-Llava-Next on A12D [2). complex diagram- based benchmark. Finally, in OCR La shows competitive performance but lags behind some of the latest AR models. This gap is primarily due our use of average pooling for vision token compression, which leads to the loss of fine- fine-grained spatial information. While this was a necessary trade-off off given our limited compute budget, it poses challenges for tasks requiring precise text recognition and layout understanding. These results highlight the strength of LaViDa, demonstrating that fefffusioo-basee approaches can scale competitively with AR models while achieving robust performance across wide range of vision- tasks.

---

Table 1: LaViDa's Performance on Visual Understanding Tasks. We report results on General, Reasoning, OCR, and Science Benchmarks. Dashes (-) denote results not reported. Open-Lnxt:
Open-LLaVA-Nex--lamma L-OV: LLaVA-OneVision- Qwen2.5: Qwen2.5-VLL-7B; Intern3: V13-MB
LaviDa-L LaViDa-D LLaV6-1.6 Open-L.xtt
L-OV
Qwen2.5
Intern3
#Params
7B
7B
8B
#lmages
0.6M
0.6M
0.6M
0.6M
0.6M
>7M
(Pretrain)
#Images
1.0M
1.0M
0.7M
1.0M
7.2M
22
21.7M
(SFT)
General
MME-P 23
1365.6
1463.5
1519.3
1610.9
1580.0
VQAv2
72.2
75.2
80.1
71.9
MMBench
47
70.5
73.8
54.6
74.4
80.8
83.5
83.4
MMMU 80
43.3
42.6
35.1
37.4
48.8
58.6
65.6
Reasoning
MME-C 23
341.1
378.9
322.5
336.8
418.0
MathVista 51
44.8
42.1
34.4
41.8
63.2
68.2
75.2
Math Verse
27.2
24.1
14.3
14.6
MathVision 74
20.4
19.4
12.8
14.1
Science
ScienceQA 52
80.2
81.4
73.2
78.3
96.0
A12D 32
70.0
69.0
66.6
70.2
81.4
83.9
85,2
OCR
TextVQA 65
56.3
57.1
64.9
61.7
84.9
80.2
VOA
55
59.0
56.1
74.4
69.9
87.5
95.7
92.7
ChartQA 53
64.6
61.0
54.8
69.7
80.0
87.3
86.6
InfoVQA
54
34.2
36.2
37.1
36.7
68.8
82.6
76.8
Table 2: Performance of Specialized stage-3 Models. We report (a) performance of LaViDa-Reason on math reasoning tasks and (b) performance of LaViDa-FIM on constrained poem completion task.
(a) Math benchmark results after long CoT distillation.
(b) Sentence and sample-level constraint satisfac- tion for poem completion tasks.
Vistat
M. Verset
M. Vision?
Sentencet
Samplet
LaViDa
44.8
27.2
20.4
La ViDa
LaViDa LaViDa-FIM
1.00
1.00
45.2
29.3
24.0
Reason
1.00
1.00
Rel. A
+1%
+8%
+18%
LLa 1.67BB
0.41
0.37
Qwen2.5-VL-7B
0.45
0.16
4.3 Reasoning Distillation
Prior work has distilled LLMs 77 and VLMs [18 73 using long Mfain-of-thhughe (CoT) data elicit strong reasoning capabilities 25 In the same spirit, we study the reasoning abilities of LaViDa by conducting additional stage-3 training using 19.2K CoT examples distilled from VL- Rethinker-7B. strong reasoning model. We refer the finetuned model as La Viia--Reasoo We evaluate it on Math Vista [5T]. Math [84) and MathVision Vision [74] with CoT generation, comparing against stage-2 results without COT. We set the maximum generation length = 1024 for these tasks. We report these results in Table 2a We find that La ViDa-Reason outperforms I.a ViDa across all benchmarks, with the most significant gains observed on the most challenging Math Vision reasoning dataset (-18% relative improvement). Further details are provided in Appendix B.3

---

COCO Caption Performance
cuni gats stands tell syrttid of good furture,
117.5
Nt5-75%
Mr2-180%
Belxid the saceed stinm a pase good forsane.
Cahon hongs alove. symtol good fortune. Dedies witch over us, place of good tort.ie
115.0
Prompt
Write pownt with Iraee
LaViDa
112.5
130.0
INC
[FM]
tort gase staedd bell Beige pi larss reach for the sky. Cansore furter the breeon
Liovo 1.678
Draft Answer
Span-Ularatant--t LaViDa
[M] (M] FFM
[M] [M] (FM
Delics watch over the world bolow
LaViDa
105.0
-FIM
100.0
YFE-25%
with 4 lines
A lovil staens tot and tright Banwoth the skylk veet reparew Grosi rg the path. Bibokkons. Disam wheapern the bewaze.
storting with A
Prompt
be neppccivsly
Owen2.55 5
1.0
Latency (sisample)
1.2
1.4
1.4
1.8
(a) Text-Infiling
(b) Speed vs. Quality Tradcoff
Figure We showcase the advantages of ViDa over AR VLMS terms of controllability and speed. (a) Qualitative comparison on constrained poem generation between LaViDa /La /LaViDa-FIM and AR models. LaViDa variants successfully satisfy line-level constraints and adapt token length per line, unlike AR baselines.
(b) Speed-quality tradeoff for image captioning on COCO 2017. By adjusting the number of discretization steps (K), LaViDa offers a tunable balance between latency and output quality (CIDEr score).
4.4 Text Infilling
LaViDa offers strong controllability for text generation, particularly in text infilling. Given a draft of L tokens containing LM masks, we jump to timestep t LM and run standard inference to reach t = 0. This directly replaces LM masks with LM tokens. However, in practice, the intended completion may require fewer tokens-e.2 e.g.. "There is a [M] [M] [M] [M] in the image" might become either "dog" or "traffic light". To allow variable-length completions, we conduct an additional stage-3 training using a 20% subset of stage-2 data and refer to this model as LaViDa-FIM. During training, we insert random-length [S] [S] [FIM] sequences mid-text. At inference, we append [FIM] to masked segments (e.g., [M] [M] DM [M] [FIM] to signal flexible termination. The model can then generate completions like [dog] [S] [S] [S] [FIM] or [traffic] [light] [S] [S] [FIM].
While FIM objectives are often discussed in the context of language tasks (e.g., code completion) 63 7 they are equally relevant to multimodal applications. Figure]4a shows showss qualitative results on constrained poem generation, where the models generate a poem describing an image, with each line starting with specific syllables. Both VaViDa and LaViDa-FIM complete the task successfully, unlike AR models. Notably, La ViD-FFIM adapts token counts per line. Table 2b shows quantitative results over 100 samples: both La variants achieve 100% constraint satisfaction, while AR baselines remain below 50%. Additional results other text-infilling use cases are provided in Appendix B.2
4.5 Speed vs. Quality Trade Off
ViDa offers a convenient way to achieve speed-quality tradeoffs by controlling the number of discretization steps K. We compare the performance on image captioning with 500 images from
the COCO 2017 val dataset 42 with varying K. We set the maximum generation length to 32, and experimented with K E {32, 24, 16, 8), equivalently. NFEE { (100%, 75,, 50%, 25%). We report the average latency per image measured on a single A5000 GPU and the CIDEr score Figure 4b) At NFF=100%, VaDi achieves a higher CIDEr score than AR baselines but is slightly slower. At
NFE=75% and NFE=50% VaDi is faster than the AR basclines and achieves better quality. At NFE=25%, it is significantly faster but trails in performance. This indicates that ViDa can flexibly adjust its inference speed based on application nelds-allowing users to trade off generation latency and output quality depending their specific requirements.
Effect of KV Cache. The speed of LaViDa relies on our proposed Prefix- -LMM setup, which allows us to cache the keys and values of visual and prompt tokens [3] In Table Ba we compare the speed and sample quality between the proposed Prefi--LLM step and the uncached full-attention-nn
step used in prior works like LLaDa. We find that Prefix-DLM significantly reduces latency and achieves maximum speedup of 3.9x, with marginal performance cost. These experiments are performed using our stage-2 model, which is trained on full-attention mask. We discuss training with customized
8

---

masks and customized kernels in Appendix B.4 In short, we found that these alternatives lead to considerable training overhead while offering little benefit.
Table 3: Speed- Tradeoff. (a) We compare COCO image captioning performance with and without Prefix- DLM caching. Latency measured in seconds/sample. (b) We compare the performance of different schedules different NFEs.
(a) Effect of KV Cache
(b) Effect of timestep shifting
Method
NFE
CIDEr
Lateney
COCO Caption (CIDEr)f
NFE
25%
50%
75%
100%
Full-DLM
100%
121.0
7.65
Prefix-DLM
100%
117.3
1.93
cosine
87.7
102.2
110.8
117.3
Full-DLM 50%
118.6
4.13
linear
84.9
105.2
108.6
117.3
Prefix-DLM 50%
114.8
1.23
a=3
48.7
74.7
92.4
117.3
Open-Lnxt-8B
111.8
1.71
o=3
101.1
114.8
117.3
117.3
Noise Schedule. To study the effect of the proposed time-step shifting schedule, compare the performance of different schedules with NFEE (100%, 75%, 50%, 25%}. We compare the proposed time-step shifting with 3, and a 3-1, well as linear and cosine schedule baselines. We report results on COCO image captioning 22 in Table 3b The convex schedule with a = 3 works the best. We also observe similar behaviors when conducting CoT inference using LaViDonReason on Math Vision [74] dataset. At NFE=50% = 3 achieves accuracy of 21.05, which is 30% higher than 16. 12 achieved by linear schedule. We provide results on Math Vision in Appendix B.3
4.6 Ablation Studies
We conducted a thorough ablation of various design choices. In the main paper, we discuss the effect of complementary masking and input image resolution. We provide further discussion of other design choices in the Appendix B.5 We conducted these experiments using a 200k subset of our (stage-2) training data. We report results in Tables and respectively. Table 4a shows that our proposed complementary masking scheme leads considerable improvements across all benchmarks, most notably, complementary masking leads to relative improvement of 67% on ScienceQA [52] with affordable compute overhead during the training (8% slowdown). Table 4bb shows that high-resolution input improves overall performance, with the gain on OCR tasks being more pronounced than generic vision tasks (c.g.. VQAv2). We did not use average pooling for the low-resolution setup. We provide additional details in the Appendix B
Table 4: Ablation Studies. We study the effect of (a) complementary masking and (b) image resolution. For (a), we also report the wall clock time of 1000 training steps with batch size of 128.
(a) Effect of complementary masking.
(b) Effect of image resolution.
w/o Co
w/ Comp.M.
3842
7682
MME1
260.00
297.00
TextVQA
48.40
55.65
Math Vistat
28.40
33.40
Doc VQAt
43.22
58.72
ScienceQAt
48.74
81.49
ChanQAt
42.20
57.70
MMMU1
38.56
41.78
InfoVQAt
26.48
36.23
Runtime
8.2 hr
8.9 hr
VQAv21
65.92
66.78
Conclusion
In conclusion, we propose LaViDa, the first family of vision-language models based on DMs. To address various challenges, we introduce several novel training and inference techniques, including complementary masking, Prefix-DLM cache, and timestep shifting. Through extensive experiments, show that these techniques significantly outperform naive adaptation of DMs for visual tasks.

---

Using a comprehensive evaluation suite, we demonstrate that LaViDa achieves competitive perfor- mance compared to AR models trained under similar settings, while offering unique advantages such as speed-quality tradeoff's and controllability via text infilling. Our work proves that ViDa can be a powerful alternative to exitsing AR VLMs, extending the prior success of DMs the language domain to the vision space.
Acknowledgement
AG would like to acknowledge support from NSF CAREER Grant #2341040, Schmidt Sciences Early Career Fellowship, and Amazon Research Award.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems,
35:23716-23736, 2022.
[2] Anthropic. Claude 3.5 family. Setps://wxx http://attarrpc..co/////aaa///// sonoe///etaau.deeenettee sonnet, 2024. Ac- cessed: 2025-05-15.
[3] Mariannee Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.09577 2025.
[4] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34: 17981-17993, 2021.
[5] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge. Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13922 2025.
[6] Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q Tran, and Mchran Kazemi. Smaller, weaker, yet better; Training I1m reasoners via compute-optimal sampling. arXiv preprint arXiv:2408. 16737, 2024.
[7] Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv: 14255, 2022.
[8] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
[9] Daniele Rege Cambrin, Gabriele Scaffidi Militone, Luca Colomba, Giovanni Malnati, Daniele Apiletti, and Paolo Garza. Level up your tutorials: Vlms for game tutorials quality assessment. arXiv preprint arXiv:2408.08 08396699,9, 2024.
[10] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023.
[11] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 11315-11322 11325, 2022.
[12] Guiming Hardy Chen, Shunian Chen, Ruifei Zhang. Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v- synthesized data for lite vision-language model, 2024.
10

---

[13] Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprini arXiv: 105. 2021.
[14] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-mode models with better captions. arXiv preprint arXiv:2311. 2023.
[15] Lin Chen and Long Xing. Open-llava-next: An open-source implementation of leava-next series for facilitating the large multi-modal model community. https://github com/ xiaoachen98/ - LLaVA- NeXT 2024.
[16] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong. Kongzhi Hu, Jiapeng Luo, Zheng Ma. al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(11):220101, 2024.
[17] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-lingustic tasks. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 24185-24198, 2024.
[18) Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai- Chang. Open- vithinker: An early exploration complex vision-language reasoning via iterative self- improvement. arXiv preprint arXiv:2503. 2025.
[19] Jacob Devlin, Ming-Wast Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume I (long and short papers), pages 4171 4171-4186, 2019.
[20] Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A pro- gramming model for generating optimized attention kernels. arXiv preprint 2rX:::241..05496 2024.
[21] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference machine learning, 2024.
[22] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.0599, 2022.
[23] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng. Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.1333 2023.
[24] Yash Goyal, Tejas Khot, Douglas Summers-Staae Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattenn Recognition (CVPR), 2017.
[25] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
[26] Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. I.ava-uhdd: an Imm perceiving any aspect ratio and high- resolution images. In European Conference on Computer Vision, pages 390-406. Springer, 2024.
[27] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
11

---

[28] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint urXiv:2404.0665. 2024.
[29] Minghui Hu, Chuanxia Zheng, Heliang Zheng, Tat-Jen Cham, Chaoyue Wang, Zuopeng Yang, Dacheng Tao, and Ponnuthurai Suganthan. Unified discrete diffusion for simultancous Vision-language generation. arXiv, 2022.
[30] Drew A Hudson and Christopher D Manning. Gqa: new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6700 1700-6709,00 2019.
[31] Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5648-564 1448-5656,20 2018.
[32] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 235-251. Springer, 2016.
[33] Geewook Kim, Teakgyu Hong. Moonbin Yim, Jeong Yeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun. Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Conference on Computer Vision (ECCV), 2022.
[34] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yey Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015-4026, 2023.
[35] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[36] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang. Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint 2rXiv:2408.03326, 2024.
[37] Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends@ in Computer Graphics and Vision, 16(1-2):1 224, 2024.
[38] Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Zichun Liao, Yusuke Kato, Kazuki Kozuka, and Aditya Grover. Omniflow: Any-to-any generation with multi-modal rectified flows. arXiv preprint 24 169, 2024.
[39] Xiang Li, John Thickstun, Ishaan Guirajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. Advances in neural information processing systems, 35:4328-4344 2022.
[40] Xiaobo Liang, Zecheng Tang, Juntao Li, and Min Zhang. Open-ended long text generation via masked language modeling. In Proceedings of the 6lst Anmual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers), pages 223-241. 2023.
[41] Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanoy, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 26689-26699, 2024.
[42] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6- 6-12, 2014, proceedings, part 13, pages 740-755, Springer, 2014.
12

---

[43] Haotian Liu. Llava v1.6 vicuna-7b.
6- vi vicuna-b. 2023. Accessed: 2025-05-14.
[44] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 26296-26306 2024.
[45] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[46] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
[47] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216-233. Springer, 2024.
[48] Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient fronticr visual language models. arXiv preprint arXiv:2412.04468. 2024.
[49] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modcling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.168.1 2023.
[50] Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q Weinberger. Latent diffusson for language generation. Advances in Neural Information Processing Systems, 36:56998-5700 2023.
[51] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wes Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:23/0.0225: 2023.
[52] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Process- ing Systems (NeurIPS), 2022.
[53] Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263-2279, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[54] Minesh Mathew, Viraj Bagal, Ruben Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawa- har. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697-1706, 2022.
[55] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200-2209, 2021.
[56] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text images. In ICDAR, 2019.
[57] Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09999 2025.
[58] OpenAl. Gpt-4o system card. arXiv preprint arXiv:2410.22 2024.
[59] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint 2r7iv22307.01952 2023.
13

---

[60] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages 7748-876.. PmL.R, 2021.
[61] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 2020.
[62] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[63] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
[64] Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:1 17:133136-13011 2024.
[65] Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317-8326, 2019.
[66] Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, and Katerina Fragkiadaki. Unified multimodal discrete diffusion. arXiv preprint arXiv:2503.2088. 2025.
[67] Wei Ren Tan, Chee Seng Chan, Herann Aguirre, and Kiyoshi Tanaka. Improved artgan for conditional synthesis of natural image and artwork. IEEE Transactions on Image Processing, 28(1):394-409, 2019.
[68] Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 36:1 36:16083-16099, 2023.
[69] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403. 055300 2024.
[70] OpenThoughts Team. Open Thoughts. https://open-thouuhts.aa ai, January 2025.
[71] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[72] Changyuan Wang, Ziwei Wang. Xiuwei Xu, Yansong Tang, Jie Zhou, and Jiwen Lu. Q-vlm: Post-training quantization for large vision-language models. arXiv preprint 1rXiv:2410.08119, 2024.
[73] Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. VI- rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.088337 2025.
[74] Ke Wang, Junting Pan, Weikang Shi. Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. The eiiht Conference Neural Information Processing Systems Datasets and Benchmarks Truck, 2024.
[75] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing langualaa model's perception of the world at any resolution. arXiv preprint 2xXiv:2409.11991 2024. 2024.
14

---

[76] Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Yao Huang. Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zett and Christoph Feichtenhofer. Demystifying clip data. arXiv preprini arXiv:2309. /6671, 2023.
[77] Chenyu Yang, Xuan Dong, Xizhou Zhu, Weijie Su, Jiahao Wang, Hao Tian, Zhe Chen, Wenhai Wang, Lewei Lu, and Jifeng Dai. Pve: Progressive visual token compression for unified image and video processing in large vision-language models. arXiv preprint arXhv:2412.09613 2024.
[78] Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Multimodal large diffusion language models. arXiv preprint arXiv:2505. 15809, 2025.
[79] Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025.
[80] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang. Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
[81] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with 2022.
reasoning. Advances in Neural Information Processing Systems,
[82] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEF/CVF international conference on computer vision, pages 11975-11986. 2023.
[83] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang. Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024.
[84] Renrui Zhang, Dongzii Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.1 14624, 2024.
[85] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint (5tv:25004/0479 2025.
[86] Alon Ziv, Itai Gat, Gael Le Lan, Tal Remez, Felix Kreuk, Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi Masked audio generation using a single autoreerrreeeee transformer. arXiv preprint arXiv:2401.0455 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction highlight three contributions. Our main contribu- tion is the proposal DM-based VLMs. Our claims of competitive performance is supported by the experimental results in Section 4.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide discussions of limitation Appendix D
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Experiment details provided in Section 4.1 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will provide open access to the data and code in camera ready version. See our reproducibility statement Appendix ??
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//ipsscccppbllii.. wwtppbblcggguaiieecceeeeeeooeeee http://ddese/eee/e sondeeSubmsssiooPooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
They are documented in Section 4.1 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the standard procedure of reporting VLM evaluations, which generally do not include error bars. Conducting multiple experiment runs with different random seeds also very expensive. However, because the large size of evaluation data, reported results should be considered as significant.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: These are detailed in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We follow the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix
B
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Appendix E
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Appendix
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
No asset released the time of submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: not used.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
We use open-sourced LLMs to generate some reasoning data. The result is fully reproducible. We did not using any close-sourced L.Mss.
Justification: See Appendix G
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
22

---

A Additional Technical Details
A.1 Formulation of Discrete Diffusion Models (DMs)
In this section, we provide a detailed review of the formulation of Discrete Diffusion Models (DMs) briefly described in 2.2 Given a text sequence of L tokens Xo [Xx] ...1 the forward process xx)) gradually coverts it to a sequence of mask tokens [M] denoted by X1 .P. over the continuous time-interval [0, 1], with 2 t 0. Formally, this process is defined as
a x = [ M),
if = M Catt(X) - + + if + M
(4)
which has the marginal
a(X:)X)) = Cat(X;; (1- 1)x3 IM)
(5)
where Cat(.) denotes a categorical distribution and M, X X are probability vectors. MDLM 64 showed that the posterior of the reversal process p(X, X Xo. can be simplified into the following
Ca X(). (at( - x M).
if + M if if = M
xX Xx)
(6)
In practice, use the categorical distribution induced by the neural network's prediction in place of X the sample from the reverse process, which gives the following parametrization
[C, X(), (Cat X - (X M))
if + M if X = M
Po(X)
Inference Algorithm. Given target length T. and discretization steps la l....k where and 1K 1, we first initialize X1L ig = XI:L = M, then use Equation to repetitively sample 1 until we reach to = 0. this process, we also assume ( (Xtx) factorize into
I Po po(X) (X.,) following previous works 57 64 49].
Training Objective. Recall that the training objective of DMs introduced in Section 2.2 is formulated as
[ (Xol)(Xo
CDLM -e,
(8)
where (X)) factorizes into IT (X However, Equation 6 shows that po (x)),,) has a closed form solution that depends only on X, when X; * M. Intuitively, this comes from the fact that in the reverse process, once token X, changes from mask token clean text token, stays the same thereafter. Based on this observation, we can remove the terms that does not depend on the neural network from the learning objective, giving us the following loss
CDLM -B,
Σ X
log po Xx)
(9)
Hence, the loss is only computed over the masked indices in X,-
23

---

Comparision of Discretization Steps
Comparision of Different Schedule f(.)
K=16
a=1/3
Remaining Masked Tokens
K=24
q=3
K-32
Cosine
Continuous
Transformed Time
Linear
0.6
15
0.4
10
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Time
Time
(a) Discretized Schedules Different K.
(b) Different Choice of Continuous Schedules
Figure 5: Visualization of Schedules (a) We visualize discretization results of the same continuous schedule (dashed line) under different numbers of sampling steps K {16, 22) at 32. (b) We visualize various choices of continuous schedules: Shift(a 3- Shift(a 3), Cosine, and Linear.
A.2 Timestep shifting Schedule
Given sequence L tokens, and a timestep e [0, 1], in expectation X, contains tL masked tokens. In practice, implement Equation such that there are exactly [tL] masked tokens and L [tL] clean text tokens at timestep during the sampling process.
Given a fixed number of sampling step K, we define the canonical discretization as t; = for i 0, 1. with to and é This forms a uniformed sampling schedule, where roughly fixed mount of R tokens is unmakes each sampling step. Any other schedule can be defined as = ftt) where f(.) is a monotonic function such that f(0) = 0 and f(1)
When f()) is convex, the slope will be steeper when get closer 1 indicating that more tokens decoded in earlier sampling steps. By contrast, when f(.) is concave, the slope will be steeper when t get closer to 0, indicating that more tokens are decoded in later sampling steps.
Choice Schedule. We explored wide range of choices for the continuous schedule f(.). The timestep shifting schedule is a family of schedule defined as
at
f(t) = sc = 1 (a 1)
(10)
where is a hyperparameter. When < the schedule is convex. When a 1, the schedule is concave. The cosine schedule is defined as
f(t) 10 ( (1+ cos(Tt))
(11)
The linear schedule is just the identity function f(t) = t. We visualize these choices in Figure 5b
Rounding in Discretization. In principle, we can pick any f(.). However, given particular choice of L and K, if [f( and (t yields the same integer, then no tokens are unmasked when we compute po P (X ....). Hence, the actual number of function calls to the model 0 may be less than, depending on the choice of í()). To make the sampling compute cost more predictable and allow for a fair comparison across different schedulers, we augment all choices of f(.) to f (.) such that [ R (1x-1) < [ [.. (i.e. at least one token decoded at each step). Note that in the sampling process, the exact real value of (tx) does not matter long does not change

---

If [t. Hence, we can parameterize the sampling process in an alternative manner using sequence of integers F Lfxh (t,) with Fo and F L. Formally, we F* by solving the following optimization objective
K
F - f((x))
min ( (c..,1..L)
k=0
subject to
Fk-1
Fi:
for k: = 1,,,....
F =
F L.
We visualize such examples Figure 5a FK is effectively linear schedule, since the only schedule with 32 steps that satisfy the constraint FR-1 < FR is uniform schedule where we unmask exactly one token per step. As K reduces to 24 and 16, we the discretized schedule becomes closer to the continuous scheduler (visualized in dashed line).
We set = 32 and K E {16, 21,,2,,30 24,32). When K = 32,
A.3 Padding
We follow the design of LLaDa [57 and apply the loss function both standard text tokens and padding tokens. AR models typically do not compute the loss over padding tokens. However, when sampling from DMs, have specified generation length L. In the generation process, L mask tokens to L non-mask tokens. Since the length of the desired answer may not be exactly L tokens, the model will generate padding tokens. To achieve this capability, we pad the sequences during the training, and apply the loss on the padding tokens following LLaDa.
unmask
B
Additional Experiment Details and Results
B.1
Setup
In this section, we document the detailed training setup, including data, hyperparameters and compute used for the main experiments. Additional details about Prefix-DLM Cache can be found in B.4 Additional details about stage-3 math reasoning experiments can be found in B.3
Training Data Composition. For the pretraining phase, use LCS-558K 46 consists of 558K image-text pairs. For the finetuning phase, we mostly use the dataset released by Appen-LLaaVa-NN
15 We made some small adjustments the weight of each data source and increased the weight of some QA dataset. This is used to compensate the fact that our model only learns from a randomly chosen round each training step for multi-round QA data, We document the precise dataset composition of our stage-2 training in Table
Table 5: Compositio of Stage-2 Training Data. We report the data sources and sample sizes used to compose the Stage-2 finetuning data.
Data Source
Size
Data Source
Size
Data Source
Size
COCOO42
349,860
GQA 30
72,140
Doe 55
10,21
ALLaVA-VFLANII
202,966
Synthdog- En 33
29,765
DVQA 31
10,000
Visual Genome 35]
86,417
TextVQA [56
21,953
SA-IB 34
8,997
OCR VQA[56]
80,000
ChartQA [53
18,317
LLaVA-1.50K
46
2,988
GeoQA+ [13]
72,318
A12D (32
12,413
WikiArt
67
500
Chare-TextVor
500
Web-Celebrity 15
500
Web-Landmark
500
Training Hyperparameters. We use AdamW optimizer with learning rate 5e-3 with cosine decay schedule for all experiments. For pretraining (Stage 1), we adopted global batch size of 256 and trained for epoch. For finetuning (Stage 2), we adopted a global batch size of 512 and trained
for two epochs.
25

---

Compute Used. We used mixture of A100s and A6000s for training experiments and A5000 for evaluations and inference speed benchmarks. Because the memory constraint, we set the per GPU batch size to 8 on A100s and 4 on A6000s. We adjust the gradient accumulation steps accordingly so that the global batch size always 256 for the pretraining and 128 for the finetuning stage.
Evaluation Setup. We implement our evlaution using LMMS-Eval [83] library. We use the default prompt provided by the library for all benchmarks. We report the split used and the generation length L in Table 6
Table 6: Evaluation Setup. We report evaluation split and generation length L used to produce results of Table the main paper. #e generation length of 100 for LaViDa and generation
length of 1024 for LaViDa-Reason.
Dataset
Split
L
Dataset
Split
L
MME-P
test
100
Math Vista*
testmini format
100
VQAv2
val
16
Math Verse*
testmini _iiiiiiiiiiion__ie dominant
100
MMBench dev
100
Math Vision* *
mathvision testmini
100
MMMU val
16
ScienceQA
scienceqa-full
16
MME-C test
16
A12D
test
16
TextVQA val
16
ChartQA
test
16
VoA
test
32
Info
test
32
B.2 Text-Infilling
In this section, we provide additional qualitative results the text-infilling capability of VaDiDa We visualize the results in Figure 6 In the first example, we ask the model to extract multiple attributes from the image in JSON format. In the second example, we ask the model to edit a sentence based on the image. This is achieved by deleting the original sentence and inserting mask tokens. In the final example, we ask the model to complete a movie script based on the image prompt. La ViDa was able successfully complete these tasks.
While may be possible to achieve similar results using an AR model, they require careful prompting. By contrast, as shown in Figure 6 using diffusion model for text-infilling is more straightfoward.
B.3 Math Reasoning
In this section, we provide additional training setup for SaViDa-Reason and provide additional experiment results on the speed-quality tradeoff on math reasoning.
Data and Training Setup. In $4.3 we train LaViDaon long of-in-of-thught (CoT) data to get LaViDa-Reason Specifically, we choose strong 7B reasoning model, VLRethinker-77 [7] teacher model to generate the long reasoning traces. Further, we choose their own ViRL-39K data that contains (image, question, final answer). Subsequently, we generate the CoT and predicted final answer from the teacher model and filter the ones that lead to the correct final answer [81 6) This led the creation of the final dataset of size 19.2K [ In particular, this data for 5 epochs using the identical training setup as stage-2 (c.g., batch size, learning rates) and chose the checkpoint that achieves the best performance on Math Vision (testmini). We observe that the same checkpoint achieved good performance on the Math Verse and Math Vista dataset too. During inference, we set the generation length 1024 since LaVi synthesizes long chain-of-thoughte for problem-solving.
finetune Vi
Speed-Quality Tradeoff. In the main paper, reported the speed-quality tradeoff results COCO image captioning and discovered that the convex schedule works the best. We conducted similar study on LaViDa-Reason for CoT inference [7 Overall, the conclusion on CoT math reasoning task is similar to that on image captioning task, with the convex schedule performing the best across different choices of sampling steps. To
Vathnision dataset. We report these results in
Wee will release this data in the camera-ready version.
26

---

Prompt
Structured Data Extraction
Classify the Image
Draft Answer
(tojjcctddee""" dieee: [M]. (MIjiig
locasoon [MI.IM]FIM
Pobject
IMIFIM
"rocation" "ouldion""
(M). (M]FFM))
"coar:: "Millow"
"vald I'person"..
"pesaton:: "conter".
"oold, location:: ('indoor" ...]
"Vold colors" [Red J
LaViDa
"valdd postion": Platt, "Migh". "oertar"])
Prompt
Visually Grounded Text Editing
story based the image."
Draft Answer
Surm hamele a segmment.
The boy climbed the hill with dog. They weru surruunded the beauty of city wi bu kiings
The ciembad the hill with his Susdanny
They
and builling twrhor, As necceed the trey shooking across sky.
the wky.
LaViDa
Prompt
Visually Grounded Script Completion
Write movie script based the image
Draft Answer
segment.
Friend A "Wo.... this is way more than expeoted."
expected."
Frind B: [M]-IM]
sky."
Friend A. "We picked parfect right come."
Friend A. "We picked the perfect night come."
LaViD
Figure 6: Additional Qualitative Results for Text Infilling, We showcase several useful applications of text-infilling capabilities.
27

---

Accuracy-Thruug hput Frontio:: LaViDa vs Open LLaVA
LWES
LitA Moxi-33
Figure 7: Speed-Quality Tradeoff Math Vista Dataset. We report the throughput and accuracy on MathVista dataset of LaViDa under a variety of inference setup.
further examine the speed-quality tradeoff, we visualize the inference throughput (tokens/s) and accuracy Math Vista dataset under different inference setup in Figure compare with Open- LLaVA-Next-8B baseline. LaViDa's frontier strictly dominates the performance of autoregressive Open-LLa Vece-LLaVA-Next-8B both in terms of speed and quality.
Table 7: Effect of different schedule on Math Vision dataset. We study the effect of different schedules on Math Vision dataset using Chain-of- ohaingof-Though inference with La LaViDa-Reasom
Math Vision Acct
NFE
25% (256 steps)
50% (512 steps)
100% (1024 steps)
cosine
8.55
13.49
24.02
linear
10.86
16.12
24.02
a=3
5.59
8.88
24.02
a=3
12.5
21.05
24.02
B.4 Prefix-DMM
this section, we discuss several alternatives to our prefix-DLM setup that we explored and document the experiment results.
Inference Algorithm. Autoregressive models employ causal attention mask. Because of this, they can leverage KV cache for effective inference. By contrast, discrete diffusion models (DMs) used a full attention mask. While DMs can decode multiple tokens in parallel, it cannot leverage attention mask for fast inference. Prefix-DLM combine the best of both worlds by introducing prefix attention mask such that the queries of image embeddings and text prompts can only interact with keys and values of image embeddings and text prompts, but not the keys and values of answer tokens. Through this mechanism, we can leverage the KV cache for the image embeddings and text prompts. In vision-language applications with long context (900+ vision tokens per image), this saves a lot of compute at inference time, while preserving full bidirectional context.
3 which uses block-wise causal attention mask. this setup, the input sequece are chunked into sequence of fixed length blocks, each containing Lg tokens. A token in Block can see all tokens in the past and current Block j with j < i, but cannot see all future blocks Block with j > While this design allows it to leverage block-wise KV cache, it limits the bi-directional context to at most Lg tokens in the future, which undesirable for tasks like text-infilling. Additionally, because of its semi autorerressvev nature, when we are generating Block i, we must see all mask-free past Blocks with j Hence, the naive training algorithm can mask at most Ln tokens in each training sample (i.e. the last block), which is inefficient. To address this issue, customized attention kernel was developed to allow for parallel training. However, this leads to considerable training overhead. By contrast, Prefix- DDM can leverage the KV cache while having the full bidirectional context. It also
alternative to our prefix-DLM was the recently proposed semi-autoregressive
28

---

x
X
X
X
Causal
Block-Cause
Prefix-DMM
Full
Figure 8: Visualization of Different Choices of Attention Mask at Inference Time. / represents the image embeddings, P represents the prompt tokens, and X2 represents the partially masked answer tokens. Each row represents a query and each column represents a key. Colored region indicts tokens queries and keys can interact with each other.
does not require any specialized training algorithms, since we adopt it as pure inference technique of DMs.
We visualize the different choices of attention masks in Figure g We also compare the properties of different choices in g
Table 8: Comparison of Different Choices of Attention Mask. We compare properties of different
choices of attention masks. The desired properties are highlighted.
Method Attn. Mask Bi-Direction Context
KV Cache
Training Overhead
AR
Causal
None
Yes
No
DM
Full
Full Seq.
No
No
Within Block
Yes
Yes
LaViDa
Prefix-LLM
Full Seq.
Yes
No
pure inference algorithm. Our training process is identical to that of standard DM with full attention mask. We made this choice mostly because of efficiency reasons. We also explored adopting the prefix-DLM attention mask during the training (Prefi-DDLM-FT) with four different but equivalent implementations. Generally, these four imple- mentations categorized into two classes: Prefix-DLM-FT1 and Refix-DLM-FT2. We visualize these setups in Figure 9
Training Algorithm. We adopt Prefix-DLM
Recall from Section 3.2 that we adopted a complementary masking scheme, wherein given each triplet of image 7, prompt and answer X, we create two versions partially masked answer X, and X with complementary masking in order to utilize all tokens in the clean answer X. Empirically, this is achieved by copying the prompt embedding and concatenate X, and X to different copics respectively (Left Column of Figure 9. By default, use the full attention for both copies. We can adopt the Prefix-DLM attention mask during the training for each copy individually. We call this setup Prefix-DLM-FT1 (Middle Column of Figure Alternatively, we a single long sequence and manipulate the attention mask such that queries of I, P.P can see keys of I, P. queries of X, see keys of I, P. and queries of X,C can keys of I, P, X." We call this setup Prefix-DLM-FF (Right Column of Figure 9)
concatenate I, P. X,, X into
For each of the two variant Prefix-DIMMFFT and Prefix-DIM-FTY we designed two concrete implementations. The first set of implementations (Prefix- DLM--FIIMMsK.PPPP simply construct a 2D attention mask of size L x L for each sample, where L is the sequence length, and pass it to the torch SDPA kernel. The second variant Drrffx-DLLMFTT--FFE. Prefix -refix-DDM--F2. Flex) merely constructs an integer tensor of size 3 per sample, containing the length of 1, P. X. We then use a customized flex_ at cention module 20 to implement the attention mask implicitly based on the length of I, X.
Notably, these four variants of Trefi-DLMMFFT generates identical loss value and will produce exactly the same training dynamics (disregarding small numerical differences between implementations). The only difference the efficiency. Hence, we only benchmarked the performance and performed the full training using the most efficient version.
29

---

P
P
X
X
X
P
X
Prefix-DLM
Prefix-DLM-FT1
Prefix-DLM-FT2
Figure 9: Visualization of Training Strategies for Prefix-DLM. Given each triplet of image prompt P and answer X, we create versions of partially masked answer X, and X with complementary masking. (Left) By default, we construct two sequence and apply full attention mask. (Middle) Trefi-DLLM-FT applies prefix attention mask each copy independently. (Right) Prefix-DLM-FT2 combines I, P. X achieve an equivalent effect Prefix-DLM-FTI1
X into single sequence and manipulate the attention mask
We report the speed benchmark in Table Overall, even the fastest finetuning version is 62% slower than the full attention mask baseline, suggesting high overhead caused by batch-dependent masking strategies during training. We also report the model performance after 1,500 training steps (roughly 200k samples from the training data) in Table
Overall, Prefix-DLA and Prefi-DDLM-F DLM-FT has mostly identical performance, with Prefix-DLM, having a small lead over many tasks. Because of this, we consider the 62% overhead as unacceptable and opt for training procedure using the full attention mask. This also gives user additional dimension for speed-quality tradeoff: they can disable Prefix-DIMM cache and use the full attention mask at inference to achieve slightly better performance (Results shown in Table 3a in main paper).
Table Training Speed of Different Variants Prefix-DLM, We report the average training speed of different setup with batch size of 128 on 8 A6000 GPUs.
Method
Speed(s/training step)
Prefix-DLM
37.2
Prefx--DL--FTT-MM
67.3
Prefix-DLMMFTT-MMass
82.6
Prefi-DDLM-FFTI-Kemed
60.2
Prefx-DDL.M-FT2-KKendd
74.4
Table 10: Performance of Prefix-DLM and Prefix-DLM-EE across benchmarks. We compare the performance of two variants after 1,500 steps of training.
MMMU
VQAv2
MME
M. Vista
M. Verse
ScienceQA
AI2D
Deffx--DLM
40.56
63.26
286.79
33.60
19.67
80.36
64.31
DLM-FT
40.10
61.02
290.71
35.00
18.15
80.15
64.15
30

---

Table 11: Ablation Studies the Choice Vision Encoders. We compare the performance of models with different vision encoders after 1,500 steps of training. On Average, SigLip improves
over CLIP by +1.86 and improves over MetaCLIP by +2.77.
MMMU
VQAv2
MME
M. Vista
Versee
ScienceQA
A12D
SigLip
40.56
63.26
286.79
33.60
19.67
80.36
64.31
CLIP
40.44
59.58
288.21
30.90
20.05
76.56
59.78
MetaCLIP
40.33
55.96
280.36
33,80
17.26
78.66
62.79
B.5
Ablation Studies
We conduct additional ablation studies over the choice of vision encoders. We experimented with SigL.p 82 CL.P (60), and Me66 We report the model performance after 1,500 training steps (roughly 200k samples from the training data) in Table 11 Overall, SigL.i achieves the strongest overall performance, with notably gains in VQAv2, ScienceQA, and AI2D.
B.6
Additional Scaling
While La ViDa outperforms AR VLMs under comparable data scale, there is still a considerable gap between LaViDa and state-of-the-an art AR VLMs. To validate the scalability of LaViDa, scale the training schedule by 2x, resulting in considerable performance on general understanding and OCR tasks. We report these results in Table [2] Due to compute constraints, left further scaling to future works.
Table 12: Additional Scaling Experiments on OCR and General Understanding Tasks.
Model
MME
MMBench
ChartQA
DocVQA
InfoVQA
LaViDa
341.1
70.5
64.6
59.0
34.2
+Additional Training
444.3
75.6
68.6
43.4
Opea-LLaVA-Next-3
336.8
74.4
69.7
69.9
36.7
C Additional Backgrounds
In this section, we provide additional discussions with relevant works not covered in the main paper.
Masked Generative Models. Masked generative modeling has a long history before the recent advancements DMs. Earliest works such BERT [9 and MAE[7] adopt the masked generative modeling objective as pretraining objective to learn rich text and vision features. They mainly concern the utility learnt features to downstream perception and understanding tasks, instead of the generation capability of the model. A series of subsequent works use mask modeling to build generative models for images [1] 10, texts 40 and audio [6]. Compared with these early works relying ad-hoc sampler designs, recent works DMs
49 64 provided principled way for
training and sampling from masked generative models.
Multi-Modal Diffusion Models. Several works have explored to build a multi-modaa diffsion models with lision-language capabilities. CoDi [68] and OmniFlow (38] build continuous diffusion model over a latent text embedding space and use autoregressive decoder to decode the generated latent mebeddings to actual texts. UniD3 (29 and UniDise (66) build discrete diffusion models for simultaneous text and image generation. Overall, these models have limited language generation capabilities. Their experiments on text generations are limited in scale and mostly focusing on captioning. They cannot perform more complex mnstruction-folloiini and understanding tasks (e.g. reasoning) like many modern autoregressive VLMs.
D Limitations
In this section, we discuss the limitations of LaViDa. There are mainly two limitations.
31

---

Scale, While LaViDeachieves competitive results when compared against similar-sized AR VLMs trained on a similar scale of data, there remain considerable gap between LaViDa's performance and that of mate-of-the--ee open sourced VL.ss such as Va-aVa-One Dieionson and Qwen2.5-VI. These models either comes with more training data or larger model sizes. Future work should study if DMs scale well with larger model and more training data.
OCR. LaViDa's performance on OCR tasks slightly worse than the baselines, this can be mainly attributed to the average pooling operation which we introduced to reduce the sequence length by compressing the visual information. Concretely, LLaVa-1.6- 8B baseline do not adopt average pooling, and uses 2880 tokens (24 x 24 x Views) to represent each image. In our setup, removing the average pooling would lead to total of 3645 tokens (24 x 24 x Views) per image. Average pooling is necessary because the base model LLaDa and Dream has context length of 4096 and 2048 respectively. Without average pooling. L.LaDa will not have enough context length to fit longer training samples, while Dream will not have enough context length even fit one image.
and Open- -aeaaaaa-----aaaaaaa -aamm---
We also tried to extend the context length of these models via techniques such as rope rescaling, but achieved limited success, We experimented with extending Dream 4096 context length and evaluate the model using the seedle-in-a-haystall task (28) at 4096 context length. We found that Dream-78 (72.4 Acc) performs worse than LLaDa-8B (91.6 Acc) and Llama-3-8B (95.4 Acc).
We hope future works on DMs with longer context will provide stronger base models to finetune on vision-language applications.
E Boarder Impacts and Safeguards
Our model may inherit the biases embedded the base model LLaDa-8B Dream-7B, as well as biases incorporated in the training data. Our model may also suffer from the same hallucination problem as the base model. Just as any LLMs and VLMs, there also the risk that our model is used to generate harmful offensive content. Overall, model is intended to be used by researchers to build strong diffusion model for vision-lanne language applications. We do not recommend be used for any other purposes.
F License Information for Data and Models
We report the licenses of the used artifacts Table 13 followed the intended use of all respective artifacts.
32

---

Table 13: Licenses and sources for datasets and models used. Datasets marked with have no published license; the accompanying paper's arXiv license is used as fallback. Training entries marked with f or reflect compound or inferred licensing conditions.
Category
Name
License
Platform
Vision Encoder
SigLIP
Apache 2.0
Hugging Face
Language Model
LLaDA-8B
MIT
Hugging Face
Language Model
Dream-7B
Apache 2.0
Hugging Face
VLM
LLaVA-NeXT (1.6)
Apache 2.0
GitHub
VLM
LLaVA-NeXT (1.6)
LLaMA License
Hugging Face
VLM
Open-LLaVA-NeXT
Apache 2.0
Hugging Face
Train Dataset
LLaVA-Pretrain
CC3M + BSD-3-Clause
Hugging Face
Train Dataset
Open-LLaVA-NeXT
CC BY-NC 4.0
Hugging Face
IixIM
Evaluation Dataset
МME (MME-P)
arXiv Non-exclusive"
arXiv
Evaluation Dataset
VQAv2
CC BY 4.0
Official Website
Evaluation Dataset
MMBench
Apache 2.0
GitHub
Evaluation Dataset
MMMU
Apache 2.0
Hugging Face
Evaluation Dataset
MME (MME-C)
arXiv Non-exclusive"
arXiv
Evaluation Dataset
Math Vista
CC BY-SA 4.0
Hugging Face
Evaluation Dataset
Math Verse
MIT
Hugging Face
Evaluation Dataset
Math Vision
MIT
Hugging Face
Evaluation Dataset
ScienceQA
CC BY-NC-SA
Official Website
Evaluation Dataset
A12D
arXiv Non-exclusive'
arXiv
Evaluation Dataset
TexIVQA
CC BY 4.0
Official Website
Evaluation Dataset
Doe
arXiv Non-exclusive"
arXiv
Evaluation Dataset
ChartVQA
GPL-3.0
GitHub
Evaluation Dataset
InfoVQA
arXiv Non-exclusive"
arXiv
Subject to the CC-3M dataset may be freely used for any purpose, although acknowledgementt of Google LLC ("Google") as the data source would be appreciated. The dataset is provided "AS IS" without any warranty, express implied. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset
Note :
Note + Our training data is based on Open-LLavVANeeTT-iin 1M, which combines ShareGPT4V data (CC BY-NC 4.0, research-only, with restrictions from LLaMA, Vicuna, and GPT-4 licenses) and AVG-LLaVA (Apache 2.0). All data used strictly for academic research.
G
LLM Usage
LLM is used to generate the math reasoning date for stage-3 training. See details
B.3
H
Concurrent Work
Concurrent to this work, MMADa 78 (released after submission deadline) proposed unified under- standing and generation modeling using DM formulation. However, they do not leverage many of the novel techniques that we propose, leading to inferior performance and slow inference speed, We list a brief comparison in Table 14
33

---

Table 14: Comparison with MMaDa. We report scores MME, MMMU, and MMB benchmarks, along with average latency for image captioning.
Model
MME
MMMU
MMB
Latency (s/image)
LaViDa-Dream
1463.5
42.6
73.8
1.13
LaViDa-LLaDa
1365.6
43.3
70.5
1.32
MMaDa
1410.7
30.2
68.5
7.68
34


Reviews:
[{"review_id": "SxZCyDIsoq", "reviewer": "Reviewer_tRob", "summary": "This paper proposes a multimodal large diffusion model called LaViDa. It introduces complementary masking, Prefix-DLM cache and timestep shifting to improve efficiency and performance. Experiments show that LaViDa achieves comparable performance with autoregressive models, with higher speed-quality tradeoffs and controllability.", "questions": "1.Though not required, it is highly suggested to include discussions on concurrent works like LLaDa-V[1], MMaDA[2] and Dimple[3].\n2.Typos:\na.Line 227: InternVL-38B -> InternVL3-8B\nb.Appendix Line 618: \"than, K depending\" -> \"than K, depending\"\nc.Appendix Table 5: compositio -> composition\nd.Appendix Line 671: LaViDaon -> LaViDa on\ne.Appendix Line 713: 8 -> Table 8\nf.Appendix Line 741: Figure 7 -> Figure 9\n[1] You, Zebin, et al. \"Llada-v: Large language diffusion models with visual instruction tuning.\" arXiv preprint arXiv:2505.16933 (2025).\n[2] Yang, Ling, et al. \"Mmada: Multimodal large diffusion language models.\" arXiv preprint arXiv:2505.15809 (2025).\n[3] Yu, Runpeng, Xinyin Ma, and Xinchao Wang. \"Dimple: Discrete diffusion multimodal large language model with parallel decoding.\" arXiv preprint arXiv:2505.16990 (2025).", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n1.The three proposed techniques help to address the issues of extending discrete diffusion models to multimodal tasks.\n2.The authors provide extensive experiments and ablations on the design choices to validate the effectiveness of each component.\n3.The writing and figures are clear and easy to follow.\n\nWeaknesses\n1.To examine the speed-quality tradeoff more thoroughly apart from Figure 4b, the authors should provide performance-latency results on more benchmarks.\n2.Results on OCRBench should be included to further validate the limitation on high-resolution understanding.\n3.Since text infilling serves as an evidence to the advantage of diffusion-based models, more **quantitative experiments** should be conducted, e.g. Compare the metrics against autoregressive methods on the tasks in appendix Figure 6.", "was_revised": true, "final_justification": "After carefully reading comments from all reviewers and authors' rebuttal, my most concerns are well addressed. Therefore, I am glad to maintain my postive score.", "rebuttal": "We thank the reviewer for the constructive feedback.\n\n**W1 Additional Performance-Latency Results**\n\nAs requested, we provide additional performance-latency results on MathVista in the table below. We report the maximum generation length (L), the number of sampling steps, the average number of non-padding tokens generated, the latency per sample, and the throughput of non-padding tokens. In accordance with review policy (no image allowed), we will include an additional visualization of these results similar to Figure 6 in the camera-ready version. For clarity, we highlight all metrics that outperform the AR baseline in **bold** in the table. With an optimal choice of parameters, our model achieves up to a 2.5x speedup compared to the AR baseline model.\n\n\n| Max Length | Steps | Avg Tokens | Acc(MathVista) $\\uparrow$ | Latency(s) $\\downarrow$| Throughput (tk/s) $\\uparrow$|\n| --- | --- | --- | --- | --- | --- |\n| LaViDa-LLaDa |  |  |  |  |  |\n| 1024 | 256 | 378 | **45.5** | 22.0 | 17.2 |\n| 512 | 128 | 333 | **43.5** | **10.05** | **33.1** |\n| 256 | 256 | 220 | **45.2** | 16.44 | 13.4 |\n| 256 | 128 | 217 | **42.9** | **8.25** | **26.3** |\n| 256 | 80 | 215 | **44.9** | **6.25** | **34.4** |\n| 256 | 60 | 211 | **44.6** | **5.17** | **40.8** |\n| 256 | 40 | 216 | **43.6** | **4.18** | **51.7** |\n| 256 | 20 | 221 | 33.2 | **3.42** | **64.6** |\n| Open-Llava-Next-8B |  |  |  |  |  |\n| NA | NA | 199.75 | 42.0 | 10.48 | 19.1 |\n\n**W2 OCR Performance**\n\nAs requested, we provide additional results on OCR-Bench in the table below. We acknowledge that our model’s performance on OCR tasks has room for improvement. One contributing factor is that diffusion language models typically require more training steps to converge than AR models, as demonstrated in a recent study [1]. To validate this hypothesis in our setting, we conducted an additional two epochs of training. This resulted in substantial improvements on both OCR tasks such as ChartQA and conventional vision tasks such as MME, confirming that OCR performance can be enhanced with additional training. We hope future research on diffusion language models will further improve their training efficiency.\n\n| Model                | MME   | MMBench | ChartQA | DocVQA | InfoVQA | OCR Bench |\n| -------------------- | ----- | ------- | ------- | ------ | ------- | --------- |\n| Paper                | 341.1 | 70.5    | 64.6    | 59.0   | 34.2    | 404       |\n| +Additional Training | **444.3** | **75.6**   | **74.9**    | 68.6   | **43.4**    | 485       |\n| Open-Llava-Next-8B   | 336.8 | 74.4    | 69.7    | **69.9**   | 36.7    | **605**       |\n\n\n**W3 Additional Quantitative Experiments on Text-Infilling**\n\nAs requested, we have included additional results on text-infilling for three tasks presented in Figure 6: JSON satisfaction, text editing, and visually grounded script completion. We randomly sampled 100 images from the LAION-2B dataset and employed GPT-4o to generate relevant questions for each image category. We also utilized GPT-4o to create optimized prompts for these VLM tasks. For the text editing and script completion tasks, the prompts explicitly instruct the VLMs to edit a specific sentence or complete a specific line while preserving the remaining text. For JSON completion, we verified the schema compliance using the Python JSON package. For text editing and script completion tasks, we employed string matching to ensure that the prefix and suffix were preserved in the generated output. The results are reported in the table below.\n\n|  | JSON Satisfaction | Text Editing | Visually Grounded Script Completion |\n| --- | --- | --- | --- |\n| *LaViDa* | 1.0 | 1.0 | 1.0 |\n| *LaViDa-FIM* | 1.0 | 1.0 | 1.0 |\n| *LLaVa-1.6-7B* | 0.89 | 0.58 | 0.49 |\n| *Qwen2.5-VL-7B* | 0.98 | 0.85 | 0.22 |\n\nThese results highlight the strength of diffusion-based VLMs, which consistently achieve a 100% constraint satisfaction rate without the need for careful prompting. We will provide additional details on these evaluations in the final version.\n\n**Q1 Concurrent Works Discussion**\n\nWe are aware of the concurrent works mentioned, which were released after our submission deadline. We will include relevant discussions and citations for these works in the revised version.\n\n**Q2 Typographical and Grammatical Errors**\n\nWe thank the reviewer for identifying these editorial errors and will correct them in the final version.\n\n[1] Prabhudesai, M., Wu, M., Zadeh, A., Fragkiadaki, K., & Pathak, D. (2025). Diffusion Beats Autoregressive in Data-Constrained Settings. arXiv preprint arXiv:2507.15857."}, {"review_id": "ZMjxWsL7eB", "reviewer": "Reviewer_kZHX", "summary": "The authors propose LaViDa, a family of diffusion-based vision-language models that integrate a vision encoder with a diffusion language model. The architecture consists of a pre-trained diffusion LM augmented with a visual encoder for image tokens embeddings and a projection layer to align vision features with the text model’s embedding space. The model is trained in two stages: a vision-language pretraining where only the projection is learned (to align visual features with the diffusion LM), and a supervised fine-tuning on multimodal instruction-following data where all components (vision encoder, projection, diffusion LM) are optimized together for tasks like image QA, captioning, etc. This two-stage setup is analogous to how AR VLMs (like LLaVA) are built by first aligning vision features and then fine-tuning on instructions. The authors address several challenges of the diffusion language models, introducing two-level complementary masking enabling the loss always be computed for the crucial answer tokens, and prefix caching for vision prefix that enables faster inference for long multimodal input.", "questions": "I have the following questions for the authors:\n1. What is the memory footprint for the diffusion-based VLMs in comparison to the similar-size autoregressive models?\n2. How do you plan to improve results on the OCR-heavy tasks?\n3. Do you have any ideas on how to adapt your approach to all-to-all models that can not only read, but also to produce images?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The main strengths of the paper are as follows:\n\n1. The authors introduced first Diffusion-Based VLM with the deep analysis and ablation study. Adapting standard adapter-based vision module integration to the diffusion-based LLMs is indeed clear and promising idea. The authors investigate different approaches on how to cope with the diffusion-based challenges, thus, providing the researcher good starting point to improve and enhance diffusion-based VLMs.\n2. New technical enhancement for diffusion-based VLMs were introduced. The authors provide interesting modifications for the masking procedure that is important for the model to take signal from the most valuable output tokens (the answer tokens). Also, it is profitable to include the prefix-caching for the multimodal input that is usually the most token-intensive.\n3. Extensive ablations with FIM objectives are valuable for further research in diffusion-based VLM.\n\nThe main weaknesses of the paper are as follows:\n\n1. While the proposed models show comparable results with autoregressive models, we still see that the diffusion-based models provide worse results on text-intensive and OCR tasks. Thus, it is still a question, how to improve this part of the model's performance.\n2. Fixed-length generation constraints of the diffusion. While the authors proposed modification for training using FIM objective, it is still a challenge, how not to train model to produce padding tokens in the case of the random-length generation.\n3. The authors propose caching for the lengthy multimodal input, however, I couldn't find the memory footprint for the final pipeline, based on the memory-speed tradeoff, for now, it still seems that diffusion-based models could be less profitable in comparison to the autoregressive ones.", "was_revised": true, "final_justification": "I would keep my score (5) and recommend to accept this paper for the publication on NeurIPS. I see this paper as one of the former works on developing diffusion-based models in vision-language domain that is quite solid in the current format, and can motivate researchers to develop this topic more.", "rebuttal": "We thank the reviewer for the constructive feedback.\n\n**W1 Performance on OCR Tasks**\n\nWe acknowledge that our model’s performance on OCR tasks leaves room for improvement. One contributing factor is that diffusion language models generally require more training steps to converge than autoregressive (AR) models, as shown in a recent study [1]. To validate this hypothesis in our setting, we conducted an additional 2 epochs of training. This resulted in substantial improvements on both OCR tasks such as ChartQA and conventional vision tasks such as MME. Hence, OCR performance can be improved through additional training. We hope that future work on diffusion language models will further enhance training efficiency.\n\n\n| Model                | MME   | MMBench | ChartQA | DocVQA | InfoVQA | \n| -------------------- | ----- | ------- | ------- | ------ | ------- | \n| Paper                | 341.1 | 70.5    | 64.6    | 59.0   | 34.2    |\n| +Additional Training | **444.3** | **75.6**   | **74.9**    | 68.6   | **43.4**    |\n| Open-Llava-Next-8B   | 336.8 | 74.4    | 69.7    | **69.9**   | 36.7    |\n\n\n\n**W2 Fixed-Length Generation Constraints of Diffusion Models**\n\nWe recognize that fixed-length generation is a significant constraint for diffusion language models. This limitation applies to all current diffusion-based language models. However, recent studies released after our submission deadline have begun to address this issue. For example, DreamOn [2] introduces special tokens that allow for expanding masked tokens or deleting unused ones. We plan to explore incorporating such advances in future work to overcome this limitation.\n\n**W3 Memory Footprint**\n\nWe use prefix caching primarily to improve inference speed (see Table 3.a). On the A5000 GPU, we do not observe a significant difference in peak memory usage between the two settings. For example, memory usage is 22,811 MB with caching and 21,668 MB without caching.\n\n**Q1 Memory Footprint**\n\nPlease refer to W3 above.\n\n**Q2 OCR Tasks**\n\nPlease refer to W1 above.\n\n**Q3 Multi-Modality**\n\nAs part of an ongoing follow-up project, we have integrated a VQ-tokenizer and additionally trained the model on text-to-image generation tasks. We achieved a GenEval score of 0.75, outperforming previous state-of-the art masked generative model Meissonic[3] (0.54) and a current work MMaDa[4] (0.63) . These are preliminary results and will be part of a separate future submission. Nonetheless, these results indicate that it is entirely feasible to extend our setup toward building a unified model for both understanding and generation using a single diffusion objective.\n\n\n\nReferences\n\n[1] Prabhudesai, M., Wu, M., Zadeh, A., Fragkiadaki, K., & Pathak, D. (2025). Diffusion Beats Autoregressive in Data-Constrained Settings. arXiv preprint arXiv:2507.15857.\n\n[2] Zirui Wu*, Lin Zheng*, Zhihui Xie, Jiacheng Ye, Jiahui Gao, Yansong Feng, Zhenguo Li, Victoria W., Guorui Zhou, Lingpeng Kong (2025). DreamOn: Diffusion Language Models for Code Infilling Beyond Fixed-Size Canvas.\n\n[3] Bai, J., Ye, T., Chow, W., Song, E., Chen, Q. G., Li, X., ... & Yan, S. (2024, January). Meissonic: Revitalizing masked generative transformers for efficient high-resolution text-to-image synthesis. In The Thirteenth International Conference on Learning Representations.\n\n[4] Yang, L., Tian, Y., Li, B., Zhang, X., Shen, K., Tong, Y., & Wang, M. (2025). Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505.15809."}, {"review_id": "MrVsJzctTm", "reviewer": "Reviewer_Y6e3", "summary": "This paper introduces LaViDa, the first family of VLMs based on diffusion models. The authors propose several novel training and inference techniques for DMs, including complementary masking, Prefix-DLM, and timestep shifting that improve the training efficiency, inference speed, and sample quality of LaViDa. LaViDa achieves competitive performance across a wide range of tasks compared to AR VLMs, while offering the unique benefits of DMs.", "questions": "1. For complementary masking, if we ignore the time cost for vision encoding, then it is actually doing 2 forward-backward pass during training. However, according to Table 4a, the training time is only increased from 8.2hr to 8.9hr under the same training steps. Why is the time not doubled?  \n2. Could the authors report the total training time (gpu hours) for different stages for LLaViDA to see whether reproducing LLaViDA is possible for the research community. \n3. What is the exact value for L when evaluating on general tasks (tasks involving short responses other than mathverse and mathvision)? What are the average **actual** response length for these tasks and what is the performance if we reduce L to the smallest possible and see the performance? This is to see whether LLaViDA is trying to modifying its earlier response in later diffusion steps.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1. This paper identifies the important efficiency challenges encountered when training a diffusion-based LLM, e.g., inefficiencies in loss computing and the incompatibility to KV-cache. Although these challenges are not restricted to diffusion MLLM, the authors propose solutions that leverage unique characteristics of MLLM. \n2.LLaViDA achieves good quality-speed trade-off on certain tasks, demonstrating promising future for diffusion-based MLLM. \n3. This paper is well-formulated and is easy to follow.\n\n## Weakness\n### Insufficient comparisons on speed-quality trade-off between AR-based MLLM and LLaViDA. \nSpecifically, the comparison in the paper (Table-3a) only include the captioning tasks on COCO with Open-LLaVA-Next 8B. However, to fully demonstrate the advantage of LLaViDA, the authors should report the speed-quality trade-off on other tasks that require long responses such as MathVerse and MathVision. \n\n### Lack of details on the evaluation of text infilling tasks.\nWhat are the datasets used and their statistics when evaluating the constrained poem task (Table 2)? Moreover, how to evaluate the Sentence and sample-level constraint satisfaction? What are the prompts used to evaluate these tasks? Is it possible that the prompts used during evaluation favours diffusion-based MLLM other than the AR-based counterpart?", "was_revised": true, "final_justification": "The authors addressed most of my concerns. However, the performance of the proposed diffusion MLLM still lags behind existing AR counterparts. This also mentioned by other revieweres. The authors are expected to scale up its performance in the future and include the additional experiments mentioned in the rebuttal in the final version. Nevertheless, we can still accept this paper due to its valuable exploration on diffusion MLLMs.", "rebuttal": "**W1 Additional Performance-Latency Results**\n\n\nAs requested, we provide additional performance-latency results on MathVista in the table below. We report the maximum generation length (L), the number of sampling steps, the average number of non-padding tokens generated, the latency per sample, and the throughput of non-padding tokens. In accordance with review policy (no image allowed), we will include an additional visualization of these results similar to Figure 6 in the camera-ready version. For clarity, we highlight all metrics that outperform the AR baseline in **bold** in the table. With an optimal choice of parameters, our model achieves up to a 2.5x speedup compared to the AR baseline model.\n\n\n| Max Length | Steps | Avg Tokens | Acc(MathVista) $\\uparrow$ | Latency(s) $\\downarrow$| Throughput (tk/s) $\\uparrow$|\n| --- | --- | --- | --- | --- | --- |\n| LaViDa-LLaDa |  |  |  |  |  |\n| 1024 | 256 | 378 | **45.5** | 22.0 | 17.2 |\n| 512 | 128 | 333 | **43.5** | **10.05** | **33.1** |\n| 256 | 256 | 220 | **45.2** | 16.44 | 13.4 |\n| 256 | 128 | 217 | **42.9** | **8.25** | **26.3** |\n| 256 | 80 | 215 | **44.9** | **6.25** | **34.4** |\n| 256 | 60 | 211 | **44.6** | **5.17** | **40.8** |\n| 256 | 40 | 216 | **43.6** | **4.18** | **51.7** |\n| 256 | 20 | 221 | 33.2 | **3.42** | **64.6** |\n| Open-Llava-Next-8B |  |  |  |  |  |\n| NA | NA | 199.75 | 42.0 | 10.48 | 19.1 |\n\n\n**W2 Evaluation of Text-Infilling Tasks**\n\nWe create a constrained satisfaction test set that requires the model to generate four-line poems, where the first word of each line must begin with two specific letters. The first letter must be a consonant from the set {'b', 'c', 'd', 'f', 'g', 'h', 'n', 'p', 'r', 's', 't'}, and the second letter must be a vowel from the set {'a', 'e', 'i', 'o', 'u'}. The task asks the model to generate a poem with specific starting syllables (e.g., \"ca\") for each line. Sentence-level and sample-level satisfaction are computed by string matching to check if each line begins with the specified characters. Sample-level satisfaction is achieved if all lines satisfy the sentence-level constraints. The prompt template is shown exactly in Figure 4. We note that this is a reasonable prompt and that better prompting may further improve the performance of AR models. However, one of the key advantages of diffusion models is their ability to achieve 100% provable satisfaction of arbitrary constraints without the need for prompt tuning.\n\nTo further validate this claim, we include additional results on text-infilling for three tasks shown in Figure 6: JSON satisfaction, text editing, and visually grounded script completion. We randomly sample 100 images from the LAION-2B dataset and ask GPT-4o to generate a relevant question of each category for each image. We also ask GPT-4o to generate optimized prompts for VLMs for these tasks. For text editing and script completion, the prompts specifically ask the VLMs to edit a specific sentence or complete a specific line while leaving the rest of the text unchanged. For JSON completion, we verify schema compliance using the Python json package. For text editing and script completion, we use string matching to verify that the prefix and suffix are preserved in the output. The results are reported in the table below.\n\n|  | JSON Satisfaction | Text Editing | Visually Grounded Script Completion |\n| --- | --- | --- | --- |\n| *LaViDa* | 1.0 | 1.0 | 1.0 |\n| *LaViDa-FIM* | 1.0 | 1.0 | 1.0 |\n| *LLaVa-1.6-7B* | 0.89 | 0.58 | 0.49 |\n| *Qwen2.5-VL-7B* | 0.98 | 0.85 | 0.22 |\n\nThese results highlight the strength of diffusion-based VLMs, which consistently achieve a 100% constraint satisfaction rate without the need for careful prompting. We will provide additional details on these evaluations in the final version.\n\n\n**Q1 Complementary Masking**\n\nWe observe that encoding and backpropagating through the vision encoder introduces considerable overhead. Since we split each image into 5 views, adding one more conversation effectively increases the batch size of the vision encoder by 5. We conducted additional measurements on 8 A100 GPUs. Using a per-GPU batch size of 2 with complementary masking results in 4.16 seconds per step, while a per-GPU batch size of 4 without complementary masking results in 7.01 seconds per step, highlighting the high overhead of vision encoding.\n\nAnother contributing factor is that we pad each batch to the longest sequence. The expected maximum length of 4 randomly selected sequences is greater than that of 2 randomly selected sequences.\n\n**Q2 GPU Hours**\n\nTotal training time is 902 A100 (80GB) hours for LaViDa-LLaDa and 1,768 A6000 (48GB) hours for LaViDa-Dream.\n\n**Q3 Generation Length**\n\nWe already report the values of L in Appendix B, Table 6. We will provide detailed average generation lengths for each task in the final version. In general, the generation length for short-answer tasks is typically fewer than 10 tokens. Changing the maximum length does not affect performance or the number of generated tokens, as the prompt often includes format instructions such as \"answer with a few words.\" However, when evaluating our reasoning model on MathVista, we found that maintaining a generation length greater than 200 tokens generally yields stable performance, whereas aggressively reducing it leads to performance degradation.\n\nAs an example, we include the results of varying generation length for AI2D and MathVista in the tables below. \n\n| **Max Length** | **Avg Tokens** | **Acc(Ai2d)** |\n| --- | --- | --- |\n| 16 | 2 | **70.0** |\n| 4 | 2 | 69.9 |\n\n| **Max Length** | **Avg Tokens** | **Acc(MathVista)** |\n| --- | --- | --- |\n| 1024 | 378 | **45.5** |\n| 512 | 333 | 43.5 |\n| 256 | 220 | 45.2 |\n| 128 | 102 | 41.2 |\n\n[1] Prabhudesai, M., Wu, M., Zadeh, A., Fragkiadaki, K., & Pathak, D. (2025). Diffusion Beats Autoregressive in Data-Constrained Settings. arXiv preprint arXiv:2507.15857.\n\n[2] Wu, Z., Zheng, L., Xie, Z., Ye, J., Gao, J., Feng, Y., Li, Z., Wu, V., Zhou, G., & Kong, L. (2025). DreamOn: Diffusion Language Models for Code Infilling Beyond a Fixed-Size Canvas."}, {"review_id": "QEzV4MNxqT", "reviewer": "Reviewer_fbuJ", "summary": "To address the challenges of fast inference and controllable generation in vision-language models (VLMs), this paper proposes a novel paradigm based on discrete diffusion models (DLMs), leveraging their parallel decoding and bidirectional context-awareness. The method follows a LLaVA-style framework, where visual features are projected into the language embedding space via an MLP, and instruction tuning is employed to unlock the pretrained LLM’s multimodal capability. To tackle practical issues in training and inference, the authors introduce several techniques: complementary masking for effective training, prefix KV caching for efficient inference, and timestep shifting for high-quality sampling. Experimental results show the method achieves comparable or slightly better performance than some autoregressive VLMs, with additional benefits in speed and controllability.", "questions": "1. The paper repeatedly emphasizes the speed-quality trade-off, but the focus should be on quantitative comparisons with AR VLMs. Specifically, assuming comparable or better quality, how much faster is the DM model compared to SoTA AR-based VLMs?\n- The COCO captioning task is not a good proxy for instruction-following benchmarks and may not fully reflect practical benefits.\n\n2. What is the maximum context length supported by the model?\n- If longer contexts (e.g., L = 4096) are required, would the speed advantage diminish significantly?\n- The limited generation length is a serious bottleneck for real-world applications.\n\n3. Bagel [ref1], under an AR framework, has shown parallel token output for image generation. Theoretically, it could also support parallel infilling of masked tokens—does this challenge the claimed advantages of the proposed approach?\n\n4. Why not fully mask the answer segment? Was there any ablation comparing cases like \"The [M] [M] dog.\" vs. \"[M] answer is [M] [M]\" vs. a fully masked \" [M] [M] [M] [M] [M]\"? What’s the impact of different masking strategies?\n\n5. The model uses fewer visual embeddings compared to recent VLMs like Qwen2.5-VL. If the number of visual tokens is increased to match state-of-the-art models, would the inference speed advantage still hold?\n\n6. InternVL-1.5 is fully open-sourced, and models like Mini-InternVL-Chat-2B/4B-V1.5 show significantly stronger overall performance than the proposed method. \n- Could the performance improvements observed in this paper be primarily attributed to the visual encoder rather than the diffusion-based architecture itself?\n\n[ref1] C. Deng, D. Zhu, K. Li, C. Gou, F. Li, Z. Wang, S. Zhong, W. Yu, X. Nie, Z. Song, G. Shi, and H. Fan. Emerging Properties in Unified Multimodal Pretraining. arXiv preprint arXiv:2505.14683, 2025.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- First VLM paradigm based on discrete diffusion models.\n- Addresses practical challenges in integrating VLM tasks into diffusion language models. The Prefix-DLM effectively accelerates inference in multimodal settings.\n- The paper is clearly written and easy to follow.\n\n**Weaknesses**\n- The model is limited in generation length, constraining its applicability.\n- While text-infilling is cited as an advantage over AR models, its practical usage scenarios are limited, making it a weaker justification.\n- There remains a notable performance gap compared to leading open-source AR-based VLMs.", "was_revised": true, "final_justification": "- First VLM paradigm based on discrete diffusion models.\n- Addresses practical challenges in integrating VLM tasks into diffusion language models. The Prefix-DLM effectively accelerates inference in multimodal settings.", "rebuttal": "**W1. Fixed-Length Generation Constraints of Diffusion Models**\n\nOverall, we recognize that fixed-length generation is a major constraint for diffusion language models. This limitation applies broadly across all such models. Notably, recent studies published after the submission deadline have begun to address this issue. For example, DreamOn [1] introduces special tokens that allow for expansion of masked tokens or deletion of unused ones. We will explore incorporating these advances in future work to address this limitation.\n\n**W2. Limited Use Cases for Text Infilling**\n\nWe incorporate additional results on text-infilling across three tasks shown in Fig. 6: JSON satisfaction, text editing, and visually grounded script completion. We randomly sample 100 images from the LAION-2B dataset and use GPT-4o to generate task-relevant questions for each image. We also ask GPT-4o to produce optimized prompts for VLMs for these tasks. For the text editing and script completion tasks, the prompts instruct the VLMs to edit a specific sentence or complete a specific line while leaving the rest of the text unchanged. For JSON completion, we validate schema correctness using Python's json package. For text editing and script completion, we use string matching to verify whether the required prefix and suffix are preserved in the generated text. The results are reported in the table below.\n\n|  | JSON Satisfaction | Text Editing | Visually Grounded Script Completion |\n| --- | --- | --- | --- |\n| *LaViDa* | 1.0 | 1.0 | 1.0 |\n| *LaViDa-FIM* | 1.0 | 1.0 | 1.0 |\n| *LLaVa-1.6-7B* | 0.89 | 0.58 | 0.49 |\n| *Qwen2.5-VL-7B* | 0.98 | 0.85 | 0.22 |\n\n\nThese results highlight the advantage of diffusion-based VLMs, which can provably achieve 100% constraint satisfaction without relying on careful prompt engineering. They also demonstrate the broader potential use cases for diffusion VLMs. We will incorporate more details of these evaluations in the final version.\n\n**W3. Performance Gap with SOTA**\n\nWe acknowledge that our model currently shows a performance gap with state-of-the-art VLMs, particularly on OCR tasks. One contributing factor is that diffusion language models typically require more training steps to converge than autoregressive (AR) models, as shown in a recent study [2]. To test this in our setting, we conducted an additional two epochs of training, which resulted in substantial improvements on both OCR tasks (e.g., ChartQA) and conventional vision tasks (e.g., MME). These results suggest that additional training can significantly improve OCR performance. We hope that future work on diffusion language models will explore strategies for improving training efficiency. It is also worth noting that state-of-the-art VLMs are generally trained on more data and for more steps, which presents opportunities for further scaling.\n\n| Model                | MME   | MMBench | ChartQA | DocVQA | InfoVQA | \n| -------------------- | ----- | ------- | ------- | ------ | ------- | \n| Paper                | 341.1 | 70.5    | 64.6    | 59.0   | 34.2    |\n| +Additional Training | **444.3** | **75.6**   | **74.9**    | 68.6   | **43.4**    |\n| Open-Llava-Next-8B   | 336.8 | 74.4    | 69.7    | **69.9**   | 36.7    |\n\n\n\n**Q1. Additional Speed-Quality Tradeoff**\n\nAs requested, we provide additional performance–latency results on MathVista in the table below. We report the maximum generation length (L), number of sampling steps, average number of non-padding tokens generated, per-sample latency, and throughput of non-padding tokens. Per review policy, we will include a visualization of these results similar to Figure 6 in the camera-ready version. To aid interpretation, we bold all metrics that outperform the AR baseline in the table. With optimal parameter choices, our model achieves up to a 2.5× speedup over the AR baseline.\n\n| Max Length | Steps | Avg Tokens | Acc(MathVista) $\\uparrow$ | Latency(s) $\\downarrow$| Throughput (tk/s) $\\uparrow$|\n| --- | --- | --- | --- | --- | --- |\n| LaViDa-LLaDa |  |  |  |  |  |\n| 1024 | 256 | 378 | **45.5** | 22.0 | 17.2 |\n| 512 | 128 | 333 | **43.5** | **10.05** | **33.1** |\n| 256 | 256 | 220 | **45.2** | 16.44 | 13.4 |\n| 256 | 128 | 217 | **42.9** | **8.25** | **26.3** |\n| 256 | 80 | 215 | **44.9** | **6.25** | **34.4** |\n| 256 | 60 | 211 | **44.6** | **5.17** | **40.8** |\n| 256 | 40 | 216 | **43.6** | **4.18** | **51.7** |\n| 256 | 20 | 221 | 33.2 | **3.42** | **64.6** |\n| Open-Llava-Next-8B |  |  |  |  |  |\n| NA | NA | 199.75 | 42.0 | 10.48 | 19.1 |\n\n\n**Q2. Context Length**\n\nThe context length is 4096 tokens. After accounting for vision and prompt tokens (~1000), the maximum possible generation length is around 3000 tokens. However, over 95% of our training data has generation lengths under 1000, which suffices for most tasks. As shown in the Q1 table above, even when AR models are allowed unlimited generation length, they typically produce only ~200 tokens on complex math reasoning tasks. When we set L = 1024 for LaViDa, it generates around 400 non-padding tokens. In summary, the current generation length is sufficient for most tasks, and within this range, the diffusion model demonstrates a speed advantage over AR models. We acknowledge that long-context generation remains underexplored for diffusion language models and hope that future work will address this area.\n\n**Q3. Bagel**\n\nBagel uses a continuous flow-matching objective with an MSE loss for image tokens, enabling parallel image generation. However, it uses autoregressive next-token prediction with a causal mask for language modeling (see Appendix Fig. 15 of their paper). Therefore, it does not support parallel language generation. We also note that it was posted on arXiv on May 20th, after our submission deadline.\n\n**Q4.Why Not Fully Mask the Answer Segment?**\n\nDuring inference, we decode a fully masked sequence into clean text tokens in a gradual manner. To support this process, it is important to train the model with varying mask rates. In practice, we sample a random mask rate between 0 and 1 during training. For mask rates less than 1, we must adopt a masking strategy that makes the best use of the training data. We already include an ablation study comparing complementary vs. random masking in Table 4a, with further discussion in Appendix B.4. We did not experiment with always masking the full sequence because that essentially reduces to learning a one-step generation model, which is extremely difficult.\n\n**Q5.If the number of visual tokens is increased to match state-of-the-art models, would the inference speed advantage still hold ?**\n\nWe extend the speed comparisons in Table 3a and Figure 4b to include an older AR model (LLaVA-1.5-7B), which uses low-resolution inputs (and thus fewer visual tokens). Our results show that LaViDa remains faster than AR models, even though it uses more visual tokens.\n\n|  | Res | CIDEr | Latency(s) |\n| --- | --- | --- | --- |\n| LaViDa | 768 | **114.8** | **1.23**|\n| Open-LlavaNext-8B | 1008 | 111.8 | 1.71 |\n| LLaVa-1.5-7B | 336 | 104.9 | 1.56 |\n\n**Q6(a). Intern-VL Vision Encoder**\n\nSmall models such as Mini-InternVL achieve superior performance by distilling knowledge from stronger models. For example, their 300M vision encoder is distilled from InternViT-6B, a significantly larger model. Therefore, this is not a fair comparison. We plan to explore distilled vision encoders in future work.\n\n**Q6(b). Could the performance improvements observed in this paper be primarily attributed to the visual encoder?**\n\nWe include ablations of different vision encoders in Appendix C, Table 11. We selected SigLIP for the final model due to its strong overall performance, although it is not the best on math tasks. For example, LaViDa-CLIP slightly outperforms LaViDa-SigLIP on MathVerse in our small-scale ablations (20.05 vs. 19.67). However, the full-scale model using the SigLIP encoder outperforms the baseline model using a CLIP encoder (27.2 vs. 14.6 in Table 1), suggesting that performance improvements do not stem solely from the vision encoder. In fact, our LaViDa-CLIP model trained on small-scale data still outperforms the AR baseline (19.67 vs. 14.6) using less data.\n\n\n[1] Zirui Wu*, Lin Zheng*, Zhihui Xie, Jiacheng Ye, Jiahui Gao, Yansong Feng, Zhenguo Li, Victoria W., Guorui Zhou, Lingpeng Kong (2025). DreamOn: Diffusion Language Models for Code Infilling Beyond Fixed-Size Canvas.\n\n[2] Prabhudesai, M., Wu, M., Zadeh, A., Fragkiadaki, K., & Pathak, D. (2025). Diffusion Beats Autoregressive in Data-Constrained Settings. arXiv preprint arXiv:2507.15857."}]

Author rebuttal:

Meta-review: The final ratings for this paper are unanimously positive  (one "Accept" and three "Borderline Accep").

The AC agrees with the reviewers' evaluation. This paper introduces LaViDa, a new exploration of vision-language models (VLMs) built upon discrete diffusion models. Reviewers acknowledged the originality of this work as one of the first to adapt diffusion models for multimodal instruction following.  Initial reviews shared several key concerns. The primary concerns included: 1) The performance gap with state-of-the-art AR models, particularly on OCR-related tasks. 2) The need for more comprehensive quantitative analysis of the claimed speed-quality tradeoff. 3) A lack of quantitative evidence for the practical advantages of text-infilling and controllability.  

The authors provided a rebuttal that effectively addressed these points. They presented new experiments, including detailed latency benchmarks demonstrating a speedup over standard AR models, additional training runs that narrowed the OCR performance gap, and new quantitative results that verified the advantages of their model in constrained generation tasks. 

Following the rebuttal, all reviewers confirmed that their major concerns had been resolved and maintained their positive ratings. While a performance gap with state-of-the-art AR models remains, the paper's value lies in its timely exploration of a new diffusion-based VLM architecture. Therefore, the AC recommends this paper for acceptance.
Decision: Accept (spotlight)