LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
Authors: name: Zebin You, name: Shen Nie, name: Xiaolu Zhang, name: Jun Hu, name: Jun Zhou, name: Zhiwu Lu, name: Ji-Rong Wen, name: Chongxuan Li

In this work, we introduce LLaDA-V, a purely diffusion-based Multimodal Large
Language Model (MLLM) that integrates visual instruction tuning with masked
diffusion models, representing a departure from the autoregressive paradigms
dominant in current multimodal approaches. Built upon LLaDA, a representative
large language diffusion model, LLaDA-V incorporates a vision encoder and MLP
connector that projects visual features into the language embedding space,
enabling effective multimodal alignment. Our empirical investigation reveals
several intriguing results: First, LLaDA-V demonstrates promising multimodal
performance despite its language model being weaker on purely textual tasks
than counterparts like LLaMA3-8B and Qwen2-7B. When trained on the same
instruction data, LLaDA-V is highly competitive to LLaMA3-V across multimodal
tasks with better data scalability. It also narrows the performance gap to
Qwen2-VL, suggesting the effectiveness of its architecture for multimodal
tasks. Second, LLaDA-V achieves state-of-the-art performance in multimodal
understanding compared to existing hybrid autoregressive-diffusion and purely
diffusion-based MLLMs. Our findings suggest that large language diffusion
models show promise in multimodal contexts and warrant further investigation in
future research. Project page and codes:
https://ml-gsai.github.io/LLaDA-V-demo/.