DS-VLM: Diffusion Supervision Vision Language Model
Authors: Zhen Sun
Yunhang Shen
Jie Li
Xing Sun
Pingyang Dai
Liujuan Cao
Rongrong Ji
Venue: ICML 2025 poster (2025)
Keywords: Vision language model;Diffusion Model
Primary area: deep_learning->large_language_models
TL;DR: We propose a novel diffuse supervised visual language model (DS-VLM) that directly utilizes the input image to supervise the visual encoder and connector via a diffusion model.

Abstract: Vision-Language Models (VLMs) face two critical limitations in visual representation learning: degraded supervision due to information loss during gradient propagation, and the inherent semantic sparsity of textual supervision compared to visual data. We propose the Diffusion Supervision Vision-Language Model (DS-VLM), a plug-and-play framework that introduces diffusion-based direct supervision for vision-language alignment. By reconstructing input images through a diffusion model conditioned on outputs of the visual encoder and the connector, our method establishes a short-path gradient propagation channel from pixel space to visual features. This approach simultaneously preserves high-level semantic alignment through conventional text supervision while enhancing visual feature quality via pixel-level reconstruction constraints. Extensive experiments conducted across various visual encoders and LLMs of different scales demonstrate the effectiveness of our approach.

VLM Diffusion Supervision Vision Language Model
Zhen Sun'
Yunhang Shen
Jie Li
Xing Sun
Pingyang Dai
Liujuan Cao
Rongrong Ji 14
Abstract
Vision- Assion-Lnnguage Models (VLMs) face two crit- ical limitations visual representation learning: degraded supervision due information loss dur- ing gradient propagation, and the inherent tic sparsity of textual supervision compared vi- sual data. We propose the Diffusion Supervision Vision-Language Model (DS-V VLM), a plug-and- play framework that introduces diffusion-based direct supervision for languaela alignment. By reconstructing imput images through a diffu- sion model conditioned outputs of the visual encoder and the connector, method establishes short-path gradient propagation channel from pixel space to visual features. This approach si- multaneously preserves high-level semantic align- ment through conventional text supervision while enhancing visual feature quality via pixel-level reconstruction constraints. Extensive experiments conducted across various visual encoders and LLMs of different scales demonstrate the effec- tiveness of our approach.
Connector
Encoder
E
Vision
LLM
Text
(a) The text knowledge propagation chain of current methods.
Vision Encoder
Connector
Diffusion
Image
(b) The image knowledge propagation chain of the proposed DS DS-V...
Figure I. The figure illustrates the knowledge propagation process of current methods and our proposed DS- VLM. comparison, the DS-VLM has shorter propagation chain and leverages the rich semantic details contained in the image.
1. Introduction
et al., 2023b) can be delineated into three components: the pre-trained vision encoder (Radford et al., 2021; Sun et al., 2023), the pre-trained LLM (Zhang et al., 2023c; Touvron et al., 2023; Chiang et al., 2023), and the connector (Alayrac et al., 2022; et al., 2023b; Liu et al., 2024b;a) trained from scratch to bridge the vision and language models.
Large language models (LLMs), exemplified by ChatGPT, have achieved remarkable progress in text understanding and generation. Meanwhile, multimodal large language models (MLLMs) have further extended the capabilities of LLMs by incorporating image understanding, evolving into models capable of integrating visual and textual modalities. This development has made MLLMs a new focal point for research and discussion.
The most critical factor affecting the performance of mod- els in understanding multi-modal content lies in the effec- tiveness of the vision encoder and connector in processing image features. Therefore, most current methods focus on improving the structure of either the Vision Encoder or the Connector components. For example, Cumo (Li et al., 2024) applies the Mixture Experts (MoE) mechanism to MILMS by integrating Top-K sparse-gating MoE blocks into the multilayer perceptron (MLP) connector. BLIP-2 (Li et al., 2023b) introduces Q-Former, which leverages set of Icarnable query vectors to filter and refine raw visual features, extracting visual information relevant to language tasks. Similarly, MG-LLava (Zhao et al., 2024b) integrates object-level features by incorporating bounding boxes iden-
In general terms, the architecture of mainstream MLLMs (Li
'Key Laboratory of Multimedia Trusted Perception and Effi cient Computing. Ministry of Education of China, Xiamen Univer- sity. 361005, P.R. China. 'Tencent YouTu Lab 'School of Infor- matics, Xiamen University. Xiamen, China. "Institute of Artificial Intelligence. Xiamen University.Xiame Xiamen, China. Correspondence to: Liujuan Cao <caoliujuan xmu.edu.cn>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

VLM: Diffusion Supervision Vision Language Model
Chtppa
Oulpad
L.M
LIM
I.IM
LeRA
L.M
Cunnechor
Weighted Sunt
Iifluskon
Connector
Quosson
Iregee
Cannectord
Connecteri
Furner
p
Diffeien Superiition
Object
Kanoder
Visio Encader
Encodlrr
Querie
Top-K Bouteer
PC
Quation
Image
Councettor
(a) BLIP2
(b) Cumo
(c) MG-LLava
(d) Our Method
Figure 2. The figure illustrates the existing mainstream methods and our proposed method. (a). (b), and (c) represent the existing mainstream methods, while (d) shows our proposed DS- VI. supervises the Vision Encoder and Connector by introducing a diffusion model during the training phase, which shortens the knowledge propagation chain and does not require the addition of extra components during the inference phase.
tified by an offline detector, aiming to enhance the model's object recognition capability, as illustrated in Figure 2c. In these methods, the optimization of the visual encoder and connector components typically relies on backpropagation through textual information during training, illustrated in Figure la. Specifically, during training, the gradients computed from the next-token prediction loss between the predicted answers generated by the LLM and the target text are backpropagated through the L.M and the connector to the visual encoder. However, this approach suffers from an excessively long knowledge propagation chain, particularly when the path traverses the L.L with its massive number of parameters, leading to significant optimization knowledge loss. This hinders effective parameter updates and optimiza- tion of these two components. Moreover, as the saying goes, "a picture is worth a thousand words," the information con- tained text is far less rich than that in images, especially in terms of semantic details related to image content, which text struggles to fully capture. Therefore, exploring methods to directly leverage image information for optimizing the visual encoder and connector components while shortening the knowledge propagation chain is highly significant yet challenging research direction.
the importance of the semantic information contained in the intermediate features of the vision encoder for enhancing the performance of vision-language models (VLMs). There- fore, we apply supervision the intermediate and final layer features of the vision encoder as well as the output features of the connector. To adapt the diffusion model to the distinct supervised features of the two modalities, we de- signed a Multi-Adapter Diffusion model based on the Stable Diffusion framework. This model can generate images con- ditioned on the supervised features. Finally, we construct a reconstruction loss to measure the differences between the generated image and the original image. Through optimiza- tion, this approach ensures that the extracted image features comprehensively capture the semantic, structural, and other critical information of the original image. During the in- ference phase, the Diffusion model is unnecessary, thereby avoiding the increase in number of model parameters and computational overhead compared to (Li et al., 2024: 2023b; Zhao al., 2024b). Extensive experiments demonstrate the effectiveness of the proposed design, showing significant improvements over the baseline across multiple datasets and frameworks.
The main contributions are summarized as follows:
Based on this, we propose a hlug-and-play method, Diffu- sion Supervision Vision Language Model (DS-VLM), which leverages pretrained diffusion model to supervise the vi- sion encoder and the connector during the training phase, as shown in Figure 2d. The knowledge backpropagation path in this method allows image information to propagate directly from the diffusion model to the vision encoder and connector components, bypassing the LLM, thereby signif- icantly shortening the propagation path. Specifically, we treat the features output by the vision encoder as image fea- tures and the features output by the connector as text feature since the connector maps image feature the space of text embeddings. Inspired by (Yao et al., 2024), we recognize
We propose the DS-VLM method, which introduces a diffusion model to leverage the rich semantic informa- tion of images for optimizing the visual encoder and connector through shorter knowledge propagation chain. Our method brings richer supervision signal and higher optimization efficiency, without extra cost for inference.
We design Multi-Adapter Diffusion to receive super- vised features from different modalities and reconstruct the original image. Serving as a medium for knowl- edge propagation, it transmits the semantic information contained in the image to the Vision Encoder and Con-
2

---

DS-: Difflusion Supervision Vision Language Model
nector components.
64x64 resolution and 1.5B text-conditional upsampling diffusion model at 256x256 resolution. DALL-E2 2 (Ramesh et al., 2022) employs diffusion model conditioned image embedding, and a prior model was trained to generate image embedding by giving a text prompt. DALL.-E 2 not only supports text prompt for image generation but also image prompt. To enhance the text understanding, Imagen (Saharia et al., 2022) adopts the T5 model (Raffel et al., 2020), a large transformer language model pretrained on text-only data, as the text encoder of diffusion model. Re-Imagen (Chen et al., 2022a) uses retrieved information to improve the fi- delity of generated images for rare or unseen entities. Stable Diffusion (SD) (Rombach et al., 2022) is built on the latent diffusion model (Rombach et al., 2022). which operates on the latent space instead of pixel space, enabling SD to generate high-resolution images with only a diffusion model. In this paper, we design Multi-Adapter Diffusion based on the SD) model, which can receive supervisory features from two modalities and generate images conditioned on them.
Extensive experiments conducted on various visual encoders and LLMSs of different scales validate the effectiveness of our method, demonstrating that the proposed DS-VLM not only brings substantial benefits but also provides valuable insights and approaches for the existing VLM research field.
2. Related Work
2.1. Vision Language Models
Vision-Language Models (VLMs) are designed to simulta- neously process visual and linguistic information. With the growing demand for multimodal tasks, VLMs have been widely applied to tasks such as image captioning (Chen et al., 2022b; Dai et al., 2023; Wang et al., 2024), visual question answering (VQA) (Xu et al., 2023; Yu et al., 2022), and image-text retrieval (Ge et al., 2024; Yang et al., 2024b) They represent prominent architectures the multimodal domain. Researchers have proposed numerous architec- tures (Li et al., 2023a; Zhu et al., 2024; Chen et al., 2023) for integrating visual features into advanced LLM inference pipelines. Llapt-Adapter (Zhang et al., 2023b) proposes to generate language answer with taking the image input as condition. Flamingo (Alayrac et al., 2022) and LLaVA (Liu et al., 2024b) blend visual tokens with text as inputs to LLM, differing in that Flamingo employs gating mecha- nisms inject encoded visual features into LLMs, while LLaVA directly concatenates visual and textual features at input. In addition, BIIP2 (Li et al., 2023b) introduces Q-Former, which extracts vision information relevant to language tasks through a set of learnable query vectors. Cumo (Li et al., 2024) applies MOE to MLLM, integrating Top-K sparse gated MOE blocks within a multi-layer per- ceptron (MLP) connector. MG-LLava (Zhao al., 2024b) introduces object-level features by incorporating bounding boxes detected by an offline detector. The optimization process of these methods involves propagating the textual information through the LLM and Connector, eventually reaching the Vision Encoder, resulting in a extended opti- mization chain. To address this, we propose an optimiza- tion method with shorter chain, DS- VLM, which uses Diffusion as an intermediary to directly propagate image knowledge to the Vision Encoder.
Aapters for Text-to-Image Models With the popularity of recent text-to-image models, adapters have also been used to provide additional control for the generation of text- to-image models. ControlNet (Zhang et al., 2023a) first proves that an adapter could be trained with pretrained eext-to-image diffusion model to learn task-specific input conditions, e.g., canny edge. Almost concurrently, T21- adapter (Mou et al., 2024) employs a simple and lightweight adapter to achieve fine- fine-grained control in the color and struc- ture of the generated images. To reduce the fine-tuning cost, Uni-ControlNer (Zhao et al., 2024a) presents multi-scale condition injection strategy to learn adapter for various local controls. In the updated version of T21-adapter, a style adapter is designed to control the style of generated images using a reference image by appending image features ex- tracted from the CLIP image encoder to text features. The global control adapter of Uni-ControlNet also projects the image embedding from CLIP image encoder into condition embeddings by small network and concatenates them with the original text embeddings, and it is used to guide the generation with the style and content of reference image. In this study, we introduce Multi-Adapter Diffusion, which in- corporates a separated cross-attentios mechanism to receive supervisory features from two modalities, enabling more effective image reconstruction.
3. Method
2.2. Diffusion Models
3.1. Preliminary
Texa-to-Image Diffusion Models Recently, diffusion mod- els (Sohi-Dickstein et al., 2015; Song et al., 2020a;b; Dhari- wal & Nichol, 2021) have emerged as the new state-of- the-art model for text-to-image generation. As a pioneer, GLIDE (Nichol et al., 2021) uses a cascaded diffusion ar- chitecture with a 3.5B text-conditional diffusion model at
As one of the most extensively adopted multi-modal LLM architectures, LLaVA consists of a vision encoder fv, an MLP projector fp. and large language model fz. Given a visual input V and textual input T, LLLVA computes the vision and language embeddings as per Eq. (I), where fr
3

---

DS-VLM: Diffusion Supervision Vision Language Model
Answer
LoRA
Large Language Model
Word Embedding
Connector
Vit
What kind of scene does picture depict?
Block2
Vison Adapter2
Block1
Vison
Text Adapter
Adapterl
Block0
Vison AdapterO
RL
Diffusion
Figure 3. The overall framework of the proposed DS-VLM It uses diffusion model to supervise the output features of the vision encoder and connector. The high-level, mid-level, and low-level features of the vision encoder are treated as image modality features, while the output features of the connector are treated text modality features. These features are input into their respective adapters and processed conditional features assist the diffusion model reconstructing the original image. represents the reconstruction loss.
represents the tokenizer of fL- The resulting embeddings, ET and Ev, are then concatenated into a single token se- quence, serving the input to the LLM. LLaVA utilizes Eq. (2) calculate the probability of the target answer XA, where 0 represents the trainable parameters and L is the length of XA- The model is trained visual instruction tuning data to maximize p(XAlV,T)
traverses an LLM with a large number of parameters, lead- ing to significant optimization knowledge loss. Moreover, the information contained in text far less rich compared to images. To address these issues, propose DS-VLMM as illustrated in Figure 3, which introduces Diffusion to su- pervise the vision encoder and connector components. The diffusion model reconstructs the original image based on the output features of these two components and optimizes them through reconstruction loss. Image knowledge directly (L) propagated these components via Diffusion, signif- icantly shortening the backpropagation path. At the same time, it leverages the richer semantic content that images can provide compared to text, thereby greatly improving the quality of the extracted features and enhancing the overall performance of the model.
E7 fr()), Ev folfv(V))
L V(X.V, II
po ( Concat(Ey, Ev, E bf
x x(i=1l)
(2)
3.2. Diffusion Supervision
Supervision Features First, it is essential to determine which features should be input into the diffusion model as conditions for reconstructing noise. Specifically, we classify the supervision features into two categories: image super- vision features and text supervision features. For image supervision features, inspired by (Yao al., 2024), which highlights the importance of intermediate layer features in the vision encoder for improving the understanding capa- bilities of vision-language models (VLMs), and to more comprehensively supervise the vision encoder features, we
Despite the encouraging results, most current methods (Li et al., 2023b; 2024; Zhao et al., 2024b) focus on optimiz- ing the structure of the visual encoder and connector com- ponents or adding additional components to enhance the performance of extracted features. The optimization of these two components typically relies on backpropagation through textual information during training. Specifically, gradients propagate through the LLM and Connector to the vision encoder. However, this results in an excessively long knowledge propagation chain, particularly when the path

---

DS-VLM: Diffusson Supervision Vision Language Model
extract low, mid and high layer features from the vision encoder as supervision features for image. The connector maps image features into the text feature domain, and there- fore, its output features naturally serve as the supervision features for text.
Vison Adapter
Cross Attention
Linear
LN
Aupte-Adapter Diffusion. Diffusion models are a general framework for generative modeling. In recent years, they have received widespread attention due their outstanding performance in the field of image generation. They consist of two processes: a diffusion process (also known as the for- ward process), which gradually adds Gaussian noise to the data using a fixed Markov chain of 7 steps, and a denoising process that generates samples from Gaussian noise with a learnable model. Diffusion models can also conditioned on other inputs, such as text the case of text-to-image diffusion models. Typically, the training objective of a diffu- sion model, denoted as ég, which predicts noise, is defined as simplified variant of the variational bound:
Cross Attention
Linear
LN
Text Adapter
Figure 4. Illustration of Multi-Adapter Diffusion, which employs multiple adapters to receive different supervised features and inte- grates these features into the Diffusion model through the MOE Cross Attention mechanism.
3.3. MOE Cross Attention
ssimpl = c.t,e ||e c,( c,
(3)
A straightforward method to incorporate image features is to concatenate the image features from various layers of the vision encoder with text features and then feed them into a cross-attentinn layer. However, we found this approach to be suboptimal. Instead, we propose an MOE (Mixture-of- Experts) cross attention mechanism, shown in Figure 4, where the cross attention layers for text features and image features from different layers are separated. The outputs of all these features are aggregated using an MOE routing mechanism. Specifically, for cach cross-attention layer in the original U-Net model, we add three new cross-attention layers to integrate low, mid and high level image features. Given the image feature G,, where i is the layer index. the output of new cross-attention Z is compute follows:
where 2a represents the real data with an additional condi- tion e, / E [0, 7 denotes the time step of diffusion process, It = 0gTo + OrE the noisy data at t step, and a₁, ot are predeffned functions of t that determine the diffusion process. In this study, we use the open-source SD model the base model to construct Multi-Adapter Diffusion. SD is a latent diffusion model conditioned on text features extracted from frozen CLIP text encoder. The architecture of the diffusion model based on a UNet (Ronneberger et al., 2015) with attention layers. In the original SD model, the text features from the CLIP text encoder are plugged into the UNet model by feeding into the cross-attention lay- ers. Given the query features Z and the text features C,, the output of cross-attention Z' can be defined by the following equation:
Q(K)
z = Attention Q, K, V) SSoftmax
Vd
(5)
QK T
Z' = Attention Q, K, V) = (Sofmax
V
(4)
where, Q ZW K' c. W and V/ W are the query, key, and values matrices from the i-th layer image features. W &s and W oi are the corresponding weight ma- trices. It should be noted that we use the same query for image cross-attention as for text cross-attention. In order to speed up the convergence, W ki and W are initialized from We & and W.. Then we employ the MOE routing al- gorithm compute the weights for the output feature of each cross-attention followed by a weighted summation to obtain the new feature representation. Accordingly, the final formulation of the MOE cross attention is defined as follows:
where Q = ZW,, q. K = c; Wx. V c; W, are the query, key, and values matrices of the attention operation respec- tively, and W,, WR. W,, are the weight matrices of the trainable linear projection layers. Inspired by SD model, We propose Multi-Adapter Diffusion, shown in Figure 4, which utilizes multiple adapters to receive and preprocess features from both image and text modalities. The pro- cessed features are then fed into the U-Net network through an MOE Cross Attention mechanism to assist the Diffusion model in reconstructing the original image. The MOE Cross Attention mechanism will be detailed in Section 3.3.
P Softmax(Pooling Q W,))
(6)
5

---

VLM: Diffusson Supervision Vision Language Model
Table 1. Performance comparison between various baselines and DS-VLMM. The best results hold.
L.M, VE, PT and IT denote
Language Model, Vision Encoder, pre- training data and instruction fine-tuning data, respectively.
Method
LM
VL
PT + II
MMB
MMS
MMMU
MV
OCRB
A12D
HB
LB
SQA
MME
Performance comparison against the baseline
LLaVA-1.5
Vicuna-7B
CLIP-L
0.5 M 0.6M
59.1
33.1
35.7
25.6
31.8
55.5
27.6
61.8
69.2
1808
DS- VL-M
Vicuna-7B
CLIP-L
0.5 0.6M
61.5
34,9
37.8
27.1
32.4
56.7
28.9
64.3
69.7
1783
LLaVA-1.5
Vicuna- 13B
CLIP-L
0.5 M
0.6M
64.0
34.3
37.0
27.7
33.7
61.1
24.5
66.1
72.6
1781
DL-
Vicuna- 13B
CLIP-L
0.5 M 0.6M
65.5
35.8
38.9
28.6
33.5
62.2
25.7
67.9
73.7
1836
Expanding to larger training datasets
LLaVA-1.5
Vicuna-7B
CLIP-L
1.2 M
1.5M
62.8
39.0
35.2
32.6
37.3
69.8
25.4
60.7
70.5
1810
DS-VLM
Vicuna-7B
CLIP-L
1.2
1.5M
63.6
39.3
37,5
33.2
36.8
70.1
25.2
61.4
71.3
1857
Expanding to robust visual encoder
LLaVA-1.5
Vicuna-7B
SigLIP-SO
0.5
0.6M
62.8
34.9
38,6
27.0
36.3
59.3
28.1
66.8
70.6
1764
DS VLM
Vicuna-7B
SigLIP-SO
M.5 + 0.6M
63.2
36.8
38.4
27.6
37,5
60.0
27.8
67.7
71.1
1814
Expanding to other LLMs
LL5
Llama3-8B
CLIP-L
0,5 M + 0.6M
66.7
38.5
40.7
26.7
33.4
61.8
27.4
64.3
74.8
1789
DS VLM
Llama3-8B 8B
CLIP-L
0,5 M 0.6M
66.3
40.4
41.5
28.3
32.5
62.0
28.6
65.1
74.3
1825
LLa
Qwen2-7B
CLIP-L
0.5 M 0.6M
70.9
42.1
43.6
32.2
33.6
65.3
28.3
65.9
74.2
1849
DS- VLM
Qwen2-7B
CLIP-L
0.5 0.6M
70.4
43.7
44.8
33.0
33.2
64.6
29.5
66.4
75.0
1953
lyze the results. Finally, we visualize the denoising process of the diffusion model to demonstrate the effectiveness of the proposed DS-VLM.
Z new = PoZ'= + Σ P.Z.
(7)
where W, E Rdxx is learnable parameter and d is the hidden dimension of the feature. P is a vector of size 4 and Po represents the weight of the output features from the textual Cross Attention, while PI, P2, and P3 represent the weights of the output features from the low, mid and high visual Cross Attention layers, respectively.
4.1. Experimental Settings
Implementation Details We implement the proposed im- provement strategy on top of LLaVA-1.5 (Liu et al., 2024a), whose general applicability in the VLM field facilitates the validation of our method's versatility. Specifically, we maintain consistency with LLaVA-1.5 by employing CLIP- ViTL/14-336px the visual encoder. To further validate the generalizability of our proposed method, we also in- corporate SigLII-S0400m-pp catnn-pathh1443330 14-384 (Zhai et al., 2023), another leading choice, for comparative analysis. In terms of LLM, we compare our method against the baseline using Vicuna-7/13B and extend our approach to Llama3-8B (Meta, 2024) and Qwen2-7B (Yang et al., 2024a), thereby demon- strating the versatility of our method. For training config- urations. we adhere strictly to the settings outlined in the original LLaVA-1.55 paper to ensure fairness, with learning rates of le-3 and 2e-5 for pre-training and instruction fine- tuning phases, respectively, and maintaining batch sizes of 256 and 128. During the LoRA fine-tuning process, the rank of all linear layers uniformly set to 8. We select the 8th, 16th, and 24th layers of the Vision encoder as the feature representatives of the low, mid and high layers, respectively. The number of iterations for the diffusion model is 50, The training process for DS- VLM utilizes the Py PyToc frame- work.
3.4. Reconstruction Loss
As diffusion models restore images from noise, selecting an effective loss function to measure the difference between the original and generated images becomes critical. Here, we attempt to construct reconstruction losses at three levels of granularity: low-level, mid-level and high-level. Specifi- cally, we employ mean absolute error (MAE) for the pixel level, structural similarity index (SSIM) for the structural level, and perceptual loss (PL) for the semantic level. By comparing the effects of these three reconstruction losses, we aim comprehensively evaluate the model's reconstruc- tion performance and identify the most suitable loss function to optimize the model. Unless otherwise specified, all the experiments and discussions in this paper are conducted using the perceptual loss.
4. Experiment
In this section, we first present the detailed experimental setup of our study. We then enumerate the improvements brought by our proposed DS- VLM over the baseline across multiple evaluation metrics, and compare our method with several state-of-the-an art (SoTA) approaches under various configurations. We also conduct ablation studies and ana-
Datasets Focusing on proposing a novel optimization method for the VLM framework, we do not incorporate any additional data beyond the LLaVA-1.5 open-source dataset (Liu et al., 2024a), which includes 558K image captions for pre-training and 665K conversations for in-
6

---

DS-: Diffusion Supervision Vision Language Model
Table 2. Comparison with SoTA methods. The best results are bold and the second-best results are underlined.
Method
I.
VE
PT
MMB
MMS
MMMU
MV
OCRB
A12D>
HB
LB
SQA
MMF
MiniGPT4
Vicuna-7B
EVA-G
+ 3.5K
20.8
16.3
23.6
20.4
17.2
28.4
31.9
45.1
39.6
1047
Qwen- VL
Qwen-7B
VIT-G/I6
1.4 B + 50M
32.9
32.5
29.6
15.5
12.7
57.7
29.9
12.9
61.1
483
VisualGLM
ChatGLM-6B
EVA-CLIP
330 M
35.7
25.9
29.9
21.9
17.0
41.2
25.0
37.3
56.1
738
PandaGPP
Vicuna-138
IB-H
160 K
34.5
25.6
32.9
25.0
26.9
48.3
21.6
57.2
61.8
1076
mPLUG-OWI2
JJama 2-7B
CLIP-L
348 M + 1.2M
60.8
34.8
34.7
25.4
25.5
55.7
29.4
59.9
69.5
1786
Emu2-chat
L33ma-33B
EVA-CLIP
52.8
40.7
35.0
30.7
43.6
49.7
29.5
56.4
68.2
1678
Yi-VL
Yi-6B
CLIP-L
100 M 26M
64.2
33.7
40.3
29.7
29.0
59.8
36.0
51.9
72.6
1915
ShareGPT-4V
Vicuna-7B
CLIP-L
1.2 M. 0.7M
61.6
35.7
37.2
26.5
37.1
58.0
28.6
66,9
69.5
1914
LLaVA-1.5
Qwen2-7B
CLIP-L
0.5 M 0.6M
70.9
42.1
43.6
32.2
33.6
65.3
28.3
65,9
74.2
1849
DS-VLM
Vicuna-7B
SigLIP-SO
0.5 M + 0.6M
63.2
36.8
38.4
27.6
37.5
60.0
27.8
67.7
71.1
1814
DS-VLM
Vicuna-7B
CLIP-L
1.2 M + 1.5M
63.6
39.3
37.5
33.2
36.8
70.1
25.2
61.4
71.3
1857
DS-VLM
Llama3-8B
CLIP-L
0,5 M + 0.6M
66.3
40.4
41.5
28.3
32.5
62.0
28.6
65.1
74.3
1825
DS-VLM
Qwen2-7B
CLIP-L
0.5 M + 0.6M
70.4
43.7
44.8
33.0
33.2
64.6
29.5
66.4
75.0
1953
struction tuning. We also apply our proposed method to the Mini-Gemini dataset (Team), which consists of 1.2M + 1.5M data, to further highlight the superiority of our approach. For evaluation, we conduct extensive experiments on widely-adopted VLM benchmarks, pro- viding robust and comprehensive performance validation for the proposed DS- V..... The evaluation datasets in- clude: MMBench (MMB) (Liu al., 2025), MMS (MM- Star) (Chen et al., 2024), MMMU (Yue et al., 2024), MV Vista) (Lu et al., 2023), OCRB (OCRBench) (Liu et al., 2023), A12D (Hiippala et al., 2021), HB (Hallu- sionBench) (Guan et al., 2024), LB (LLaVABench) (Liu et al., 2024b), SQA (ScienceQA) (Saikh et al., 2022), and MME (Fu al., 2024).
sual feature representations. MMBench emphasizes comprehensive multimodal understanding, including layout reasoning and structured chart interpretation. Our method's multi-level supervision effectively im- proves the model's ' sensitivity to spatial and structural details. Furthermore, L.L.VABene emphasizes not only multimodal alignment but also the model's abil- ity to perform knowledge grounding in complex real- world scenarios. DS-VLM addresses these challenges by introducing pixel-level reconstruction supervision, which enhances the semantic richness and fidelity of visual features. For VL- with Vicuna-13B. we also achieve an average performance improvement of 1 %. Specifically, we see +1.5 gain on MMStar, a +1.9% gain on MMMU, and +1.2% gain on Hal- lusionBench. The improvement on MMStar, which emphasizes fine- fine-grained multimodal alignment and reasoning under instruction -ostuuction-oolloiing settings, demon- strates DS- ability to preserve semantic coher- ence across modalitics through enhanced visual feature quality. Meanwhile, HallusionBench is designed to assess a model's 's robustness against hallucinated or misleading visual-linguistic content. The stronger vi- sual representations induced by diffusion supervision help DS- VLM better anchor its predictions to grounded visual evidence, effectively reducing semantic drift and hallucination. These impressive results further validate the contribution of the proposed DS-VLM architec- ture to visual feature optimization, highlighting the favorable impact of our method.
4.2. Improvement Over the Baseline
In table 1, we present the performance improvements of the proposed method across various configurations compared to the baseline. According to the experimental results, we can draw several phenomenons:
The proposed DS-VLM demonstrates substantial im- provements over the baseline. Compared with the original LL. DS-VLM achieve much better performance. As shown in the first four rows. our method leads on the majority of evaluation datasets. It is noteworthy that DS- demonstrates aver- age improvement of 1.4% over the original VL1.5 across ten datasets when using Vicuna-7B, highlight- ing the method's significant value. When compared with the baseline LLaVA-1.5 Vicuna-7B model, we enhance performance metrics by +2.4% on MMBench, +2.1% on MMMU, and +2.5% on LLa VABenchs,, re- spectively. Specifically, MMMU involves complex vi- sual reasoning across multidisciplinary domains such as medicine and engineering, which requires precise interpretation of structural visual content and domain- specific symbols. DS- VS-V excels in such scenarios by leveraging pixel-level reconstruction to enhance vi-
The proposed Diffusion- supervision method can be widely applied as a modular plugin within main- stream VLM frameworks. As shown in the rest part of Table 1. we further validate the versatility of our proposed method under various settings. We replace CLIP with SigLIP and substitute Vicuna with Llama3 and Qwen2 on top of the original LLaVA-1.5 frame- work. We compare these settings with our method as the baseline. The results in Table confirm that
7

---

DS-: Diffusson Supervision Vision Language Model
Table Ablation study results different types of supervision features, where TS represents text modality supervision and IS represents image modality supervision.
Components
No.
MMB
MMS
MMMU
MV
OCRB
A12D
HB
LB
SQA
MME
TS
IS (High)
IS (Mid)
IS (Low)
0
64.0
34.3
37.0
27.7
33.7
61.1
24.5
66.1
72.6
1781
1
/
64.3
34.9
37.4
28.0
32.8
61.4
24.8
66.5
72.9
1798
2
/
64.8
35.2
38.0
27.9
33.1
61.6
24.6
66.9
72.8
1804
3
/
65.2
35.5
38.3
28.1
33.3
61.9
25.0
67.3
73.3
1820
4
65.0
35.5
38.7
28.4
33.3
62.0
25.5
67.6
73.6
1827
5
65.5
35.8
38.9
28.6
33.5
62.2
25.7
67.9
73.7
1836
our method continues to maintain a leading advantage across most datasets, demonstrating that the proposed DS-VLM exhibits excellent generalizability and pos- sesses strong potential for adaptation a wide range of VLM architectures.
Reconstruction Loss Ablation Additionally, to explore which reconstruction loss performs best in improving model performance, we conducted ablation experiments on recon- struction losses, with the results shown in Table 4. Specif- ically, the model uses Vicuna-7B as the LLM and CLIP-L as the visual encoder. We evaluated reconstruction losses at three levels of granularity: Mean Absolute Error (MAE), Structural Similarity Index (SSIM), and Perceptual Loss (PL). The experimental results across five datasets show that Perceptual Loss (PL) consistently achieved the best perfor- mance. This indicates that Perceptual Loss can help the supervised features retain more image details and semantic information, thereby enhancing the model's performance.
4.3. Quantitative Comparison with SoTAs
We further compare our method with several leading ap- proaches. The methods included in the comparison are MiniGPT4 (Zhu et al., 2023), Qwen-VL. (Bai et al., 2023), VisualGLM (GLM et al., 2024), PandaGPT (Su et al., 2023), mPLUG-Owl2 (Ye et al., 2024), Emu2-chat (Sun et al., 2024), Yi-VL (Young et al., 2024) and ShareGPT-4V (Chen
et al., 2025). Table 2 presents the performance comparison across multiple benchmarks.
Table 4. Ablation study results on different reconstruction losses.
MAE, SSIM and PL denote mean absolute error. structural simi- larity index and perceptual loss, respectively.
Remarkably, despite relying solely on settings from LLa VA- 1.5, DS-VLM achieves performance that matches surpasses the benchmarks set by leading SoTA methods, with a comparatively smaller volume of pretraining and instruction fine-tuning data.
RL
MMB
MMS
MMMU
MV
A12D
SQA
MAE
60.4
33.5
36.1
25.8
55.4
68.7
SSIM
61.1
34.3
36.6
26.2
56.0
69.3
PL
61.5
34.9
37.8
27.1
56.7
69.7
4.4. Ablation Studies
Ablation Study on the Effectiveness of the Independent Cross-Attention Mechanism To further assess the effec- tiveness of our proposed Independent Cross-Attentie Attention mech- anism, we conduct an ablation study comparing it against the commonly used Shared Cross-Attention strategy. As illustrated Table 5, the independent crossaattention design consistently outperforms the shared variant across all bench- mark evaluations. These performance gains primarily attributed to the modular separation of attention heads for textual and visual features, which facilitates more precise and independent modeling of modality-specifft semantics. In contrast, the shared cross-attention approach is more susceptible modality interference, thereby reducing the efficiency of feature fusion. The results demonstrate that the Independent Cross- mechanism enables more robust multimodal alignment and enhances the model's rea- soning and comprehension capabilities across diverse set of tasks.
Supervised feature ablation The ablation study of super- vised features is shown in Table 3. The model selects Vicuna-13B as the Language Model and CLIP-L as the Vision Encoder. The experimental results demonstrate that combining text supervision (TS) and image supervision at different levels (IS) significantly improves the model's per- formance. Text supervision (TS) alone already leads to improvements, and progressively adding high-level, mid- level, and low-level image supervision further enhances the model's performance, particularly in terms of detail capture and global semantic understanding. The optimal combina- tion using text supervision along with multi-level image supervision (TS IS High IS Mid + IS Low), which per- forms best across multiple datasets, especially achieving the highest scores on metrics such as SQA and MME. This indicates that multi-level supervision from both text and image modalities can complement each other, comprehen- sively enhancing the model's multi-tase learning ability and multimodal understanding capability.
Ablation Study on Adapter Architecture To evaluate the effectiveness of our proposed linear adapter, we conduct an
8

---

VL-: Diffusion Supervision Vision Language Model
process fully demonstrates that supervised features play crucial role in the generation process of the diffusion model. They provide rich semantic information, enabling the model to effectively recover image details and global structure during the denoising process. Experimental results further validate that the proposed DS-VLM framework can learn visual representations more efficiently through shorter knowledge propagation chain, thereby achieving superior performance in image understanding tasks.
Table Ablation study results of different Cross-Attention mecha- nisms. SCA and ICA denote Shared Cross Attention and Indepen- dent Cross Attention, respectively.
Method
MMB
MMS
MMMU
MV
A12D
SQA
SCA
64.1
34.4
37.6
28.3
61.7
73.4
ICA
65.5
35.8
38.9
28.6
62.2
73.7
ablation study by replacing it with the Q-Former adapter utilized in BLIP-2 (Li et al., 2023b). As shown in Table 6, our linear adapter consistently outperforms the Q-Former- based design across six diverse vision-language benchmarks. Unlike Q-Former, which introduces additional query token mechanisms, the linear adapter employs more parameter- efficient structure without increasing architectural complex- ity. These results validate the superiority of our lightweight and efficient adapter design.
5. Conclusion
In this paper, we propose a novel Diffusion -uppeiii-- Vision-Language Model (LS-VLM), which addresses the issue of information loss caused by the lengthy propagation path of text-based supervision signals in traditional meth- ods by directly leveraging input images supervise the visual encoder and connector through diffusion model, Ex- perimental results demonstrate that DS- VLM significantly improves the quality of feature extraction and achieves su- perior performance across various tasks and datasets. This approach provides an efficient and generalizable solution for multimodal learning, and its potential applications in broader scenarios be further explored in the future.
Table 6. Ablation study results of different adapter structures.
Method
MMB
MMS
MMMU
MV
A12D
SQA
Q-Fomerr
64.3
34.6
37.4
28.0
61,5
73.1
MLP(Ours)
65,5
35.8
38.9
28.6
62,2
73.7
Acknowledgements
This work was supported by the National Science Fund for Distinguished Young Scholars (No.62025603), the Na- tional Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. U23A20383, No. U21A20472, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), and the Natural Science Foundation of Fujian
Province of China (No. 2021J06003, No.2022106001).
Initial moise
20 timesteps
timesteps
fimesteps
Original Image
Impact Statement
Figure The figure illustrates the process by which the diffusion model gradually reconstructs a clear image from random noise over multiple timesteps.
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
4.5. Qualitative Analysis via Visualization
To verify the effectiveness of our proposed method, we conducted a visualization analysis of the image generation process the diffusion model under given supervised fea- tures, as shown in Figure Specifically, we present the image evolution at different stages of the denoising process to illustrate how the diffusion model progressively restores high-quality images. Starting from an initial random noise image, as the denoising timesteps progress, structural infor- mation gradually emerges, contours become clearer, details are continuously enriched, and eventually, a high-quality im- age with complete semantic information is generated. This
References
Alayrac. J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few- shot learning. Advances in neural information processing systems, 35:23716-23783 2022.
Bai, J., Bai, S., Yang, S., Wang. S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: frontier large vision- language model with versatile abilities. arXiv preprint

---

DS-VLM: Diffusson Supervision Vision Language Model
trXiv:2308.12966 2023.
Guan, T., Liu, F,, Wu, X., Xian, R., Li, Z.,, Liu, X., Wang, X., Chen, L.,, Huang, F., Yacoob, Y., et al. Hallusionbench: advanced diagnostic suite for entangled language hal- lucination and visual illusion in large vision-language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14375- 14385, 2024.
Chen, L.. Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J.. Qiao, Y., Lin, D., et al. Are we on the right way for evaluating large lision-languae models? arXiv preprint arXiv:2433.20333 2024.
Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pp. 370-387. Springer, 2025.
Hippaaa, T., Alikhani, M., Haverinen, J., Kalliokoski, T., Logacheva, E.,, Orekhova, S., Tuomainen, A., Stone, M., and Bateman, J. A. Ai2d-rst: A multimodal corpus of 1000 primary school science diagrams. Language Re- sources and Evaluation, 55:661-688, 2021.
Chen, W., Hu, H., Saharia, C., and Cohen, W. W. Re- imagen: eetrieval-augmentt text-to-image generator. arXiv preprint arXiv:2209. /4491, 2022a.
Langley, Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000). pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Gryener, A., Mustafa, B., Beyer, L.., et al. Pali: A jointly-scaled multilingual aegguage-imagee model. arXiv preprint arXiv:2209.0677 2022b.
Li, B., Zhang, Y., Chen, L.., Wang, J., Pu, F. Yang. J., Li, C., and Liu, Z. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint NKKi:22306004222 2023a.
Chen, X., Wang. X., Beyer, L., Kolesnikov, A., Wu, J., Voigtlaender, P., Mustafa, B., Goodman, S., Alabdul- mohsin, I., Padlewski, P., et al. Pali-3 vision lan- guage models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199, 2023.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-imasn image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19744 PMLR, 2023b.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E.,, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See hutps:/vvicuna, Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Li, J., Wang, X., Zhu, S., Kuo, C.-W., Xu, L.,, Chen, F., Jain, J., Shi, H., and Wen, L. Cumo: Scaling multimodal Ilm with co- co-upeyeled arXiv:2405.000 05999, 2024.
arXiv preprint
Liu, H., Li, C.. Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 86296-26306 2024a.
Dai, W., Li, Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B. A., Fung, P., and Hoi, S. C. Instructblip: Towards general-purpose purpose vision-lange languug models with instruction tuning. arxiv abs/2305. hls/2305.06500 (2023), 2023.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024b.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780 2021.
Liu, Y.. Li, Z., Yang, B., Li, C., Yin, X., Liu, C.-L., Jin, L., and Bai, X. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.078 077099 2023.
Fu, C., Zhang, Y.-F. Yin, S., Li, B., Fang, X., Zhao, S., Duan, H.. Sun, X., Liu, Z., Wang, L., et al. Mme-survey: A comprehensive survey on evaluation of multimodal lims. arXiv preprint arXiv:241 1.2266, 2024.
15296,
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang. J., He, C., Liu, Z., et al. Mmbench: Is your multi- modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer, 2025.
Ge, H., Jiang, Y., Sun, J., Yuan, K., and Liu, Y. LIm-enhanced composed image retrieval: An intent uncertainty-awar linguistic- visual dual channel match- ing model. ACM Transactions on Information Systems, 2024.
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models visual contexts. arXiv preprint arXiv:2310.0225, 2023.
GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Zhang, D., Rojas, D., Feng, G., Zhao, H., et al. Chatglm: A family of large language models from glm- 130b to glm-4 all tools. arXiv preprint arXiv:2406. /2793, 2024.
10

---

DS-:LM: Diffusson Supervision Vision Language Model
Meta, A. Introducing meta Ilama 3: The most capable openly available IIm to date. Meta A1, 2024.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on PMLR, 2015.
Mou, C., Wang, X., Xie, L., Wu, Y., Zhang, J., Qi, Z., and Shan, Y. T2i-adapter: Learning adapters to dig out more controllable ability for -xx-tt-imaae diffusion models. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 38, pp. 4296 4304, 2024.
machine learning, pp.
Song. J., Meng, C., and Ermon, S. Denoising diffusion im- plicit models. arXiv preprint arXiv:2010.02502 2020a.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.1344 2020b.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P. McGrew, B., Sutskever, I,, and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
Su, Y. Lan, T., Li, H., Xu, J.. Wang, Y., and Cai, D. Pandagpt: One model to instruction-foll followw them all. arXiv preprint arXiv:2305.16355 2023.
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learming, pp. 8748-8763, PMLR, 2021.
Sun, Q., Fang, Y., Wu, L.., Wang, X., and Cao, Y. Eva- clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303. /5389, 2023.
Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q. Wang, Y., Rao, Y., Liu, J., Huang, T., and Wang, X. Generative mul- timodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14398-14409, 2024.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Team, G. G. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context (2024). URL hutps://goo. gle/GeminiV1-5.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical cext-conditional image generation with clip latents. arXiv preprint 22Xiv22204.06/55 1(2):3 2022.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288: 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent difffusion models. In Proceedings of the IEEE/CVF con- ference computer vision and pattern recognition, pp. 10684-10695, 2022.
Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y., et al. Visionllm: Large language model is also an open-ended decoder for vision- centric tasks, Advances in Neural Information Processing Systems, 36, 2024.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image compuling and computer-ussisted intervention-MICCS 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015.
Xu, H. Ye, Q., Yan, M., Shi, Y., Ye, J., Xu, Y., Li, C., Bi, B., Qian, Q., Wang. W., et al. mplug-2: A modularized multi-modal foundation model across text, image and video. In International Conference on Machine Learning, pp. 01722-38748.... PMLR, 2023.
Saharia, C., Chan, W., Saxena, S., Li. L., Whang. J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in neural information processing systems, 35: 36494, 2022.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D.,, Huang, F,, Wei, H., et al. Qwen2. 5 technical report. arXiv preprint 1rXiv:2412.153 151 15, 2024a.
Yang, Z., Xue, D.,, Qian, S., Dong, W., and Xu, C. Ldre: Llm-based divergent reasoning and ensemble for zero- shot composed image retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 80-90, 2024b.
Saikh, T., Ghosal, T., Mittal, A., Ekbal, A., and Bhat- tacharyya, P. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):289-300 2022.

---

DS- Diffusion Supervision Vision Language Model
Yao, H., Wu, W. Yang, T., Song, Y., Zhang, M., Feng, H., Sun, Y., Li, Z.,, Ouyang, W., and Wang. J. Dense connector for milms. arXiv preprint arXiv:2405.1380 2024.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2300 10592, 2023.
Ye. Q. Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q. Zhang, J., and Huang, F. mplug-ow12: Revolution- izing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition, pp. 13040-13051, 2024.
Zhu, L., Ji, D., Chen, T,, Xu, P., Ye, J., and Liu, J. lbd: Alleviating hallucinations in large vision-language models via image-biased decoding. arXiv preprint arXiv:2402. 2en::2402.847700 18476, 2024.
Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., et al. Yi: Open foundation models by 0l. ai. arXiv preprint WrXiv2243.046622 2024.
Yu, J., Wang, Z., Vasudevan, V., Yeung, L.,, Seyedhosseini, M., and Wu, Y. Coca: Contrastive captioners are image- text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G.,, Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. Mmmu: A massive multi- muuti-dissciplie multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9556-9567, 2024.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In Proceed- ings of the IEEE/CVF international conference com- puter vision, pp. 11975-11988 2023.
Zhang, L., Rao, A., and Agrawala. M. Adding conditional control to text-to-image diffusion models. Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023a.
Zhang, R., Han, J., Liu, Gao, P., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., and Qiao, Y. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.1 16199, 2023b.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al.
Opt: Open pre-trained transformer language models, 2022. URL https:/arxiv. org/ubs/2205.0/ 200000002000.00888, 3:19-0, 2023c.
Zhao, S., Chen, D., Chen, Y-C.,, Bao, J., Hao, S., Yuan, L., and Wong, K.-Y. K. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024a.
Zhao, X., Li, X., Duan, H., Huang, H., Li, Y., Chen, K., and Yang, H. Mg-llava: Towards multi- mulliigrranularity vi- sual instruction tuning. arXiv preprint axXiv:2406.17770 2024b.
12

---

DS-VLM: Diffusion Supervision Vision Language Model
A. Comparative Experiments with LA with Trainable Encoder
To further validate effectiveness of our proposed DS-VLM under the setting of a trainable visual encoder, we conduct an additional set of comparative experiments to evaluate the overall performance of DS-VLM VLM against the mainstream method LLaVA-1.5 when the visual encoder is trainable. Specifically, we perform comprehensive evaluations under two different visual encoder configurations CLIP-L and SigLIP-SO while keeping the language model (Vicuna-7B) and the training data (0.5M + 0.6M) consistent. As shown in Table 7, DS-V DS-VLM significantly outperforms LLaVA-IA across most evaluation benchmarks, This demonstrates that our proposed diffusion basedd supervision mechanism can more effectively optimize the visual encoder through a shorter gradient propagation path, thereby improving the quality of multimodal semantic alignment and enhancing the model's ' overall reasoning capabilities.
Table Comparative Study with LLaVA Framework Equipped with a Trainable Encoder
Method
LM
VE
PT + IT
MMB
MMS
MMMU
MV
OCRB
A12D
HB
LB
SQA
MME
LLaVA-L.5
Vicuna-7B
CLIP-L
0.5M 0.6M
59.4
33.6
36.2
26.3
31.5
55.8
27.8
62.2
68.9
1817
DS-VLM
Vicuua-7B
CLIP-L
0.5M + 0.6M
61.5
34.9
37.8
27.1
32.4
56.7
28.9
64.3
69.7
1783
LLaVA-1.5
Vicuua-7B
Sig
0.5M 0.6M
62.6
35.2
38.1
27.5
67.0
70.4
1782
DS- DS-VLM
Vicuna-7B
SigLIP-SO
0.5M + 0.6M
63.2
36.8
38.4
27.6
37.5
60.0
27.8
67.7
71.1
1814
B. Validating the Critical Role of the Vision Encoder in the Training Process
To verify that the optimization process in the DS-VLM framework is solely dominated by the adapter module, conducted controlled experiment comparing two configurations: training only the adapter (with the vision encoder frozen) and jointly training both the vision encoder and the adapter. As shown in table 8, the reconstruction loss was 0.5 when only the adapter was trained, whereas it significantly decreased to 0.1 when both the vision encoder and the adapter were trainable. This result indicates that relying solely on the adapter insufficient for effective visual reconstruction, and that the vision encoder indeed learns meaningful representations during training. This experiment effectively rules out the hypothesis that the adapter dominates the training process and further validates the effectiveness and necessity of our proposed diffusion supervision mechanism in optimizing the vision encoder.
Table Reconstruction Loss under Different Freezing Strategies
Method
Reconstruction Loss
Frozen E Adapter Trainable Encoder+Trrinabl Adapter
0.5
0.1
13


Reviews:
[{"review_id": "nRP9KpaiEk", "reviewer": "Reviewer_gwC3", "summary": "The authors design a framework to train the visual encoders in VLMs, with diffusion supervision. The authors design a mechanism to incorporate visual features from multiple levels of the encoder to reconstruct the image using a frozen diffusion model during training.  The authors evaluate their method, built on top of existing VLMS, on various VLM benchmarks and demonstrate some improvement over the VLM by itself.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The improvements are marginal ~1-2% and not \"substantial\" as claimed by the authors. Also for many benchmarks there is no improvement at all in Table 1.\n> \n\n> The authors need to analyse this inconsistency in performance a cross the different benchmarks.\n> \n\nA1: Thank you for pointing out the modest improvements and performance inconsistency. We understand your concern and clarify that DS-VLM is not just about fine-tuning gains, but proposes a new optimization paradigm: diffusion-based image reconstruction supervision. It delivers dense semantic signals from pixel space to the encoder and connector, shortening the gradient path in traditional text supervision and mitigating semantic sparsity, thus enhancing structural and semantic representation. This is particularly effective on datasets like MMMU, MMBench, and MMStar. In OCRB tasks requiring high-res text recognition, our reconstruction focuses on structure and semantics, lacking detail for small text. We plan to explore enhanced text-area reconstruction in future work. DS-VLM adds no extra inference cost, ensuring practicality and scalability. We will include this analysis in the final version.\n\n> Do the authors roll out the diffusion process during training to backpropagate gradients. This needs to be clarified in the paper.\n> \n\nA2: The gradient backpropagation path in DS-VLM follows: Image Reconstruction Loss → Diffusion → Adapter → Vision Encoder. Below, we derive the backpropagation process of the model. For simplicity, we set the number of diffusion iterations to 2. We begin by defining:\n\n$z_e=E(f;a)$\n\n$z_a=A(z_e;b)=A(E(f;a);b)$\n\n$z_{d1}=D(x,z_a;c)=D(x,A(E(f;a);b);c)$\n\n$z_{d2}=D(z_{d1},z_a;c)=D(D(x,A(E(f;a);b);c),A(E(f;a);b);c)$\n\n$L=L(z_{d2})$\n\nHere, $E$ denotes the encoder with parameters $a$, $A$ denotes the adapter with parameters $b$, and $D$ denotes the diffusion module with parameters $c$. $L$ denotes the reconstruction loss.\n\nAccording to the chain rule, the gradient of the Adapter parameter can be deduced as:\n\n$\\frac{\\partial L}{\\partial b}=\\frac{\\partial L}{\\partial z_{d2}}[\\frac{\\partial D(z_{d1},z_a;c)}{\\partial z_{d1}}\\cdot \\frac{\\partial D(x,z_a)}{\\partial z_a}+\\frac{\\partial D(z_{d1},z_a)}{\\partial z_a}]\\cdot \\frac{\\partial A}{\\partial b}$\n\nThe gradient of the Encoder parameters is：\n\n$\\frac{\\partial L}{\\partial b}=\\frac{\\partial L}{\\partial z_{d2}}[\\frac{\\partial D(z_{d1},z_a)}{\\partial z_{d1}}\\frac{\\partial D(x,z_a)}{\\partial z_a}+\\frac{\\partial D(z_{d1},z_a)}{\\partial z_a}]\\frac{\\partial A(z_e;b)}{\\partial z_e}\\frac{\\partial E(f;a)}{\\partial a}$\n\nI hope the above can answer your questions. We look forward to further communication.\n\n> The authors add a bunch of new layers to feed the multi-level features into the diffusion model. I assume these layers are not used during inference since their only purpose is to adapt the features for diffusion model. Seems to me that this could result in the primary encoder not learning anything useful since the new layers(unused during inference) maybe doing most of the heavy lifting. Have the authors figured out a way to solve this issue ?\n> \n\nA3: To address concerns about the Adapter dominating learning, we ran a controlled experiment (Table 1 below) comparing: (1) freezing the encoder and (2) training both. Training both gave lower reconstruction loss, confirming the encoder is effectively optimized. The lightweight Adapter acts as a bridge to the diffusion model, supporting—not replacing—the encoder, ensuring efficient training and better visual representation.\n\nTable1\n\n| Method | Reconstruction Loss |\n| --- | --- |\n| Frozen encoder+Trainable adapter | 0.5 |\n| Trainable encoder+Trainable adapter | 0.1 |\n\n> In Table 2 the authors show the best results using the proposed framework + Qwen2-7B but they do not show results on Qwen2-7B by itself\n> \n\nA4: The experimental results for Qwen2-7B are presented in the third row of the \"Expanding to other LLMs\" section in Table 1 in the article. We will add these results to Table 2 in the final version.\n\n> This method is not particularly new except the proposed use of Diffusion models, as illustrated by the authors in Fig 2.\n> \n\nA5: Thank you for your interest. Instead of simply using diffusion for supervision, we propose a novel design to tackle long supervision paths and sparse textual signals in VLM training. Our Multi-Adapter Diffusion fuses multi-level visual and connector features, while MOE Cross Attention dynamically routes multimodal inputs for fine-grained fusion. Together, they create a compact, semantically rich supervision path that enhances visual representation. We will further clarify these contributions in the final version.\n\n> line 242-243, \"/////\" added for no reason.\n> \n\nA6: Thanks for pointing this out. The slashes will be removed in the final version.", "claims_and_evidence": "The improvements are marginal ~1-2% and not \"substantial\" as claimed by the authors. Also for many benchmarks there is no improvement at all in Table 1.", "methods_and_evaluation_criteria": "It is not clear to me how the authors train their framework. Typically diffusion training involves sampling a noise level and predicting and supervising the noise. Instead the authors say that they supervise the image with reconstruction loss, which would mean the authors need to make several iterative passes through the Diffusion model. Do the authors roll out the diffusion process during training to backpropagate gradients. This needs to be clarified in the paper.\n\nThe authors add a bunch of new layers to feed the multi-level features into the diffusion model. I assume these layers are not used during inference since their only purpose is to adapt the features for diffusion model. Seems to me that this could result in the  primary encoder not learning anything useful since the new layers(unused during inference) maybe doing most of the heavy lifting. Have the authors figured out a way to solve this issue ?", "theoretical_claims": "NA", "experimental_designs_or_analyses": "In Table 2 the authors show the best results using the proposed framework + Qwen2-7B but they do not show results on Qwen2-7B by itself. Also it is not the best across all benchmarks. The authors need to analyse this inconsistency in performance a cross the different benchmarks.", "relation_to_broader_scientific_literature": "The authors introduce a new diffusion based framework to finetune the Vision Encoder in VLMs. This method is not particularly new except the proposed use of Diffusion models, as illustrated by the authors in Fig 2.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "line 242-243, \"/////\" added for no reason.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EYXcK4xBg3", "reviewer": "Reviewer_dHBy", "summary": "VLMs integrate visual and textual information to perform tasks such as image captioning, visual question answering, and image-text retrieval. However, current VLMs face two critical limitations: degraded supervision due to information loss during gradient propagation through LLMs, and the inherent semantic sparsity of textual supervision compared to visual data. To address these issues, the authors propose the Diffusion Supervision Vision-Language Model (DS-VLM), which uses a diffusion model to directly supervise the visual encoder and connector components, bypassing the LLM and providing richer semantic information from images.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The suggestions for additional analyses (e.g., statistical significance, computational overhead) would further strengthen the paper and provide a better evaluation of the proposed method.\n> \n\n> The training process involving diffusion models is computationally intensive. The paper does not provide any analysis on the training time and resource requirements, which are critical for practical applications. This lack of information raises concerns about the feasibility of deploying DS-VLM in real-world scenarios.\n> \n\nA1: Thank you for your constructive suggestion. We conducted full training of DS-VLM on 8 H100 GPUs. As shown in Table 1 below, compared to the original LLaVA-1.5, DS-VLM increases training time by approximately 15%. However, during inference, the diffusion module is not involved, so there is no additional inference overhead compared to the baseline. Given the performance gains on multiple benchmarks (e.g., a 2.1% improvement on MMMU and 1.8% on MMS), we believe this moderate training cost is justified. We will include this analysis in the final version to better illustrate the practicality of our method.\n\nTalbe1: Comparison of two-Stage training time consumption under different settings.\n\n|  | Vicuna-7B |  | Vicuna-13B |  |\n| --- | --- | --- | --- | --- |\n|  | PT | IT | PT | IT |\n| LLaVA-1.5 | 2.0 | 6.7 | 2.8 | 10.8 |\n| DS-VLM | 2.5 | 7.8 | 3.4 | 12.1 |\n\n> The paper provides a good review of related work. However, there are a few related works and concepts that are relevant but not currently cited or discussed in the paper:\n> \n\nA2: Thank you for highlighting the missing references. We appreciate your suggestions and will include and discuss them in the final version.\n\n> While the idea of using diffusion models to supervise vision-language models is interesting, it appears to be an incremental extension of existing work rather than a new approach. For instance, the integration of diffusion models in VLM has been extensive explored in previous studies such as DiffTPP, and also been explored in other contexts (e.g., text-to-image generation), and the application to VLMs seems like a straightforward adaptation.\n> \n\nA3: Thank you for your valuable comments. We understand the concern about the novelty of using diffusion to guide VLMs. Our approach is not a direct application of existing methods but an innovative integration of diffusion into VLMs.\n\nUnlike methods such as DiffTPP, which uses diffusion for generation, we first use it as a supervision signal during training. This enables the construction of a short-path gradient propagation chain from the image pixel space to visual features. Such a mechanism effectively mitigates the supervision information loss caused by long gradient paths in current VLM training. At the same time, the image reconstruction task provides fine-grained, multi-level optimization signals for both the visual encoder and the connector.\n\nFurthermore, our proposed Multi-Adapter Diffusion architecture flexibly receives supervision features from different modalities and employs MOE Cross Attention to aggregate information across multiple layers—a design that, to the best of our knowledge, is unprecedented in prior work.\n\nTherefore, we believe this work introduces substantial and original contributions in coupling diffusion models with VLMs, rather than being a generic application of diffusion techniques to VLMs. We will clarify this point further in the final version of the paper.\n\n> The core components of DS-VLM, such as the Multi-Adapter Diffusion model and the MOE Cross Attention mechanism, are built on well-established concepts. The paper does not introduce any new mechanisms or architectures that advance the field.\n> \n\nA4: Thank you for your thoughtful review. We sincerely acknowledge that while the Multi-Adapter Diffusion and MOE Cross Attention mechanisms in DS-VLM are inspired by some existing foundational concepts, our method introduces key innovations in how these components are integrated and applied. Specifically, Multi-Adapter Diffusion is the first to leverage a diffusion model for supervising intermediate features during the training phase. It receives features from multiple layers of the visual encoder as well as from the connector, and reconstructs the image through a unified U-Net framework. Meanwhile, the MOE Cross Attention mechanism processes these heterogeneous features through separate attention pathways and dynamically fuses them via expert routing, enabling the injection of fine-grained supervision signals. This design not only establishes a short supervision path from image pixels to multi-level visual features but also significantly enhances the structural coherence and semantic expressiveness of the learned representations. We believe this constitutes a substantial advancement over traditional VLM optimization paradigms. We will further highlight the uniqueness and novelty of these mechanisms in the final version of the paper.", "claims_and_evidence": "The claims made in the submission are generally supported by empirical evidence. The authors provide experimental results, ablation studies, and visualizations to validate their proposed DS-VLM framework.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are suited for the problem of improving vision-language alignment in VLMs. The DS-VLM framework is technically sound, addressing the limitations of traditional supervision methods. The evaluation criteria, including benchmark datasets, performance metrics, ablation studies, and visualizations, provide a assessment of the method's effectiveness. The results demonstrate improvements over baseline models and state-of-the-art methods, validating the claims made by the authors.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental designs and analyses are generally sound. The authors have conducted a set of experiments to validate their proposed DS-VLM framework, including comparisons with baselines, ablation studies, and visualizations. The results demonstrate improvements over baseline models and state-of-the-art methods, supporting the claims made. The suggestions for additional analyses (e.g., statistical significance, computational overhead) would further strengthen the paper and provide a better evaluation of the proposed method.", "relation_to_broader_scientific_literature": "The key contributions of the DS-VLM paper are aligned with the broader scientific literature on VLM, diffusion models, and multimodal learning. The paper leverages recent advancements in diffusion models to address a few limitations in traditional VLMs, providing a solution for enhancing visual feature extraction. By building on prior work, DS-VLM represents one step forward in the development of robust and efficient multimodal models.", "essential_references_not_discussed": "The paper provides a good review of related work. However, there are a few related works and concepts that are relevant but not currently cited or discussed in the paper:\n\n1. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., Le Scao, K., Wolf, T., ... & Riedel, S. (2019). Parameter-efficient transfer learning for NLP. International Conference on Machine Learning, 2790-2799.  \nThis work provides a theoretical and practical foundation for using adapter modules to enhance model performance without extensive retraining.\n\n2. Li, J., Chen, D., Hoi, S., & Lu, J. (2019). VisualBERT: A simple and strong visual-language fusion model. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 1-12.\n3. Lu, J., Batra, D., Parikh, D., & Lee, S. (2019). ViLBERT: Pretraining task-agnostic visual-language representations for vision-language tasks. Advances in Neural Information Processing Systems, 32, 13-23.\nThese models laid the groundwork for integrating visual and textual information using transformer architectures, which is relevant to the design of DS-VLM.", "other_strengths_and_weaknesses": "Pro：\n1. This paper is generally well-written.\n2. The experiments demonstrate the effectiveness of their proposed method.\n3. The authors also conduct a set of ablation studies to better understand the proposed method.\n\nCon:\n1. While the idea of using diffusion models to supervise vision-language models is interesting, it appears to be an incremental extension of existing work rather than a new approach. For instance, the integration of diffusion models in VLM has been extensive explored in previous studies such as DiffTPP, and also been explored in other contexts (e.g., text-to-image generation), and the application to VLMs seems like a straightforward adaptation.\n2. The core components of DS-VLM, such as the Multi-Adapter Diffusion model and the MOE Cross Attention mechanism, are built on well-established concepts. The paper does not introduce any new mechanisms or architectures that advance the field. \n3. The training process involving diffusion models is computationally intensive. The paper does not provide any analysis on the training time and resource requirements, which are critical for practical applications. This lack of information raises concerns about the feasibility of deploying DS-VLM in real-world scenarios.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See \"Other Strengths And Weaknesses\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5NltVx6ukO", "reviewer": "Reviewer_qD28", "summary": "This paper proposes DS-VLM,  which introduces diffusion-based supervision to improve VLMs. To shorten the gradient propagation path, this paper reconstructs images using diffusion models, thereby improving supervision for the vision encoder and connector. Experimental results confirm the effectiveness of proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback. We’ll address each point in detail.\n\n> The paper reports good results across various test sets. Section 4.3 shows accuracy gains from different supervision features and Section 4.5 show typical reconstruction process for diffusion models. However, the connection between proposed method and performance improvements are still not clear, and more discussions are recommended.\n> \n\n> This paper shows result improvement on multiple benchmarks. However, only the numbers are discussed in Section 4.2. More discussions on the improvement of model ability are recommended.\n> \n\nA1: Thank you for pointing this out. The performance gains come from our diffusion-based pixel-level supervision, which creates a short gradient propagation chain from image pixels to visual features, greatly enhancing the visual encoder and connector’s representation capabilities.\n\nOur method incorporates multi-level visual feature supervision (high, middle, and low layers) along with an MOE-based fusion mechanism, allowing the model to comprehensively perceive semantic details and structural information within the image—effectively compensating for the semantic sparsity of traditional text-only supervision in VLMs. Furthermore, the visualization results in Section 4.5 clearly demonstrate how DS-VLM progressively reconstructs the original image, providing intuitive evidence of the effectiveness of supervised features in guiding the visual understanding process.\n\nThe 1.9% gain on MMMU highlights our method’s strength in structured image understanding and semantic reasoning. We’ll clarify this causal link in the final version.\n\n> More ablation studies for different modules are recommended, such as the effect of adapters, the layer selection, cross-attentions.\n> \n\nA2: As shown in Table 3 (No.2–4) of the paper, we explored layer choices in our ablation study. Joint supervision across visual layers significantly boosts performance, as the layers complement each other and enhance supervision quality, improving model understanding and generalization.\n\nFollowing your suggestion, we conducted ablation studies on Adapter types (Table 1 below) and Cross Attention mechanisms (Table 2 below). Our proposed Adapter and decoupled Cross Attention show clear performance advantages. The multi-adapter structure enables finer adjustment of features from different modalities compared to the Q-Former-style Adapter. The decoupled Cross Attention avoids interference across feature levels, enabling each to fully express its semantic information through its dedicated attention path. These results validate the strength of our architectural design.\n\nTalbe1: Adapter type comparison\n\n| Method | MMB | MMS | MMMU | MV | OCRB | AI2D | HB | LB | SQA | MME |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Q-Former | 64.3 | 34.6 | 37.4 | 28.0 | 32.8 | 61.5 | 25.2 | 66.8 | 73.1 | 1805 |\n| Ours(MLP) | 65.5 | 35.8 | 38.9 | 28.6 | 33.5 | 62.2 | 25.7 | 67.9 | 73.7 | 1836 |\n\nTable2: Comparison using shared Cross-Attention mechanism\n\n| Method | MMB | MMS | MMMU | MV | OCRB | AI2D | HB | LB | SQA | MME |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Shared Cross attention | 64.1 | 34.4 | 37.6 | 28.3 | 33.1 | 61.7 | 24.9 | 67.0 | 73.4 | 1821 |\n| Ours(Decoupling Cross Attention) | 65.5 | 35.8 | 38.9 | 28.6 | 33.5 | 62.2 | 25.7 | 67.9 | 73.7 | 1836 |\n\n> Visualizations of model outputs are recommended to clearly show the difference between proposed method and baseline.\n> \n\nA3: We agree that visual comparisons are important for validating our method. In a fine-grained description task (e.g., “Describe the accessories worn”), the baseline missed the earring detail, while DS-VLM correctly generated “a silver hoop earring on the left ear.” This shows the benefit of our diffusion-based supervision in capturing fine details. We will add visual examples to highlight DS-VLM’s strength in fine-grained understanding.\n\n> Line 324 mentions the Mini-Gemini dataset, where is the result?\n> \n\nA4: The Mini-Gemini dataset is included in the \"Expanding to larger training datasets\" section of Table 1 in the paper. We'll clarify this in the final version.\n\n> Does the proposed method achives comparable results if trained on partial training data (such as 2/3) of LLaVa?\n> \n\nA5: As shown in Table 3 below, reducing training data leads to performance drop, highlighting the importance of full data for effective diffusion-based supervision. This analysis will be added in the final version.\n\nTable3: Comparison using partial training data\n\n| Method | LM | VE | PT+IT | MMB | MMS | MMMU | MV | OCRB | AI2D | HB | LB | SQA | MME |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| DS-VLM | Vicuna-7B | CLIP-L | 0.3M+0.4M | 60.7 | 34.8 | 37.3 | 26.4 | 32.3 | 56.5 | 28.7 | 63.9 | 70.3 | 1763 |\n| DS-VLM | Vicuna-7B | CLIP-L | 0.5M+0.6M | 61.3 | 35.5 | 38.1 | 26.7 | 32.6 | 56.9 | 29.2 | 65.0 | 70.6 | 1779 |", "claims_and_evidence": "The paper reports good results across various test sets. Section 4.3 shows accuracy gains from different supervision features and Section 4.5 show typical reconstruction process for diffusion models. However, the connection between proposed method and performance improvements are still not clear, and more discussions are recommended.", "methods_and_evaluation_criteria": "The method and evaluation in this paper makes sense.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "This paper shows result improvement on multiple benchmarks. However, only the numbers are discussed in Section 4.2. More discussions on the improvement of model ability are recommended.", "relation_to_broader_scientific_literature": "This paper improves the VLM by diffusion supervision, which may inspire the VLM community.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strength:**\n1. This paper propose an interesting method that uses diffusion models as supervision to improve the VLM. \n2. Experimental results show that the proposed method improves the performance over baseline model on multiple benchmarks. \n\n**Weaknesses**:\n1. The paper reports good results across various test sets. Section 4.3 shows accuracy gains from different supervision features and Section 4.5 show typical reconstruction process for diffusion models. However, the connection between proposed method and performance improvements are still not clear, and more discussions are recommended.\n\n2. This paper shows result improvement on multiple benchmarks. However, only the numbers are discussed in Section 4.2. More discussions on the improvement of model ability are recommended.\n\n3.  More ablation studies for different modules are recommended, such as the effect of adapters, the layer selection, cross-attentions. \n\n4. Visualizations of model outputs are recommended to clearly show the difference between proposed method and baseline.\n\n5. Line 324 mentions the Mini-Gemini dataset, where is the result?\n\n6. Does the proposed method achives comparable results if trained on partial training data (such as 2/3)  of LLaVa?", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please refer to weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xf5QLekmQd", "reviewer": "Reviewer_cFgh", "summary": "This paper proposed Diffusion Supervision Vision-Language Model (DS-VLM), which uses a diffusion module to provide additional supervision on the vision encoder and connector in the convential MLLM frameworks. Specifically, a frozen Stable Diffusion module takes the multi-level vision encoder features and the output of the connector as conditions to reconstruct the input image. It is claimed that DS-VLM facilitates the learning of visual semantic information of the vision encoder and connector. The experiments show that the newly introduced diffusion module can consistently improve the MLLM performance across different choices of vision encoders, LLM backbones, and training datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> However, the default input space of the Text Adapter (CLIP features) and the LLM backbone (word embeddings) are different. Why are the newly learned information assumed to be able to leveraged by the LLM backbone?\n> \n\nA1: In response to the reviewer’s question, our DS-VLM framework leverages a diffusion model to supervise both the connector outputs and multi-level visual encoder features, effectively shortening the long knowledge propagation chain often seen in MLLM training.\n\nAs for the difference between the Text Adapter’s input (CLIP features) and the LLM’s backbone (word embeddings), our key view is that the connector’s goal is to semantically align visual features to serve as effective textual embeddings. The diffusion model provides strong supervision and helps align feature distributions across modalities, allowing the connector to produce features that carry both visual detail and semantic meaning. These features, once projected into the LLM’s embedding space, can be naturally understood and used by the language model.\n\nOur experiments show that DS-VLM significantly improves multimodal task performance. We will further clarify this mechanism’s rationale and contributions in the final version.\n\n> The original LLaVA framework freezes the vision encoder during training. However, this work fine-tunes the vision encoder. It is unclear whether the so-called \"LLaVA\" in Table 1 freezes the vision encoder or not. If yes, comparing the proposed method to LLaVA with a trainable vision encoder is necessary to verify the effectiveness.\n> \n\nA2: Thank you for your suggestion. In Table 1, the LLaVA baseline freezes the visual encoder during training. Following your advice, we have conducted additional experiments comparing DS-VLM with a version of LLaVA where the visual encoder is trainable. As shown in the Table 1 below, DS-VLM still demonstrates clear advantages even when compared with LLaVA using a trainable visual encoder.\n\nTable1: Comparative experiments with LLaVA with trainable encoder\n\n| Method | LM | VE | PT+IT | MMB | MMS | MMMU | MV | OCRB | AI2D | HB | LB | SQA | MME |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| LLaVA-1.5 | Vicuua-7B | CLIP-L | 0.5M+0.6M | 59.4 | 33.6 | 36.2 | 26.3 | 31.5 | 55.8 | 27.8 | 62.2 | 68.9 | 1817 |\n| DS-VLM | Vicuua-7B | CLIP-L | 0.5M+0.6M | 61.5 | 34.9 | 37.8 | 27.1 | 32.4 | 56.7 | 28.9 | 64.3 | 69.7 | 1783 |\n| LLaVA-1.5 | Vicuua-7B | SigLIP-SO | 0.5M+0.6M | 62.6 | 35.2 | 38.1 | 27.2 | 36.6 | 58.7 | 27.5 | 67.0 | 70.4 | 1782 |\n| DS-VLM | Vicuua-7B | SigLIP-SO | 0.5M+0.6M | 63.2 | 36.8 | 38.4 | 27.6 | 37.5 | 60.0 | 27.8 | 67.7 | 71.1 | 1814 |\n\n> It would be appreciated if the paper can analyze qualitative results of the model and summarize on what types of questions the proposed method improves the performance. This might be helpful to support that the diffusion module enhances the learned semantic visual information.\n> \n\nA3: We agree that analyzing performance across question types helps validate the diffusion module’s impact. DS-VLM shows strong results on tasks requiring structural understanding and fine-grained perception, with notable gains on datasets like MMMU, MMB, and MMS. Specifically, MMMU covers multi-disciplinary visual reasoning (e.g., medicine and engineering), requiring accurate interpretation of complex visuals and domain-specific symbols—areas where DS-VLM excels due to its enhanced visual representations learned via pixel-level reconstruction. MMB evaluates broad multimodal capabilities, including structured diagrams and spatial layouts, where DS-VLM’s multi-layer supervision improves detail sensitivity. MMS further emphasizes semantic alignment between modalities, which benefits from our diffusion-enhanced visual features. We will include qualitative examples and analyses in the final version to better demonstrate DS-VLM’s effectiveness across these question types.\n\n> Additional slashes in Line 242-243.\n> \n\nA4: Thank you for pointing this out. We will remove the unnecessary slashes in lines 242–243 in the final version to avoid confusion.\n\n> Figure 3 shows that the LLM backbone is finetuned with LoRA. It would be better to also mention it in the Implementation Details in Section 4.1.\n> \n\nA5: Thank you for pointing out this issue. We will include the relevant information in the final version to ensure consistency between the text and figures, thereby improving the completeness and readability of the paper.\n\n> In Line 302-303, Section 4.1. The MLLM frameworks that use SigLIP as the vision encoder should be cited.\n> \n\nA6: We will include citations to MLLM frameworks that use SigLIP as the vision encoder in the final version to improve the discussion of related work.\n\n> It would be better to also include the performance of LLaVA in Table 2.\n> \n\nA7: Thank you for your suggestion. We will include the performance of LLaVA in Table 2 of the final version to enable a more comprehensive comparative analysis.", "claims_and_evidence": "The paper claims that the proposed DS-VLM can enrich the visual information learned by the vision encoder and connector. Their experiment results support this claim. The experiments are conducted across four different LLM backbones, two different vision encoders, and two different sets of training data. The method can improve the performance on most of the benchmarks under every setting.", "methods_and_evaluation_criteria": "The proposed DS-VLM aims to address the long knowledge propagation chain in MLLM training. It uses a diffusion model to supervise the connector output and multi-level vision encoder features. Especially, the output of the connector is treated as text embedding and fed into the Text Adapter of the diffusion module. I agree that this supervision can help the connector learn more visual semantic information. However, the default input space of the Text Adapter (CLIP features) and the LLM backbone (word embeddings) are different. Why are the newly learned information assumed to be able to leveraged by the LLM backbone? As the experiment results suggest that the DS-VLM can indeed improve the performance, the discussion about this problem would bring valuable insights to the multimodal learning community.", "theoretical_claims": "This paper does not involve theoretical claims.", "experimental_designs_or_analyses": "I checked the soundness/validity of the experimental designs and analysis. One issue is:\n1. The original LLaVA framework freezes the vision encoder during training. However, this work fine-tunes the vision encoder. It is unclear whether the so-called \"LLaVA\" in Table 1 freezes the vision encoder or not. If yes, comparing the proposed method to LLaVA with a trainable vision encoder is necessary to verify the effectiveness.", "relation_to_broader_scientific_literature": "Effective vision encoder and connector are essential to improve the MLLMs. While the previous works mainly focus on optimizing their architecture and training data, this work introduces additional training objectives on the vision encoder and connector. Moreover, the employment of the diffusion model also lies in the topic of representation learning via diffusion models. This paper extends the application of diffusion models to MLLM training.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Other Strengths:\n1. The proposed model outperforms the state-of the-art models in comparable sizes on a few benchmarks (Table 2).\n2. The paper is well-written and easy to follow.\n\nOther Weaknesses:\n1. It would be appreciated if the paper can analyze qualitative results of the model and summarize on what types of questions the proposed method improves the performance. This might be helpful to support that the diffusion module enhances the learned semantic visual information.", "other_comments_or_suggestions": "Typos:\n1. Additional slashes in Line 242-243.\n\nSuggestions on writing:\n1. Figure 3 shows that the LLM backbone is finetuned with LoRA. It would be better to also mention it in the Implementation Details in Section 4.1.\n2. In Line 302-303, Section 4.1. The MLLM frameworks that use SigLIP as the vision encoder should be cited.\n3. It would be better to also include the performance of LLaVA in Table 2.", "questions_for_authors": "See **Methods And Evaluation Criteria** and **Experimental Designs Or Analysis**. Those are of my greatest concern.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received mixed reviews: one Accept, two Weak Accepts, and one Weak Reject. Reviewer dHBy raised concerns that the work is primarily built upon existing concepts and techniques. In contrast, the Area Chair emphasized that a strong submission should address a practically meaningful and technically challenging problem. From this perspective, the issue of feature alignment between visual embeddings and the input space of large language models (LLMs)—as tackled by the proposed connector—remains a critical challenge in the development of multimodal large language models (MLLMs). Although the diffusion model utilized in this work may not introduce substantial technical novelty, the AC highly valued the insight of leveraging pretrained diffusion models to address the challenging problem of modality alignment in MLLMs. As such, the AC recommends accepting this paper.
Decision: Accept (poster)