Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation
Authors: Jitesh Jain
Zhengyuan Yang
Humphrey Shi
Jianfeng Gao
Jianwei Yang
Venue: NeurIPS 2025 poster ()
Keywords: Representation Learning
Multimodal Large Language Models
Visual Perception
Primary area: deep_learning

Abstract: In recent times, the standard practice for developing MLLMs is to feed features from vision encoder(s) into the LLM and train with natural language supervision. This approach often causes models to lean towards language comprehension and undermine the rich visual perception signals present in the data, which are critical for tasks involving spatial reasoning in the domain of embodied AI and robotics. Is it possible to optimize both at the same time? In this work, we propose VisPer-LM, the first approach that infuses visual perception knowledge from expert vision encoders into the LLM's (of an MLLM) hidden representations. We start by investigating MLLMs trained solely with natural language supervision and identify a positive correlation between the quality of visual representations within these models and their downstream performance. Given this insight, we formulate the objective during the pretraining stage in MLLMs as a coupled optimization of predictive visual embedding and next (text) token prediction.  Moreover, through extensive probing, we observe improved visual representation quality due to embedding optimization, underscoring the effectiveness of our probing setup. We demonstrate that our VisPer-LM outperforms the single and multi-encoder baselines, proving our approach's superiority over explicitly feeding the corresponding features to the LLM. In particular, VisPer-LM boosts performance by an average margin of up to 2.5% on various benchmarks, with a notable improvement of 8.7% on the Depth task in CV-Bench.

Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation
Jitesh Jain Zhengyuan Yang² Humphrey Shitf
Jianfeng Gao
Jianwei Yang**t
SHI Labs @ Georgia Tech Miicosooft Research, Redmond
Meta Superintelligence Labs
https: //ttthub.tb//////// com/ SHI- com//SHt-Labbs/issPe----a
glase
(a) Single Input to
Wooking i Mesiiu
PC
Pradecors
Reprseetition Cuality
Enabodting Loss
Proj
LLM
Iomalle
E
Autliss
(b) Input
(c) Single Encoder Input Multi-Encoddrr Embedding Distillation into LLM (ours)
Figure I: Different Paradigms for Incorporating Visual Information into LLMs. (a, b) Existing approaches [45, 67] feed features from the visual encoder(s) into the LLM and train the model solely with natural language supervision, i.e., next token prediction (NTP) to align the embedding space of the vision encoder(s) and the LLM. (c) We propose distilling target visual information into the intermediate representations of the LLM from a set of auxiliary vision encoders (E We adopt predictive embedding [2] optimization approach at selected LLM layers during training to minimize the embedding losses and the NTP loss function, resulting in a vision-centric approach to training the Multimodal Large Language Model. We only single base vision encoder during inference.
Abstract
In recent times, the standard practice for developing MLLMs is to feed features from vision encoder(s) into the LLM and train with natural language supervi- sion. This approach often causes models to lean towards language comprehension and undermine the rich visual perception signals present in the data, which are critical for tasks involving spatial reasoning in the domain of embodied Al and robotics. Is it possible optimize both at the same time? In this work, we pro- pose VisPer-LM, the first approach that infuses visual perception knowledge from expert vision encoders into the L.LM's (of an MI.LM) hidden representations. We start by investigating MI.LMs trained solely with natural language supervision and identify positive correlation between the quality of visual representations within these models and their downstream performance. Given this insight, we formulate the objective during the pretraining stage in MLLMs as a coupled optimization predictive visual embedding and next (text) token prediction. Moreover, through extensive probing, we observe improved visual representation quality due to em- bedding optimization, underscoring the effectiveness of our probing setup. We demonstrate that our VisPer-LM outperforms the single and multi-encoder base- lines, proving our approach's superiority over explicitly feeding the corresponding features to the I.M.. In particular, VisPer-I.M boosts performance by an average margin of up to 2.5% on various benchmarks, with notable improvement of 8.7% on the Depth task in CV-Bench.
"Wok done during JJ, JY's time with Microsoft Research. 'Equal advising.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Probing agaan
Suaketing MLMM
Figure 2: Probing reveals a positive corre- lation between depth representation qual-
ity and performance on CV-Bench. (a)
Increasing training data and using only the next-token prediction objective improves vi-
5964
1008
1994
4404
1220X
1256K
sual representation quality in the LLM, well as downstream performance. (b) Our method, VisPer-LM, outperforms LLaVA- 1.5 [45] in both probing and downstream tasks under the same settings.
(a) Scaling Training Data with Latural Supervision
Deyth Features
on EV Eo-Booch
696
ViPen
11906-15
VaPannMM loun)
(b) Predictive Embeddling Optimization Notural Language Supervition
Introduction
In the last couple of years, Multimodal Large Language Model (MLLM) [45, 81J development has witnessed rapid growth, owing mainly to the increasing number of powerful LLMs [65, 64, 18, 66] and large-scale datasets [67, 70, 11]. A visual reasoning MILM generally has three main compo- nents: vision encoder(s), a projector, and decoder LLM. Given the presence the decoder LLM responsible for producing the final outputs, the established practice to train MLLMs is with a next text token prediction objective [56] on relevant image-text datasets [37, 67] with supervision from the ground-truth text. Specifically, training an MLLM typically involves two primary stages [46]: (i) Pre-Training (PT), training a projector to align the embeddings from the vision encoder(s) to those inside the LLM, and (ii) Instruction Fine- Fine-Tuning (IFT), training the projector and LLM on conversation data for downstream instruction following tasks. Traditional MLLMs [46] usually use pre-trained visual encoder like CLIP-VIT [55, 14, 46] to process the visual inputs.
Although effective on general visual reasoning tasks, the visual encoder's features usually lack the fine-grained visual information required to perform tasks like spatial/depth reasoning [68, 71, 32], which are important visual perception abilities. Consequently, several recent works propose simply scaling the number of visual (input) encoders [67, 41, 35, 32] to incorporate auxiliary information amicable for improving the model's visual perception abilities. However, existing works often re- quire relatively large amounts of training data [35, 67] and compute [16, 1] for convergence, making these impractical for resource and data limited settings. Moreover, gains from using multiple visual encoders often come at the cost of latency during inference. In this work, we hypothesize that scal- ing the number of visual inputs and/or training data improves visual perception performance owing to its positive effect on the visual representations inside the LLM. Therefore, we explore a hidden opportunity to optimize the visual quality of representations inside the LLM directly. that end, propose to distill knowledge from set of expert visual encoders into the LLM''s 's representations during the PT stage through set of embedding losses (Fig. Ic). During inference, we only use a sin- gle encoder, resulting in a better trade-off between visual understanding performance and efficiency as compared to feeding multiple visual inputs to the L.M (Fig. la.b).
We first conduct extensive experiments to establish relationship between the (M)LLM's visual representation quality and its downstream VQA performance. To that end, we train probe each layer inside the LLM (of the MILM, LLaVA-1.5 [45], in our case due to its wide usage in the academic community) to analyze their quality against expert visual features. We choose features from visual encoders trained for three tasks as targets: image segmentation [31, 36), depth estima- tion [74, 75], and image generation [59, 57], owing to their well-studied, fundamental nature and the first two being seminal perception abilities [27, 21]. Based on choice of target features,
refer
to benchmarks [67] evaluating the depth and spatial reasoning ability as our target tasks, i.c., we aim to improve the model at visual perception without harming its general reasoning abilities. We find that the LLM representations also improve with more data, indicating enhanced visual perception ability and downstream performance, proving the effectiveness of our probing setup (Fig. 2).
From our probing experiments, we also observe that the middle LLM layers are optimal for em- bedding visual information inside the LLM based on the layer-wise representation quality trend. Consequently, we investigate the effect of optimizing the intermediate L.M representations against the expert visual features at specific layers. Inspired by the recent works in embedding predictive

---

degts
Scaking 5
Cosine Simileity
the multi eroader and basalines. servin on thodkoff buthwaan and repeosertition quality.
Cosine Siilurity
Preher for modele tra (mtagea) show inprowed Visul rage canristion the improves traiiin datts
Cosine Cosin--Simllaitty
Figure 3: Probing Visual Representation Quality across LLM layers in MLLMs. (1) As shown in the first row, the multi-encoder baseline has the best probing performance owing to the addi- tional feature inputs. The performance of probes trained on our VisPer-LM falls between the two baselines, demonstrating the effectiveness of our embedding distillation approach in learning proved projector while only using a single encoder during inference. (2) We observe that the probing performance for single-encoder models trained solcly with natural language supervision improves as the training data the base MLLM increases, indicating that the LLM improves its visual rep- resentations of the world with more training data. In the last row, we observe that our VisPer-LM (base setting) outperforms LLaVA-1.5 trained with more data during the PT stage, demonstrating the effectiveness of our approach with limited (data/compute) resources.
im-
architectures for self-supervised learning [2]. we propose VisPer-LM (Visual Perception Language Model) to distill [28] knowledge from the expert visual encoders into the LLM's representations during the pre-training (PT) stage. Specifically, feature and the embedding predictor output at the corresponding LLM layer (Fig. Ic). Note that we still use features from a base encoder inputs to the LLM. Furthermore, we incorporate special- ized set of tokens, (t), enriched with target information into the LLM's input sequence, fostering an implicit visual chain of thought [24, 78] while enhancing the model's ability to handle spatial and depth reasoning queries. Our experiments Sec. 5 illustrate the effectiveness of our approach on various benchmarks while outperforming the baselines. To summarize, contributions are:
optimize an embedding loss between the target
Through probing existing MILMs, we observe a positive correlation between visual repre- sentation quality and downstream performance our knowledge, we are the first to analyze the visual quality of MLLMs' representations. We present VisPer-LM, approach to distill knowledge from expert visual encoders into LLM's representations to improve the model's visual (depth/spatial) perception abilities.
benchmarks ks like CV-Bench. To the best
We conducted extensive experiments demonstrate the superiority of VisPer-LM over the corresponding single and multi-encode baselines on various benchmarks, including boosts up to 8. 7% and 5. 6% the depth and distance task in CV-Bench, respectively.
2 Related Works
2.1 MLLMs for Visual Reasoning
Contemporary MILMs have three primary components: vision encoder(s), projector, and LLM. One line of work [81, 37, 45, 39, 46] uses a single pretrained vision encoder [55, 20, 50, 14, 79] and trains projector, like an MLP [45] or QFormer [17] to align the visual features with the LLM.

---

A few recent approaches [63, 19] attempt to develop native MLLMs by directly feeding image patches into the LLM without using any visual encoder. Some works [1, 16] train cross-attention modules inside the L.M,, modifying the LLM architecture and often requiring millions of training data samples. Another line of work explores using either multiple encoder features [67, 80, 61, 41] or multiple visual modalities [32, 47, 13] as inputs to the LLM for improved visual (spatial) reasoning performance. Our approach uses a single-base vision encoder while distilling information from expert visual encoders into the LLM's representations.
2.2 Perception Probing in Foundation Models
OthelloGPT [40] probed the features from a GPT-2 [56] trained on sequences from a board game, Othello, and found that the probes were able to learn the board state, indicating the ability of se- quence models to learn spatial world representations. A recent work probes the features from foun- dational vision encoders [3] for 3D tasks. In our work, we probe the representation from the LLM layers against visual features from expert perception models. Furthermore, we establish a relation- ship between visual representation quality inside the LLM and downstream VQA performance.
2.3 Self-Supervised Learning
Distilling information [28] from a target encoder into a source encoder is well-established tech- nique to improve the source encoder's embeddings for a downstream task [9, 10, 25]. Recently, I- JEPA [2] proposed an embedding predictive architecture improve representations inside source encoder by comparing the target encoder features and mapped source encoder features with a trained predictor. A concurrent work, REPA [76] improved DiTs [54] at image generation by distilling in- formation from DINOv2 [53]. Recently, few works [58, 71] first distill information from target encoders into a single model and then leverage the resulting model as the vision encoder in an MLLM, requiring relatively longer and more training data for distillation. Unlike previous works, we distill information from multiple vision encoders into the LLM embedding space during the PT stage, resulting in a more vision-centric training approach without any extra training data.
3 Visually Probing LLM's Embeddings
In this section, we systematically analyze the quality of LLM representations within the MLLM through probing with the expert visual features as the targets. We define high-quality representations those that map easily to the target feature space, demonstrated by a high cosine similarity to the corresponding expert encoder's features. We choose target encoders from models trained for three visual tasks: image segmentation, generation, and depth estimation, guided by the mentioned tasks' fundamental and interpretable nature, i.e., visualizing the representations' quality through the respective decoders. We use the encoder outputs from Swin-L OneFormer [31, 49], DINOv2-L Depth Anything-v2 [20, 75]. and CLIP-ViT-L unCLIP-SD-2.1 [57, 55] as the segmentation (E**), depth (Edépt) and generation (Esa) as target probe features, respectively.
Probing Setup. We train a single-layer Perceiver Resampler [30] as the probe head at every LLM layer for each of the three target features. We use resampler probe head (similar to the Emb Predictor shown in Fig. 4) to accommodate the different sequence lengths of the target features and representations inside the LLM. For each LLM layer, input learnable queries into the probe head while using the layer's hidden states (for all tokens) as keys for cross-attention inside the resampler block. During probing, we set the learning rate as 1 and minimize the smooth-L1- loss objective with a batch size of 256. We train the probes for two epochs on the 118k images from the COCO-train201 7 [43] with the text query: "Describe the image in Iwo lines.". ". We find a positive correlation of 0.98 between the depth probing performance and CV-Bench accuracy in Tab. X, proving the effectiveness of our probing setup. We analyze the following
models with Llama-3-8h [64] as the LLM and CLIP CLIP-ConNNeXT-XXI [14] as the visual encoder:
Single-Encoder MLLM with features from husse passed through an MLP into the LLM. Multi-Encoodrr MLLM with features from Ebase Egen Edepth and Esss concatenated the feature dimension (in order) and passed through an MLP into the LLM.
VisPer-LM with features from Eoase passed through an ML. into the L.M and information from Egen, Edepth and E distilled into the LLM representations.

---

my PT
btt entrs-y loss
1: head
Emb Predictor
Uiarr Harm
R G
Predictor (P
Q
Logan Hanggn mile
Feed Fawertbook
Tolam predidur
emb loss
5+ 1:g 1
into tng
Peere on Aftelion
unclip CLIP-VIT (g
sexts
Emb Predictor (Pine)
Q
sog emb loss
OneFormer Swin (E**1)
Qatk
d
Predictor (Plnges)
Q
<d>
<g>
2:d
depth emb loss
Vision Encoder (base)
<cys>
<bxt>
DAv2 DINOv2 (Edepst)
MLP Projector
<ays> <iigg Lgg <dd -ss <bx>>
Figure 4: Architecture for VisPer-LM. During Pre-Trainn Training (PT), optimize an embedding loss at specific layers for each target encoder: layers & ID; E S, and E G for the depth, seg- mentation, and generation tasks, respectively. We resampler- embedding predictor [30], denoted as Pf (task) at each layer I, to output predictions. Each predictor takes in two inputs: a
set learnable queries (Qmsk) and the token sequence from layer I. with special tokens for other tasks omitted. The task tokens are derived from the corresponding embedding predictor's learnable queries. During IFT, we train with only the next-token prediction objective while keeping the special
tokens frozen to not affect their nature as we found it to perform empirically better in Sec. 5.
We compute the cosine similarity between the probe outputs and the corresponding target features over the 5k images from the COCO-va12017 [43] set to get the probing performance for evaluation.
Layerwise Trend. Upon evaluating the probes in the single encoder baseline, we observe that the middle (1 (12-24) layer probes show the best representation quality for the depth and seg probing tasks, with an upward trend in quality in the initial layers and a downward trend the deeper layers, as shown in Fig. 3. We attribute the observed trend the fact that the middle layer representations contribute the most to the (visual) reasoning (primarily spatial/depth) in LLMs [33, 34, 62]. It is an important finding when deciding the position of the embedding losses, since optimizing the middle layer representations should be optimal. Surprisingly, the probes trained to predict the features from Egen have fairly high cosine similarity (greater than 0.7) for all layers. We attribute the mentioned phenomenon to the choice of Egen that already has language-aligneed features, unlike Edepth and Eseg
Visual Encoding Approach. As shown in the first row of Fig. 3, the probes for multi-encoder MLLM expectedly learn better representations than the probes for the single-encoder MLLM. In- terestingly, for the former, the depth and seg probe performance remain at the same level till layer 20 and then follow downward trend, indicating the possibility of the deeper layer representations becoming text-rich in deeper layers [44]. We do not observe downward trend for gen probing, owing to the text-aligned target gen features. We observe that the VisPer-LM probes fall between the two baselines, serving as a good trade-off between efficiency and visual representation quality,
Training Data. We study the effect of scaling training data on the visual representation quality inside the single-encoder MILMM the second row of Fig. We analyze four models: (i) PT: model after the PT stage, (ii) PT 50% IFT: model with complete PT and trained on 50% IFT data, (iii) PT + IFT: model with complete PT and IFT, and (iv) PT VPT IFT: model with an extra training stage ALLaVA-Caphae that with an increase in the amount training data for the probed model, the probes show a gradual improvement, indicating that the representations of visual world inside MI.MMs and, therefore, downstream performance improve with just natural language supervision more data (Fig, 2a)!
[6] data, during which the whole model is trained. We observe
5

---

For experimental completeness, we also report probing for the single-encoder LLaVA-1.5 trained with additional LLLaVA-Captiinn [6] data during the PT stage. As illustrated in the third row of Fig. 3, while the inclusion of additional PT data improves L.L.V--1.55 performance on CV-Bench, our VisPer-LM achieves superior results with lesser PT data, highlighting the effectiveness of our aaproachh with limited data. Notably, there a significant improvement in representation quality for the seg probing task with the use of ALLaVA VA-Laatootiion during PT, which we attribute to the dataset's s detailed captions fostering enhanced semantic alignment [6]. Please refer to Tab. for more results.
4 Embedding Visual Information into LLM
The current trend in developing MLLMs for visual reasoning is gluing vision encoder(s) to a decoder LLM with a projector and training the model to minimize the cross-entropy loss for classification over the LLM's vocabulary for next token prediction (NTP). In our attempt to add a vision per- spective to training MLLMs while only using a single visual encoder during inference, we aim to optimize a predictive visual embedding objective for the intermediate LLM representations with the standard NTP objective during the PT stage. In other words, we distill auxiliary visual (task) in- formation into the LLM's intermediate representations rather than feeding the visual features into the LLM from the expert encoders. We hypothesize that such an approach leads to better projec- tor initialization (Tab. VIII) for the IFT stage. Our hypothesis is validated in Fig. 3, as probes for our VisPer-LM significantly outperform the single encoder baseline, especially in the initial layers. During IFT, we train the MLLM using the NTP objective, without the embedding predictors.
4.1 Multi-Encoder Feature Distillation
Given set of target visual encoders, Edepth E and E we distill information from their feature outputs into the LLM's representation space by minimizing embedding losses between predictor outputs and target features at certain layers. Since the token sequence inside the LLM has a different length as compared to the target representations, we use a single-layer Perceiver Resampler [30] as the embedding predictor (task)) that takes as inputs: learnable latent queries (Q (Q(kk)) along with
the embeddings from layer of the LLM and outputs prediction, p'. We set the number of queries in Q ( such that it matches the number of tokens in the corresponding (task) features, i.e., 576, 576, and 1, for deepth Q and Qarnn respectively.
To amplify the effect of the contextual information about the target task (spatial/depth reasoning) inside the LLM [7, 24], we use a set of special N task tokens. (t). We append (f) to the image tokens in the LLM's input sequence. Specifically, we average pool Qdepth and Qaey into N number of tokens to obtain the (d) and (s) tokens, respectively. Since the number of target tokens for generation features is only one, we average pool (g) to obtain Qeen. During PT, we only train the MLP projector, embedding predictors, and the special tokens (1). We extract the tokens corresponding to the system prompt, input image, the corresponding (t), and the text query from a layer's output
sequence as the input keys to the embedding predictor, as shown in Fig. 4.
4.2 Predictive Embedding Optimization
At the core of our approach is indirectly optimizing the projector during the PT stage by minimizing an embedding loss for each target representation at specific layers. As shown in Fig. 4, we feed the outputs from the d E ID. S, and g E G layers of the LLM into the corresponding embedding predictor to obtain the predictions for computing an embedding loss (Lemb), which is a weighted sum of Smooth-Li-Loss [23] and contrastive (InfoNCE) loss [69]. We denote the sets of layers for embedding distillation from Eeeph E and E as D, S, and G, respectively. The final embedding loss for each target feature is a sum of losses over all corresponding layers. We compute the final loss during PT as the sum of the NTP objective and embedding losses, as shown in Eq. (4).
We denote p' and t as the embedding predictor (P (task)) outputs at layer I and the target task fea- tures, respectively, D denotes the 1h element in a batch of embeddings with a global batch size of B aggregated over all GPUs. We denote 7 (initialized to 2.0) as the learnable scaling factor for the con- trastive loss. The weights for smooth-L.l-and contrastive losses are A&..1 and Acontrastive = 0.3, respectively, at all selected layers. We select these values to ensure their magnitudes are comparable

---

Table 1: Comparisons Baselines. Our VisPer-LM outperforms the single and multi-encoder LLa [45] by up to 2.5% and 0.9% on average across various benchmarks, respectively. The best numbers are set bold for every base-encoder and decoder LLM combination.
CV-Bench
General
Method
Countil
Depth 3
Relation"
MMStar
Phiz- BizZ-4ommini
1LaVA 1.5 VisPosLLM (OBES)
CLIP VIL
52.4
67.2
75.2
56.2
36.5
57.1
56.7
57.3
CLIP-VITLL
52,4
68.7
76.0
56.7
36.0
58.01
56.4
57.7
1LaVA 1,5
CLIP ConNNeX XXL
51.8
70,8
74.0
55.3
36.4
58.0
55.9
57.4
VisPasLM (omss)
49.4
72.5
77.2
60.3
38.4
58.4
56.5
58,9
Llomas 56
1L
CLI--VIEL
50.4
72.5
64.9
45.7
38.8
57,8
56.9
55.1
C Elre & 4
45.3
75.5
70.9
54.3
36
57.5
58.3
56.8
LLaVA-15
CLIR-II-L grnt
68.9
VisPer-LM
69.4
Canbrian- [67]
CLI--CoNNcX--NNL CLIP-VIT-L
S0LIP-S0-40M DINOY2.G
LLAAAL1
RADROWTLLSS (Ss]
LLaVA-15
CLIP-CoWNeCT-AN
Table 2: Results on Additional Benchmarks. Our VisPer-LM outperforms LLaVA-1.5 1.5 on classical benchmarks like POPE [42] and GQA [29] showing its intact general reasoning ability.
Method
Encoder
POPE
GQA
MMMU
VizWiz
Avg
LLaVA-155
RADIO-ViTLL [58]
86.3
62.8
36.9
50.5
59.1
1LAVA-1.5
VITL
VisPer-LM (ours)
CLIP-VITLL
86.4
63.7
38.7
54.0
60.7
Table 3: Scalability with VPT (more training data). VisPer-LM outperforms LLa aver- age across different CV-Bench tasks. We use CLIP--ConNeXTT-XXLL as the base encoder.
Method
LLM
Co
Depth
Relation
Distance
Overall
LLaVA-1.5
Phi3 Miiiii
49.7
70.0
72.6
58.7
61.8
VisPer-LLM (ours)
Phi3 2ki3-4k:na mini
53.7
72.0
73.1
58.5
63.4
LLaVA-1.5
Llama3-88 88
56.3
76.8
73.1
50.3
VisPer-LM (ours)
Llama3-8b
60.0
75.0
70.8
55.2
64.6
and their weighted sum aligns in scale with LNTP at convergence to maintain training stability. We set Adepth, Aueg and Agen as 0.5.
C/u (p', 0.5 (p' t)2 if Ip'- <1 else -/'-t/-02. 0.5
exp(sim(p), t,)/7)
C coomrastive log
(2)
Σラ exp(sim(pp) t2)/7)
D/8/G emb
(Da + Acontrustive Cl coomtrastiv)) LEB/S/G
Σ
(3)
CpT = CNTP + Adopth + Asse Ce + Agen C
(4)
DINOv2-L [53] from Depth Anything v2 [75] as Edepth Swin-L. [49] from OneFormer [31] (trained on COCO-train2017 [43]) as E and CLIP-ViT-L [55] from unCLIP-SID-2. [57] Egen In the input sequence the LLM, append the gen, depth, and seg tokens in that order to the image tokens. Therefore, given an image- text pair, the input token arrangement is {sys)) (img))
Other Architecture Details. As shown in Fig. 4,
(in)))(img)) (g) (s) (tzt)), where (sys), (img), and (txt) denote the tokens corresponding to the system prompt, image, and text query, respectively. The target feature dimensions from Egen Edepth and Eseg are (1, 1024), (576, 1024), and (576, 1536). respectively, corresponding to their final layer outputs.
Experiments
In this section, we first provide comprehensive comparison of our method's performance to that of the base MILM, 1.555 [45] across different base vision encoder and decoder L.M choices in Tab. 1. Next, we provide experimental results with a 2.5-stage training strategy, composed of an extra Visual Pre-Training (VPT) stage that involves training the ALLaVA-Caption [6] dataset to
7

---

benchmark. Our VisPer-LLM improves significantly over LLaVA-1.5 [45] on depth and relation reasoning, our target perception tasks.
Table 4: Comparison on the Perception tasks using BLINK [22]
Method
Avg
Spatial Relation
Relative Depth
Count
LLaVA-1.5
55.6
72.0
51.4
43.3
MADDOO-LLLLVA-I
55.8
65.7
52.4
49.2
Cambrian-1
56.9
74.1
51.6
45.0
VisPer-LM (ours) 58.8
75.5
51.6
49.2
Table 5: Ablations on Layer sets for Lemb. Setting DD 20), $=(10, 18), and G=(12, 203 per- forms the best. Following the findings in Sec. 3, we only experiment with middle layers.
BD
S
G
CV-Bench CV-Bench
MMStar
Avg
[20)
(18)
{20)
57.6
60.8
38.8
52,4
[8:20]
10:18)
[12:20]
58.6
64.2
39.5
54.1
(18:20)
18:20)
[16:20]
55.8
59.5
40.8
52.0
[16:18;20]
16:18820)
116:18:20)
61.3
37.0
51.7
Table 6: Training stages for Lemb. Using the embedding losses only during PT is optimal.
PT
IFT
CV-Bench CV-
MMStar
RWQA
Avg
single-encoder
57.8
57.7
62.9
38.8
57.5
54.2
58.6
64.2
39.5
57.9
55.1
59.1
58.3
38.3
56.2
53.0
demonstrate the scalability of our approach to more data. Lastly, we methodically study various design factors, including the optimal choice of layers for embedding additional visual information and the number and nature of special tokens ((1) through a series of ablations.
5.1 Implementation Details
Training. During the PT stage, we the LLaVA-558K [45] dataset train our model for an epoch with Ir of 1e-3, We only train the (MLP) projector, the embedding predictors, and the special the LLaVA-665K [45] dataset and train the projector and
tokens (()). During the IFT stage, LLM for one epoch with an Ir of 2e-5 with the vision encoder and (t) kept frozen. When using VPT, we leverage the ALLaVA-Caption-66n one epoch with an (r of 2e- Unless mentioned otherwise, we report results with models trained with PT and IFT stages. During the PT stage, we train our model with the NTP objective and the embedding prediction objectives. During the VPT and IFT stages, we solely use the NTP objective.
[6] dataset to train the whole model (except (t)) for
We train all models 16 AMD 192G-M1300X GPUs with batch size of 256 during PT and 128 during IFT and VPT. We use CLIP-ViT-L [55] and Llama3-8b [64] as the base vision encoder and decoder L.MM unless mentioned otherwise. By default, set N. ID, S, and G 8, (8, 20), {10, 18), and (12, 20), respectively. For other hyperparameters, we follow LL- [45].
Evaluation. We primarily evaluate VisPer-LM for vision-centric abilities on CV-Bench [67] and report results for all four tasks CV-Bench: Count (2D), Relation (2D), Depth (3D), and Distance (3D). For ablations, and distance) task categories. We also evaluate our models' general visual reasoning capabilities on the MMStar [8], RWQA [72], and OK-VQA [52] benchmarks. We select MMStar our primary benchmark for general visual reasoning, as addresses the well-known limitations [8] of existing benchmarks [48, 51, 77, 38] through careful filtering. Nonetheless, report results several classical general reasoning benchmarks [29, 42, 26, 77] in Tab. 2 for experimental completeness.
report the average accuracy over the 2D (count and relation) and 3D (depth
5.2 Main Results
As shown Tab. 1, our VisPer-LM outperforms the single encoder baseline, i.e., LLaVA-1.5 [45] across different base encoder [14, 55] and decoder LLM [64, 65] combinations. Specifically, Llama3-8b [64] based VisPer-LM outperforms 1L. by 5.6 % and 4.5 on the Distance and Relation task, respectively for the CLIP-ViT-L base encoder and by 8.7 % on the Depth task for the

---

Table Ablation on the nature of special tokens during IFT. Keeping (t) frozen during IFT aids in keeping their task-specific nature intact, resulting in better performance.
(t) during IFT CV-Bench
CV-Bench
MMStar
RWQA
Avg
frozen
58.6
64.2
39.5
57.9
55.1
learnable
56.9
56.1
39.0
57.3
52.3
base encoder. Furthermore, we compare our approach to the corresponding two variants of multi-encoder baselines: feat concat.: features from all encoders are concatenated along the feature dimension and passed into a single MLP; and (ii) token concat:: features from all encoders are first passed through separate MLPs and then concatenated along with token dimension after average pooling the sequence outputs from Edepih and Ex into eight tokens each. As shown in Tab. I, our VisPer-LM outperforms the multi-encoder baselines on average, showing the effec- tiveness of our approach while using a single encoder during inference. Note that the multi-encoder baselines are better than VisPer-LM on the Depth task in CV-Bench owing to their superior depth representation quality observed in Sec. 3.
We also compare our VisPer-LM with a RADIO- ViT-L [58] and Llama3-8b-based LLaVA-1.5. Un- like our approach, which distills information directly into the LLM representations, RADIO distills knowledge from multiple expert encoders into a single student encoder. As shown in Tab. 1, our VisPer-LM outperforms the -ITO--vT--- LLaVA-115, underscoring the effectiveness of our approach over encoder distillation for developing MLLMs. Note that we train the RADIO-VITA based model using the same setting as the single encoder baseline.
Additionally, we train a Llama3-8h 8b based Cambrian-1 [67] model (with a batch size of 512, fol- lowing Tong al. [67]) using the LLaVA-558K. (PT) plus LLaVA-665K (IFT) dataset mix. As shown in Tab. 1, the Cambrian- model not only underperforms as compared to our VisPer-LM but also LLa underscoring the effectiveness of our approach with limited data, the publicly released Cambrian-1 models were trained on millions of samples, requiring immense resources.
Additional Benchmarks. Although not the target tasks for our method, we report results on clas- sical visual reasoning benchmarks like POPE [42], GQA [29], MMMU [77], and VizWiz [26] in Tab. 2. VisPer-LM outperforms VL-15. demonstrating its superior visual perception ability without any loss general reasoning abilities.
Scalability with VPT. We report results using the 2.5-stage training setup (PT+VPT+IFT) for VLa [45] and VisPer-LM in Tab. 3. Our VisPer-LM outperforms the baselines on average across all CV-Bench tasks, highlighting the scalability of our approach with more training data.
Results on BLINK [22] benchmark. We report results visual perception tasks under the BLINK benchmark (val set) in Tab. 4. We also report results using the data-matched Cambrian-1 and VLaVA-1.5 models. We find that our VisPer-LM significantly outperforms all other baselines on average, especially on the Spatial Relation task, demonstrating the effectiveness and generalization of our approach.
MADIO-ViT-L-barr
5.3 Ablations
Layer Sets for Embedding Losses. The choice of layers in D, S, and G for the corresponding embedding losses is a crucial design choice with significant effect the performance. Based on the findings about middle layers Sec. 3, we ablate on embedding loss positions only in the middle layers of the LLM. As shown in Tab. setting D. S, and G {8, 20,. {10, 18), and (12,203, respectively, performs the best overall for the 32-layer Llama3-8b.
Training Stage for Embedding Optimization. In Table 6, we observe that adding Lemb during IFT results worse performance than doing so only during the PT stage. We attribute this to the interference of vision-centio supervision with task-aligned natural language supervision during IFT. Additionally, the second row of Table 6 shows that using learnable tokens in the sequence [24] without Lemb slightly improves performance over the baseline but remains inferior VisPer-LM.
Nature of Special tokens during IFT. As shown in Tab. keeping the special tokens (()) frozen during IFT performs better as compared to making them learnable. We attribute frozen tokens

---

Table 8: Throughput Analysis. Our VisPer-LM has superior inference throughout compared to that of the multi-encoder baseline and similar throughput to that of the single-encoder baseline with much better performance.
inference
single-encoder
multi encoder
VisPer-LM
throughput (samples/sec)
9.92 + 0.03
5.32 + 0.02 9.86 + 0.01
Table 9: Experiments with SigLIP-VIT-S0400N [79] as the base vision encoder. Our VisPer- LM improves significantly over LLaVA-1.5 [45] on depth and relation reasoning, our target percep- tion tasks.
Method
CV-Bench
Count
Depth
Relation
LLa
56.3
47.8
60.3
57.4
VisPer-LM (ours)
57.3 (+1.0)
48.7 (+0.9)
63.7 (+3.4)
66.3 (+8.9)
performing better to the gradients from natural language supervision interfering with the vision- centric information stored in the special tokens during IFT if those are learnable parameters.
Throughput Analysis. We demonstrate the superior inference throughput of VisPer-LM over the multi-encoder baseline in Tab. 8. We record the throughput on single NVIDIA 80G A100 GPU for a single forward pass on the CV-Bench evaluation set with a batch size of 1. We report the mean and standard deviation across 10 runs. We use the CLIP-ConVNeXT-XXC [14] and Llama3-8b 8b [64] based models for the throughput analysis.
Experiments with SigLIP [79]. In Tab. 1, we achieved performance gains with CLIP-ConVNeXT: XXI., one the strongest visual encoders, as also found in Cambrian-1 [67] (ranked second overall in their Tab. 2). Here, we experiment with SigLIP-ViTE ------TTO00OO (ranked first in Cambrian-1 [67]] our base visual encoder. As expected, we notice performance boosts on the perception tasks in CV- Bench in Tab. 9, which have high positive correlation to the depth and seg representation quality (Appendix D), underscoring the effectiveness of our approach with various vision backbones.
Conclusion
In this work, we probed MLLMs and established a positive correlation between visual representation quality within the LLM and downstream performance. Building on these insights, we introduced VisPer-LM, the first approach to distill knowledge from target encoders into the LLM via predictive embedding optimization during the pre-training stage, complementing the next token prediction objective. We validated that our embedding optimization results in better vision- langune alignment before the IFT stage. Through extensive experiments, we demonstrate EsPer-LM's superiority to the corresponding baselines in terms of both representation quality and VQA performance.
Limitations and Future Work. With VisPer-LM. we enhanced the quality of representations in- side the LLM with guidance from expert visual perception encoders. That said, our work mainly focuses on improving perceptual reasoning without any loss of general reasoning abilities. Incor- porating more general-purpose teacher encoders, such as Intern [11, 12], offers promising pathway improve general reasoning abilities. Secondly, in this work, we mainly work with the image modality. Applying predictive embedding optimization for low-level information like motion control [4. 5] while training on videos could improve MILMs' spatial and temporal reasoning in the future. Lastly, due to resource constraints, we could not scale VisPer-LM to larger LLMs like Llama3-70b [64] or Qwen2-72b [66], and it remains an intriguing experiment.
Acknowledgements. We extend our heartfelt gratitude Fangrui Zhu, Reuben Tan, Min Shi, Bhavika Devnani, Fiona Ryan, and Chieh- Yun Chen for their valuable feedback and insightful dis- cussions. We also sincerely thank the GCR team Microsoft for their support in resolving frequent infrastructure challenges, which enabled our experimentation. This work was in part supported by NSF CAREER Award #2239840, and the National Institute for Exceptional Education (Award #2229873) by National Science Foundation and the Institute of Education Sciences, U.S. Depart- ment of Education. Lastly, we thank the MI. Center @ GGorgia Tech and Microsoft Research for supporting this work.
10

---

References
Bran-Baptiste Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds. Roman Ring, Eliza Rutherford. Serkan Cabi, Tengda Han, Zhitao Gong. Sina Samangooei. Mariannn Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski. Ricardo Barreira, Oriol Vinyals, An- drew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot leamnig. In NeurlPS, 2022. 2,
[2] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-super learning from images with a joint-embedding predictive architecture. In ICCV. 2023. 3,4
[3] Mohamed El Banani, Amit Raj. Kevis-Kokits, Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d awareness of visual foundation models. CVPR, 2024.
[4] Adrien Bardes, Quentin Garrido, Jean Ponce, Michael Rabhat, Yann LeCun, Mahmoud Assran, and Nico- las Ballas. Revisiting feature prediction for learning visual representations from video. arXiv, 2024. 10
[5] Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak. Yaniv Taigman, Lior Wolf, and Shelly Sheynin. Videojam: Joint appearance-motion representations for enhanced motion generation in vidco models. arXiv, 2025. 10
[6] Guiming Hardy Chen, Shumian Chen, Ruifei Zhang, Junying Chen, Xiangho Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing spt4v-synthesized data for a lite vision-language model, 2024. 5, 6, 7, 2,
[7] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krish- namoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: Large language model as a unified interface for vision-language multi-task learning, arXiv, 2023. 6
[8] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? In NeurlPS, 2024. 8
[9] Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton, A simple framework for con- trastive learning visual representations. ICML, 2020. 4
[10] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, 2020.
[11] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint ar0iv:2404. 16821, 2024. 2, 10
[12] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijic Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv, 2023. 10
[13] An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. In NewrIPS, 2024. 4
[14] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. CVPR, 2023. 2, 3, 8, 10, 23, 28
[15] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chathot impressing gpt-4 with 90%* chatgpt quality, March 2023. 28
[16] Wenliang Dai, Naycon Lec, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shocybi, Bryan Catanzaro, and Wei Ping. Nvlm: Open frontier-class multimodal llms, arXiv,
2024. 2, 4
[17] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language language models with instruction tuning. 2023. 3

---

[18] DeepSeek-AL. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,
2024. 2
[19] Haiwen Diao, Yufeng Cui, Xiaotong Li. Yueze Wang. Huchuan Lu, and Xinlong Wang. Unveiling encoder-free vision-language models. In NewrlPS, 2024. 4, 25, 26, 28
[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 3,4
[21] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale doep network. In Advances in Neural Information Processing Systems, pages 2366-2374, 2014. 2
[22] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In ECCV, 2024. 8,9
[23] Ross Girshick. Fast r-cnn. In ICCV, 2015. 6
[24] Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Na- garajan. Think before you speak: Training language models with pause tokens. In ICLR, 2024. 3, 6,
[25] Jean-Bastien Grill, Florian Struh, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: new approach to self-supervised learning, 2020.
[26] Danna Gurari, Qing Li, Abigail J Stangl, Yinan Guo, Chaz Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. CVPR, 2018. 9.9
[27] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, pages 2961-2969, 2017.
[28] Geotfrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. NeurlPS
Deep Learning Workshop, 2014, 3,
[29] Drew A Hudson and Christopher D Manning. Gqa: new dataset for real-world visual reasoning and compositional question answering. CVPR, 2019. 7,
[30] Andrew Jaegle, Sebastian Borgeaud, Saan-Bapiiste Alayrac, Carl Doersch, Catalin Tonescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Pereeiver io: A general architecture for structured inputs & outputs. In ICLR, 2022. 4, 5,6
[31] Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. OneFormer: One Transformer to Rule Universal Image Segmentation. In CVPR, 2023. 2, 4, 7, 30, 31
[32] Jitesh Jain, Jianwei Yang, and Humphrey Shi. VCoder: Versatile Vision Encoders for Multimodal Large Language Models. In CVPR, 2024. 2,4
[33] Omri Kaduri, Shai Bagon, and Tali Dekel. What's in the image? a deep-dive into the vision of vision language models. arXiv, 2024. 5, 23
[34] Guy Kaplan, Matanel Oren, Yuval Reif, and Roy Schwartz. From tokens to words: On the inner lexicon of llms. arXiv, 2024. 5
[35] Oguzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. BRAVE: Broadening the visual encoding of vision-language models. In ECCV, 2024.
[36] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland. Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In ICCV, 2023. 2
[37] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision Easy visual task transfer. arXiv preprint arXiv:2408.0333 03326 2024. 2,
12

---

[38] Bohao Li, Rui Wang. Guangzhi Wang, Yuying Ge. Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal Ilms with generative comprehension. 2024. 8
[39] Jiachen Li, Xinyao Wang. Sijie Zhu, Chia-wen Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, and Longyin Wen. CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of- Experts. NeurIPS, 2024. 3
[40] Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring sequence model trained on a synthetic task. In JCLR, 2023.
[41] Yanwei Li, Yuechen Zhang, Chengyao Wang. Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv:2403. 18814, 2023. 2,4
[42] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. EMNLP, 2023. 7,8.9
[43] Tsung-Y- Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context. In ECCV, 2014, 5,5,7 7.
[44] Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. arXiv, 2024. 5
[45] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024. 2, 7,8, 8, 9, 10, 18, 25, 29, 30, 31
[46] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023. 3
[47] Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar. Prismer: A vision-language model with multi-task experts. TMLR, 2024.
[48] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Wangbo Zhao Songyang Zhang, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? In ECCV, 2024. 8
[49] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 4,7
[50] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022. 23, 27, 28
[51] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Pe- Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. In NeurlPS, 2022. 8
[52] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019. 8
[53] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo. Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Russell Howes, Po- Yao Huang, Hu Xu, Vasu Sharma, Shang-Wer Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Syn- naeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2:
Learning robust visual features without supervision, 2023. 4,
[54]
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Tlya Sutskever. Learning transferable visual models from natural language supervision. arXiv, 2021. 2, 3, 4, 8, 23
[56] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. arXiv, 2019. 2,4
[57] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv, 2022. 2, 4,
13

---

[58] Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In CVPR. 2024. 7,9
[59] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 2
[60]
Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. https:/
pytorch- fid, August 2020. Version 0.3.0. 30, 31
[61] Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao, Jan Kautz, Zhiding Yu, and Guilin Liu. Eagle: Exploring the design space for multimodal Ilms with mixture of encoders. In ICLR, 2025,
[62] Oscar Skean, Md Rifat Arefin, and Ravid Shwartz-Ziv. Does representation matter? exploring intermedi- layers in large language models. In Workshop Machine Learning and Compression, NeurlPS 2024, 2024.
[63] Chameleon Team. Chameleon: Mixed-modal fusion foundation models. arXiv, 2024. 4, 25, 26, 27
[64] LLaMA-3 Team. The llama 3 herd of models, 2024. 2, 4, 8, 10, 23, 27, 28
[65] Phi3 Team, Phi-3 technical report: A highly capable language model locally on your phone, arXiv, 2024.
2, 8
[66] Qwen2 Team. Qwen2 technical report. arXiv, 2024, 10,
[67] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang. Shusheng Yang, Adithya Iyer. Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-I: fully open, vision-centric exploration of multimodal Ilms, 2024. 1, 2, 4, 7,
8, 9, 24
[68] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xic. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, 2024, 2
[69] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv, 2018.
[70] Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang. Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song. Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models. arXiv, 2023. 2
[71] Wenxuan Wang, Quan Sun, Fan Zhang, Yepeng Tang, Jing Liu, and Xinlong Wang. Diffusion feedback helps clip see better. arXiv, 2024. 2,4
[72] xAL grok, 2024. 8
[73] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv, 2023. 23
[74] Lihe Yang. Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing: Unleashing the power of large-scale unlabeled data. In CVPR, 2024. 2
[75] Lihe Yang. Bingyi Kang. Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv:2406.094 04041 2024. 2, 4, 30 31
[76] Sihyun Yu, Sangkyung Kwak, Huiwon Jang. Jongheon Jeong. Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is casier than you think. arXiv, 2024. 4
[77] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024. 8,
[78] Stuart Shieber Yuntian Deng, Yejin Choi. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv, 2024.
[79] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 3, 10
14

---

[80] Xiangyu Zhao, Xiangtai Li, Haodong Duan, Haian Huang. Yining Li, Kai Chen, and Hua Yang. Towards semantic equivalence of tokenization in multimodal Ilm. arXiv preprint, 2024.
[81] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv, 2023. 2,3
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Please refer to Sec. and Sec. 5 for verifying the claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the end of Sec. 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
We do not include any theoretical results or claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer to Sec, 3 for details about our probing experiments and Sec. and Sec. 5.1 for our main experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data also be provided via
often one good way accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models. it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes] for data; [No] for code.
Justification: We publicly available datasets which be easily downloaded from the web. Our codebase built on top of the LLaVA-1.5 [45] codehase, and we will release our code with the camera-ready version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: /nips.
public/ guides/ Codeeeeeeessseeeee Suiomi ss sessssooPooo oolccy for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips... cc ppublic. c/ c/quiiiiee quuides C////CoSSSute Submi ssi onnolicy)) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer:
[Yes]
Justification: Please refer to Sec. 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: We conduct all our experiments with a fixed seed, resulting in a deterministic nature of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative еπτог rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Sec. 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ /tterr////euip.p.eee http://blcc/tthcccieeelleeee
Answer: [Yes]
Justification:
To the best of our knowledge, we follow the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We do not believe our work has any societal impacts except the usual hallu- cination concern involved with the usage of MLLMM.. Our embedding distillation approach does not affect this in any form or way.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: We shall release the code and models under non-commereial license upon acceptance ensure only research use.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have credited all prior works with citations and will include the corre- sponding license our code release upon acceptance.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi wwwwwtheeeee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We did not use L.Ms except for training LLM-based MILMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /t//urrrippp pis..pps.... cc/ /Ct.feeeeeeecc///////c 2025/LLM) for what should or should not be described.
22

---

Table Using additional data during PT v/s embedding optimization. Our VisPer-LM demon- strates superior performance than the LLaVA-1.5 model trained on with additional ALLaVA- Caption [6] data during the PT stage, underscoring the effectiveness of our approach with limited data. We use CLIP-ConwNeXT-XX- [14, 50] as the base visual encoder.
Method
PT
IFT
CV-Bench CV-Bench
MMStaa
OK-VQA
Avg
LLaVA-S58K LLaVA-1:5 LLaVA-558K MLLLaV-Captton-dfit
LLaVA-1:5
LLaVA-665k
60.0
56.3
37.4
56.0
52.4
ILaVA-6653
56.8
60.8
37.1
57.5
53.1
VisPer-LM
LLaVA-558K
LLaVA-665K
Table II: Setting N = 8 is optimal with N = 0 setting also giving boosts on CV-Bench and MMStar.
N
CV-Bench2D
CV-Bench
MMStar
RWQA
Avg
single-encoder
56.0
61.0
38.8
57.8
53.4
56.1
62.0
40.1
56.3
53.6
58.6
64.2
39.5
57.9
55.1
56.6
63.6
37.1
54.5
52.9
24
55.7
60.0
39.3
57.4
53.1
Table III: Key input to the Embedding Predictor. Feeding the tokens corresponding to the system prompt, image, corresponding special tokens, and the text query is optimal.
key imput emb. predictor
CV-Bench"
CV-Bench
MMStar
RWQA
Avg
(img)
(t)
$3.0
54.6
38.4
56.7
50.7
sys) (img) (t)
58.7
63.0
38.8
57.4
54.5
(sys)
(img) (0) (tat)
58.6
64.2
39.5
57.9
55.1
Appendix
In this appendix, we first share additional ablations, including the effect of the order of different special tokens (d).())(()) in the input sequence to the LLM and the different key input possibil- ities to the embedding predictor in Appendix A. We use CLIP-ViT-L [55] and Llama3-8b [64] as the base vision encoder and decoder LLM, respectively, for the ablations, unless mentioned oth- erwise. Next, demonstrate the effectiveness of our probing setup for encoder-free MLLMs in Appendix C and establish a clear correlation between probing performance and CV-Bench accuracy in Appendix D. Lastly, we provide qualitative and quantitative analysis the downstream probing tasks in Appendix E.
A Additional Ablations
Number of Special Tokens. We analyze the effect the number of special tokens per target feature ((t)) the performance in Tab. II. Setting N eight results in the best average performance across benchmarks. Furthermore, VisPer-LM without (t) also outperforms the baseline on CV-Bench 3D and MMStar, respectively, demonstrating the effectiveness of our embedding optimization.
Input tokens Embedding Predictor. As shown in Tab. III, we find that including the tokens corresponding to the system prompt in the key input to the embedding predictor is critical for per- formance. We attribute it to system tokens having high attention scores and effect on the genera- tion [73]. Therefore, distilling target information into the system tokens is crucial for performance. Moreover, including the text query tokens in the key input to the embedding predictors also results in slight performance boost as the text tokens hold global image information [33].
Embedding Optimization Mode. In this ablation study, we evaluate various combinations of em- bedding losses applied during pretraining (PT). Our results, summarized in Tab. IV, reveal that the optimal performance achieved when all three embedding losses depth, seg, and gen-are are used together. Interestingly, we observe that utilizing only depth or gen embedding losses still leads to notable performance improvements, whereas relying solely on seg embedding loss does not yield significant gains. This suggests that different types of target information contribute uniquely to the distillation process. Investigating how the distillation of one type of target information influences the effectiveness of others presents intriguing direction for future research.
23

---

Table IV: Embedding Optimization Modes. simultaneously is optimal.
Using the depth, seg, and gen embedding losses
mode
CV-Bench
CV-Bench
MMStar
Avg
L.LVA-1.5
56.0
61.0
38.8
51.9
depth
58.6
63.5
38.8
53.6
seg
56.2
57.6
38.2
50.7
gen
56.2
65.8
39.3
53.8
depth + seg
58.6
61.8
38.6
53.0
depth + gen
53.6
61.8
38.8
51.4
seg gen
54.2
60.2
39.3
51.2
depth + + gen
58.6
64.2
39.5
54.1
Table V: Order of different special tokens in the input sequence to the LLM. Appending the
gen, depth, and seg tokens (in that order) in the LLM's input sequence after the image tokens is the optimal setup.
(1) order
2ount
Depth 3D
Relation
Distance
Overall
50.4
73.3
64.9
48.7
58.5
(d) (
(
49.4
68.7
69.2
56.2
59.9
(8)
51.6
72.8
70.3
54.5
61.4
9g
48.7
71.3
65.2
52.5
58.5
(8)
9
(d)
46.7
71.3
71.2
50.8
58.9
9
(d)
(8)
51.3
74.2
69.4
54.3
61.4
(0)
(8)
(4)
50.9
68.8
70.0
50.5
59.2
Table VI: Embedding Loss weights during PT. Setting each embedding loss' weight to 0.5 is optimal.
Adepth
Aseg
Agen
CV-Bench
CV-Bench
MMStar
Avg
LL5
56.0
61.0
38.8
51.9
0.10
0.10
0.10
60.5
61.3
38.3
53.4
0.25
0.25
0.25
56.3
59.4
37.1
50.9
0.50
0.50
0.50
58.6
64.2
39.5
54.1
0.75
0.75
0.75
57.9
59.4
37.6
51.6
1.00
1.00
1.00
55.8
61.7
38.1
51.9
Table VII: Ablations on components of embedding losses. Using both smooth-Ll-loss and con- trastive loss to compute the final embedding loss is optimal.
LLL1
Coontrative CV-Bench CV-Bench
MMStar
Avg
56.8
62.3
38.3
52.5
58.6
64.2
39.5
54.1
Order of Special Tokens. In Tab. V, we ablate the order of different special tokens in the L.M''s input sequence. We find that {(g) (d) {s)} and {(d) (g) (s)} show the best performance on
CV-Bench [67]. We choose {g) (d) (s) as our default order due to its performance being better
than the baseline all sub-tasks in CV-Bench.
Embedding Loss weights. In Tab. VI, we ablate on different values of Adepti- Aueg and Agen during for the corresponding embedding losses during the pre- pre-training stage. We find that setting each loss weight to 0.5 is optimal.
Effect of Contrastive Embedding Loss. In Tab. VII, analyze the impact of the contrastive loss component within the embedding loss. Our findings show that incorporating the contrastive loss significantly enhances performance, highlighting its positive influence on the model's effectiveness.
24

---

Table VIII: Enhanced Projector post Predictive Embedding Optimization. We observe that the projector features in VisPer-LM are better than those in LLaVA-1.5 after the PT stage (proj PT), indicating a positive effect of the embedding losses during the PT stage on the learned projector. We use TLIP-CConNeXT-XXX and Llama3-8B as the base visual encoder and decoder L.M,, respec- tively.
Category
Model
Module Output Probed
Stage
Probe cosine-sim Scores
depth
seg
gen
avg
vision.encede
CLIP ConyNeXT-XXL
0.545
0.512
0.719
0.592
LLaVA-1.5
Projector
PT
0.531
0.495
0.781
0.602
proj PT
VisPer-LM (ours)
Projector
PT
0.554
0.503
0.776
0.611
bncod
Deodl
Figure I: Visualizing Embedding Predictor Outputs after the PT stage. The quality of the de- coded representations indicates the effectiveness of our embedding optimization.
Image
Depth Anything
One
Figure Ground-trute outputs from the target models used for Probing MLLMs.
Table IX: Evaluating Encoder-Free MILMs on CV-Bench. Our VisPer-LM significantly outper- forms native encoder free VLMs like EVE-7B [19] and Chameleon-7B [63].
Method
Visual Encoder
LLM
Count
Depth
Relation
Distance
Overall
Chamelcon [63]
Llama2-7b
13.6
51.7
49.5
50.8
39.6
EVE [19]
Vicuna-7b
52.2
61.3
69.2
53.8
58.4
LLaVA-1.5
ELIP-CoINNeXT-XLL ILIP-COMNNeXT-XX
Llama3-8b
54.1
62.8
69.5
49.8
58.2
VisPer-LMM (ours)
Llama3-8b
57.4
71.5
66.8
52.8
61.5
We keep the smooth L1 loss as default component to ensure the embedding predictions maintain the same magnitude the target features, which is crucial for meaningful visualization.
Qualitative Comparisons. We provide qualitative comparisons demonstrating the difference be- tween LLaVA-115 [45] and VisPer-LM for the different tasks in CV-Bench in Fig. VI, Fig. VII, Fig. VIII, and Fig. IX.
Visualizing Embedding Predictions for Target Tasks. We visualize the visual quality of LLM representations after the PT stage for our VisPer-LM using the decoders from the corresponding target models, as shown in Fig. I. We observe that the decoder outputs have good object shape and boundary quality, demonstrating the successful representation optimization with our embedding losses.
25

---

Cosine- Simileritttt
depth
Cosine Similarity
Figure III: Probing Encoder- Free MLLMs. We probe Chameleon- [63] and EVE-7B [19] mod- els show the effectiveness of our probing setup for encoder-free MILMs. In the first row, compare the mentioned models to LLaVA-1.5 and our VisPer-LM from Sec. 3 of the main text. On the one hand, we notice that EVE-7B shows worse probing performance in the initial layers than LLaVA-1.5, and the representation quality follows an upward trend even in deeper layers, resulting features of similar quality LLaVA-1.5, which is also reflected the CV-Bench accuracy. the other hand, Chameleon- 7B has the best representation quality in the early middle layers, with the quality dropping significantly in the deeper layers. We include only the EVE-7B and Chameleon in the second row for better readability.
B Enhanced Projector post Fre-Training
In this section, probe the features from the visual encoder and the projector for the LLaVA-1.5 and VisPer-LM models from Sec. 3 of the main text, i.c., using CLIP-ConWNeXT-XXL and the Llama3-8B the base visual encoder and decoder LLM, respectively. Note that only the projector trained during the PT stage, while the projector and the LLM are trained during the IFT stage. The visual encoder is kept frozen during both stages.
We compare the probing scores for the feature outputs from the projector module of the LLaVA-1.5 and VisPer-LM models between checkpoints obtained after the PT stage (proj_PT) of training. As shown in Tab. VIII, the representations in models under proj.P are better VisPer-LM than LL. indicating the predictive embedding optimization assisting in learning improved projector before the IFT stage.
We observe that the quality of depth and representations in LLaVA-115 under proj PT are worse than those of the base visual encoder (visi wwl.siooeencooo.oo encoder). indicating loss of depth/spatial information after the features are fed into the projector in LLaVA-115. However, for VisPer-LM. the depth representations have better quality than those from the vision encoder. Note that OLA=VLM's seg representations are slightly worse than those from CLIP-ConwNeXT-XXL but still better than those from 1..
C Probing Encoder-Free MLLMs
Following the settings described in Sec. 3 of the main text, probe all layers from the publicly available encoder-free Chameleon-77 [63] and EVE-7B [19] model checkpoints against the depth, seg, and gen target features in Fig. III.
notice that the representation quality features from EVE-7B relatively worse in the initial layers, owing to the patch embedding operation that outputs coarse features. However, the probing performance improves in the deeper layers, which could attributed to the patch-alignment op- eration [19]. Notably, EVE-7B uses about 35M [19] training samples but still lags slightly behind
26

---

Table X: Correlation of CV-Bench performance to probing performance. We notice the high positive correlation of 0.98 between depth probing performance and CV-Bench accuracy. Moreover, find that performance the Count and Depth subtasks also highly correlates with the seg and depth probe cosine-similarity scores, respectively. Additionally, we find low correlation between the gen probing performance and CV-Bench performance.
Probe Mode
Count RD
Depth 3D
Relation 3D
Distance
CV-Bench
CV-Bench
CV-Bench
depth
0.90
0.98
0.70
-0.31
0.96
0.94
0.98
seg
0.98
0.93
0.50
-0.15
0,91
0.96
0.98
gen
0.34
0.46
0.27
-0.31
0.37
0.36
0.37
LL in performance (Tab. IX), with our VisPer-LM showing better representation quality and performance, underscoring the effectiveness of our approach.
Unlike EVE-7B, features from Chameleon-7B [63] have surprisingly low depth and seg represen- tation quality in the deeper layers. Unlike other models. Chameleon-78 7B follows a steep downward trend in the deeper layers, which we attribute to its relatively low training on VQA data with heavy focus text-to-text and text-to-image tasks in the SFT data mixture, which is also evident from its low performance on CV-Bench. Note that during evaluation, we noticed that Chameleon cannot output option letters required to evaluate on MCQ-type CV-Bench. Therefore, we leverage Dwwn-2.5-772--Insiii Instrruutt [66] to extract the option letters corresponding to the sentence output from Chameleon to obtain the numbers for Tab. IX.
For both the models, gen probes perform relatively better due to the text-aligned nature of the gen target features.
These findings provide critical insights into developing future encoder- free MLLMs, eetabaablg the importance of VQA data in the SFT mixture, and supervising the vision tokens inside the LLM from an expert encoder during training. Our proposed embedding optimization opens up a promising avenue to this end.
D Correlation between VQA Performance and Probing Performance
Takeaways
1. As shown in Tab. X, we observe the highest positive correlation between CV-Bench accuracy and seg/depth probing performance.
2. Moreover, we find a high correlation between the performance on the Depth/Relation task and Count tasks with the depth and seg probing performance, respectively.
3. As expected, we find low performance correlation with gen probe scores, indicating that gen features act as good proxy for general visual representation targets.
4. Surprisingly, we observe a negative correlation between the Distance task and all probe scores, which could be attributed to the lack of "real-world distance" estimation queries in the training dataset.
In this section, we analyze the correlation between the performance on CV-Bench (evaluates models' visual perception ability) and probing performance (measure of model's representation quality), basing our conclusion on trends for six models and their corresponding depth/seg/gen probes:
LLaVA-L.5-8B-500 %-FFT: A LLaVA-1.5 model trained with complete pre-training (PT) but only 50% completion of the instruction fine-tuning (IFT) stage. It employs CLIP- ConxNeXT-XXLL [14, 50] as the visual encoder and Llama3-8B [64] as the decoder LLM. -LLVVA-15--B: LLaVA-1.5 model trained with complete PT+IFT stages. employs
FLIP-ConwNeXTT-XXL [14, 50] as the visual encoder and Llama3-88 -B [64] as the decoder L.M.
VisPer-L.M-BB: Our VisPer-LM model trained with complete PT+IFT stages while us- ing predictive embedding optimization during the PT stage. It employs CI.P--CovNeeTT XXL [14, 50] as the visual encoder and Llama3-8B [64] the decoder LLM.
27

---

CV-Rwnch
CV-Bmnch-30
CV-Bench-DD
CorManc
Volee 0818377
14:78 wuc La un
430
120
1200
0.99
500
CDD
50D
Coreade 0ay
Cenecden (X)
uuis 12 Mt
475
14 WT
Figure IV:
Plotting Correlation between Probing performance and CV-Bench accuracy. We find a high correlation between the eval cosine-sim scores and the accuracies on the CV-Bench as well as the 2D and 3D categories under CV-Bench for the depth and seg probes, validating their choice as the target task to improve the MLLM's perception ability.
LLaVA-L.S-BB-VPT: VL- model trained with complete PT+VPT+IFT stages. It employs CLIP--CowNeeT--XX. [14, 50] as the visual encoder and Llama3-8B [64] as the decoder L.M..
VisPer-LMMSBB-PT:: Our VisPer-LM model trained with complete PT+VPT+IFT stages using predictive embedding optimization during the PT stage. It employs CLIP- ConvNeXT-XXL [14, 50] as the visual encoder and Llama3-8B [64] as the decoder LLM. EVE--B: The publicly available EVE-7B model checkpoint from [19]. It does not use a visual encoder and is built on Vicuna-7B [15]
We follow the same settings as mentioned in Sec. and Sec. 5 of the main text to train our probes and models, respectively.
Probing Mode: depth. We observe a high correlation of 0.98 between the CV-Bench accuracy and depth probe eval cosine-sim scores, as shown in Fig. IV. It is critical proof of our claim that improved visual representations and the corresponding VQA performance go hand-in-hand. We also find a correlation with a 0.98 and 0,80 factor between the accuracy on the Depth and Relation tasks (in CV-bench), respectively, with the depth probe scores in Fig. V, explaining the gains achieved by VisPer-LM.
Probing Mode: seg. We also observe a high correlation of 0.98 between the CV-Bench accuracy and seg probe eval cosine-sim scores, as shown in Fig. IV. The high correlation indicates that CV-Bench requires strong perception abilities, validating our decision to adopt it as the primary benchmark for our work. We also find a correlation of 0.98 between the accuracy on the Count task (in CV-bench) with the seg probe scores in Fig. V.
28

---

Court
Depth
Fatarin
Clstance
Scation XM
Figure V: Plotting Correlation between Probing performance and CV-Bench sub-tasks accuracy. We find a high correlation between the depth eval cosine-sim scores and the DeptthRelltion/CCunt task performances. We make the same observation about seg probe scores and the Count/Depth task. Interestingly, we find a low negative correlation between the probe scores and Distance task performance.
Count (2D)
How many pictures are in the image? Select from the following choices.
(A) 3
(B)
(C) 0
(D)
LLaVA-1.5: (C)
VisPer-LM: (D)
Ground-Truth: (D)
How many trees in the image? Select from the following choices.
(A)
(B) 0
(C)
(D)
LLaVA-1.5: (A)
VisPer-LM: (C)
Ground-Truth: (C)
Figure VI: Qualitative Examples for the Count task in CV-Bench. Our VisPer-LM can accurately predict the presence of one picture and one tree, unlike LLaVA-1.5 [45].
Probing Mode: gen. We observe that the gen probe scores do not correlate highly with CV-Bench or any its tasks, as shown Fig. IV and Fig. V. We attribute this finding to the generative features' text-aligned nature, which is not optimal for developing the model's perception abilities.
29

---

Depth (3D)
Which object is closer to the camera taking this photo, the lamp (highlighted by red box) or the books (highlighted by blue box)?
(A) lamp (B) books
LLaVA-1.5: (A)
VisPer-LM: (B)
Ground-Truth: (B)
Which object is closer camera taking photo, keyboard (highlighted by red box) or the chair (highlighted by blue box)?
keyboard
(B) chair
LLaVA-1.5: (B)
VisPar-LM: (A)
Ground-Truth: (A)
Figure VII: Qualitative Examples for the Depth task in CV-Bench. Our VisPer-LM can accurately predict that the lamp and keyboard ar closer to the camera in the respective samples.
Relation (2D)
Considering the relative positions of the trees (annotated by the red box) and the shore in the image provided, where the trees (annotated by box) located with respect shore? Select the following choices.
(A) above (B) below
LLaVA-1.5: (B)
VisPer-LM: (A)
Ground-Truth:
Considering the relative positions of the bottle (annotated the red box) and the person in the image provided, where the bottle (annotated by the red box) located with respect to the person? Select from the following choices.
(A) left
(B) right
LLaVA-1.5: (A)
VisPer-LM: (B)
Ground-Truth: (B)
Figure VIII: Qualitative Examples for the Relation task in CV-Bench. Our VisPer-LM can accu- rately predict that the positions the trees and the bottle in the respective samples.
E Evaluating Probes on Downstream Tasks
We evaluate the probes trained against the target features on the corresponding downstream target tasks, i.c.. image generation, depth estimation and image segmentation. To obtain the predictions, we feed the outputs from the probes into the decoder from the corresponding target model. Specifi- cally, we report the FID [60] scores 5k images from COCO-va12017 [43] for gen probes, accuracy on the DA-2K [75] benchmark for depth probes, and mloU [31] on the COCO-vaI200 [43] set for seg probes. We average the scores over all layers for easier comparison, As shown in Tab. XI, probes trained for our VisPer-LM outperform those for the baseline VA-.5- [45] model across all prob- ing tasks, proving the improved visual representation quality owing to our embedding optimization approach.
30

---

Distance (3D)
Estimate wealdworld distances between objects this image. Which object closer to the books (highlighted by red box), the lamp (highlighted by blue box) or the table (highlighted by green box)?
(A) lamp (B) table
LLaVA-1.5: (B)
VisPer-LM: (A)
Ground-Truth: (A)
Estimate the real-world distances between objects this image. Which object closer stationary (highlighted a red box), the chair
(highlighted by a box) or the mouse (highlighted by green box)?
(A) chair (B) mouse
LLaVA-1.5: (A)
VisPer-LM: (B)
Ground-Truth: (B)
Figure IX: Qualitative Examples for the Distance task in CV-Bench. Our VisPer-LM can accu- rately predict that the distances between the respective pair objects.
Quantitative Evaluation target probing task. Probes trained for our VisPer-LM perform significantly better as compared to the probes trained on baseline LLaVA-1.5 [45].
Table XI:
Probed Model
FID [60] (L) DA-2K % Acc. [75] (1)
% mloU [31]
LLaVA-1.5
23.1
66.4
39.3
VisPer-LM (ours)
22.4
77.8
45.4
Target Encoder
18.1
97.3
64.5
LLaVA-1.5
VisPer-LM (ours)
M
M
M
C
MKC
K
M
MO
PD
DRT
WVT
MRD
CRT
SEE
M
ADC
MRC
M
Figure X: Layerwise visualizations for the depth probes. For LLaVA-1.5 [45], the probes generate blob-like outputs up the eighth layer, with visualizations progressively improving in the middle layers, aligning with the findings presented in Sec. 3 of the main text. Notably, probes trained on VisPer-LM begin producing distinguishable object shapes and boundaries as early the third layer, attributed to the incorporation of embedding losses leading to improved representations in the initial layers.
31

---

LLaVA-1.5
VisPer-LM (ours)
Figure XI: Layerwise visualizations for the seg probes. The LLaVA-1.5 probes often fail to seg- ment the third car in the background for the first sample during the initial layers (layers two to eight), whereas the VisPer-LM probes demonstrate relatively better performance in this scenario.
LLaVA-1.5
VisPar-LM (ours)
Figure XII: Layerwise visualizations for the gen probes. The probe outputs for both the models are of fairly good quality.
Visualizing Probe Outputs. We present visualizations for the outputs for the probes trained on the single-encoder LLaVA-1.5 model Sec. in the main text. We provide the ground-truth visualizations from the teacher models in Fig. II.
As shown in Fig. X, the probe visualizations for the first eight layers of LLaVA-1.5 exhibit blob- like patterns, while the later layers progressively enhance the shape and boundary details of the foreground objects. In contrast, the probe visualizations for our VisPer-LM demonstrate improved object shapes and boundaries starting as early as layer-4, consistent with the findings on represen- tation quality and layer-wise trends discussed in Sec. 3 of the main text. Additionally, we present probe visualizations for the seg and gen representations in Fig. XI and Fig. XII, respectively.
32


Reviews:
[{"review_id": "HOKwzL99Z5", "reviewer": "Reviewer_KZsA", "summary": "* This paper identifies that standard MLLM training with only a next-token prediction objective can result in models with underdeveloped visual perception.  \n* To address this, the authors propose VisPer-LM, a novel training approach that distills knowledge from multiple “expert” visual encoders (eg, for depth, segmentation) directly into the intermediate hidden layers of the LLM.  \n  * This is accomplished via an auxiliary embedding loss during the pre-training stage (and *only* this stage)  \n  * ⇒ This is a key advantage of this method: the expert encoders are only used during training, adding no latency at inference time.  \n* The authors first use a probing methodology to establish a correlation between internal visual representation quality and downstream performance, and then show that VisPer-LM improves performance on perception-heavy benchmarks compared to standard MLLM baselines.", "questions": "1. **Dimensionality of the \"Generation\" Probe Target**  \n   - In Section 4.1, you note that the target for the \"generation\" features (Etextgen) has only one token, unlike the 576 tokens for the depth and segmentation features  \n   - Could this significant difference in the prediction target's dimensionality explain the highly stable and distinct probing performance observed for the gen features in Figure 3?  \n2. **Using Multi-Token Generative Features for a Cleaner Comparison**  \n   - Following up on Q1… Other works have successfully extracted multi-token features from generative model  \n      - E.g., Cambrian-1 uses a 1024-token representation from Stable Diffusion  [[paper]](https://arxiv.org/abs/2406.16860)  [[diffusion_encoder.py]](https://github.com/cambrian-mllm/cambrian/blob/main/cambrian/model/multimodal_encoder/diffusion_encoder.py)  \n      - → from their code, it seems this is based on methods from \"Emergent Correspondence from Image Diffusion\" (DIFT)  [[paper]](https://arxiv.org/abs/2306.03881)  [[DIFT code]](https://github.com/Tsingularity/dift)  \n   - Have you considered using a similar multi-token generative feature extractor? This would create a much cleaner comparison against your multi-token depth and segmentation experts and help isolate the effect of *what* kind of knowledge is being distilled from the *dimensionality* of the distillation target.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. Introduces a novel and clever method for improving the visual perception of MLLMs. The core idea of distilling knowledge from expert encoders directly into the LLM's intermediate layers is well-motivated and insightful  \n2. Provides an excellent trade-off between performance and efficiency. It successfully infuses rich visual information into the model during training without incurring the inference-time latency of traditional multi-encoder approaches  \n3. Supported by a strong and novel probing analysis (Section 3\\) that empirically grounds the motivation for the proposed method by correlating internal representation quality with downstream task performance  \n4. The paper is well-executed, with extensive ablations that explore the key design choices of the proposed method\n\n**Weaknesses:**\n\n1. The final performance gains on the main benchmarks, while consistent, are relatively modest (e.g., \\~2.5% average improvement in Table 1\\)  \n   - ⇒ While the methodological contribution is strong, the practical impact in terms of state-of-the-art performance is not dramatic  \n2. While the distillation method is effective for the targeted perception tasks, its reliance on a curated set of specialized \"expert\" encoders raises questions about its generality.  \n   - It's unclear if this approach is a scalable path forward for improving MLLMs broadly, or if it imposes too much specific structure, potentially at the expense of more general reasoning capabilities that might emerge from simpler, larger-scale training.", "was_revised": true, "final_justification": "Most of my concerns are addressed. This is a good paper worthy of acceptance.", "rebuttal": "We thank the reviewer for their feedback and comments.\nWe are glad they found our idea *“well-motivated and insightful”* and our method *“novel and clever.”*\nWe appreciate their recognition of our method's *“excellent trade-off between performance and efficiency”* and the *“empirically grounded motivation from a strong and novel probing analysis.”*\nWe are also grateful for their positive remarks on the our work's execution and extensive ablations.\n\nWe respond to their questions and concerns below:\n\n### **Pt #1: Dimensionality of the \"Generation\" Probe Target**\n\n> Reviewer's Comment: In Section 4.1, you note that the target for the \"generation\" features (Etextgen) has only one token, unlike the 576 tokens for the depth and segmentation features.\n>\n> Could this significant difference in the prediction target's dimensionality explain the highly stable and distinct probing performance observed for the gen features in Figure 3?\n\nThat is an interesting point and perspective! In the main text (L#147-148), we attribute the superior gen probing performance to the choice of $\\textbf{E}^{\\text{gen}}$ (from unCLIP-SD-2.1 [56]) which produces visual features that are already language-aligned owing to its training setup, making it easier for the probes to learn a mapping from the LLM representation (already in the language space).\n\nTherefore, we believe in this particular case, the nature of expert features plays the major role and not the number of tokens. Nonetheless, ablating different sets of expert encoders for probing and embedding distillation remains a promising direction for future work. We thank the reviewer for this intriguing question!\n\n[56] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv, 2022.\n\n### **Pt #2: Using Multi-Token Generative Target Features**\n\n> Reviewer's Comment: Following up on Q1… Other works have successfully extracted multi-token features from generative model\n>\n> E.g., Cambrian-1 uses a 1024-token representation from Stable Diffusion\n> from their code, it seems this is based on methods from \"Emergent Correspondence from Image Diffusion\".\n>\n> Have you considered using a similar multi-token generative feature extractor? This would create a much cleaner comparison against your multi-token depth and segmentation experts and help isolate the effect of what kind of knowledge is being distilled from the dimensionality of the distillation target.\n\nThank you for your suggestion and sharing the exact reference! We share results with the usage of 1024-token outputs from the SD-2.1 as the gen expert features in the table below. We observe that the features from unCLIP-SD-2.1 (used in the main text) performs much better than using the 1024 token representation from SD-2.1 (VAE+UNet). Therefore, we believe that the information captured by the feature embeddings plays a more critical role than their dimensionality and a strong generative prior may even harm the MLLM's understanding performance in some cases, therefore it is important to choose the experts wisely. Nonetheless, future research into comparison of different experts is an intriguing direction.\n\n| gen expert features             | Depth | Relation | Count | Distance | MMStar | Avg   |\n|--------------------------------|:-----:|:--------:|:-----:|:--------:|:------:|:-----:|\n| single token from unCLIP-SD-2.1 | 69.4 |  74.2    | 51.3  |   54.3   |  39.5  | **57.7** |\n| 1024 tokens from SD-2.1 (VAE+UNet) | 61.2  |  66.8    | 47.6  |   55.7   |  35.8  | 53.4 |\n\n### **Pt #3: Impact of VisPerLM**\n\n> Reviewer's Comment: The final performance gains on the main benchmarks, while consistent, are relatively modest (e.g., ~2.5% average improvement in Table 1)\n>\n> ⇒ While the methodological contribution is strong, the practical impact in terms of state-of-the-art performance is not dramatic\n\nWe appreciate the reviewer’s recognition of our methodological contribution's strength. Enhancing MLLMs in visual perception and reasoning remains an active area of research, with several recent works—such as Cambrian-1 [66], which leverages multiple encoders to address the limitations of single-encoder MLLMs, and AM-RADIO [57], which distills knowledge from multiple teacher encoders into a single, enhanced encoder.\n\nWe compare our VisPerLM against these approaches in Table 1 of the main text, demonstrating that our method outperforms both under identical data settings, highlighting the effectiveness of our proposed approach.\n\nWe believe a 2.5% improvement on average is not modest in an academic setting where we do not use any extra/different data to train the MLLM compared to the baselines. However, we acknowledge the reviewer’s concern regarding the practical impact in terms of state-of-the-art performance. As noted in our limitations, we were unable to conduct experiments at larger scale due to resource constraints. Nonetheless, we believe the current results provide strong empirical evidence supporting a new paradigm for improving MLLMs. We also hope our probing framework serves as a valuable tool for understanding the underlying factors driving performance gains for the community.\n\n[66] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms, NeurIPS 2024.\n\n[57] Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In CVPR, 2024\n\n### **Pt #4: VisPerLM does not harm general reasoning**\n\n> Reviewer's Comment: While the distillation method is effective for the targeted perception tasks, its reliance on a curated set of specialized \"expert\" encoders raises questions about its generality.\n>\n> It's unclear if this approach is a scalable path forward for improving MLLMs broadly, or if it imposes too much specific structure, potentially at the expense of more general reasoning capabilities that might emerge from simpler, larger-scale training.\n\nWe remind the reviewer that VisPerLM's core idea is to improve an MLLM at target perception tasks without any loss of its general reasoning abilities. For example, our approach provides a promising direction to improve current video understanding models at tasks like motion understanding. We leave the exploration of our method with large scale data and more target tasks to future work as noted under limitations."}, {"review_id": "Gh70HXgUUV", "reviewer": "Reviewer_jTWX", "summary": "Current  MLLMs is trained with natural language supervision, causing models to ignore the rich visual perception signals present in the data. This work finds that the quality of visual representations is positively correlated with the performance on downstream tasks. Therefore, the authors proposes to optimizes visual embedding prediction alongside next (text) token prediction during training. Specifically, it adds supervision from target visual information to the embeddings of selected LLM layers to improve visual embedding prediction. The proposed method, VisPer-LM, outperforms both single- and multi-encoder baseline methods on various benchmarks.", "questions": "1. During training, computing the three target probe features takes time. Compared to the multi-encoder approach, is the training time actually reduced? Specifically, where is the efficiency improvement reflected?\n2. In the results shown in Figure 3, for the first and third rows, the performance of VisPer-LM is almost indistinguishable from the single encoder in the deeper LLM layers, and in the generation task, the single encoder even performs slightly better. This suggests that training with these three tasks does not significantly improve the quality of the visual embeddings. Therefore, I have doubts that the performance improvement on the benchmarks is actually due to the use of more data and an extended training stage, rather than an improvement in the quality of the three target visual embeddings.\n3. Why does the multi-encoder have the highest visual embedding quality in Figure 3, but perform worse than VisPer-LM in Table 1? Doesn’t this contradict the paper’s claim that the quality of visual representations is positively correlated with downstream task performance?\n4. As training progresses and the number of steps increases, does the quality of the visual embeddings and the model’s performance continue to improve accordingly?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. This paper proposes a method to enhance visual representations by distilling three target visual information, achieving a trade-off between efficiency and representation quality in both the single encoder and multi-encoder settings.\n2. Extensive experiments demonstrate the effectiveness of the proposed approach.\n3. Additional experiments analyze the choice of key parameters and the design of the model structure.\n\nWeakness\n1. The training time for each stage and the efficiency compared to the multi-encoder are not reported. The total loss includes three additional depth, segmentation, and generation tasks, respectively. Compared to the multi-encoder approach, is the training time actually reduced?\n2. Since both increasing the training data and distilling the three target visual information are performed simultaneously, a further comparative explanation of the impact of these two operations is needed.\n3. In Figure 3, it is confusing that the multi-encoder baseline has the best probing performance but achieves lower performance than VisPer-LM. Also, the probing performance of VisPer-LM that training with these three tasks does not significantly improve the quality of single encoder.", "was_revised": true, "final_justification": "The rebuttal addresses my concerns. I am now convinced that the performance improvements are primarily due to the enhanced visual embedding quality. I will update my scores to Borderline accept.", "rebuttal": "We thank the reviewer for their feedback and comments. \nWe are glad that they acknowledge the effectiveness of our approach supported by extensive experiments and analysis.\n\nWe respond to their questions and concerns below:\n\n### **Pt #1: Training Time and Inference Time Efficiency**\n\n> Reviewer's Comment: The training time for each stage and the efficiency compared to the multi-encoder are not reported. The total loss includes three additional depth, segmentation, and generation tasks, respectively. During training, computing the three target probe features takes time. Compared to the multi-encoder approach, is the training time actually reduced? Specifically, where is the efficiency improvement reflected?\n\nWe would like to clarify that by performance-efficiency trade-off in the main text (L#48), we mainly refer to the inference time efficiency. Nonetheless, we report numbers for both inference and training time below to demonstrate the efficient nature of VisPer-LM compared to the multi-encoder baseline.\n\n**Inference time:** We demonstrate the superior inference throughput of VisPer-LM over the multi-encoder approach in Tab. VIII in appendix A. We report the numbers again below for the reviewer's reference:\n\n| inference                  | single-encoder      | multi-encoder      | **VisPer-LM (ours)**        |\n|---------------------------|---------------------|--------------------|----------------------|\n| throughput (samples/sec)  | 9.92 ± 0.03       | 5.32 ± 0.02       | 9.86 ± 0.01       |\n\nOur VisPer-LM has superior inference throughput compared to that of the multi-encoder baseline and similar throughput to that of the single-encoder baseline with much better performance. We record the throughput on a single NVIDIA 80G A100 GPU for a single forward pass on the CV-Bench evaluation set with a batch size of 1. We report the mean and standard deviation across 10 runs.\n\n**Training time:** We report stage-wise training time for the different approaches below for CLIP-ConvNeXT-XXL as our base visual encoder and Llama3-8b as the llm decoder on 8 A100 GPUs.\n\n| training                 | single-encoder      | multi-encoder      | **VisPer-LM (ours)**        |\n|---------------------------|---------------------|--------------------|----------------------|\n| PT stage   | 9 hours       | 16 hours       | 16 hours       |\n| IFT stage  | 15 hours       | 23 hours       | 15 hours       |\n| total training time (PT+IFT) | 24 hours       | 39 hours       | 31 hours       |\n\nAs shown in the above table, our VisPerLM reduces the total training time by 8 hours as compared to the multi-encoder approach. Note that we only use the auxiliary losses during the PT stage, resulting in IFT training time comparable to that of the single encoder approach.\n\n### **Pt #2: All baselines (single and multi encoder) use the same training data and steps as VisPerLM**\n\n> Reviewer's Comment: Since both increasing the training data and distilling the three target visual information are performed simultaneously, a further comparative explanation of the impact of these two operations is needed.\n\nWe clarify that we use the same amount of training data and steps as the baselines for the numbers reported in Tab. 1 in the main text while demonstrating the superior performance of VisPerLM. Moreover, we show the impact of extra training data in Tab. 3 (with an extra VPT stage) in the main text, which also shows gains, showing the scalability of VisPerLM with more data. Therefore, our experiments already include separate experiments to demonstrate the inidividual effect of visual embedding distllation and increasing the amount of data.\n\n### **Pt #3: The probing experiments explain the multi-encoder performance on the Target tasks**\n\n> Reviewer's Comment: In Figure 3, it is confusing that the multi-encoder baseline has the best probing performance but achieves lower performance than VisPer-LM. Also, the probing performance of VisPer-LM that training with these three tasks does not significantly improve the quality of single encoder.\n>\n> Why does the multi-encoder have the highest visual embedding quality in Figure 3, but perform worse than VisPer-LM in Table 1? Doesn’t this contradict the paper’s claim that the quality of visual representations is positively correlated with downstream task performance?\n\nThat is a good question and observation! As explained on L#267-269 in the main text, the multi-encoder baselines are slightly better than VisPer-LM on the Depth task in CV-Bench owing to their superior depth representation quality, aligned with our claim that the *quality of visual representations is positively correlated with the downstream performance on the target tasks (Depth and Relation under CV-Bench)*. Note that we probe the LLaVA-1.5 (feat concat.) baseline in Fig. 3.\n\nThe multi-encoder baseline performs worse on average than our VisPerLM in Tab. 1 as the former is unable to retain general reasoning ability (on MMStar) same as the single-encoder baseline, unlike VisPerLM, demonstrating the effectiveness of our approach.\n\n### **Pt #4: Scalability with More training data**\n\n> Reviewer's Comment: As training progresses and the number of steps increases, does the quality of the visual embeddings and the model’s performance continue to improve accordingly?\n\nYes, as shown in the second row of Fig. 2(a) of the main text, as we increase the amount of training data (and therefore, the number of steps), the probing score and downstream performance gradually increase as expected. We observe the same phenomenon in row 2 of Fig. 3 as explained on L#156-170.\n\n### **Pt #5: Gen Probing Performance has low correlation with downstream performance**\n\n> Reviewer's Comment: In the results shown in Figure 3, for the first and third rows, the performance of VisPer-LM is almost indistinguishable from the single encoder in the deeper LLM layers, and in the generation task, the single encoder even performs slightly better. This suggests that training with these three tasks does not significantly improve the quality of the visual embeddings. Therefore, I have doubts that the performance improvement on the benchmarks is actually due to the use of more data and an extended training stage, rather than an improvement in the quality of the three target visual embeddings.\n\nAs explained under **Pt #2**, we use the same amount of data for training our VisPerLM model as the single-encoder baseline. Therefore, the the auxliary losses are solely responsible for the observed performance and visual representation quality improvement.\n\nMoreover, in appendix D, we find a high positive correlation of 0.98 between the gen/seg probing scores and downstream performance on CV-Bench while only a correlation of 0.37 for the gen probing scores. Therefore, single encoder baseline having slightly better gen probing scores is not a surprise due to the nature of our target tasks (Depth and Relation under CV-Bench).\n\nAs explained on L#143 in the main text, prior works [32, 33, 61] have shown that early-to-middle layers have the most significant effect on the reasoning performance in (M)LLMs, and our VisPerLM shows far superior (depth/seg) representation quality in those layers compared to the single encoder baseline, explaining its superior performance. Nonetheless, understanding how different layers contribute to an MLLM's reasoning abilities remains an interesting research direction for the future.\n\n[32] Omri Kaduri, Shai Bagon, and Tali Dekel. What’s in the image? a deep-dive into the vision of vision language models. arXiv, 2024.\n\n[33] Guy Kaplan, Matanel Oren, Yuval Reif, and Roy Schwartz. From tokens to words: On the inner lexicon of llms. arXiv, 2024.\n\n[61] Oscar Skean, Md Rifat Arefin, and Ravid Shwartz-Ziv. Does representation matter? exploring intermediate layers in large language models. In Workshop on Machine Learning and Compression, NeurIPS 2024."}, {"review_id": "D5T56JRvEE", "reviewer": "Reviewer_Khkk", "summary": "This paper introduces VisPer-LM, a Multimodal LLM pre-trained with visual embedding distillation from multiple vision experts to enhance visual perception. VisPer-LM follows the established structure Vision Encoder $\\rightarrow$ MLP $\\rightarrow$ LLM. \n\nThe work starts with a preliminary analysis, examining whether representations within the LLM decoder can be probed to visual representations of different vision experts (depth, text-to-image, and segmentation models). To achieve this, it utilizes a Resampler block, which enables control over the output sequence length through learnable queries, and experiments with a single- and multi-encoder MLLM design. The analysis shows that different layers lead to different probing performance and, more importantly, that probing performance positively correlates with performance on CV-Bench, a vision-centric benchmark. \n\nThis motivates the rest of the work in designing a pretraining scheme to distill visual representations from vision experts to (subsets of) LLM blocks. The approach utilizes the probing setup described above and optimizes both a Smooth-L1 objective and an InfoNCE objective to align Resampler outputs with those of experts, pairing these objectives with the standard language modeling loss. Following the LLaVA recipe, only the MLP is trained during this phase (alongside the Resampler heads and some auxiliary task tokens).  Empirical results show that further using this objective during Instruction Tuning hurts performance.\n\nThe paper evaluates vision-centric abilities on CV-Bench, as well as standard multimodal abilities, showing improved visual perception with little to no degradation on common academic benchmarks.", "questions": "For a productive rebuttal, I would start expanding the weaknesses above into the following questions:\n\n1. How much does the downstream performance of VisPer-LM suffer from informed decisions based on CV-Bench? Does improved visual perception persist when evaluating on other vision-centric benchmarks?  \n2. Did the authors experiment with other probing setups (*.e.g.* omitting textual tokens as well as / or tokens from the system prompt)? \n2. About the RADIO-based LLaVA: did the authors train it themselves, or does it come from some previous work I am unaware of?  \n3. Is there any evidence that some vision experts are unnecessary? This question mostly arises from the consistently good probing performance for the image-to-text generation expert, hence I wonder if this can be removed with little to no performance degradation;  \n4. Were multiple runs executed for these experiments? If so, can the authors please report the standard deviation as well? This would be helpful to understand which performance gains are significant and which are not. \n5. How are the sets of layers contributing to embedding distillation from different vision experts selected for the experiments with Phi3-4k-mini (Table 1)? \n\nI am definitely open to increasing my score after a productive rebuttal.", "limitations": "This work contains a Limitations section in the main body of the paper, which I think is sufficient.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths.** \n- This paper tries to improve Visual Perception in Multimodal LLMs, which I believe is an importance aspect due to their well-known overreliance on text; \n- Pairing the embedding distillation objective with the standard language modeling objective allows to maintain strong multimodal understanding alongside improving visual perception; \n- I think Section 3 (Preliminary Analysis) provides cool insights that can be useful for future research (*i.e.* results about layer importance according to the visual task, as well as probes improving consistently with the addition of data); \n- I overall think the experimental section is comprehensive; I particularly liked the \"data-matched\" Cambrian-1 experiment and the comparison against a RADIO-ViT-L based LLaVA; \n\n\n**Weaknesses.**  \n- I wonder how much some of the downstream results may suffer from the chicken-and-egg dilemma when it comes to CV-Bench. All preliminary analyses conduct extensive experiments on all CV-Bench tasks, which inform all design choices of VisPerLM, and the main evaluation for vision-centric capabilities is conducted on CV-Bench as well. I think it is important for this paper to test downstream performance on other vision-centric benchmarks (*e.g.* BLINK [a]) to ensure this confounder disappears, and check whether the improved vision-perception abilities are stable and consistent; \n- Reading the paper, it is a bit unintuitive that all tokens within LLM decoder blocks are used for probing, *i.e.*, since the goal of the resampler is to probe visual representations, I wonder whether only visual tokens alongside task tokens may be sufficient (or even better), compared to also using text tokens.\n\n**References.**  \n[a] Fu, Xingyu, et al. \"Blink: Multimodal large language models can see but not perceive.\" ECCV 2024.", "was_revised": true, "final_justification": "The discussion with the authors was positive, but some of my primary concerns (primarily regarding the transfer of visual understanding capabilities, the significance of the reported outcomes, and the generalizability of design choices to other LLM architectures) remain partially unaddressed. I do, however, acknowledge that this paper has merits as well. I am therefore inclined to lean towards acceptance and keep my original \"Borderline Accept\" rating.", "rebuttal": "We thank the reviewer for their feedback and comments.\nWe are glad that they recognize the importance of our effort to improve MLLMs at visual perception and that *\"Section 3 (Preliminary Analysis) provides cool insights that can be useful for future research\"*. We are encouraged that they found the experimental section *\"comprehensive\"* and *\"particularly liked the \"data-matched\" Cambrian-1 experiment and the comparison against a RADIO-ViT-L based LLaVA\"*.\n\nWe respond to their questions and concerns below:\n\n### **Pt #1: VisPerLM shows improvements on BLINK benchmark**\n\n> Reviewer's Comment: I wonder how much some of the downstream results may suffer from the chicken-and-egg dilemma when it comes to CV-Bench. All preliminary analyses conduct extensive experiments on all CV-Bench tasks, which inform all design choices of VisPerLM, and the main evaluation for vision-centric capabilities is conducted on CV-Bench as well. I think it is important for this paper to test downstream performance on other vision-centric benchmarks (e.g. BLINK [a]) to ensure this confounder disappears, and check whether the improved vision-perception abilities are stable and consistent;\n> \n> How much does the downstream performance of VisPer-LM suffer from informed decisions based on CV-Bench? Does improved visual perception persist when evaluating on other vision-centric benchmarks?\n\nThank you for the suggestion to evaluate our method on the BLINK benchmark! We report results on visual perception tasks under the BLINK benchmark (`val` set) in the table below. We use CLIP-ConvNeXT-XXL and Llama3-8b as the base visual encoder and base LLM, respectively. We also report results using the data-matched Cambrian-1 RADIO-ViT-L based LLaVA-1.5 models for the reviewer's reference.\n\n| Method                | Spatial Relation | Relative Depth |  Avg  |\n|-----------------------|:----------------:|:--------------:|:-----:|\n| LLaVA-1.5             |       72.0       |      51.4      | 61.7  |\n| RADIO-L LLaVA-1.5     |       65.7       |    **52.4**    | 59.1  |\n| Cambrian-1            |       74.1       |      51.6      | 62.9  |\n| **VisPer-LM (ours)**  |     **75.5**     |      51.6      |**63.6**|\n\nWe find that our VisPer-LM significantly outperforms all other baselines on average, especially on the `Spatial Relation` task, demonstrating the effectiveness of our approach.\n\n### **Pt #2: Effect of system prompt and text tokens on the probing performance**\n\n> Reviewer's comment: Reading the paper, it is a bit unintuitive that all tokens within LLM decoder blocks are used for probing, i.e., since the goal of the resampler is to probe visual representations, I wonder whether only visual tokens alongside task tokens may be sufficient (or even better), compared to also using text tokens.\n>\n>Did the authors experiment with other probing setups (.e.g. omitting textual tokens as well as / or tokens from the system prompt)?\n\nWe clarify that the goal of our probing setup is to evaluate the representation quality of all the tokens from the LLM as a whole and not just the tokens corresponding to the visual features. This is an important detail as prior works [32, 72] have shown the importance of system and text tokens during the LLM's token generation process. Therefore, in order to establish a correlation between the MLLM's representation quality and downstream performance, we believe it is critical to probe all tokens from a given sequence.\n\nNonetheless, we agree this is an interesting experiment! Therefore, we share the average probe scores (across all layers) for our VisPer-LM model under different token input settings to the resampler below:\n\n| Input to Probe Resampler                  | Depth   |  Seg   |  Gen   |\n|------------------------------------------|:-------:|:------:|:------:|\n| &lt;img&gt; &#124; &lt;t&gt;                   | **0.613** | **0.564** | 0.744 |\n| &lt;img&gt; &#124; &lt;t&gt; &#124; &lt;text&gt;   | 0.608   | 0.562  | 0.748  |\n| &lt;sys&gt; &#124; &lt;img&gt; &#124; &lt;t&gt; &#124; &lt;txt&gt; | 0.603   | 0.563  | **0.764** |\n\n&lt;sys&gt;: system tokens; &lt;img&gt;: visual tokens; &lt;t&gt;: task-specific tokens; &lt;txt&gt;: text tokens\n\nAs expected, dropping the system and text tokens, leads to the best depth and seg probing scores. Interestingly, using all tokens leads to the best gen probing performance which we attribute to the language aligned nature of the gen expert features (L#147 in the main text).\n\nHowever, note that we also conduct a similar experiment while training the model in appendix A Tab. III, i.e., ablating the tokens input to the embedding predictors for the auxiliary losses. We found that using all tokens as input to the embedding predictor leads to the best performance owing to the importance of the system and text tokens for LLM's reasoning.\n\n[32] Omri Kaduri, Shai Bagon, and Tali Dekel. What’s in the image? a deep-dive into the vision of vision language models. arXiv, 2024.\n\n[72] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. ICLR, 2024\n\n### **Pt #3: RADIO based LLaVA**\n\n> Reviewer's Comment: About the RADIO-based LLaVA: did the authors train it themselves, or does it come from some previous work I am unaware of?\n\nWe trained our own version of RADIO ViT-L LLaVA-1.5 based on Llama3-8b. We will clarify this in the main text. Thank you for pointing this out!\n\n### **Pt #4: Removing the Gen expert**\n\n> Reviewer's Comment: Is there any evidence that some vision experts are unnecessary? This question mostly arises from the consistently good probing performance for the image-to-text generation expert, hence I wonder if this can be removed with little to no performance degradation;\n\nThis is an interesting experiment! We provide ablations on the different combinations of auxiliary losses during training in appendix A (L#539-545).\nWe share the analysis from the article here for the reviewer's reference and direct the reviewer to Tab. IV in the appendix for the numbers:\n\n*Our results reveal that the optimal performance is achieved when all three embedding losses: depth, seg, and gen, are used together. Interestingly, we observe that utilizing only depth or gen embedding losses still leads to notable performance improvements, whereas relying solely on seg embedding loss does not yield significant gains. This suggests that different types of target information contribute uniquely to the distillation process. Investigating how the distillation of one type of target information influences the effectiveness of others presents an intriguing direction for future research.*\n\n### **Pt #5: Fixed seed for experiments**\n\n> Reviewer's Comment: Were multiple runs executed for these experiments? If so, can the authors please report the standard deviation as well? This would be helpful to understand which performance gains are significant and which are not.\n\nDue to resource constraints, we could not perform multiple runs. However, to ensure experimental validity and significance, we perform all our experiments with a fixed seed as also pointed out under point 7 in the checklist.\n\n### **Pt #6: Embedding Distillation Layers for Phi3-4k-mini**\n\n> Reviewer's Comment: How are the sets of layers contributing to embedding distillation from different vision experts selected for the experiments with Phi3-4k-mini (Table 1)?\n\nBecause Phi3-4k-mini has the same number of layers as Llama3-8b (32 layers), we use the same set of layers for both the LLMs based on the ablation results from Tab. 4 in the main text. We verified that Phi3-4k-mini based models showed similar probing trends as that of Llama3-8b based models while making this decision."}, {"review_id": "xDSXvzcCrY", "reviewer": "Reviewer_JZQJ", "summary": "The paper introduces VisPer-LM to enhance the visual perception of MLLMs. The model distill expert visual knowledge directly into the hidden representations of the LLM. VisPer-LM optimizes a predictive embedding loss from auxiliary vision encoders trained on segmentation, depth, and generation tasks. Extensive experiments demonstrate that VisPer-LM outperforms both single-encoder and multi-encoder baselines.", "questions": "I don't have further questions.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength:\n\n1. The task is meaningful. It's important for LLMs to use visual input more precisely.\n\n2. The proposed method uses different expert visual encoders. This new training pipeline shows good results on various benchmarks.\n\n3. The paper is well written and easy to follow.\n\n\nWeakness:\n\nThe proposed model and pipeline look promising but more experiments need to be performed, especially for spatial/depth reasoning. Please try benchmark in SpatialRGPT and SpatialBench: \nhttps://github.com/AnjieCheng/SpatialRGPT\nhttps://huggingface.co/datasets/RussRobin/SpatialBench\n\nAlso, comparing with LLaVA seems to be unfair, because LLaVA is based on RGB image encoder only, and is not specifically trained for spatial/depth understanding and reasoning. Please also compare with: SpatialVLM, SpatialRGPT and SpatialBot.\n\nIn all, the model and pipeline is good, but the paper lacks suuficient experiments. I'm more than happy to raise my ratings if the authors can provide more experiment results in the revised manuscript.", "was_revised": true, "final_justification": "The rebuttal has addressed my concerns so my final rating is 4.", "rebuttal": "We thank the reviewer for their feedback and comments.\nWe are encouraged that they find our task as \"*meaningful\"* and *\"important\"*. We are glad that they acknowledge our method's *\"good results on various benchmarks.\"* and that they find our paper easy to follow and well-writen.\n\nWe respond to their questions and concerns below:\n\n### **Pt #1: More Benchmarks**\n\n> Reviewer's Comment: The proposed model and pipeline look promising but more experiments need to be performed, especially for spatial/depth reasoning. Please try benchmark in SpatialRGPT and SpatialBench\n\nThank you for your suggestions! \n\nWe provide results on the `positional` and `counting` tasks (which are aligned with our target tasks) under SpatialBench [2]. As shown in the table below, our VisPer-LM outperforms the baseline LLaVA-1.5 model and both variants of SpatialBot (RGB and RGBD inputs). For SpatialBot, we report results as mentioned by the authors under issue #13 of the official repo due to an update of the model's checkpoint since its publication.\n\n| method                       |   positional                 |   counting            | \n|-----------------------|:----------------------------:|:----------------------------:|\n| LLaVA-1.5             |            55.9              | 86.3 |\n| SpatialBot (RGB)  |          50.0               | 87.4 |\n| SpatialBot (RGBD)  |          **61.8**              | 87.4 |\n| **VisPer-LM (ours)**  |          **61.8**               | **89.1** |\n\nWe could not evaluate our model on SpatialRGPT-Bench [1] because the benchmark is designed for grounded spatial reasoning, i.e., it requires the model to be trained with region-aware referring question-answer data which is not present in our LLaVA-1.5 based training set, making it hard for the model to understand and answer the benchmark's questions in way that could help the evaluation of its abilties. We hope the reviewer understands the special nature of this benchmark and our inability to evaluate on it without a special training recipe.\n\n[1] Cheng, An-Chieh et al., \"SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models.\" NeurIPS 2024\n\n[2] Cai, Wenxiao et al., \"SpatialBot: Precise Spatial Understanding with Vision Language Models.\" ICRA 2025\n\n\n### **Pt #2: Comparison to MLLMs with non-RGB encoders**\n\n> Reviewer's Comment: Also, comparing with LLaVA seems to be unfair, because LLaVA is based on RGB image encoder only, and is not specifically trained for spatial/depth understanding and reasoning. Please also compare with: SpatialVLM, SpatialRGPT and SpatialBot.\n\nWe respectfully disagree that comparing with LLaVA is unfair just because it only uses an RGB image as input. Most existing MLLM works use RGB encoders including Cambrian-1 [66] which proposed CV-Bench (including the spatial and depth reasoning tasks) for evaluating MLLM's visual perception abilities. We believe it is critical to develop general-purpose MLLMs that are accurate at depth and spatial reasoning with only an RGB image input and our VisPer-LM is a significant advancement in that direction.\n\nNonetheless, we thank the reviewer for their suggestion and provide comparisons to SpatialRGPT and SpatialBot on the Depth and Relation tasks under CV-Bench below. Interestingly, although our VisPerLM does not show the best performance on either task, it performs best on average. SpatialBot shows the best performance on the Depth task while SpatialRGPT shows the best performance on Relation task but their performance lags on the other task. We believe a plausible reason for this could be the large scale data used by these models, making them specialized for particular tasks despite using additional non-RGB inputs. We could not compare our approach to SpatialVLM [1] since the authors of that paper do not release their code, models or benchmark.\n\n| Method                | Training Data | Depth | Relation | Avg  |\n|-----------------------|:-------------:|:-----:|:--------:|:----:|\n| LLaVA-1.5             |     ~1.2M     | 70.8  |   74.0   | 72.4 |\n| SpatialBot            |     ~2.7M     | **77.2**  |   70.0   | 73.6 |\n| SpatialRGPT           |     >50M      | 59.1  |   **80.3**   | 69.7 |\n| **VisPer-LM (ours)**  |     ~1.2M     | 72.5  |   77.2   | **74.9** |\n\n[1] Chen, Boyuan et al., \"SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities.\" CVPR 2024\n\n[66] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms, NeurIPS 2024."}]

Author rebuttal:

Meta-review: In this paper, authors proposed VisPer-LM that infuses visual perception knowledge from expert vision encoders into the LLM's hidden representations. They formulated the objective during the pretraining stage in MLLMs as a coupled optimization of predictive visual embeddings and next token prediction. They showed that VisPer-LM outperformed the single and multi-encoder baselines.

This paper got 4 borderline accept ratings. 

The strength of this paper suggested by reviewers are:
1) proposed method is novel, clever, well-motivated and insightful. (Reviewer KZsA)
2) excellent trade-off between performance and efficiency. (Reviewer KZsA, jTWX)
3) well-executed, extensive ablations, extensive experiments. (Reviewer KZsA, jTWX, Khkk, JZQJ)
4) solves an important problem. (Reviewer Khkk, JZQJ)
5) provides cool insights (Reviewer Khkk)
6) paper is well written (Reviewer Khkk)

The weaknesses are:
1) final performance gains on the main benchmarks, while consistent, are relatively modest (Reviewer KZsA)
2) questions about its generality. (Reviewer KZsA)
3) training time for each stage is not reported. (Reviewer KZsA)
4) some additional analysis or experiments needed (Reviewer Khkk, JZQJ)

After rebuttal, Reviewer KZsA, jTWX, JZQJ thought their concerns are addressed by authors and gave borderline accept rating. Reviewer Khkk suggested that some of their concerns for example "primarily regarding the transfer of visual understanding capabilities, the significance of the reported outcomes, and the generalizability of design choices to other LLM architectures" remain partially unaddressed. but overall positive about this paper and gave borderline accept rating.

Given these AC decided to accept this paper.
Decision: Accept (poster)