Law of Vision Representation in MLLMs
Authors: Shijia Yang
Bohan Zhai
Quanzeng You
Jianbo Yuan
Hongxia Yang
Chenfeng Xu
Venue: COLM 2025 (2025)
Keywords: Multimodality Large Language Models; Computer Vision; Vision Representation
TL;DR: MLLM performance is correlated to cross-modal alignment and correspondence of its vision representation

Abstract: We introduce the "Law of Vision Representation" in multimodal large language models (MLLMs), revealing a strong correlation among cross-modal alignment, vision representation correspondence, and overall model performance. We quantify the these factors using the cross-modal Alignment and Correspondence score. Extensive experiments across fifteen distinct vision representation settings and evaluations on eight benchmarks show that the A and C scores correlate with performance following a quadratic relationship. By leveraging this relationship, we can identify and train the optimal vision representation for an MLLM, achieving a 99.7% reduction in computational cost without the need for repeated finetuning of the language model.

Published as a conference paper at COLM 2025
Law of Vision Representation in MLLMs
Shijia Yang'
Bohan Zhai*
Quanzeng You
Jianbo Yuan
Hongxia Yang
Chenfeng Xu
STANFORD UNIVERSITY 2UC BERKELEY THE HONG KONG POLYTECHNIC UNIVERSITY #NDDEPENDENT RESEARCHER
Abstract
We introduce the "Law of Vision Representation" in multimodal large language models (MLLMs), revealing a strong correlation among cross- modal alignment, vision representation correspondence, and overall model performance. We quantify these factors using the cross-modal Aligmment and Correspondence scores. Extensive experiments across fifteen distinct vision representation settings and evaluations on eight benchmarks show that A and C scores correlate with performance following a quadratic relationship. By leveraging this relationship, we can identify and train the optimal vision representation for an MLLM, achieving a 99.7% reduction in computational cost without the need for repeated finetuning of the language model. The code available at https: //github. com/bronyayang/ Law of Vision Representation in MLLMs
Introduction
Current multimodal large language models (MLLMs) (Chen et al., 2024a; Liu et al., 2024e;d) have achieved remarkable advancements by integrating pretrained vision encoders with powerful language models (Touvron et al., 2023; Zheng et al., 2023). Among the core components of general MLLM, vision representation plays critical role. Many researchers have utilized CLIP (Radford et al., 2021) or SigL.PP (Zhai et al., 2023b; Tschannen et al., 2025) as the primary image feature encoder, but their limitations are becoming increasingly noticeable (Tong et al., 2024b; Geng et al., 2023; Yao et al., 2021). As a result, alternative vision representations (Tang et al., 2025) and the combination of multiple vision encoders are being actively explored (Tong et al., 2024a; Lin et al., 2023).
Despite this growing attention, selection of vision representation has largely been empirical. Researchers typically test a set of vision representations on a specific MLLM and choose the one that yields the highest performance constrained by the number of representations tested and does not address the underlying factors that drive performance differences. As result, the optimal vision representatio tion for a specific MLLM is often determined by empirical performance rather than a deep under- standing of the factors that contribute to success. The question of what fundamentally makes feature representation achieve the highest performance remains largely unanswered.
benchmark tasks. This approach, however, is
To address this gap in understanding what makes vision representation optimal for MLLMs, we propose the Law of Vision Representation in MLLMs. It aims to explain the key factors of vision representation that impact MILLM benchmark performance. Our findings reveal that cross cross-modal Aligument (A) and Correspondence (C) of the vision repre- sentation are strongly correlated with model performance.; specifically, higher A and C lead to improved performance. To quantify this relationship, we define A and C scores that measures cross- modal alignment and correspondence vision representation. The A and C
* *Corresponding author: xuchenfeng@berkkleyy.du

---

Published as a conference paper at COLM 2025
Law of Vision Representation in MLLMS
models and predicts MLLM performance with different vision representations through cross- pross-model Aligmment
Correspondence
aleat]
Law works with dienetent:
MI LM Performance (MMBench
ML MLLM Perfonmancc (MM MMMBench
Soddd 3iz SgLiP ans
SULLIP Lage
Sigli
70
[CLIP DINO
Teptire Connonaaooo PoR
60
88
8
SOXL
2015 type
40
20XL
50
gn2.1
30
gm
30
241
-35 Cross-nod Alignment
-2.0
Crass inedal Alignment
-2.
-2
* Giound Truhh Prediction
-2
-3.0
-3
Correspundanee (PCK@0. 10)
10
15
20 Corresonndane (PCK@0011)
15
20
25
Figure Visualization of the Law Vision Representation in MLLMs.
scores well model performance exhibit quadratic relationship, with coefficient of determination of 94.06%.
Furthermore, the Law of Vision Representation guides the selection of an optimal vision representation for MLLMs. Originally, this process was extremely costly because even subtle changes in vision encoding- such as switching encoder types, altering image resolution, or testing feature combinations- example, using a top data-efficient MLLM pipeline with a 7B language model requires 3,840 NVIDIA A100 GPÜ hours to test the 10 encoders, amounting to a cost of approximately $20,000 Testing additional encoders leads to linear increase in cost. Moreover, the recent trend of feature combination, which often results in better performance, necessitates combinatorial testing of vision encoders. Testing all possible combinations of 10 encoders results in 1023 combinations, exponentially increasing the cost and energy consumption. This process consumes approximately 100,000 kilowatt-hours? enough to drive an electric vehicle around the Earth 13 times.
finetuning the language model (Lin et al., 2024). For
Thus, we are the first to propose policy, AC policy, that selects the optimal vision rep- resentation using AC scores within the desired search space. Unlike traditional meth- ods that rely on benchmarking performance, the AC policy enables the expansion of the search space-allowing for an increased number of vision representations ereh-without incurring additional costs. We demonstrate that this approach enhances both accuracy and efficiency compared to randomly searching for the optimal representation. The policy successfully identifies the optimal configuration among the top three choices in 96.6% of cases, with only three language model finetuning across a 15-setting search space.
be consid-
2 Related Works
2.1 Vision Representations for MLLMs
Recent studies have explored various vision representations in MLLMs (Beyer et al., 2024; Ge et al., 2024; Liu et al., 2024e; Wang et al., 2024b; Sun et al., 2023; Luo et al., 2024). Interestingly, some findings indicate that relying solely on encoders outside of the CLIP family (Cherti
https://replicte..com/iiici datasheet.pdr
2

---

Published as a conference paper at COLM 2025
et al., 2023; Zhai et al., 2023b; Li et al., 2022), such as DINOv2 (Oquab et al., 2023) and Stable Diffusion (Rombach et al., 2021), often leads to lower performance (Karamcheti et al., 2024; Tong et al., 2024a). However, combining features from these encoders with CLIP features, such as concatenating image embeddings in the channel dimension, significantly enhances performance beyond using CLIP alone (Tong et al., 2024a;b; Liu et al., 2024c; Kar et al., 2024). Researchers intuitively suggest that these additional encoders provide superior detail-oriented capabilities, but no studies have thoroughly analyzed the underlying causes of the performance change (Wei et al., 2023; Lu et al., 2024a). This suggests that the attributes of an optimal vision representation remain not fully understood.
2.2 Cross-modal Alignment
Cross-modal alignment refers to the alignment between image and text feature spaces (Duan et al., 2022). This concept emerged with the introduction of text-image contrastive learn- ing (Radford et al., 2021; Jia et al., 2021). Although current MLLMs utilize contrastively pretrained image encoders, the challenge of achieving effective alignment persists (Ye et al., 2024; Zhai et al., 2023a; Woo et al., 2024). Despite efforts to critique the limitations of CLIP family representations and explore alternative vision representations, many approaches continue to rely on contrastively pretrained encoders or adding contrastive loss without fully eliminating them (Zhang et al., 2024b; Lu et al., 2024a; Tong et al., ;024a;;; Liu et al., 2024b). In our work, we point out that alignment in vision representation is essential for improved model performance and is crucial for data efficiencyy Without pre-aligned vision representations, extensive data pretraining is required to achieve cross-modal alignment within the language model (Ge et al., 2024; Chen et al., 2024b; Li et al., 2024c).
2.3 Visual Correspondence
Visual correspondence is a fundamental component in computer vision, where accurate correspondences can lead to significant performance improvements in tasks, such image detection (Xu et al., 2024; Nguyen & Meunier, 2019), visual creation (Tang et al., 2023; Zhang et al., 2024c), and MLLMs (Liu et al., 2024a), etc. Correspondences are typically categorized Semantic correspondences (Zhang et al.,
into semantic- and 2024c; Min et al., 2019) involve matching points that represent the same semantic concept not necessarily representing the same instance. Geometric correspondences (Sarlin et al., 2020; Lindenberger et al., 2023), on the other hand, require matching the exact same point across images, which is often crucial for low-level vision tasks, such as pose estimation (Sarlin et al., 2020; Lindenberger et al., 2023; Zhang& & Vela, 2015), and SLAM tasks, etc.
Several studies have pointed out that the CLIP family's ss vision representation "lacks visual details" (Lu et al., 2024a; Tong et al., 2024b; Ye al., 2024). We explain this observation through the concept of correspondence. Current MLLMs convert images into embeddings, with each embedding representing patch of the image. Image features with high cor- respondence increase the similarity within internal image patches on similar semantics, thereby enabling the retrieval of more detailed information.
3 Law of Vision Representation in MILMs
We introduce the Law of Vision Representation in Multimodal Large Language Models (MLLMs). It states that the performance of MLLM, denoted as Z, can be estimated by two factors: cross-modal alignment (A) and correspondence (C) of the vision representation, as- suming vision representation is the sole independent variable while other components (e.g., language model and alignment module) remain fixed. This relationship can be expressed
Z x f(A,C)
(1)
where f is quadratic function of A and C.
3

---

Published as a conference paper at COLM 2025
3.1 Assumptions
Following NVLM (Dai et al., 2024), we categorize MLLMs into the following types: (1) Decoder-only MLLMs (Tong al., 202Aa; Liu al., 2024e; Li al., 2024a; Liu et al., 2024f; Dai et al., 2024; Lu et al., 2024b; Zhang et al., 2024a; Wang et al., 2024a): These MLLMs consist of vision encoder(s) and an alignment module, such as multilayer perceptron (MLP), which maps the vision representation into vision tokens. These tokens are designed to have a similar distribution as language tokens and are directly input into a language model in the same manner as language tokens. (2) Cross-attention-bass
MLLMs (Dai et al., 2024; Bai et al., 2023; Alayrac et al., 2022; Laurençon et al., 2024; Chen al., 2024c): These MLLMs include vision encoder(s) and additional module, often serving as downsampling component, such as a perceiver resampler. The vision tokens generated are integrated into the language model through cross-attention mechanisms.
The Law of Vision Representation specifically focuses on decoder-only MLLM architecture due to their widespread adoption and their simplicity, which facilitates controlling variables in training recipes and enables clear mathematical modeling.
We further assume vision representation is the only independent variable, while the alignment module and LLM architecture remain fixed. In the case of a unfrozen vision encoder, we cannot guarantee that the vision encoder does not take the func- tion of the alignment module. This causes the architecture and role of the alignment module to change alongside the encoder, making the experiment uncontrolled and the models no longer comparable.
3.2 Theoretical Justification
In this section, we theoretically analyze how an increase A and C leads to improved model performance. When vision representation demonstrates high cross-modal alignment and accurate correspondence, the MLLM exhibits the following desired properties:
When training a MLLM, if the vision representation is closely pre-aligned with the language distribution, the pretrained language model requires less computational effort bridge the gap between different modalities during finetwning. In Section A.1, we provide theo- retical justification that finetuning on well-aligned multimodal data is about equivalent to finefuning on text-only data, eliminating additional effort beyond language finetuning. This efficiency can lead to improved performance, especially scenarios where the available training data for finetuning is limited.
If the vision representation ensures accurate correspondence, the attention within the image embeddings is precise. Consequently, the MLLM develops a refined focus on visual content, capturing even details that cannot be derived solely from text-to-image attention, leading to a more detailed interpretation of the image. We provide theoretical justification in Section A.2.
3.3 Empirical Justification
In this section, empirically show that A and C scores are strongly correlated to model performance. To quantify the correlation between A and C as well as model performance, we first propose methods to measure cross-modal alignment and correspondence within the vision representation:
To quantify cross-modal alignment, we define a metric A SCORE, that measures how well the vision representation is mapped into the language model's space. With both the vision encoder and the LLM frozen, if visual features aligns with the LLM's language space effectively, the LLM's prediction error will be minimized. In other words, a well-aligned vision embedding leads to higher likelihood for the correct caption tokens.
Formally, for an input image and its associated caption y = (1/,.22...,, ,r), where T is the sequence length, let f(T) denote the projected visual representation (i.e., the
4

---

Published as a conference paper at COLM 2025
output of the vision encoder projector). The conditional probability of generating token y/ is given by:
Ply ((),)=))
with y-t representing the tokens preceding yt. The alignment score is then defined as the average log likelihood over all tokens:
1 A SCORE(1,y) T Σ log P Ply (().)=))
A higher A SCORE indicates that the visual features are more effectively aligned with the language model's representation, reflected by the increased log- likelihood of the correct caption.
To quantify correspondence, we measure how accurately key points in one image can be matched to their semantically corresponding locations in another image. Given a pair of image with annotated, semantically matching key points, we first extract features from each image pair. Let F and F denote the feature maps of the source and target images, respectively.
Using the feature vectors at the labeled key point positions in Ft, we predict the corresponding key points in F' by selecting the location with the maximum simi- larity, yielding set of predicted key points {r ..... .. Pmm for key points. The ground-truth key points for the image pair are denoted by "pi"r,..., )T The correspondence score is then defined as the Pencentage of Correct Keypoints (PCK), computed as follows:
m
C SCORE
(2)
<T
where T is threshold proportional to the bounding box size of the object in the image, and (-) is the indicator function that returns when the condition is satisfied and 0 otherwise.
A higher C ScORE indicates more accurate key point correspondence, reflecting better vision feature matching.
To capture the overall performance, integrate the A SCORE and the SCORE into single metric called the AC SCORE. We do this by fitting a second-degree polynomial to benchmark performance, which allows us to model potential nonlinear interactions between and C. Formally, the AC Score is defined as:
2 2-4
AC SCORE
Σ
Σ
a
A CBB
(3)
8=0
where WaB are trainable parameters that are optimized to best fit the benchmark perfor- mance.
This formulation allows the AC Score capture both the individual contributions of alignment and correspondence, as well as their interaction effects.

---

Published as a conference paper at COLM 2025
Results. We fit a simple regression model using 15 vision representations across
Coefficient of Deternination (%6
91.48%
94.06%
vision-based MLLM benchmarks. As shown Figure 2, the average coefficient
80
of determination (R²) obtained is 94.06% when using the AC score of the vision
60
56.56%
representations. For comparison, we also
36.90%
fit models using 15 random scores, the A
score alone, and the C score alone, all with quadratic functions. The random scores and single-factor models show lower correla- tions with performance. This result high- lights the strong correlation between the AC score and MLLM performance, validating the Law of Vision Representation. Refer to Sec- tion 5.4 for details.
20
Random
A Score
C Score
AC Score
Figure 2: R² values for regression models fit- ted on various scores.
AC Policy
Feature extraction from any vision encoder
Score features with AC
Image
CLIP+DINOv2
Features
Cross-modal Alignment Correspondence
CLIP@2224 DINOv2
SD CLIP@336
SigLIP
with AC score
331
Fit + predict performance with AC
Train only optimal model
Vision Model
Performance
Predict
Performance
Fit
Connector
L.LM
Figure Overall framework AC policy.
Problem Formulation. The MLLM architecture assumed in this framework consists of frozen vision encoder, followed by a trainable connector (alignment module) and the pretrained language model. To determine the optimal out of k vision representations for the MLLM, we originally needs finetune LLM k times, making the scaling of k difficult. Therefore, we propose AC policy, illustrated in Figure 3, to efficiently estimate the optimal vision representation from a search space consisting of vision representations. We finetune only k' LLMs to obtain downstream performance, allowing k to scale without significant cost, where K' < k. The value of should be determined based on the computational budget allocated for vision representation selection.
Policy Fitting. Let X E R be the matrix containing AC scores of vision representation
in the search space. We subsample data points from X, denoted as X₃ E RR to serve as the input to the regression model:
X-w +
(4)
Here, w E 6 is the vector of model parameters, e E IRK is the vector of error terms, and y R represents the downstream performance desired benchmark.

---

Published as a conference paper at COLM 2025
91.70%
91.30%
90.90%
Vision Representation
Resolution
80
Aveeraee Pledicted Optinal
$3..50%
AC Palie Imprevenne ceneenncca
Single rision emnoder: fred-forusard models
OpenAT CLIP VIT-1./14
224
CLIP ViI-L/14 (Radford 2021)
60
-61.02%
OpenCLIP VIT-L/14 (Cherti et al, 2023)
224
Receligg
DINOv2 VIT-L/14 (Oquab
2023)
224
SigL VTT-B/16 (Zhai et al,, 20236)
224
SigLIP ViT-L/16 (Zhai et al,, 2023b)
256
40
SigLIP2 VIT-L/16 (Ischannen et al., 2025)
256
Single Dision encoder: diffusion models
20-
Randonly Tesling VR.26. 7%
SD.1.5 (Rombach et al., 2022)
768
SD2.11 (Rombach et al., 2022)
768
Image Variations
SD XL (Podell et al., 2023)
512
OKVQ4
TeuVQA
MMBeaahh
SERD Bonch
VirWar
Science
DIT (Peebles Xie, 2023)
512
SD (Esser et al,, 2024)
312
Multiple vision encoslers: feature combination
CLIP-DINOv2 WIT-L/14
224
CLIP-DINOv2 VIT-I./14
336
Figure 4: Given a limited budget of 4 finetun- ings, AC policy achieves 87.72% Recall@3 in predicting the optimal vision representation.
Table 1: Vision representations explored.
Sampling Strategy. The selection of k' can impacts the function fit and, consequently, the accuracy of predictions. To avoid sampling points that are too close in terms of their A and scores, we employ a sampling strategy based on the coordinates.
The normalized A and C score pairs of vision representation can be plotted on 2D graph as coordinates (A, C), ensure diverse sampling, we divide the graph into regions. For each iteration j in which the total sampled points do not yet fulfill K', we divide the graph into 4 equal regions. We then remove empty regions and those that contain previously sampled points. The next data point is randomly selected from a remaining region.
Results. In Figure 4, we demonstrate that the AC policy consistently predicts the optimal vision representation using minimal resources within a finite search space of 15 settings. Our aim is finetune only a small subset of this space while ensuring that the optimal vision representation is among the top-3 predictions (Recall@3). With a computational budget equivalent 4 full finetuning runs, random subset selection achieves only 26.7% Recall@3. In contrast, the AC policy achieves 87.72 87.72% Recall@3 (averaged over 6 benchmarks) while still requiring just 4 full training runs. For further details, see Section 5.5.
Empirical Result Details
5.1 Experiment Settings
For our MLLM pipeline, we deploy LLaMA-based LLM, specifically Vicuna-7B 1.5 (Zheng et al., 2023) as well as Qwen-14B 2.5 Instruct (Qwen al., 2025), and utilize a widely used layer GeLU-MLP connector as the projector. For vision representation, we explore a variety of encoder types and sizes, input resolutions, training paradigms, and feature combinations, as detailed in Table 1.
Our training process consists of two stages. Stage 1, we perform alignment using the LLa VA 1.5 dataset with 558K samples (Liu et al., 2024e), training only the projector. In Stage 2, we train both the connector and the language model on an expanded LLaVA 1.5 dataset containing 665K samples.
The MLLM benchmarks used in this paper include four vision-based benchmarks (MM- Bench (Liu et al., 2023), MME (Fu et al., 2023), OKVQA (Marino et 2019), (Li et al., 2024b)) and four QCR-based benchmarks (MMMU (Yue et al., 2024), TextVQA (Singh et al., 2019), VizWiz (Gurari et al., 2018), ScienceQA (Lu et al., 2022)).
7

---

Published as a conference paper at COLM 2025
5.2 AC Score
To compute the cross-modal alignment score, we perform Stage 1 training on all vision representations to obtain the projected vision representations. This stage requires sigmifi- cantly less computation than Stage 2, involving only 0.298% of the trainable parameters. The image-caption pairs are taken from the LLaVA-558K dataset, and the alignment score is computed by averaging the results across 100 randomly sampled images.
For the correspondence score, we follow common practices using the SPair-71k dataset (Min et al., 2019). Note that all benchmarks share the same A and C scores, while the quadratic parameters the function fitting adapt to capture variations across tasks.
5.3 Feature Extraction
Both MLLM training and score computation involve image feature extraction. Below, we introduce the approach for obtaining two types of vision representations.
Given an image R process it either in its raw form for U-Net models or in a patchified form for transformer models. For transformers, we extract the last hidden state F E R where | is the sequence length and c is the hidden dimension. the case of the U-Net model, we take the interme- diate activation F R fxxwcc after the first upsampling block. Note that the features from these two types of models are interchangeable between sequence and grid formats through reshaping and flattening. For consistency, the following sections assume that all features have been pre-converted into the same format.
Vision Representation from Feed-forward Models.
Vision Representation from Diffusion Models. Diffusion model is primarily used for generating images via multi-step denoising, yet recent trend is to use diffusion model as the vision representation model (Xu et al., 2024; 2023; Zhang et al., 2024c; Tong et al., 2024a). Specifically, for diffusion models, given an image E RHxWK3, we first add noise to the VAF-encoded representation of I:
X Va. VAE(I) (V1 a+) a+)
where (0, and a; is determined by the noise schedule. Note that utilize the little-noise strategy by setting the f 1. In that case, the diffusion model only denoises the noise-latents once and treat the one-step denoising latents the vision representation features.
5.4 Additional Results on the Law of Vision Representation
In Section 3, we demonstrate the strong correlation between the AC score and MLLM performance by analyzing the coefficient of determination (R2) ob- tained from fitting quadratic regression model. In this section, we further ablate the experiments by adding baselines, fitting model performance with random scores, scores, and C scores separately. Additionally, we explored the relationship between the score and score by fitting a linear regres- sion model. We provide more insights about the AC relationship in appendix A.3. Besides, we avoid higher-degree transformations to prevent overfitting, which could obscure the true relationship between A and C scores.
Fitting Data
R² (Vision)
R² (OCR)
No transformation on fitting data Random
4.03%
1.75%
A Score
80.53%
58.00%
C Score
39.02%
14.57%
AC Score
80.55%
62.06%
Polymomial transformation on fitting data
Random
36.90%
31.26%
A Score C Score AC Score
91.48%
79.67%
56.56%
30.11%
94.06%
83.85%
Table 2: Averaged R² results of AC and other baselines fitting on MILM benchmarks.
As shown in Table 2, the results indicate that using the AC score consistently outperforms all other set-
8

---

Published as a conference paper at COLM 2025
tings in terms of R² values. While this observation
holds regardless of the degree of fitted function, using second-degree polynomial on A and C scores yields the highest correlation with model performance. This suggests an inherent trade-off between and C scores: vision representations with high cross-modal alignment often exhibit lower correspondence, and vice versa.
Interestingly, we observe a lower correlation between OCR-based benchmark performance and C scores, which leads to reduced correlation between the AC score and DCR-based benchmark performance. In Section 6, we discuss how the use of the SPair-71k correspon- dence dataset across all benchmarks fails to adequately capture correspondence in images containing text.
Here, we additionally provide MLLM with Qwen 2.5 14B Instruct as LLM backbone fitting results, as shown in Table Specifically, we use the exact same settings (data, vision representations, hyperparam- eters), changing only the LLM backbone to Qwen 2.5 14B. We again evaluate the resulting MLLMs on 8 benchmarks, recompute the A and C scores, report- ing the fitting R². The averaged fitting R² are above 90% for vision based benchmarks and above 80% for OCR based benchmarks, comparable to Vicuna 1.5 7B, showing that our conclusion applies to different types and sizes of LLM backbones.
LLM Backbone
R (Vision)
R² (OCR)
Vicuna 1.5 7B
94.06%
83.85%
Qwen 2.5 14B
91.50%
82.27%
Table 3: Averaged law fitting R² com- parison for different types and sizes of L.M backbones.
5.5
Additional Results on the AC Policy
SFF.D-Bench
MMBench
OKVQA
0.75
0.75
0.5
0.59
0.59
0125
AC Predicted
6008
10:11
14
0.50
10 13
0.00
9 10
13 15
Prediction Recal
TextVQA
VizWiz
ScienceQA
1.00
1.00
1.00
Optimal VR
Random Predicted
0.50
0.55
0.50
0.25
0:35
0.00
Number of Full Training
Figure 5: Number of full training (LLM finetuning) cycles required to include the optimal vision representation within the top-3 predictions (Recall@3).
In Section 4, we demonstrate that fitting the AC score consistently predicts the optimal vision representation with minimal resources, given finite search space-in this case, 15 settings. In this section, we provide detailed visualization for Figure 4.
When performing ablation experiments on vision encoders, it's common to randomly select subset to train on. However, as shown in Figure 5, with 1000 runs of simulated ablation experiments, we found that to include the optimal vision representation 85.6% of the time, at least 13 out of the 15 settings need to be trained. This suggests that running small subset of vision representations is unreliable, especially as the search space expands, making increasingly unlikely to identify the true optimal representation by training only subset.
In contrast, the AC policy requires only 4 full training runs on average to reach 87.7 87.72% Recall@3. For the most successful prediction benchmark, OKVQA, the policy successfully identifies the optimal configuration among the top three choices in 91.7% of cases, with only four language model finetuning runs across 15-setting search space. This result shows that

---

Published as a conference paper at COLM 2025
AC policy significantly reduces the effort and cost of exploring vision representations for MLLMs.
Obtaining the AC policy results in significant computational cost reduction. After fitting the AC function, testing a new vision encoder only requires MLLM Stage training (i.e. training small projector), while traditional full MLLM post-training requires training both the projector and LLM. Concretely, we quantify computational cost using the number of trainable parameters, which is commonly correlated with both memory and compute. A decoder-only MLLM's Stage trains a small two-layer MLP, while Stage involves fine- tuning large language model with 7B parameters. This leads parameter ratio of about 0,003, yielding 99.7% reduction.
6 Limitation
Correspondence of Nature Image
Correspondence of Image with Text
CLIP @
DINOv2 &
CLIP d
DINOv2
Figure Visualization of correspondence on natural images and images containing text for CLIP and DINOv2.
We find that OCR-based benchmarks correlate less with the AC score than vision-based ones, making MME and MMMU outliers. For example, Sig L55 outperforms CLIP-Large@336 in both alignment (-1.81 vs. -1.97) and correspondence (16.75 vs. 15.66) but underperforms on MME due to OCR-heavy categories.
This discrepancy arises because vision representations exhibit different correspondence accuracy on different domain of images, as shown in Figure 6. The SPair-71k dataset (for the C score) focuses tasks not captured in SPair-71k. Likewise, our alignment score, calculated using LLa VA 558K (a subset of LAION, CC, SBU), lacks sufficient OCR, numeric, and symbolic data. Consequently, both the A and C scores underrepresent performance on OCR-related tasks.
natural images (e.g., cats, trains), whereas CLIP excels at text-based
To our knowledge, an OCR-specific correspondence dataset does not currently exist, and systematically designed OCR short caption datasets are scarce. We intend to pursue further investigation in this direction and encourage other researchers do the same, as advance- ments in this area would be valuable for the broader MLLM communniyypprtiiullla for tasks requiring the understanding of tables and charts, fundamental capability.
Conclusion
In this work, we present the Law of Vision Representation in MLLMs, uncovering a strong and quantifiable relationship between cross-modal alignment, feature correspondence, and overall model performance. Our proposed A and C scores capture this relationship with high fidelity, enabling principled way to reason about vision representation quality. Through extensive evaluation, we demonstrate that model performance exhibits clear quadratic correlation with AC scores, providing not only an interpretable diagnostic but also predictive signal for selecting optimal visual features.
By leveraging this insight, we introduce the AC policy, cost-efficient strategy to identify high-performing vision representations without repeatedly finetuning the language model. This yields up to 99.7% reduction in compute cost, transforming what was previously an expensive and combinatorially large search problem into tractable one. Our findings offer a new perspective on vasion-language alignment and open the door to more scalable MLLM development.
10

---

Published as a conference paper at COLM 2025
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-223333 23736, 2022.
35:
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-languagee model with versatile abilities. arXiv preprint arXivv2308..12 122666,, 2023.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vim for transfer. arXiv preprint arXiv:2407.07726, 2024
Kaibing Chen, Dong Shen, Hanwen Zhong, Huasong Zhong, Kui Xia, Di Xu, Wei Yuan, Yifei Hu, Bin Wen, Tianke Zhang, et al. Evim: An efficient vision-language model for visual understanding. arviv preprint ar arXio:240..14177 2024a.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. ar Xiv preprint arXio:2404.1681,, 2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024c
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-imagg learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nvlm: Open frontier- class multimodal llms. Xiv preprint ar Xrx::22499.1144 2024.
Jiali Duan, Liqun Chen, Son Tran, Jinyu Yang, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Multi-modal alignment using representation codebook. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 15651-15660, 2022.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arv preprint ar (ttt::2306.13994 2023.
Chunjiang Ge, Sijie Cheng, Ziming Wang, Jiale Yuan, Yuan Gao, Jun Song, Shiji Song, Gao Huang, and Bo Zheng. Convllava: Hierarchical backbones visual encoder for large multimodal models. Xiv preprint drXiv:2405.15738, 2024.
Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, and Yongfeng Zhang. Hiclip: Con- trastive language-image pretraining with hierarchy-aware attention. arXiv preprint ar arXiv=2303.02995, 2023.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEÉE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
11

---

Published as a conference paper at COLM 2025
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and langual representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Oguzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. Brave: Broadening the visual encoding of vision-language models. aixivv preprint ar arXi::244.007204 2024.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-condittoned language models. ar Xrv preprint axXiv:2402.07866, 2024.
Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self- attention. In International Conference on Machine Learning, pp. 5562-5571. PMLR, 2021.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. Xiv preprint ar Xi0:24408.033,6 2024a.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13399,13308, 2024b.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888-12900. PMLR, 2022.
Yifan Li, Yikai Wang, Yanwei Fu, Dongyu Ru, Zheng Zhang, and Tong He. Uni- fied lexical representation for interpretable visual-language alignment. arv preprint ar #XXv=2407.17827 2024c.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26689-26699, 2024.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXio preprint ar arXio:2311.07575 2023.
Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Pollefeys. Lightglue: Local feature matching at light speed. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 17627-17638, 2023.
Benlin Liu, Yuhao Dong, Yiqin Wang, Yongming Rao, Yansong Tang, Wei-Chiu Ma, and Ranjay Krishna. Coarse correspondence elicit 3d spacetime understanding in multimodal language model. arXiv preprint ar fto:2400.00754, 2024a.
Chenglong Liu, Haoran Wei, Jinyue Chen, Lingyu Kong, Zheng Ge, Zining Zhu, Liang Zhao, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Focus anywhere for fine-grained multi-page document understanding. arXiv preprint ar Xiv:2440 142295,, 2024b.
12

---

Published as a conference paper at COLM 2025
Fanfan Liu, Feng Yan, Liming Zheng, Chengjian Feng, Yiyang Huang, and Lin Ma. Robouni- view: Visual-language model with unified view representation for robotic manipulaiton.
arXio preprint ar Xivv2406.189777, 2024c.
Haogeng Liu, Quanzeng You, Xiaotian Han, Yiqi Wang, Bohan Zhai, Yongfei Liu, Yunzhe Tao, Huaibo Huang, Ran He, and Hongxia Yang. Infimm-hd: A leap forward in high- resolution multimodal understanding. ar Xto preprint arXio:2403.01487, 2024d.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, oct, and world knowledge, 2024e.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024f.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi multi-modal model an all-around player? arXiv preprint ar Xiv:23307 0.200.0 2023.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, lingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real-world lision-language understanding, 2024a.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision- language understanding. arviv preprint XrXo:2243.00555, 2024b.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song- Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting- En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, et al. Deem: Diffusion models serve as the eyes of large language models for image perception. Xrv preprint Xio:2405.15232, 2024.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cof conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair-7lk: A large-scale benchmark for semantic correspondence. arxiv prepreint. arXiv preprint wxXiv:1908.10544 6:12-14, 2019.
Trong-Nguyen Nguyen and Jean Meunier. Anomaly detection in video sequence with appearance-motion correspondence. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1273-1283, 2019.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. ar preprint arXi::2304 000999, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high- resolution image synthesis. arXiv preprint arXiv:2307.01995 2023.
Qwen, ;, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men,
13

---

Published as a conference paper at COLM 2025
Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://arxiv, http.///.ggabb//11///// 15115.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 7748-8763 PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolttion image synthesis with latent diffusion models, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Su- perglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1938-4947, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp- 8317-8326, 2019.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In The Tweeffh International Conference on Learning Representations, 2023.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. In Thirth-seventh Conference Neural In- formation Processing Systems, 2023. URL
Zineng Tang, Long Lian, Seun Eisape, XuDong Wang, Roei Herzig, Adam Yala, Alane Suhr, Trevor Darrell, and David M. Chan. Tulip: Towards unified language-image pretraining, 2025. URL https: //wxxva./vvv///a////////aa/aiivl org/abs/2503. 15485.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal lims. arXiv preprint arXiv:2406.16866, 2024a.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568-9578, 2024b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09286 2023.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Al-
abdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. Xiv preprint ar XrX::2250..14786 2025.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. ar preprint ar arXio:2409.121191, 2024a.
Wenxuan Wang, Quan Sun, Fan Zhang, Yepeng Tang, Jing Liu, and Xinlong Wang. Diffusion feedback helps clip see better. Xiv preprint arXit:2407.20177 2024b.
14

---

Published as a conference paper at COLM 2025
Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language models. arXio preprint ar xrXi::2312.06109, 2023.
Sangmin Woo, Donguk Kim, Jachyuk Jang, Yubin Choi, and Changick Kim. Don't miss the forest for the trees: Attentional vision calibration for large vision language models. arXiv preprint ar Xit::405.17820, 2024.
Chenfeng Xu, Huan Ling, Sanja Fidler, and Or Litany. 3difftection: 3d object detection with geometry-awars diffusion features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10617-10627, 2024.
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2955-2966, 2023.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-gran teneggained interactive language-image pre-training. Xiv preprint ar axXio:2111.07783, 2021.
Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, et al. X-vila: Cross-modality aligmment for large language model. Xiv preprint ar 1rXio:2405.19335 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the [EEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, and Manling Li. Halle- switch: Controlling object hallucination in large vision language models. arXiv e-prints, pp. arXiv-2310, 2023a.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975-11986, 2023b.
Guangcong Zhang and Patricio A Vela. Good features to track for visual slam. In Proceedings of the IEEF conference on computer vision and pattern recognition, pp- 1373-1382, 2015.
Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mml. 5: Methods, analysis insights from multimodal Ilm fine-tuning. ar preprint ar Xiw::ii49..055555 2024a.
Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An im- proved baseline for referring and grounding with large language models. aiv preprint arXiv:2404.07973, 2024b.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Eric Chen, Varun Jampani, Deqing Sun, and Ming- Yang. Telling left from right: Identifying geometry-aware semantic correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024c.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
15

---

Published as a conference paper at COLM 2025
A
Appendix
A.1 Theoretical Justification of Vision Representation with High Cross-modal Alignment
In Section 3.2, we state that when training an MLLM, if the vision representation is closely pre-aligned with the language distribution, then the pretrained language model requires less computational effort to bridge the gap between different modalities during finetuning. In this section, we show that using well-aligned vision representation, finetuning multi- modal data is about equivalent to finetuning on text-only data, eliminating additional effort beyond language finetuning.
Assume the vision embedding distribution Dimage and text embedding distribution Diext are well-aligned in the MLLM. For a shared concept e, the image embedding after the alignment Emmage
module and its corresponding text embedding, E. D Dimage and Efext Dtext are close
in distance, meaning:
image Etext S e
(6)
where e is a small constant. Given this condition, we can show that the output of the MLLM
with multimodal embeddings [E] [E!lmage E1, E2, En] is close to the output with text-only embeddings [E E,, .,,, En
Since our language model f is well-trained and pre-normed, the input space to each trans- former layer bounded and compact, meaning that the values of the input are bounded by small constant. This implies that the continuously differentiable function f is Lipschitz (Kim et al., 2021). This property ensures small changes in the input of the language model of the MLLM result in small, controlled changes in the output:
((E imagg Er, Ez...., Enl) f( Eg, Ez,... Enl)
< L E image EE, Ez,..., En]
pllext Ei, Eg, En]
< Le.
where L is the Lipschitz constant. This closeness in output distance implies that even with multimodal data, the pretrained language model mimics the training dynamics closely resemble language-only finetuning.
A.2 Theoretical Justification of Vision Representation with Accurate Correspondence
In Section 3.2, we state that if the vision representation ensures accurate correspondence, the attention within the image embedding is precise. In this section, we show that vision representation with accurate correspondence can help vision information retrieval the attention mechanism. Therefore, more visual details are considered even if not attended by the text token.
Consider an input [ image
,F, pimage E2,..., En] to the transformer, where the image embed- dings Eimage and E gimage are derived from different patch of high correspondence vision
representation. By definition, the dot product E, E. immage is large if the two corresponding original image patches share related information.
iimage
Suppose a text token E2 attends to E pimage We show that it is also able to retrieve E vice versa. This can be demonstrated as follows:
pimage and
(E2WR)
(E) (Eimaae W
score(Fz, F
rimage
(8)
Vdz
16

---

Published as a conference paper at COLM 2025
If score(Ez, Eo Eo ) is high, and (Eo (Em W
(E, (Eimak W is also large (assuming W does not distort the vectors drastically), then by transitivity, score F F is also likely to be high. This transitivity ensures that attention is effectively spread across related visual information, enhancing the model's ability to interpret visual content in greater detail.
A.3 Analysis of Quadratic Relationship between A and
We first provide intuitive explanation of why we choose to capture a quadratic relation- ship between cross-modal alignment and correspondence. Then, we analysis the coefficent of the fitted function to reveal more in-depth relationship between these two factors.
Here's our insight:
Good semantic correspondence means the vision part of MLLM can really tell what things are in picture, even if they look bit different. Like knowing all dogs are dogs, no matter their breed. But this alone isn't enough for the MLLM to "talk" about them. Those visual details need to be understood by the LLM.
Good language alignment means the visual information is perfectly understood by the language model. But if the initial visual information isn't 't very good at telling different things apart (poor semantic correspondence), then aligning it perfectly won't help much when the MLLM sees something new or slightly different.
So, for MLLMs to work best, both semantic correspondence and language alignment need to work together. More importantly, they interplay with each other more than just adding up:
the visual features are already well-aligned with language (like in models such as CLIP), it actually helps the MLLM understand the semantic details better. That's because if you can map "dog" to the word "dog" consistently, it reinforces the concept of "dog" in the visual world.
But why is this interplay represented in a quadratic form (like a hill)? Think of it in this way:
If language alignment is very weak, even if you make semantic correspondence the MLLM's performance won't ton'jjjnn jump much. It's like trying to have a good conversation when you don't speak the same language very well. Take model like DINOv2. It's great at seeing things, but it might not be as good at linking those visual details to language as, say, CLIP. In this case, making the visual part even better won't help MLLM performance much because the language alignment is the bottleneck.
little better,
As language alignment improves, the MLLM can then really benefit from better semantic correspondence. It's like they're The working together to reach that "sweet spot" where the performance really shines. Both getting better at the same time, or one unlocking the potential of the other, leads to this curved, quadratic rise in performance.
In short, quadratic models include 2 and C² and interaction term RAC which includes exactly the relationship we want to capture. our paper, we test linear, second-degree polynomial relationships. Our result shows that the second-degre relationship fitted the best among others while still maintaining simplicity. More complex functional forms would obscure the clear individual contribution of A and C, well as interaction between alignment and correspondence.
Next, We discuss the interation alignment and correspondence from two aspects: the ideal relationship between A and C suggested by the law, and the actual relationship we observe from existing encoders. To provide the ideal relationship, extract the cross-term coefficient AC) of the fitted functions as shown in Table 4.
17

---

Published as a conference paper at COLM 2025
We observe two kind of interactions:
Benchmark
AAC
Synergy AC > O): MMBench, MME, TextVQA, VizWiz, ScienceQA, and BeedBench all have &AC > O. For these six tasks, performance improves more than additively when both A and C increase.
MMBench
+1.1301
MME
+1.5819
MMMU
-0.4872
OKVQA
-0.0364
Mild trade-off AC < 0): MMMU and OKVQA show aAc < O, indicating a slight penalty if A and C are both pushed upward (i.e., diminishing returns).
TextVQA
+4.9804
VizWiz
+2.9964
ScienceQA
+4.4644
These results demonstrate that most benchmarks, align- ment and correspondence actually reinforce each other (pos- itive interaction). Therefore, A and C strengthen each other, so the ideal scenario would be increasing both to achieve the best performance.
SeedBench
+0.5772
Table Interaction term AAC of fitted function for all eight benchmarks.
However, from our observations of A and C of vision encoders, the factors often have a trade-off: Intuitively, measures how well image features align to a shared text embedding, whereas captures imtra-image consistency (e.g,, visual cluster compactness). Forcing very high alignment often collapses visual cluster structure (hurting correspondence). In practice, looking at the A and C scores in Appendix Section A.5, common visual encoders often have high A but low C (CLIP and SigLIP) or low A but high C (DINOv2 and Diffusion features).
A.4 All Settings Benchmark Performance
In this section, we present the performance results of all 15 vision representation settings, as summarized in Table 5. The benchmarks we evaluated include:
MMBench (Liu et al., 2023): A set of multiple-choice questions designed to assess 20 different ability dimensions related to perception and reasoning.
MME (Fu et al., 2023): A dataset focused yes/no questions, covering areas such as existence, counting, position, and color, primarily based on natural images.
MMMU (Yue et al., 2024): Multiple-choice questions targeting college-level sub- ject knowledge and deliberate reasoning, primarily testing the language model's abilities.
OKVQA (Marino et al., 2019): Open-ended questions based on the MSCOCO (Lin al., 2014) dataset, spanning 10 different knowledge categories.
TextVQA (Singh al., 2019): Open-ended questions designed evaluate the model's OCR capabilities.
VizWiz (Gurari et al., 2018): Open-ended questions sourced from people who are blind, aimed at testing the model's OCR capabilities.
ScienceQA (Lu et al., 2022): A multiple-choice science question dataset, with 86% of the images being non-natural, covering topics in natural science, social science, and language science.
SEED-Bench (Li et al., 2024b): benchmark consisting of multiple-choice questions designed to assess both spatial and temporal understanding.
A.5 All Settings AC Scores
We provide the AC scores of all 15 vision representation settings, as summarized in Table
A.6
More Visualization of Correspondence
We provide additional visualizations correspondence for four different vision represen- tations: CLIP, SIgL.I, DINOv2, and Stable Diffusion 1.5. Figures 7 and display pairs
18

---

Published as a conference paper at COLM 2025
CLIP@336
CLIP@224
OpenCLIP
SigLIP Base
SigLIP Large
MMBBnch
64.26
64.18
63.40
61.86
65.46
MME
1502.70
1449.64
1460.28
1425.00
1455.18
MMMU
35.0
36.2
37.2
35.8
36.6
OKVQA
53.20
56.13
56.36
54.01
57.02
TextVQA
46.04
42.67
40.13
36.00
42.44
VizWiz
54.27
51.69
52.11
53.17
51.49
ScienceQA
69.26
68.82
67.87
66.88
70.20
SEED-Bench
66.09
65.13
64.71
64.40
66.28
C+D0224
C+D@336
SigLIP2 Large
DINOv2
DIT
MMBench
65.72
65.12
67.35
58.50
33.68
MME
1436.42
1475.19
1486.66
1295.47
902.00
MMMU
36,9
34.6
34.2
34.6
32.7
OKVQA
55.94
56.92
57.57
54.78
33.75
TextVQA
40.04
46.17
47.2
14.27
10.82
VizWiz
54.04
53.44
56.55
49.67
49.92
ScienceQA
69.11
67.63
69.41
65.15
63.46
SEED-Bench
65.39
66.38
68.22
61.39
40.66
SDXL
SD3
SD2.1
SD1.5
SDim
MMBench
43.73
32.82
28.87
42.53
52.84
MME
1212.69
843.43
905.27
1163.90
1205.33
MMMU
32.8
32.4
32.8
33.9
33.7
OKVQA
41.78
34.95
34.41
39.14
46.04
TextVQA
11.81
10.77
10.46
11.64
13.77
VizWiz
47.14
47.12
46.59
50.14
47.33
ScienceQA
65.25
62.27
62.67
63.31
66.34
SEED-Bench
53.78
38,94
38.82
50.00
50.33
Table 5: Benchmark performance of all 15 settings. C+D means feature combination of CLIP and DINOv2. The table provides data points for function fitting and is not intended for comparison.
CLIP@336
CLIP@224
OpenCLIP
SigLIP-B
SigLIP-L
Correspondence PCK@0.10
15.66
14.30
16.22
12.89
13.66
Cross-modal Alignment
Log likelihood
-1.97
-1.98
-1.93
-1.92
-1.83
C+D@224
C+D@336
SigL.IP--L
DINOv2
DIT
Correspondence PCK@0.10
23.62
26.08
16.75
24.51
1.91
Cross-modal Alignment
Log likelihood
-1.96
-1.95
-1.81
-2.32
-3.76
SDXL.
SD3
SD2.1
SDL.5
SDim
Correspondence PCK@0.10
16.52
3.09
6.99
22.02
20.90
Cross-modal Alignment
Log likelihood -2.69
-4.13
-2.81
-2.53
-2.37
Table 6: AC scores all 15 settings. C+D means feature combination CLIP and DINOv2. The table provides data points for function fitting and is not intended for comparison.
19

---

Published as a conference paper at COLM 2025
of source-target images for each of the four vision representations. In each pair, the left image is the source, and the right image is the target. The red dot on both images indicates the predicted key points using the vision representation. Ideally, these key points should correspond to the same semantic meaning. For example, a red dot on the "left cat ear" in the source image should correspond to the "left cat ear" in the target image. The green areas highlight regions of relatively high similarity with the source points.
In Figure 7, DINOv2 demonstrates superior correspondence for natural images compared to the other vision representations. It accurately matches small parts of the cat between the left and right images, whereas CLIP struggles to correctly identify and align features such as left, right, front, and back.
In Figure 8, the CLIP family shows precise correspondence for text within images. For instance, when the source image points text like "LLaVA" or "VQAv2", CLIP accurately matches all instances of the text the target image. contrast, other vision representations known for "accurate correspondence" in computer vision, such as DINOv2 and Stable Diffusion, fail to provide the same level of accuracy when dealing with images containing text. This emphasizes key distinction in selecting vision representations for computer vision tasks versus multimodal large language models (MLLMs).
CLIP
SigLIP
DINOv2
SDL.5
Figure 7: Correspondence of natural images for different vision representations.
A.7
Pseudo Code
Computing the A score is simple loss calculation using the frozen MLLM; therefore, provide pseudocode for the other algorithms, including the computation of the score at Algorithm 1, region-based sampling at Algorithm 2, and the AC policy at Algorithm 3.
A.8 Limitation of AC Policy
Figure 9 shows two benchmarks-Min MMEE and MMMU-where the AC policy fails to predict the optimal vision representation. For details on the reason this behavior, please refer to Section 6.
A.9 AC Policy Recall@1
20

---

Published as a conference paper at COLM 2025
Algorithm 1: COMPUTE C SCORE
Input: Set of paired images with key points S from SPair-71k; Vision encoder E;
Threshold T.
Output: score for vision encoder E.
// Initialize correspondence lists
G G |]
1/ Ground truth keypoint correspondences
P+ 1;
// Predicted keypoint correspondences
foreach K, K2) E do
// Extract feature representations
Fi + E(1);
B + E(l2);
// Compute similarity matrix
Ssim + F..
// Transform keypoints from to 12
R2 calculate kelyulate keypoint. hraiitattttttt.ttoooootttoonooo
K1);
// Store ground truth and predicted keypoints G.apend
P.append (K2))
// Compute correctness score
Eerror & buulidean-distancce G);
Ceorrect +- sum Eerror < T);
Cscore total keypoints K₂²
Coonect
Algorithm 2: REGION-BASED SAMPLING
Input: k A and C score pairs from models ACs; past_san pas__sa models; current sampling level (1 K,, increments when regions are exhausted as each region is sampled only once)
Output: Sampled model to train next
regions <- ();
for AC E ACs do
region + and С coordinates
; // Identify the region based on
regions (reiion (reyj.append(mmaag A, A,C));
Remove models past_sampled from regions; sanatininnrregione + keys of regions; chosen_region +- randomly select from remaining-regione
model +- randomly select from regions((hossnnreginn)
21

---

Published as a conference paper at COLM 2025
LLBVAyB 7B
LLaVA7B
LLaVAyB
LLaVAyB
LLaVA7B
LLaVA7B
LLaVAyB
LLaVA7B
LLaVA7B
LLaVA7B
LLaVAyB
LLaVA7B
LLaVA7B
LLaVA7B
LLaVAzB
LLa LAyB
LLaVAyB
LLaVAyB
VABB
LLaVA
LLaVA7B
LLaVA7B
LLaVAzB
LLaVA7B
CLIP
SigLIP
DINOv2
SD1.5
Figure 8: Correspondence of images with text for different vision representations.
Algorithm 3: AC POLICY
Input: & vision encoders with pretrained projectors V; computation budget K' Output: A ranking of k MLLMs based on performance
ACs + (Compute_A..c Acoorre))) Compute (ttpputee..ccoo(e)))) D V];
past_sampled <- [];
train + []:
frainn.errormance <- []: for i - to K' do
model + www.Reio.bbaeedSSaaeeeeeeeeoe Sampling( ACs, past-sampled); ferformance < Fully train model;
train_ACs.append(AA of model);
train perrorrrunnccappennneerrrorno (orformancc
past ppsllssommpllll.peedmmmtttt
poly + Pww.nomillFaatuuusssggggg
2);
transformeedLrrinnnACs + poly.fit.transfffe wtt..trrannffmmttrinnAACCCC regression <- SineerRegresssoon()
engression.iittraameee formed formed train ACs, train per formacec;));; ranking +- Rank V by regression predictions ACs;
MME
MMMU
In Section 5.5, we used Recall@3 as a metric for AC policy effective- ness to capture potential perfor- mance fluctuations during MILMM training and evaluation. Our AC policy's ' core aim is to narrow the search space to a small set of promising vision encoders, rather than pinpointing single best en- coder. Recall@3 directly quantifies whether the true top-performers appear within our top-3 recommendations, which:
110
1.00
0.5
0.50
025
0.25
Figure 9: Limitation of AC policy.
22

---

Published as a conference paper at COLM 2025
SEED-Bench
MMBench
OKVQA
1.00
1.50
0.5
0.15
0.15
0.25
0.25
0.25
AC Predicted
000
12 13 14
0.50
10
10
Prediction Recall
TexIVQA
VizWiz
ScienceQA
Optimal VR
Random Predicted
0.15
0.15
0.59
0.59
0.50
0.00
1011
Number of Full Training
Figure 10: Number of full training (LLM finetuning) cycles required to include the optimal vision representation within the top-1 predictions (Recall@1).
Reflects practical
typically inspect multiple ("top 3-5")
candidates, not just one.
Smooths score noise-unlike Top-1 accuracy, Recall@3 tolerates minor ranking fluctuations while still ensuring coverage of strong encoders.
Focuses on coverage-rather than overall ranking correlation (e.g. Pearson's r) or mean reciprocal rank (which over-emphasizes exact positions), Recall@3 measures actionable retrieval quality: "Is the best encoder in my shortlist?" In other words, we put coverage higher priority, "Is the best encoder in my shortlist" is more important than is the top-3 order correct.
While we believe this makes Recall@3 the most appropriate metric for AC's intended use- case, we also report Recall@1 for predicting the optimal vision representation. As shown in Figure 10, AC policy consistently outperforms random testing even under this stricter metric.
23


Reviews:
[{"review_id": "56bjfc9Q5k", "reviewer": "Reviewer_s8cX", "summary": "This paper presents a method of optimally selecting a frozen vision encoder for training multimodal large language models by using two metrics - cross-modal alignment (A) and vision representation correspondence (C). A refers to the alignment between the vision and text encoders and C refers to the ability of matching points in one image to their semantically corresponding points in other images. The theoretical justification presented for the correlation between high A and C scores and overall model performance is that high A score leads to less computation to bridge the gap between different modalities during training, and a high C score leads to precise attention in the image embeddings. The paper presents implementations of A score and C score and a combined AC score, verify the predictive power of the AC score and present a framework to sample vision representations from a set of vision encoders and given a language encoder and predict the optimal vision encoder using the AC score at a cost significantly less than finetuning all combinations to compare downstream performance. Finally, the paper presents experiments conducted with 15 vision encoders and 8 benchmarks. While the approach works on vision based benchmarks, its performance is limited on OCR-based benchmarks. The paper theorizes that this is due the C score being focused on natural images, and can be improved by adding OCR-based correspondence datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ Overall this is a comprehensive paper:\n  + proposes a novel idea of using 2 metrics to predict downstream model performance.\n  + provides a sound theoretical justification for the correlation between these scores and downstream performance\n  + provides all the details required to understand the implementation of the scores\n  + proposes a clear framework to use the scores in practice to select an optimal vision encoder\n  + presents ample empirical validation of the idea with 15 vision encoders and 8 benchmarks\n  + highlights the limitations of the approach and future directions.\n+ The paper achieves a 99.7% reduction in the computational cost of selecting vision encoders and is able to predict downstream model performance using the A and C score with a coefficient of determination of 94.06%. This seems to be a promising result and could significantly simplify multimodal large language model training.", "reasons_to_reject": "These are not strong reasons to reject.\n+ The authors could include the computational cost of learning the AC scores and predicting the optimal vision encoder and contrast it with the traditional approach. The 99.7% reduction in computational cost is only mentioned in the abstract and not explained in the paper.\n+ The method proposed in the paper is applicable only to multimodal models with a pretrained vision and language encoder. A future direction could be exploring the incorporation of A and C scores when learning the language and vision representations as part of model training.\n+ As acknowledged by users, this approach works well only for natural image datasets and doesn’t do well for OCR datasets. Since these are an extremely important aspect of multimodal learning, performing well on these datasets would be pivotal for the applicability of this method.\n+ It would be interesting to see how this approach scales to other modalities like audio.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zYACa76uK2", "reviewer": "Reviewer_Qc4i", "summary": "This paper investigates the relationship between vision representations and the performance of Multimodal Large Language Models (MLLMs). The authors propose the \"Law of Vision Representation,\" suggesting that cross-modal alignment (A) and vision representation correspondence (C) are key determinants of MLLM performance. They introduce metrics to quantify A and C (A SCORE, C SCORE) and find a quadratic relationship between these scores and performance on various benchmarks. Based on this observed relationship, an \"AC policy\" is presented to efficiently select vision representations, aiming to reduce the extensive computational costs associated with empirically finetuning multiple MLLMs. Experiments are conducted with 15 vision representations and 8 benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Empirical Exploration and Practical Utility:\nThe paper conducts a systematic empirical study across a considerable number of vision encoders and benchmarks, which is valuable for the community. The proposed AC policy, if robust, offers a tangible benefit by potentially reducing the computational resources needed to identify effective vision representations for MLLMs. The claimed 99.7% cost reduction is a significant practical motivation.\n\n2. Quantification of Factors:\nThe effort to quantify concepts like cross-modal alignment and correspondence through specific metrics (A SCORE, C SCORE) is a step towards a more principled understanding, moving beyond purely heuristic selection of vision encoders.", "reasons_to_reject": "1. Limited Conceptual Novelty and Overstated Claims (\"Law\"):\nThe primary concern with this paper is the perceived limited conceptual novelty. The core components used – the MLLM architecture (decoder-only), the vision encoders (CLIP, DINOv2, SigLIP, etc.), and the projector – are all well-established. The identified factors, cross-modal alignment and correspondence, are not new concepts in multimodal research. The contribution appears to be more in the systematic quantification and correlation analysis of these known factors within specific MLLM setups, rather than the discovery of fundamentally new principles or components.\n\n2. Scope and Generalizability of the \"Law\" and AC Policy:\nThe definition of A SCORE (caption likelihood) and C SCORE (PCK on SPair-71k) might limit the \"law's\" applicability. As the authors rightly point out in their limitations, the C SCORE is not well-suited for OCR-heavy tasks. This suggests the current formulation of A and C might not be universally optimal predictive factors for all MLLM capabilities, weakening the claim of a general \"law.\"\n\n3. AC SCORE Formulation and Potential Overfitting:\nThe AC SCORE is derived by fitting a second-degree polynomial (with 6 trainable parameters $\\tau v_{\\alpha\\beta}$) to benchmark performance using A and C scores from 15 vision representations. While the $R^2$ value is high (94.06%), fitting a 6-parameter model to 15 data points warrants caution regarding potential overfitting. The robustness of these $\\tau v_{\\alpha\\beta}$ parameters across different sets of vision encoders or slight variations in benchmarks would be important to establish.\n\n4. Practicality of A SCORE Calculation:\nWhile the AC policy aims to save finetuning costs for the full LLM, the calculation of the A SCORE for *all* candidate vision encoders still requires Stage 1 training (training the projector). For a very large search space of vision representations, this initial step could still be computationally intensive.", "questions_to_authors": "See the Reasons To Reject", "extra_scores": {}}, {"review_id": "hbeoxgF9SN", "reviewer": "Reviewer_UhUv", "summary": "This paper proposes the \"Law of Vision Representation\" in Multimodal Large Language Models (MLLMs), demonstrating a strong correlation between cross-modal alignment (A-score), representation correspondence (C-score), and overall model performance. Extensive experiments across diverse vision representations validate that model performance exhibits a quadratic correlation with the transformed function of A-score and C-score, which can be used to reduce computational costs when deciding the optimal vision representations for MLLMs.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose a novel methodology utilizing visual-semantic alignment (A-score) and correspondence (C-score) metrics for optimal visual feature selection. This approach presents a theoretically sound and innovative solution.  \n2. The proposed method demonstrates significant computational efficiency improvements over traditional fine-tuning approaches, achieving substantial resource savings.\n3. The experimental design is thorough and comprehensive, covering multiple vision representation. And the results are analyzed in depth.", "reasons_to_reject": "1. In this paper, the authors propose a linear relationship between the AC score and model performance, termed the \"Law of Vision Representation in MLLMs.\" However, have other potential factors been considered that might affect or weaken this conclusion? For instance:\n - Exploring a broader range of vision encoders\n - Investigating different types and scales of LLM backbones\n - Considering alternative visual representations or varying downstream tasks\n\n2. Why is Recall@3 used as the evaluation indicator for the AC strategy, and what are its advantages compared to other indicators?\n\n3. The illustration and writing can be improved for better understanding.", "questions_to_authors": "There are some unclear parts requiring further clarification:\n\nAbout the Equation 3:\n>  $\\omega_{\\alpha\\beta}$ are trainable parameters optimized to best fit benchmark performance\n  \nFor different downstream tasks, are these $\\omega_{\\alpha\\beta}$ parameters learned from scratch, or updated through incremental learning? Is there a better way for cross-task adaption?", "extra_scores": {}}, {"review_id": "PeRX4eD5V1", "reviewer": "Reviewer_ufKw", "summary": "This paper introduces the \"Law of Vision Representation\" for MLLMs, establishing a correlation between cross-modal alignment, visual representation correspondence, and model performance. Using quantitative A-score and C-score metrics, experiments across 15 visual representations and 8 benchmarks reveal a quadratic relationship between these scores and performance. The authors propose an AC policy to identify optimal visual representations without repeated language model fine-tuning, reducing computational costs by 99.7%.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.This paper addresses an important challenge in the MLLM field—how to select optimal visual representations, which is currently often an empirical trial-and-error process consuming substantial computational resources. The writing is good, with a clearly defined and important problem statement.\n\n2.The motivation is clear, with a relatively solid theoretical foundation. The authors not only propose a quantitative relationship between visual representation and model performance but also provide theoretical proofs in the appendix explaining why high alignment and high correspondence lead to better model performance.\n\n3.The experimental design is relatively reasonable, covering 15 different visual representation configurations, including various encoder types, sizes, input resolutions, and feature combinations, while evaluating on 8 different types of benchmarks, including visual understanding and OCR tasks.\n\n4.The work has practical value, as the proposed AC policy significantly reduces computational costs, which is important for both researchers and industrial applications.", "reasons_to_reject": "1.The Law of Vision Representation specifically focuses on decoder-only MLLMs, which limits its application scope. How should other types of MLLMs be handled, and are there alternative approaches? For example, how should optimal visual representation models be selected for cross-attention-based MLLMs?\n\n2.The paper proposes that A and C scores have a quadratic relationship with performance, but does not deeply explore why it's a quadratic relationship rather than another functional form. This point needs more in-depth analysis.\n\n3.The paper does not sufficiently analyze the potential trade-off relationship between A and C, merely stating in Formula 3 that \"AC Score to capture both the individual contributions of alignment and correspondence.\" What is the intrinsic connection between these two factors?\n\n4.The paper experiments with different visual representation models. However, it only uses one base language model (Vicuna-7B 1.5) and does not verify applicability across language models of different scales and architectures (such as T5 model[1]).\n[1] Raffel C, Shazeer N, Roberts A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer[J]. Journal of machine learning research, 2020, 21(140): 1-67.\n\n5.The radar chart on the right side of Figure 6 is rather blurry, making the text illegible.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper emphasizes the correlation between cross-modal alignment and correspondence scores (A and C scores) and MLLM performance. High A and C scores are indicative of superior model performance. 
The authors further propose an AC policy for selecting optimal vision representations.
The AC policy selects the optimal vision representation for MLLMs, which significantly reduces computational costs and enhances both accuracy and efficiency in predicting the optimal vision representation

Pros
- The authors present comprehensive results on 15 vision encoders and 8 benchmarks.
- The AC policy has been demonstrated to efficiently predict the optimal vision representation, achieving a high prediction recall rate (e.g., 87.72% Recall@3) with minimal training resources, which is a significant improvement over random selection methods.
- The AC policy is versatile and can be applied to various types of vision representations.

Cons
- The applicability of the AC policy to other vision encoders and LLM architectures beyond those tested in the paper may be limited without further validation. The authors provide the additional results on Qwen2.5 and promise to include in the revised version.
- The paper acknowledges that the AC policy has a lower correlation with tasks that involve a significant amount of text in images (OCR-heavy), since the datasets used to compute the A and C scores do not adequately represent tasks that require understanding of tables, charts, and other text-heavy content.
Decision: Accept