Law of Vision Representation in MLLMs

## Abstract

Abstract We present the “Law of Vision Representation” in multimodal large language models (MLLMs). It reveals a strong correlation between the combination of cross-modal alignment, correspondence in vision representation, and MLLM performance. We quantify the two factors using the cross-modal A lignment and C orrespondence score (AC score). Through extensive experiments involving thirteen different vision representation settings and evaluations across eight benchmarks, we find that the AC score is linearly correlated to model performance. By leveraging this relationship, we are able to identify and train the optimal vision representation only, which does not require finetuning the language model every time, resulting in a 99.7% reduction in computational cost. The code is available at https://github.com/bronyayang/Law_of_Vision_Representation_in_MLLMs .

## 1 Introduction

Current multimodal large language models (MLLMs) (Chen et al., ; Liu et al., ; ) have achieved remarkable advancements by integrating pretrained vision encoders with powerful language models (Touvron et al., ; Zheng et al., ). As one of the core components of a general MLLM, the vision representation is critical. Many researchers have utilized CLIP (Radford et al., ) as the primary image feature encoder, but its limitations are increasingly noticeable (Tong et al., ; Geng et al., ; Yao et al., ). As a result, alternative vision representationand the combination of multiple vision encoders are being actively explored (Tong et al., ; Lin et al., ).

Despite this growing attention, the selection of vision representation has largely been empirical. Researchers typically test a set of vision representations on a specific MLLM and choose the one that yields the highest performance on benchmark tasks. This approach, however, is constrained by the number of representations tested and does not address the underlying factors that make certain feature representations perform better than others. As a result, the optimal vision representation for a specific MLLM is often determined by empirical performance rather than a deep understanding of the features that contribute to success. The question of what fundamentally makes a feature representation achieve the highest performance remains largely unanswered.

To address this gap in understanding what makes a vision representation optimal for MLLMs, we propose the Law of Vision Representation in MLLMs. It aims to explain the key factors of vision representation that impact MLLM benchmarks performance. Our findings reveal that the cross-modal Alignment and Correspondence (AC) of the vision representation are strongly correlated with model performance. Specifically, an increase in the AC of the selected vision representation leads to improved model performance. To quantify this relationship, we define an AC score that measures cross-modal alignment and correspondence in vision representation. The AC score and model performance exhibit a linear relationship, with a coefficient of determination of 95.72%.

Furthermore, the Law of Vision Representation guides the selection of an optimal vision representation for MLLMs. Originally, this process was extremely costly because even subtle changes in vision encoding methods—such as switching encoder types, altering image resolution, or testing feature combinations—require finetuning the language model (Lin et al., ). For example, using a top data-efficient MLLM pipeline with a 7B language model requires 3,840 NVIDIA A100 GPU hours to test the 10 encoders used in this study, amounting to a cost of approximately $20,000(^1^11https://replicate.com/pricing). Testing additional encoders leads to a linear increase in cost. Moreover, the recent trend of feature combination, which often results in better performance, necessitates combinatorial testing of vision encoders. Testing all possible combinations of 10 encoders results in 1023 combinations, exponentially increasing the cost and energy consumption. This process consumes approximately 100,000 kilowatt-hours(^2^22https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet.pdf), enough to drive an electric vehicle around the Earth 13 times.

Thus, we are the first to propose a policy, AC policy, that selects the optimal vision representation using AC scores within the desired search space. Unlike traditional methods that rely on benchmarking performance, the AC policy enables the expansion of the search space—allowing for an increased number of vision representations to be considered—without incurring additional costs. We demonstrate that this approach enhances both accuracy and efficiency compared to randomly searching for the optimal representation. The policy successfully identifies the optimal configuration among the top three choices in 96.6% of cases, with only three language model finetuning across a 13-setting search space.

## 2 Related Works

### 2.1 Vision for MLLMs

Recent studies have explored various vision representations in MLLMs (Beyer et al., ; Ge et al., ; Liu et al., ; Wang et al., ; Sun et al., ; Luo et al., ). Interestingly, some findings indicate that relying solely on encoders outside of the CLIP family (Cherti et al., ; Zhai et al., ; Li et al., ), such as DINOv2 (Oquab et al., ) and Stable Diffusion (Rombach et al., ), often leads to lower performance scores (Karamcheti et al., ; Tong et al., ). However, combining features from these encoders with CLIP features—such as concatenating image embeddings in the token or channel dimension—significantly enhances performance beyond using CLIP alone (Tong et al., ; ; Liu et al., ; Kar et al., ). Researchers intuitively suggest that these additional encoders provide superior detail-oriented capabilities, but no studies have thoroughly analyzed the underlying causes of the performance change (Wei et al., ; Lu et al., ). This suggests that the attributes of an optimal vision representation remain not fully understood.

### 2.2 Cross-modal Alignment

Cross-modal alignment refers to the alignment between image and text feature spaces (Duan et al., ). This concept emerged with the introduction of text-image contrastive learning (Radford et al., ; Jia et al., ). Although current MLLMs utilize contrastively pretrained image encoders, the challenge of achieving effective alignment persists (Ye et al., ; Zhai et al., ; Woo et al., ). Despite efforts to critique the limitations of CLIP family representations and explore alternative vision representations, many approaches continue to rely on contrastively pretrained encoders or adding contrastive loss without fully eliminating them (Zhang et al., ; Lu et al., ; Tong et al., ; ; Liu et al., ). In our work, we point out that alignment in vision representation is essential for improved model performance and is crucial for data efficiency. Without pre-aligned vision representations, extensive data pretraining is required to achieve cross-modal alignment within the language model (Ge et al., ; Chen et al., ; Li et al., ).

### 2.3 Visual Correspondence

Visual correspondence is a fundamental component in computer vision, where accurate correspondences can lead to significant performance improvements in tasks, such as image detection (Xu et al., ; Nguyen & Meunier, ), visual creation (Tang et al., ; Zhang et al., ), and MLLMs (Liu et al., ), etc. Correspondences are typically categorized into semantic- and geometric-correspondences. Semantic correspondences (Zhang et al., ; Min et al., ) involve matching points that represent the same semantic concept not necessarily representing the same instance. Geometric correspondences (Sarlin et al., ; Lindenberger et al., ), on the other hand, require matching the exact same point across images, which is often crucial for low-level vision tasks, such as pose estimation (Sarlin et al., ; Lindenberger et al., ; Zhang & Vela, ),and SLAM tasks, etc.

Several studies have pointed out that the CLIP family’s vision representation ”lacks visual details” (Lu et al., ; Tong et al., ; Ye et al., ). We explain this observation through the concept of correspondence. Current multi-modal large language models (MLLMs) convert images into embeddings, with each embedding representing a patch of the image. Image features with high correspondence increase the similarity within internal image patches on similar semantics, thereby enabling the retrieval of more detailed information.

## 3 Law of Vision Representation in MLLMs

We introduce the Law of Vision Representation in Multimodal Large Language Models (MLLMs). It states that the performance of a MLLM, denoted as $Z$, can be estimated by two factors: cross-modal alignment ($A$) and correspondence ($C$) of the vision representation, assuming vision representation is the sole independent variable while other components (e.g., language model and alignment module) remain fixed. This relationship can be expressed as:

$$ $Z\propto f(A,C)$ (1) $$

where $f$ is a linear function on second-degree polynomial transformations of $A$ and $C$.

### 3.1 Theoretical Justification

In this section, we theoretically analyze how an increase in $A$ and $C$ leads to improved model performance. When a vision representation demonstrates high cross-modal alignment and accurate correspondence, the MLLM exhibits the following desired properties:

- •
When training a MLLM, if the vision representation is closely pre-aligned with the language distribution, the pretrained language model requires less computational effort to bridge the gap between different modalities during finetuning. In Section A.1, we provide theoretical justification that finetuning on well-aligned multimodal data is about equivalent to finetuning on text-only data, eliminating additional effort beyond language finetuning. This efficiency can lead to improved performance, especially in scenarios where the available training data for finetuning is limited.
- •
If the vision representation ensures accurate correspondence, the attention within the image embeddings is precise. Consequently, the MLLM develops a refined focus on visual content, capturing even details that cannot be derived solely from text-to-image attention, leading to a more detailed interpretation of the image. We provide theoretical justification in Section A.2.

### 3.2 Empirical Justification

In this section, we empirically show that AC is strongly correlated to model performance. To quantify the correlation between AC and model performance, we first propose methods to measure cross-modal alignment and correspondence within the vision representation:

- •
To quantify cross-modal alignment, we aim to compare the image and text embeddings of the same concept. However, finding the same concept is difficult since it requires alignment. To address this, we use the CLIP vision embedding as a reference. We calculate the maximum cosine similarity $\textsc{S}_{\textsc{c}}$ between vector pairs from the CLIP embedding $\hat{E}$ and target vision representation embedding $E$:
$\textsc{A Score}=\frac{1}{n}\sum^{n}_{i=0}\ \max\limits_{u,v}\ \textsc{S}_{
\textsc{c}}(\hat{E}_{i}^{(u)},E_{i}^{(v)})$
(2)
where $n$ is the total number of image samples, and $E_{i}^{(v)}=MLP(F_{i})^{(v)}$ is the $v$-th embedding vector, resulting from the vision feature $F$ of the $i$-th image.
- •
To compute the correspondence score, we extract features from paired images, resulting in $F^{i}_{s}$ and $F^{i}_{t}$ from a specific source and target pair. Given ground truth key points $\{p^{s}_{1},\dots,p^{s}_{m}\}$, we obtain the predicted key points $\{p^{t}_{1},\dots,p^{t}_{m}\}$ using the features. The correspondence score is the Percentage of Correct Keypoints (PCK) calculated using the following equation:
$\textsc{C Score}=\frac{1}{m}\sum^{m}_{i=0}\mathbbm{1}_{\left\|p^{t}_{j}-p^{s}_
{j}\right\|_{2}<T}$
(3)
where $T$ is a threshold defined as proportional to the bounding box size of the object instance in the image.

Finally, AC score is a second-degree polynomial transformation of the $A$ and $C$ score:

$$ $\textsc{AC Score}=\sum^{2}_{i=0}\sum^{2-i}_{j=0}\beta_{ij}A^{i}C^{j}$ (4) $$

#### Results.

Figure: Figure 1: $R^{2}$ values for linear regression models fitted on various scores.
Refer to caption: extracted/5820567/figure/law_results.png

We fit a simple linear regression model using 13 vision representations across 4 vision-based MLLM benchmarks. As shown in Figure 1, the average coefficient of determination ($R^{2}$) obtained is 95.72% when using the AC score of the vision representations. For comparison, we also fit models using 13 random scores, the A score alone, and the C score alone, all with polynomial transformations. The random scores and single-factor models show significantly lower correlations with performance. This result highlights the strong correlation between the AC score and MLLM performance, validating the Law of Vision Representation. Refer to Section 5.4 for details.

## 4 AC Policy

Figure: Figure 2: Overall framework of AC policy.
Refer to caption: x1.png

#### Problem Formulation.

A general MLLM architecture consists of a frozen vision encoder followed by a trainable connector (alignment module) and the language model, similar to the setup in LLaVA (Liu et al., ). To determine the optimal out of $k$ vision representations, we originally needs finetune LLM $k$ times, making the scaling of $k$ difficult. Therefore, given $N$ vision encoders, we propose AC policy, as illustrated in Figure 2, to efficiently estimate the optimal vision representation from a search space consisting of $k$ vision representation out of all $2^{N}-1$ possible feature combinations. We finetune only $k^{\prime}$ LLMs to obtain downstream performance, allowing $k$ to scale without significant cost, where $k^{\prime}\ll k$.

#### Policy Fitting.

Let $\mathbf{X}\in\mathbb{R}^{k\times 6}$ be the matrix containing AC scores of vision representation in the search space. We subsample $k^{\prime}$ data points from $\mathbf{X}$, denoted as $\mathbf{X}_{s}\in\mathbb{R}^{k^{\prime}\times 6}$, to serve as the input to the linear regression model:

$$ $\mathbf{y}=\mathbf{X}_{s}\mathbf{w}+\mathbf{\epsilon}$ (5) $$

Here, $\mathbf{w}\in\mathbb{R}^{6}$ is the vector of model parameters, $\mathbf{\epsilon}\in\mathbb{R}^{k^{\prime}}$ is the vector of error terms, and $\mathbf{y}\in\mathbb{R}^{k^{\prime}}$ represents the downstream performance on a desired benchmark.

#### Sampling Strategy.

The selection of $k^{\prime}$ can impacts the function fit and, consequently, the accuracy of predictions. To avoid sampling points that are too close in terms of their A and C scores, we employ a sampling strategy based on the coordinates.

The normalized A and C score pairs of $k$ vision representation can be plotted on a 2D graph as coordinates $(A,C)$, To ensure diverse sampling, we divide the graph into regions. For each iteration $j$ in which the total sampled points do not yet fulfill $k^{\prime}$, we divide the graph into $4^{j}$ equal regions. We then remove empty regions and those that contain previously sampled points. The next data point is randomly selected from a remaining region.

**Table 1: Number of LLM finetunings required to achieve approximately 90% Recall@3 in predicting the optimal vision representation.**
| Benchmark | Number of Fintuning | Recall@3 |
| --- | --- | --- |
| Random | 12 | 92.04% |
| MMBench | 4 | 90.1% |
| MME | 3 | 90.6% |
| OKVQA | 3 | 90.0% |
| SEED-Bench | 3 | 86.0% |
| MMMU | 7 | 96.5% |
| TextVQA | 5 | 83.1% |
| VizWiz | 3 | 84.6% |
| ScienceQA | 3 | 96.6% |
| Average | 3.88 | 89.69% |

#### Results.

In Table 1, we show that AC policy consistently predicts the optimal vision representation with minimal resources, given a finite search space—in this case, 13 settings. Our goal is to finetune only a small subset of the search space while still identifying the optimal vision representation within the top-3 predictions (Recall@3). However, if we randomly select a subset to train on, we need 12 out of 13 finetuning to achieve over 90% Recall@3. In contrast, the AC policy requires only 3.88 full training runs on average to reach 89.69% Recall@3. Refer to Section 5.5 for details.

## 5 Empirical Result Details

### 5.1 Experiment Settings

**Table 2: Vision representation explored.**
| Vision Representation | Resolution |
| --- | --- |
| *Single vision encoder: feed-forward models* |  |
| OpenAI CLIP ViT-L/14 | 224 |
| OpenAI CLIP ViT-L/14  (Radford et al., ) | 336 |
| OpenCLIP ViT-L/14  (Cherti et al., ) | 224 |
| SigLIP ViT-L/16  (Zhai et al., ) | 224 |
| DINOv2 ViT-L/14  (Oquab et al., ) | 224 |
| *Single vision encoder: diffusion models* |  |
| SD 1.5  (Rombach et al., ) | 768 |
| SD 2.1  (Rombach et al., ) | 768 |
| SD XL  (Podell et al., ) | 768 |
| SD Image Variations | 768 |
| SD 3  (Esser et al., ) | 512 |
| DiT  (Peebles & Xie, ) | 512 |
| *Multiple vision encoders: feature combination* |  |
| CLIP+DINOv2 ViT-L/14 | 224 |
| CLIP+DINOv2 ViT-L/14 | 336 |

For our specific MLLM pipeline, we follow the training procedure, general architecture, and dataset outlined in LLaVA (Liu et al., ). The training process consists of two stages: in the first stage, we train a 2-layer GeLU-MLP connector using the LLaVA 1.5 dataset with 558K samples. In the second stage, we train both the connector and the language model on the expanded LLaVA 1.5 dataset with 665K samples. It is important to note that for each training, all factors are held constant except for the vision representation being changed. The 13 vision representation in this paper are outlined in Table 2. The MLLM benchmarks used in this paper includes 4 vision-based benchmarks, MMBench (Liu et al., ), MME (Fu et al., ), OKVQA (Marino et al., ), SEED-Bench (Li et al., ), and 4 QCR-based benchmarks including, MMMU (Yue et al., ), TextVQA (Singh et al., ), VizWiz (Gurari et al., ), ScienceQA (Lu et al., ).

### 5.2 AC Score

To compute the cross-modal alignment score, we perform stage 1 training with all the vision representations to obtain the MLPs. This process requires significantly less computation than stage 2, involving only 0.298% of the trainable parameters. The alignment score for each benchmark is averaged across 100 randomly sampled images. For the correspondence score, we follow common practices using the SPair-71k (Min et al., ) dataset. Consequently, each benchmark has its own alignment score, while the correspondence score remains consistent across all representations.

### 5.3 Feature Extraction

Both MLLM training and score computation involve image feature extraction. Below, we introduce the approach for obtaining two types of vision representations.

#### Vision Representation from Feed-forward Models.

Given an image $I\in\mathbb{R}^{H\times W\times 3}$ we process it either in its raw form for U-Net models or in a patchified form for transformer models. For transformers, we extract the last hidden state $F\in\mathbb{R}^{l\times c}$ where $l$ is the sequence length and $c$ is the hidden dimension. In the case of the U-Net model, we take the intermediate activation $F\in\mathbb{R}^{\hat{H}\times\hat{W}\times c}$ after the first upsampling block. Note that the features from these two types of models are interchangeable between sequence and grid formats through reshaping and flattening. For consistency, the following sections assume that all features have been pre-converted into the same format.

#### Vision Representation from Diffusion Models.

Diffusion model is primarily used for generating images via multi-step denoising, yet a recent trend is to use diffusion model as the vision representation model (Xu et al., ; ; Zhang et al., ; Tong et al., ). Specifically, for diffusion models, given an image $I\in\mathbb{R}^{H\times W\times 3}$, we first add noise to the VAE-encoded representation of $I$:

$$ $x_{t}=\sqrt{a_{t}}\cdot\textsc{VAE}(I)+(\sqrt{1-a_{t}})\cdot\epsilon$ (6) $$

where $\epsilon\sim\mathcal{N}(0,\mathbf{I})$ and $a_{t}$ is determined by the noise schedule. Note that we utilize the little-noise strategy by setting the $t=1$. In that case, the diffusion model only denoises the noise-latents once and we treat the one-step denoising latents as the vision representation features.

### 5.4 Additional Results on the Law of Vision Representation

**Table 3: Averaged $R^{2}$ results of AC and other baselines fitting on MLLM benchmarks.**
| Fitting Data | $R^{2}$ (Vision) | $R^{2}$ (OCR) |
| --- | --- | --- |
| *No transformation on fitting data* |  |  |
| Random | 2.94% | 5.08% |
| A Score | 50.80% | 55.06% |
| C Score | 46.12% | 19.45% |
| AC Score | 83.24% | 66.18% |
| *Polynomial transformation on fitting data* |  |  |
| Random | 45.09% | 35.77% |
| A Score | 76.56% | 77.45% |
| C Score | 56.91% | 29.07% |
| AC Score | 95.72% | 85.21% |

In Section 3, we demonstrate the strong correlation between the AC score and MLLM performance by analyzing the coefficient of determination ($R^{2}$) obtained from fitting a linear regression model. In this section, we further ablate the experiments by adding baselines, fitting model performance with random scores, A scores, and C scores separately. Additionally, we explored the relationship between the A score and C score by applying two different data transformations: no transformation and second-degree polynomial transformation. We avoid higher-degree transformations to prevent overfitting, which could obscure the true relationship between A and C scores.

As shown in Table 3, the results indicate that using the AC score consistently outperforms all other settings in terms of $R^{2}$ values. While this observation holds regardless of transformation, applying a second-degree polynomial transformation to the A and C scores yields the highest correlation with model performance. This suggests an inherent trade-off between A and C scores: vision representations with high cross-modal alignment often exhibit lower correspondence, and vice versa.

Interestingly, we observe a lower correlation between OCR-based benchmark performance and C scores, which leads to a reduced correlation between the AC score and OCR-based benchmark performance. In Section 6.3, we discuss how the use of the SPair-71k correspondence dataset across all benchmarks fails to adequately capture correspondence in images containing text.

### 5.5 Additional Results on the AC Policy

Figure: Figure 3: Number of full training (LLM finetuning) cycles required to include the optimal vision representation within the top-3 predictions (Recall@3).
Refer to caption: x2.png

In Section 4, we demonstrate that fitting the AC score consistently predicts the optimal vision representation with minimal resources, given a finite search space—in this case, 13 settings. In this section, we provide detailed visualization for Table 1.

When performing ablation experiments on vision encoders, it’s common to randomly select a subset to train on. However, as shown in Figure 3, with 1000 runs of simulated ablation experiments, we found that to include the optimal vision representation 81.2% of the time, at least 11 out of the 13 settings need to be trained. This suggests that running a small subset of vision representations is unreliable, especially as the search space expands, making it increasingly unlikely to identify the true optimal representation by training only a subset.

In contrast, the AC policy requires only 3.88 full training runs on average to reach 89.69% Recall@3. For the most successful prediction benchmark, ScienceQA, the policy successfully identifies the optimal configuration among the top three choices in 96.6% of cases, with only three language model finetuning runs across a 13-setting search space. This result shows that AC policy significantly reduces the effort and cost of exploring vision representations for MLLMs.

## 6 Discussion and Limitations

### 6.1 Finding Vision Representations with High AC

Figure: Figure 4: Normalized performance improvements across 8 benchmarks by increasing resolution and combining features, thereby enhancing correspondence and cross-modal alignment.
Refer to caption: x3.png

Since AC score is highly correlated with MLLM benchmark performance, to improve MLLM from the vision side, it is essential to identify vision representations with high AC scores and add them into the search space. We suggest two strategies to achieve this: increasing resolution and combining features.

Figure 4 shows the normalized performance improvement summed across eight benchmarks. Increasing the resolution of well-aligned features directly enhances correspondence. For example, increasing the image resolution from 224 to 336 for CLIP, while maintaining cross-modal alignment, resulted in a performance increase from 7.1 to 7.3 out of 8.

Additionally, feature combination—merging two features with high A and C scores along the channel dimension—can enhance cross-alignment while preserving correspondence. We chose to concatenate features along the channel dimension to preserve context length and to align with the intuition from our high correspondence attention proof in Appendix A.2. Specifically, the high correspondence feature can support the retrieval of information in the attention mechanism for the feature with high cross-modal alignment. Combining CLIP with DINOv2, for instance, leads to an increase in cross-modal alignment, thereby improving model performance.

### 6.2 Refining A Score Design

We design the A score calculation as the maximum cosine similarity between each pair of embedding vectors from the CLIP embedding and the target embedding. However, correspondence effects can be unintentionally included if the features obtained from CLIP and the target encoder differ in resolution. For example, the A score computed for CLIP@224 with CLIP@336 is not 1, which it is supposed to be, as our assumption is that CLIP always has an A score of 1. This shows that correspondence is not fully disentangled in the A score calculation in this scenario. The best practice is to always use the same input resolution for both CLIP and the target encoder. However, since we only have CLIP at resolutions 224 and 336, we cannot compute the alignment score if the target encoder operates at a higher resolution. To mitigate this effect, we compute the averaged A score for each target vision representation using both CLIP@224 and CLIP@336, aiming to average out the influence of correspondence.

Another limitation is the use of CLIP as a reference metric. This could be problematic if another encoder with better cross-modal alignment exists in the search space. However, we believe the error should not be significant enough to cause the predicted optimal vision representation to fall outside the top-3 or top-5. If a vision representation with the same correspondence as CLIP yields significantly higher MLLM performance than CLIP, it would be advisable to switch to this model as a more accurate cross-modal alignment reference model.

### 6.3 Refining C Score Design

Figure: Figure 5: Visualization of correspondence on natural images and images containing text for CLIP and DINOv2. The left image shows the source image, while the right one shows the target image. Blue dots indicate key points with the highest similarity, and the green areas represent regions with relatively high similarity.
Refer to caption: x4.png

We have noted that OCR-based benchmark performance shows a weaker correlation with the AC score compared to vision-based benchmarks. The primary cause of this discrepancy lies in the correspondence dataset we selected to compute the C score. The SPair-71k dataset measures feature correspondence for natural images, such as objects like cats and trains. As shown in Figure 5, the CLIP encoder demonstrates poorer correspondence in natural images compared to DINOv2. However, when it comes to images containing text, CLIP exhibits significantly better correspondence than DINOv2 or any other encoders. Therefore, using SPair-71k to calculate the C score does not accurately capture true correspondence across all scenarios. Ideally, each benchmark should have its own keypoint-labeled images for correspondence evaluation. At a minimum, an OCR-specific correspondence dataset would be highly beneficial for assessing MLLMs. To our knowledge, no such dataset currently exists. We encourage further investigation in this direction, as it would be valuable across fields in MLLM, particularly for understanding tables and charts—a fundamental capability.

## Appendix A Appendix

### A.1 Theoretical Justification of Vision Representation with High Cross-modal Alignment

In Section 3.1, we state that when training an MLLM, if the vision representation is closely pre-aligned with the language distribution, then the pretrained language model requires less computational effort to bridge the gap between different modalities during finetuning. In this section, we show that using well-aligned vision representation, finetuning on multimodal data is about equivalent to finetuning on text-only data, eliminating additional effort beyond language finetuning.

Assume the vision embedding distribution $D_{image}$ and text embedding distribution $D_{text}$ are well-aligned in the MLLM. For a shared concept $c$, the image embedding after the alignment module and its corresponding text embedding, $E_{c}^{image}\sim D_{image}$ and $E_{c}^{text}\sim D_{text}$, are close in distance, meaning:

$$ $\|E_{c}^{image}-E_{c}^{text}\|\leq\epsilon$ (7) $$

where $\epsilon$ is a small constant. Given this condition, we can show that the output of the MLLM with multimodal embeddings $[E_{c}^{image},E_{1},E_{2},\dots,E_{n}]$ is close to the output with text-only embeddings $[E_{c}^{text},E_{1},E_{2},\dots,E_{n}]$.

Since our language model
$f$ is well-trained and pre-normed, the input space to each transformer layer is bounded and compact, meaning that the values of the input are bounded by $c$, a small constant. This implies that the continuously differentiable function $f$ is Lipschitz (Kim et al., ). This property ensures that small changes in the input of the language model of the MLLM result in small, controlled changes in the output:

$$ $\displaystyle\|f([E_{c}^{\text{image}},E_{1},E_{2},\dots,E_{n}])-f([E_{c}^{ \text{text}},E_{1},E_{2},\dots,E_{n}])\|$ $\displaystyle\leq L\|[E_{c}^{\text{image}},E_{1},E_{2},\dots,E_{n}]$ (8) $\displaystyle\quad-[E_{c}^{\text{text}},E_{1},E_{2},\dots,E_{n}]\|$ (9) $\displaystyle\leq L\epsilon$ (10) $$

where $L$ is the Lipschitz constant. This closeness in output distance implies that even with multimodal data, the pretrained language model mimics the training dynamics closely resemble language-only finetuning.

### A.2 Theoretical Justification of Vision Representation with Accurate Correspondence

In Section 3.1, we state that if the vision representation ensures accurate correspondence, the attention within the image embedding is precise. In this section, we show that vision representation with accurate correspondence can help vision information retrieval in the attention mechanism. Therefore, more visual details are considered even if not attended by the text token.

Consider an input $[E_{0}^{\text{image}},E_{1}^{\text{image}},E_{2},\dots,E_{n}]$ to the transformer, where the image embeddings $E_{0}^{\text{image}}$ and $E_{1}^{\text{image}}$ are derived from different patch of a high correspondence vision representation. By definition, the dot product $E_{0}^{\text{image}}\cdot E_{1}^{\text{image}}$ is large if the two corresponding original image patches share related information.

Suppose a text token $E_{2}$ attends to $E_{0}^{\text{image}}$. We show that it is also able to retrieve $E_{1}^{\text{image}}$ and vice versa. This can be demonstrated as follows:

$$ $\text{score}(E_{2},E^{\text{image}}_{0})=\frac{(E_{2}W^{Q})\cdot(E^{\text{ image}}_{0}W^{K})}{\sqrt{d_{k}}}$ (11) $$

If $\text{score}(E_{2},E^{\text{image}}_{0})$ is high, and $(E^{\text{image}}_{0}W^{K})^{\top}(E^{\text{image}}_{1}W^{K})$ is also large (assuming $W^{K}$ does not distort the vectors drastically), then by transitivity, $\text{score}(E_{2},E^{\text{image}}_{1})$ is also likely to be high. This transitivity ensures that attention is effectively spread across related visual information, enhancing the model’s ability to interpret visual content in greater detail.

### A.3 All Settings Benchmark Performance

In this section, we present the performance results of all 13 vision representation settings, as summarized in Table 4. The benchmarks we evaluated include:

- •
MMBench (Liu et al., ): A set of multiple-choice questions designed to assess 20 different ability dimensions related to perception and reasoning.
- •
MME (Fu et al., ): A dataset focused on yes/no questions, covering areas such as existence, counting, position, and color, primarily based on natural images.
- •
MMMU (Yue et al., ): Multiple-choice questions targeting college-level subject knowledge and deliberate reasoning, primarily testing the language model’s abilities.
- •
OKVQA (Marino et al., ): Open-ended questions based on the MSCOCO (Lin et al., ) dataset, spanning 10 different knowledge categories.
- •
TextVQA (Singh et al., ): Open-ended questions designed to evaluate the model’s OCR capabilities.
- •
VizWiz (Gurari et al., ): Open-ended questions sourced from people who are blind, aimed at testing the model’s OCR capabilities.
- •
ScienceQA (Lu et al., ): A multiple-choice science question dataset, with 86% of the images being non-natural, covering topics in natural science, social science, and language science.
- •
SEED-Bench (Li et al., ): A benchmark consisting of multiple-choice questions designed to assess both spatial and temporal understanding.

Table: Table 4: Benchmark performance of all 13 settings. C+D means feature combination of CLIP and DINOv2.

### A.4 All Settings AC Scores

We provide the AC scores of all 13 vision representation settings, as summarized in Table 5.

Table: Table 5: AC scores of all 13 settings. C+D means feature combination of CLIP and DINOv2.

### A.5 More Visualization of Correspondence

We provide additional visualizations of correspondence for four different vision representations: CLIP, SigLIP, DINOv2, and Stable Diffusion 1.5. Figures 6 and 7display pairs of source-target images for each of the four vision representations. In each pair, the left image is the source, and the right image is the target. The red dot on both images indicates the predicted key points using the vision representation. Ideally, these key points should correspond to the same semantic meaning. For example, a red dot on the ”left cat ear” in the source image should correspond to the ”left cat ear” in the target image. The green areas highlight regions of relatively high similarity with the source points.

In Figure 6, DINOv2 demonstrates superior correspondence for natural images compared to the other vision representations. It accurately matches small parts of the cat between the left and right images, whereas CLIP struggles to correctly identify and align features such as left, right, front, and back.

In Figure 7, , the CLIP family shows precise correspondence for text within images. For instance, when the source image points text like “LLaVA” or “VQAv2”, CLIP accurately matches all instances of the text in the target image. In contrast, other vision representations known for ”accurate correspondence” in computer vision, such as DINOv2 and Stable Diffusion, fail to provide the same level of accuracy when dealing with images containing text. This emphasizes a key distinction in selecting vision representations for computer vision tasks versus multimodal large language models (MLLMs).

Figure: Figure 6: Correspondence of natural images for different vision representations.
Refer to caption: x5.png

Figure: Figure 7: Correspondence of images with text for different vision representations.
Refer to caption: x6.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Typo in Table 1 'Number of Fintuning' -> 'Number of Finetuning'.
2. The experimental setting for the chosen LLM and projector is unclear. Could you give more information about the experimental setting?
soundness: 2
strengths: 1. The paper identifies the limitations in current methods for choosing vision representations, highlighting the need for a deeper understanding rather than an empirical, trial-based approach.
2. The proposal of the Law of Vision Representation is an innovative concept that attempts to quantify the relationship between cross-modal alignment, correspondence in vision representation, and MLLM performance through the AC score. 
3. The paper highlights an approach that achieves computational efficiency by reducing the need for finetuning with each vision representation change. This leads to a substantial reduction in computational cost.
summary: The paper presents the Law of Vision Representation in multimodal large language models (MLLMs), aiming to establish a strong relationship between vision representation quality, cross-modal alignment, and model performance. The authors introduce the AC score to quantify cross-modal Alignment and Correspondence, demonstrating a linear correlation between AC scores and MLLM performance across multiple benchmarks. The AC policy is also proposed as a method to efficiently select optimal vision representations, potentially reducing computational costs and energy consumption significantly compared to traditional methods.
weaknesses: While this work raises an important question in MLLM, there are certain weaknesses that prevent me from assigning a higher score.
1. The experiments in this paper are conducted on a specific MLLM setup, with a particular LLM and projector configuration, which raises doubts about whether the conclusions can be applied if the LLM or projector settings are changed. Given that the paper aims to establish a "law of vision representation," the current experiments seem too limited in scope to substantiate such a universal law.  Additional experiments should be carried out using various LLM sizes and projector designs to showcase wider applicability.
2. The choice of linear regression for modeling the relationship between the AC score and MLLM performance lacks sufficient explanation. It is unclear why a linear regression model is specifically used, and there is no discussion on whether other types of regression (e.g., polynomial, logarithmic) could potentially better capture this relationship. A justification for the linear assumption or a comparison with other regression models would strengthen the analysis. A comparison of various regression models (such as linear, polynomial, and logarithmic) should be performed, and their respective performance metrics should be reported to support the selection of linear regression.
3. Although the authors provide a theoretical explanation of the impact of the AC score on performance, the paper would be strengthened by a more rigorous proof or formal derivation of the mathematical relationship. Additionally, a more in-depth discussion on the design and application of AC scores, encompassing both explainable and theoretical aspects, should be included.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. In section 3.2, the explanation of key points and their prediction process is missing.
2. In Table 1, could you provide the top-1/2 predictions for a more comprehensive analysis?
3. Why does the 13 setting only require 12 finetuning runs to predict the optimal vision representations in a random setting?
soundness: 3
strengths: 1. The experiments are well-conducted and quite comprehensive.
2. The analyses of A, C, and AC scores, as well as their correlation with model performances are quite clear and valuable, highlighting the effectiveness of the proposed AC score.
summary: In this paper, the authors analyze the relationship between the cross alignment and correspondence (AC) score and MLLM performance. The AC score is a second-degree polynomial transformation of the alignment and correspondence score. Through a few training runs, the AC Policy can be determined and subsequently used to predict the optimal configurations of MLLMs. Thus, the AC Policy can reduce computational costs when deciding the optimal vision representations for MLLMs.
weaknesses: 1. How should we choose the number of $k'$ for Policy fitting? In Table 1, each benchmark has its own number of finetuning runs required to successfully predict the optimal vision representation. While 3 is the most common number across the benchmarks, there are still some outliers. It is difficult to determine the exact number of $k'$ for a benchmark without having the groundtruth. This suggests that the AC policy may not be very practical.
2. All the experiments are conducted under the condition of freezing the Vision Encoders. The conclusions may change when the Vision Encoders are unfrozen, which limits the generalizability of the AC Policy and these conclusions.
3. It is confusing that “if the vision representation ensures accurate correspondence, more visual details are considered even if not attended by the text token”. This statement implies that a high correspondence vision representation preserves visual details, which is ungrounded.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Please clarify the meaning and origin of the dimension 6 in  $X ∈ R^{k×6}$ in Policy Fitting. Is it related to the coefficients of the AC score?
2. Provide a pseudo code for the AC POLICY algorithm to enhance understanding.
3. Explain the choice of Recall@3 as the main metric and supplement with other metrics (like Recall@1, Recall@2, etc.).
4. Present the specific format (contains the weights and bias) of Policy Fitting and analyze the differences in fitting results across benchmarks. 
5. SigLIP is supposed to have better cross-modal alignment as an improved version of CLIP, but it shows a lower score in your setting. Please explain.
6. Does a higher AC score always mean better performance?
7. Besides the A and C scores, are there other factors that need to be considered?
soundness: 3
strengths: 1. AC policy consistently predicts the optimal vision representation with minimal resources.
2. Provide a insight of the choice and design of vision encoder for LVLM.
summary: This paper introduce a AC score that can be used to predict the performance of LVLM with specific vision encoders.
weaknesses: 1. The calculation of the A score by using the image feature of CLIP might not fully fulfill the motivation for quantifying the cross-modal alignment, as it assumes that the image-text features are perfectly aligned in CLIP.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: * How would using a stronger contrastive learning model, like BLIP, or a weaker model affect the results when calculating cross-modal alignment score?
* Does calculating the correspondence score require training an MLP to predict key points? If so, different vision representations may require different training hyper-parameters. Consequently, what approach should be taken to determine these optimal hyper-parameters? If the representation dimensions of the vision representation are not the same, then how can they be fairly compared?
* Why does the AC score not exhibit strong performance on the TextVQA and VizWiz benchmarks? Are there any significant differences between these two benchmarks and other benchmarks?
* How is the number of fine-tuning in Tab. 1 determined? Why does the number of fine-tuning vary across different benchmarks?
soundness: 3
strengths: * The motivation is sound. Building the relationship between vision representation and MLLMs' performance is very interesting and helpful.
* By identifying an optimal vision representation without extensive fine-tuning, the paper contributes to reducing computational costs and makes MLLM training more efficient.
summary: The authors propose the AC score, which combines cross-modal alignment and correspondence in vision representation. Then, they find a linear correlation between the AC score and MLLMs' performance. Extensive experiments across different vision representations validate the linear relationship between the AC score and MLLMs' performance, with an R-squared value of 95.72%.
weaknesses: * Cross-modal alignment score is the maximum cosine similarity between each pair of embedding vectors from the CLIP embedding and the target embedding. However, there may be bias in CLIP characterization. At the same time, CLIP is also one of the vision representations involved in the test, which results in the CLIP representation having an artificially high score on the cross-modal alignment score. 
* The authors only tested the proposed method on LLaVA v1.5. However, I am concerned that the vision representation law will change after changing the MLLM structure or the base LLMs. Therefore, I strongly recommend the authors to verify the generalization of the proposed method on more MLLM structures or the base LLMs.
* To fairly compare different vision representations, it is usually necessary to control the computation cost (e.g., flops) and model parameters of different vision representations within a similar level, but the proposed method does not take this into account. I suggest providing the computational cost and parameter counts for the different vision representations used and discussing how these factors might impact the AC scores.

Meta-review: Paper was reviewed by four expert reviewers and received 2 x marginally below the acceptance threshold and 2 x marginally above the acceptance threshold ratings. While generally the reviewers liked the premise of the work, they also pointed out a number of concerns, including (1) over-claiming of the results, as they only focus on a particular class of LLaVA models [JSYS]; (2) lack of clear exposition (pointed out by multiple reviewers, e.g., with respect to experimental setup and conclusions [JSYS] and underlying assumptions [AFzz]); and (3) lack of justification for the design beyond empirical evidence and verification that such design is adequate for border class of models, e.g., BLIP that has more complex transformer-based alignment and losses was not explored. Authors have provided comprehensive rebuttal addressing these issues and reviewers found some of the responses satisfactory. However, during the discussion, it was apparent that some concerns remained (e.g., concerned of limited scope that may prevent generalization of results to a broader range of MLLMs). 

AC has read the reviews, rebuttal and the discussion, as well as the paper itself. AC agrees that the overall idea of the work is intriguing, i.e., to be able to select visual representation / encoder that is best suited for an MLLM. However, AC also agrees that (i) paper, even after revisions, is difficult to read and understand (despite the underlying concept being relatively simple), and (ii) further investigation is need to identify the full extent of the findings and whether what is presented could really be called a "Law". Speaking from the formal definition, a "law" is a statement that describes or predicts a range of phenomena. Certainly the findings in the paper suggest that AC score predicts a phenomena, but fall short of the "range" implied. Authors are encouraged to revise the manuscript to improve quality and to conduct additional experiments with architectures that differ by form of projection and losses for resubmission to the next top-tier venue.

Decision: Reject