Law of Vision Representation in MLLMs

Authors: Shijia Yang, Bohan Zhai, Quanzeng You, Jianbo Yuan, Hongxia Yang, Chenfeng Xu
Primary area: foundation or frontier models, including LLMs
Keywords: Multimodality Large Language Models; Computer Vision; Vision Representation

Abstract:
We present the "Law of Vision Representation" in multimodal large language models (MLLMs). It reveals a strong correlation between the combination of cross-modal alignment, correspondence in vision representation, and MLLM performance. We quantify the two factors using the cross-modal Alignment and Correspondence score (AC score). Through extensive experiments involving thirteen different vision representation settings and evaluations across eight benchmarks, we find that the AC score is linearly correlated to model performance. By leveraging this relationship, we are able to identify and train the optimal vision representation only, which does not require finetuning the language model every time, resulting in a 99.7% reduction in computational cost.

Decision: reject
Reviewer scores: 5, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730552031894

thread_type:
official_review

<thread_object>
creator: reviewer_9bvh

<content>
summary:
The authors propose the AC score, which combines cross-modal alignment and correspondence in vision representation. Then, they find a linear correlation between the AC score and MLLMs' performance. Extensive experiments across different vision representations validate the linear relationship between the AC score and MLLMs' performance, with an R-squared value of 95.72%.

strengths:
* The motivation is sound. Building the relationship between vision representation and MLLMs' performance is very interesting and helpful.
* By identifying an optimal vision representation without extensive fine-tuning, the paper contributes to reducing computational costs and makes MLLM training more efficient.

weaknesses:
* Cross-modal alignment score is the maximum cosine similarity between each pair of embedding vectors from the CLIP embedding and the target embedding. However, there may be bias in CLIP characterization. At the same time, CLIP is also one of the vision representations involved in the test, which results in the CLIP representation having an artificially high score on the cross-modal alignment score. 
* The authors only tested the proposed method on LLaVA v1.5. However, I am concerned that the vision representation law will change after changing the MLLM structure or the base LLMs. Therefore, I strongly recommend the authors to verify the generalization of the proposed method on more MLLM structures or the base LLMs.
* To fairly compare different vision representations, it is usually necessary to control the computation cost (e.g., flops) and model parameters of different vision representations within a similar level, but the proposed method does not take this into account. I suggest providing the computational cost and parameter counts for the different vision representations used and discussing how these factors might impact the AC scores.

questions:
* How would using a stronger contrastive learning model, like BLIP, or a weaker model affect the results when calculating cross-modal alignment score?
* Does calculating the correspondence score require training an MLP to predict key points? If so, different vision representations may require different training hyper-parameters. Consequently, what approach should be taken to determine these optimal hyper-parameters? If the representation dimensions of the vision representation are not the same, then how can they be fairly compared?
* Why does the AC score not exhibit strong performance on the TextVQA and VizWiz benchmarks? Are there any significant differences between these two benchmarks and other benchmarks?
* How is the number of fine-tuning in Tab. 1 determined? Why does the number of fine-tuning vary across different benchmarks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730613236976

thread_type:
official_review

<thread_object>
creator: reviewer_nwyo

<content>
summary:
This paper introduce a AC score that can be used to predict the performance of LVLM with specific vision encoders.

strengths:
1. AC policy consistently predicts the optimal vision representation with minimal resources.
2. Provide a insight of the choice and design of vision encoder for LVLM.

weaknesses:
1. The calculation of the A score by using the image feature of CLIP might not fully fulfill the motivation for quantifying the cross-modal alignment, as it assumes that the image-text features are perfectly aligned in CLIP.

questions:
1. Please clarify the meaning and origin of the dimension 6 in  $X ∈ R^{k×6}$ in Policy Fitting. Is it related to the coefficients of the AC score?
2. Provide a pseudo code for the AC POLICY algorithm to enhance understanding.
3. Explain the choice of Recall@3 as the main metric and supplement with other metrics (like Recall@1, Recall@2, etc.).
4. Present the specific format (contains the weights and bias) of Policy Fitting and analyze the differences in fitting results across benchmarks. 
5. SigLIP is supposed to have better cross-modal alignment as an improved version of CLIP, but it shows a lower score in your setting. Please explain.
6. Does a higher AC score always mean better performance?
7. Besides the A and C scores, are there other factors that need to be considered?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730636294720

thread_type:
official_review

<thread_object>
creator: reviewer_afzz

<content>
summary:
In this paper, the authors analyze the relationship between the cross alignment and correspondence (AC) score and MLLM performance. The AC score is a second-degree polynomial transformation of the alignment and correspondence score. Through a few training runs, the AC Policy can be determined and subsequently used to predict the optimal configurations of MLLMs. Thus, the AC Policy can reduce computational costs when deciding the optimal vision representations for MLLMs.

strengths:
1. The experiments are well-conducted and quite comprehensive.
2. The analyses of A, C, and AC scores, as well as their correlation with model performances are quite clear and valuable, highlighting the effectiveness of the proposed AC score.

weaknesses:
1. How should we choose the number of $k'$ for Policy fitting? In Table 1, each benchmark has its own number of finetuning runs required to successfully predict the optimal vision representation. While 3 is the most common number across the benchmarks, there are still some outliers. It is difficult to determine the exact number of $k'$ for a benchmark without having the groundtruth. This suggests that the AC policy may not be very practical.
2. All the experiments are conducted under the condition of freezing the Vision Encoders. The conclusions may change when the Vision Encoders are unfrozen, which limits the generalizability of the AC Policy and these conclusions.
3. It is confusing that “if the vision representation ensures accurate correspondence, more visual details are considered even if not attended by the text token”. This statement implies that a high correspondence vision representation preserves visual details, which is ungrounded.

questions:
1. In section 3.2, the explanation of key points and their prediction process is missing.
2. In Table 1, could you provide the top-1/2 predictions for a more comprehensive analysis?
3. Why does the 13 setting only require 12 finetuning runs to predict the optimal vision representations in a random setting?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730990684738

thread_type:
official_review

<thread_object>
creator: reviewer_jsys

<content>
summary:
The paper presents the Law of Vision Representation in multimodal large language models (MLLMs), aiming to establish a strong relationship between vision representation quality, cross-modal alignment, and model performance. The authors introduce the AC score to quantify cross-modal Alignment and Correspondence, demonstrating a linear correlation between AC scores and MLLM performance across multiple benchmarks. The AC policy is also proposed as a method to efficiently select optimal vision representations, potentially reducing computational costs and energy consumption significantly compared to traditional methods.

strengths:
1. The paper identifies the limitations in current methods for choosing vision representations, highlighting the need for a deeper understanding rather than an empirical, trial-based approach.
2. The proposal of the Law of Vision Representation is an innovative concept that attempts to quantify the relationship between cross-modal alignment, correspondence in vision representation, and MLLM performance through the AC score. 
3. The paper highlights an approach that achieves computational efficiency by reducing the need for finetuning with each vision representation change. This leads to a substantial reduction in computational cost.

weaknesses:
While this work raises an important question in MLLM, there are certain weaknesses that prevent me from assigning a higher score.
1. The experiments in this paper are conducted on a specific MLLM setup, with a particular LLM and projector configuration, which raises doubts about whether the conclusions can be applied if the LLM or projector settings are changed. Given that the paper aims to establish a "law of vision representation," the current experiments seem too limited in scope to substantiate such a universal law.  Additional experiments should be carried out using various LLM sizes and projector designs to showcase wider applicability.
2. The choice of linear regression for modeling the relationship between the AC score and MLLM performance lacks sufficient explanation. It is unclear why a linear regression model is specifically used, and there is no discussion on whether other types of regression (e.g., polynomial, logarithmic) could potentially better capture this relationship. A justification for the linear assumption or a comparison with other regression models would strengthen the analysis. A comparison of various regression models (such as linear, polynomial, and logarithmic) should be performed, and their respective performance metrics should be reported to support the selection of linear regression.
3. Although the authors provide a theoretical explanation of the impact of the AC score on performance, the paper would be strengthened by a more rigorous proof or formal derivation of the mathematical relationship. Additionally, a more in-depth discussion on the design and application of AC scores, encompassing both explainable and theoretical aspects, should be included.

questions:
1. Typo in Table 1 'Number of Fintuning' -> 'Number of Finetuning'.
2. The experimental setting for the chosen LLM and projector is unclear. Could you give more information about the experimental setting?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731927292011

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Reply to All Reviewers

comment:
We carefully read and sincerely appreciate the reviewers’ comprehensive suggestions. We are really grateful to see the reviewers find our work as an **innovative concept** (JSYS) and **well-motivated** (9bVh), almost all of the reviewers (JSYS, Nwyo, 9bVh) agree with the **contribution of computation reduction for finetuning MLLM**. Specifically, reviewer (JSYS) **agree with the need of exploring and deepen understanding of current methods for choosing vision representation**. We also appreciate reviewer (AFzz) appreciate our **extensive experiments**.

In this work, we reveals a strong correlation between the combination of cross-modal alignment, correspondence in vision representation, and MLLM performance, we quantify these factors as cross-modal Alignment and Correspondence score (AC score), and through extensive experiments, we find the relationship between AC score and model performance.

We encourage active and multi-round discussions, so we will have the final updated PDF at the end of discussion phase, which will be merging all the reviewers’ suggestions. In this round, we added pseudo code of A, C score calculation, as well as AC Policy in Appendix Section 6 in blue text. 

**Several common misunderstandings that we want to highlighted here:**

> *1. How does the law behave on settings other than LLaVA 1.5, such as different LLMs? (Reviewer JSYS, 9bVh)*

By adding extra experiments on additional type and size of LLM, we show that the fitting $R^2$ remains strong, and in some cases even higher, when using different LLM types and sizes. The variation in $R^2$ falls within a reasonable range, indicating that the effect of LLM and vision representation compatibility, if it exists, is negligible compared to the influence of the A and C factors. These results demonstrate that the Law of Vision Representation relationship holds across different LLMs.

> *2. Concern on CLIP embedding as reference embedding in A score. (Reviewer Nwyo, 9bVh)*

In our paper, we aim to elucidate the relationship between performance, alignment, and correspondence. We employ "A score" as an initial approach to quantify the concept of alignment in the field of MLLM. A score is designed to be flexible and can be adjusted based on individual preferences, such as the belief that another model has the highest cross-modal alignment or to minimize fitting error. 

Addressing the concerns raised by Reviewers Nwyo and 9bVh, Nwyo think SigLIP is the better model, and 9bVh think BLIP should be a good try. We acknowledge that different researchers may have varying preferences for the strongest contrastive vision encoder. Nevertheless, many papers published post-CLIP have compared their results with CLIP across numerous benchmarks, thereby demonstrating the widespread acceptance of CLIP. 

It's worth noting that most vision representations are not specifically designed to optimize the concept of alignment, hence there's no consensus on which model performs the best in this regard. Part of our proposal of A score seeks to establish a baseline method for quantifying cross-modal alignment, making CLIP an apt choice widely accepted by researchers.

We would like to compare additional settings, but as mentioned in our paper, the experiments are costly and time-consuming. We have included the SigLIP experiments in our response, which demonstrate that SigLIP has a poorer fitting result compared to CLIP. Therefore, we assert that selecting CLIP for our settings and experiments remains the optimal choice.

> *3. Why not evaluate AC Policy Recall@1 and Recall@2. Why Recall@3? (Reviewer AFzz, Nwyo)*

There maybe confusions on why Recall@3 is used. As we showed in the response, we provide the MLLM performance percentage difference between the Top-1 and Top-2 vision representations to illustrate that, in most cases, the performance differences are minimal and their order can be attributed to fluctuations. We also added the Recall@1 and Recall@2 in the response. We observe that, given the same computation budget where only 7 MLLM finetunings, using Recall@1 provides a significant 20% improvement over random trials in the chance of finding the optimal vision representation. However, using Recall@3 as a metric further optimizes this chance, demonstrating its practicality. This observation implies that, rather than assuming an absolute "optimal vision representation", we should acknowledge that each finetuning and inference process is subject to fluctuations, making "optimal" predictions inherently challenging. Employing Recall@3 accounts for these fluctuations, providing a more robust and practical approach without increasing computation costs.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Paper was reviewed by four expert reviewers and received 2 x marginally below the acceptance threshold and 2 x marginally above the acceptance threshold ratings. While generally the reviewers liked the premise of the work, they also pointed out a number of concerns, including (1) over-claiming of the results, as they only focus on a particular class of LLaVA models [JSYS]; (2) lack of clear exposition (pointed out by multiple reviewers, e.g., with respect to experimental setup and conclusions [JSYS] and underlying assumptions [AFzz]); and (3) lack of justification for the design beyond empirical evidence and verification that such design is adequate for border class of models, e.g., BLIP that has more complex transformer-based alignment and losses was not explored. Authors have provided comprehensive rebuttal addressing these issues and reviewers found some of the responses satisfactory. However, during the discussion, it was apparent that some concerns remained (e.g., concerned of limited scope that may prevent generalization of results to a broader range of MLLMs). 

AC has read the reviews, rebuttal and the discussion, as well as the paper itself. AC agrees that the overall idea of the work is intriguing, i.e., to be able to select visual representation / encoder that is best suited for an MLLM. However, AC also agrees that (i) paper, even after revisions, is difficult to read and understand (despite the underlying concept being relatively simple), and (ii) further investigation is need to identify the full extent of the findings and whether what is presented could really be called a "Law". Speaking from the formal definition, a "law" is a statement that describes or predicts a range of phenomena. Certainly the findings in the paper suggest that AC score predicts a phenomena, but fall short of the "range" implied. Authors are encouraged to revise the manuscript to improve quality and to conduct additional experiments with architectures that differ by form of projection and losses for resubmission to the next top-tier venue.