Intriguing Properties of Large Language and Vision Models

Authors: Young-Jun Lee, Byungsoo Ko, Han-Gyu Kim, Yechan Hwang, Ho-Jin Choi
Primary area: foundation or frontier models, including LLMs
Keywords: LLVM;clustering;projector;investigation

Abstract:
Recently, large language and vision models (LLVMs) have received significant attention and development efforts due to their remarkable generalization performance across a wide range of tasks requiring perception and cognitive abilities. A key factor behind their success is their simple architecture, which consists of a vision encoder, a projector, and a large language model (LLM). Despite their achievements in advanced reasoning tasks, their performance on fundamental perception-related tasks (\eg MMVP) remains surprisingly low. This discrepancy raises the question of how LLVMs truly perceive images and exploit the advantages of the vision encoder. To address this, we systematically investigate this question regarding several aspects: \textit{permutation invariance}, \textit{robustness}, \textit{synthetic data}, \textit{alignment preserving} and \textit{importance}, by evaluating the most common LLVM's families (\ie LLaVA) across 13 evaluation benchmarks. Our extensive experiments reveal several intriguing properties of current LLVMs: (1) they internally process the image in a global manner, even whenthe order of visual patch sequences is randomly permuted; (2) they are sometimes able to solve math problems without fully perceiving detailed numerical information; (3) the cross-modal alignment is overfitted to complex reasoning tasks, thereby, causing them to lose some of the original perceptual capabilities of their vision encoder; (4) the representation space in the lower layers ($<25\%$) plays a crucial role in determining performance and enhancing visual understanding. Lastly, based on the above observations, we suggest potential future directions for building better LLVMs and constructing more challenging evaluation benchmarks.

Decision: reject
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730548077153

thread_type:
official_review

<thread_object>
creator: reviewer_tcxa

<content>
summary:
This paper presents a detailed investigation of how vision is perceived in large language and vision models. Comprehensive experiments reveal the properties of LLVMs from multiple aspects including permutation invariance, robustness, math reasoning, alignment preserving and importance.

strengths:
This paper elaborates on a critical problem in multimodal LLMs that how vision is perceived. It is a profound question which could benefit the future development of MLLMs.

weaknesses:
The examples to be investigated in this paper are mainly from the LLaVA family whose vision representation is continuous. An analysis on discrete vision representation would be interesting.

questions:
Please see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730596707952

thread_type:
official_review

<thread_object>
creator: reviewer_epc1

<content>
summary:
This paper presents an empirical study of large vision language models with a focus on the llava model families using perception and reasoning tasks across various benchmarks. The authors revealed several findings of LVMs including image patch permutation invariance, overfitted cross-modal alignment, importance of lower block layers, etc..

strengths:
The paper comprehensively evaluated several VLMs across various perception and reasoning benchmark to better understand their behavior and performance. These empirical findings could be valuable providing insights for model developments.

weaknesses:
1. While the paper shows some good empirical studies of the LVM performance, they are loosely connected to the suggested future directions. This makes the contribution of the paper less clear. For example, the authors suggested "deeply consider innovative model architectures" in Sec 4 for enhancing cross modal alignment, yet it's not clear how this related to the empirical findings discussed in the previous sections and what necessary enhancements are entailed from their analysis.

2. The authors should consider toning down the rhetoric of the study from general LVMs to llava model families. Reported results do not show whether these findings would generalize to other LVMs as they are not evaluated yet.

3. Some of the findings are more or less expected, e.g., some degree of robustness to occlusion due to the robustness from image encoder. The conclusion from these findings are not clear. I suggest the authors append more focused discussions on a few key findings and tie them to any future directions or proposed improvements.

questions:
Please see weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730596896429

thread_type:
official_review

<thread_object>
creator: reviewer_v3xh

<content>
summary:
The paper analyzes some properties of LLVMs with respect to permutation invariance, mathematical reasoning, robustness to perturbations, and other aspects. The problem tackled is interesting and relevant to the VLM community. The authors analyze a wide variety of VLMs properties by focusing on the LLAVA family and a diverse set of classification/instruction following benchmarks.

strengths:
* The paper addresses an important and timely problem that is of interest to the VLM community.
* Some observations made by the authors are insightful and have the potential to contribute to the understanding of VLMs.
* The set of analysis is diverse and has the potential to inform the  development of a new set of architectures/benchmark.
* Detailed set of benchmark datasets in the Appendix.

weaknesses:
* Many conclusions are not clearly backed up by evidence or rigorous analysis, which undermines their validity (I've given some examples in the questions section)
* Some observations appear to be cherry-picked (by dataset or sample?), which raises concerns about the representativeness and generalizability of the results. For instance, the authors want to evaluate the LLaVA family of models but some results are given on a single model and data sample while the conclusion is general (Figure 1 and Section 3.7 for instance, i've given more specific questions in the next section)
* The choice of datasets, metrics, and evaluation methods for different parts of the analysis is often unclear or unjustified, making it difficult to assess the generalizability of the findings.

questions:
* The article focus on the LLaVA family. Could the authors  explain their rationale for focusing on the LLaVA family and discuss potential limitations in generalizing these results to other LVLM architectures?
* In Section 3.3, how is the grouping done to quantify local information in patches? How are neighbours defined, and what is the impact of this definition on the results?
* The explanation of backpropagation and loss signal coming from the assistant in line 234 is unclear. Can the authors provide more details or clarify the average performance drop, which seems to be benchmark-dependent?
* In Table 2, what setting is used to measure the frequency of outputting 1 (synthetic or original)? Is there a difference between these cases? Can the authors also report the frequency of outputting "no" and precision/recall for yes/no questions?
* Evaluating the loss of  cross-modal alignment is interesting. However the evaluation method used in Table 3 is unclear and may not be fair compared to a contrastive approach. Could the authors give more details about their evaluation setup (prompt used?) as it is known that prompt optimization impacts even zero-shot CLIP models classification performance.  Can the authors consider alternative approaches, such as log-likelihood comparison of description prompts (e.g., "Describe the image: This is an image of a [class label]")? As it is done in captioning models evaluation for instance?
* How is the 1k subset obtained in Table 3?
* The section on shared world concepts is hard to follow. What metric is used to measure alignment, and why was the DOCCI dataset chosen? How is alignment measured with long captions given CLIP's 77-tokens context?
* Are the local scores in Figure 8 averaged over the whole dataset or just one example? Same question for Figure 1 ? 
* Can the authors comment on the choices of datasets for different parts of the analysis (e.g., MMVet for localization of information, MMStar for layers importance)? Are these results model-, dataset-, or sample-specific, and how do they impact the generalizability of the findings? I know this question is quite general but the justification is not clear to me and the drawn conclusions seem to be a general so I would except such visualizations to be performed on a subset of the different datasets.
* The conclusion drawn from Figure 9 lacks a clear explanation of how the importance score is obtained. Can the authors provide more details on the method used (e.g., noised input or weights, which proportion)?

* In general it would help if the authors could : 
1. Provide explicit justifications for their dataset choices in each analysis section
2. Discuss potential limitations in generalizing results from specific datasets to broader conclusions
3. Consider extending their analysis to multiple datasets where feasible, or explain why this might not be possible or necessary
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620399273

thread_type:
official_review

<thread_object>
creator: reviewer_ldvk

<content>
summary:
This paper looks at how Large Language and Vision Models (LLVMs) like LLaVA handle different tasks. It finds that LLVMs don’t rely much on image order, sometimes solve math without exact details, and lose some visual skills when fine-tuned for language tasks. The study also notes that the first layers process visuals best. The authors suggest building tougher benchmarks to better test and improve these models.

strengths:
This paper is useful because it digs into how large language and vision models (LLVMs) really work with visual information. It shows that LLVMs are flexible, able to handle scrambled image pieces and solve math problems without all the visual details. It also highlights that when LLVMs are tuned for complex reasoning, they lose some basic visual skills. These findings can help make LLVMs better by balancing complex reasoning with simpler perception tasks, potentially guiding the creation of new, stronger models.

weaknesses:
The paper would benefit from more precise and carefully scoped conclusions. While the experiments provide interesting observations, the claims drawn from them are often overly broad. For example: 1) The claim about permutation invariance is based on VQA tasks, but this alone cannot support a general conclusion about LLVMs' visual processing capabilities.
2) The benchmarks used don't adequately test basic visual skills to support such sweeping statements about visual understanding

Specificity Needed:
Each conclusion needs clear boundaries about when it applies and when it doesn't.
The paper makes approximately 10 major claims, but lacks sufficient context about their limitations and specific application scenarios.
More precise scoping of these findings would make them more actionable for future research.
Currently, the broad generalizations make it difficult for other researchers to build upon these results effectively.

Minor points:
Figure 5's text is too small to read effectively, hampering understanding of key results.

questions:
Besides the weakness, to improve, I suggest the authors:

1. Clearly specify the conditions under which each conclusion holds
2. Acknowledge the limitations of their experimental setup
3. Provide more nuanced interpretations of their results
4. Consider additional experiments to support broader claims
</content>
</thread_object>

</thread>
<thread>
creation date:
1732422129109

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response for Reviewers (1/9)

comment:
We appreciate all reviewers' thoughtful feedback regarding the **broadness of our claims and the need to present limitations and future directions for each finding.** We acknowledge that many conclusions from our observations lacked the clarity needed to effectively guide future research. To address this, we have revisited each observation and provided more specific, well-defined conclusions. We have outlined the experimental limitations of each observation and highlighted practical application scenarios where these findings may be most beneficial. We have also proposed potential future research directions that build upon these observations. These clarifications and refinements are presented below (and shown in detail in Appendix A):

---

### **Findings 1: Each visual patch token encapsulates localized visual information.**

In this finding, we observe that each visual patch token from the projector captures a localized understanding of the patch area corresponding to its position in the image.

- **Limitations**: In our experiments, this observation was investigated solely on the LLaVA-1.5 model using two evaluation benchmarks: MMStar and MME. Both benchmarks primarily assess the integrated capabilities of LLVMs (e.g., perception and reasoning). This limited scope means that we cannot guarantee the generalizability of our findings, despite using widely recognized evaluation benchmarks and a popular LLVM model. Furthermore, more diverse and challenging benchmarks, as well as advanced and highly intelligent LLVMs, were not considered in our study. To reduce computational complexity, we simplified the evaluation by grouping neighboring visual patch tokens. While this approach streamlined the analysis, it does not verify the localized information within each visual patch token in detail. To address this limitation.
- **Future Directions**: We verify that each visual patch token encodes varying degrees of locality within the given image. Notably, visual patch tokens corresponding to the central part of the image tend to contain higher visual information compared to those at the edges, leading to an information discrepancy. Building on this observation, **selectively focusing on the more informative localized visual patch tokens could reduce the overall number of tokens, enhancing the efficiency of both training and inference.** Typically, increasing the number of visual tokens enhances visual perception capabilities. However, it also increases the final maximum sequence length of LLMs. For example, in the Qwen2-VL series, the number of visual patch tokens often exceeds 1,000, which can cause GPU Out-of-Memory (OOM) issues depending on the available computational resources. **By reducing the number of visual patch tokens based on the observed information discrepancy, we can mitigate such issues while maintaining performance.** For this, we can consider **adaptively reducing visual tokens based on the input image and text using Gumbel softmax, allowing the model to determine which visual tokens are important based on the given inputs, similar to DynamicViT [28].** Additionally, with fewer visual patch tokens, it becomes feasible to implement an interleaved format. This format is particularly practical for real-world applications, such as multi-modal conversations, where images can appear at arbitrary positions within the dialogue context. This future direction aligns with trends observed in prior studies, such as **Flamingo**, **HoneyBee**, and **BLIP-3**. While these studies have explored and adopted similar approaches, our work provides the first concrete evidence supporting this strategy. As such, our findings offer meaningful insights and lay a strong foundation for future research in this area.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732456190224

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response for Reviewers

comment:
We sincerely thank all reviewers for their valuable feedback, which has been instrumental in improving the quality of our paper. Specifically, the comments on clarifying our findings and discussing the generalizability of our results from both model and dataset perspectives have been particularly helpful. Based on the reviewers' feedback, we have made the following updates to the paper:

### Main Paper

- **Additional Interpretations**: We have added further explanations and interpretations, highlighted in blue for clarity.
    - We added more detailed results in Section 3.3 regarding benchmark-dependent "permutation invariance," providing a more precise analysis.
    - We added an explanation for why LLVMs are sensitive to spatial structures in text-rich benchmarks (e.g., MathVista, AI2D) in Section 3.3.
    - We clarify in Table 2 whether "Freq." belongs to "Orig." or "Syn."
    - We conducted five experimental runs of the zero-shot image classification task and updated the results in Table 3.
- **Adjustments for Page Limit**: To comply with the 10-page limit, we resized Figures and Tables where necessary and moved the related work regarding "model-stitching" to Appendix F.
    - We reduced the size of Figure 4, Table 3, and Figure 7.

### Appendix

- **Limitations and Further Discussions**: In Appendix A, we added the limitations of our experiments and included further discussions for each of our findings.
- **Platonic Representation Hypothesis**: Appendix B now includes a detailed explanation of the "platonic representation hypothesis," how cross-modal alignment is measured using our alignment metric, and the motivation for selecting the DOCCI dataset.
- **Importance Score**: In Appendix C, we provide a comprehensive explanation of the "importance score," including its calculation methodology.
- **Experimental Setup**: Appendix D includes detailed descriptions of our experimental setup and specific conditions necessary to derive our findings.
- **Generalization of Findings**: In Appendix E, we present additional experimental results to strengthen the generalizability of our findings. These include evaluations of Qwen2-VL and Fuyu-8B models, experiments using image-captioning datasets, and detailed analyses of our observations.
- **More Examples:** In Appendices G, H, and I, we present additional examples of synthesized, shuffled, and occluded images used in our experiments.

We deeply appreciate the reviewers' insightful suggestions. While we acknowledge that extending our observations to additional LLVMs would further enhance the rigor of our study, we focused on models such as the LLaVA family and Qwen2-VL, which are currently recognized as significant LLVMs. We believe that our observations on these models are meaningful and impactful. Moving forward, we are committed to expanding our research, and—if we secure sufficient resources—plan to include additional experiments on specific benchmarks that require access to OpenAI API or other large-scale resources.

Once again, we thank all reviewers for their constructive feedback and for helping us refine our work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733044883197

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Friendly Reminder

comment:
Dear Reviewers, this is a friendly reminder that the reviewer discussion period concludes on December 2. We would be sincerely grateful to receive your feedback at your earliest convenience. Your insights are invaluable to us. If you need any additional information or have further questions, please do not hesitate to reach out. Thank you very much for your time and thoughtful consideration!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper considers a critical problem of how LLVMs perceive and process visual information and aims to provide a comprehensive study relevant to the community through a diverse set of benchmarks and experiments. In principle, findings could inform the future models and benchmarks. While the paper provides valuable insights, the reviewers find that conclusions were overgeneralized, lacked methodological justification, and the implications were limited to a limited set of models. While the rebuttal phase brought improvements and strengthened the work, the AC feels that a major revision is required to incorporate all these findings.