Title: Vision Function Layer in Multimodal LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Large Language Model

Vision Function Layer
Abstract: This study identifies that visual-related functional decoding is distributed across different decoder layers in Multimodal Large Language Models (MLLMs). Typically, each function, such as counting, grounding, or OCR recognition, narrows down to two or three layers, which we define as Vision Function Layers (VFL). Additionally, the depth and its order of different VFLs exhibits a consistent pattern across different MLLMs, which is well-aligned with human behaviors (e.g., recognition occurs first, followed by counting, and then grounding). These findings are derived from Visual Token Swapping, our novel analytical framework that modifies targeted KV cache entries to precisely elucidate layer-specific functions during decoding. Furthermore, these insights offer substantial utility in tailoring MLLMs for real-world downstream applications. For instance, when LoRA training is selectively applied to VFLs whose functions align with the training data, VFL-LoRA not only outperform full-LoRA but also prevent out-of-domain function forgetting. Moreover, by analyzing the performance differential on training data when particular VFLs are ablated, VFL-select automatically classifies data by function, enabling highly efficient data selection to directly bolster corresponding capabilities. Consequently, VFL-select surpasses human experts in data selection, and achieves 98% of full-data performance with only 20% of the original dataset. This study delivers deeper comprehension of MLLM visual processing, fostering the creation of more efficient, interpretable, and robust models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper presents an analysis study about the importance of different LLM layers for various VQA tasks using MLLMs.\n- The authors find that for different tasks, different layers are important and term those as the “Vision Function Layers” (VFL) for the Qwen-2.5-VL model.\n- The authors also try finetuning a qen2.5-vl model with LoRA applied to only the VFLs and find it to perform at par or better than applying LoRA to all layers.\n- Finally, they select samples with the highest value based on the probability based heuristic at the most important VFL for a given domain and show that using a subset of the dataset can result in fair performance\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- Good analysis which provides insights into what layers are important for different tasks.\n- The VFL select method could be useful for future methods.\n\n### Weaknesses\n\n- There is little reference to previous works in the experiment section in order to support their findings. For example, previous works (like FastV and [vision-of-vlm](https://vision-of-vlm.github.io/)) have found that the attention scores to visual tokens is the highest in the earlier layers, so why do changing the visual tokens there does not affect the answer? This finding is counter-intuitive and should be explained.\n- There is little effort to explain the findings of the paper about the presence of VFL layers, I would have liked the presence of some hypothesis.\n- Analysis is only done with qwen2.5-vl model which is very strong model in itself. I would like to see how the findings change with different MLLMs like LLaVA-1.5, Cambrian-1 (multi-encoder approach), Intern2.5-VL and also an MoE based model like Deepseek-VL2. Because this is an analysis driven paper, multiple models should be analyzed to make the findings well grounded and widely applicable.\n- How does VFL-select compare to methods using the probability of answers from the final LLM layer and also after the head? that is an important baseline which is missing. If the results are similar, VFL-select has lesser value with much analysis to be done for every different family of model.\n- The finding about MMMU and ScienceQA-Img is well known (as found in MM-Star, I would have liked to see that citation and also evaluation on that benchmark).\n\nQUESTIONS:\n- How does the findings change for video benchmarks? do we see the presence of VFL layers for video inputs too?\n- How does VFL transfer to encoder-free MLLMs like chameleon or EVE-2/EVE?\n- In the fig. 3 caption, the authors state: “contrasting with strong language priors for Distance and Relation.”, what is the basis of saying this? Relation and distance tasks do require good spatial reasoning abilities.\n\nLIMITATIONS:\nThe authors do not discuss any limitations but a few that come to mind:\n- No analysis for encoder free MLLMs or video benchmarks.\n- No analysis on different MLLM architectures (including MoEs).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates vision-related functional decoding in Multimodal Large Language Models (MLLMs), revealing that visual functions like counting, grounding, and OCR are localized within narrow decoder layer blocks, termed Vision Function Layers (VFLs). \n\nUsing a novel Visual Token Swapping framework, the study finds consistent VFL depth and order across MLLMs, aligning with human visual processing (recognition → counting → grounding → OCR).\n\nThe insights enable practical applications: VFL-LoRA, a targeted fine-tuning method, outperforms full LoRA by focusing on relevant VFLs.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The Visual Token Swapping and Dropping methods provide a systematic way to probe layer-specific visual functions in MLLMs, enabling precise localization of functions like OCR, counting, and recognition to narrow layer blocks. This framework addresses the \"black box\" problem of MLLMs and sets a new standard for interpretability research.\n2. VFL-LoRA reduces tunable parameters by ~50% while improving out-of-domain generalization compared to standard LoRA\n3. The study validates findings across 8+ benchmarks (e.g., ScienceQA, MMMU) and uses controlled image pairs to isolate single visual functions, ensuring robust results.\n\nQUESTIONS:\n1. Experiments focus on basic visual functions (such as recognition and counting), whereas there are numerous complex and diverse tasks in the real world. The VFL theory has not been validated in these real-world tasks.\n2. This paper proposes the visual token swapping and token dropping strategy to validate the VLF theory. However, many papers [1] drop vision tokens directly, according to the attention map or other strategies. Since these visual tokens can be directly discarded, can your operation still prove your theory?\n3. Please provide a more detailed explanation of how  visual token swapping and token dropping are specifically implemented.\n\n\n[1] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-PLay Acceleration for VLLM Inference\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the internal \"black box\" mechanisms of visual processing in MLLMs. The authors propose an analytical framework called Visual Token Swapping, which modifies targeted KV cache entries to precisely identify the function of specific layers during decoding. A core discovery is the existence of Vision Function Layers, which are narrow blocks of 2-3 layers where specific visual functions like counting or OCR are exclusively handled. These VFLs exhibit a consistent hierarchical order across different MLLMs. Leveraging these insights, the paper introduces practical applications for model optimization. VFL-LORA applies LoRA only to relevant VFLs, outperforming full LoRA in out-of-domain generalization while using fewer parameters. Furthermore, VFL-select, an automated data curation strategy, surpasses human experts and achieves 98% of full-data performance with only 20% of the original dataset. This work provides a deeper understanding of MLLM visual processing, enabling more efficient and interpretable models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is clear, structured, and effectively explains the experimental design and results. The writing is concise yet comprehensive.\n2. This paper provides a novel perspective on understanding MLLMs by introducing the concept of \"Vision Function Layers\". This perspective offers a novel approach to explain how MLLMs process visual inputs and presents new insights into how these models function internally. \n\nWeaknesses:\n1. Lack of technical contributions. While this paper introduces an interesting framework, it does not present sufficient technical contributions. The primary contribution lies in the analysis of vision functions across layers, rather than offering technical innovations that could push the field forward. This limits the technical impact of the paper.\n2. Limited scope of visual functions. The paper only considers four visual functions (recognition, counting, grounding, and OCR), which is too narrow. There are many other important visual reasoning tasks, such as object relationship inference and emotion recognition, that are not covered. This limited scope means the analysis may not fully capture the complexity of visual functions in MLLMs.\n3. The analysis is based solely on the contribution of individual layers, without considering the interaction between multiple layers. In reality, the functionality of different layers may interact and jointly influence the model's performance. By ignoring this interaction, the study may not provide a complete understanding of how visual features are processed and integrated across layers.\n4. The evaluation benchmarks used in the paper do not rely solely on single visual functions, which may undermine the accuracy of the results in assessing specific vision functions.\n\nQUESTIONS:\n1. The authors could incorporate a wider range of visual functions into the framework, as MLLMs exhibit complex visual capabilities and therefore require a more comprehensive analysis.\n\n2. The authors should design corresponding benchmarks to evaluate different visual functions, since current MLLM benchmarks usually rely on multiple visual capabilities.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a nice experiment design to assess where reasoning takes place in VLMs. The token dropping experiment is simple yet effective in isolating and pinpointing the layers with the most influence on a vision subtask. Empirically, the authors test their hypothesis on 4 different types of tasks - OCR, Grounding, Counting, and Recognition, and present insights about the particular layer where the task is \"performed\". Next, they show how fine-tuning only the pinpointed layers improves performance, reaffirming the insight. Finally, the authors utilize this insight into data selection for pre-training VLMs and achieve almost similar performance with ~20% of the data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Interpreting the inner workings of VLMs is an important topic with significant interest recently.\n2. The paper is very clear with a simple but intuitive methodology.\n3. Empirical results are exhaustive with a wide coverage of task types and datasets.\n4. Pre-training data selection is a smart application.\n\nWeakness:\n\n1. Only low-level tasks considered: Currently, the tasks are simple enough to be effective through direct prompting strategies. A lot of the insights presented are novel but do not take into account the high-dimensional vision-language interplay, which elicits the majority of VLM reasoning. For example, for each task, the prompt itself is not the most important aspect of the representations - OCR, recognition, etc., can be performed directly through probing vision tokens. It would be significantly impressive if these insights hold onto more difficult tasks like VQA (like on datasets like A-OKVQA), where VLMs truly shine.\n\n2. Building on this, I would be excited to see how prompting techniques change the visual functions across layers. For example, consider in-context learning, where $k$ exemplars are pre-appended to the prompt. Now there are $k$ times more visual tokens - how does the performance change then? How about prompts with longer context? \n\n3. The authors should also evaluate their approach on step-by-step prompting techniques like Chain-of-thought (CoT) reasoning, where a big visual reasoning task is broken down into simpler steps. If the author's findings confirm that each step of CoT activates similar layers, it will be a very deep contribution as multiple future works like PPO/GRPO can benefit from layer-specific policy learning.\n\nTypos:\n\nSection 2.1 \"Multi-Model Large Language Models\"\n\nQUESTIONS:\nRefer Weakness.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}