Vision Function Layer in Multimodal LLMs
Authors: Cheng Shi
Yizhou Yu
Sibei Yang
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Model
Vision Function Layer
Primary area: deep_learning

Abstract: This study identifies that visual-related functional decoding is distributed across different decoder layers in Multimodal Large Language Models (MLLMs). Typically, each function, such as counting, grounding, or OCR recognition, narrows down to two or three layers, which we define as Vision Function Layers (VFL). Additionally, the depth and its order of different VFLs exhibits a consistent pattern across different MLLMs, which is well-aligned with human behaviors (e.g., recognition occurs first, followed by counting, and then grounding). These findings are derived from Visual Token Swapping, our novel analytical framework that modifies targeted KV cache entries to precisely elucidate layer-specific functions during decoding. Furthermore, these insights offer substantial utility in tailoring MLLMs for real-world downstream applications. For instance, when LoRA training is selectively applied to VFLs whose functions align with the training data, VFL-LoRA not only outperform full-LoRA but also prevent out-of-domain function forgetting. Moreover, by analyzing the performance differential on training data when particular VFLs are ablated, VFL-select automatically classifies data by function, enabling highly efficient data selection to directly bolster corresponding capabilities. Consequently, VFL-select surpasses human experts in data selection, and achieves 98% of full-data performance with only 20% of the original dataset. This study delivers deeper comprehension of MLLM visual processing, fostering the creation of more efficient, interpretable, and robust models.

Vision Function Layer in Multimodal LLMs
Cheng Shi 2,
Shi Yizhou Yu², Sibei Yanglt
Sun Yat-sen University, SScooo Computing and Data Science, The University of Hong Kong shicheng2025ecommect hku hk, yizhouy@acm. org, yangsb3email sysu. edu.
httpa:/ com com/ChengShiett Saangghieat/Visiin Ceciaion-Funntti Function-Layy
Abstract
This study identifies that visual-related functional decoding is distributed across different decoder layers in Multimodal Large Language Models (MLLMs). Typi- cally, each function, such as counting, grounding, or OCR recognition, narrows down two or three layers, which we define as Vision Function Layers (VFL). Additionally, the depth and its order of different VFLs exhibits a consistent pattern across different MI..Ms, which well-aligned with human behaviors (e.g., recog- nition occurs first, followed by counting, and then grounding). These findings are derived from Visual Token Swapping, our novel analytical framework that modifies targeted KV cache entries to precisely elucidate layer-specific functions during decoding. Furthermore, these insights offer substantial utility in tailoring MLLMs for real-world downstream applications. For instance, when LoRA training is selectively applied to VFLs whose functions align with the training data, VFL- LoRA not only outperform full-LoRA but also prevent out-of-domain function forgetting. Morcover, by analyzing the performance differential on training data when particular VFLs ablated, VFL-select automatically classifies data by function, enabling highly efficient data selection to directly bolster corresponding capabilities. Consequently, VFL-select surpasses human experts in data selection, and achieves 98% full-data performance with only 20% of the original dataset. This study delivers deeper comprehension of MILM visual processing, fostering the creation of more efficient, interpretable, and robust models.
Introduction
Large language models (2, 3, 8, 17, 47, 55, 59, 60), built on deep transformer layers [6 [. have become the dominant paradigm in natural language processing, demonstrating remarkable versatility and human-level performance across diverse tasks [16, 45, 53, 56, 69]. Recent studies [11, 38, 44] further reveal that frontier LLMs develop hierarchical internal structures and problem-solving strategies analogous to human cognition, in which simple features are combined into complex representations and tasks are decomposed into sub-components. Understanding these mechanisms has become central to improving interpretability [20, 32], efficiency [11, 76], and driving architectural advances like Mixture of Experts [1, 15, 66].
Extending LLMs, Multimodal Large Language Models (MLLMs) [3, 6, 7, 23, 64] achieve joint text-vision understanding and reasoning by processing integrated visual and textual inputs. Through supervised fine-tuning on vision instruction data, these MLLMs have progressed beyond simple image captioning to address diverse tasks requiring visual perception and understanding. However, despite these remarkable advancements in visual understanding capabilities, the internal workings of these MLLMs- particularly how they process and reason with vision tokens remain largely
* WWok done during Cheng's intersship at SYSU.
Coorespponing author is Sibei Yang.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

OCR--unntioon
Count-Function
Vision Function Layer
Please Describe this image in detaill
TEXTI
TEXT2
Layer K
Qwen2.5.VL. VL
Pairnd Image
Language
Vision
00000
A The image features cat holding white sign with the "Language" written on it. The cat appears
Tolion
Text
What text can you read in the white paper?
How many cale are there in the image??
Imagel
Tolkan
24 OCR-Function Layer
Please Describe this
Vision Token Swapping
f In the image, a cat holdingg white sign displaying word vision. The
Recog- Fuccog-Funntion
Crounddin-Functton
image detail.
"No Image"
Image1
Image2
Vision
Tokan
000000
K - 18 Recoo--Functinn Function Layer
(b The picture features dog that holding white sign with the word "Language" on The
Text
Token
there imagee?
Imsaee2
MLLM Layer
Probing specific visual functions using minimally different image pairs and targeted questions. Right: The token swapping mechanism, where vision tokens from a source image replace those of a target image at a specific layer within the MLLM's KV cache during decoding. The example with Qwen2.5-VL demonstrates how swapping at different functional layers (Layer=24 for OCR, Layer=18 for Recognition) directly alters the model's output to reflect the swapped content.
Figure 1: Overview of the Vision Token Swapping Framework. Left:
unclear, often characterized as a "black box." The heightened challenge in understanding the internal mechanisms of MLLMs, compared to their text-only counterparts, stems primarily from two aspects:
First, the diversity of vision-language tasks [22, 30, 35, 36, 68] tackled by MLLMs demands mastery of a wide range of fundamental visual functions, where we define visual function as a distinct perceptual capability essential for solving a specific category of vision tasks such as object recognition, counting, text reading, or spatial reasoning- each reflecting a particular type of visual understanding. These atomic visual functions serve as building blocks for more complex and integrative vision-language tasks, where multiple perceptual capabilities are jointly required. For instance, solving math problem in a diagram may require reading handwritten equations (OCR), counting geometric elements, and reasoning about spatial relationships each engaging different visual functions in concert. Unlike text-only models that operate in uniform linguistic space [28, 31J. MLLMs must learn to activate and combine heterogencous visual functions to interpret diverse visual inputs. This multi-faceted requirement adds substantial complexity their internal mechanisms, making it difficult to pinpoint how specific visual functions are represented, combined, and aligned with textual reasoning.
Second, unlike text-only LLMs, which have largely adopted standardized architectures, MLLMS still present diverse designs, especially in their visual branches. They employ different types of visual encoders [10, 39, 43, 63] and various connector modules [6, 25, 29, 58] to integrate vision tokens with text tokens. This architectural variability further complicates the understanding of their internal mechanisms - akin to assuming that different species, each sensing the world through distinct systems, would nonetheless process these signals through identical neural pathways [54].
While recent MILM studies [4, 9, 37, 71] have explored token importance and cross-modal interac- tions, they largely overlook how diverse visual functions are internally represented and coordinated. key challenge lies in the lack of diagnostic frameworks that isolate individual functions, as most ceneral-purpose tasks require multiple abilities simultaneously. This limitation hinders targeted analysis and leads to only coarse conclusions (e.g., shallow layers extract visual features while deeper perform reasoning), leaving core questions about MILMss internal visual mechamisn unanswered.
To address this challenge, we examine the layer-wise functional roles of vision tokens within the LLM backbone of MLLMs, aiming to understand how different layers contribute to the realization of specific visual functions. We propose a two-level, step-by-step evaluation framework: single- function evaluation via visual tokens swapping and multi-function evaluation via visual tokens drop. Our fngle-function evaluation assesses the functional roles of visual representations across layers by swapping visual tokens between pairs of images differing solely in one visual function. This controlled perturbation reveals how layer-specific visual features contribute to the model's output, as illustrated in Fig. 1. Building on this, we extend our evaluation to multi-function general benchmarks [12, 22, 24, 30, 35, 68]. In these more complex settings, where precisely designing pairs of images that differ by only single visual function for token swapping is challenging,

---

instead employ a token dropping strategy. By analyzing performance degradation after dropping visual tokens from different layers, we can identify the importance of specific layers for various tasks within these benchmarks and subsequently infer the critical visual functions these layers support.
We comprehensively test across diverse range of vision functions and MILLM architectures, leading to the surprising discovery of consistent internal MLLM mechanism. This mechanism proves broadly applicable, from early MLLM iterations like the LLaVA series [23, 29] recent models such as the Qwen series [3, 62]. Our key findings are as follows:
MLLMs feature Vision Function Layers, where specific visual functions executed within remarkably narrow layer blocks (typically 2-3 layers). -wen-2.5-Vi. VL, for example, restricts count-function to layers 14-16 and OCR to layers 22-24. This division of labor is sharply defined: these functional layers operate with exclusivity, and other layers contribute negligibly these specific tasks.
Vision Function Layers exhibit a consistent arrangement in diverse MLLMs, where recognition typically occurs earliest, followed by counting in middle layers, then grounding, and finally OCR in later layers. This observed sequence holds true across MLLM generations (from LLaVA-v1.5 to Qwen2.5-VL) and scales (3B to 70B).
Vision Function Layers are redundant within MILMs. For tasks like ScienceQA and MMMU, which do not rely on function-specific layers, MLLMs often maintain or even improve performance when redundant Vision Function Layers, typically constituting over half of the model's 's depth, are omitted.
This mechanisms shed light on the "black-box" nature of MLLMs, offering explanations for previously puzzling phenomena and the diverse behaviors of MLLMs across applications. We believe these insights are fundamental numerous MILM applications, such guiding the development of canction-layer-targeee parameter-efffcient fine-tum tuning strategies and enabling more principled vision instruction data selection based active function layers.
In summary, our key contributions are as follows:
1. We propose a novel evaluation framework centered on visual token swapping and dropping. This framework operates by replacing carefully designed paired image data to precisely locate the functional layer of different tasks, providing unique methodology for analyzing MLLMs behavior.
We provide comprehensive findings obtained through the application of our framework to a wide range of MLLMs and various visual tasks. These evaluations reveal consistent layer-wise functional arrangement across different model families, successive versions, and model sizes, with specific visual functions consistently mapped to narrow, dedicated layer blocks.
We demonstrate the profound practical utility of our insights, showcasing that functional-laye targeting enables: (a) LoRA, matches its in-domain performance while boosting out-of-domain generalization; (b) data selection strategies surpassing human experts under identical budget constraints; and (c) achieving 98% of full-data performance with merely 20% of the data.
LoRA, using only one-third the tunable parameters of full
2 Vision Function Layer
2.1 Preliminaries on Multi-Model Large Language Models
Let I e "xxxxCC denote the imput image and T the tokenized text prompt. MLLMs first employ a vision encoder & to map I into N, dense embeddings V. These raw visual tokens are then projected into the language model's embedding space by a lightweight connector P. yielding aligned vision embeddings U. In parallel, the text sequence T is embedded into W. MILMS concatenate vision and text embeddings along token dimension to form the joint input [U; W). which then processed by successive Transformer layers ($(!) 1=1:L. the prefilling stage, in which the model builds up its multimodal context before any token is generated as follows:
[U()) w 4 (((UU))) w(I))))
(I)
3

---

During autoregressive decoding, each generated token continuously gathers information from the vision and text token representations at every layer, thereby enabling dynamic cross-modal interaction as follows:
P(y U. W)
II t=1
P(ue
yen U. W):
where Ngen denotes the total number of tokens generated and y = (1/1 ....... denotes the sequence of output tokens, predicted by projecting the final layer representation through linear layer followed by softmax over the vocabulary.
2.2 Decoding with Vision Token Swapping and Dropping
In this work, we systematically probe the layer-wise vision representations U quantify their individual contributions the predicted token. To probe the function role of vision tokens at different layers, we first introduce Vision Token Swapping: at layer k, we replace the original vision tokens UTk) with alternative set U(k,, while keeping all other layers unchanged. The resulting decoding probability becomes:
N II P( Uet, U(ER), (()), W),
P Pswap (y U(2R) U w)
where (A) can be substituted with vision tokens from any other image, even replaced with NULL tokens. By carefully designing (k),, we assess how vision tokens at layer & influence the generated output, thereby revealing their causal contribution to multimodal decoding.
Next, we consider an alternative probing method for scenarios where generating specific alternative set of vision tokens (4)) is not feasible or desired. In this approach, instead of swapping tokens, we directly drop them. We observed that merely nullifying the vision tokens at a single layer (i.e. removing ((k) without replacement and without new information from an alternative source ((()) often yields changes in the output that are too subtle to be clearly indicative. To elicit a more discernible impact and assess the cumulative importance of vision information processed up certain depth, we adopt strategy of progressively dropping all vision tokens from a given layer onwards, named as Vision Token Dropping. The decoding probability when all vision tokens from layer & onwards are dropped is formulated as:
N
Parup (y
U
w)
II P(u Use, U(<A) w),
where U(*A) denotes the visual tokens propagated up to layer k. In the subsequent experiments, we adopt Equ. 3 whenever valid replacement ((k) is available; otherwise, we adopt Equ. 4.
2.3 Targeting Vision Function Layer by Vision Token Swapping and Dropping
Experiment Setting. To precisely identify the layers for key visual functions within MLLMs, employ our Vision Token Swapping methodology, which measures the "change rate" in the outputs after token swapping. We construct dedicated paired image datasets for four key visual functions: Optical Character Recognition (OCR), Object Counting (Count), Object Recognition (Recognition), and Object Grounding (Grounding). designed to isolate single visual attribute, ensuring minimal differences between paired images and we random choose one as targe image and another as source image. Specifically:
exemplified in Fig. 1. Each image pair is meticulously
OCR pairs consist of distinct words (sampled from a deduplicated arXiv corpus [61]) rendered onto visually uniform blank canvases, designed to evaluate the model's capacity for textual information extraction. Change rate is quantified by whether the model's output text changes.
Grounding pairs present identical objects placed at varying random locations within otherwise clean backgrounds, aiming to probe spatial sensitivity. Change rate is the proportion of instances where the Intersection-over-Unn (loU) between the predicted bounding box and the swapping- ground-truth bounding box exceeds 0.5.

---

Counting pairs, adapted from the CLEVR dataset [19], differ primarily in the quantity of target object type, with associated queries focused on enumeration. Change rate is computed based on whether the predicted number changes.
Recognition pairs, drawn from COCO [27], contrast images containing a target object (e.g., a cat) with blank canvases; queries ask whether this target object is present. The change rate the proportion of "No" predictions after token swapping.
These experiments primarily utilize the Qwen-2.5-V Diven-2.5-VL-7- model [64] which contains 28 layers, and we have observed similar functional localization patterns across other MLLMs.
Experiment Results. Our interventions reveal that specific visual functions are handled in remark- ably narrow Vision Function Layers within MLLMs, as illustrated for Qwen-2.5-VL-775 in Fig. 2. "Results Change Rate (%)" is detailed in experiment settings. Collectively, these results highlight clear hicrarchical processing strategy within the MLLM, with distinct layers specializing in different visual functions, from foundational identity cues in early layers to complex OCR-related textual cues in deeper layers.
100
Recogn, shows high sensitivity in early layers (0-10), with basic visual features established early. though effect persists deeper. Count- ing peaks around layer 12 (87.4%) and Grounding peaks at layer 18 (100.0%), suggesting mid-layer pro- cessing for spatial and numerical reasoning. OCR peaks sharply in later layers (layer 22, 92.8%), in- dicating that visual-linguistic rep- resentations are finalized at last stages.
Results Change Rate (%)
80
Only Recognition
Grounding Peak (100.0%)
OCR Peak (92.8%)
Count Peak (87.4%)
60
Still Present -1.0% +
40
Visual Function
OCR
20
Count
Recognition
Grounding
14 16
18 20
26
Swapping Layer Index
This layer-wise order reflects MLLM strategy of progres- sive abstraction: starting with coarse object identification, ad- vancing to conceptual under- standing, and culminating in highly specialized representa- tions for tasks like OCR.
Figure 2: Vision Function Layer emerges in MILMs. A higher Results Change Rate (%) quantifies the involvement of the corre- sponding layer in processing the corresponding vision function. Functional peaks are sharply localized to specific layers, while other layers contribute negligibly. An exception is recognition function, which peaks in early layers but exhibits distributed influ- ence across almost all layers.
2.4 Targeting Vision Function Layer by Vision Token Dropping
Experiment Setting. To assess whether the layer-specific dependencies observed with our curated paired datasets generalize to broader visual question answering (VQA) contexts, we extend our investigation to general-purpont VQA benchmarks. In these experiments, progressively dropping vision tokens as defined in Equ. 4. We test models such as LLaVA-v1.5 (7B, 13B) [23, 29] and Qwwn2.55VV. (3B, 7B) [3, 64] across suite of benchmarks including SQA-I [33]. MMMU [68], POPE [22], SEED [24], CVBench [58], TextVQA [52], OCR [30], and ChartQA [35]. Performance is measured using the standard accuracy metries pertinent to each benchmark. Tab.
evaluate the impact of
presents the detailed results, showing performance when all vision layers are used (baseline) versus when increasing number of final layers are omitted.
Experiment Results. Across different models and scales (see Tab. 4), our conclusions are:
1. Different MILMS show remarkable consistency in the hierarchical arrangement of the vision function layer. This hierarchical order strongly corroborates findings from our earlier paired-image swapping experiments. Both experimental approaches reveal that MLLMs process visual information hierarchically, where OCR capabilities decline first (with both
5

---

General & Knowiedge
Recognition
Spatial
OCR & Chart
MMMU1668
CVBench [58]
TextVQA [52]
ChanQA [35]
SQA-1
MME
PORE [22]
SEED [24]
OCR OCR [30]
Method
L.VAA-V1.5-75 -----2255-7B-22 10--7B322-aay- llylr
68.8
34.7
1455.9
86.4
67.3
56.2
47.2
33.0
22.0
drop v-layery
68.8
34.4
1460.6
86.4
67.3
56.4
44.8,
31.7,
21.0,
drop 16 v-layers
68.7
34.3
1457.3
86.0
67.3
53.7.
18.1
10.1
15.9
drop 20 v-layers
69.0
35.2
1470.4
83.2,
65.8,
43.0
13.1
3.7
13.8
drop 24 v-layers
63.7.
33.9,
835.5
38.1
45.6
37.2
9.7
1.9
12.8
LLaVA-v1.5-13B--
72.7
35.4
1522.0 G
85.9
08.2
53.0
48.7
33.5
22.6
drop v-layers
72.6
35.8
1528.3
86.0
68.2
53.2
44.8
31.2,
21.1,
drop 16 v-layers
72.7
35.4
1547 0
84.9
68.0 U
53.2
16.6
6.7
15.8
drop 20 v-layers
72.2
37.2
1458.2,
74.9,
65.8,
51.2,
12.0
2.2
15.0
drop 24 v-layers
70.1
34.6,
783.1
11.5
48.2
52.2
9.2
2.1
13.5
Qwen2.5-VI.3B 10amn.5-Vi-3--3--laae la-eerr
80.3
46.3
1530.9
87.0
74.8
72.9
77.8
77.8
83.4
drop v-layers
80.0
46.9
1528.1
86.9
74.9
72.9
59.3
56.4
78.6
drop v-layers
80.1
46.3
1530.8
86.9
74.9
72.6
22.0
17.9
60.0
drop v-layers
79.6
46.2
1400.9,
82.7+
66.2,
56.
12.4
13.0
drop 20 v-layers
76.7.
45.2,
905.1
1944
54.1
17.2
10.9
2.3
13.2
Swen2.5-VL-7B-28--BB
87.2
30.7
1696.4
86.1
77.6
80.8
82,8
82.2
83.2
drop v-layers
87.4
50.8
1693.6
86.3
77.5
81.0
74.1
76.3
82.7
drop y-layers
87.4
50.6
1683.1
86.2
77.5
80.6
15.3
5.5
20.5
drop v-layers
50.2
1633.9,
79.5.
74.5
69.1
17.4
drop v-layers
77.3.
45.8
1111.5
52.4
44.2
12.2
Table 1: Vision Token Dropping on General Benchmarks. A, indicates the onset of performance degradation, while A highlights significant drops. The results reveal consistent hierarchical order of vision function layers across diverse MLLMs. Results of other MLLMs are provided in Appendix.
models starting to lose OCR functionality between layers 4-8), followed by spatial reasoning, and finally, object recognition.
Many tasks do not necessitate visual tokens from every layer, and critically, some tasks achieve superior performance when specific, seemingly non-contributory vision function layers are omitted. For example, on the MMMU task, all models achieved their highest performance when some vision tokens were dropped, and the highest increase could be 1.8%.
Driving Progress in Multimodal LLMs with Vision Function Layer Insights
3.1 Vision-Funcctin--LoRR
Motivation. Fine-tuning pre-trained MLLMs is widely used to strengthen specific abilities such as spatial reasoning [58, 65]. Due to their large size, PEFT methods like LoRA [18] have become standard. However, LoRA is typically applied uniformly across layers, which is suboptimal: as analysis (Fig. 2) shows, different visual functions are localized to specific layers and dropping useless function layers can improve the performance. Moreover, task-specific fine-tuning risks degrading general performance through catastrophic forgetting. To address this, we propose Vision-Function LoRA (VFL), PEFT method that selectively applies LoRA updates only to layers critical for the target visual function(s), thereby enhancing desired skills while preserving overall model capability.
Experiment Setting. To evaluate the efficacy and benefits of VFL-LoRA, we focus on enhancing spatial reasoning fundamental visual understanding capability where current MLLMs often exhibit deficiencies. It is important to note that, to evaluate VFL-LoRA's generalizability and the robustness of the identified Vision Function Layers, count-function from Fig. 2, without any access to the training or test data of the downstream spatial reasoning benchmarks. For example, for (wwen.55VVL-77 we use layers 10-17, 20, 21, 22, and 23.
directly select the layers with non-zero change rate of
We utilize the SAT [46] as training dataset, specifically its single-image ceestion-answering tasks prob- ing spatial understanding, Our base architectures the Qwen2.5-VV models [64]. We benchmark

---

CV-Bench Count
CV-Bench Depth
In-Domain
Out-of-Deuna
Relation [58]
Distance [558]
CharQA [35]
MMMU (68)
Accuracyyt)
6568.7
Parum. (S)
Depth [58]
Rou (22)
Jaccam Cupes 2005
Average
Average
Count
40
Dwen2 5-8-38
Qwen2.5-VI-3 -3V
75.0
68.1
77.8
79.3
69.7
72.2
83.4
46.3
87.0
5wen2.5 VL-78
Lora
3.1
82.7
70.6
91.2
86.3
87.8
71.8
82.0
46.1
87.3
CV-Bench Distance
CV-dendh Relation
Reversed-VFI.
2.1
82.0
70.3
89.3
86.6
87.0
71.9
83.0
46.8
86.
VFL (Ours)
83.5
72.3
90.0
88.3
86.3
72.9
83.4
47.3
88.0
80
Lorn
1.9
82.1
84.4
70.9
68.0
91.3
91.2
87.3
91.1
88.3
80.5
74.3
73.3
83.2
86.2
50.1
50.7
86.6
86.1
70
64.8
68/2
Reversed-VFF VFL.
0.9
82.7
69.0
91.0
88.1
87.1
74.0
85.9
51.2
84.9
+1.8%
* 14.9
VEL (Ours)
0.9
85.0
72.6
91.4
91.0
86.8
75.0
86.4
51.7
86.9
50
Ri porn Guise (50%)
fomm Cuess (0%)
20
10
20
10
Drop layer
Figure 3: VFL-LoRA Efficiency and Diagnostic Analysis on CV-Bench. Left Table: VFL-LoRA, which trains LoRA adapters exclusively on Count-Functions Layers, achieves significant parameter efficiency while maintaining competitive in-domain performance and demonstrating superior out-of- domain generalization across diverse benchmarks. Right Figure: Analysis of CV-Bench sub-tasks (Count, Depth, Distance, Relation) using vision token dropping. Results show high visual dependency for Count and Depth sub-tasks, contrasting with strong language priors for Distance and Relation. Attributing its strength in vision-heavy tasks like counting to its focus on Vision Layers, this analysis shows VFL yields clear improvements in this domain, with less impact on language- focused tasks.
VFL-LoRA against two primary baselines: (1) Standard LoRA, where LoRA is applied uniformly across all adaptable layers, and (2) Seversed-VFL, an ablation study where LoRA is applied to layers excluding the count-functo fonnt-fnntionn layer range. The evaluation conducted on comprehensive test set comprising both in-domain spatial reasoning tasks from CV-Bench (which includes sub-tasks like Count, Relation, Depth, and Distance) and diverse suite of out-of-domain benchmarks (such as ChartQA [35], OCRBench [30], MMMU [68], and POPE [22]) to assess broader generalization. Detailed results are presented Fig. 3
Experiment Results. The performance of VFL-LoRA, primarily benchmark against standard LoRA, detailed in Fig. 3. Notably, VFL-LoRA achieves substantial reduction in tunable pa- rameters, requiring nearly 50% fewer (155M vs. 309M for standard LoRA the tested models). Beyond this increased parameter efficiency, VFL.-LORA attains marginally superior average in- domain accuracy on the CV-Bench spatial reasoning tasks (84.4% vs. 85.0% for standard LoRA), with particular improvements on specific sub-tasks such as CV-Count (72.6% vs. 70.9%). However, standard LoRA showed a lead CV-Distance (A detailed analysis is provided in Deeper Lock at CV-Bench) More critically, for out-of-domain generalization, VFL-LoRA consistently surpasses standard LoRA, achieving a higher average performance (75.0% vs. 74.3%) across the diverse suite of benchmarks including ChartQA, OCRBench, MMMU, and POPE. In essence, these results indicate that VFL-LoRA not only provides significant parameter savings but also largely maintains enhances performance, especially in out-of -f--o---mmmin generalization, compared to the standard LoRA.
A Deeper Look at CV-Bench. To further dissect visual dependencies across spatial reasoning tasks in CV-Bench, we apply vision token dropping to Qwen2.5-VL. As shown in Fig. 3, we observe clear task-specific patterns. Performance Count and Depth drops sharply as more layers are removed, eventually nearing random sevell-conffrmingg their strong dependence on processed visual input. In contrast, Distance and Relation remain robust even with heavy layer dropping, suggesting they rely more on language priors and statistical biases rather than detailed visual features. This explains why VFL-LoRA consistently improves count-centric and other secception-hawwe tasks, but offers limited gains on tasks like CV-Distance that rely less on the targeted visual functions.
3.2 Data Selection through the Lens of Vision Function Layers
The demand for large-scale instruction datasets to train MLLMs presents significant challenges, as these datasets, while rich in diverse signals, often have heterogeneous quality, making it difficult to determine the specific contribution of individual instances to enhancing distinct model capabilities. This ambiguity complicates efficient training and targeted skill development, thereby necessitating more efficient data selection strategies. To address this, we propose VFL-Select,
Motivation.
7

---

General
Knowledge
OCR & Chart
Vision-Centrie
MMMU
CharzQA
TexiVQA
Data
MME
SEED
GQA
SQA!
A12D
OCR
Count
Dist
Oracle
665k
1476.9
67.3
63.0
86.4
34.7
62.5
22.0
33.0
47.2
34.1
43.0
150k
1306.6
59.3
50.0
64.0
33.4
50.9
27.0
30.3
44.7
34.4
49.7
250k
1411.6
52.7
59.3
37.3
52.9
28.0
38.1
53.8
Random
350k
1358.3
62.5
54.5
61.8
36.1
53.9
31.7
24.2
47.0
37.6
665k
1410.8
64.7
56.7
60.4
36.5
57.1
33.7
22.8
48.2
33.1
49,2
150k
1338.3
56.3
51.8
64.5
33.8
52.1
28.0
15.9
44.0
38.3
51.1
250k
1337.6
59.7
53.5
62.7
35.1
53.4
29.5
17.0
44,9
35.1
46.8
Expert [58]
350k
1360.8
60.5
55.1
62.4
34.4
56.0
31.4
16.8
45.6
35.8
55.1
665k
1421.0
62.6
56.7
66.0
34.4
56.3
34.3
25.1
47.4
35.2
48.2
A 10.2
2.1
0.0
2.1
0.8
0.6
2.3
0.8
2.1
1.0
150k
1357.1
60.8
55.8
66.5
36.9
53.9
30.6
28.9
45.4
36.9
52.6
250k
1444.3
62.5
$6.8
69.0
37.1
55.8
32.1
32.5
46.7
35.3
55.5
VFL (Ours) 350k
1462.6
63.7
58.0
69.5
37.1
57.0
33.4
33.2
47.0
36.6
50.8
665k
1526.3
68.2
64.1
86.0
38.3
63.1
37.5
34.1
49.5
35.3
48.2
115.5
3.5
25.6
1.8
6.0
3.8
11.3
1.3
1.0
Table 2:
Comprehensive Benchmark Results for Data Ratio Experiments. We compare data subset selection Ptrategies-Orakee Random, Expert [58], and our VFL acr-scross sample sizes ranging from 150k to 665k. Results show that VFL consistently outperforms both Expert and Random baselines, with particularly notable gains on general, knowledge, and OCR tasks. In the table, results the optimal 665k setting are bolded, while the best scores for other subset sizes are underlined.
novel approach that leverages our understanding VFLs as a guiding principle for data curation. The core idea is that data instances most beneficial for improving MLLM capabilities are those that effectively engage, or are predicted to refine, these functionally specialized layers. By analyzing data "through the lens" of VFLs, VFL-Select aims to curate smaller, higher-quality, and more targeted datasets, prioritizing data based on its predicted utility.
Experiment Setting. We construct a diverse data pool consisting of 20 million vision instruction samples, covering wide range of tasks and modalities. To implement VFL-Select, we first determine the functional value of a given sample (æ., y) (input æ, ground-truth answer y) in layer k as:
Ply (xx)((A) W)
Ra y) P(y ((z)((k-L)) W)
(w) W) is the probability of generating y in Equ. 4. A higher value Rg for a given sample suggests greater reliance layer A: for correctly processing that sample, thus associating the sample with the vision function in that layer. This allows for a functional categorization of data without requiring explicit prior knowledge semantic labeling of what kind of vision function each specific layer represents. In practice, we partition the entire dataset based on the highest R& score for each sample, effectively grouping data according their dominant layer- ayser influence. From each partition, we then uniformly sample data to construct balanced subsets for training. A crucial aspect for practical application is the scalability of this data classification process.
where P(y
Notably, our findings indicate consistent VFL hierarchical trends across diverse MLLMs. This allows the computationally intensive VFI.-Seleci data classification to be efficiently executed using smaller proxy models (e.g., TinyLLLaVA [75]), with the derived insights directly informing data curation for much larger target models (e.g., 7B LLaVA model), substantially reducing computational overhead and enhancing VFL-Select's practical scalability.
Experiment Results. Tab. 2 demonstrates that VFL-Select consistently outperforms both Random and Expert data selection strategies [58] across all tested subset sizes (150k to 665k instances). VFL-Select particularly excels Knowledge-intensiv benchmarks. For instance, on SQA with 665k data subset, Vele-Select achieves a score of 86.0, substantially outperforming Random selection (c.g., 60.4) and Hluman-Exper selection (c.g.. 72.1) as detailed in Tab. 2. This robust outperformance
8

---

Shallow-layer
Deep-layer
SQA!
MMMU
MME
POPE
SEED
VQAv2
GQA
TeXVVQA
OCR
ChanQA
Method
Rel.
Rcl.(%)
Full
68.4
34.7
1476.9
86.4
67.3
79.1
63.0
58.2
33.0
22.0
Random
68.5
33.2
1483.0
84.7
62.2
97.3
75.7
58.9
55.3
30.3
19.7
93.1
D2-Pruning [34]
69.3
34.1
1391.2
85.7
63.1
97.4
73.0
58.4
51.8
30.9
20.3
92.0
EL2N [41]
65.5
34.0
1439.5
84.3
63.1
96.5
76.2
58.7
53.0
30.1
21.2
93.6
COINCIDE [21]
69.2
34.1
1495.6
63.8
76.5
59.8
55.6
29.1
20.8
94.0
VFL (Ours)
70.4
34.2
1504.2
86.1
63.5
99.5
77.4
61.4
57.1
31.0
22.0
97.4
Table 3: Performance of Data Selection Methods using a 20%
LLaVA-665k Subset. All strategies,
excluding "Full" (trained on 100% of LLaVA-665k), utilize only a 20% subset of the LLaVA-665k
data for fine-tuning. Performance on shallow-layer and deep-layer task categories presented relative to the "Full" model's scores (Rel.(%)
confirms that VFL-Select efficiently identifies higher-utility data instances from large, heterogeneous pools, leading to enhanced model performance for fixed data budget and demonstrating the value of leveraging VFL insights for intelligent data curation.
Experiment on LL5VA-665k. To further assess the versatility and effectiveness of our VFL-Select methodology, we conduct experiments focusing on its ability to identify high-utility data within a more constrained and established dataset, specifically VLaVA-665k [23]. The objective was to curate an optimal 20% subset from the LLaVA-665k dataset itself for fine-tuning. We compared VFL-Select against other selection strategies (Random, D2- Pruning, EL2N, COINCIDE) operating under this 20% data constraint. The performance of models fine-tuned on these subsets was evaluated relative to model trained on the complete LLa VA-665k dataset ("Full")). As detailed in Tab. 3, with only 20% of the LLLVA VA-65-6 data, models fine-tuned using VFL-Select achieved 99.5% of the full-data performance shallow-layer task benchmarks and 97.4% deep-layer task benchmarks. These results significantly surpassed those of other data selection methods, underscoring VFL-Select's s efficacy identifying the most impactful training instances.
Related Work
Layer-wise Representations in LLMs. Recent studies investigate the role of individual layers within LLMs. AdaInfer [11] finds that many layers in LLMs are redundant, with only about 20% of layers being essential for general tasks and around 50% for sentiment analysis. Their method assesses the contribution of each layer by directly ablating it. Building on the same hypothesis, DSA [26] introduces pruning strategy that leverages per-layer importance scores to search for a computation rule that determines the pruning ratio for each layer. In similar vein, LISA [40] shows that many parameters introduced during LoRA fine-tuning are also redundant, and proposes selecting layers to fine-tune based their weight norms.
Layer-wise Representations in MILMs. There has been limited focus on layer-wise representa- tions in MILMS. The most relevant line of work comes from token pruning studies [5, 67, 72], which reveal that MLLMs do not require all vision tokens to perform accurate reasoning. Methods [67, 72] have shown that the acceptable token reduction rate varies across different layers. While prior work focuses on token-level efficiency, it leaves the reasons unexplained. In contrast, we investigate layer-wise vision functions, offering deeper insights into how vision tokens contribute across layer.
Conclusion
This paper reveals that Multimodal Large Language Models (MLLMs) develop narrow, hierarchical Vision Function Layers (VFLs) specialized for tasks like counting, localization, and OCR recognition. Using Visual Token Swapping and token dropping, we show these structures emerge consistently across models. Leveraging this, our VFL-targeted fine-tuning cuts parameter costs while preserving performance, and VFL.-guided data selection achieves 98% of full-data results with just 20% of data. Our findings offer new paths toward more interpretable and efficient multimodal systems.

---

Acknowledgement: This work is supported in part by the National Natural Science Foundation of China under Grant No.62206174 and 66.62576365, and Hong Kong Research Grants Council under NSFC/RGC Collaborative Research Scheme (Grant URS_HHK703/24)..
10

---

References
[1] Mistral AL Mixtral of experts, 2023.
[2] Rohan Anil, Yinpeng Bai, Aakanksha Chowdhery, et al. Palm 2 technical report, 2023.
[3] Bai, Yang Gu, Yong Liu, et al. Qwen technical report. arXiv preprint urXiv:3310.02404 2023.
[4] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baohao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, In European Conference on Computer Vision, pages 19-35. Springer, 2024.
[5] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-pil inference acceleration for large vision- lannuala models. In European Conference on Computer Vision, pages 19-35. Springer, 2024.
[6] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:24/2.05277 2024.
[7] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185 24198, 2024.
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, et al. Palm: Scaling language models with pathways. arXiv preprint arXiv:2204.00 023111, 2022.
[9] Yunkai Dang, Kaichen Huang, Jiahao Huo, Yibo Yan, Sirui Huang. Dongrui Liu, Mengxi Gao, Jie Zhang, Chen Qian, Kun Wang, et al. Explainable and interpretable multimodal large language models: A comprehensive survey. arXiv preprint arXi::24/2.02 104, 2024.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.10 11929, 2020.
[11] Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Pong Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. Not all layers of iims necessary during inference. arXiv preprint 2r300:2403.02181, 2024.
[12] Chaoyou Fu, Peixian Chen, Yunhang Shen. Yulei Qin, Mengdan Zhang, Xu Lin. Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
[13] Zheng Ge, Qian Jiaye, Tang Jiajin, and Yang Sibei. Why Ivims are more prone hallucinations in longer responses: The role of context. ICCV 2025, 2025.
[14] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint 2407 2024.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[16] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song. and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 00X70:2004.08758, 2021.
[17] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
[18] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
[19] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedingy of the IEEE conference computer vision and pattern recognition, pages 2901-2910, 2017.
[20] Tianjie Ju, Weiwei Sun, Wei Du, Xinwei Yuan. Zhaochun Ren, and Gongshen Liu. How large language models encode context knowledge? layer-wiss wise probing study. arXiv preprint 4xXiv:2402.1606/ 2024.
11

---

[21] Jaewoo Lee, Boyang Li, and Sung Ju Hwang. Concept-skill transferabillty-bssee data selection for large vision-language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024.
[22] Aojun Li, Mike Zhang, and Wallace Hallucination. Evaluating object hallucination in large vision-languag models, arXiv preprint arXiv:2305. 10355, 2023.
[23] Bo Li. Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:248..03:: 033366, 2024.
[24] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang. Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognittin, pages 13299-13308, 2024.
[25] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2 Bootstrapping language-inan pre pre-training with frozen image encoders and large language models. In International conference on machine learning. pages 19730-19742. PMLR, 2023.
[26] Lujun Li, Peijie Dong Zhenheng Tang, Xiang Liu, Qiang Wang. Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo. Discovering sparsity allocation for layer-wise pruning of large language models. Advances in Neural Information Processing Systems, 37:141292-141317 2024.
[27] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer VECi 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. 755.0755.505 Springer. 2014.
[28] Zihao Lin, Samyadeep Basu, Mohammad Beigi. Varun Manjunatha, Ryan A Rossi, Zichao Wang. Yufan Zhou, Sriram Balasubramanian, Arman Zarei, Keivan Rezaci, al. survey on mechanistic interpretability for multi-modal foundation models. arXiv preprint arXiv:2502.17516, 2025.
[29] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. 2023.
[30] Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr large multimodal models. arXiv preprint +xXix:3355008895 2023.
[31] Siqu Long, Feiqi Cao, Soyeon Caren Han, and Haiqin Yang. lision-and-languge pretrained models: A survey. arXiv preprint vrXiv:2204.07356 2022.
[32] Miguel Ataez-Otal, Jorge Gracia, Jordi Bernad, Carlos Bobed, Lucía Pitarch- Bearrc-Baallesteros and Emma Anglés-Herrero. Linguistic interpretability transformer-basee language models: a systematic review. arXiv preprint (xXx::2504.08000, 2025.
[33] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS). 2022.
[34] Adyasha Maharana. Prateek Yadav, and Mohit Bansal. D2 pruning: Message passing for balancing diversity and difficulty in data pruning. arXiv preprint 11X 2023.
[35] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Md Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
[36] Gavin Mischler, Yinghao Aaron Li, Stephan Bickel, Ashesh D Mehta, and Nima Mesgarani. Contextual feature extraction hierarchies converge in large language models and the hrain. Nature Machine Intelligence, pages 1-11, 2024.
[37] Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing in vision-language models. arXiv preprint arXiv:2410.07 2024.
[38] SubbaReddy Oota, Manish Gupta, and Mariya Toneva. Joint processing of linguistic properties in brains and language models. Advances in Neural Information Processing Systems, 36: 18001-1411,, 2023.
[39] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafrannicc Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby. et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199. 2023.
12

---

[40] Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: layerwise im- portance sampling for memory-efficien large language model line-tuning Advances Neural Information Processing Systems, 37:57018-57049 2024.
[41] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 34:20596 24-20596-20007,, 2021.
[42] Jiaye Qian, Zheng Ge, Zhu Yuchen, and Yang Sibei. Intervene-all-paths: Unified mitigation of Ivim hallucinations across alignment formats. NewrIPS 2025, 2025.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR, 2021.
[44] Maryam Rahimi, Yadollah Yaghoobzadeh, and Mohammad Reza Daliri. Explanations of large language models explain language representations in the brain. arXiv preprint arXiv:2302. 14677, 2025.
[45] Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Pency Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, 2016.
[46] Arijit Ray. Jiafei Duan, Ellis Brown, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A. Plummer, Ranjay Krishna, Kuo-Hao Zeng, and Kate Saenko. Sat Dynamic spatial aptitude training for multimodal language models, 2025.
[47] Teven Le Scao, Angela Fan, Christopher Akiki, et al. Bloom: A 176h-parameter open-access multilingual language model. arXiv preprint arXiv:221 1.05100, 2022.
[48] Cheng Shi and Sibei Yang. Edadet: Open-vocabulary object detection using early dense alignment. In Proceedings of the IEEE/CVF international conference on computer vision, pages 15724-15734 2023.
[49] Cheng Shi and Sibei Yang. Logoprompt: Synthetic text images can be good visual prompts for vision- language models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2932-2941, 2023.
[50] Cheng Shi, Yulin Zhang, Bin Yang, Jiajin Tang, Yuexin Ma, and Sibei Yang. Part2ohject: Hierarchical unsupervised 3d instance segmentation. Eurupean Conference on Computer Vision, pages 1-18. Springer. 2024.
[51] Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao, Jan Kautz, Zhiding Yu, and Guilin Liu. Eagle: Exploring the design space for multimodal IIms with mixture of encoders. arXiv:2408.1594 2024.
[52] Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition, pages 8317-8326, 2019.
[53] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, pages 163 1631-1642, 2013.
[54] Peter Sterling and Simon Laughlin. Principles of neural design. MIT press, 2015.
[55] Yu Sun. Shuohuan Wang, Yukun Li, et al. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. Science China Technological Sciences, 2089)2208772 2023.
[56] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4149 4158, 2019.
[57] Gemini Team, Rohan Anil, Yinpeng Bai, et al. Gemini: A family of highly capable multimodal models. arXiv preprint 18Xi 2023.
13

---

[58] Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adidhya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu. Ziteng Wang, al. Cambrian-1: A fully open. vision-centric exploration of multimodal Ilms. Advances in Neural Information Processing Systems. 1197710--73560 2024,
[59] Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. Llama: Open and efficient foundation language models. arXiv preprint urXiv:2302.13977, 2023.
[60] Hugo Touvron, Louis Martin, Kevin Stone, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[62] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- -anguugg model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[63] Sanghyun Woo, Shoubhik Debuath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16133- 16142, 2023,
[64] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.11113 2024.
[65] Jihan Yang, Shusheng Yang. Anjali W. Gupta, Rilyn Han, Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces. arXiv preprint
141111
[66] Alex Young. Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403. 046522 2024.
[67] Qianhao Yuan, Qingyu Zhang, Yanjiang Liu, Jiawei Chen, Yaojie Lu, Hongyu Lin, Jia Zheng. Xianpei Han, and Le Sun. Shortv: Efficient multimodal large language models by freezing visual tokens in ineffective layers, arXiv preprint arXiv:2504.00500 2025.
[68] Xiangsay Yue, Tiao Yu, Kai Zhang, Keyu Wang, Siyuan Wang, Zixu Kuang, Wenhui Lin, Yiming Wang. Bo Zheng, Kaiming Hc, et al. MMMU: A massive muuti-discipline multimodal understanding and reasoning benchmark for expert AGI. arXiv preprint 1.Xi:23311.16002 2023.
[69] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, pages 649-657,20 2015.
[70] Yulin Zhang, Cheng Shi. Yang Wang. and Sibei Yang. Eyes wide open: Ego proactive video-Ilm for streaming video. arXiv preprint wwXiv:2510.145000 2025.
[71] Qinyu Zhao, Ming Xu, Kartik Gupta, Akshay Asthana, Liang Zheng, and Stephen Gould. The first to know: How token distributions reveal hidden knowledge in large vision- language models? In European Conference on Computer Vision, pages 142. Springer. 2024.
[72] Shiyu Zhao, Zhenting Wang, Felix Juefei-Xu, Xide Xia, Miao Liu, Xiaofang Wang, Mingfu Liang, Ning Zhang, Dimitris N Metaxas, and Licheng Yu. Accelerating multimodel large language models by scarching optimal vision token reduction. arXiv preprint arXiv:2412.00556 2024.
[73] Ge Zheng, Bin Yang, Jiajin Tang, Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of- thouh prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36:5168 5191, 2023.
[74] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging Ilm-as-a-judge with me-bench and chatbot arena, 2023.
[75] Baichuan Zhou. Ying Hu, Xi Weng. Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models, 2024.
[76] Changhai Zhou, Shijie Han, Lining Yang. Yuhua Zhou, Xu Cheng, Yibin Wang, and Hongguang Li. Rankadaptor: Hierarchical rank allocation for efficient fine-tuning pruned Ilms via performance model. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5781-5795 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
For experiment results section, we discuss the limitations of our method under specific conditions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide detailed experiment information.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Code will be released after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
mublic/guides// wtweees/oooeSuumisiiioooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss cttppsccccpbccccccceecccecccclcc htt.gguddde//eeeebiiiiieeeee /Cc///eodiiisseeeeeeeea ssssonoliccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide detailed experimental settings.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The performance gains are substantial, and present detailed analyses to support our findings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide resources details in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurIPS Code of Ethics https: httpp/://////us./e/eeee/ http://eeerrssccccpubbbee cctpublic/Ettiiecceeeececoo cttpliiieetiisauieeeeliiiiee
didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: / /ttpr//erspp///// httplppss..//offfeeeeeeeeeeeeeee /22/oeeereeeee222/////// 2LLM/LLM for what should or should be described.
21

---

In the supplementary material, we provide additional information regarding,
Evaluation Protocol (In Section A).
More Results of other MILM on Vision Token Dropping Experiment, (In Section B).
Implementation Details of VFL-LoRA (In Section C).
Implementation Details of VFL-Select (In Section D).
Discussion, Limitation and Future Work (In Section E).
A
Evaluation Protocol
In this section, we provide detailed evaluation settings. Our test set covers a wide range of capabilities, including general knowledge, visual recognition, spatial reasoning, well as chart and our own tasks.
Evaluation tool. We observed that the official evaluation tools provided by some benchmarks inefficient in large-scale experiments. Therefore, we adopt two well-maintained toolkits: 1mms -emn- (v0.33) 3 and VLMEvalKit (v0.2) 4, perform all our evaluations.
is important to note that the version of the evaluation toolkit can significantly affect the final reported performance. For fair comparison, especially for experiments that require external baselines (e.g.. Tab.3), we follow the default evaluation protocol used by each benchmark. For all other experiments, we adopt Imms-eval and VLMEvalKit to improve efficiency, To ensure reproducibility, we release all the question prompts used during evaluation.
Question Prompt
[ScienceQA']
inAnswer with the option's letter from the given choices directly.
[MMMU]
Multi-Choice Answer with the option's letter from the given choices directly.
Open-Ended Answer the question using a single word or phrase.
[MME]
inAnswer the question using single word or phrase.
[POPE]
InAnswer the question using single word or phrase.
[SEED]
InAnswer with the option's letter from the given choices directly.
[TextVQA]
'nAnswer the question using single word or phrase.
[ChartQA]
inAnswer the question using single word.
Unlike the default CV-Bench protocol, report the final performance by directly averaging the results over the four subsets.
CV-Bench.
More Results on Vision Token Dropping Experiment
In this section, present additional results from the vision token dropping experiments. We first provide a qualitative analysis on the Math dataset, followed by a comprehensive quantitative evaluation across multiple datasets by different MLLMs.
Vision Function Layer Different MLLMs. We provide additional experimental results on vision token dropping across various Multimodal Large Language Models (MLLMs) in Tab. 4. The models are categorized based on their vision encoders and language models as follows:
eval
'https://githar hwwpeithu..oooooeeeeeeeeeoeeeeee nom/open-ooe cowpass/VLMBvallee
22

---

General & Knowledge
Recognition POPE [22]
Spatial CVBench [58]
OCR
Vision Encoder & Language Model
[33-1[33]
MMMU [68]
ChartQA [35]
CLIP [43] & Vicuna [74]-7B
68.8
34.7
86.4
56.2
22.0
drop 8 v-layers
68.8
34.4
86.4
56.4
21.0,
dmp v-layers
68.7
34.3
86.0
53.7,
15.9
dmp v-layers
69.0
35.2
83.2,
43.0
13.8
drp v-layers
65.7.
33.9,
38.1
37.2
12.8
CLIP [43] & Vicuna [74]-13B
72.7
35.4
85.9
53.0
22.6
drop 8 v-layers
72.6
35.8
86.0
53.2
21.1,
drop 16 v-layers
72.7
35.4
84.9
53.2
15.8
drop 20 v-layers
72.2
37.2
74.9,
51.2,
15.0
drop 24 y-layers
70.1
34.6,
11.5
52.2
13.5
DINOv2 [39] & LLaMA-3 [14]-8B
71.4
32.7
85.5
64.8
73.1
drop 8 v-layers
70.1
34.0
85.0
66.4
70.8,
drop 16 y-layers
70.5
31.3
84.5
68.9
18.6
drop 20 y-layers
69.6
34.8
78.2,
66.5.
13.4
dmp v-layers
67.7.
36.5
6.6
44.3
11.6
DINOv2 [39] & Vicuna [74]-13B
74.3
37.7
85.6
65.4
72.7
drop 8 v-layers
74.3
40.7
84.6
66.0
62.0,
drop 16 y-layers
74.3
37.4
84.9
66.2
17.8
drop 20 v-layers
74.6
37.2
76.9,
57.3,
14.0
drop 24 v-layers
72.4.
36.6,
17.6
51.0
13.1
Qwen-ViT & Qwen2.5-LM (3)-3B
80.3
46.3
87.0
72.9
83.4
drop 4 v-layers
80.0
46.9
86.9
72.9
78.6
drop 8 v-layers
80.1
46.5
86.9
72.6
60.0
drop 16 v-layers
79.6
46.2
82.7,
56.4
13.0
drop 20 y-layers
76.7,
45.2,
19.4
47.2
13.2
Qwen-ViT & Qwen2.5-LM (3]-7B
87.2
50.7
86.1
80.8
83.2
drop 4 v-layers
87.4
50.8
86.3
81.0
82.7
drop 8 v-layers
87.4
50.6
86.2
80.6
20.5
drop 12 v-layers
87.2
50.2
79.5.
69.1
17.4
drop 18 v-layers
77.3,
45.8
37.1
44.2
14.3
Table 4: Vision Token Dropping on General Benchmarks. A, indicates the onset of performance degradation, while A highlights significant drops. The results reveal a consistent hierarchical order of vision function layers across diverse MLLMs.
Vision encoders: CLIP [43], DINO [39], and Qwen-Vit
Language models: Micuna-BB/13B [74], LLaMA-3 [14] 8B, and Qwen2.5-LM--3B/7B [3].
Across different combinations of these components, all MLLMs consistently exhibit the phenomenon of Vision Function Layer, showing similar trends. This demonstrates that the observed behavior is an isolated case, but rather shared internal mechanism among different MLLMs.
Interestingly, for MLLMs equipped with DINO as the vision encoder, we find that removing later- stage Vision Function Layers (VFLs) can actually enhance performance certain CV-related tasks. On CV-Bench, for instance, dropping 20 layers results higher accuracy than using the full model (68.9 64.8), indicating potential redundancy task misalignment in deeper VFL.s
Vision Token Dropping on Math We provide a qualitative analysis of vision token dropping Math Vista, illustrating how different vision token layers affect complex visual reasoning. For example, when interpreting a bar chart, the full model accurately read bar values (8 and 6, correctly finding none smaller than 6) and text. Dropping later vision token layers (VFLs-L20-L28) showed nuanced impact. While the model still correctly reasoned that no bars were smaller than 6 (reporting values 6 and 8), its OCR capability was severely compromised, outputting generic placeholders like "X-axis labels" instead of actual text. This suggests these layers are critical for fine- fine-grained detail like but less so for gross quantitative estimation. More aggressive dropping (VFLS-L10-L28) led to significant degradation in fundamental visual perception. The model then incorrectly identified bar values as approximately and 2, wrongly concluding two bars were smaller than These examples demonstrate that different layers uniquely contribute to diverse visual reasoning abilities essential for Math Vista tasks, with later layers crucial for OCR broader range vital for accurate visual magnitude interpretation.
23

---

Vision Token Dropping Math Vista
buft
How many bars have values smaller than 6?
There are no bars in the chart that have values smaller than 6. The two bars shown in the chart have values of 8 and 6, respectively.
Drop VFLs-L20-L28
There are no bars in the chart that have values smaller than 6. The two bars shown in the chart have values of 6 and 8, respectively.
Drop VFLs-L10-L28
In the given bar chart, there are two bars that have values smaller than 6. The first bar has value of approximately 4, and the second bar has value of approximately
What text can you read from the image?
The text in the image reads:-
Title: Title- Values: 0, 2, 4, 6, 8, 10- Categories: butt, sick.
Drop VFLs-L20-L28
The text in the image reads:- "X-axis labels: X-axis labels"- "Y-axis label: Y-axis label" "Title: Title.
Subtract all large yellow metal blocks. Subtract all gray metallic cylinders. How many objects are left?
In the image, there are three objects:1. A large yellow metallic cube. 2. A small red metallic c. A small cyan sphere. Since we need to subtract all large yellow metallic blocks and all gray metallic cylinders, we can see that there is one large yellow metallic block in the image. There are no gray metallic cylinders in the image. After removing the large yellow metallic block, the remaining objects are: 1. The small red metallic cube. 2. The small cyan sphere. So, there are 2 objects left.
Drop VFLs-L20-L28
In the image, there are two large yellow metal blocks and one small gray metallic cylinder. However, since the question asks to subtract all large yellow metal blocks and all gray metallic cylinders, we will remove these objects from the count. There is large yellow metal block. There gray metallic cylinder. After removing these objects, the remaining objects in the image are: small gray metallic cylinder (which was not subtracted because is not large yellow metal block). small gray metallic cylinder (which was not subtracted because it is not a large yellow metal block). So, after subtracting the specified objects, there are no objects left in the image.

---

Vision Token Dropping Math Vista-cont.
12
11
10
2
9
3
8
4
7
5
6
What time does the clock show?
The clock shows 4:30. The hour hand is pointing at 4, and the minute hand is pointing at 6, which represents 30 minutes past the hour.
Drop VFLs-L25-L28
The clock shows 4:30. The hour hand is pointing at 4, and the minute hand is pointing at 6, which represents 30 minutes past the hour.
Drop VFLs-L20-L28
The clock shows 3:30.
San
Organiam
Organism
Organism C
Organiem
Nordionni souk)
What would happen to organism if organism increased?
the food chain depicted in the image, Organism B (a dragonfly) is a predator of Organism A (a plant), and Organism C (a fish) is predator of Organism B. If the population of Organism increases, means there will be more predators for Organism C. Here's 'ss what could happen: 1.... summary, if the population of Organism B increases, it is likely that the population of Organism C would decrease due to increased predation pressure and possibly resource competition.
Drop VFLs-L25-L28
If the population organism B (the herbivore) increases, it would likely have positive impact on the population of organism C (the primary consumer). Here's why:1. **Increased Food Supply**: Organism B consumes organism G its food source. If there is an increase in the number of organism B, there will be more consumers for organism C. This could lead to a higher demand for organism C, potentially increasing its population. 2. "Competition"" However, this increase in organism might also lead increased competition among the organisms for
Drop VFLs-L20-L28
the population of organism B increases, it could have several effects on organism
C: 1. "Competition for Resources**. Organism B and organism C may compete for the same resources such as food, water, or space. An increase in the .... an increase in the population organism B would likely have negative impact on organism C, potentially leading to a decrease in the population of organism C.
25

---

C Implementation Details of VFL-LoRA
In this section, we provide detailed overview of our VLF-LoRA, including the training data, experimental setup, and implementation details.
Training Data. We leverage SAT [46], simulated spatial aptitude training dataset that includes both static and dynamic spatial reasoning tasks, comprising 175K question-answer (QA) pairs across 20K scenes. The dataset features three types of questions:
1. Relative Spatial Relations: determining whether object X is to the left, right, above, or below object Y, and identifying whether object A or B is closer to another object C.
2. Relative Depth:
assessing whether object X is closer to the camera than object Y by computing object distances within the simulator.
3. Counting: determining the number of specific objects in a scene.
After filtering out multi-image questions, retain a total of 151,724 training samples for use in experiments.
Experimental Setup and Implementation Details. We use popular open-source MILMs, Qwen2.5-VL [64] and (wwn2.55VL--77 [64], for our experiments. The experiments are conducted using a well-maintained GitHub repository'. We use 8 H20 GPUs, and the entire training process takes approximately 3 hours. For Qwen2.5-VL-7B, we apply LoRA fine-tuning to layers 10 through 17, as well as layers 20, 21, 22, and 23, while keeping all other parameters frozen. We the LoRA rank to 32, the scaling factor (alpha) to 64, and the dropout rate to 0.05. For the 3B variant, we also conduct Vision Token Swapping experiments on the Count task. In this setting, LoRA fine-tuning is applied to layers 16 through 26, and all other experimental configurations are kept identical to those used for the 7B model.
Detailed analysis on VFI.-LORA performance. Fig.3 in main paper shows that our VFL.-LORA method consistently outperforms standard LoRA and Reversed- VFL. on both Qwen2.5-VIL-3B and Owen2.5-VL-77 models across In-Domain and Out-of-Domail tasks, while utilizing significantly fewer trainable parameters. For Qwen2.5-VL-3B, VFL (Ours) achieves an average In-Domain score of 83.5% (vs. 72.7% for LoRA) and an Out-of-Domain score of 72.9% (vs. 71.8% for LoRA), using only 0.9% parameters compared to LoRA's 3.1%. Notably, it improves the Count task to 72.3% and MMMU to 47.3% On the Qwen2.5-VL-7B model, VFL (Ours) demonstrates advantages. It achieves an average In-Domain score of 85.0% (LoRA: 84.4%) and an Out-of-Domain score of 75.0% (LoRA: 74.3%). This superior performance is achieved with only 0.9% parameters, less than half of LoRA's 1.9%. Compared to Reversed- -FL. which uses a similar 0.9% parameters, our VFL method still shows clear improvements, particularly in the average In-Domain (85.0% vs 82.7%) and Out-of-Domain (75.0% vs 74.0%) scores. Key improvements include Count (72.6%) and MMMU (51.7%). These results underscore VFL-LoRA's effectiveness in enhancing model performance and generalization capabilities with remarkable parameter efficiency.
stronger
It is worth noting the performance on the Distance task, where VFL (Ours) (86.3% for 3B, 86.8% for 7B) does not consistently outperform the baselines (Reversed- VFL: 87 57.0% for 3B; LoRA: 88.3% for 7B). Our preliminary analysis suggests that the Distance * task might be predominantly language- driven. Tasks that rely more heavily on nuanced linguistic understanding or generation, rather than fine-grained visual feature manipulation targeted by VFI., may not benefit as significantly from approach. In such language-centee scenarios, methods allowing broader adaptation of language components might yield more competitive results.
D Implementation Details of VFL-Select
In this section, we provide detailed overview of our VLF-Select, including the data pool we collect, experimental setup, and implementation details.
httpe::///gtuuu.omm//// VL-Finetune
26

---

Data Pool. To simulate real-world data distributions, we begin with large-scale data pool. Specifi- cally, we construct this pool by merging all training data from Cambrian- 1 [58]. EAGLE-1 [51], and EAGLE-2 [51]. This diverse dataset provides broad coverage of scenarios encountered in practical applications.
Experiment Setup and Implementation Details. We use TinyLLaVA-L.SD-SiLLLII [75] as the MLLM for data classification. Although its model size is smaller than the commonly used 7B models, it outperforms LLaVA-1.5-7BB in our evaluations. Moreover, based on our observations, models of different sizes share the same ordering of vision function layers. This consistency allows the classifi- cation results obtained from the smaller model to be effectively transferred the training of larger models. TinyLLaVA- L....A 155--IigI consists of 22 layers. We compute R3 for k = 8, 10, 12, 16, 20 as defined in Equ.5. For each annotated sample, we group based on the value of &o that yields the highest RR, and then perform stratified sampling within each group according to Rz2. This ensures that samples of varying difficulty levels are evenly represented in the selected subset. Note that although we compute Rg six times, the computational cost remains low. This is because the dominant cost in MLLM inference comes from processing long sequences of image tokens. In our case, the image tokens are dropped after encoding, so computing multiple Rīg values incurs cost comparable to a single pass with the full image tokens. We perform the data classífication process on 16 H20 GPUs, which takes approximately 40 hours to complete. It is important to note that, as generalizable preprocessing step, this feature extraction only needs to be done once and can be reused across different training settings. Therefore, consider the computational cost to be acceptable. For the human-expert baseline, we follow the optimal dataset-type distribution reported in Cambrian [58] after multiple rounds of empirical tuning. Specifically, the composition is as follows: Language (21.00%). General (34.52%), OCR (27.22%), Counting (8.71%), Math (7.20%) Code (1 (1%) and Science (1%) ()).
Detailed analysis on VFL-Select performance. Tab.2 in main paper details performance com- parison between our VFL data selection strategy and a method based on human-expert curated data. This comparison spans various data subset sizes from 150k to 665k and covers range of task categories. A clear and consistent trend indicates that VFL (Ours) generally achieves superior or highly competitive results when compared to Expert, with this advantage often becoming more pronounced as the volume of data increases.
understanding tasks such as MME SEED', and GQA VFL (Ours) consistently surpasses Expert at all data quantities. For example, at the 665k data scale, VFL (Ours) scores 1526.3 on MME*. 68.2 on SEED'. and 64.1 on GQA which are notably higher than Expert 's respective scores of 1421.0, 1,000006 62.6, and 56.7. This pattern extends robustly into knowledge-intensive tasks (SQA', MMMUY AI2D) and OCR & Chart interpretation tasks (ChartQA, OCR, TextVQA), In the knowledge domain, VFL (Ours) demonstrates a particularly strong lead, with SQA' performance at 665k reaching 86.0 for VFL (Ours) versus 66.0 for Expert. Similarly, in OCR tasks, VFL (Ours) achieves 34.1 compared to Expert 's 25.1 at the 665k level, highlighting VFL's efficacy in selecting data pertinent to textual and schematic understanding. For instance, across these more reasoning- heavy categories, VFL (Ours) generally establishes a significant performance margin, suggesting its data selection is more effective for enhancing complex cognitive capabilities in models.
Across general visual-
The comparison in vision-centric tasks, specifically "Count" and "Dist," reveals a more varied landscape. For the "Count" task, Expert shows a slight advantage at the smallest data size (150k: 38.3 vs. VFL's 36.9). However, with increasing data, VFL (Ours) tends to match or slightly exceed Expert, exemplified by scores of 35.3 (VFL.) and 35.2 (Expert) at 665k. The "Dist" task shows fluctuating relative performance: VFL (Ours) leads at 150k (52.6 vs. 51.1) and 250k (55.5 vs. 46.8), while Expert is ahead at 350k (55.1 vs. VFL's 50.8). At the largest data point of 665k, both methods converge to an identical score of 48.2. It's noteworthy that the Oracle score for "Dist" (43.0) is unusually lower than many achieved scores, indicating potential specificities with this benchmark.
ScienceQA / is considered an outlier in our dataset. In our initial experiments, we retained at least the first 8 layers of vision tokens, based on the assumption that removing these early layers-which are responsible for fundamental visual recognition- would render the MLLM nearly non-functional. However, we found this assumption does not hold for ScienceQA'. Specifically, the model achieves reasonable performance using only the first 4 layers of vision tokens.
27

---

As result, in our data selection experiments, if no special handling is applied to the data required by ScienceQA' the final performance tends to plateau around 60. Remarkably, when we supplement the dataset with image-Vis VQ samples that do not rely on vision tokens, the performance improves significantly to 86. This suggests that, for ScienceQA', , vision tokens serve less as a source of rich visual information and more as a structural placeholder to maintain unified input format for training.
E Discussion, Limitation and Future Work
The advantages of VFL-LORA are most pro- nounced for tasks heavily reliant the specific visual functions localized in the targeted layers. For tasks that depend more on language priors or statistical biases rather than detailed visual features (e.g., the CV-Distance sub-task), VFL-LORA offers limited gains compared to standard LoRA, which updates all layers. This suggests that the effectiveness of VFL-LORA is conditional on the nature of the downstream task and its alignment with the functions of the selected VFLs. Scope of Investigated Visual Functions: The current research primarily focuses on four key visual functions: OCR, Object Counting. Object Recognition, and Object Grounding. MLLMs are designed to handle a much wider array of visual concepts and reasoning types. The existence, localization, and hierarchical order of VFLs for other, potentially more abstract or complex, visual functions remain be explored.
Limitations. Task-Spun Pask-Specifiic Benefits of VFL-LORA:
Future Works. Expanding the Repertoire of VFLs: Future studies could extend the VFL analysis to broader spectrum of visual questions beyond the four investigated, including more complex relational reasoning [73], open-world detection and recognition [48, 49], streaming video understand- ing [70], 3D scene understanding [50] or even visual hallucinations problems [13, 42]. This would provide more comprehensive map of functional specialization within MLLMss VFL--Guided MILLM Design: The insights from VFI. localization could directly inform the development of novel MILMM architectures. For instance, architectures could be designed with explicit layer specializations or rout- ing mechanisms guided by VFL principles, potentially leading to models that are more interpretable, efficient, and easier to train for specific capabilities.
28


Reviews:
[{"review_id": "uvmDgfpHCI", "reviewer": "Reviewer_5N5o", "summary": "The paper provides a nice experiment design to assess where reasoning takes place in VLMs. The token dropping experiment is simple yet effective in isolating and pinpointing the layers with the most influence on a vision subtask. Empirically, the authors test their hypothesis on 4 different types of tasks - OCR, Grounding, Counting, and Recognition, and present insights about the particular layer where the task is \"performed\". Next, they show how fine-tuning only the pinpointed layers improves performance, reaffirming the insight. Finally, the authors utilize this insight into data selection for pre-training VLMs and achieve almost similar performance with ~20% of the data.", "questions": "Refer Weakness.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Interpreting the inner workings of VLMs is an important topic with significant interest recently.\n2. The paper is very clear with a simple but intuitive methodology.\n3. Empirical results are exhaustive with a wide coverage of task types and datasets.\n4. Pre-training data selection is a smart application.\n\nWeakness:\n\n1. Only low-level tasks considered: Currently, the tasks are simple enough to be effective through direct prompting strategies. A lot of the insights presented are novel but do not take into account the high-dimensional vision-language interplay, which elicits the majority of VLM reasoning. For example, for each task, the prompt itself is not the most important aspect of the representations - OCR, recognition, etc., can be performed directly through probing vision tokens. It would be significantly impressive if these insights hold onto more difficult tasks like VQA (like on datasets like A-OKVQA), where VLMs truly shine.\n\n2. Building on this, I would be excited to see how prompting techniques change the visual functions across layers. For example, consider in-context learning, where $k$ exemplars are pre-appended to the prompt. Now there are $k$ times more visual tokens - how does the performance change then? How about prompts with longer context? \n\n3. The authors should also evaluate their approach on step-by-step prompting techniques like Chain-of-thought (CoT) reasoning, where a big visual reasoning task is broken down into simpler steps. If the author's findings confirm that each step of CoT activates similar layers, it will be a very deep contribution as multiple future works like PPO/GRPO can benefit from layer-specific policy learning.\n\nTypos:\n\nSection 2.1 \"Multi-Model Large Language Models\"", "was_revised": true, "final_justification": "I have read the rebuttals, and I maintain my ratings.", "rebuttal": "We thank the reviewer for the insightful comments and questions. Our response is as follows:\n\n> ***W1. VFL Still holds on high-dimensional vision-language task***\n> \n\nFollowing your suggestion, we conducted both qualitative and quantitative vision token dropping experiments on A-OKVQA, a dataset that includes questions requiring reasoning over various types of knowledge such as commonsense, world knowledge, and visual understanding.\n\n| Qwen-2.5-VL-7B | Full | Drop OCR-VFL (18-28 layers) | Drop OCR-Gounding-Count-VFL (12-28 layers) | Drop (6-28 layers) |\n| --- | --- | --- | --- | --- |\n| A-OKVQA | 89.25 | 88.94 | 76.68 | 57.90 |\n\n- **Foundational VFLs remain critical even in complex tasks that need high-dimensional vision-language interplay.**  As shown above, removing **OCR-VFL** has no impact on performance, as A-OKVQA does not include OCR-related questions. In contrast, dropping **Grounding-Count-VFL** causes a 10% drop, with errors concentrated in questions like *“How many people will dine at this table?”*, which require counting objects (e.g., bowls) before reasoning. Even after partially removing **Recognition-VFL**, the model correctly answers questions like *“What does the man who sits have trouble doing?”* (*“Walking”*), relying mainly on language priors. In summary, **VFLs remain active and functionally aligned** even in complex tasks requiring multi-step vision-language reasoning.\n- **Atomic VFLs can integrate with language priors to support reasoning.**  These results are also intuitively reasonable, as **VLMs tend to process visual and language cues relatively independently**. For example, we newly introduced two VFLs—Emotion Recognition and Object Relation (see detailed information in our response to Reviewer Sn1h)—and found that the visual version of the *Sentiment* task, which has been shown in [1] to be handled in the middle and deeper layers of LLMs, is instead processed in the early layers of VLMs. This discrepancy not only highlights the fundamental differences between LLMs and VLMs, but also underscores the significance of identifying and analyzing **Vision Functional Layers**.\n\n> ***W2. VFL Still holds on in-context learning and long-context:***\n>\n\nFollowing your suggestion, we used VL-ICL Bench [2] to evaluate in-context learning. Specifically, we tested CLEVR Count and TextOCR to verify the Count and OCR VFLs, respectively, under zero-shot, 1-shot, 4-shot, and 8-shot settings. Notably, the 8-shot setup exceeds 10,000 tokens, approaching Qwen-VL’s pretraining context length. CLEVR Count requires models to answer questions like “How many red objects are there in the scene?” by learning from in-context examples. TextOCR evaluates OCR ability by asking the model to output text from red-highlighted regions in images. Zero-shot results are obtained via direct text prompting without any examples.\n\n| **CLEVR Count** | zero-shot | 1-shot | 4-shot | 8-shot |\n| --- | --- | --- | --- | --- |\n| Qwen-2.5-VL | 55.5 | 60.0 | 61.5 | 59.0 |\n| Drop-OCR-VFL(20-28) | 55.0 | 60.5 | 60.5 | 59.0 |\n| Drop-Count-VFL(12-28) | 33.5 | 31.5 | 25.5 | 22.0 |\n|  |  |  |  |  |\n| **TextOCR** | zero-shot | 1-shot | 4-shot | 8-shot |\n| Qwen-2.5-VL | 46.5 | 48.0 | 47.5 | 49.0 |\n| Drop-OCR-VFL(20-28) | 12.0 | 11.5 | 9.0 | 5.5 |\n- **VFLs persist under both in-context and long-context settings.** As shown above, VFLs remain active in the in-context learning setup, and their layer-wise behavior is consistent with that observed in the zero-shot setting. Moreover, under the long-context condition (e.g., 8-shot), VFLs continue to function as expected, further confirming their stability across different prompting regimes.\n- **VFLs activate whenever a visual function is needed, not just during answer generation.** We further tested dropping only the vision tokens in the context while keeping those in the question. The performance drop is same as dropping all visual tokens, suggesting that both **task understanding** and **answering** rely on the same function layer. This confirms that **in-context prompts** and **long-context inputs** do not change the functionality or layer position of VFLs.\n\n> ***W3. Application in CoT and GRPO:***\n>\n\nWe sincerely appreciate the reviewer’s insightful suggestion, which closely resonates with our own perspective.\n\n- **VFL in CoT.** During **Chain-of-Thought reasoning**, when a generated token demands a specific visual skill (e.g., OCR), the same **OCR-VFL** is activated as expected (As discussed in W2). We further observed this behavior in **math reasoning tasks**. Using the RL-trained version of Qwen2.5-vl-7B with enhanced reasoning capabilities, we conducted case analyses. When the reasoning process required a specific vision function, we fixed the random seed, reverted, and dropped the corresponding vision function layer identified earlier. This caused the model to lose that specific capability, demonstrating that **VFL consistency persists not only during chain-of-thought (CoT) generation but also remains preserved in the RL-enhanced base model**.\n- **VFL in GRPO.** We agree that applying VFLs in GRPO is a very promising direction. Current GRPO rewards focus on format and accuracy, lacking supervision or interpretability over intermediate reasoning steps (e.g., within `<\\think>`). With our layer-wise localization of vision functions, the model could autonomously decide which function to invoke next (e.g., `<\\OCR>`, `<\\Count>`, `<\\Factual>`), skipping irrelevant layers to save computation. Moreover, our method could help verify if the model truly activates the intended function, improving interpretability. However, we consider this beyond the scope of the current work but welcome further discussion and look forward to your response.\n\n> ***Typos:***\n> \n\nWe will correct the typos and incorporate the above content into the revised version of the paper. Once again, we sincerely thank you for providing such valuable and constructive suggestions.\n\n[1] Not All Layers of LLMs Are Necessary During Inference. 2024\n\n[2] VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning ICLR 2025"}, {"review_id": "8VvKsv1lLe", "reviewer": "Reviewer_Sn1h", "summary": "This paper investigates the internal \"black box\" mechanisms of visual processing in MLLMs. The authors propose an analytical framework called Visual Token Swapping, which modifies targeted KV cache entries to precisely identify the function of specific layers during decoding. A core discovery is the existence of Vision Function Layers, which are narrow blocks of 2-3 layers where specific visual functions like counting or OCR are exclusively handled. These VFLs exhibit a consistent hierarchical order across different MLLMs. Leveraging these insights, the paper introduces practical applications for model optimization. VFL-LORA applies LoRA only to relevant VFLs, outperforming full LoRA in out-of-domain generalization while using fewer parameters. Furthermore, VFL-select, an automated data curation strategy, surpasses human experts and achieves 98% of full-data performance with only 20% of the original dataset. This work provides a deeper understanding of MLLM visual processing, enabling more efficient and interpretable models.", "questions": "1. The authors could incorporate a wider range of visual functions into the framework, as MLLMs exhibit complex visual capabilities and therefore require a more comprehensive analysis.\n\n2. The authors should design corresponding benchmarks to evaluate different visual functions, since current MLLM benchmarks usually rely on multiple visual capabilities.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is clear, structured, and effectively explains the experimental design and results. The writing is concise yet comprehensive.\n2. This paper provides a novel perspective on understanding MLLMs by introducing the concept of \"Vision Function Layers\". This perspective offers a novel approach to explain how MLLMs process visual inputs and presents new insights into how these models function internally. \n\nWeaknesses:\n1. Lack of technical contributions. While this paper introduces an interesting framework, it does not present sufficient technical contributions. The primary contribution lies in the analysis of vision functions across layers, rather than offering technical innovations that could push the field forward. This limits the technical impact of the paper.\n2. Limited scope of visual functions. The paper only considers four visual functions (recognition, counting, grounding, and OCR), which is too narrow. There are many other important visual reasoning tasks, such as object relationship inference and emotion recognition, that are not covered. This limited scope means the analysis may not fully capture the complexity of visual functions in MLLMs.\n3. The analysis is based solely on the contribution of individual layers, without considering the interaction between multiple layers. In reality, the functionality of different layers may interact and jointly influence the model's performance. By ignoring this interaction, the study may not provide a complete understanding of how visual features are processed and integrated across layers.\n4. The evaluation benchmarks used in the paper do not rely solely on single visual functions, which may undermine the accuracy of the results in assessing specific vision functions.", "was_revised": true, "final_justification": "The authors' rebuttal has largely addressed my concerns. Consequently, I will adjust the rating to borderline accept.", "rebuttal": "We appreciate the reviewer’s comments. For better clarity, we respond in a reorganized order rather than the original question sequence.\n\n> ***W2, Q1. More visual functions.***\n>\n\n***More visual functions can be easily identified using our pipeline.** W*e truly appreciate your positive assessment of our work on VFLs and the framework we proposed. One of the key contributions of our paper is **the discovery and localization framework for VFLs**. With the help of this framework, one can easily identify and analyze any vision function of interest, detailed as follows:\n\n- **Two additional VFL.** As an additional demonstration, we present two more VFLs, namely the **Object Relation Layer** and the **Emotion Recognition Layer**, as suggested by the reviewer. We simply follow the procedure outlined in **Section 2.3** and replace the paired images accordingly to identify any vision function layer (VFL) of interest. Specifically:\n    - **(1) Object Relation Pairs:** Adapted from the CV-Bench Depth Order benchmark [1], this task requires the model to determine which of two distinct objects is closer to the camera. The **change rate** is defined as the proportion of prediction changes after swapping vision tokens with those from an empty image. For **Qwen2.5-VL-7B**, the change rate peaks at **layer 20**, indicating that object relation reasoning is primarily handled at this specific layer—consistent with the patterns observed in other VFLs.\n    - **(2) Emotion Recognition Pairs:** We first use **Qwen3** [2] to generate 200 scene descriptions involving different emotions, including *joy, sadness, anger, fear,* and *disgust*. Based on these descriptions, we use **FLUX.1** [3] to generate corresponding images. We then randomly pair images representing different emotions and formulate multiple-choice questions to query the model. The **change rate** is calculated as the proportion of prediction changes after swapping vision tokens between images depicting different emotions. For **Qwen2.5-VL-7B**, the change rate peaks at **layer 10**, indicating that emotion recognition is primarily handled at this specific layer—consistent with the patterns observed in other VFLs.\n    \n    |  | Peak Layer | Peak Value | Other Layer Average |\n    | --- | --- | --- | --- |\n    | OCR-VFL | 22 | 92.8% | 0% |\n    | Object-Relation-VFL | 20 | 81.5% | 7% |\n    | Grounding-VFL | 18 | 100% | <5% |\n    | Count-VFL | 14 | 87.4% | <5% |\n    | Emotion Recognition-VFL | 10 | 91% | <5% |\n    | Recognition-VFL | 0-8 | 90% | 37% |\n- **Any vision function of interest.** It is important to note that all models used in the above process are publicly available and open-sourced. By leveraging an LLM to generate prompts and using an image generation model to create paired images, **this pipeline—as described in the main paper—provides a general and reproducible approach to localizing any vision function of interest.**\n- **More analysis for different VLFs.** We summarize the layer-wise localization of the six identified VFLs on **Qwen2.5-VL-28-layer**, along with their **change rate distributions** across all layers. The results (table above) show that for each vision function, the corresponding **VFL** exhibits a distinct peak in change rate, while the function remains largely inactive in other layers—indicating a clear specialization of layers for different visual capabilities. As indicated in the supplementary material, we consistently observed this phenomenon across all tested MLLMs. We hope that the additional VFLs we provided, along with the proposed pipeline for broadly discovering VFLs, have effectively addressed your question.\n\n> ***W4, Q2. Each experiment is designed to ablate a single visual capability:***\n>\n\nIn this work, we propose two approaches for analyzing and localizing VFLs: **vision token swapping** on our custom-designed benchmarks, and **vision token dropping** on carefully-selected MLLM benchmarks. They are all designed to ablate a single visual capability.\n\n- ***Vision token swapping ablates a single visual capability using paired images.** W*e construct **paired images that differ only in one specific visual function** being tested. When vision features are swapped at a particular layer and the model's prediction changes to match that of the swapped image, the **change rate** indicates that the corresponding visual capability is executed at that layer. Since the paired images are carefully designed to differ solely in the targeted visual function, this setup effectively rules out interference from multiple visual capabilities. For example, in the **OCR pairs**, we vary only the text content, while keeping the font, font size, font color, and text position identical. This ensures that only the OCR capability is being evaluated, without interference from other visual factors.\n- **Vision token dropping ablates one core visual capability through carefully selected dataset types, validated by experimental results.**\n    - The commonly used MLLM benchmarks (Table 1) we selected are each primarily designed to evaluate one core **visual capability**. This categorization is also supported by [1], which groups these datasets into four categories: *General & Knowledge*, *Recognition*, *Spatial*, and *OCR & Chart*.\n    - **The sharp performance drop (rather than a gradual decline) in Table 1 further validates the necessity of only one core visual function required by these benchmarks.**  Specifically, the accuracy on these benchmarks does **not gradually decline** as vision tokens are dropped; instead, a **sharp performance drop** is observed precisely when the vision tokens from the corresponding **VFL** are ablated. *(As a comparison, see our response to Reviewer 5N5o regarding A-OKVQA, where performance gradually declines as multiple VFLs are dropped, due to the task requiring a combination of vision functions.) For example, OCR-related benchmarks show a sharp drop in performance after removing the OCR layer (layer 22 in Qwen2.5-VL-7B), while others remain unaffected. Similarly, each benchmark only degrades after its corresponding VFL is dropped. In contrast, general knowledge tasks rely less on visual input and perform well as long as basic recognition VFLs are intact. These results provide strong and comprehensive support for our formulation of Vision Function Layers (VFLs).*\n\n> ***W3. Interaction between different VFLs.***\n>\n\n- **The effects of different VFLs are largely independent of one another.** Thank you for raising this suggestion—this is indeed a very interesting direction. In our preliminary exploration, we observed that the activation of different VFLs tends to follow a **linear additive pattern**, with each VFL operating **relatively independently**. As discussed in the main paper, specific vision functions are localized to **dedicated layers**, further highlighting the importance of understanding each **atomic VFL** individually.\n- **Atomic VFLs can be interacted and composed to handle complex tasks.** We have also explored the application of different VFLs in Chain-of-Thought (CoT) and GRPO settings in our response to **Reviewer 5N5o**—please refer to that response for more details. While we have observed instances where multiple VFLs are engaged within a single task, developing **systematic methods to explore their interactions**—such as building complex tasks that require coordinated visual functions and solving them step-by-step—remains a promising direction for future work. In this paper, our focus is to **discover and define** the concept of VFLs—a previously overlooked but fundamental phenomenon in MLLMs.\n\n> ***W1. Technical Contributions:***\n>\n\n- **To the best of our knowledge, we are the first to discover, define, and provide a systematic toolset for understanding Vision Function Layers (VFLs)** in MLLMs. This work establishes VFLs as a fundamental structural and functional unit in vision-language processing—an aspect previously overlooked in the literature.\n- **Our contributions span from VFL discovery framework and foundational insights to practical applications.** Building upon this foundation, we introduce two practical extensions: **VFL-LoRA**, a parameter-efficient fine-tuning method targeting specific VFLs, and **VFL-Select**, a technique for automatic dataset construction based on visual function needs. **It even outperforms methods specifically designed for these particular tasks!**\n- More broadly, our work provides an end-to-end perspective that spans the entire MLLM development process—from data selection and model training to the interpretation of internal mechanisms. Concretely, we address key questions: how to select training data, how to train models based on targeted visual functions, and how to understand the model’s internal behaviors. This comprehensive understanding not only clarifies how current MLLMs function but also offers practical insights for developing the next generation of more effective and interpretable multimodal models.\n\nWe believe these contributions are technically substantial, novel, and impactful. We hope the reviewer finds them meaningful. \n\n[1] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs.\n\n[2] Qwen3 Technical Report.\n\n[3] FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space."}, {"review_id": "FcCve3z4fd", "reviewer": "Reviewer_TCaw", "summary": "The paper investigates vision-related functional decoding in Multimodal Large Language Models (MLLMs), revealing that visual functions like counting, grounding, and OCR are localized within narrow decoder layer blocks, termed Vision Function Layers (VFLs). \n\nUsing a novel Visual Token Swapping framework, the study finds consistent VFL depth and order across MLLMs, aligning with human visual processing (recognition → counting → grounding → OCR).\n\nThe insights enable practical applications: VFL-LoRA, a targeted fine-tuning method, outperforms full LoRA by focusing on relevant VFLs.", "questions": "1. Experiments focus on basic visual functions (such as recognition and counting), whereas there are numerous complex and diverse tasks in the real world. The VFL theory has not been validated in these real-world tasks.\n2. This paper proposes the visual token swapping and token dropping strategy to validate the VLF theory. However, many papers [1] drop vision tokens directly, according to the attention map or other strategies. Since these visual tokens can be directly discarded, can your operation still prove your theory?\n3. Please provide a more detailed explanation of how  visual token swapping and token dropping are specifically implemented.\n\n\n[1] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-PLay Acceleration for VLLM Inference", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The Visual Token Swapping and Dropping methods provide a systematic way to probe layer-specific visual functions in MLLMs, enabling precise localization of functions like OCR, counting, and recognition to narrow layer blocks. This framework addresses the \"black box\" problem of MLLMs and sets a new standard for interpretability research.\n2. VFL-LoRA reduces tunable parameters by ~50% while improving out-of-domain generalization compared to standard LoRA\n3. The study validates findings across 8+ benchmarks (e.g., ScienceQA, MMMU) and uses controlled image pairs to isolate single visual functions, ensuring robust results.", "was_revised": true, "final_justification": "The author's response effectively addressed my concerns. Based on the issues from other reviewers and rebuttal from author, I have decided to improve my rating.", "rebuttal": "We thank the reviewer for the insightful comments and questions. Our response is as follows:\n\n> ***Q1. VFL has been successfully validated on real-world tasks, demonstrating its practical effectiveness.***\n>\n\n- **We systematically validated the presence of Vision Function Layers across diverse real-world tasks.**\n    - In Table 1 of the main text, we validate the presence of Vision Function Layers across nine widely used MLLM benchmarks—spanning General & Knowledge, Recognition, Spatial, OCR, and Chart categories—thereby encompassing a comprehensive range of vision functions. This broad empirical coverage has also been recognized by other reviewers, who noted that the results are “exhaustive with a wide coverage of task types and datasets.”\n    - Our pipeline is extensible to a wide range of vision functions, as discussed in our response to Reviewer Sn1h, W2, Q1 (\"More visual functions\"), particularly in the paragraph of W2, where we demonstrate how to systematically identify additional Vision Function Layers using our proposed framework.\n- To further demonstrate the applicability of our approach to real-world tasks, we evaluate our method under the following scenarios:\n    - **Fundamental vision capabilities serve as the essential foundation for tackling complex visual reasoning tasks.**  We have identified six distinct Vision Functions—namely OCR, Object Relation, Grounding, Count, Emotion Recognition, and Recognition—that collectively encompass a wide spectrum of common visual capabilities. The compositional integration of these atomic vision functions facilitates the resolution of more complex and higher-order tasks.\n    - **VFLs remain valid under different prompting techniques**, including in-context learning, chain-of-thought (CoT) prompting, and long-context settings. For details, please refer to our responses to Reviewer 5N5o: W2 *“VFL Still holds on in-context learning and long-context”* and W3 *“Application in CoT and GRPO.”*\n    - **VFLs also hold when the input modality is video rather than static images.** For supporting evidence, please refer to our response to Reviewer 5Arb, Q1 *“VFL still present on video benchmark.”*\n\n\n> ***Q2. The concept of Vision Function Layers (VFL) offers a theoretical justification for inference acceleration techniques that leverage selective vision token dropping.***\n>\n\n- **Visual tokens can NOT be directly discarded.** For token reduction methods like FastV, at least 25% of the tokens must be retained to maintain final performance; discarding all tokens directly results in significant performance degradation.\n- **VFL addresses fundamentally different concerns compared to token reduction techniques.** VFL focuses on the **vertical** behavior of vision tokens across each Transformer layer in MLLMs, analyzing the functional role of each layer by dropping ALL tokens at specific layers. In contrast, token reduction methods address the **horizontal** redundancy among vision tokens within a single layer, exploiting this redundancy to selectively drop tokens and accelerate inference.\n- **VFL offers a theoretical justification for *acceleration methods based on* token reduction.** For token reduction methods such as FastV, achieving optimal pruning efficiency requires carefully selecting parameters such as the starting Transformer layer for pruning and the proportion of tokens to drop. These hyperparameters are typically determined through empirical tuning and are not easily adaptable to different datasets or task types. In contrast, our discovery of Vision Function Layers (VFLs) provides a principled framework to guide such decisions. For example, as shown in Table 1 of our paper, for knowledge-based tasks, all vision tokens can be safely discarded starting from layer 10, offering clear guidance for layer-wise pruning strategies.\n\n> ***Q3.* Detailed explanation of Vision Token Swapping and Vision Token Dropping**\n>\n\nBoth Vision Token Swapping and Vision Token Dropping are implemented at the decoding stage of the MLLM's generation process. Specifically, during inference, we operate directly on the **key-value (KV) cache** corresponding to the image tokens.\n\n- **Vision Token Swapping:** For a given layer, we replace the KV cache entries corresponding to the image tokens of the original input with those from a paired image. This intervention effectively simulates how the model would behave if it were processing a different visual input at that specific layer, while keeping all other layers and tokens unchanged.\n- **Vision Token Dropping:** Similarly, we remove the KV cache entries corresponding to the vision tokens at a given layer, effectively nullifying the visual input at that layer while leaving the rest of the computation unchanged. This allows us to isolate the contribution of visual information at different layers.\n\nThese interventions are **causal** and **non-invasive** with respect to the rest of the model pipeline, and enable precise probing of where and how visual functions are executed across layers."}, {"review_id": "UfIfU32R2U", "reviewer": "Reviewer_5Arb", "summary": "- The paper presents an analysis study about the importance of different LLM layers for various VQA tasks using MLLMs.\n- The authors find that for different tasks, different layers are important and term those as the “Vision Function Layers” (VFL) for the Qwen-2.5-VL model.\n- The authors also try finetuning a qen2.5-vl model with LoRA applied to only the VFLs and find it to perform at par or better than applying LoRA to all layers.\n- Finally, they select samples with the highest value based on the probability based heuristic at the most important VFL for a given domain and show that using a subset of the dataset can result in fair performance", "questions": "- How does the findings change for video benchmarks? do we see the presence of VFL layers for video inputs too?\n- How does VFL transfer to encoder-free MLLMs like chameleon or EVE-2/EVE?\n- In the fig. 3 caption, the authors state: “contrasting with strong language priors for Distance and Relation.”, what is the basis of saying this? Relation and distance tasks do require good spatial reasoning abilities.", "limitations": "The authors do not discuss any limitations but a few that come to mind:\n- No analysis for encoder free MLLMs or video benchmarks.\n- No analysis on different MLLM architectures (including MoEs).", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- Good analysis which provides insights into what layers are important for different tasks.\n- The VFL select method could be useful for future methods.\n\n### Weaknesses\n\n- There is little reference to previous works in the experiment section in order to support their findings. For example, previous works (like FastV and [vision-of-vlm](https://vision-of-vlm.github.io/)) have found that the attention scores to visual tokens is the highest in the earlier layers, so why do changing the visual tokens there does not affect the answer? This finding is counter-intuitive and should be explained.\n- There is little effort to explain the findings of the paper about the presence of VFL layers, I would have liked the presence of some hypothesis.\n- Analysis is only done with qwen2.5-vl model which is very strong model in itself. I would like to see how the findings change with different MLLMs like LLaVA-1.5, Cambrian-1 (multi-encoder approach), Intern2.5-VL and also an MoE based model like Deepseek-VL2. Because this is an analysis driven paper, multiple models should be analyzed to make the findings well grounded and widely applicable.\n- How does VFL-select compare to methods using the probability of answers from the final LLM layer and also after the head? that is an important baseline which is missing. If the results are similar, VFL-select has lesser value with much analysis to be done for every different family of model.\n- The finding about MMMU and ScienceQA-Img is well known (as found in MM-Star, I would have liked to see that citation and also evaluation on that benchmark).", "was_revised": true, "final_justification": "I believe the findings from this paper are helpful to the community and it may help in designing better MLLMs or even training a few layers for some tasks in future MLLMs.", "rebuttal": "We thank the reviewer for the insightful comments and questions. Our response is as follows:\n\n> ***Q1 VFL still present on video benchmark:***\n>\nWe further validated the presence of VFLs on Video-MME, which includes a variety of fine-grained tasks. Following the vision token dropping protocol described in Table 1 of the main paper, we applied this analysis to Video-MME. The results are presented below.\n\n| Video-MME | OCR Set | Spatial Perception | Counting | Object Recognition |\n| --- | --- | --- | --- | --- |\n| Full | 68.4 | 80.0 | 40.0 | 64.3 |\n| Drop-OCR-VFL | 38.6 | 83.3 | 38.4 | 64.5 |\n| Drop-Grounding/Spatial-VFL | 30.3 | 56.7 | 40.0 | 64.3 |\n| Drop-Count-VFL | 30.1 | 44.3 | 31.4 | 61.2 |\n| Drop-Recognition-VFL | 30.3 | 36.7 | 27.2 | 32.1 |\n\nAs shown in the table above, **dropping the corresponding VFL leads to a sharp performance drop on the associated task set**, while performance on other task sets remains unaffected—consistent with the patterns observed in image-based benchmarks. For example, in the **OCR task** within Video-MME, performance drops significantly from **68.4 to 20.0** after removing the **OCR-VFL**.\n\n- **In summary, our results demonstrate that VFLs are not limited to static image inputs, but also extend to video-based tasks.** The persistence of VFLs in MLLMs when processing video inputs can be attributed to the fact that, fundamentally, a video can be viewed as a **long-context image**. Since extending the context length does not alter how MLLMs process visual tokens, the underlying behavior of VFLs remains unchanged. *(For more analysis on in-context and long-context settings, please refer to our response to Reviewer 5N5o.)*\n\n> ***Q2,W3,W2 VFL still present on Multi-encoder/Encoder-free/MoE-based MLLM:***\n>\n\nWe further validated the presence of VFLs across a diverse set of MLLM architectures, demonstrating the generality of our findings. The results are presented below.\n\n| Multi-Encoder | MMMU | POPE | CVBench | ChartQA |\n| --- | --- | --- | --- | --- |\n| Cambrian-1 | 37.7 | 85.6 | 65.4 | 72.7 |\n| Drop-OCR-VFL | 40.7 | 84.9 | 66.2 | 17.8 |\n| Drop-Grounding/Spatial-VFL | 37.4 | 84.5 | 57.3 | 14.0 |\n| Drop-Recognition-VFL | 36.6 | 17.6 | 51.0 | 13.1 |\n\n| Encoder-free | MMMU | POPE | CVBench | ChartQA |\n| --- | --- | --- | --- | --- |\n| EVE-2 | 39.3 | 87.6 | 66.4 | 73.9 |\n| Drop-OCR-VFL | 39.3 | 85.6 | 66.2 | 20.5 |\n| Drop-Grounding/Spatial-VFL | 38.2 | 86.4 | 43.0 | 17.3 |\n| Drop-Recognition-VFL | 38.2 | 11.5 | 37.2 | 14.3 |\n\n| MoE-Based | MMMU | POPE | CVBench | ChartQA |\n| --- | --- | --- | --- | --- |\n| Deepseek-VL2 | 40.7 | 88.7 | 69.5 | 86.0 |\n| Drop-OCR-VFL | 40.7 | 88.7 | 69.2 | 40.4 |\n| Drop-Grounding/Spatial-VFL | 39.5 | 88.4 | 33.5 | 23.5 |\n| Drop-Recognition-VFL | 40.7 | 23.5 | 33.5 | 22.4 |\n\nAs shown in the table above, we examined representative models with different architectures and conducted experiments on datasets from four task categories. The results demonstrate that **VFLs are consistently present across MLLMs with diverse architectural designs**.\n\n- **In summary, our findings regarding VFLs are robust and generalizable across model families.** We chose **Qwen2.5-VL** as the primary model for our main experiments because it is one of the most widely used, high-performing, and representative open-source MLLMs available. Additionally, in the Appendix, we provide experimental results on other model families (e.g., **LLaVA**), and the **VFL findings remain consistent** across these different MLLM architectures.\n- **We hypothesize that the emergence of VFLs** in MLLMs reflects an internal hierarchical structure of visual processing. This is analogous to the human visual system, where perception proceeds through layered stages—from low-level feature extraction to high-level semantic reasoning—with each layer supporting the next. Similarly, MLLMs may allocate specific visual functions to distinct layers, such that a given capability (e.g., OCR or object counting) only emerges after prerequisite features are computed. This view is also consistent with findings in LLMs [1,2], where textual information is processed in a **coarse-to-fine**manner across layers. **We believe this hypothesis provides a principled explanation** for the layer-specific specialization observed in VFLs.\n\n> ***Q3,W5 Language priors in CVBench and Results in MM-Star***\n>\n\n- The statement **\"strong language priors for Distance and Relation\"** in **Figure 3** refers to the **Distance** and **Relation** subsets of CV-Bench. For these two tasks, the model achieves significantly higher-than-random accuracy even **without access to visual tokens**. For example, on the **Distance** task, the MLLM exceeds random guess performance by over **10 percentage points** without any visual input. This indicates that the model leverages strong **language priors**, meaning it can often answer these questions correctly **without actually interpreting the image**.\n- We further validated the presence of VFLs in MM-Star\n\n|  | MM-Star | SEED | ScienceQA | MMMU |\n| --- | --- | --- | --- | --- |\n| Full | 63.9 | 77.6 | 87.2 | 50.7 |\n| Drop-OCR-VFL | 64.0 | 77.5 | 87.4 | 50.6 |\n| Drop-Recognition-VFL | 57.6 | 52.4 | 77.3 | 45.8 |\n\nSince MM-Star is a composite benchmark constructed from multiple datasets (e.g., SEED, ScienceQA, MMMU, etc.) with samples filtered to exclude those that can be answered without visual information, we also report performance on the original datasets for comparison. As shown, the VFL phenomenon still holds on MM-Star. For example, since the constituent datasets of MM-Star do not require OCR capabilities, dropping the OCR-VFL has no impact on performance, consistent with our expectations.\n\n> ***W4 VFL-Select Baseline***\n>\n\nUsing only the final-layer output for selection performs on par with random choice, showing no significant advantage. We observed that relying solely on the final layer introduces dataset selection bias, causing performance to fall below that of random selection. This observation is corroborated by the results in [3].\n\n> ***W1  Clarity on the VFL***\n>\n\n- **Our findings are consistent with prior studies on attention in early layers.** Thank you for the insightful comment. Our findings do *not* contradict prior works such as **FastV**. As these works [1,2] have noted, early layers primarily perform **feature extraction and fusion**, which explains the high attention scores to visual tokens in shallow layers. This is a natural part of the perception pipeline and does not conflict with our results.\n- **Our analysis focuses on vision function rather than attention patterns.** While early layers may attend strongly to visual tokens, attention magnitude does not imply that a layer is functionally responsible for the final prediction. Our **VFL framework** is based on **causal intervention**—we perturb visual tokens at a specific layer while keeping others fixed. If swapping or dropping tokens at that layer does not change the output, it indicates that the required visual function is not processed there. The fact that modifying early-layer tokens does not affect the answer supports this view: these layers handle general encoding, while task-specific visual reasoning emerges in deeper layers. We will clarify this distinction and cite related works in the revised version.\n\n[1] Not All Layers of LLMs Are Necessary During Inference. 2024\n\n[2] Zero-Shot Vision Encoder Grafting via LLM Surrogates ICCV2025\n\n[3] Concept-skill transferability-based data selection for large vision-language models."}]

Author rebuttal:

Meta-review: This paper introduces Vision Function Layers (VFLs) in MLLMs, showing that core visual functions (e.g., recognition, counting, OCR, grounding) are localized in narrow blocks of layers. Using vision token dropping and swapping, the authors provide a causal framework for probing visual functions, and demonstrate practical applications through VFL-LoRA (efficient fine-tuning) and VFL-Select (data selection).

Reviewers initially raised concerns about limited scope, technical novelty, and applicability to complex tasks. However, the rebuttal by the authors added substantial new experiments, including A-OKVQA, in-context learning, CoT reasoning, video benchmarks, and diverse architectures (multi-encoder, encoder-free, MoE). These additional inputs demonstrated the robustness and generality of the findings, and convinced the reviewers on the significance of this submission.

The ACs like the findings of this work regarding the vision function layer and the application of this discovery for improve the parameter-efficient fine-tuning and data selection for model training. While the work is more analytical than architectural, its clarity and novelty make it a valuable and timely contribution to MLLM interpretability and efficiency. With reviewers converging to borderline accept or higher (one raising to full accept), the ACs recommend acceptance.
Decision: Accept (poster)