Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
Authors: Shiqi Chen
Jinghan Zhang
Tongyao Zhu
Wei Liu
Siyang Gao
Miao Xiong
Manling Li
Junxian He
Venue: ICML 2025 poster (2025)
Keywords: Interpretability
Model Merging
Vision Language Models
Primary area: deep_learning->large_language_models
TL;DR: We explore whether model merging can be used to improve the reasoning ability of VLMs and we further adopt the model merging as an interpretability tool to analyze the inner workings of VLMs.

Abstract: Vision-Language Models (VLMs) combine visual perception with the general capabilities, such as reasoning, of Large Language Models (LLMs). However, the mechanisms by which these two abilities can be combined and contribute remain poorly understood.
In this work, we explore to compose perception and reasoning through model merging that connects parameters of different models.  
Unlike previous works that often focus on merging models of the same kind, we propose merging models **across modalities**, enabling the incorporation of the reasoning capabilities of LLMs into VLMs. 
Through extensive experiments, we demonstrate that model merging offers a successful pathway to transfer reasoning abilities from LLMs to VLMs in a **training-free** manner.
Moreover, we utilize the merged models to understand the internal mechanism of perception and reasoning and how merging affects it. We find that perception capabilities are predominantly encoded in the early layers of the model, whereas reasoning is largely facilitated by the middle-to-late layers. After merging, we observe that all layers begin to contribute to reasoning, whereas the distribution of perception abilities across layers remains largely unchanged. These observations shed light on the potential of model merging as a tool for multimodal integration and interpretation.

Bring Reason to Vision:
Understanding Perception and Reasoning through Model Merging
Shiqi Chen *1
Jinghan Zhang" 22
Tongyao Zhu Wei Liu Siyang Gao
Miao Xiong? Manling Li*
Junxian He?
Abstract
1. Introduction
Multimodal reasoning is crucial for a variety of important applications, such as interpreting charts and figures in sci- entific publications and government reports. Despite the great successes of Vision- Langua-L Models (VLMs) in tasks requiring perceptual and linguistic integration (Li al., 2024; Liu et al., 2024; Bai et al., 2023; Wang al., 2024b), these models struggle with complex multimodal reasoning tasks (Lu et al., 2024; Zhang et al., 2024b). This limitation partly due to the searcity of multimodal reasoning data- leaves them lagging far behind their language model counter- part which has made remarkable advancement in reasoning tasks (Yang et al., 2024; DeepSeek-Al et al., 2025).
Vision-Language Models (VLMs) combine visual perception with the general capabilities, such reasoning, of Large Language Models (LLMs). However, the mechanisms by which these two abilities can be combined and contribute remain poorly understood. In this work, we explore to compose perception and reasoning through model merging that connects parameters of different models. Unlike previous works that often fo- cus merging models of the same kind, we propose merging models across modalities, en- abling the incorporation of the reasoning capa- bilities of LLMs into VLMs. Through extensive experiments, we demonstrate that model merging offers a successful pathway to transfer reasoning abilities from LLMs to VLMs in training-free manner. Moreover, we utilize the merged models to understand the internal mechanism of percep- tion and reasoning and how merging affects it. We find that perception capabilities are predom- inantly encoded in the carly layers of the model, whereas reasoning is largely facilitated by the middle-to-lat layers. After merging, we observe that all layers begin to contribute to reasoning, whereas the distribution of perception abilities across layers remains largely unchanged. These observations shed light on the potential of model merging as a tool for multimodal integration and interpretation. Our code is publicly available at: https//giiuub..o///iiqggee 7/VLM Merging.
Perception and reasoning are two fundamental components in this context. While language models primarily represent the reasoning ability, VLMs demand both to succeed. There- fore, it is natural to ask: can we incorporate the reasoning ability of LMs into VLMs? Achieving such a combination is challenging, as the interaction between perception and reasoning within VLMs remains poorly understood. In this work, investigate these questions through the lens of model merging (Ilharco et al., 2023), straightforward ap- proach to explore whether perception and reasoning can be combined across modalities and how these two abilities are embedded within VLMs.
Concretely, model merging generates a new model by per- forming arithmetic operations the parameters of existing models, without requiring additional training. This strategy works based on the assumption that models fine-tuned from shared initialization reside in connected subspace of the parameter space. While previous works on model merg- ing focus on models of the same kind (Yadav et al., 2023; Yu et al., 2024b), remains unknown that whether models across different modalities are connectable to yield benefits.
'Equal contribution 'City University of Hong Kong 'Hong Kong University of Science and Technology "Nationa Univer- sity of Singapore "Northwestem University. Correspondence to: Shiqi Chen <schen438 c@ cityu..duuu.. Jinghan Zhang <jzhangjv@ csc.ust.hk>, Junxian He <junxianh@ @ ese. hk?.
In this work, we specifically focus the textual compo- nents of the VLMs and select L.MSs with task-specific rea- soning abilities that match the VLM's configuration, per- forming a weighted average operation on their parameters as demonstrated in Figure 1.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We conduct extensive experiments by merging commonly

---

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
Early Layer
Merged VLM
VLM
LLM
Vision Tower Projector
Late Layer
Question: Subtract brown objects. Subtract all large purple cylinders. How many objects are left?
? Fail to do reasoning
Equipped with CoT reasoning
VLM: There are objects left.
There are three objects in total: one brown egg, small blue ball, and large purple cylinder. If we subtract the brown egg and the large purple cylinder,
Math Empowered Merged VLM:
VLM with CoT prompt: There are two objects left: blue sphere and purple cylinder.
Ground Truth: 1,
with small blue ball. The
Figure 1: Illustration of our work investigating how model merging works when transferring reasoning ability from a Math-specific LLM to the VLM, showcasing the effects and results of the merged model, as well as the interpretation of layer-wise abilities. The abilities are represented in corresponding colors: red indicates perception ability hidden early layers, while blue denotes reasoning ability hidden in relatively later layers.
used VLMs with LLMs trained on diverse math reason- ing datasets ($4). Our findings demonstrate that model merging consistently improves the reasoning capabilities of VLMs across all math benchmarks, with minimal impact on the perception-dominant tasks. For instance, merging with Dart (Tong et al., 2024) enhances LLaVA's performance on Vistass (Lu al., 2024) math-related subset, yielding 3.6-point absolute improvement. Even in Vision-Only mode of Verse (Zhang et al., 2025), where questions presented in images, it also achieves 1.4-point absolute improvement. Experiments on multiple reasoning-relates benchmarks and various VLMs consistently show improve- ment. These results underscore the potential of parameter merging as a simple yet powerful mechanism for capability transfer across model architectures.
for enhancing multi-modal reasoning systems.
2. Model Merging Across Modalities
In this section, we introduce model merging for transferring the reasoning abilitics of textual LMs to VLMs. typical VLM consists of three key components: vision tower, language Model, and projector that bridges these two parts. The vision tower processes images, enabling the model to "sce" visual content, while the language model serves as the reasoning engine, processing knowledge and generating responses. Therefore, we target the language model (0vtm) for merging while keeping the vision tower and projector unchanged.
Model merging has emerged as promising "free lunch" technique, enabling performance improvements by reusing existing models through simple arithmetic operations on their parameters, without requiring additional training. For simplicity, we adopt linear merging (Ilharco et al., 2023), a widely used and robust merging strategy, in our main ex- periments. We also experiment with TIES merging (Yadav et al., 2023) in some cases compare both methods in Appendix C.
Furthermore, delve deeper into understanding how model merging works and how perception and reasoning interplay in this context. Specifically, we analyze parameter changes during the merging process to investigate whether different capabilities, such as high-level reasoning and low- level perception, are disentangled within distinct subspaces of the model's parameter space ($5). Through knockout analysis, identify two key observations: (1) Image per- ceptual abilities and world knowledge are predoninantly embedded in the early layers of the model, whereas mathe- matical reasoning skills are concentrated in the middle-to- late layers; and (2) Merging with reasoning models brings reasoning abilities to all the layers, while having a minimal impact on the layer distribution perception ability. These findings contribute better understanding of how reason- ing be transferred between models and provide insights into model compositionalliy, offering promising approach
The core idea of model merging relies on lask vectors, the modifications made during fine-tuning which is usually the information necessary to do well on given task. Given a base model bhsee and tne-ttuned model on. the correspond- ing task vector Task defined as:
Trakk On Orase-
Task vectors provide an interpretable way to understand how fine-tuning adapts model to a particular task. In the
2

---

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
Category
Name
Size
Base Model
LLTVA-NeXT (Liu et al., 2024)
8B
LLaMA-series
VLMs
Idefics2-8B (Laurençon et al., 2024)
8B
Mistral-series
InterVL2 (Chen et al., 2024b)
76B
LLaMA-series
Domain
Base Model
Dart-Math (Tong et al., 2024)
Math Domain
LLMMA/Mistraa-srrisss
MetaMath (Yu et al., 2024a)
Math Domain
L.aMA-series
Task Vectors
M--moTH-1 (Yue et al., 2024a)
Math Domain
I.LaMA-series
MAmmoTH-2 (Yue et al., 2024b)
General Domain
ILLMAAMiistral-erriss
Magpie-v0.3 (Xu et al., 2024)
Math Domain
LLLMA-series
Deepseek-RR RI-Distill (DeepScck-Al al., 2025)
Math Domain
LLaMA-scries
Table 1: Overview of Vision-Languaae Models (VLMs) and Task Vectors with the attributes -size, base models and domains.
context of VLMs, we define the adaptation of the language model component as:
2024), which includes two variants: Dart-Uniform and Dart- Prop2diff. In this paper, we refer the latter as Dart-Prop. Additionally, we examine MAmmoTT-1 (Yue et al., 2024a), Magpie-v0.3 (Xu et al., 2024), MetaMath (Yu et al., 2024a), and Deepseek- 1-iistill (DeepSeek-AA et al., 2025). Our scope further extends to broader reasoning task vectors ob- tained in MAmmoTH-2 (Yue et al., 2024b). The base mod- els and task vectors are detailed Table 1.
Tylm = fymm Ouase-
where Tvlm captures the changes introduced when adapting the base LLM into the VLM. Similarly, for a reasoning- specialized LLM Ocasson we define its corresponding task vector:
Treason
Oreason
Opase-
To enhance the reasoning ability of the VLM, we merge its language model with strong geasoning-specialinn LLM by linear merging:
Hyperparameters In our main analysis and experimental sections, we employ a linear merging strategy for all task vectors under the same hyperparameter settings to ensure a fair comparison. This approach assigns a weight of 0.9 to the textual component of LLaVA-Nex-LLaMMA--5 and 1.1 to the reasoning task vector, where À 0.9. This parameter is tuned on Math Vista based on Dart- Part-Prop (Tong et al., 2024), We choose the best value from the range (0.8, 0.85, 0.9).
Bbase
ATvim + (1 A) Treason-
Here, determines the weight assigned to the VLM task vector, allowing us to control the balance between the orig- inal multi-modal capabilities of the VLM and the newly introduced reasoning strength from the LLM.
For our additional experiments analyzing the effects of merg- ing across different base VLMs, we adjust the parameter within range of 0.05 across the intervals (0.8, 0.85, 0.9) based Dart-Prop Math Vista and apply the same hyper- parameter across all benchmarks and other task vectors if they exist.
3. Experiment settings
In this section, we describe our experimental setup, detail- ing the selected models, datasets, and evaluation protocols used to evaluate the effectiveness of model merging and understand its internal workings.
Evaluation We evaluate the performance series of VLM benchmarks. We apply five benchmarks: Math- Vista (Lu et al., 2024), Math Verse (Zhang et al., 2025), Math Vision (Wang et al., 2024a), Dynamath (Zou et al., 2024) and MMStar (Chen et al., 2024a).
VLMs
We span models in different sizes and base models verify the generalization ability of merging. For VLMs, use LLaVA-Nex--LLaMAB.... (Liu al., 2024), Idefics2- 8B (Laurençon et al., 2024), and InternVL2-LLaMM 76B (Chen et al., 2024b) (Abbreviated as LLaVA, Idefics, and InternVL. our paper) ranging from 8B to 76B and including both Mistral-based and LLaMA-based models.
Among these benchmarks, Math is a diverse bench- mark that includes both math-related reasoning tasks and general visual question answering tasks. Fach data sam- ple in Vist is meticulously annotated with meta- information such as source, task etc., allowing us to evaluate various aspects of improvement, such as identifying the spe- cific scenarios where our method most effective.
Reasoning Task Vectors We span task vectors across different reasoning domains, beginning with mathemati- cal reasoning tasks featured in Dart-Math (Tong et al.,
3

---

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
4. Can Model Merging Enhance VLM Capabilities?
giometry weasoning
algebraic neasoning
3.56
2.5
1.42
0.36
1442
textbook QA
0.00
9.16
127
0.00
2.53
In this section, we present model merging viable ap- proach for enhancing inherent VLM capabilities, specifi- cally exploring how merging with specialized models can augment core functionalities such as underdeveloped rea- soning abilities in standard VLMs.
reasoning-
0.00
geometry problem solvingg
3:37
1.22
0.48
0.48
0.00
Accuracy Change
math word problem
3:23
1.08
-1.08
-0.54
4.08
stattstical masoning
2:33
-1.33
-1.33
-.33
Overa
0.50
0.70
-2.70
-8.80
-0.10
sceetffi casoning.
-1.64
-1.64
-0.82
0.82
vicuel QA
-1.68
0.00
-0.55
-1.12
Merging VLMs with math-specialized models consis- tently improves performance across all math bench- marks. We posit that VLMs primarily rely two fun- damental capabilitiis: perception and chain-of-thought rea- soning ability to solve the visual reasoning problems. Ad- ditionally, they also leverage the knowledge-recal ability to enhance their decision-making. Among these, chain- of-thought reasoning remains as a bottleneck in current VLMs (Zhang al., 2024a), despite significant advances in this area by LLMs. This observation motivates our investi- gation into whether merging VLMs with math- math-specialized LLMs can enhance their inherent reasoning capabilities.
arthhmetic casoning.
0.28
0.57
1.70
1.18
1.98
numeric connor
2,08
-0.99
fiuure GA
-1.86
-1.49
-2.23
-1.85
-0.37
dant proptal
dert undom
maggle
mammithh
mammitth2
Merged Models
Figure 2: Accuracy changes after merging compared to the baseline. Generally, datasets directly requiring math- related and text-dominant capabilities, such as textbook QA and math word problem, exhibit clear improvements while domains requiring visual processing such as figure QA show performance degradation.
To explore this hypothesis, we use LLaVA-Next-LLMMa 8B, a commonly employed VLM, as the base model, and integrate it with 5 ofate-of-the-art reasoning models (see task vectors in Table 1) that were specifically fine-tuned on reasoning tasks: Dart-Math (Tong et al., 2024) (Dart has two variants: Mart-Uniform and Dart-Prop2difft abbreviated as Dart-Prop in our paper), MAmmoTT-1 (Yue et al., 2024a), MAmmoTH-2 (Yue et al., 2024b), and Magpie-v0.3 (Xu et al., 2024). For a fair comparison of the task vectors, we employ the linear merging strategy to all task vectors in the same syper-parameter setting, which assigns a weight of 0.9 to the textual component of -LL.VA--exxLLLMMA. 0.1 to the math task vector. The merged model is evaluated across five datasets (see $3), and the results are summarized in Table 2. As indicated by the green arrows, integrating VLMS with math-speciallee specialized models such as Dart (Tong et al., 2024) consistently improves performance over the baseline across all five mathematical datasets. Notably, on the Math Verse Benchmark in Text-Dominan evaluation mode, merging with Dart-Prop yields 30% relative im- provement over the baseline (a 6-point absolute increase). Additionally, it boosts performance on math-related VQA in Math MathVista by 3.5 absolute points. Whereas merging with a ceneral-purposs reasoning LLM like MagPie (Xu et al., 2024) offers only modest gains.
benefit. We attribute this to Idefics already being extensively fine-tuned on large-scale text-only data, including math SFT data, leading to a high degree of overlap with existing task vectors, which limits further improvement through merging, For Intern Intern VL2-766, integrating Dart improves all bench- marks by approximately 1 absolute point, demonstrating that merging can also be beneficial for large-scale VLMs.
Model merging exhibits minimal improvement or even decreased performance on vision-dominant tasks and general knowledge-centric tasks. A closer analysis of performance across different subtasks reveals that on the Math Verse benchmarks, model merging yields higher per- formance gains for math-related and text-dominant samples, while vision-only questions show limited improvement (see Table 2). This phenomenon also observed in the Math- Vista dataset (Figure 2), where we visualize performance changes relative to the baseline for each sub-domain dataset using heatmap. Notably, datasets that directly require math-related and text-dominant capabilities, such as geometric reasoning, algebraic reasoning and textbook question answering, exhibit consistent improvements. However, domains requiring extensive visual processing (e.g., visual question answering and figure question answer- ing) show slight performance degradation. Vision-only and rision-dominas tasks consist of questions embedded in fig- ures, which require robust image perception to accurately recognize the question before employing knowledge-recwd and reasoning to derive the answer. The inability of model merging to improve these tasks raises a critical question: if the bottleneck lies in image perception, then enhancing the
and
Moreover, we extend our methods to other VLMs and task vectors. Table 3 presents results for Idefics2-8B (Yadav et al., 2024) and Intern VL.2-76B (Chen et al., 2024b) using task vectors available in the open-source community. For Idefics, MAmmoTH-1 achieves the best performance, im- proving accuracy by approximately 1.0 absolute points on average, while most reasoning task vectors fail to provide a
4

---

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
Math Vista
MathVerse Benchmarks
MMStar
Model
Method
DM
MV
All
General
Math
Overall
T-D
T-L
V-I
V-D
V-O
All
Math
Baseline
37.4
51.7
25.4
20.1
25.9
20.8
21.1
16.5
16.0
43.8
30.0
22.7
13.8
+Dart-Unifom
38,2
49.8
28.3 12.9
23.6
13.5
32.0
25.6
25.4
19.3
17.4
42.5
31.2
11.2
24.5 11.8
15.8 12.0
+MAmmoTHH1
36.7
50.0
10.0
21.1
11.0
26.9
23.1
22.6
16.6
16.4
44.1
30.8
10.8
22.6 1o.1
15.8 12.0
+--mmoTH-2
37.4
51.1
25.7 25.7 10.3
20.6
10.5
26.0
22.0
22.8
16.4
16.1
43.8
30.0
10.0
22.5 (0.2
14.1 10.3
LLaVA
+ +Magpie-v0.3
36.8
49.7
25.9 10.5
20.7
10.6
26.8
22.2
22.2
16.6
15.5
44.1
30.8
10.8
22.7 10.0
16.4 12.6
DeepSeek-RR--Die
38.1
51.1
27.0 11.6
21.2
28.4
22.7
22.5
17.3
15.1
43.7
33.2
13.2
24.3 11.6
15.1 11.3
+Dart-prop
38.0
48.7
28.9 13.5
23.7
13.6
30.7
24.8
25.5
19.8
17.4
43.6
33.6
13.6
24.5 11.
14.8 11.0
Table 2: The performance of LLaVA-Nex--ZLMA---0 model with merged task vectors across math-related Benchmarks: MathVista (All, General, and seathrelated categories), MathVerse (Overall, Text-Dominant Text-Lite, Vision-Integrate Dssion-Dominant, and Vision-Only categories), MMStar (All and Math split), DynaMath (annotated as DM), Mision (annotated as MV). We include both variants of Dart (Tong et al., 2024) for comparison. We bold the highest value in each benchmark, and the gray row indicates the best task vectors on average.
Math Vista
Math Verse Benchmarks
MMStar
Model
Method
DM
MV
All
General
Math
Overall
T-D
T-L
V-I
V-D
V-O
All
Math
Baseline
51.8
57.0
47.4
19.4
24.4
21.3
20.7
19.7
11.0
49.5
39.6
21.8
17.1
+MetaMath
53.2
57.8
49.3 11.9
20.0 10.6
25.3
22.3
21.1
18.7
12.4
48.1
39.2
J0.4
22.7
11.8 15.3
+ +Dart-Prop
51.6
58.0
46.1 L1.3
20.0 10.6
26.3
21.8
21.6
18.9
11.2
48.4
39.6
10.0
22.7 10.9
14,8 12.3
Idefies
DDart-nnffrm
51.6
57.0
47.0 10.4
20.5 f1.1
27.3
22.6
21.1
19.5
12.2
47.9
38.4
11.2
22.7 10.9
14.8 12.3
+ +--mmoTTH11
53.0
58.5
48.3 10.9
20.4 11.0
26.0
22.5
21.3
19.8
12.1
48.3
40.8
11.2
23.2 11.4
16.8 10.3
+ -M-mmoTH-2
52.8
58.3
48.1 10.7
18.4 11.0
25.6
22.7
20.7
19.4
12.4
48.5
40.0 10.4
24.0 12.2
16.8 10.3
Baseline 65.6
67.0
64.4
43.1
54.1
47.5
44.8
43.8
25.3
67.3
75.2
38.7
23.7
VnternVI.
+Dart-Uniform
66.1
67.2
65.2
10.8
44.3
11.
53.9
48.1
46.3
44.5
28.6
67.5
74.8
J0.4
39.6
10.9
25.3 11.6
Table 3: The performance of Idefics2-8B model and
model with merged task vector across math- related Benchmarks: Vista (All, General, and Math-related categories), Marhe (Overall, Text-Dominant, Text-Lite, Vision-Integrated,
and Vision-Only categories), MMStar (All and Math split), DynaMath (annotated as DM) and Math Vision (annotated as MV). For benchmarks with Math subsets, only the Math score is included in the average score calculation. We bold the highest value in each benchmark, and the gray row indicates the best task vectors on average.
textual component through model merging may fail to yield performance gains.
length, even showing a decrease in performance. These re- sults suggest that merging with a Dart not only enhances the detail and depth of responses in reasoning-intensive tasks but also maintains efficiency and stability in perception-driven tasks, highlighting the adapt- ability of the merged framework across diverse domains. We show the details in Figure 9.
model like
Merging with math models brings inference time scaling ability. We hypothesize that the reasoning ability trans- ferred VLMs is primarily reflected in the improvement of chain-of-thought capabilities. To support this hypothesis, we analyze answer lengths before and after merging with Dart, highlighting significant shifts in task-specifie behav- iors. As shown in Figure 3, the performance improvement exhibits a nearly linear relationship with the increase in an- swer length, indicating that merging enables VLMs to scale inference time effectively. When looking closer the spe- cific tasks, tiatt--o--tough--ittneee try problem solving", "geometry reasoning". and "algebraic reasoning" experienced substantial increase in average prediction length, exceeding 250% of the original answer length. In contrast, changes in visual-intensive tasks like "figure question answering" and "visual question answering" were relatively modest, with nearly the same original answer
5. Merging as Interpretability Tool - Dive into the inner parameter space of LLaVA
In this section, we leverage model merging as an analytical tool to decompose and understand the internal mechanics of VLMs. By analyzing parameter modifications during the merging process, we aim to identify and isolate distinct parameter subspaces responsible for specific capabilities, such visual perception and reasoning. We conduct fine- grained analysis of LLaVA on the Math (Lu et al., 2024) dataset, which categorizes the examples into "Gen- eral VQA" and "Math VQA". ". We hypothesize that "General
tasks such as "geome-
5

---

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
formance on math reasoning tasks significantly improves through merging. Furthermore, we demonstrate in $4 that this improvement stems from the infused inference-time scaling ability. To pinpoint where these infused ch thought abilities reside in the parameter space, we masked the parameters of the "Dart-Merged LLaVA" using those from the original LLaVA model. 2 and 4 at Figure indi- cates that masking the later layers has a more pronounced impact on math-relater tasks, suggesting that the later five or more layers are crucial for math reasoning. Moreover, merging with math-focused model boosts math reason- ing abilities while only minimally affecting general VQA performance. This implies that integrating a math model with small weights incurs only a slight trade-off VQA ca- pabilities while yielding significant gains for math-specific questions. Overall, these findings suggest that perception and -fain-of-thought reasoning abilities occupy distinct regions within the LLa VA parameter space and can be largely disentangled.
Accuracy Difference (16)
aigbbric reasoning
grometry probiem solving
2-
math wer probiem
Statstical reasoning
log reasoring
texthoo: QA
0-
arthimetic reasoning
sclentifcc reasoning
Rgure
numoric commensence
150
200
250
300
150
Answer Length Difference (16)
Figure 3: The relationship between answer length change (characters) and accuracy improvement after merging (The x-axis represents the relative change from the original an- swer, while the y-axis shows the absolute change in accu- racy). classified by task and skills.
VQA" primarily evaluates perception ability and knowledge recall, while "Math VQA" " assesses perception ability and reasoning ability.
Expore the threshold of the absolute ability of LLaVA for each module In previous experiments, we masked in- dividual modules using those from another model to compar- atively evaluate specific capabilities. To precisely quantify the role of each module-such as attention or MLP layers- in VLMs for both general VQA and math VQA tasks, we replace each module's weights with uniform distribution (i.e., assigning each parameter value of 1/N, where N is the size of the weight matrix's 's first dimension). This substi- tution introduces significant noise, effectively disabling the module's s functionality. By measuring the resulting perfor- mance drop, we can assess each layer's absolute contribution to the model's performance on both tasks.
Mask Out We begin by using the masking out technique to assess the influence of each module. Specifically, for each layer, we replace the parameters of the MIP and at- tention modules with those from alternative modules (e.g.. a uniform distribution or parameters from another model) to evaluate the absolute and relative impact of each module. Our hypothesis is that greater performance drop when masking particular module indicates its higher impor- tance for the task, while a smaller drop suggests more trivial effect on the task.
Locate the perception ability in the parameter space We first analyze the impact of supervised fine-tuning (SFT) of LLaVA compared to the pretrain LLaMA, by progres- sively masking out LLaVA's parameters and replacing them with LLaMA's parameters, layer by layer for each module. Our hypothesis is that LLaVA's SFT training improves the model's perception ability, and we use the Masking Out technique to identify key regions where perception abil- ity is located. O and at Figure 4 shows that, from layer-wise perspective, masking out the early layers has greater impact on general VQA tasks than masking the later layers, suggesting that perception abilities are primarily located in the early layers. We also observe that while general VQA performance declines after masking out, Math VQA performance improves consistently. This implies that LLa'A's s SFT enhances perceptual abilities at the cost of rea- soning capabilities, which motivates our efforts to improve reasoning ability in VLMs.
Figure 5 shows the performance drops of disabling LLaVA S and 7 and Dart-Merged LLaVA 6 and 8 The first observation is that 1) early-to-middle layers are more crucial for both general and math-related tasks in the LLaVA model, as evidenced by the significant drops, i.e., 25% absolute accuracy drop in general tasks and 10% in math-targeted VQA (highlighted in red gray square). This suggests that the early-to-nnn miidde layers play unique and in- dispensable roles in VLMs, this is intuitive since early layers handle perception, and accurately perceiving the image is a prerequisite for answering correctly while the later layers are less important and more robust to noise. Secondly, math- targeted VQA shows a smaller performance drop than general VQA after masking out parameters, due to its inherently weak math reasoning ability. When certain pa- rameters are masked out, general visual question answering (VQA) tasks (left side of 5 and 7 exhibit a larger perfor- mance decrease than math-related questions (right side of 5 and ①. This can be explained by the LLa model's 's limited knowledge of math tasks and its inherent weak rea-
Locate the chain-of-thought reasoning ability in the parameter space As shown in Table 2, LLaVA's per-
6

---

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
Where is the perception ability located?
Where is the reasoning ability located?
by nuth targ
Accuracy Change by Blocking- cartain Lapera
Tail: perersivag
1.00
1:30
305
3.05
Mask MLP layers
-000
-0:0
+100
+0,0
-22
-200
+3.0
Change Blocking certain Self
④
Blocking oyt certain Atttencion
1:31
po
9:99
Trub muth teryclad
4:11
3:01
reath Corpcted
Mask
1.03
9:00
6:33
3:10
Attention layers
3:00
-2.0
-200
-1.2
-200
44
Finding 1: Perception Ability mainly lies in Early Layers
Finding2: Reasoning Ability mainly lies in Later Layers
Figure 4: LLaVA LLaMA ( and 3) ): the accuracy changes after replacing the parameters of each MIP ① and Attention ( layer of the LLaVA model with parameters from LLaMA. We find that masking out the early layers has greater impact on general VQA tasks than masking the later layers, suggesting that perception abilities gained from VA-sft training are primarily located in the early layers. Dart-Merged LLaVA - LLaVA 2 and ): the accuracy changes after replacing each MLP 2 and Attention layer A of the Mart-Merged LLaVA model to that of LLaVA. A significant drop in accuracy in math-targeted VQA tasks is observed from layer onwards. (highlighted in blue), suggesting that the reasoning ability is mainly located on these layers.
soning abilities compared the abilities demonstrated in VQA tasks (e.g., perception and world knowledge). In other words, the model's limited mathematical reasoning ability makes it less sensitive to parameter alterations, which also enhances our motivation to incorporate external reasoning ability into it.
6. Related Work
Large Vision-Language Models (VLMs) consist of three main components: a visual encoder for pro- cessing images, such as CLIP (Radford et al., 2021) or SigLip (Zhai et al., 2023); a language model (e.g., a LLaMA model (Dubey al., 2024) or Mistral model (Jiang et al., 2023)) for processing textual inputs and image features to generate responses; and a projector, typically implemented as multilayer perceptron (MLP), to bridge the gap between the visual and language components. This module maps fea- tures from the visual space to the language space, facilitating interaction between the two modalities.
VLMs
Merging with Reasoning Models Enhances Almost All Layers' Math Reasoning Ability As shown in Figure 5, after merging with Dart (right side of 6 and $ ) we observe that nearly all more in math reasoning tasks compared to the base LLaVA (right side of 2 and 7) where fewer layers are influenced. This suggests that reasoning ability has been successfully integrated into all layers without substantially affecting the layer distribution of perception ability. Several qualitative examples supporting this are presented in Figure 6. The first and third examples illustrate how the model better perceives key entities and makes decisions through chain-of-though reasoning. However, for general VQA tasks, we also see reduced activation in the early layers, indicating a slight loss in world knowledge due to the model merging.
by the blue mask drop
Model Merging Model merging offers a "free lunch" by repurposing fine-tuned models for downstream tasks (Worts- man et al., 2022; Ilharco et al., 2023; Zhang et al., 2023). It creates new model through simple arithmetic on existing parameters, requiring no extra training or inference cost.
Several model merging techniques have been proposed to improve performance, such as calculating different weights for model parameters using data and internal model acti-
7

---

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
Where is the Overall ability=
After merging, Math ability for all layers improves!
perception world knowledge reasoning ability located?
5
Accuracy Change by Brocking Out cartain NLP Layers
Accuratyy Change Blocking Out certain NLP Layers
raigetdd
Back gooedd
Bek wuth taggtedde
Mask MLP layers
-0.33
-0:33
-035
13
23
L
Finding 3: Early-to-Midon Moddeee layers are indispensable while later layers are more robust to noise.
Finding 4: Contribution of the middle-to-late MLP layers and almost all attention layers to math reasoning has increased.
7
Accuracy Change by Blocking Out certain Self Attention Layers posool
8
Accuracy Change by Blocking Sut cortain Attention Layoo
Doyyte way
Mask Attention layers
-000
-1.0
145
14
S and 7 y: the accuracy changes after replacing the parameters of each MLP 55) and Attention O layer of the LLaVA model with N. where N is the first dimension of the weight matrix. The highlighted red area shows that early-to-middle layers are more crucial for both general and math-related tasks in the LLaVA model, as evidenced by the significant drops, i.e., 0.25 absolute accuracy drop in general tasks and 0. 10 in math-targeted VQA. Dart-Merged LLaVA + 1/N O and : the accuracy changes after replacing the parameters of each MLP (6) and Attention (8 layer of the Mert- LLaVA model with N Comparing before and after merging when applied masking out, observe larger drop in accuracy in math-targeted VQA tasks across all layers (highlighted in blue), suggesting that the contribution of all most all layers to math reasoning has increased.
Figure 5: LLaVA - I/N
Perception
Chain-of-thought
Perception
Qt: According to the table, what was the rate of change between August and September? (Unit: minutes per month)
Q1: In the figure above. the ratio the length of line AB to the lenith of line AC 2: 5. If AC 25, what is the langth of line AB?
Q1: According to the table, what was the rate of change between August and September? (Unit: minutes per month)
Hugust
27
Seezenter
14
Oewber
28
Roweroo
Docariee
The change between August and Saptember 7 minutes par
(E base
The correct
(.0)
0.5
base
base
The ratio of the langth of line AB to the length of line AC is 2: 5, which means that for every 2
The change between August and September is 14 17 -3 minutes per month.
AB, there then AB 2/5 AC, which correct answer Is:
of
25,
The fraction females facing 4/5.
ours
215 The
(B) 10
Figure Qualitative study: three examples that can be fixed by merging with Dart (Tong et al., 2024).
vations (Matena & Raffel, 2022; Jin et al., 2023). Some approaches initially sparsify the models to reduce conflicts across different functions (Yadav al., 2023; Yu et al., 2024b). Recently, Layer Swapping (Bandarkar et al., 2024)
was introduced, retains specific layers while merging oth- ers to enhance transfer learning. Despite their differences, simple averaging is often preferred for its simplicity and robustness. As model scales grow, performance gaps among
8

---

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
merging techniques shrink (Yadav et al., 2024), making linear merging natural choice.
Anonymous. REMEDY: Recipe merging dynamics in large vision-language models. In The Thirteenth In- ternational Conference Learning Representations, 2025. URL / 1 id=1x7eHHE5TX.
http://pppereiiieeeeeeee net /orumm
Merging for VLMs Although much prior research has focused on merging vision models for tasks such as image recognition and scene understanding (Ilharco et al., 2023; Yang al., 2023), the composition for VLMs remain insuffi ciently explored. particularly in terms of transferring special- ized skills across modalities. Recently, REMEDY (Anony- mous, 2025) proposes a practical merging recipe, which merges projector and front-layer modules of VLMs, focus- ing on multitasking and transfer in low-shot settings across various VQA types. Additionally, Sakana Al demonstrates the potential for multilingual capabilities in VLMs by trans- ferring Japanese comprehension and generation abilities from an expert LLM to VLMs (Akiba et al., 2024). How- ever, key challenge lies in transferring reasoning skills across modalities. Our study addresses this by integrating the reasoning capabilities of LLMs into VLMs with com- prehensive analysis.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, Qwen-vl: A versatile vision- language model for understanding, localization, text read- ing, and beyond. arXiv preprint wrXiv:2308.12960 2023.
Bandarkar, L., Muller, B., Yuvraj, P., Hou, R., Singhal, N., Lv, H., and Liu, B. Layer swapping for zero-shot cross- lingual transfer in large language models. ArXiv preprint, abs/2410.01335, 2024. URL htthttps:// http://aaxivv....o abs/ 0tss/2410.0133
Chen, L., Li, J., Dong. X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., et al. Are we on the right way for evaluating large vision-language models? ArXiv preprint, abs/2403.20330, 2024a. URL https: /ttp//rxivvv.////////.a/////atva wwwpwawwwww..aaaaaaaooaaaaaaaaae
7. Conclusion
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q. Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual- lingusiii tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024b.
In this paper, we investigate the use of model merging meth- ods to bridge the intelligence of cross-modalities, specif- ically transitioning from pure textual modality to vision- textual modalities. By incorporating various math-specific LLMs into different VLMs through model merging, we demonstrate that this approach effectively enhances the rea- soning capabilities of VLMs. Our experimental results in- dicate an improvement of up to 12% in performance on math reasoning tasks compared to the baseline. And by employing model merging as an interpretability tool, we further unlock the parameter space of VLM. We find that for VLMs trained solely on image-text pairs, the abilities of perception and reasoning can be decomposed within the parameter space. Specifically, perception ability resides in the early layers, while reasoning ability is concentrated in the later layers. This decomposition further validates our experimental results.
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang. P., Bi, X., Zhang. X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., and Liu..., A. Deepseek-rl: Incen- tivizing reasoning capability in llms via reinforcement learning, 2025. URL https: /tt///axvvvorg///eee abs/ 2501.1294
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. ArXiv preprint, abs/2407.2117 217.2773., 2024. URL https: http///xxivv.r//// abs/ / 0220/220222228
Impact Statement
Ilharco, G., Ribeiro, M. T. Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: / / oppprevveee.. viww. net wwwe:/pfttdddeeeeeeeeeeepp
This paper presents work whose goal is advance the field of Machine Learning Interpretability. There many po- tential societal consequences of our work, none of which we feel must be specifically highlighted here.
References
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. ArXiv preprint, abs/2310.06825,, 2023. URL https: / //aaxivvv.../////////a///////av/ .org/ abs/2310. 06825.
Akiba, T., Shing, M., Tang, Y., Sun, Q. and Ha, D. Evolutionary optimization of model merging recipes. ArXiv preprint, abs/2403.13187, 2024. URL https: // /arxiv.orrg/ab//240. 131877
9

---

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
Jin, X., Ren, X., Preotiuc-Pietro, D., and Cheng, P. Data- less knowledge fusion by merging weights of language models. In The Eleventh International Conference Learning Representations. ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: http://wwwwweeeeeeedddddaoccaona FCw.dernoohuuuunnncc
J., Krueger, G., and Sutskever, 1. Learning transferable visual models from natural language supervision. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748- 8763. PMLR, 2021. URL http: http://proceedingg. mir.prest v11./v11999/// https//119/aaafodd222e. http///ffffodd.....aaa htm.lll.....m...m.mmmmm
Laurençon, H., Tronchon, L., Cord, M., and Sanh, V. What matters when building vision-language models? Ad- vances in Neural Information Processing Systems, 37: 87874-87907, 2024.
Sun, K., Bai, Y., Qi, J., Hou, L.,, and Li, J. Mm-math: Advancing multimodal math evaluation with process evaluation and fine- fine-grained classification, 2024. URL https://arxiee org/ abs /tw./aaaae 2404 05091.
Laurençon, H., Tronchon, L., Cord, M., and Sanh, V. What matters when building dision-language models?, 2024.
Li, B., Zhang, Y., Guo, D., Zhang. R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., al. Llava- onevision: Easy visual task transfer. ArXiv preprint, abs/2408.03326, 2024. URL https://///// /ttpa//.//xvvoo///////// abs/2408. 03326.
Tong, Y., Zhang, X., Wang, R., Wu, R., and He, J. Dart- math: Difficulty-waare rejection tuning for mathematical problem-solving ArXiv preprint, abs/2407. 136000,,.1..,... 2024. URL https: http///aavvvv..../ab// www//og/////.ooooooooo 13660
Wang. K., Pan, J., Shi, W., Lu, Z., Zhan, M., and Li, H. Mea- suring multimodal mathematical reasoning with math- vision dataset, 2024a.
Liu, H., Teng, Z., Cui, L., Zhang, C., Zhou, Q. and Zhang, Y. LogiCoT:: Logical chain-of-though instruc- tion tuming. In Bouamor, H., Pino, J. and Bali, K. (eds.), Findings of the Association for Computa- tional Linguistics: EMNLP 2023, pp. 2908-2921, Singa- pore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2022 findo/0g0...ddnggeeennll 191. URL https: / /aaaaaaaanntloooyo org/2023. emmn-mmnpppp....m 191//
Wang. P., Bai, S., Tan, S., Wang, S., Fan, 7., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y. Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.1 000::2409.1299111 12191, 2024b.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved rea- soning, ocr, and world knowledge, 2024. URL https://llave. http://laaaav..githhee vl.github. io/blog/ 2024
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965- 23998. PMLR, 2022.
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-.., Galley, M., and Gao, Math- vista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024.
Xu, Z., Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y., and Lin, B. Y. Magpie: Alignment data synthesis from scratch by prompting aligned Ilms with nothing. ArXiv preprint, abs/2406.08464, 2024. URL https: // //arxiv. org/ abs wwwp///...022.. 08464.
Matena, M. and Raffel, C. Merging models with weihht- averaging. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Pro- cessing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans. LA, USA, November 28 De- cember 2022, 2022. URI. http::/papers http:/ppee__leeeeppppr/////////a T0c-6911bEE3440006111100003 Abstract-Confference..t
Yadav, P., Tam, D., Choshen, L., Raffel, C. A., and Bansal, M. Ties-merging: Resolving interference when merging models. Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. htt:: /ptttrrs/pppprs... Saps.cc/pappr. htt.cc//aaaerr.ilese/ppppee/e//e 1644c9a2280011188844666222222222 Abstract-Conferennee html.
Neural Information Processing
/hash/
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark,
10

---

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
Yadav, P., Vu, T., Lai, J., Chronopoulou, A., Faruqui, M., Bansal, M., and Munkhdalai, T. What matters for model merging at scale? ArXiv preprint, abs/2410.03617, 2024. URL https: arxiv.. org/ abs/ 2410 03617.
abs/2410.16198, 2024a. URL URL.tttss:/arxii.orr abs/2410.16199
Zhang. R., Jiang, D., Zhang, Y., Lin, H., Guo, Z., Qiu, P., Zhou, A., Lu, P., Chang, K.-W., Qiao, Y., et Math- verse: Does your multi-modal I1m truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp. 169-186. Springer, 2025.
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., Lu, K., Xue, M., Lin, R., Liu, T., Ren, X., and Zhang, Z. Qwen2.5-math techni- cal report: Toward mathematical expert model via self- improvement, 2024. URL https: http///axxvvvoooeeeee abs/2409.122 12122.
Zhang, Y., Bai, H., ZHANG, R., Gu, J., Zhai, S., Susskind, J. M., and Jaitly, How far are we from intelligent visual deductive reasoning? In First Con- ference Language Modeling, 2024b. URL https: / //openreview net forum? iiemmnid=pppp Y=phEEGZGNY..
Yang. E., Wang, Z., Shen, L., Liu, S., Guo, G., Wang, X., and Tao, D. Adamerging: Adaptive model merging for multi-task learning, ArXiv preprint, abs/2310.02575, 2023. URL. http:://arxxi..or//bb////2 02575.
Zou, C., Guo, X., Yang. R., Zhang, J., Hu, B., and Zhang, H. Dynamath: A dynamic visual benchmark for evalu- ating mathematical reasoning robustness of vision lan- guage models, 2024. URL https: httpa/aavve/.////ee//ee/eeeeeeee org/ abs /bb/2411.00836.
Yu, L., Jiang, W., Shi, H., YU, J., Liu, Z., Zhang, Y., Kwok, J., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Confer- ence on Learning Representations, 2024a. URL https: // //openreview net foet/for forum? d=N8NChgNDRt.
Yu, L.. Yu, B., Yu. H., Huang. F., and Li, Y. Language models super mario: Absorbing abilities from homol- ogous models free lunch. In Forty-first International Conference on Machine Learning, 2024b.
Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. МAmmoTH: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference Learning Representations, 2024a. URL https: /poperevvvi eww. net/ forum? id id-yLC1Gs770I
Yue, X., Zheng, T., Zhang, G., and Chen, W. Mammoth2: Scaling instructions from the web. Advances in Neural Information Processing Systems, 2024b.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L.
Sigmoid loss for language image pre- pre-training.
IEEE/CVF International Conference on Computer Vi- sion, ICCV 2023, Paris, France, October 1-6, 2023, pp. 11941-11952. IEEF, 2023. doi: 10.1 10.1109//CCV510700
2023.01100. URL https: /tt///iiogg/....111/ ICCV51070. 2 2023. 01100.
Zhang, J., Liu, J., He, J., et al. Composing parameter- efficient modules with arithmetic operation. Advances in Neural Information Processing Systems, 36:12589- 12610, 2023.
Zhang, R., Zhang, B., Li, Y., Zhang, H., Sun, Z., Gan, Z., Yang, Y., Pang, R., and Yang, Y. Improve vision lan- guage model thaougt reasoning. ArXiv preprint,

---

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
A. The checkpoints used in experiments
show all the checkpoints we use for experiments Table 4.
Category
Huggingface ckpt
Task Vectors
Huggingface ckpt of task vectors
MAmmo
BrasiGuuh/llaaaa--namno----hhh
MAmmoT112
TIGER-LLbDMAAmmoTI1--nn www.:///l.aaaa-------e--------ee v0.3
Magpie-v0.3
LLa VA-Net Imms-lab/llama3-llaa--ai
Dart-Uniform
Dart-Prop
DeepSeek-RI-Dee
MAmmo
TIGER-LLb/MAAAmm07117BBB..
MAmmo
[IGER-Lab/MAmmen
Idefics2-8B
HugggingFaaceMMiiddeiiss..bbbb
MetaMath
meta geeta-math/Meen Metaath/Metaaaahhba -aitttaaMaMaMMstta-----
Dart-Uniform
hkust-nlp/dart-maaee hattt---istttl----ieee
Dart-Prop
InternVL2
OpenGVVaab//neem2222LLmmaa
Dart-Prop
hkust hlusstnlp/dart- math-llama'ss Iaamlllmmmm7----- 700--000ppmmm 70b-prop2diff'
Table 4: All the huggingface checkpoints we in our experiments
B. More analysis for Math Vista
MathVista includes various metadata, such as "Task" and "Till& allowing the dataset to be classified into subsets using different methods. We present Math Vista's performance improvement after merging with Dart (Tong et al., 2024) across different "Tasks" (there in total 5 tasks in Math Vista) Figure 7, along with the correlation between answer length and accuracy improvement for each task. We see that two figures both show consistent pattern with findings Section 4. Figure suggests that the math-specialized task vectors primarily benefit math reasoning tasks, but may hinder performance on knowledge-intensive general VQA tasks. Figure shows that merging with Dart enhances the model's inference-tine time scaling ability.
Heatmap for task
question anowering
D.C
Aecumey Offfrencee
0.C2
ACCUFaC Change
mach word problem-
3.55
9.0
0.31
0.00
gecieety problem sah math word prodler
visual queniton unswering.
6.02
CM
3.2
0.00
0.01
00.0
vosua ta
figar questtion emswering
4:37
-0.02
150 200
dar: prop2diff
uniform
magple Merged Models
marmooth2
Figure 8: The relationship between answer length change and accuracy improvement after merging, classified by task.
Figure 7: The accuracy difference after merging across several subtasks in MathVista for LLaVA.
C. Different merging methods perform generally comparable
We are interested whether different merging methods affect the performance of reasoning ability transfer. To investigate this, we employ TIES (Yadav al., 2023), Dare merging (Dare-TIES and Dare-Linear) (Yu et al., 2024b), and Layer Swapping (Bandarkar et al., 2024), which are popular merging methods in practice, to compare their performance with that of linear merging. We adopt the hyperparameter search strategy as linear merging, parameterized by (01, a22, where is tuned for the VLM vector and 02 for the Math LLM task vector, to obtain the most comparable checkpoints on benchmarks emphasizing visual and textual reasoning.
As shown Figure 10, we present results of each configuration for TIES merging on both the visual- and text-dominant tasks,
12

---

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
Answer Length and Accuracy Difference Before and After Merging with Dart
Data Points deseline Lincar 0.2
MathVeers [lex Dominent Score
22
120
Merged with Dert
4.0%
2.2, 8.2
Accuracy Difference
1.0, 0.2
Accuracy Difference (%)
20
0.8, 0.2
0.4
Average Longth
2.0%
2.2. 9
100,04
28
0.8, 0.4
60
1.0%
40
0.0%
26-
-1.0%
20
-2.0%
MathVison Score
groomlry reasornting
geomatty probien solving
algatralc reasoving
math world problem
toxtbook question answering
logical massiing
russoning
arthnelic reasoning
visual question antwering
scientiffc reasoniing
fiuure questinn Bnswering
numanic commoneente
Figure 10: Performance comparison of TIES and lin- merging methods across Math Verte-Text- and Math benchmarks. Baseline presents the re- sult of LLaVA-Nex--LLaMAA-- sult of merging Dart-prop vector with hyperparameters of (0.9, 0.1). Performance reached by TIES merging with the same two task vectors are labeled as the hyper- parameter pair used.
statistical
Linear shows the re-
Figure 9: The relationship between answer length change and accuracy improvement after merging, classi- fied by task.
Method
Weights
Math
Wath
Baseline
13.8
25.9
Linear
(0.9, 0.1)
14.8
30.7
TIES
(1.6, 0.2)
14.5
31.4
Dare-TIE
(1.0, 0.2)
17.8
22.7
Dare-Linear
(1.2, 0.2)
17.4
21.7
Swap5layers
0.19
15.1
29.6
Table 5: Performance comparison of different merging methods on Math Vision and Math
datasets.
Math Vision and Math Verse-Text-Dominggg accordingly. Compared to the performance yielded by linear merging, most combinations of TIES trade off gains in one domain against losses in the other. Additionally, the constraint of on + 02 = 1 commonly dropped in practice because of sparsification during TIES merging process, As result, TIES merging requires more hyperparameter tuning to achieve comparable performance in expanded searching space.
We also expand our experiments the other methods and results shown in Table 5. The results performs comparably TIES merging, consistent with our finding that different merging methods yield similar performance, with none significantly outperforming simple averaging. This supports our choice to adopt the linear merging method. This finding is also consistent with the previous finding in Yadav et al. (2024) that different merging methods tend to exhibit similar behaviors at larger scales. Given this, we chose not to focus extensively on exploring alternative merging methods but instead to explore more about the interpretability and composition of the model's internal abilities.
D. dditional results demonstrating generalization from logical to mathematical reasoning
To investigate whether reasoning skills from other domains, such as logical reasoning, can generalize to multi-modal math reasoning, we fine-tune LLaMA3-SB on LogiCoT (Liu et al., 2023)-a logical chain-of-though bataset-and merge with LLaVA. As shown Table 6, this logic-focused model improves performance on math reasoning tasks, suggesting generalization capabilities across the reasoning domains and highlighting the potential for transferring reasoning skills from textual to multi-modal settings.
13

---

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
Math Vista
Math Verse Benchmarks
Model
Method
MV
All
General
Math
Overall
T-D
T-L
V-I
V-D
V-O
Bascline
37.4
51.7
25.4
20.1
25.9
20.8
21.1
16.5
16.0
13.8
LLaVA
+logic
37.0 J0.4
49.1 42.6
26.7 11.3
23.5
13.4
30.5
14.6
25.0
14.2
26.4
15.3
20.3
13.8
15.1 J0.9
11.2
J2.6
Table Performance (%) of LLaVA before and after merging with the LLM fine-tuned on logical training data (Liu et al., 2023) Math Vista, Math Verse and Math Vision benchmarks. Arrows denote absolute change from the baseline.
Vist
Math Benchmarks
MMStar
Model
Method
DM
MV
All
General
Math
Overall
T-D
T-L.
V-I
V-D
V-O
All
Math
Baseline
61.2
69.6
54.1
31.8
35.9
31.4
31.5
33.1
26.9
59.9
59.2
34,4
21.1
Qwen2-VL
+Qwen2-Math
60.2
68.0
53.5 10.6
31,9 10.1
37.1
31.7
31.5
32.5
26.7
59.5
58.4
J0.8
35.0 10.6
21.7 10.6
Table 7: Performance (%) of Qwen2-VL before and after augmenting with Qwen2-Math Math Vista, Math Benchmarks, MMStar, DM and MV. Arrows denote absolute change from the baseline.
E. More results on Qwen2-VL.
To assess the generality of this approach, we further extend our experiments stronger vision-language model, Qwen2- VL. In Table 7, we present the results of merging B-een--V--B--nnsse with Qwen2-Math-7B. The merged model shows significant improvements on Math Verse, DynaMath and Math Vision, but a performance drop on Math Vista and MMStar. This uneven pattern mirrors our observations with Idefics and Intern suggesting that VLMs already pretrained on mathemmtics-focussee text corpora-as is common in many state-of-the-art models today-derive less benefit from integration with specialized reasoning modules (Laurençon al., 2024).
F. More results on MM-Math
Model
Method
MM-Math
Baseline
0.61
+Dart-prop
0.71 10.10
DDart-nnifomm
0.86 10.25
+MAmmoTH-1
0.68 10.07
LLaVA
+ MAmmoTH-2
1,46 10.85
+ +aggic--v..3
1.30 10.69
+ DDeepSek-RR
0.27 J0.34
+Dart-kee--Sle
1.05 10.44
Baseline
4.00
MetaMatth
4.68 10.68
Dart-prop
2.63 J1.37
Idefies
+ Dart-uniform
2.73 41.27
+MAmmoTH-1
4.03 10.03
+MAmmoTH-2
3.80 J0.20
Baseline
22.70
Intern
+ +Dart-uniform
22.80 10.10
Table Addition experiment result on MM-Math.
In Table 8, we provide detailed results our method applied to the MM-Math (Sun et al., 2024) benchmark, presenting general improvement brought by model merging in visual reasoning.
14

---

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
G. Significance Test
To evaluate whether the improvements from merging are significant, we conduct significance test and mark in Table ??. Merging general reasoning models like Mammoth2 and Magpie does not yield statistically significant enhancements, exhibiting pattern of marginal accuracy gains, shown in Table 2 of our paper. In contrast, merging math-focused LLMs, such as Dart-Prop, results in statistically significant improvements. This supports our conclusion that merging with math-related models offers the greatest advantage.
Vista
Math Verse Benchmarks
MMStar
Model
Method
DM
Math
Overall
T-D
T-L
V-I
V-D
V-O
Math
Baseline
25.4
20.1
25,9
20.8
21.1
16.5
16.0
30.0
22.7
Unifor
28.3 12.9
23.6 13.5
32.0"
25.6'
25.4"
19.3
17.4
31.6
24.5 11.8
TH-2
25.7 10.3
20.6 10.5
26.0
22.0
22.1"
16.4
16.1
30.0
22.5 J0.2
LLaVA
+Magpie-v0.3
25.9 10.5
20.7 10.6
26.8
22.2
22.6
16.2
15.5
30.8
10.8
22.7 10.0
+Dart-prop
28.9 13,5*
23.7 "3.6"
30.7"
24.8*
25.5'
19.5
17.4
33.6 13.6"
Table 9: Significant test results across the models and datasets.
marks statistically significant improvements (p 0.05).
15


Reviews:
[{"review_id": "N0XX1ziS08", "reviewer": "Reviewer_wWNd", "summary": "This paper investigates the impact of integrating math-specific LLMs into VLMs through model merging. The experimental results demonstrate that this approach effectively transfers reasoning abilities from math-specific LLMs to VLMs in a training-free manner. Furthermore, the authors conduct extensive experiments to analyze the distribution of perception and reasoning abilities within the model’s parameter space by modifying different layers. Their findings indicate that perception capabilities primarily originate from the early layers, while reasoning abilities are predominantly derived from the middle-to-late layers. Additionally, they conclude that merging with reasoning models enhances the mathematical reasoning ability across most layers of the model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and detailed comments! We address your questions below.\n\nQ1: \n>The authors assert that \"after merging, we observe that all layers..whereas the distribution of perception abilities across layers remains largely unchanged.\" I could not find any text discussing this observation.\n\nThanks for your advice! This observation is briefly discussed in Section 4 (line 376–383) in paper. After merging with Dart, nearly all layers exhibit a larger performance drop in math reasoning tasks compared to LLaVA, suggesting that reasoning capabilities are more widely integrated across layers post-merging. \n\nThe statement about “distribution of perception abilities across layers remains largely unchanged” is an **implicit deduction**. \n- the higher influence of all layers in math reasoning task (Math subset of MathVista) implies that reasoning is layered on top of perception abilities, rather than replacing them. Since this task requires both perception and reasoning, the observation that early (perception-focused) layers continue to show sensitivity by masking supports this interpretation. If reasoning overrides perception, the influence of early layers would diminish—but it is not observed.\n- Besides, in general VQA tasks (see left side of  ⑤ and ⑦ vs. ⑥ and ⑧), we observe no substantial shift in the behavior of individual layers after merging. This further supports the idea that perception-related representations remain stable.\n\nWe appreciate your feedback and will revise the discussion in Section 4 of the next version to clarify this point.\n\nQ2: \n>More layers show an impact on accuracy does not necessarily imply an improvement in their reasoning ability. \n\nThank you for your thoughtful feedback! We agree that the original phrasing may convey a stronger claim than intended. We will revise it to clarify that our analysis suggests a potential correlation, not conclusive evidence. Specifically, we hypothesize that the presence of more layers may correlate with improved reasoning performance, as the amount of reasoning-related latent knowledge stored across layers may also increase. \n\nQ3:\n> it is unclear why the authors did not explore the use of other powerful logical LLMs.\n\nThank you for your valuable feedback. We share your interest in exploring the generalization of reasoning capabilities across domains, including logical reasoning. \n\n- Our paper already includes diverse LLMs trained on various reasoning tasks. Specifically, we incorporated Magpie, which covers planning and coding tasks (see [1] for details), and Mammoth2, a model trained on various reasoning domains including Socratic reasoning - a form of logical reasoning. The summary of training data as shown below.\n\n|Domain|Size|Subjects|\n|-|-|-|\n|MathStackExchange|1484630|Mathematics|\n|ScienceStackExchange|317209|Physics, Biology, Chemistry, Computer Science|\n|Socratic|533384| Mathematics, Science, Humanities (Logical)|\n\nPerformance results for both are at table 1 in our paper. They show only minor gains, suggesting that combining general-purpose reasoning models remains insufficient for strong performance on math-heavy benchmarks.\n- To explore further on logical reasoning, we add the experiment by fine-tuning a LLaMA3-8B on LogiCoT [3] (a logical chain-of-thought dataset) and merging with LLaVA. As results shown below, this pure logic-focused approach helps the math-reasoning tasks, indicating good generalization ability, we will update these experiment results in our paper. \n|**Model**|**Method**|**MathVista**|||**MathVerseBenchmarks**||||||**MV**|\n|-|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|||All|General|Math|Overall|T-D|T-L|V-I|V-D|V-O|||\n|LLaVA|Baseline|37.4|51.7|25.4|20.1|25.9|20.8|21.1|16.5|16.0|13.8|\n||+logic|37.0|49.1|26.7↑1.3|23.5↑3.4|30.5|25.0|26.4|20.3|15.1|11.2↓2.6|\n\nQ4: \n>Some of the character sizes of figures appear to be too small.\n\nThank you for the feedback! We will enlarge the font sizes in these subfigures to ensure clarity  in the revised version.\n\nQ5: \n>I am curious about the performance improvements observed in Figures 4 and 5 when modifying or replacing the model parameters. \n\nFor instance, in the math-targeted VQA tasks ① and ③ (right panel) shown in Figure 4, performance notably improves after masking by LLaMA, indicating that the image SFT training—designed to equip LLaVA with image embedding comprehension—**may overemphasize learning image perception**, and cause forgetting issues to diminish LLaMA’s inherent mathematical reasoning skills. Restoring the original parameters recovers this capability, highlighting the need to preserve strong reasoning skills in VLMs to counteract domain shift introduced by image training.\n\n[1] Magpie: Alignment Data Synthesis From Scratch By Prompting Aligned Llms With Nothing, Zhangchen et al. ICLR 2025.  \n[2] MAmmoTH2: Scaling Instructions from the Web. Xiang et al. NeurIPS 2024.  \n[3] LogiCoT: Logical Chain-of-Thought Instruction Tuning. Liu et al.EMNLP 2023", "claims_and_evidence": "Most claims are supported by corresponding empirical evidence. However, the authors assert that \"after merging, we observe that all layers begin to contribute to reasoning, whereas the distribution of perception abilities across layers remains largely unchanged.\" I could not find any text discussing this observation in further detail.", "methods_and_evaluation_criteria": "Model merging is a common and effective approach in the field of vision-language models.", "theoretical_claims": "This paper is empirically focused, with minimal theoretical equations or claims.", "experimental_designs_or_analyses": "The experimental results (e.g., Tables 2 and 3) and analysis (e.g., Figures 4 and 5) are detailed and well-presented.  I appreciate the thorough analysis of Figures 4 and 5.  However, I find the conclusion stated in line 375—\"Merging with Reasoning Models Enhances Almost All Layers’ Math Reasoning Ability\"—to be inconsistent with Finding 4, which states, \"The contribution of the middle-to-late MLP layers and almost all attention layers to math reasoning has increased.\"  The observation that more layers show an impact on accuracy does not necessarily imply an improvement in their reasoning ability.  Instead, I believe this section presents a more robust analysis rather than definitive evidence of enhanced reasoning ability.", "relation_to_broader_scientific_literature": "This paper demonstrates that integrating math-specific LLMs can consistently enhance the reasoning capabilities of VLMs, offering valuable insights into the field of multimodal integration.", "essential_references_not_discussed": "As far as I know, most essential references have been appropriately discussed in the manuscript.", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper provides a comprehensive analysis of integrating the reasoning abilities of large language models (LLMs) into vision-language models (VLMs). It includes detailed performance comparisons and insightful visualization analyses, effectively highlighting the impact of this integration.\n\nWeaknesses:\n\nA major concern is the experimental setting. Although merging LLMs with extensive mathematical knowledge into VLMs is rational, given that math abilities are crucial for logical reasoning, it is unclear why the authors did not explore the use of other powerful logical LLMs.", "other_comments_or_suggestions": "Some of the character sizes of figures, particularly in Figures 4 and 5, appear to be too small, which may make them difficult for readers to interpret.", "questions_for_authors": "I am curious about the performance improvements observed in Figures 4 and 5 when modifying or replacing the model parameters. Intuitively, such changes would be expected to either maintain or decrease performance rather than enhance it. Could you clarify this discrepancy?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KkDdeC1JER", "reviewer": "Reviewer_V8q8", "summary": "This paper proposes merging models across modalities to enable the incorporation of the reasoning capabilities of LLMs into VLMs. Effectiveness of this training-free recipe is verified via extensive experiments. It also evaluates VLMs in different sizes to verify the generalization ability of merging. Additionally, this work provide interesting insights about perception and reasoning abilities across different layers by investigating into merged VLMs.\n\n\"## update after rebuttal\nWhile I appreciate the authors’ efforts to address most of my concerns, their response to my questions about leveraging the finding—that perception and reasoning abilities lie in different areas of MLLMs—for improved merging performance is not insightful enough. It's claimed that different merging methods behave very similarly at larger scales. Consequently, I maintain my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and detailed comments! Your suggestions will be helpful in improving the paper. \n\nQ1:\n> Lack of baselines of other merging methods. Only TIES is provided in the appendix.\n\nWe appreciate the suggestion and have expanded our experiments to evaluate DARE merging [3] (Dare-TIES and Dare-Linear), another SOTA merging method to explore additional merging techniques. We follow the hyperparameter search strategy same as TIES, parameterized by ($\\alpha_1$, $\\alpha_2$), where $\\alpha_1$ is tuned for the VLM and $\\alpha_2$ for the Math LLM, in order to obtain the most comparable checkpoints on benchmarks that emphasize vision and textual reasoning. The results are shown below. \n\n||**MathVision**|**MathVerse-Text-Dominant**|\n|:-:|:-:|:-:|\n|Baseline|13.8|25.9|\n|Linear|14.8|30.7|\n|TIES(1,0.4)|15.1|29.7|\n|TIES(1.6,0.2)|14.5|31.4|\n|Dare-TIES(0.8,0.2)|17.8|22.7|\n|Dare-Linear(0.8,0.2)|17.4|21.7|\n\nThe results show that DARE performs comparably to linear merging and TIES, consistent with our finding that different merging methods yield similar performance (line 657), with none significantly outperforming simple averaging. This supports our choice to adopt the linear merging method in our paper. This finding is also consistent with the previous finding in [2] that different merging methods tend to exhibit similar behaviors at larger scales. Given this, we chose not to focus extensively on exploring alternative merging methods but instead to explore more about the interpretability and composition of the model’s internal abilities.\n\nQ2:\n>It would be more interesting to also include other model families or larger models.\n\nWe focused on LLaVA for our interpretability experiments mainly due to resource limits. Specifically, we conducted masking-out experiments on each even layer (for MLP/Attention, respectively) across four configurations (e.g., masking LLaVA to LLaMA). These experiments, presented in Figures 4 and 5, involved a total of 128 runs. We use GPT-4 to evaluate these runs (by VLMEvalKit), which incurred a substantial cost of approximately $3,000. Due to the high computational cost, our study currently focuses on LLaVA. However, we recognize the value of extending this analysis to other model families and leave it as promising future work.\n\nQ3:\n>Lack of significance test…\n\nThank you for your valuable feedback! We acknowledge that some of the performance improvements in Table 2 fall within a 1% margin. We conduct t-tests to compare the results of the merged model with the base LLaVA. The resulting p-values are presented in the table below, with statistically significant results (p < 0.05) shown in bold. Merging general reasoning models such as Mammoth2 and Magpie does not lead to statistically significant improvements, showing a similar pattern of marginal accuracy gains as shown in Table 1 of our paper. In contrast, merging Math LLMs, such as Dart-Prop, leads to statistically significant improvements. This aligns with our conclusion in the paper (line 204) that merging with math-related models provides the greatest benefit.\n\n|**Model**|**Method**|**MathVista**|**MathVerse**|||||**MMStar**|**DM**|\n|-|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|||Math|T-D|T-L|V-I|V-D|V-O|Math||\n|LLaVA|+Dart-Uniform|0.15|**0.00**|**0.00**|**0.00**|**0.02**|0.46|0.56|0.21|\n||+MAmmoTH-2|0.77|0.58|0.30|**0.04**|0.70|0.35|1.00|1.00|0.82|1.00|\n||+Magpie-v0.3|0.67|1.00|0.06|0.07|0.84|0.62|0.56|0.96|\n||+Dart-prop|0.06|**0.00**|**0.01**|**0.00**|**0.01**|**0.02**|**0.05**|**0.00**|\n\nQ4: \n> I'm interested in how to leverage the findings that perception and reasoning abilities lie in different areas of VLMs, so that we can achieve better merging performance.\n\nDuring our exploration, we wondered whether insights from the decomposition of perception and reasoning abilities could be leveraged to develop better merging methods, similar to the approach proposed by [1]. To investigate this, we experimented by keeping the early layers unchanged to preserve perceptual ability while only merging the later layers to enhance reasoning ability (see Q3 to Reviewer xnvF for details). Our results indicate that this approach does not yield any performance gains.  Given no clear performance gain, we opted for the simpler solution. Our finding is also consistent with previous findings from [2] that **different merging methods behave very similarly at larger scales**. \n\n[1] Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models, Lucas et al. ICLR 2025.  \n[2] What Matters for Model Merging at Scale?. Yadav et al. arXiv 2024.  \n[3] Language models are super mario: Absorbing abilities from homologous models as a free lunch. Yu et al.ICML. 2024.", "claims_and_evidence": "Yes, via extensive experiments and case study of LLaVA.", "methods_and_evaluation_criteria": "Yes, the reasoning ability is measured by math reasoning benchmarks, while perception ability is measured by general VQA benchmarks.", "theoretical_claims": "There's no theoretical claims in this paper.", "experimental_designs_or_analyses": "Yes, I've checked the main experiments and analysis of merged LLaVA.", "relation_to_broader_scientific_literature": "The main contribution of this work is to apply a training-free recipe to merge VLMs with reasoning expert models. It's more a combinational innovation because the linear merge recipe was proposed by prior work.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "## Strength:\n1. The problem addressed is timely and interesting, it might provide a new perspective for inference time scaling in VLMs.\n2. It conducted extensive experiments on different size VLMs, demonstrating noticeable improvements and good generalization ability.\n3. The analysis focusing on merged LLaVA provide insights about perception and reasoning patterns across different layers.\n\n## Weakness:\n1. Lack of baselines of other merging methods. Only TIES is provided in the appendix. \n2. The investigation about perception and reasoning pattern across layers only considers LLaVA. It would be more interesting to also include other model families or larger models, such as Idefics2-8B, InternVL2-LLaMA3-76B.\n3. Lack of significance test since many of the performance improvements in table 2 are within a 1% margin.", "other_comments_or_suggestions": "See Strengths And Weaknesses", "questions_for_authors": "I'm interested in how to leverage the findings that perception and reasoning abilities lie in different areas of VLMs, so that we can  achieve better merging performance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bXabTPQYlC", "reviewer": "Reviewer_xnvF", "summary": "This is an experimental paper. This paper enhances VLMs with reasoning capabilities using Reasoning LLMs. Specifically, this paper integrates reasoning capability of Reasoning LLMs into VLMs through linear merging. Extensive experiments demonstrates the effectiveness of this simple model merging method on math reasoning tasks. Additionally, this paper experimentally disentangles perceptual and reasoning abilities within the model parameter space.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments! We address the concerns below.\n\nW1: \n>The workload of experimental analysis and discovery in this paper is not enough…For examples…why not try a dynamic model merging method considering this finding?\n\nWe sincerely appreciate your feedback on our work. But we respectfully disagree with the comments on “insufficient workload” for the following reasons:\n- A Simple Method Does Not Mean Less Work: We actually tried it in our early experiments but it worked similarly to the simple linear merging method [see response to Q3]. Given no clear performance gain, we opted for the simpler solution. \n - We want to emphasize here again that our goals are:\n    - **To see whether the perception and reasoning abilities can be infused by model merging**. We incorporate three VLMs of varying architectures and sizes, along with six reasoning models across diverse domains, and evaluate on five multimodal reasoning benchmarks.This evaluation process accruing costs of around $1,000. \n    - **To understand how perception and reasoning abilities are distributed in parameter space** within a VLM. To achieve this: In Figures 4 & 5, we conducted masking experiments on each even layer (MLP/Attention) under various configurations (e.g., masking LLaVA to LLaMA or random noise), evaluating each independently—resulting in 128 runs in total. In addition, we conducted 4–5 such preliminary experiments. We use GPT-4 (via VLMEvalKit) to evaluate model outputs, accruing costs of around $3,000. \n     - Although **GPU hours and monetary costs** do not directly equate to the quality of the work, these data points **do illustrate one aspect of the hidden workload** we have put into this work. \n\nBesides, we also add results based on Qwen model [see response to Q1], hopefully these added results can help recognize the workload of this paper.\n\nQ1/2: \n>include Qwen-based model.., report the results of InternVL on other benchmarks considering that LLaVA and Idefics .\n\nThank you for the advice! We expand our experiments for Qwen-based model here. Specifically, we merge Qwen2-VL-7B-Instruct with Qwen2-Math-7B, the results are:\n|**Model**|**Method**|**MathVista**|||**MathVerseBenchmarks** ||||||**MMStar**||**DM** |**MV**|\n|-|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|||All|General|Math|Overall|T-D|T-L|V-I|V-D|V-O|All|Math|||\n|Qwen2-VL|Baseline|61.2|69.6|54.1|31.8|35.9|31.4|31.5|33.1|26.9|59.9|59.2|34.4|21.1|\n||+Qwen2-Math|60.2|68.0|53.5↓0.6|31.9↑0.1|37.1|31.7|31.5|32.5|26.7|59.5|58.4↓0.8 |35.0↑0.6|21.7↑0.6|\n\nFrom the results, we see that merging benefits certain benchmarks like MathVerse, DynaMath and MathVision. However, there is a decline in performance for benchmarks MathVista and MMStar. The unstable increase pattern is similar to what we observe with Idefics and InternVL, indicating that VLMs that have undergone math-related pure text training  (as many state-of-the-art models do today [1]) benefit less from merging with reasoning models.\n\nAs for the large-scale model InternVL-76B, we have already evaluated it on the same benchmarks, as shown in the InternVL row of Figure 3 in our paper.\n\nQ3: \n> Did the author try to only merge reasoning llm in the later layers? would this allow VLMs to achieve a good balance between perception and reasoning?\n\nThank you for the great suggestion of trying a dynamic model merging method! Like we have briefly discussed this in W1 response, we did test a dynamic approach based on our layer-wise insights.Inspired by the observation that perceptual abilities lie in early layers, we merge only the later layers to preserve them.\n\n|**Model**|**Method**|**MathVista**|||**MathVerseBenchmarks**||||||**MMStar**||**DM**|**MV**|\n|-|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|||All|General|Math|Overall|T-D|T-L|V-I|V-D|V-O|All|Math|||\n|LLaVA|Baseline|37.4|51.7|25.4|20.1|25.9|20.8|21.1|16.5|16.0|43.8|30.0|22.8|11.8|\n||+Dart-prop|38.0|48.7|28.9↑3.5|23.7↑3.6|30.7|24.8|25.5|19.8|17.4|43.6|33.6↑3.6|24.5↑1.7|14.8↑3.0|\n||+keep5layers|37.9|48.5|28.9↑3.5|22.0↑1.9|29.6|23.2|23.2|17.8|16.4|43.8|30.8↑0.8|23.7↑0.9|15.1↑3.3|\n\nAlthough the results show clear gains over the baseline, they are only on par with linear merging. Given no clear performance gain, we opted for the simpler solution. One possible reason is that interpretation-based analysis is inherently subjective and only captures coarse-grained trends, often with substantial noise. For instance, while later layers may show stronger reasoning, earlier layers still play a role. Thus, merging only the later layers in reasoning-focused LLMs may not yield performance gains. Decomposing the two abilities may enable successful merging but doesn’t necessarily yield a better merging method. This also consists with previous findings from [2] that **different merging methods behave very similarly at larger scales**. \n\n[1] What matters when building vision-language models? NeurIPS 2024.  \n[2] What matters for model merging at scale? Prateek et al. 2024", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "They experimentally proof their claim in Sec. 4 and Sec. 5.", "experimental_designs_or_analyses": "Recently, there are many reasoning LLMs based on Qwen-series, I think the author should consider including this series in experimental designs to further valid their claims.", "relation_to_broader_scientific_literature": "Methods in this paper is widely used. But as far as I know, I haven't seen similar findings of this paper in other literature.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "1. To valid the effectiveness of merging model to enable reasoning capability of VLMs, more comprehensive experiments should be conducted. Like Qwen-based model, Results of InternVL in other benchmark.\n2. The technical contribution is minimal. I would not be inclined to reject a paper based solely on the method. But the workload of experimental analysis and discovery in this paper is not enough to support it to be accepted by a top-tier conference. There is still a lot to do after getting the analysis and discovery. For examples, as the author found that the perception ability primarily located in the early layers and the reasoning ability mainly lied in later layers, why not try a dynamic model merging method considering this finding?", "other_comments_or_suggestions": "No", "questions_for_authors": "1. Can the author try to include Qwen-based model to futher valid the effectiveness of model merging?\n\n2. Why not report the results of InternVL on other benchmarks considering that LLaVA and Idefics report them. Can the author also reports them to make the claim more confidential.\n\n3. Based on the finding in Sec. 5, Did the author try to only merge reasoning llm in the later layers? would this allow VLMs to achieve a good balance between perception and reasoning? For multimodal math reasoning benchmarks, there is no doubt that VLMs also needs a strong perception to understand the image input.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission investigates merging vision-language models (VLMs) with large language models (LLMs) specialized in reasoning, demonstrating effective transfer of reasoning capabilities into VLMs via a simple linear merging method. Reviewers acknowledged the paper’s comprehensive experimental validation and insightful layer-wise analyses revealing that perception capabilities predominantly reside in early model layers, whereas reasoning emerges from middle-to-late layers. The authors successfully addressed most reviewer concerns through detailed rebuttals, providing additional experimental validations, such as incorporating Qwen-based models and significance testing. However, reviewers highlighted limitations, including the modest novelty of the merging technique itself and the scope limited primarily to math-based reasoning models. Despite these, the reviewers collectively agreed the paper makes valuable contributions in interpretability, clearly demonstrating the utility of cross-modal model merging for understanding internal model dynamics. Considering the detailed responses and additional results provided, the reviewers leaned towards acceptance, emphasizing the meaningful empirical insights presented.
Decision: Accept (poster)