REMEDY: Recipe Merging Dynamics in Large Vision-Language Models
Authors: Didi Zhu
Yibing Song
Tao Shen
Ziyu Zhao
Jinluan Yang
Min Zhang
Chao Wu
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-Modal Large Language Models
Zero-shot Generalization
Model Merging
Primary area: foundation or frontier models, including LLMs

Abstract: Model merging has emerged as a powerful technique for combining task-specific vision models into a unified and multi-functional model. Previous methods represented by task arithmetic, have demonstrated effectiveness and scalability in this domain. When large vision-language models (LVLMs) arise with model size scaling up, this design becomes challenging to fuse different instruction-tuned LVLMs for generalization enhancement. The large scale and multi-modal nature of LVLMs present unique obstacles, including constructing reusable and modular components to accommodate the multi-component architecture of LVLMs and the requirement for dynamic fusion based on multi-modal input tokens. To address these challenges, we propose the \textbf{RE}cipe \textbf{ME}rging \textbf{DY}namics (REMEDY) method, a scalable and flexible paradigm for model merging in LVLMs. We first define reusable modules termed \textit{recipes} including the projector and shallow LLM layers, enhancing visual-language understanding. Then, we introduce a modality-aware allocator dynamically generates weights in a one-shot manner based on input relevance to existing recipes, enabling efficient cross-modal knowledge integration. REMEDY thus offers an adaptive solution for LVLMs to tackle both seen (i.e., multi-task learning) and unseen (i.e., zero-shot generalization) tasks. Experimental results demonstrate that our method consistently improves performance on both seen and unseen tasks, underscoring the effectiveness of REMEDY in diverse multi-modal scenarios.

Published as a conference paper at ICLR 2025
REMEDY: RECIPE MERGING DYNAMICS IN LARGE VISION-LANGUAGE MODELS
Didi Zhu Yibing Song Tao Shen Ziyu Jinnuuan Yang* Min Zhang', Chao Wu Zhejiang University, DAMO Academy, Alibaba Group, *Fast China Normal University. Academy of Social Governance, Zhejiang University, *Hupan Lab
Academy Computer Science and Technology, Zhejiang University didi ridi_zhuēzju. edu. en, wuazju www.wu.edu.cccooo edu.co
ABSTRACT
Model merging has emerged as powerful technique for combining task-specific vision models into a unified and multi-functional model. Previous methods rep- resented by task arithmetic, have demonstrated effectiveness and scalability in this domain. When large vision-language models (LVLMs) arise with model size scaling up, this design becomes challenging to fuse different instruction-tume LVLMs for generalization enhancement. The large scale and multi-modal nature of LVLMs present unique obstacles, including constructing reusable and modu- lar components to accommodate the multi-componee the requirement for dynamic fusion based on multi-modal input tokens. To ad- dress these challenges. we propose the REcipe MErging DYnamics (REMEDY) method, a scalable and flexible paradigm for model merging in LVLMs. We first define reusable modules termed recipes including the projector and shallow LLM layers, enhancing visual-language understanding. Then, we introduce a modality- aware allocator dynamically generates weights in a one-shot manner based on input relevance to existing recipes, enabling efficient cross-modal knowledge inte- gration. REMEDY thus offers adaptive solution for LVLMs to tackle both seen (i.e., multi-task learning) and unseen (i.e., zero-shot generalization) tasks. Experi- mental results demonstrate that method consistently improves performance both seen and unseen tasks, underscoring the effectiveness of REMEDY in diverse multi-modal scenarios.
architecture of LVLMs and
INTRODUCTION
Visual recognition strives to establish a robust alignment between visual perception and linguistic understanding (Liu et al., 2020). It has evolved from basic image classification (Krizhevsky et al.,
2012: Huang et al., 2024b; 2023; 2022; Zhang et al., 2024; 2023b; Zhu et al., 2023b; Tong et al., 2023: Zhu et al., 2023c; 2024a; Chen al.) to complex tasks like detection and segmentation (Anderson et al., 2018; Feng et al., 2022; Bian et al., 2024; Feng et al., 2025; Zhu et al., 2023d; 2024c). This progression has led advanced research in visual grounding (Plummer al., 2015), referring expression comprehension (Yu et al., 2016), and referring image segmentation (Hu et al., 2016). Along with model scaling up. the alignment between vision and language has been advanced in the Large LisguaLLanguuge Models (LVLMs) form (Lyu et al., 2022a; Dai et al., 2023; Liu et al., 2023: Zhu et al., 2023a; Wang et al., 2023; Zhu al., 2024b; Lv et al., 2025a; Zhao al., 2024a). The visual content is projected into the language space for visual question answering (VQA) via large language models (LLM). As the variety and complexity of vasion- (VL) tasks continue to grow, the demand for quickly and efficiently adapting L.LLMs to downstream tasks has become increasingly critical.
In light of this, model merging emerges promising paradigm, facilitating the synthesis of task- specific models into unified model capable of simultaneously addressing diverse downstream tasks (Ilharco al., 2022; Yadav al., 2023). Prior studies such the task arithmetic (Ilharco et al., 2022) design develop 'task vectors' for model merging via arithmetic operations, which benefit knowledge
"Corresponding author

---

Published as a conference paper at ICLR 2025
transfer across different tasks. Building on this, TIES- (Yadav et al., 2023) and AdaMerging (Yang et al., 2024b) advance the merging process to achieve greater versatility by model pruning and adapting merging coefficients, respectively. They have achieved significant success in vision models for recognition scenarios while leaving LVLMs under-explored.
Merging LVLMs encounters emerging challenges, which analyzed in two aspects. (1) Large Scale of LVLMs: LVLM typically consists of a visual encoder, projector, and LLM (Dai et al., 2023; Liu al., 2023). The parameters of the visual encoder and LLM are typically huge (e.g., in LLaVA 1.5 there are 0.3B vision encoder and 7B or 13B LLM). Adapting these large models to specific tasks challenging due limited task-specific data. For instance, the ScienceQA dataset has only 16,96 image-question pairs (Lu et al., 2022), insufficient for comprehensive fine-tuning. Therefore, focusing on specifie LVLM submodules for efficient knowledge transfer becomes crucial. (2) Specificity of VI. Task Discrepancies: In LVLMs, the inputs contain both visual and language data, which differs from prior model merging designs where only visual content is processed. Un- like unimodal scenarios, task discrepancies in LVLMs can arise from either the visual or linguistic aspects, or both, As shown in Figure the same globe image might require inferring a city name in one task, while in another task it may need generate image description. This variability extends to zero-shot scenarios, where LVLMs are expected to handle novel visual- linguis-ii combinations, a challenge surpassing traditional unimodal zero-shot tasks. To enhance both multi-task learning and zero-shot generalization, there is a need for dynamic fusion mechanisms capable of adjusting model merging strategies for each visual-lane hauuulla pair.
To address the challenges outlined above, we propose REcipe MEEgingg DYnamics (REM- EDY), a paradigm for merging models in LVLMs that overcomes the constraints of tra- ditional vision-based model merging strategies. As illustrated in Figure 1, REMEDY is com- posed of two main steps: (1) Recipe Construc- tion: We define the reusable modules men- tioned in the challenges as recipes. Through extensive experiments on LVLMs, we have determined the effective composition of these recipes, which includes the projector and the shallow layers of the large language model. This configuration enhances the model's visual perception capabilities and improves visual- language interaction understanding, rather than merely mimicking output styles (Ghosh al., 2024). (2) Recipe Merging: After con- structing the recipes, We introduce modality- aware allocator and employ few-shot learning paradigm to train it, enabling efficient adaptation with minimal data requirements. This allocator can instantly capture the knowledge relevance between input image-text tokens and existing recipes, and perform one-shot weight allocation based on this captured relevance. By dynamically adjusting the multi-modal nature of inputs, it facilitates effective integration of knowledge across different tasks and modalities. Results from extensive experiments demonstrate the empirical effectiveness of REMEDY multimodal benchmarks, showcasing substantial improvements in performance over previous methods.
Customization
Image
Question:
Unseen
Please
LVLMS
Image Question:
Quastion:
Seen
inference the city
Give caplion this image.
Recipe
Recipe
naith.
Construction
Construction and Customization Process of REMEDY. The construction phase uses datasets from "Seen" tasks to create model recipes, while the customization phase applies these recipes to "Unseen" tasks.
Figure 1:
The main contributions of this work are threefold. First, analyze LVLM components and con- struct reusable recipes for efficient adaptation. Second, we propose REMEDY with modality- aware allocator that enables dynamic, one-shot weight allocation through few-shot learning. Finally, through extensive experiments, we demonstrate REMEDY's effectiveness in model merging and knowledge transfer across tasks.
2

---

Published as a conference paper at ICLR 2025
Mutti Modal Input
Instruction: Scrutinize the question and correspanding multimedia input, select the right respanse from the thoices, and compose clear, concise ex planation that covers the essential aspocts your reasoning and the scientific concepts behind your decision.
Answer by Finetuned on the last two layers of LLM and Projecotr:
answer
Answer by Finetuned on the former two layers of LLM:
The answer B The colony shown in the image is Rhode island. The map pinpolnnts the location of the and chares coastl with the Atlantic corner further emphasizes its location map.
colony. bordered Connerticut, New York, and New
Ocean presence the colony's ss flag X
What the name of the colony shown? New York
Answer by Finetured
two layers
Prajector:
Rhode Island
The correct answer is because the colony called "Rhode Island". The multimedia input features map and highllihtt the location of Rhode island, which found in the northnastern region of the Lnittd States, It is not New York or New jersey, as those are other colonies with different nomes,The The answer B.
Carolina
New Jersey
Figure 2: Comparison of fine-tuning strategies for LLaVA model performance on multi-modal question answering. Fine-tuning the former two layers yields superior results compared to fine- tuning the last two layers, demonstrating both accurate answers and comprehensive explanations. This suggests that focusing on earlier layers during fine-tuning may be more effective in enhancing LLaVA's comprehension and output quality for multi-modal tasks.
2 EXPLORING RECIPE CONSTRUCTION IN LVLMS
Large vision-language models represented by architectures like LLaVA, consist of three core com- ponents: a vision encoder, a language model, and a cross-modal fusion module that integrates visual and textual representations. Due to the multi-component nature of LVLMs, we introduce recipes as modular functional building blocks for flexible fine-tuning, designed for efficient combination and reuse. These recipes not only enable efficient adaptation to diverse tasks but also reduce com- putational overhead, which in turn facilitates the subsequent fusion of task- tusk-specific knowledge. In developing these recipes for LVLMs, critical question arises: which specific components should be fine-tuned to construct an effective recipe?
To address this, we conducted a comprehensive analysis of LLaVA's structure identify the key elements that contribute most significantly to task-specific adaptation.
Deep Layers. Our ex- periments reveal a significant performance dis- parity when applying Low-Rank Adaptation (LoRA) to shallow versus transformer layers of the LLM. As demonstrated in Table fine- tuning the first two layers of the LLM with projector consistently achieves performance comparable to, even surpassing, full model tuning across various datasets. Notably, on the TextVQA dataset, this shallow tuning approach outperforms even the fully-tuned model. In Fig- ure 5, the model fine-tuned on the former two layers provides accurate answers with comprehensive explanations, whereas the model tuned on the last two layers fails produce correct responses. Specifically, the latter model outputs an incorrect and lack of explanatory context. This behavior likely stems from the format of the training data, where answers may have been presented to merely mimic the output text style without truly comprehending the mudti- input or the question at hand, aligning with the findings in (Ghosh al., 2024).
Shallow Layers
Table Performance comparison across differ- ent fine-tuning strategies. Fine-tuning the first two layers with projector achieves comparable or superior performance to full model tuning.
Dasasei
All
Fist w/ w/Pr
Last w/Pro
wlo
ScrenceQAA
83,23
83.15
78.45
75.06
Flickrll
91.4
91.1
89.6
84.5
CDCD
192.8
152.4
131.8
126.4
TouVQA
59.77
61.89
60.39
59.34
concise, direct manner. As result, the model appears
Effectiveness of Projectors As illustrated in Table 1, across all datasets, the configuration with pro- jectors consistently outperforms the one without. This performance gap is particularly pronounced in tasks requiring fine-grained visual understanding, such as TextVQA, where the projector-enhancee model surpasses even the fully-tuned model. In Figure 5, neglecting to fine-tune the projector can significantly impair the model's visual perception capabilities. This deficiency may lead to visual hallucinations, such as the erroneous mention a "flag" that is absent from the image. The impact underscores the critical role of the projector in maintaining the integrity of visual information pro- cessing within the model. Based on the above findings, we establish the following guidelines for constructing effective recipes:
3

---

Published as a conference paper at ICLR 2025
Stage Recipe Construction
Stage Recipe Marging Strategy Learning
Stage Recipe Merging Dynamics
anuvee
Thiica page
Original Recipes
ait pait
selae mage briety,
MTN
PTN
Aln
FFN
FFN
Wharls Zee mage?
Projecar
Valao Dreader
Sedezithe the coned option
image brarty.
IFN
ADD
Viin Gneader
Prjeekr
Reepe Mergin
that'sshetsc
Video Groader
Capdia
Figure 3: Overview of Recipe Merging Dynamics for large vision-language models. Stage 1: We build task-specific recipes using the former layers of LLM and task task-specific projectors for different tasks. Stage 2: We employ a modality-aware allocator to learn generalizable merging strategies for the original recipes based on few seen vision-language tokens. Stage 3: We apply the learned allocator to new multi-modal imputs, dynamically capturing their relationships with existing recipes and assigning layer-wise weights adaptively based on the specific characteristics of each input.
Fine-tuning the former LLM layers with LoRA and incorporating projectors yields reusable, efficient functional recipes that enhance multi-modal integration and visual understanding.
Building upon our experimental insights, a recipe R for LVLMs is defined as follows: Definition 1 (Recipe). Given an LVLM M with initial parameters W for its recipe layers in LLM and Op for its projector, a recipe R is defined as:
R= A0r),
where AW E R&MN represents the LoRA-induced weight updates for the LLM, with M being the number of adapted Transformer layers, and N being the number of linear layers within each Transformer layer. Each element of AW is defined as AW = As. Bi, with At, B.g E ddxr being low-rank matrices and r d. A denotes the changes in parameters of the projector.
For each of the M Transformer layers of the LLM, the updated weight matrices are given by W W AWW, The projector's updated parameters are O @ p- A++ The projector transforms the visual feature vector v by o' = P(t)= Wr + b/2 where W and b,, are components of O'p.
This finding provides valuable guidance for efficient fine-tuning of other LVLMs with similar ar- chitectures, it suggests that focusing adaptation efforts on the initial layers may be sufficient to achieve strong performance while minimizing computational costs.
3 METHOD
3.1 OBJECTIVE OF RECIPE MERGING
We define two sets of tasks: Twem {T.) representing the seen tasks which the model has been fine-tuned through different recipes, and Tunseen } S+U representing the unseen tasks that have not been encountered during fine-tuning. Each task T, from the set Treen has yielded PPP)
corresponding recipe R; (AW() A as shown in the left side of Fig. 3.
Our goal is to develop a fusion function, denoted F, that combines all these task-specifh recipes into a single set of fused parameters. This fusion function will produce the fused parameters W and Pp:
(W*, Op) =F (Wo, 0%, (R.)e_))
(2)
4

---

Published as a conference paper at ICLR 2025
where W and Op the initial parameters of the LLM and projector, respectively. It is crucial to note that during the fusion process, only the recipes {R,};- fi=1 are merged, while the remaining parts [R]);-1
of the model retain their pre-trained values. The model M(W*, p is expected to maintain high performance on the tasks Teeen while generalizing effectively to the unseen tasks in Tunseen-
We formalize the optimization objective for this fusion maximizing the combined performance over both seen and unseen tasks:
max Σ W TIET.O
P(M(W", Op),, + Σ P(M(W" Op), 7j),
TTET
where PMM(WW, 07), T) represents the performance metric on task T for the given model pa- rameters (W*, OP)..
important to note that while our objective function includes performance on unseen tasks, we only have access data from the seen tasks Tseen during the training process. This presents key challenge: how to optimize for generalization to unseen tasks without direct access to their data. Our approach addresses this challenge by leveraging the diverse knowledge captured in the task-specific recipes and designing a fusion mechanism that promotes robust tross-task generalization.
3.2 MODALITY-AWARE ALLOCATOR
Building upon the above recipes, we introduce the modality- aware allocator, a mechanism designed dynamically fuse multiple task-specific recipes. As illustrated in the middle part of Fig. 3, the allocator takes the input token sequence and outputs a single set of dynamic fusion weights for all layers of the recipes.
3.2. TOKEN UNDERSTANDING COMPONENT.
At the core the Modality-aware Allocator the token understanding module, which employs self-attention mechanism process and analyze the incoming multimodal token sequences. This module captures complex inter-token relationships and modality-specifie relevance, going beyond simple token encoding. Specifically, given input sequence of tokens (æ1, .., ,,, .T). where each token I, may belong to different modalities (e.g., visual or textual), the token understanding module computes modality-sensitivv embeddings h, for each token: h, SelfAttention(2, x). where SelfAAttentoo(() represents the self-attention mechanism that captures dependencies between tokens while accounting for modality information.
By generating these modality modality-sensitive sensitive embeddings h, the module provides a nuanced foundation for the subsequent fusion of multiple recipes, allowing the model adapt its behavior based on the relative importance of visual versus textual information in any given input.
3.2.2 TOKEN-CONDITIONAL WEIGHT GENERATOR
Complementing the token understanding module, introduce the token-conditional weight gen- crator. This component leverages the modality-aware embeddings the entire input sequence to compute single set of dynamic fusion weights for all layers of the recipes.
Given the sequence of token embeddings [h1, by.... hr} produced by the token understanding module, the weight generator generates set of weights y E (MMxxx a one-shot manner:
= fweight (h,, he, hy Owa),
(4)
where fwoighe(:) is a function parameterized by @wG (the parameters of the weight generator), S is the number of task-specific recipes, M is the number Transformer layers, N the number of linear layers within each Transformer layer, and K is the number of layers in the projector. This function maps the sequence of token embeddings to a single set of weights that determine the contri- bution of each recipe to each layer of the model. We ensure that the weights are normalized across recipes for each layer by Ei 7i.j 1, Vj (1,-,,, M....,MA N K}.

---

Published as a conference paper at ICLR 2025
Difference from Routers. This approach distinguishes itself from conventional mixture-of- routers two key aspects. Firstly, it generates single set of weights for the entire input sequence in one-shot manner, rather than producing token-specific weights. Secondly, unlike traditional routers that require deployment at each layer of the model, our method operates globally, generating weights for all layers simultaneously. This holistic, sequence-level weight generation enables more efficient and context-aware processing of multi-modal inputs.
3.2.3 TRAINING THE MODALITY-AWARI ALLOCATOR
To train the modality aware allocator, randomly sample small number of image-text pairs from each seen task as training data. Based on the fused parameters defined in Equation 4, we obtain a merged model M(W P) Given training sample (I, y). where is the input image-text pair and = (y1,-.,, Vr) the target text sequence, we define the training objective for the allocator as:
Callocator O Oww) = Σ log plylleeee W W'))))
(5)
By optimizing this objective with respect to only OwG while keeping all recipe parameters fixed, this autoregressive training objective aligns with the pre-training objective of LVLMs.
3.3 RECIPE MERGING DYNAMICS DURING INFERENCE
The modulation of LoRA weights and projector parameters performed dynamically during infer- ence, as depicted in the right side of Fig. 3. At the start of each forward pass, the input sequence of tokens processed through the Modality-aware Allocator, which outputs single set of fusion weights 7. These weights are then used to fuse the task-specific recipe components. The fused LoRA update AWT, for each layer (i,j) is computed as:
AWT, = Σ Was (W i
Vi E [I,E [],
(6)
where "k,i is the fusion weight for the k-th recipe at the i-th Transformer layer and j-th linear layer. Similarly, the fused parameter update A0P; for each layer of the projector is computed as Δθμμ -S Th, (a()) VI E [1, K]. Ya, is the fusion weight for the k-th recipe the I-th projector layer, and K is the number of layers the projector.
The model's parameters are then updated using these fused components during the forward pass:
W Wt, + AW 0/r1 =0.=0pp- + A0P.
This adaptive fusion mechanism allows the model to effectively leverage knowledge from multiple task-specific recipes, addressing both seen and unseen tasks with improved generalization. By inte- grating the Modality-awane Allocator into our LVLM, we enable dynamic fusion multiple recipes, allowing the model effectively address diverse set of tasks.
EXPERIMENT
4.1 EXPERIMENT SETUP
Architectures and Datasets. To evaluate the effectiveness of REMEDY, we conducted ex- periments using LLaVA-1.5 (Vicuna-7B), widely adopted large vision-language model. Our evaluation was performed on two categories of datasets: seen tasks and unseen tasks. Seen tasks refer to datasets that were used during the LLaVA recipe fine-tuning phase to create task- specific recipes. This category includes four datasets: Flickr30k (Young et al., 2014) and COCO (Lin al., 2014) for image captioning, and ScienceQA (Lu al., 2022) and TextVQA (Singh

---

Published as a conference paper at ICLR 2025
Table Performance comparison of model fusion methods on seen and unseen tasks. H-score represents the harmonic mean of the performance on seen and unseen tasks, providing balanced measure of the model's capability across both task categories. REMEDY demonstrates superior performance compared to other fusion methods and the zero-shot baseline.
Method
Scen tacks COCO
Unseen tasks MMB-EN
All tasks Hscore
TexVQA
SQAGimg)
Flickr30k
Avg
MM-NNi
MMB-CN
VizWiz
POPE
Avg
Zero-shot
67.72
130.7
74.1
71.70
31.1
58.3
64.3
66.31
Simple Average
117.8
84.28
28.4
Tisk
84.5
60.9
Thes Merging
50.45
63.45
109.2
79.1
76.80
23.8
51.19
57.3
08.5
90.32
52.24
64.52
62.21
TW AdaMerging
19.34
72.89
114.6
82.8
82.41
27.9
54.82
39.3
50.9
83.99
53.38
68.90
66.21
LW AdaMerging
00.99
13.67
116.4
86.9
84.74
28.1
53.24
60.1
51,2
$4.41
55.41
70.08
67.07
REMEDY
60.34
75.34
116.9
88.2
85.20
30.9
$8.69
64.9
52.2
54.88
58.31
71.76
09.25
Table 3: Performance comparison of individual recipes finetuned recipes demonstrate transferability on a few tasks. REMEDY leverages this transferability to learn dynamic knowledge integration from multiple recipes.
and unseen tasks. Single-task
Method
Scen
Unsten
All tacks
TexVQA
SQAtimg)
COCO
Flickr30k
Avg
MM--VL
POPE
Avg
Avg
Zero-shot
Recipe VQA
61.59
64.85
113.2
53.5
74.54
30.2
44.12
60.4
49.34
83.55
53.52
64.03
62.23
Recipe-SQA
53.75
83.18
100.8
61.4
74.78
28.3
57.24
65.8
49.36
83:36
36.85
65.82
64.52
Recipe COCO
54.74
62.17
132.4
76.2
$1.28
27.4
50,82
61.3
52.94
84.62
55.42
68.40
65.91
Flicks-
54.58
64.40
107.9
91.1
79.50
26.5
52.24
63.2
53.57
82.54
55.61
67.56
65.44
REMEDY 60.34
75.34
116.9
88.2
85.20
30,9
58.69
64,9
52.2
84.58
58.31
71,76
69.25
et al., 2019) for visual question answering (VQA). Unseen tasks are datasets that were not in-
volved in the recipe construction process. These tasks are used to evaluate the zero-shot gen- eralizability of our method. Specifically, we employed the MM-Vet (Yu et al., 2024), MM- Bench (Zhang et al., 2023a), MMM-ench-Chhinsse (Zhang al.. 2023a), VizWiz (Gurari al., 2018), and POPE (Li et al., 2023) datasets to assess this capability. Additionally, we utilize TextCaps (Sidorov et al., 2020), which shares identical images with TextVQA but differs in task instructions, as a special case to analyze REMEDY's allocation behavior across related tasks. For more detailed information about these datasets, please refer to Section A. 1.1. in the Appendix.
Comparison Methods. Our compara- tive analysis encompasses three primary areas: First, we assess the Zero-Shot per- formance of pre-trained models across var- ious datasets, evaluating their inherent pabilities without any task-specific fine- tuning. Second, investigate the per- formance of models fine-tuned on spe- cific datasets, testing these task-specific recipes "Recipe-TextVV VQA, Recipe-SQA, Recipe-COCO, Recipe-Flickr300 across all datasets in our study. Third, we compare our method against existing model fusion techniques, including Simple Averaging, Task Arithmetic (Asai et al., 2022), TIES-merging (Yadav et al., 2024), Task-wise AdaMerging (Yang et al., 2023), and Layer-wise AdaMerging (Yang et al., 2023). These techniques aim to combine the strengths of multiple models or efficiently adapt models to new tasks. For more detailed information about these methods and implementation details, please refer to Sec- tion A.2 and A.4 in the Appendix.
5.5my
14
185
K7YOA Kibe Rcbco TEPEOY
mosiel
individull
Figure 4: Performance comparison on Related Tasks (TextCaps). TextCaps shares identical images with TextVQA but differs in task instructions.
4.2 MAIN RESULTS
Table 2 presents a comprehensive performance comparison between REMEDY and baseline meth- ods across various vision-language tasks, demonstrating REMEDY's superior performance in terms of average performance and H-score. Our experimental results reveal several significant findings:
REMEDY demonstrates superior multi-task learning capabilities across tasks. REM- EDY significantly outperforms existing model fusion algorithms across seen tasks. This perfor- mance consistently high across diverse seen tasks such image captioning, visual ques-
7

---

Published as a conference paper at ICLR 2025
tion answering, and visual reasoning. It demonstrates its efficacy in mitigating task interference through dynamic integration. Unlike conventional approaches that often suffer from negative trans- fer, REMEDY maintains high performance across diverse tasks, indicating successful synergistic knowledge sharing.
REMEDY exhibits exceptional zero-shot generalization to unseen tasks. Notably, current model fusion algorithms generally underperform compared to the simple average method on un- seen tasks. This suggests that existing fusion techniques may struggle with generalization, po- tentially overfitting to the seen tasks. In contrast, REMEDY is the only method that consistently outperforms both the simple average method and the zero-shot baseline across diverse unseen tasks. This improvement demonstrates that REMEDY effectively identifies and leverages relevant cross-modal knowledge for novel scenarios. Specifically, it surpasses the zero-shot performance on MM-Bench and VizWiz datasets without additional training.
REMEDY demonstrates effective knowledge transfer between related tasks. As shown in Table 4, on TextCaps which shares images with TextVQA but has different task objectives, REM- EDY outperforms all model fusion methods and individual recipes except Recipe-TextVQA This indicates that REMEDY can effectively leverage shared visual understanding while maintaining task-specific capabilities, validating our modality-aware allocation strategy for balancing knowl- edge transfer between related but distinct tasks.
4.3 TASK SIMILARITY ANALYSTS
While our main results demonstrate REMEDY's superior performance across both seen and unseen tasks, it is crucial to understand the underlying factors that contribute to knowledge transfer in multi- task learning scenarios.
As illustrated in Table 3, we further analyze the performance of single-task fine-tuned recipes across various tasks. While catastrophic forgetting is prevalent, with many recipes performing worse than zero-shot baselines out-of-domain tasks, we also observe instances of positive transfer and suc- cessful generalization. Several instances demonstrate positive transfer and generalization. Key ob- servations include: (1) Recipe-SQA outperforms zero-shot baselines MMBench; (2) Recipe- COCO and Recipe-Flickr30k show improved results on VizWiz; (3) Recipe TexiV-A generalizing effectively to TextCaps.
The effectiveness of this transfer can be attributed to shared characteristics between task do- mains. For instance, Tex and TextCaps share similar image distributions but differ in their instructions. allowing the recipe to leverage common visual understanding skills. For VizWiz, we observe the Flickr30k recipe receives higher weights possibly because both tasks require general visual understanding capabilities. Meanwhile, despite both being VQA tasks, VizWiz's 's unique char- acteristics (images taken by visually impaired users) may explain why TextVQA recipe receives lower weights. This observation is supported by Table 3, where the Flickr30k recipe shows better transferability to VizWiz (53.57% accuracy).
However, these positive transfer cases are limited to specific task pairs, highlighting the constraints of single-task fine-tuning approaches. In contrast, REMEDY's dynamic knowledge integration mechanism allows it to effectively leverage transfer opportunities across much wider range of tasks, both seen and unseen. This capability results in consistently superior performance and better generalization, underscoring the advantages of REMEDY'S approach over single-task recipes.
4.4 FURTHER ALLOCATION ANALYSIS
In this subsection, we provide a comprehensive analysis of REMEDY's performance from both qual- itative and quantitative perspectives. Figure illustrates four representative examples that highlight the distinctions between our method and simple averaging, along with visualizations of layer-wist wise allocation scores.
Qualitative Analysis. Comparing REMEDY with simple averaging reveals significant differences in output quality across various tasks. For caption tasks, recipe fused by simple averaging tends to produce shorter descriptions, often omitting crucial details. For instance, in the Flickr30k example, fails to mention "woman" and "picture". : In TextCaps, it overlooks the small text on the paper and the phrase "I' m trying to read". REMEDY, in contrast, generates more comprehensive and accurate
8

---

Published as a conference paper at ICLR 2025
Seen Tasks
Unseen Tasks
Flieette
Prouine de shen reptoo for this mage
Instuction:
Layer-Mtte
Captian TactCala
prion
Ground Truth Arswer:
aicmi likes pecuure baabyy weating
sper
a pins hat while man carnas he".
Shh!
Answer by iiplle Average
S Answer timple Avetages
Read!!
Arower REPcOY::
Arower REMIDD::
Arabic
protographad
witting onit and sticker saya (Tawrd
VQA Tagk:
Layer mire Score
Vizinle
Tanki
f intrrction:
Layerratie Score
FRetActive
proudeed information
Antwer by simple Avarage
Atswer by Simpla Averape
Pumplin
Quantian:
Armwnrt by REMEDY:
Lt Question: What sduut thrs?
what book lexet you?
de you study
what frowet (ood liem?
boore Recupe
25k Rodie
SOX Inturnl Husip
Figure 5: Comparison and Score Visualization of READY on difffrrnt multi-modal question answering. Each row represents different linear layer, progressing from the projector to the first two sections of the LLM. The colored segments within each row indicate the proportion of scores allocated to different recipes, with segment length corresponding to score magnitude.
captions, capturing these nuanced details. For VQA Tasks, recipe fused by simple averaging exhibits limitations both visual and textual comprehension. In the TextVQA example, it misinterprets the question, providing an irrelevant answer. For VizWiz, it neglects to address the first part of the question entirely. REMEDY demonstrates superior understanding, providing more accurate and complete responses to these complex queries.
Quantitative Analysis. The layer-wise score visualizations in Figure offer insights into REM- EDY's allocation mechanism. For caption-related inputs (e.g.. Flickr30k and COCO), we observe larger score proportions for the Flickr30k and COCO recipes (yellow and blue segments). TextVQA inputs show higher allocations to SQA-finetuned and TextVQA KeetVQA-finetuun QA-finetuned recipes. Interestingly, for recipes. This aligns
the unseen VizWiz dataset, allocations are primarily to Flickr30k-fineee with the observation in Table 3, where Recipe-Flickr30k performed well on VizWiz, demonstrating REMEDY's s ability to leverage relevant knowledge from seen tasks for unseen scenarios.
ABLATION STUDY
Infulunce of Layer number. To construct
reuse and effective recipe, we conducted an in-depth investigation into the large language
model, a key component of the recipe, focusing particularly how the number of LoRA fine-
tuned transformer layers affects performance. We carried out a series of experiments by ad- justing the depth of LoRA fine-tuning, with re- The results show
2 Lay Lay Lay
32 Lay.
0lyy
2138 3
4
32 Lay
fnneuning Stoweey/
Tneeuuiin 9rabayy
sults presented in Figure that fine-tuning the first 2 layers typically re- sults in substantial performance gains, while ex- tending fine-tuning to deeper layers yields only marginal improvements. This suggests that the initial layers play crucial role in adapting the model to mudti-modal tasks, while fine-tume tuning additional layers may lead to overfitting. Based on these observation, we selected the strategy of fine-tuning the first 2 layers of the I.I.M
Performance TertvOA
Performance SOA
Straiegy
Ein-tuning Stratggy
Parformance on Performance Filckratk
Figure 6: Impact of fine-tuning different num- bers of LLM layers on task performance.

---

Published as a conference paper at ICLR 2025
Table 4: Performance comparison between REMEDY with iitentiion-ased and MLP-based modality-aware allocators on seen and unseen tasks. The attention- based Allocator consistently outperforms the MLP-based version across all tasks, demonstrating the effectiveness of the attention mechanism in fusing multi-modal information.
Method
Seca tasks
Unseen tacks MMBBEN
All tacks
TeaVQA
SQAGingi
COCO
Flickr30k
MM--Vl
MMBBCN
VieWir
POPE
Avg
Hscore
REMEDY REMEDY
58.27
85.1
25.4
62.1
83.19
55.93
68.82
66.39
75
85.20
84.88
69.25
as the effective trade-off between performance
gain and computational efficiency.
Effectiveness of Modality-awand Allocator. We investigated the token understanding component allocator. Table presents a performance comparison between using atten- tion layers and MLP layers the token understanding component. In the MLP version, we directly averaged the input tokens before feeding them into the MLP, instead of using an attention mecha- nism. The results show that the attention-basee based REMEDY consistently outperforms the MLP-based version across seen and unseen tasks. This consistent improvement demonstrates the crucial role of the attention mechanism in capturing complex relationships between different modalities.
of the modality-awel
6 RELATED WORKS
Model merging in visual recognition models. Recent years have witnessed extensive research and application of model merging in visual recognition models. Task Arithmetic (Ilharco et al., 2022) introduced the concept of "task vectors," enabling multi-task learning through arithmetic op- erations on these vectors. TIES-Merging (Yadav et al., 2023) proposed pruning individual models and eliminating parameter sign conflicts to reduce interference before merging sparse models using Task Arithmetic. AdaMerging (Yang et al., 2024b) presented an adaptive method to learn merging coefficients by minimizing entropy on unlabeled test data as a proxy loss. Representation Surgery (Yang et al., 2024a) observed representation bias in merged models and introduced "representa- tion surgery" module to calibrate this bias. Consensus (Jain et al., 2023) explored using the Fisher Information Matrix to guide model merging. Additionally, Weight-Ensembllng MoE (WEMoE) (Tang et al., 2024) combined model merging with Mixture of Experts (MoE), proposing a dynamic merging Transformer architecture.
MoE in Large Vision-Language Models. LLaVA-MoLE (Chen et al., 2024) proposed a sparse mixture of LoRA experts, creating set of LoRA experts for MLP layers in Transformer layers. This method effectively mitigates conflicts between different instruction datasets while maintain- ing computational costs. MoCLE (Gou et al., 2024) introduced cluster-conditional LoRA expert mixture approach. It clusters instructions from all training data, constructs task-specific experts for each cluster, and incorporates universal expert trained on all data balance specialization and generalization. TaskGalaxy (Chen et al., 2025) further pushes the boundaries by scaling multi- modal instruction fine-tuning to handle thousands of vision task types Although these methods show promising results in L.L.Ms, they still have some limitations. These approaches primarily address data conflicts and necessitate concurrent access to all visible task data for joint training of L.M LoRA and routers. However, this requirement may not be feasible in certain practical scenarios. Moreover, the scalability these methods when applied to large-scale multimodal task sets remains unverified, raising questions about their applicability in more complex, real-world environments.
7 CONCLUSION
This paper introduces REMEDY (Recipe Merging Dynamics), novel approach to model merg- ing for LVLMs. REMEDY addresses the challenges posed by the complex, multi-modal nature of LVLMs, offering scalable and flexible solution for combining task-specific knowledge. The method consists of two main components: Recipe Construction, which defines reusable modules including the projector and shallow LLM layers, and Recipe Merging, which uses modality awwar allocator for dynamic weight allocation based on input relevance. This approach enables efficient cross-modal knowledge integration and enhances the model's 's ability handle diverse VI. tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (62441605, 62376243), the Zhejiang Provincial Key Research and Development Project (2023C0104), and the Academy Of Social Governance Zhejiang University.
REFERENCES
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6077-6086, 2018.
Akari Asai, Timo Schick, Patrick Lewis. Xilun Chen, Gautier Izacard, Sebastian Riedel, Han- nanch Hajishirzi. and Wen-tau Yih. Task-aware retrieval with instructions. arXiv preprint (XXv:22/1.09200 2022.
Ang Bian, Wei Li, Hangjee Yuan, Chengrong Yu, Zixiang Zhao, Mang Wang, Aojun Lu, and Tao Feng. Make continual learning stronger via c-flat. NeurIPS, 2024.
Jiankang Chen, Tianke Zhang, Changyi Liu, Haojie Ding, Yaya Shi, Feng Cheng, Huihui Xiao, Bin Wen, Fan Yang, Tingting Gao, et Taskgalaxy: Scaling multi-mode instruction fine-tuning with tens of thousands vision task types. arXiv preprint arXiv:2502.00 09925, 2025.
Shaoxiang Chen, Zequn Jie, and Lin Ma. Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning milms. arXiv preprint arXiv:2401. /6160, 2024.
Zhikang Chen, Min Zhang, Sen Cui, Haoxuan Li, Gang Niu, Mingming Gong, Changshui Zhang, and Kun Zhang. Neural collapse inspired feature alignment for out-of-distribution generalization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Wenliang Dai, Junnan Xu, Bingchen Ni, Zihuiwen Liu, Jiatao Gu, Xiang Chen, Yixiao Lin, Bolei Zhou, and Linjie Li. Instructblip: Towards general-purpost vision-language models with instruc- tion tuning, arXiv preprint 2305:::2330000000.. 2023.
Tao Feng, Mang Wang, and Hangjie Yuan. Overcoming catastrophic forgetting in incremental object detection via elastic response distillation. In CVPR, 2022.
Tao Feng, Wei Li, DiDi Zhu, Hangjie Yuan, Wendi Zheng, Dan Zhang, and Jie Tang. Zeroflow: Overcoming catastrophic forgetting is easier than you think. arXiv preprint 201102550..1004, 2025.
Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Deepali Aneja, Zeyu Jin, Ramani Du- raiswami, Dinesh Manocha, et al. A closer look at the limitations of instruction tuning. arXiv preprint arXiv:2402.05113 2024.
Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Xin Jiang, Zhenguo Li, Dit- Yan Yeung, James T Kwok, et al. Mixture of cluster-conditional lora experts for vision-language instruction tuning. arXiv preprint arXiv:2312.12239, 2024.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5608-3617, 2018.
Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expres- sions. In European Conference on Computer Vision pp. 108-124. Springer, 2016.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Ef- ficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13266 2024a.
11

---

Published as a conference paper at ICLR 2025
Wenke Huang, Mang Ye, and Bo Du. Learn from others and be yourself in heterogeneous federated learning. CVPR, 2022.
Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Rethinking federated learning with domain shift: A prototype view. In CVPR, 2023.
Wenke Huang, Mang Ye, Zekun Shi, Guancheng Wan, He Li, Bo Du, and Qiang Yang. A federated learning for generalization, robustness, fairness: survey and benchmark. TPAMI, 2024b.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint 22Xiv:2212.04089, 2022.
Michael S Jain, Cole Matena, and A Raffel. Merging models with fisher-weighted averaging. Ad- vances Neural Information Processing Systems, 35:17703-17716, 2023.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. In Advances in neural information processing systems, volume 25, 2012.
Wenbo Li, Jumnan Li, Guangyi Li, and Qi Huang. Pope: Probing open-ended visually grounded gen- eration from vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20623-20633, 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304. 08485, 2023.
Li Liu, Wanli Ouyang, Xiaogang Wang, Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen. Deep learning for generic object detection: A survey. International journal of computer vision, 128:261-318. 2020.
Shuqi Liu, Han Wu, Bowei He, Xiongwei Han, Mingxuan Yuan, and Linqin Song. Sens-merging: Sensitivity-guided parameter balancing for merging large language models. arXiv preprint urXiv:2502.12420 2025a.
Shuqi Liu, Han Wu, Bowei He, Zehua Liu, Xiongwei Han, Mingxuan Yuan, and Linqi Song. lbit-merging: Dynamic quantized merging for large language models. arXiv preprint ar02v:2502. 10743, 2025b.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, volume 35, pp. 9517-9532, 2022.
Zheqi Lv, Wenkai Wang, Jiawei Wang, Shengyu Zhang, and Fei Wu. Cascaded self-evaluation aug- mented training for efficient multimodal large language models. arXiv preprint arXiv:2501.0566 2025a.
Zheqi Lv. Tianyu Zhan, Wenjie Wang. Xinyu Lin, Shengyu Zhang, Wenqiao Zhang, Jiwei Li, Kun Kuang, and Fei Wu. Collaboration of large language models and small recommendation models for device-cloud recommendation. arXiv preprint arXiv:2501.0554 2025b.
Weimin Lyu, Xinyu Dong, Rachel Wong, Songzhu Zheng, Kayley Abell-Hart, Fusheng Wang, and Chao Chen. A multimodal transformer: Fusing clinical notes with structured ehr data for inter- pretable in-hospital mortality prediction. In AMIA Annual Symposium Proceedings, volume 2022, pp. 719. American Medical Informatics Association, 2022a.
Weimin Lyu, Songzhu Zheng. Tengfei Ma, and Chao Chen. A study of the attention abnormality in trojaned berts. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4727-4741, 2022b.
12

---

Published as a conference paper at ICLR 2025
Weimin Lyu, Songzhu Zheng, Lu Pang, Haibin Ling, and Chao Chen. Attention Attentionenhanccng backdoor attacks against bert-based models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10672-10690, 2023.
Weimin Lyu, Zexin Bi, Fusheng Wang, and Chao Chen. Badelm: Backdoor attack in clinical lan- guage models for electronic health records. arXiv preprini arXin:2407.05213, 2024a.
Weimin Lyu, Lu Pang, Tengfei Ma, Haibin Ling, and Chao Chen. Trojvlm: Backdoor attack against vision language models. European Conference on Computer Vision, 2024b.
Mohammed Mugeeth, Haokun Liu, Yufan Liu, and Colin Raffel. Learning to route among special- ized experts for zero-shot generalization. arXiv preprint arXiv:2402.05889, 2024.
Oleksiy Ostapenko, Zhan Su, Edoardo Maria Ponti, Laurent Charlin, Nicolas Le Roux, Matheus Pereira, Lucas Caccia, and Alessandro Sordoni. Towards modular Ilms by building and reusing library of loras. arXiv preprint arXiv:2405. 11157, 2024.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting gagioontto-hhaan correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference computer vision, pp. 2641 (641,2649, 2015.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In European Conference on Computer Vision, pp. 742-758. Springer, 2020.
Amanprect Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2317-8326, 2019.
Anke Tang, Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight-ensembling mixture of experts. arXiv preprint 24X1:22402.00433, 2024.
Yunze Tong, Junkun Yuan, Min Zhang, Didi Zhu, Keli Zhang, Fei Wu, and Kun Kuang. Quantita- tively measuring and contrastively exploring heterogeneity for domain generalization. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2023, 2023.
Hanqing Wang, Bowen Ping, Shuo Wang, Xu Han, Yun Chen, Zhiyuan Liu, and Maosong Sun. Lora-flow: Dynamic lora fusion for large language models in generative tasks. arXiv preprint arXiv:2402.11455, 2024.
Jinze Wang, Yixuan Zhang, Hang Sun, Jiahua Xu, Jingxiang Zhang, Ychao Lu, Qifan Yang, and Zhilin Yang. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.1220 12966, 2023.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Resolving interfer- ence when merging models. arXiv preprint wwXiv:2306.0/708 2023.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Re- solving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint 10.00:2310.02555. 2023.
Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. arXiv preprint arXiv:2402.01445 2024a.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arXiv:2402.054411 2024b.
13

---

Published as a conference paper at ICLR 2025
Jinluan Yang, Anke Tang, Didi Zhu, Zhengyu Chen, Li Shen, and Fei Wu. Mitigating the backdoor effect for multi-task model merging via safety-aware subspace. arXiv preprint arXiv:2410.13910, 2024c.
Jinluan Yang. Dingnan Jin, Anke Tang. Li Shen, Didi Zhu, Zhengyu Chen, Daixin Wang. Qing Cui, Zhiqiang Zhang, Jun Zhou, et al. Mix data or merge models? balancing the helpfulness, honesty, and harmlessness of large language model via model merging. arXiv preprint arXiv:2502. 06876, 2025.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. In Proceed- ings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2:
Short Papers), pp. 67-77, 2014.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European Conference on Computer Vision, pp. 69-85. Springer, 2016.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International conference on machine learning. PMLR, 2024.
Chaoyou Zhang, Yudong Zhang, Kanjian Gong, Yue Song, Xingqiang Zuo, Jiayi Zhang, Xiang Hu, Zhenlan Ren, Ziyue Chen, Mingyu Li, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint FrXiv22307.0628 2023a.
Min Zhang, Junkun Yuan, Yue He, Wenbin Li, Zhengyu Chen, and Kun Kuang. Map: Towards balanced generalization of iid and ood through model-agnostic adapters. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1921-11931, 2023b.
Min Zhang, Haoxuan Li, Fei Wu, and Kun Kuang. Metacoco: A new few-shot classification bench- mark with spurious correlation. arXiv preprint arXiv:2404.19644 2024.
Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, and Donglin Wang. Cobra: Extending mamba to multi-modal large language model for efficient inference. arXiv preprint arXiv:2403. 145.00 2024a.
Ziyu Zhao, Leilei Gan, Guoyin Wang, Yuwei Hu, Tao Shen, Hongxia Yang, Kun Kuang, and Fei Wu. Retrievallaugmented mixture of lora experts for uploadable machine learning. arXiv preprint arXiv:2406. 16989, 2024b.
Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu. Loraretriever: Input-aware lora retrieval and composition for mixed tasks in the wild, arXiv preprint arXiv:2402.0999) 2024c.
Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Kun Kuang, and Fei Wu. Merging loras like playing lego: Pushing the modularity of lora to extremes through rank-wise clustering. arXiv preprint 2409 /6/67, 2024d.
Ziyu Zhao, Yixiao Zhou, Didi Zhu, Tao Shen, Xuwu Wang. Jing Su, Kun Kuang, Zhongyu Wei, Fei Wu, and Yu Cheng. Each rank could be an expert: Single-ranked mixture of experts lora for multi-task learning. arXiv preprint arXiv:2501.151 15150, 103, 2025.
Xiongtao Zhou, Jie Hc, Yuhua Ke, Guangyao Zhu, Victor Gutiérrez- Bassuloo, and Jeff Z Pan. An empirical study parfieter-efffccient fine-tuning for multimodal large language models. arXiv preprint arXiv:2406.055 05 130, 2024.
Deyao Zhu, Jun Chen, Xiaogian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En hancing language understanding with advanced large language models. arXiv preprint arXiv:2304 10592, 2023a.
Didi Zhu, Yinchuan Li, Yunfeng Shao, Jianye Hao, Fei Wu, Kun Kuang, Jun Xiao, and Chao Wu. Generalized universal domain adaptation with generative flow networks. In ACM International Conference Multimedia (MM) 2023, 2023b.
14

---

Published as a conference paper at ICLR 2025
Didi Zhu, Yinchuan Li, Junkun Yuan, Zexi Li, Kun Kuang, and Chao Wu. Universal domain adapta- tion via compressive attention matching. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 6974-6985, 2023c.
Didi Zhu, Zexi Li, Min Zhang, Junkun Yuan, Jiashuo Liu, Kun Kuang, and Chao Wu. Neural collapse anchored prompt tuning for generalizable vision-language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4631 16440, 2024a.
Didi Zhu, Zhongyi Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Kun Kuang, and Chao Wu. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. ICML 2024, 2024b.
Lanyun Zhu, Tianrun Chen, Jianxiong Yin, Simon See, and Jun Liu. Continual semantic segmenta- tion with automatic memory sample selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3082-3092, 2023d.
Lanyun Zhu, Tianrun Chen, Jianxiong Yin, Simon See, and Jun Liu. Addressing background context bias in few-shot segmentation through iterative modulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3370-3379, 2024c.
15

---

Published as a conference paper at ICLR 2025
A DETAILED EXPERIMENT DESCRIPTIONS
A.1 DETAILED DATASET DESCRIPTIONS
To evaluate the effectiveness of proposed method, we conducted experiments on a diverse set of vision-language tasks, encompassing both seen and unseen scenarios. Table provides an overview of the datasets used in our study.
Seen tasks. For recipe construction and initial evaluation, we utilized four datasets representing different aspects of vision-language understanding: (1) Flickr30K and COCO: Both are image cap- tioning datasets featuring natural scenes. We use CIDEr as the evaluation metric, which captures the consensus between generated and reference captions. (2) SQA(img): A knowledge grounded visual question answering dataset focused on science topics. presents multiple-choice questions and is evaluated using accuracy. It's worth noting that the original SQA dataset contains both text-only and image-related questions. However, as we focus on multimodal tasks, we only consider the subset with images, consistent with the approach in Zhou et al. (2024). (3) TextVQA: This dataset tests reading comprehension in visual contexts, requiring models to recognize and understand text within images. Performance is measured by accuracy on phrase-level answers.
knowledge--rounded
Unseen Tasks. To assess zero-shot generalization capabilities, we employed five additional datasets not used in recipe construction: (1) MM- Vet: Evaluating the integrated performance of large multi- modal models, covering various core visual-language abilities, including recognition, OCR (Optical Character Recognition). knowledge, language generation, spatial perception, and mathematics. (2) MM-Bench: A comprehensive benchmark for multimodal understanding and reasoning, featuring a variety of question types with multiple-choice answers. (3) MM-Bench-Chinese: A Chinese- language subset of the MMBench dataset, designed evaluate the performance of multimodal mod- els on Chinese cultural and linguistic understanding. (4) VizWiz: This visual question answering dataset uniquely challenging as it contains images taken by visually impaired individuals, often with quality issues. It requires phrase-level responses. (5) POPE: An object-basee based VQA dataset designed to probe for object hallucination in vision-language models. uses yes/no questions to evaluate model performance.
Additionally, we included TextCaps as a related task. While sharing the same image base as Tex,VQA, this dataset presents a distinct linguistic challenge by requiring the model to generate de- scriptive captions that incorporate OCR information, rather than answering specific questions. This unique setup allows us to evaluate REMEDY's ability to transfer and combine knowledge across different linguistic objectives.
A.2 DETAILED COMPARED METHODS
In our study, we compared our proposed approach with several model merging methods. These meth- ods represent different strategies for combining knowledge from multiple tasks or models. Below, provide detailed description of each compared method, highlighting their key characteristics and potential advantages:
Task Arithmetic: This method, introduced by Ilharco et al. (2022), involves computing task vectors by subtracting the parameters of a base model from those of task-specific fine-tuned model. These task vectors can then be added to or subtracted from other models to transfer task-specific knowledge. Task Arithmetic offers simple yet effective way to combine knowledge from different tasks.
TiES-Merging:
Proposed by Yadav al. (2023), TIES-Merging addresses the challenge of negative transfer in multi-task learning. It employs task-specific importance estimation strategy to identify and preserve crucial parameters for each task. By pruning less important parameters and aligning the remaining ones, TIES- TIES-Merging aims to create merged model that maintains high performance across all seen tasks.
Task-wist AdaMerging: This approach, developed by Yang et al. (2024b), introduces adaptive merging coefficients at the task level. It dynamically adjusts the contribution of each task-specific model to the final merged model based on the input. This method allows for more flexible knowledge integration, potentially improving performance on diverse in- puts by leveraging the most relevant task-specific knowledge for each instance.
16

---

Published as a conference paper at ICLR 2025
Table Detailed Dataset Deseriptions. "Recipe Construction" indicates whether the dataset was used in the model's recipe finetuning process. This table presents the key characteristics of datasets used in our experiments, including both seen and unseen tasks.
Dutaset
Recipe Construction
Task
Metric
Description
Answer type
Hickr30K
Yes
Image Caption
CIDEr
Image dataset with captions for Caption ural secoes.
COCO
Yes
Image Cuption
CIDEr (1)
Image dataset with captions for nat- Caption
SQArimg)
Yes
Knowledge Grounded VQA
Accuracy (t)
Science-focused focused mulliple-choice
Option
VQA
TextVQA
Reading Comprehension VQA
Accuracy (f)
VQA ages.
recognition
Vet
No
General VQA
Acturacy (f)
Evalunting the integrated perfor- Sentence
mance large muhtimodal
MM Bench
No
General VQA
Accuracy (1)
Comprehensive benchmark for mul-
Option
timodal understanding ing
MM Bench Chinese
No
Gemeral VQA
Accuracy (?)
Comprehensive benchmark for Chi- cultural and linguistic under-
Option
standing.
VizWiz
No
General VQA
Accuracy (?)
VQA dataset focused images taken by visually impaired individ-
Phrase
uals.
POPE
No
Object-hused VQA
Accuracy (f)
Probing dataset for object hallucina- Yes/No
Isson-language models,
TexiCops
No
Image Caption
CIDEr (1)
Image dataset with captions for nat-
Caption
ural scenes.
Table 6: Recipe Finetuning Configurations
Datasct
Data Size
Leaminng Rate
Training Epochs
Training Instruction
Flickr30K
145,000
2c-5
Provide one-sentence caption the provided image.
COCO
113,287
2e:5
Provide one-sentence caption for the provided image.
SQA
16,967
26-4
Answer with option's 's index (start from from the given choices directly.
TextVQA
34,602
2c-5
5
Answer the question using single word phrase.
Layer-wise AdaMerging: An extension of the AdaMerging concept, this method applies adaptive merging at the layer level rather than the task level. It allows for finer-grained control over the merging process, capturing layer-specific expertise from different models.
A.3 EVALUATION METRICS
To comprehensively evaluate the performance our method, we employ two primary metrics: H- score and Avg. H-score represents the harmonic of the performance seen and unseen tasks, providing balanced measure of the model's capability across both task categories. Avg represents the average performance across all tasks.
A.4 DETAILED IMPLEMENTATION DETAILS
For the recipe construction phase, we focus two key components of the LLaVA architecture. First, we target the initial two layers of the L.M component. For fine-tuning the L.M layers, we employ the LoRA technique. The second component utilize standard fine-tuning techniques, allowing for more comprehensive update the projection parameters. For the recipe merging phase, train modality allocator by standard fine- tuning process.
fine-tune is the projector. For this part,
Implementation Details of Recipe Construction. Table 6 shows the configurations used in recipe construction. For image captioning tasks (Flickr30K and COCO), we employed a conservative approach with a low learning rate (2e-5) and single training epoch, given the large dataset sizes and
17

---

Published as a conference paper at ICLR 2025
Table 7: Computational analysis of different fusion methods. While REMEDY requires moder- ately more computational resources than baseline methods, it achieves significantly better zero-shot performance, demonstrating a favorable trade-off between computational cost and model capability.
Method
Training (h)
Inference (mskquery)
Training GPU Memory (GB)
Inference GPU Memory (GB)
GFL.PPs
Simple Average
38.5
380
LW AdaMerging
3.84
52.8 60.6
38.5
425
REMEDY
5.25
123
40.3
483
the relatively straightforward nature of the task. In contrast, for more complex tasks like SQA(img) and TextVQA, we adjusted our strategy to account for the increased task difficulty and smaller dataset sizes. This involved using higher learning rate (2e-4) for SQA(img) and increasing the number of training epochs to for both SQA(img) and TexIVQA. These adjustments allowed for more thorough learning these challenging tasks without risking overfitting. The rank of LoRA is set to 128.
Implementation Details of Recipe Merging. The modality- aware allocator is implemented neural network module that dynamically adjusts the contribution of different modalities. consists of an input projection layer implemented by self-attention mechanism, and multi-layer perceptron with normalization and dropout. The learning rate of the allocator is set to 2c-4. For training the modality-award allocator, we randomly sampled 1000 data points from each of the four seen datasets. This sample size represents approximately 1 % to 5% of the original training sets, providing a diverse yet manageable subset for efficient training of the allocator. The training objective to minimize task-specific losses of the merged model on the sampled data. Specifically, we aim to find optimal merging weights generated by the allocator such that when applied fuse the original recipes, the resulting merged model achieves minimal loss each task.
A.S COMPUTATIONAL ANALYSIS
Table 7 presents detailed analysis of computational costs and performance metries for different model fusion approaches. We evaluate these methods across multiple dimensions including training time, inference latency, memory requirements, and computational complexity (GFLOPs).
These results REMEDY introduces moderate additional costs while achieving substantial perfor- mance gains. Specifically, the training phase requires about 1.5 more hours compared to LW AdaMerging, mainly due to the modality-aware allocator training. During inference, REMEDY adds marginal latency per query for weight computation. Given the significant performance improve- ment (+3.1 1.% over simple averaging on unseen tasks), we believe these computational trade-offs are well justified.
B MORE RELATED WORKS
Model merging in LLMs. LoraHub (Huang et al., 2024a) first trains several LoRA weights on upstream tasks, then uses a gradient-free method to search for optimal combination coefficients for downstream tasks. LoRA-Flow (Wang et al., 2024) introduces fusion gates at each Transformer layer, generating weights to average the outputs of pre-trained LoRAs Arrow (Ostapenko al., 2024) explores building and reusing a library of LoRA experts for zero-shot task generalization, adding linear routers at each layer for dynamic selection using expert prototypes. PHATGOOSE (Muqeeth et al., 2024) proposes slightly modified training procedure, adding sigmoid gates for each training task to learn token importance. LoraRetriever and its variant (Zhao et al., 2024c:b) train a sentence embedding model to map input queries to an embedding space for expert model selection and routing. Recent works have explored rank-wise operations for LoRA fusion by treat- ing each rank a separate module (Zhao et al., 2024d; 2025). Security concerns have also been addressed with approaches focusing on mitigating backdoor effects and balancing safety in merged models ( (Yang et al., 2024c; 2025; Lyu et al., 2023; 2022b; 2024b;a). Meanwhile. efficiency improve- parameter balancing and dynamic quantized
ments have been achieved through sensitivity-guiide merging strategies (Liu al., 2025a;b). Ly et al. (2025b) demonstrates the effectiveness of collabo-
18

---

Published as a conference paper at ICLR 2025
rative approaches between large language models and specialized small models in recommendation scenarios.
While these methods show impressive performance in LLMs, they have some limitations. Firstly, they are primarily designed for pure text LLMs and cannot be directly applied to Multimodal Large Language Models (LVLMs). They don't consider the interaction between image and text tokens, which is crucial in LVLMs. Secondly, many methods adopt token-wise fusion strategies and deploy routers every layer, design that might lead computational inefficiency and increased model complexity in LVLMs.
C LIMITATIONS AND FUTURE WORK
While REMEDY shows improved zero-shot generalization, there is still room for further enhance- ment in this area. Additionally, the current recipe construction process relies heavily on experimental findings, which may limit its adaptability to new scenarios. Future work could focus on developing more dynamic recognition mechanisms for recipe construction, potentially incorporating adaptive techniques that can automatically identify effective module compositions based on task characteris- ties. This could lead to even more flexible and generalizable model merging strategies for LVLMS, further advancing their capabilities in handling diverse and novel multi-modal tasks.
D ETHICAL STATEMENT
We declare that our research does not present any potential ethical issues. The study does not involve human subjects, sensitive data, or methodologies that could result harmful outcomes or biases. All data this work uses is publicly available, and privacy security concerns implicated.
E REPRODUCIBILITY STATEMENT
We have made significant efforts ensure the reproducibility of work. A code example is provided as supplementary material, demonstrating the core components of our approach. Upon acceptance, will release all of the data and the complete training and testing code facilitate the full reproducibility of our results.
19


Reviews:
[{"review_id": "6Ed8tRU5yX", "reviewer": "Reviewer_6SYF", "summary": "The paper introduces REMEDY (Recipe Merging Dynamics), a model merging framework tailored for large vision-language models (LVLMs). REMEDY leverages modular \"recipes\" and a dynamic, modality-aware allocator for effective cross-modal knowledge integration, enhancing multi-task learning and zero-shot generalization across diverse vision-language tasks.", "questions": "As shown in weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper presents a unique \"recipe\" construction framework, creating modular, reusable components that facilitate task-specific adaptation and multi-task learning within large vision-language models.\n\n2.  Extensive experiments on multiple datasets demonstrate REMEDY’s efficacy, highlighting superior performance over baseline methods on multi-task and zero-shot generalization. \n\n3. The study provides a thorough analysis of task similarity and allocation strategies, offering insights into how specific recipes contribute to various tasks.", "weaknesses": "1.  The process of constructing recipes heavily relies on experimental insights, such as which layers and modules to fine-tune. This empirical approach may limit REMEDY's scalability and adaptability to new tasks or models without extensive trial-and-error adjustments.\n\n2.  Although REMEDY introduces modularity, the recipe construction phase lacks dynamic or automated selection mechanisms to identify optimal modules or layers. \n\n3. The paper does not extensively discuss the computational demands of the proposed approach, especially in the context of training and inference with the modality-aware allocator.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fqiQkts0uQ", "reviewer": "Reviewer_Fe8D", "summary": "This paper proposes REMEDY, a new technique for merging large vision-language models (LVLMs). It addresses the challenges of scale and multi-modal input by creating reusable fine-tuned modules called \"recipes\" and dynamically combining them using a \"modality-aware allocator.\" This allocator analyses the input image and text to generate weights for each recipe, enabling adaptive fusion and improved performance on both seen and unseen vision-language tasks.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The analysis of finetuning strategies is interesting. The paper shows that fine-tuning the projector and shallow layers of the LLM leads to better improvements.\n2. The motivation that uses the most relevant experts to better answer user questions is reasonable.", "weaknesses": "1. There is a typo in line 054: the vision encoder should be 0.3B.\n2. There are some redundant definitions in this paper and the \"recipes\" are actually MoE-based LoRA layers. The use of global and module-wise routing is not novel.\n3. The experiments include only 4 seen tasks and evaluation benchmarks. And I suspect that the proposed method can generalize well. It is expected that REMEDY surpasses the zero-shot setting on seen tasks since it is trained on these datasets. My main concern is the improvement on unseen tasks, which appears weak overall.\n4. TextCaps and TextVQA use the same images, so TextCaps is not an unseen task if TextVQA is already seen.\n5. The baseline is somewhat weak. There are stronger open-source baselines available, such as LLaVA-NeXT and LLaVA-OneVision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LDfNvXPXQG", "reviewer": "Reviewer_QEaG", "summary": "This paper introduces REMEDY, a model merging technique for LVLMs that addresses the challenges of combining knowledge across multi-modal tasks. REMEDY explores components of LVLMs and propose recipes for learn specific tasks. It further propose a dynamic allocator that learns weights for different recipes. Experiments demonstrate the effectiveness of the proposed method.", "questions": "How about the performance of REMEDY compared to LLaVA fine-tuned on the seen tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The method propose a new approach for model merging for LVLMs, which is effective for enhancing the multi-task learning capabilities of LVLMs.\n2. The paper is well-writing and the experiments are comprehensive.", "weaknesses": "1. The work trains the model in few-shot way, e.g., on seen tasks and expected to generalize to unseen tasks. But current MLLMs are expected to have strong zero-shot capabilities. And from the comparison results of Table 2, REMEDY seems not have explicit performance improvement compared to the zero-shot way or simple average way. Considering the complexity it adds on the training and inference stage, it may limit the practical use case of it.\n2. Does the model needs to compute the weights for different recipes and then update the model weights for each inference time. If so, how about the extra inference time introduced and the comparison with the baseline?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "82w29E4Oiu", "reviewer": "Reviewer_dj7R", "summary": "This paper introduces a straightforward approach to addressing the multi-task learning challenge in Large Vision-Language Models (LVLMs) using model merging techniques. Given the large parameter scale of LVLMs, the authors first select layers to merge, referred to as 'recipes,' based on preliminary experiments. These layers include the projection layer and the shallow layers of the LVLM. An attention-based module is then trained to dynamically assign merging weights to the 'recipes' based on the input multi-modal token, allowing for adaptive weighting to handle the complexity of VL tasks. The proposed method demonstrates improvements on both learned tasks and zero-shot generalization across multiple datasets.", "questions": "I have some additional concerns:\n\n- **Evaluation Metric**: What does the Hscore metric indicate? Is the average accuracy calculated as a direct average of the accuracies on the different datasets or is it weighted by the number of samples in the dataset?\n\n- **Score Visualization Analysis**: In Figure 4, the weight distribution across different layers shows considerable variation, with some layers dominated by distinct 'recipes.' This phenomenon is intriguing, and I would appreciate further explanation from the authors. For VizWiz, an unseen dataset with the same VQA task, why does the \"Recipe\" of TextVQA have a smaller layer-wise score?\n\nFinally, the proposed Modality-aware Allocator is designed for multimodal tasks, which is one of the advantages over previous research on visual models. Therefore, I am curious whether using only visual or textual input as clues for allocation will result in a performance degradation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- **Clarity**: The paper is well-structured and easy to follow.\n- **Motivation**: As the size of the visual language model continues to grow, it becomes costly to maintain a separate model for each specific task. It makes sense to combine task-specific models into a unified model through model merging that can simultaneously solve different downstream tasks.\n- **Novelty**: This paper designs a merging strategy that takes into account the characteristics of multimodal tasks and the difference in fine-tuning paradigms between LVLM and traditional models. Meanwhile, this paper also analyzes the influence of each component in LVLM on fine-tuning performance, which is suggestive for the follow-up works.\n- **Results**: The experimental results across multiple datasets effectively highlight the potential and robustness of the method.", "weaknesses": "- **Detail in Weight Generation**: The paper does not explain how the proposed attention-based module is trained to generate the merging weights. Neither the main text nor the supplementary materials provide insights into the optimization objective used. While the authors may consider this aspect elementary, I believe a detailed explanation is necessary.\n- **Analysis of Computational Overhead**: The work in this paper is carefully designed when selecting an extended module, which may reduce the cost of calculating overhead and model parameters. However, the experiment lacks analysis of the training and inference overheads.\n- **Details Error**: The introduction states that the new challenges facing the LVLM merger will be analyzed from three aspects, but only two aspects are introduced below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)