Multi-modality Expansion and Retention for LLMs through Parameter Merging and Decoupling

Authors: Junlin Lee, Guodong DU, Jing Li, Sim Kuan Goh, Ho-Kin Tang, Yequan Wang, Fangming Liu, Saleh Alharbi, Daojing He, Min Zhang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Multimodal Large Language Models, Model Merging, Parameter Decoupling, Knowledge Localization

Abstract:
Extensive fine-tuning of the synthesis between multimodal encoders and Large Language Models (LLMs) on modality-specific data can expand the modalities that LLM can handle, leading to the formation of Multimodal Large Language Models (MLLMs).
However, this paradigm to expanding modalities heavily relies on initiating fine-tuning from scratch with new multimodal data, which is both resource-intensive and inflexible. 
In this paper, we propose $\textit{MMER (Multi-modality Expansion and Retention)}$, a novel $\textit{training-free}$ approach that reuses and composes existing MLLMs to facilitate effective multimodal expansion while retaining the original performance of each MLLM. 
In particular, MMER maintains the multimodal encoders of the MLLMs while merging their LLM parameters.
By comparing the original LLM parameters with the merged ones, MMER can create binary masks that enable an approximate separation of the LLM parameters for each modality.
This process allows the decoupled parameters to independently process modality-specific inputs, thereby reducing parameter conflicts and maintaining the fidelity of the original MLLMs.
Additionally, MMER integrates strategies to prevent catastrophic forgetting by employing a similar approach to separately decouple the parameters fine-tuned on new tasks from the original parameters. 
Experiments on three multimodal tasks and fourteen dual-modal tasks show significant improvements over recent baselines, demonstrating that MMER can effectively expand multimodal capabilities of LLMs while retaining 99.6\% of the original performance. 
Further experiments in both single-task and cross modalities multi-task scenarios reveal that MMER significantly mitigates catastrophic forgetting.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730626475184

thread_type:
official_review

<thread_object>
creator: reviewer_egsn

<content>
summary:
This paper introduces a novel training-free approach called MMER (Multi-modality Expansion and Retention). The approach aims to expand the capabilities of Large Language Models (LLMs) to handle multimodal data while retaining their original performance on specific modalities.
MMER merges LLM parameters from multiple MLLMs while constructing binary masks to decouple modality-specific parameters. This allows for independent processing of modality-specific inputs and reduces interference.

strengths:
The paper introduces a new parameter decoupling strategy that effectively separates modality-specific parameters, reducing interference and enhancing performance.

The authors have clearly articulated the problem, the proposed solution, and the advantages of MMER over existing methods. 

The paper is clear and well-organized, making it easy for readers to follow the authors' line of reasoning.

weaknesses:
In general, cross-modal tasks exhibit diversity, and the paper reports performance on only one cross-modal task for each modality combination, such as MUSIC-AVQA and MCUB. This makes the validation of the proposed method's effectiveness insufficiently comprehensive.

Lack of the performance of the unimodal model on the tri-modality task for a fairer comparison.

The author decouples modalities using addition and subtraction; however, models are typically nonlinear, which confuses me. The paper also lacks sufficient proof for this approach. Additionally, the justification for using Manhattan distance when calculating the mask is not strong enough.

questions:
Can the author clarify how many parameters are decoupled for each modality?

I hope the author can provide a more detailed explanation and proof of the proposed method. I will consider changing my perspective if the author's response is valuable.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730673554256

thread_type:
official_review

<thread_object>
creator: reviewer_q9ea

<content>
summary:
- The paper proposes MMER (Multi-modality Expansion and Retention), a novel training-free approach that reuses and composes existing MLLMs to facilitate effective multimodal expansion while retaining the original performance of each MLL
- Extensive and rigorous experiments on various multimodal benchmarks have been conducted to validate the effectiveness and robustness of MMER.

strengths:
- The paper proposes a training-free MMER approach that enables seamless multimodal expansion for LLMs through multimodal parameter merging and decoupling.
- The paper also leverages MMER for mitigating catastrophic forgetting in MLLMs, demonstrating its potential for continual learning applications.
- Extensive experiments have been performed to demonstrate the effectiveness of MMER on multi-modality expansion,  multi-modality retention and mitigating catastrophic forgetting.

weaknesses:
- In terms of mitigating catastrophic forgetting, although MMER demonstrates good performance, considering the amount of storage required for MMER, it is unclear how this improvement compares to simply tuning task-specific adapters for new tasks.
- The paper mentioned that the hyper-parameters, such as $\alpha$ and $\lambda$, are selected based on the validation set. However, the construction of this validation set is not well-detailed, making it hard to tell how well these hyper-parameters can generalize to different tasks.
- The paper does not provide sufficient theoretical grounding for the proposed method, lacking analysis or reference on key aspects such as the relationship between model parameter differences and corresponding performance differences, the measurement of modality interference, or positive transfer between difference modality-specific LLMs.

questions:
- In Equation 2, what is the rationale for using the Manhattan distance to minimize the distance between the reconstructed model parameters and the original LLM parameters?
- What will be the suggested way to quantify the difference between the original MLLM and its approximate version reconstructed by MMER? Furthermore, is there a relationship between this measured difference and the performance discrepancy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685498709

thread_type:
official_review

<thread_object>
creator: reviewer_psfy

<content>
summary:
The paper proposes a training-free approach, MMER, to enhance the multimodal capabilities of LLMs without extensive fine-tuning. To process different modalities, MMER maintains the vision encoder and introduces binary masks on the LLM parameters for each modality, facilitating modality-specific input processing and reducing parameter conflicts. Additionally, MMER incorporates a similar method to mitigate catastrophic forgetting by decoupling parameters fine-tuned on new tasks from the original parameters. The experimental results on multimodal and dual-modal tasks indicate that MMER achieves notable improvements over recent baselines, demonstrating its potential effectiveness.

strengths:
- MMER’s training-free approach makes it practical and resource-efficient, avoiding the computational costs of extensive fine-tuning.
- The method effectively retains the original performance of merged models, mitigating catastrophic forgetting and preserving their capabilities.
- MMER demonstrates versatility, applicable to various modalities, and maintains performance across diverse multimodal tasks.
- The paper includes extensive experiments that show consistent performance improvements over baseline methods, validating MMER’s robustness.

weaknesses:
- The paper could better explain the advantages of MMER over existing modular approaches and provide a clearer justification for adopting this monolithic method.
- While the paper demonstrates effectiveness, it lacks comparisons with certain mainstream MLLMs and does not evaluate larger-scale models. Including these aspects would strengthen the argument for MMER’s superiority and provide insights into its performance at scale.

questions:
Refer to the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720009250

thread_type:
official_review

<thread_object>
creator: reviewer_vhbg

<content>
summary:
This paper introduces a training-free parameter-merge framework, called MMER, for MLLMs. This method compresses key parameter differences between each MLLM and original LLM into a merged task vector and a multimodal mask. When addressing tasks involving multimodal expansion and retention, MMER restores the parameters specific to the required modality using the task vector and mask, making the merged model approximate the corresponding MLLM. Extensive experiments demonstrate MMER’s effectiveness across various benchmarks and its capability to mitigate catastrophic forgetting.

strengths:
1. The paper offers several clear illustrations that is greatly helpful in understanding the mechanisms of parameter merging, expansion, and retention.
2. By decoupling the merged task vector and mask, the model can utilize parameters approximating to specific modality to process corresponding modality inputs, which sounds make sense.
3. This method merges the newly fine-tuned MLLM as an additional task vector, while maintaining performance on the original task.

weaknesses:
For audio modality, the merged model only selects 2.2% parameters from merged task vector (Figure 4a), which indicates the majority of key parameters in the audio MLLM deviate from the direction of the original LLM according to the merging mechanism. However, the merged model (98% parameter from original LLM + 2% parameter from merged task vector activated by audio mask) achieves 1.5x improvement compared to original audio MLLMs, which is confused. 

The author attributes this result to the redundant parameters in the audio model, consistent with the observations in [1]. However, even in [1], as more parameters are dropped, performance declines rather than an exaggerated 1.5x improvement. Considering that 2% of the parameters are not from the original audio MLLM, I wonder if that 2% of the parameters overlap with other modalities (such as video), thus providing more prior knowledge?

[1] Yu et al. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch.

questions:
According to the merging mechanism, merged task vector retains the parameters of target MLLM that are consistent with those of original LLM in direction. As more modalities are merged, the merged task vector will also accumulate key parameters from other modalities. Will the parameters of a single modality correlate to the final merged task vector? Will this correlation weaken with the introduction of more MLLMs, leading to the failure of the merging mechanism? In the merged task vector, what is the parameter overlap like across different modalities?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732112845580

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We appreciate your consideration in taking the time to review our comments. We received valuable feedback from four reviewers, who provided insightful and constructive comments. Below, we summarize the key strengths highlighted in their feedback:

1. **Clarity and Presentation**: 

   The paper is well-organized, clearly articulates the problem, and presents the proposed solution effectively. The use of clear illustrations and tables greatly aids in understanding the mechanisms of parameter merging, expansion, and retention, making the paper accessible and easy to follow.

2. **Innovation and Practicality**: 

   The training-free MMER approach offers a practical, resource-efficient solution for multi-modality expansion in LLMs, eliminating the need for costly extensive fine-tuning. The novel parameter decoupling strategy enables the separation of modality-specific parameters, reducing interference and enhancing performance.

3. **Effectiveness and Versatility**: 

   MMER shows versatility by supporting multi-modality expansion, retaining original performance, and mitigating catastrophic forgetting across diverse modalities.

4. **Comprehensive Experiments**: 

   Extensive experiments validate MMER’s effectiveness, showcasing consistent improvements over baseline methods. The results underline MMER’s robustness in multi-modality expansion, retention, and continual learning applications.

However, there are still some questions and concerns from the reviewers. We have summarized and addressed these in four key points: 

1. **Principles of Model Merging and Decoupling**:

   The principles and mechanisms of model merging and decoupling based on task vectors have been explored and proven in previous studies [1,2,3,4]. Building on this foundation, we provide a comprehensive summary and analysis. Specifically, we analyze the merging mechanism from the perspective of the number of merged models (**see Response to Reviewer VHBG, Question 1**) and explain why arithmetic operations can be applied to task vectors to achieve model merging and decoupling (**see Response to Reviewer egSN, Comment 3**).

2. **Rationale for Using Manhattan Distance**:

   The reasons for using the Manhattan distance for calculating the mask are threefold. First, to save computational resources and improve efficiency, we opted for a binary mask that directly eliminates irrelevant parameters in the merged task vector, retaining only the key information related to the modality. Second, the Manhattan distance naturally encourages parameter sparsification, which aligns well with the binary mask, whereas the Euclidean distance makes the mask tend towards smoothness. Finally, the Manhattan distance measures element-wise differences, whereas the Euclidean distance emphasizes the "total distance" in the entire parameter space. And experiments are conducted to validate this analysis (**see Response to Reviewer egSN, Comment 4**).

3. **Experimental Supplements:**

   We have additionally included comparison experiments with LoRA on mitigating catastrophic forgetting (**see Response to Reviewer Q9ea, Comment 1**), comparisons with mainstream MLLMs (**see Response to Reviewer PSfy, Comment 2**), the results of four original unimodal models on tri-modality tasks (**see Response to Reviewer egSN, Comment 2**), and results from new cross-modal tasks (**see Response to Reviewer egSN, Comment 1**). The results of these experiments further strengthen the validation of the effectiveness of our MMER method.

4. **Analysis of Methods and Experiments:**

   We have supplemented additional analyses on the method and experiments to validate the rationality of our approach while providing theoretical support. Specifically, we analyzed the overlap in decoupled parameters (**see Response to Reviewer VHBG, Comment 1 & Question1**) and examined the measurement of modality interference (**see Response to Reviewer Q9ea, Comment 3**). Additionally, we explored the positive transfer between different modality-specific LLMs (**see Response to Reviewer Q9ea, Comment 4**) and investigated the relationship between parameter differences and performance discrepancy (**see Response to Reviewer Q9ea, Question 1**).

Once again, we sincerely thank you for your involvement and thoughtful feedback! We have provided detailed responses to each question from each reviewer.

**Reference**

[1] Ortiz-Jimenez et al. Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. NeurIPS, 2023.

[2] Ilharco et al. Editing models with task arithmetic. ICLR, 2023.

[3] Yu et al. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch. ICML, 2024.

[4] Yadav et al. Ties-merging: Resolving interference when merging models. NeurIPS, 2023.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces MMER, a training-free approach for multimodal expansion and retention in large language models (LLMs), leveraging parameter merging and decoupling. Strengths include its resource-efficient design, robustness in mitigating catastrophic forgetting, and extensive experiments demonstrating competitive performance across multimodal tasks. However, reviewers noted limitations in theoretical grounding, insufficient diversity in task evaluations, and limited clarity on certain methodological choices. While the authors provided detailed rebuttals and addressed several concerns, the revisions only partially resolved key issues.