CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging

Authors: zongzhen yang, Binhang Qi, Hailong Sun, Wenrui Long, Ruobing Zhao, Xiang Gao
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Model Merging, Pruning Technique, Task Vectors, Language Models, Conflict-Aware Sparsity (CA), Balanced Sparsity (BS)

Abstract:
Model merging based on task vectors, i.e., the parameter differences between fine-tuned models and a shared base model, provides an efficient way to integrate multiple models without retraining. This approach can be used to combine task-specific models into a multitask model, improve generalization, or address model deficiencies. One of the significant challenges faced by model merging is the conflicts between task vectors. Existing works aim to mitigate these conflicts through sparsification; however, two issues observed in our experiments significantly limit their performance: $\textit{high parameter overlap}$ and $\textit{unbalanced weight distribution}$. To address these issues, we propose a simple yet effective framework called CABS (Conflict-Aware and Balanced Sparsification), consisting of $\textbf{C}$onflict-$\textbf{A}$ware Sparsification (CA) and $\textbf{B}$alanced $\textbf{S}$parsification (BS). CA can reduce parameter overlap by applying masks during sequential pruning, ensuring that each task vector retains distinct, non-overlapping parameters. BS leverages $n$:$m$ pruning to preserve critical weights while maintaining an even distribution across layers. Our comprehensive experiments demonstrate that CABS outperforms state-of-the-art methods across a range of diverse tasks and model sizes. Notably, in experiments with 7B-parameter language models, CABS surpasses the average performance of an "ideal" model, a virtual model that selects the highest score from individual fine-tuned models for each task (CABS: 76.50 vs. Ideal Model: 76.30 vs. Baseline: 76.02 vs. Fine-tuned Model: 75.86). Our results highlight the importance of addressing both high parameter overlap and unbalanced weight distribution to achieve robust and high-performance model merging.

Decision: reject
Reviewer scores: 5, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730312029966

thread_type:
official_review

<thread_object>
creator: reviewer_jfhp

<content>
summary:
This paper presents CABS (Conflict-Aware and Balanced Sparsification), a framework for improving model merging through task vectors. The key contribution is identifying and addressing two critical issues in existing sparsification approaches: high parameter overlap and unbalanced weight distribution. The method introduces Conflict-Aware (CA) Sparsification to reduce parameter overlap and Balanced Sparsification (BS) using n:m pruning for better weight distribution. Experiments on Mistral-7B and RoBERTa-Base demonstrate consistent improvements over SOTA.

strengths:
- Comprehensive empirical validation across both encoder-based (RoBERTa) and decoder-based (Mistral-7B) architectures

- Strong quantitative results, notably surpassing the "ideal" model baseline (76.50 vs 76.30)

- Thorough ablation studies demonstrating the individual and combined effects of CA and BS components

- Clear practical impact with minimal computational overhead and straightforward implementation

weaknesses:
- Limited exploration of sparsity levels below 0.25 and above 0.75

- No investigation of the method's applicability to cross-architecture model merging

- Lack of analysis on the impact of different task orderings in the sequential pruning process

- Experiments focused primarily on English language tasks, leaving questions about multilingual applicability

questions:
- How sensitive is the sequential pruning order in CA to task difficulty or model performance? Would a different ordering strategy (e.g., based on task complexity) potentially yield better results?

- Could CABS be extended to merge models across different architectures or pre-training objectives?

- In cases where sparsity levels exceed 1 and overlap cannot be completely eliminated, how do you determine which overlapping parameters to prioritize?

- What is the computational overhead of CABS compared to simpler merging approaches when scaling to larger models (>13B parameters)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730508995341

thread_type:
official_review

<thread_object>
creator: reviewer_so7i

<content>
summary:
This work introduces model merging based on task vectors, aiming to address high parameter overlap and unbalanced weight distribution through n:m pruning and inverse masks.

strengths:
* Adopting n:m pruning and inverse masks to sparsify task vectors is an interesting approach. 
* The method is simple and straightforward. 
* Analyzing the impact of different overlap rates on performance in Figure 4 is insightful.
* Overall, this paper is well-written and easy to follow.

weaknesses:
* Since sequential pruning is utilized, I believe the task order could impact performance. For instance, extracting the mask for task A and then using its inverse mask for task B might yield different results compared to extracting the mask for task B and using its inverse mask for task A. Could the authors analyze this aspect?

* Additionally, I am uncertain about the rationale behind the inverse mask strategy. When extracting the mask for task A and then using its inverse mask for task B, the inverse mask may remove important patterns for task B, as it is derived without consideration of task B. Could the authors provide a theoretical explanation and/or a more detailed analysis to justify this method?

* Magnitude and random pruning are commonly regarded as baselines in network compression literature. What might happen if alternative pruning criteria, such as the geometric median criterion (https://arxiv.org/abs/1811.00250), were used instead?

* The score improvements in Tables 2 and 3 appear quite marginal compared to the baselines, and the results are not particularly compelling. Additionally, I believe it would be helpful to include the score of the pretrained model before fine-tuning to better understand the influence of the base model (W_base​ in Algorithm 1).

* It would be helpful to present results from merging 3–5 models and compare them with the baselines to demonstrate the scalability and generalizability of the method.

* Could the authors provide guidelines on setting λ_A,B​ (the weighting constants of masked task vectors)? Are these values sensitive, and do they significantly impact performance?

* Would it be possible to extend the experiments to include a generation-based benchmark, such as IFEval (https://arxiv.org/abs/2311.07911)? I am curious about the model's generation capabilities and instruction-following ability after merging.

questions:
Please refer to the above weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622490273

thread_type:
official_review

<thread_object>
creator: reviewer_qave

<content>
summary:
This paper presents a new model merging approach called CABS. CABS has two steps: 1) Conflict-Aware sparsification (CA), which essentially applies masks sequentially during weight pruning for different tasks to avoid parameter conflicts. 2) Balanced sparsification (BS), which leverages existing n:m pruning technique to maintain weight balancing. Experiment results on both encoder-only and decoder-only models demonstrate the effectiveness of the proposed approach.

strengths:
Model merging is a promising field in terms of compositing LLM capabilities without retraining.

weaknesses:
The proposed method is very ad-hoc, and the improvement seems not that convincing. In particular:
- CA is simple, but questionable. However, what if you exchange the pruning order to task B first and then task A? The resulting model will not be equivalent, right? Also, what if you have > 2 models to merge? The later tasks will almost aways has less effective weights to be pruned from.
- CA can be combine with any pruning technique, I do not see why BS here is a particularly good pruning option. n:m pruning is a paper from 2021, and there are multiple state-of-the-art pruning techniques recently. While the paper empirically proves BS is better than basic magnitude based pruning, what about other advanced pruning techniques (e.g., https://arxiv.org/pdf/2305.11627). If CA + any pruning technique better than n:m pruning results in better performance, then we should not over-sell the importance of BS.
- The improvements are small compared to competing methods, what are the confidence intervals of these results?
- Sparsification is only a necessary for pruning-based model merging techniques. That said, I would like to see how the method compares against other categories of model merging techniques such as evolutionary model merge (https://arxiv.org/abs/2403.13187) and pack of llms (https://arxiv.org/abs/2404.11531).

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677021511

thread_type:
official_review

<thread_object>
creator: reviewer_goun

<content>
summary:
This paper presents CABS (Conflict-Aware and Balanced Sparsification), a framework designed to address two key challenges in model merging: parameter overlap and unbalanced weight distribution. The framework consists of two main components:

- CA: A sequential pruning approach that reduces parameter overlap between task vectors
- BS: An n:m pruning strategy that maintains balanced weight distribution

The authors evaluate CABS on both large-scale models (Mistral-7B) and smaller models (RoBERTa), demonstrating improvements over existing methods, particularly at high sparsity levels (0.75). The method achieves better performance than an "ideal" model baseline on some tasks, though improvements are modest.

strengths:
- Novel perspective on handling conflicts in model merging
- Simple but effective solution
- Practical value demonstrated in experiments- Clear problem formulation with empirical validation
- Comprehensive ablation studies
- Results generally support main claims
- Reasonable experimental design and evaluation metrics
- Well-organized structure with clear flow
- Good visualization of key concepts (Fig. 1)
- Detailed experimental results

weaknesses:
Theoretical Foundation:

- No mathematical derivation or proofs
- Lack of mechanism analysis
- Missing theoretical basis for n:m ratio selection

Experimental Design:

- Limited model coverage (only Mistral and RoBERTa)
- Missing statistical significance analysis for Small model like RoBERTa
- Insufficient variance analysis across seeds
- Incomplete baseline comparisons
- Small performance improvements


Technical Limitations:

- Only applicable to homogeneous models
- Multi-task vector case not addressed
- Insufficient analysis of computational overhead
- Unknown performance on larger models (>7B)
- Simple linear combination for merging

questions:
Theoretical:


- Can you provide some theoretical proof or analysis for why reducing parameter overlap improves performance?

Technical:

- How would the method extend to merging multiple (>3) task vectors?
- What is the impact of different pruning orders on final performance?
- How do you determine optimal n:m ratios for different model sizes?
- Why choose simple linear combination when there are more sophisticated approaches available, such as Fisher Information-based merging, gradient-based adaptive weighting, or attention-based parameter fusion? Have you considered incorporating these methods to potentially improve the merging performance? 


Experimental:

- Why weren't other major architectures (LLaMA, GPT) tested?
- Can you provide complete computational and memory overhead analysis?
- How does the method perform on cross-architecture merging?


Scalability:

- How would the method perform on larger models (30B+)?
- How does computational complexity scale with number of task vectors?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732257355266

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Rebuttal by Authors

comment:
We thank all reviewers for their time and constructive feedback, which have provided valuable insights to improve the paper. We are pleased that reviewers recognized the following strengths of our work:  

- **Novelty and Practical Impact**: A novel perspective on addressing conflicts in model merging (Goun), with practical value demonstrated through experiments (Goun, jFhp).  
- **Simplicity and Effectiveness**: A simple yet effective method that is computationally efficient and straightforward to implement (Goun, qavE, so7i, jFhp).  
- **Experimental Rigor**: Comprehensive empirical validation across encoder-based and decoder-based architectures, with strong quantitative results surpassing the "ideal" baseline (jFhp). Thorough ablation studies demonstrating the impact of key components, CA and BS (Goun, jFhp).  
- **Clarity and Presentation**: Well-organized structure with clear problem formulation (Goun), insightful analysis of overlap rates (so7i), and effective visualization of concepts (e.g., Fig. 1, Goun).  

We appreciate these positive comments and aim to address all concerns raised by reviewers in our detailed responses below.
# Common question

**Question1**：How would the method extend to merging multiple (>3) task vectors?（ Goun，qavE，so7i）

|    | COLA  | SST2 | RTE  | MRPC | avg  |
| ---- | ---- | ----- | ----- | ----- | -------- |
| fine-tuned on cola | 85.04 | 50.92 | 45.85 | 67.4  | 62.3025     |
| fine-tuned on sst2 | 68.74 | 94.04 | 46.93 | 68.38 | 69.5225          |
| fine-tuned on rte  | 31.83 | 51.15 | 79.42 | 25.98 | 47.095           |
| fine-tuned on mrpc | 49.47 | 51.15 | 47.29 | 91.18 | 59.7725          |
| task-arithmetic    | 76.32 | 90.83 | 69.68 | 81.37 | 79.55            |
| dare               | 76.99 | 90.14 | 70.76 | 81.13 | 79.755（+0.2）   |
| Ties-dare          | 77.66 | 90.94 | 69.31 | 81.62 | 79.8825（+0.33） |
| magnitude          | 82.07 | 87.04 | 65.34 | 79.66 | 78.5275（-1.02） |
| Ties-magnnitude    | 82.36 | 86.93 | 61.01 | 79.41 | 77.4275（-2.12） |
| **CABS(Ours)**        | 76.89 | 92.09 | 74.73 | 83.09 | 81.7（+2.15）    |

**Answer:** As shown in the table, we performed additional experiments to merge 4 models at 90% sparsity using the **same λ value for each task vector** with an search interval of 0.01. This decision was made to manage the computational complexity of λ selection. Specifically, if we were to search for optimal λ pairs for each task combination, the search space would grow exponentially with the number of models \(k\), making it infeasible for larger \(k\). 

Our method (CABS) achieves the best average score of **81.7**, with a **+2.09 improvement** compared to the baseline task-arithmetic method. It significantly outperforms other methods, such as Ties-dare, which achieves only **79.8825 (+0.33)** (the best result among other comparison methods). 

As the number of task models increases, the advantages of CABS become even more evident. It effectively resolves conflicts and maintains balanced sparsity, demonstrating its superior scalability and performance, even under a unified λ setting.

---


**Question2**：What is the impact of different pruning orders on final performance?（ Goun，qavE，so7i，jFhp）

**Answer**: As discussed in **Appendix A.12** of our paper, we explored the effect of different sparsification sequences in a 7b model experiment. Specifically, we analyzed the order in which source models (e.g., “wild” and “west”) are sparsified during the merging process. For example, under the same setting (sparsity ratio 64:256), **CABS-wild-first achieves 76.37 (+0.07)**, while **CABS-west-first achieves 76.39 (+0.09)**, showing that the order has a minimal effect on the overall average performance.

In our additional experiments merging four models (as shown in the table), we found that different pruning orders (e.g., CSRM, SCMR, RCMS, MRSC) do slightly influence performance on individual tasks, but the overall average remains robust, ranging narrowly from **81.6375 to 81.7**. This demonstrates that CABS effectively handles pruning sequence variations while maintaining high average performance.

Issues caused by pruning order could potentially be addressed by techniques such as using **adaptive lambda** to give greater importance to later models or adopting **variable sparsity ratios** to better balance model contributions. However, given the minor impact of pruning order on overall performance, we currently consider this less critical for our method's practical application.

|       | COLA  | SST2  | RTE   | MRPC  | avg              |
| ------------- | ----- | ----- | ----- | ----- | ---------------- |
| task-arithmetic    | 76.32 | 90.83 | 69.68 | 81.37 | 79.55|
| CABS(CSRM)         | 78.24 | 92.32 | 74.37 | 81.62 | 81.6375（+2.09） |
| CABS(SCMR)         | 78.52 | 91.97 | 73.65 | 82.6  | 81.685（+2.14）  |
| CABS(RCMS)         | 77.76 | 92.09 | 75.09 | 81.62 | 81.64（+2.09）   |
| CABS(MRSC)         | 76.89 | 92.09 | 74.73 | 83.09 | 81.7（+2.15）    |
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents an approach for model merging called CABS, which consists of two parts: 1) Conflict-Aware sparsification and 2) Balanced sparsification.

Despite its strength such as its task significance, simple but effective approach, and good experimental results, most reviewers pointed out critical concerns such as lack of theoretical justification, insufficient comparisons, and unclear scalability effectiveness.

During the rebuttal, the authors failed to address these issues sucessfully.

Considering the limitations, AC recommends rejecting this paper.